Compare commits

..

3 Commits

Author SHA1 Message Date
Xavier Roche
01d64d407f Store the cache self-tests' X-Save verbatim, and feed the sniff BOM as hex
The cache self-tests built the X-Save path with fconcat(), which rewrites
every '/' to '\' on Windows -- including the path_html_utf8 prefix the
reader matches on. cache_readex then took the pre-3.40 compat branch and
re-rooted the name into a path that does not exist, so -#test=cache's
disk-fallback read and -#test=cache-corrupt's headers-only and in-memory
cases all failed on MSVC builds. The engine keeps save names in '/' form and
DOS-ifies only at the syscall boundary; the self-tests now do the same.

The sniff test passed a UTF-8 BOM as raw bytes through argv, where Windows
routes it via the ANSI codepage and U+FEFF has no mapping, so it arrived as
'?'. Use the hex: body form the self-test already accepts.

Both are self-test bugs: neither path is reachable from the crawler, which
never sniffs argv and never DOS-ifies a stored save name.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 14:00:10 +02:00
Xavier Roche
bc4b2c7b8e Keep the mirrored file when a coded re-fetch fails to decode (#560)
* Keep the mirrored file when a coded re-fetch fails to decode

An --update re-fetch of a content-coded URL decoded straight into url_sav: the
live mirror was truncated (filecreateempty) before the decode was even attempted,
so a truncated gzip, a corrupt br/zstd stream, a decode-size overrun or a coding
we have no decoder for destroyed the good copy we were merely refreshing. The
uncompressed path has been rollback-safe since #522, but its backup is explicitly
skipped for coded bodies, so nothing covered them.

Decode into a temporary file instead and only commit it over the mirror once the
decode succeeded; on failure url_sav is never touched. The failure path also has
to filenote() the surviving copy, or the end-of-update purge (in old.lst, absent
from new.lst) would delete the very file the decode took care not to overwrite.

Closes #557

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Never drop the decoded body when the mirror replace fails, and probe the good path

replace_file() unlinks the destination and retries when the first rename is
refused (Windows never clobbers). If that retry also fails, the previous copy is
gone, so deleting the decoded temp as well destroyed the last copy of the body:
keep it and log where it went, the way the backup restore does. That restore is
the same remove-then-rename, so it now shares the helper.

The test grew the missing positive control: freshdisk.bin decodes on both passes
on the direct-to-disk path, so the update pass renames a decoded temp over an
existing mirror file, which no case exercised. The leftover-temporaries sweep in
local-crawl.sh now covers the .z/.u decode temps for every crawl test, not just
the three names this one asserted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore the 0644 mode on a decoded direct-to-disk file

Committing the decode by renaming the temp bypassed filecreate(), which is what
chmods the mirrored file to HTS_ACCESS_FILE. Under a restrictive umask a coded
binary landed 0600 while every other mirrored file stayed 0644, so a mirror
served from a web root or shared with a group silently lost read access. chmod
it at the rename, as the plain direct-to-disk path already does.

The test now crawls under umask 077 and asserts the mode, via a --file-mode
assertion in local-crawl.sh; without the chmod it fails with 0600.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the codec-update assertions to the URLs and to fresh content

The exact error count did not say which links failed: an engine that logged one
URL twice while another failed silently still counted three and passed. Match the
three decode errors per URL instead. The content checks only looked for the new
marker, so a fix that appended the decoded body to the old file rather than
replacing it would pass too; assert the pass-1 marker is gone. Also assert the
mode on a kept file, not just on the two rewritten ones.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:10:28 +02:00
Xavier Roche
4fafe283b4 Store the .rc blobs LF-normalized, as .gitattributes declares (#561)
.gitattributes marks *.rc as "text eol=crlf": the blob is LF, the checkout is
CRLF. The two resource scripts added in #558 went in with CRLF blobs, so git
renormalizes them on every add and every checkout reports them as modified.
Renormalized with git add --renormalize; version.rc was already correct. The
working-tree files stay CRLF, which is what the Windows toolchain reads.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 12:42:19 +02:00
4 changed files with 19 additions and 18 deletions

View File

@@ -232,11 +232,10 @@ static int disk_fallback_selftest(httrackp *opt) {
static const char body[] = "BINARY-on-disk-body-0123456789-no-trailing-nul";
const size_t body_len = sizeof(body) - 1;
/* X-Save must start with path_html_utf8 so the reader resolves it verbatim
(otherwise it re-roots it as a pre-3.40 relative path); then the body we
create at fconv(save) is exactly where cache_readex looks for it. */
fconcat(save, sizeof(save), StringBuff(opt->path_html_utf8),
"example.com/blob.bin");
/* a DOS-ified X-Save loses the path_html_utf8 prefix the reader matches on,
and gets re-rooted as a pre-3.40 relative path; fconv() only on access */
concat(save, sizeof(save), StringBuff(opt->path_html_utf8),
"example.com/blob.bin");
/* write only the header (X-In-Cache: 0); the body stays on disk */
selftest_open_for_write(&cache, opt);
@@ -1253,8 +1252,9 @@ static void corrupt_build_disk(httrackp *opt) {
memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
remove(reconcile_st_path(opt, "hts-cache/new.zip"));
remove(reconcile_st_path(opt, "hts-cache/old.zip"));
fconcat(save, sizeof(save), StringBuff(opt->path_html_utf8),
CORRUPT_ADR "/victim.bin");
/* X-Save stored verbatim: see disk_fallback_selftest */
concat(save, sizeof(save), StringBuff(opt->path_html_utf8),
CORRUPT_ADR "/victim.bin");
selftest_open_for_write(&cache, opt);
store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
"OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,

View File

@@ -1,5 +1,5 @@
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"

View File

@@ -1,5 +1,5 @@
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"

View File

@@ -35,7 +35,8 @@ chk image/avif hex:0000001C6674797061766966 "$yes"
chk image/avif hex:0000001C6674797068656963 "$no" # heic brand is not avif
chk image/heic hex:0000001C6674797068656963 "$yes"
chk image/svg+xml '<svg xmlns="x">' "$yes"
chk image/svg+xml $'\xef\xbb\xbf <?xml version="1.0"?>' "$yes" # BOM+ws skip
# BOM+ws skip; hex, as Windows argv cannot carry the raw BOM through the ANSI codepage
chk image/svg+xml hex:EFBBBF20203C3F786D6C2076657273696F6E3D22312E30223F3E "$yes"
# audio / video
chk audio/mpeg 'ID3xxx' "$yes"