Compare commits

..

4 Commits

Author SHA1 Message Date
Xavier Roche
8b1ac57916 Pin the codec-update assertions to the URLs and to fresh content
The exact error count did not say which links failed: an engine that logged one
URL twice while another failed silently still counted three and passed. Match the
three decode errors per URL instead. The content checks only looked for the new
marker, so a fix that appended the decoded body to the old file rather than
replacing it would pass too; assert the pass-1 marker is gone. Also assert the
mode on a kept file, not just on the two rewritten ones.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 12:52:05 +02:00
Xavier Roche
814f4e7b2e Restore the 0644 mode on a decoded direct-to-disk file
Committing the decode by renaming the temp bypassed filecreate(), which is what
chmods the mirrored file to HTS_ACCESS_FILE. Under a restrictive umask a coded
binary landed 0600 while every other mirrored file stayed 0644, so a mirror
served from a web root or shared with a group silently lost read access. chmod
it at the rename, as the plain direct-to-disk path already does.

The test now crawls under umask 077 and asserts the mode, via a --file-mode
assertion in local-crawl.sh; without the chmod it fails with 0600.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 09:28:21 +02:00
Xavier Roche
0141fb7f66 Never drop the decoded body when the mirror replace fails, and probe the good path
replace_file() unlinks the destination and retries when the first rename is
refused (Windows never clobbers). If that retry also fails, the previous copy is
gone, so deleting the decoded temp as well destroyed the last copy of the body:
keep it and log where it went, the way the backup restore does. That restore is
the same remove-then-rename, so it now shares the helper.

The test grew the missing positive control: freshdisk.bin decodes on both passes
on the direct-to-disk path, so the update pass renames a decoded temp over an
existing mirror file, which no case exercised. The leftover-temporaries sweep in
local-crawl.sh now covers the .z/.u decode temps for every crawl test, not just
the three names this one asserted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 09:13:40 +02:00
Xavier Roche
11c214bcfd Keep the mirrored file when a coded re-fetch fails to decode
An --update re-fetch of a content-coded URL decoded straight into url_sav: the
live mirror was truncated (filecreateempty) before the decode was even attempted,
so a truncated gzip, a corrupt br/zstd stream, a decode-size overrun or a coding
we have no decoder for destroyed the good copy we were merely refreshing. The
uncompressed path has been rollback-safe since #522, but its backup is explicitly
skipped for coded bodies, so nothing covered them.

Decode into a temporary file instead and only commit it over the mirror once the
decode succeeded; on failure url_sav is never touched. The failure path also has
to filenote() the surviving copy, or the end-of-update purge (in old.lst, absent
from new.lst) would delete the very file the decode took care not to overwrite.

Closes #557

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 08:46:53 +02:00
4 changed files with 18 additions and 19 deletions

View File

@@ -232,10 +232,11 @@ static int disk_fallback_selftest(httrackp *opt) {
static const char body[] = "BINARY-on-disk-body-0123456789-no-trailing-nul";
const size_t body_len = sizeof(body) - 1;
/* a DOS-ified X-Save loses the path_html_utf8 prefix the reader matches on,
and gets re-rooted as a pre-3.40 relative path; fconv() only on access */
concat(save, sizeof(save), StringBuff(opt->path_html_utf8),
"example.com/blob.bin");
/* X-Save must start with path_html_utf8 so the reader resolves it verbatim
(otherwise it re-roots it as a pre-3.40 relative path); then the body we
create at fconv(save) is exactly where cache_readex looks for it. */
fconcat(save, sizeof(save), StringBuff(opt->path_html_utf8),
"example.com/blob.bin");
/* write only the header (X-In-Cache: 0); the body stays on disk */
selftest_open_for_write(&cache, opt);
@@ -1252,9 +1253,8 @@ static void corrupt_build_disk(httrackp *opt) {
memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
remove(reconcile_st_path(opt, "hts-cache/new.zip"));
remove(reconcile_st_path(opt, "hts-cache/old.zip"));
/* X-Save stored verbatim: see disk_fallback_selftest */
concat(save, sizeof(save), StringBuff(opt->path_html_utf8),
CORRUPT_ADR "/victim.bin");
fconcat(save, sizeof(save), StringBuff(opt->path_html_utf8),
CORRUPT_ADR "/victim.bin");
selftest_open_for_write(&cache, opt);
store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
"OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,

View File

@@ -1,5 +1,5 @@
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"

View File

@@ -1,5 +1,5 @@
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"

View File

@@ -35,8 +35,7 @@ chk image/avif hex:0000001C6674797061766966 "$yes"
chk image/avif hex:0000001C6674797068656963 "$no" # heic brand is not avif
chk image/heic hex:0000001C6674797068656963 "$yes"
chk image/svg+xml '<svg xmlns="x">' "$yes"
# BOM+ws skip; hex, as Windows argv cannot carry the raw BOM through the ANSI codepage
chk image/svg+xml hex:EFBBBF20203C3F786D6C2076657273696F6E3D22312E30223F3E "$yes"
chk image/svg+xml $'\xef\xbb\xbf <?xml version="1.0"?>' "$yes" # BOM+ws skip
# audio / video
chk audio/mpeg 'ID3xxx' "$yes"