* Release 3.49.16
Version bump across configure.ac, htsglobal.h, version.rc and the AppStream
metainfo, plus the release notes for history.txt and debian/changelog.
VERSION_INFO 3:7:0 -> 3:8:0: no installed struct or exported signature moved
this cycle, so the soname stays .so.3 and Debian needs no package rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Correct the 3.49.16 release notes
The AppStream metainfo describes WebHTTrack and is read by Linux software
centres, so it now lists only WebHTTrack-visible changes: the macOS bullet and
the nine option-dialog strings (WinHTTrack-only, no hits outside lang/) are out,
the charset re-encoding is in. Drop the task-switcher claim, which webhttrack
cannot make: it is a script that launches a browser.
Fix two over-claims in history.txt. Only the trailer section after the
terminating chunk was rejected, not any chunked page. The cache aborted the
mirror on an over-long URL rather than merely refusing one it had accepted.
Cite #901 rather than the PR that closed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Installed headers do not compile under -std=c99
htssafe.h spells typeof without the underscores in three GNU-only
branches and calls POSIX strnlen from its inline bodies. __GNUC__
survives -std=c99 but neither the keyword nor the declaration does, so
htssafe.h and the three installed headers including it stop a strict-ISO
consumer.
Use __typeof__, and route the inline helpers through a private wrapper
that calls the libc strnlen wherever it is actually declared. tests/206
compiles every installed header under -std=c99 and -std=c11.
Closes#972
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test 206 asks for more POSIX than it needs
The two socket headers get -D_POSIX_C_SOURCE so getnameinfo() and
NI_NUMERICHOST resolve. At 200809L that also un-hides strnlen, so
htsnet.h and htsopt.h stopped covering the strnlen half of #972: delete
the __STRICT_ANSI__ guard from htssafe_strnlen_ and both still compile
clean. 200112L exposes the networking surface and leaves strnlen hidden,
so both catch that mutant again.
The level is probed rather than hard-coded, because only glibc is
verified here. A libc that needs 200809L still passes, and the test
reports the lost coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test 206 never expands the macros it is meant to protect
Two of the three __typeof__ sites this PR fixes sit in
htsbuff_must_be_array_, which is a macro. A macro body only reaches the
compiler where it expands, and 206 compiles a bare #include of each
header, so reverting either of those two lines to plain typeof left the
test green. Only HTS_IS_CHAR_BUFFER was covered, and then only because
htsarrays.h happens to expand it.
Add a translation unit that instantiates htsbuff_array(), htsbuff_catn()
and strcpybuff(). Reverting any of the three sites now reds 206; before
this, only one of them did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test 206 never runs the byte loop it protects
206 is -fsyntax-only throughout, so htssafe_strnlen_'s fallback was covered
for parsing and nothing else: changing its bound to `i <= maxlen` or its
result to `i - 1` left the test green. That loop is the strict-ISO
consumer's strnlen, and this header bounds copies from the wire.
Compile and run one strict-ISO TU that differentials htssafe_strnlen_
against memchr over every content/bound pair up to 8 bytes, on a poisoned
buffer so an overshot bound cannot land on a NUL and read as correct. The
gate is asserted first, or a build that fell back to libc would grade libc.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fail rather than silently drop strnlen coverage in test 206
The 200809L fallback un-hides strnlen, so it would report PASS while covering
less than it claims. No CI libc reaches it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Mark hts_record_assert_memory_failed HTS_UNUSED (#1002)
A C++ consumer of the installed headers hits -Wunused-function on this
static function: it is only called from a TypedArray macro expansion,
so a translation unit that just includes htsarrays.h leaves it unused.
g++ only runs that check on a real compile, not -fsyntax-only, which
is why 206's C loop (all -fsyntax-only) missed it; the new C++ leg
uses a real -c compile for exactly this reason. htssafe.h already
tags the same shape HTS_UNUSED.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lang/Slovak.txt line 166 read Hľdanie where the windows-1250-encoded
Slovak word is Hľadanie, missing the "a". Pre-existing typo left over
from #963's charset-declaration move, unrelated to that PR's changes.
Closes#994
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frame match keys on addr2line's " at " separator, which binutils
translates: under fr_FR.UTF-8 it reads " a" and the test fails on a
report whose frames are all present.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Nothing checks that the data directory comes from the OS path, not argv[0]
Reverting the startup lines to argv[0] alone left the whole suite green: the
part -#test=datadir cannot see is which path gets handed to
hts_resolve_datadir(). 215_engine-datadir-ospath runs a copy of the engine from
a fabricated bin/ under two lying argv[0]s, one naming a decoy tree that has
templates of its own and one a bare name as a PATH lookup leaves, and asserts
the templates the run resolved are the copy's.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fall back to src/httrack when the build needed no libtool wrapper
--disable-shared leaves the program in src/ with no .libs copy, and the test
then failed on a missing HTTRACK_BIN instead of running.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Plant a binary at the decoy path so an existence-checked argv[0] is caught
The decoy tree had templates but no program, so an engine that prefers argv[0] only when it names an existing file reached the OS path by accident and passed all three runs. Gating that preference on access(argv[0], F_OK) reproduces it: the test as merged passes that engine, this one fails it on the decoy run.
The decoy also goes first on PATH for the bare-name run, so an engine resolving argv[0] by lookup has something wrong to find.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Make six lang/*.txt files agree with their own LANGUAGE_CHARSET
Croatian, Slovak, Polski and Slovenian are stored in CP1250 and Macedonian in
CP1251, but they declared an ISO-8859 charset. html/server/*.html copies
LANGUAGE_CHARSET into its <meta> and htsserver.c decodes the POST body with it,
so every s-caron, z-caron and Cyrillic letter came out wrong in webhttrack.
Strings added since #588 followed the declaration instead of the surrounding
bytes, leaving the files mixed.
The declaration moves to the codepage the bytes are actually in, and the minority
lines are transcoded to match. Going the other way was not an option: Latin-1
cannot spell Slovenian at all, and ISO-8859-2 would lose the typographic quotes
and ellipsis Polski uses. windows-1250 and windows-1251 are already in the
engine's codepage table and already declared by Cesky, Russian, Bulgarian,
Ukrainian and Uzbek.
Svenska declared ISO-8859-2 over Latin-1 bytes, which agree on everything it uses
except a-ring; only the declaration changes there.
62_lang-integrity now decodes every file through its declared charset and rejects
both a byte the charset has no mapping for and a decode landing in the C1 range,
which is the tell of a Windows codepage wearing an ISO-8859 label. Two
deliberately mislabelled copies guard the check itself, one per branch.
Closes#963
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the charset guard honest on macOS
BSD sed does not read \r in an s/// replacement, so the C1 probe would have
been labelled ISO-8859-2r there and failed on the unknown charset name instead
of on the C1 branch it exists to cover. The extractor strips CR anyway, so the
escape bought nothing. Also default n to 1, since a grep that errors leaves it
empty and the comparison then reads as a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Name the executable's own frames in a crash report
dladdr() reports the main program's path as argv[0], which is a bare name
when the binary was found on PATH. The symbolizer's access() check then
fails and it drops the whole module, so every frame of the executable stays
a raw offset. A --disable-shared build puts the entire engine there, which
is why 105_suite-timeout failed and 80_engine-crash-symbolize skipped.
Resolve /proc/self/exe once at init and match frames against the main
program's load base, so the symbolizer gets a path it can open.
Not a link-flag problem: relinking with -rdynamic exports none of these
symbols, since -fvisibility=hidden and file-scope statics make them
STB_LOCAL and --export-dynamic only promotes global ones.
Closes#889
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Reject a clipped /proc/self/exe path
readlink() cannot report truncation, so an executable path of BT_PATH_SIZE
bytes or more came back as a prefix that find_main_object() kept. The prefix
fits copy_bounded(), and access() accepts it whenever it names a readable
directory, so the crash report gained a 1023-byte header pointing at that
directory and addr2line's "is a directory" complaint in place of the
executable's frames. A full buffer now reads as unresolved, which falls back
to argv[0] as before.
Checked by hand: an httrack installed 1024 bytes deep and run off PATH prints
the raw trace only, and 1023 still symbolizes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin test 157 to one executable block and the symbol column
A second block for the executable means library frames were misattributed to
it, and the bare symbol match could hit a path component named main.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add a weekly networked AppStream metainfo check
The per-PR lint job and tests/210 validate with --no-net so an httrack.com
outage never reds an unrelated PR, but that also hides the one failure a
software centre user would notice: a dead screenshot URL. Add a scheduled
workflow that runs appstreamcli validate without --no-net weekly, and on
demand via workflow_dispatch; it does not touch the existing --no-net gates.
Closes#971
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten the appstream-network comments to one line each
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
fuzz/Makefile.am's noinst_PROGRAMS only builds under --enable-fuzzers, so #960
left fuzz/ alone. An anchored per-binary list keeps the tracked fuzz-*.c
sources out of the pattern, unlike a fuzz-* glob that would swallow them too.
Closes#970
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A relocated install cannot find its own shared library
httrack and htsserver now link with a loader-relative rpath,
$ORIGIN/../lib (@loader_path/../lib on macOS), beside the absolute
libdir libtool records, so a tree configured for one prefix and
unpacked somewhere else finds libhttrack instead of failing before
main.
Not every loader expands that token, so configure probes it instead of
trusting the linker: it builds a small shared library and an executable
that needs it, then runs the executable from an unrelated directory
with the library path cleared. Solaris and the older BSDs get a second
attempt with -Wl,-z,origin. Nothing is added when libdir is already on
the loader's search path, which covers every distribution package.
Closes#906
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Put the configured libdir ahead of the binary-relative rpath
Emitting $ORIGIN/../lib first shadowed the configured libdir whenever
$(bindir)/../lib is a different directory, which a --libdir=$prefix/lib64
or a split-bindir layout makes routine: a stale library sitting there
won the lookup on an ordinary, non-relocated install. Naming $(libdir)
ourselves before the token puts it back in front, so the token only
answers once the configured libdir is gone.
The system-libdir gate no longer reads libtool's
sys_lib_dlsearch_path_spec, which is only augmented when /etc/ld.so.conf
exists and would let a distribution libdir through in a sysroot or a
minimal container; it pattern-matches /lib, /lib64, /usr/lib and
/usr/lib64 and their subdirectories instead. /usr/local is no longer
treated as a system libdir, so a default configure now enables the
rpath and the tests exercise it.
Cross compiling no longer enables the feature unprobed. It defaults to
off with --enable-origin-rpath as the override, and that override takes
the -Wl,-z,origin spelling Solaris and the older BSDs need, since
nothing can run to tell the two apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the rpath ordering and the cases that must not get one
195_install-relocate.test asserted only that the token was present, so
it could not see it landing ahead of the configured libdir, and it
skipped outright wherever the rpath was suppressed. It now reads the
rpath entries in order and requires the libdir to come first, requires
proxytrack to carry no token at all, and in a suppressed build asserts
the absence rather than skipping.
Proving the relocated copy started was also not enough: a libhttrack
elsewhere on the loader path would have answered. The copy's own library
is now moved away and the binary has to stop working, which replaces the
ldconfig guard (found by `command -v ldconfig`, which fails for a user
without /sbin on PATH, and matching any soname including a .so.2 that
could never satisfy .so.3) and works on macOS, where there is no ldd.
Staging through DESTDIR rather than overriding the paths keeps libtool
from rewriting the build tree while the rest of the suite runs.
196_install-rpath-gates.test configures a system prefix, a multiarch
libdir, --disable-origin-rpath and --disable-shared and asserts each
one emits no rpath, with a private prefix first so a probe that never
succeeds cannot make the rest vacuous. One shared cache keeps the five
runs to a few seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Suppress the rpath for the 32-bit multilib libdirs too
/lib32, /libx32 and their /usr counterparts are on the loader path, and
libtool's sys_lib_dlsearch_path_spec used to cover them, so replacing
that read with an enumerated list handed a 32-bit distribution build an
rpath it never had and lintian's binary-or-shlib-defines-rpath with it.
Still enumerated rather than globbed as /lib*, which would swallow
/libfoo, /usr/library and /usr/libexec.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Configure a symlink farm, not the already-configured srcdir
196_install-rpath-gates.test configures out of $abs_top_srcdir, which
automake rejects when a config.status sits there. CI builds in-tree, so
every leg that runs the full suite failed; the out-of-tree local runs and
the msan leg (restricted TESTS) never saw it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Refresh mergeability after master moved
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Turn the binary-relative rpath off on Darwin, and read Mach-O with otool
dyld consults LC_RPATH only for an @rpath/ load path, and libtool stamps
libhttrack with an absolute -install_name, so the @loader_path entry the
Darwin arm emitted was never read. The copied tree stayed just as broken.
Gate the feature off there instead of shipping a dead load command.
That left the two bugs the macOS leg was actually failing on. Test 195
picked its object dumper by whatever happened to be installed, and an ELF
reader handed a Mach-O prints no rpath rather than an error, so a present
rpath read as an absent one; it now picks by platform. Test 196's nested
configures inherit none of the parent's CPPFLAGS/LDFLAGS, so brew's
keg-only openssl turned the default --enable-https into a hard configure
error. The gate has nothing to do with TLS, so they ask for =auto.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A stack-overflow SIGSEGV kills httrack with no diagnostic at all
sig_fatal went in with plain signal(), so it ran on the faulting thread's
own stack. When the fault is stack exhaustion there is no room left for a
signal frame, the kernel falls back to the default action, and the process
dies at 139 having printed nothing. Register the four fatal signals through
sigaction() with SA_ONSTACK, over a 64 kB alternate stack allocated at
startup, and prime backtrace() at init so glibc's lazy libgcc_s dlopen()
does not happen inside the handler.
Only the main thread is covered: htsbacktrace.c is linked into the httrack
binary rather than into libhttrack, so the engine's worker threads cannot
reach the installer. They behave as before, SA_ONSTACK being ignored where
no alternate stack exists.
Closes#866
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test 180 could not tell a stack overflow from an ordinary fault
Three mutants passed it: a crash_stack() that only dereferences NULL, a
backtrace() truncated to one frame, and a backtrace that returns nothing at
all, since the frame assertion accepted "No stack trace available" as a
match. The reports now have to differ: a runaway recursion repeats the same
return address ~250 times where the segv control repeats none, and the
control asserts the threshold still separates them.
Also stop skipping on macOS and the BSDs, where sigaltstack() exists and the
fix is live; only the trace text is gated on Linux. Each assertion now names
what failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* The alt stack leaked, and took over the sanitizer's
LeakSanitizer called the 64 kB alt stack a direct leak: nothing retains the
pointer once sigaltstack() owns it. Map it instead of allocating it, which
also keeps the handler's own stack out of the heap whose corruption it may
be reporting.
The leak was the tell for the worse bug. ASan already has an alt stack
installed by the time signal_handlers() runs, 32 kB of it, and the old size
floor rejected it and installed ours over the top, moving ASan's handlers
onto our mapping. Defer to any alt stack already installed regardless of
size: whoever put it there sized it for their own handler, and ours needs
about 10 kB of the 32 that the smallest of them offers.
Test 181 pins both directions through an LD_PRELOAD observer that compares
the mapping the process ends up with against the one it was handed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Refresh mergeability after master moved
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Installed dev headers do not compile standalone
Three headers in DevIncludes_DATA fail on their own from $(includedir)/httrack
once HTS_INTERNAL_BYTECODE is defined: htswrap.h includes coucal.h, which is
never installed and which the file does not use; htsmodules.h uses HTSEXT_API
without htsglobal.h; htsdefines.h uses size_t without <stddef.h>.
205_install-headers.test installs the list into a temp dir and compiles a
one-line TU per header in both preprocessor states.
Closes#943
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Split CC so the test runs on the linux-i386 leg
That job configures with CC="gcc -m32", and looking the whole string up as one
executable made the test exit 77. A skip reads as success, so the one leg with a
different data model was checking nothing. Same for CC="ccache gcc".
The compiler now also has to compile a bare <stdio.h> first, or a box without
gcc-multilib blames every installed header instead of reporting its own toolchain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pass the build's CPPFLAGS to the installed-header compile
htsbasenet.h includes <openssl/ssl.h> under HTS_USEOPENSSL, so the test needs
OpenSSL's include path like any consumer does. Linux resolves it from
/usr/include, which is why only the macOS leg failed: brew's openssl@3 is
keg-only and reachable only through the CPPFLAGS given to configure.
Reproduced by hiding /usr/include/openssl behind a -nostdinc mirror and exposing
it from a private prefix, which reds htsnet.h and htsopt.h exactly as CI did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Tooltip JavaScript breaks on any translation containing an apostrophe
The onMouseOver handlers interpolate through ${html:...}, which escapes
for HTML text. The browser decodes an attribute value before compiling
it as JavaScript, so ' arrives as a bare quote and closes the string
literal; a translation carrying a double quote ends the attribute
outright. Add a js: template mode escaping for both layers and move all
304 handler sites onto it.
Closes#864
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Close the step2.html and title-attribute gaps in the js: escaping test
The static check could not match ${html:html:...}, the very shape it was
added to catch, and the runtime probe only fetched option1.html: reverting
all eight step2.html handlers left the test green. Nothing asserted that a
title= attribute stays on html: outside option1.html either.
Walk every server template instead, pairing each ${...} with the attribute
holding it, and fetch step2.html as well. Also feed the fixture a newline
and a tab, which had no coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Emit the backslash as \x5c so a DBCS trail byte cannot swallow it
shift-jis, BIG5 and gb2312 all accept 0x5c as a trail byte, and those are
the declared charsets of three shipped lang files, so the browser really
does run a DBCS decoder over the page. An orphan lead byte followed by the
emitted \\ pair decodes as one glyph plus a stray backslash, which escapes
whatever comes next. Emitting every escape as a \xNN group leaves the
decoder nothing to pair with: the only bytes cat_js_escaped adds are '\',
'x' and hex digits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Nine lang.def strings are untranslated in 26 of the 30 language files
LANG_F15b, LANG_I6c, LANG_I23c/d/e, LANG_I35c and LANG_I43c/d/e were
translated only in English, Francais, Dansk and Portugues-Brasil, so the
Windows GUI's option dialogs fell back to English everywhere else. Fill
them in for the other 26 files, each encoded in that file's declared
LANGUAGE_CHARSET and inserted where English.txt orders it.
Svenska.txt declares ISO-8859-2 but holds Latin-1, and Chinese-BIG5.txt
needs cp950 rather than strict big5. Romanian.txt and Slovenian.txt
declare ISO-8859-1, which cannot spell either language, so both stay
unaccented ASCII the way #862 left them.
Test 62 waived these nine msgids by name and fails once a waiver is no
longer needed, so drop them from that list: it reports 231 untranslated
msgids against master.
Closes#863
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Romanian: restore the two circumflexes ISO-8859-1 can spell
Only the-breve, s-comma and t-comma are unrepresentable in the declared
charset; a- and i-circumflex are 0xE2 and 0xEE, and the rest of the file
already spells them that way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Icon=httrack never resolves through the share/pixmaps fallback
share/pixmaps only ever held the sized names, so the bare-basename lookup a
desktop falls back on when no icon theme is installed never matched. Install
httrack.xpm (a 32x32 copy) alongside them rather than renaming one: Debian
globs the whole directory and a rename would strand the old names on upgrade.
Closes#932
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the pixmaps fallback to the icon, not to any 32x32 image
The geometry check passed on a solid-red square, so compare the installed
fallback against the installed 32x32 copy, declaration line aside. Also drop
the banned pipe into grep -q while here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The suite watchdog may kill its own taskkill, and an empty category reads as a failing test
ci_suite_heartbeat runs in a subshell of the process it targets, and kill_tree
on Windows is taskkill /F /T /PID, so taskkill is a grandchild of its own
target. A leaves-first /T would reap the watchdog before reaching the root: the
annotation is printed, nothing dies, and the step runs on to the 45-minute
cancel with no log. Signal the target directly first, then through the tree, so
the kill no longer depends on taskkill outliving its own ancestors.
The driver's category globs also expanded to themselves when they matched
nothing, handing test-timeout.sh a literal pattern that exits 127 and lands in
the tally as a failing test named after the glob. Categories now carry a label
and expand under nullglob; an empty one names itself and stops the suite before
any test runs.
Closes#952Closes#953
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the empty-category gate cover the one category that names a single test
runnable:00_runnable.test carries no metacharacter, so nullglob cannot empty it
and the gate never fired for it: the name reached test-timeout.sh unexpanded and
was counted as a test failing 127, exactly what the gate exists to prevent. Give
it a pattern like every other category, and empty it in the driver test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools/Info.plist is the one that matters: configure generates it from
Info.plist.in, so a stray "git add -A" would commit a snapshot that
shadows it and pins CFBundleShortVersionString to whatever version built
it (#884, one directory over).
Closes#905
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
#897 pinned the metainfo version against htsglobal.h, but nothing checked
the file was valid AppStream: a broken one passes the suite and a software
centre then drops WebHTTrack without saying so.
The lint job gains an appstreamcli step beside the .vcxproj well-formedness
pass, and 210_appstream-metainfo.test runs the same check under make check,
skipping when appstreamcli is absent. --no-net keeps the gate off the
screenshot host's reachability; errors and warnings fail, hints do not.
The test validates a copy with <id> stripped as a positive control, so a
validator that never fails cannot make it vacuous.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
windows-build.yml was the only workflow without a concurrency group, so
every push left the runs before it going. Match ci.yml and codeql.yml.
Closes#942
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Derive the bundle's minimum macOS from its payload
tools/Info.plist.in hardcoded LSMinimumSystemVersion 11.0, but nothing
built the payload for 11.0: with no MACOSX_DEPLOYMENT_TARGET set, clang
on the macos-15 runner targets the host, and all nine Mach-Os in the
released DMG report minos 15.0. Finder believes the plist and launches
on macOS 11 through 14, then dyld refuses the binaries.
macos-app.sh now takes the highest minos in the assembled bundle and
writes it to LSMinimumSystemVersion before the ad-hoc signature seals
Info.plist, so the two cannot drift again. The floor has to be the
maximum rather than the main executable's: five of those nine are
Homebrew bottles whose minos was fixed when the bottle was built, so a
deployment target on our own compile would not move them.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the declared minimum macOS instead of rewriting it
Review pushed back on deriving LSMinimumSystemVersion from the payload:
it makes the shipped minimum follow whatever runner image built the
bundle, so bumping a CI label would move HTTrack's advertised floor with
nobody deciding it. Declare 15.0, which is what the binaries need today,
and fail the build when the payload disagrees.
macho_floor also swallowed an otool failure: the call sat non-final in a
pipeline, so a Mach-O otool could not map contributed nothing and the
floor came from the rest, which is the same wrong-floor bug this fixes.
deps() already guards its otool the same way.
Dropping the plist rewrite drops plist_set with it, and with it the
awk-on-XML edge cases and the read-back check that was only meaningful
while the derived value differed from the template's.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the comments the review flagged
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The CI watchdog deadlines stretch under the starvation they exist to catch
Every deadline in the test harness counts poll iterations and assumes each
costs exactly the tick it asked for. Under the CPU and fork starvation that
wedges a runner, an iteration costs several times that, so the budget arrives
late in wall-clock terms or never: the 45-minute step timeout cancels first,
and a cancelled step keeps neither its log nor its artifacts. That is why the
wedged Windows jobs in #795 name no test.
Measure $SECONDS instead, in test-timeout.sh and in run_with_timeout,
wait_bounded and reap_bounded, which restores the wall-clock contract
run_with_timeout's comment already described.
The guards are checked against a reverted copy via a slow-sleep shim: with
each poll stretched fourfold, the counting code takes 44s to honour a 1s
budget and 16s a 3s one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Compare the deadlines strictly, and make the guards kill their mutants
$SECONDS is floored, so a reading equal to the budget can be a fraction
under it: -ge fired up to a second early, which kills healthy work. Compare
strictly instead, at the cost of at most a second late.
The 58_watchdog guard also passed on a watchdog that killed everything on
sight, because one stretched reap poll lifted the elapsed time over its lower
bound, and its control raced (the child exited before the first poll). Give
the control a command that outlives several polls, and widen the timing bound
to sit a full poll quantum either side of 8s healthy and 16s counted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools/macos-release.sh signs every Mach-O with the hardened runtime, notarizes and staples the app, packs a DMG and notarizes that too. A new macos-release workflow runs it on a tag or on demand, with a Developer ID imported into a throwaway keychain from repo secrets. Signing is per-Mach-O rather than --deep, which reaches nested code but never applies the runtime, and notarytool's status line is read rather than its exit code.
The bundle stub is now a Mach-O (tools/httrack-launcher.c) rather than a shell script, which Apple treats as a resource and TCC cannot attribute a prompt to. The Mach-O walk and the Info.plist reader are shared with macos-app.sh via tools/macos-bundle.sh, and ci.yml's macos-app job runs the same script ad-hoc on every PR.
Closes#901
* The Windows suite driver is 140 lines of shell inlined in YAML
Move the "Run the engine test suite" step body to tests/ci-windows-suite.sh,
where shellcheck and shfmt reach it and it can be run by hand. The two
deliberate word-splits in the skip-set compare needed a directive; nothing
else changed, verified by diffing the shfmt-normalized old body against the
new file.
ci_annotate and ci_suite_heartbeat move with it, out of the test library
every test sources on every platform. 171_watchdog-heartbeat.test sources
the driver, which returns early unless run directly.
Closes#948
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test the driver on this platform, and ask the shell whether it was sourced
The guard read "${BASH_SOURCE[0]}" = "$0", which is only false by accident of
what the caller put in $0: bash -c '. "$0"' <driver> makes the two equal and
falls through into the suite. Ask the shell instead.
172_ci-windows-driver.test pins that case, the bindir contract and the loop's
accounting against a stub bindir, none of which a Windows-only leg proves
before merge.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Silence two shellcheck findings in the new driver test
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the driver test survive distcheck's read-only srcdir
cp carries the source's mode over, so under distcheck the neutered testlib
copy came out read-only and the append failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Bundle the non-system dylibs, or a downloaded HTTrack.app cannot launch
src/Makefile.am links $(OPENSSL_LIBS) into libhttrack, which on a build
machine resolves to Homebrew. The bundle check only rejected the staging
prefix, so /opt/homebrew paths sailed through: fine for someone who ran
brew install, fatal for anyone who mounts a DMG.
macos-app.sh now copies the transitive closure of non-system dylibs into
Contents/Frameworks, rewrites the load commands to @rpath, and adds a
depth-correct @loader_path rpath to every Mach-O. install_name_tool
invalidates the signature and arm64 refuses to run an unsigned binary, so
each rewritten file is re-signed ad-hoc.
The check is now that every load command resolves to /usr/lib,
/System/Library, or a file present in Frameworks, and it fails if it
scanned no Mach-O at all. Because a static check cannot fail while the
loader is quietly falling back to Homebrew, CI also hides Homebrew's
openssl@3 and re-runs the smoke against the moved bundle.
Part of #901.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Give the linker header room for the @rpath rewrite
install_name_tool refuses when the new load commands do not fit the
existing header: "changing install names or rpaths can't be redone".
The bundle build now passes -Wl,-headerpad_max_install_names, and the
script says so when the rewrite fails rather than surfacing the
toolchain's own message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix two bundling bugs the review found, and three checks that could not fail
machos() ended with a while loop, so its status was the last iteration's
file(1) test. The bundle always holds shell scripts, and find walks APFS in
directory-hash order, so whenever a non-Mach-O came last the script died
under set -e with no diagnostic and a half-populated Frameworks. Green CI
here was luck, not evidence.
Two dependencies sharing a basename were both copied in the same pass, the
second clobbering the first, and relink() then pointed both references at
the one survivor. Frameworks is flat and cannot express the difference, so
this now fails with both paths named.
The checks that could not fail:
- machos() prefiltered on mode and suffix, and the same list drove the copy,
the relink and the audit, so a non-executable Mach-O was unbundled AND
unchecked. It now enumerates every file.
- deps() piped otool into awk, so a failing otool yielded no dependencies and
the binary was recorded as clean. Its status is now checked. Reading the
indented lines rather than tail -n +2 also stops a universal binary's
per-slice headers from parsing as dependencies.
- The CI probe hid only the openssl@3 opt symlink, which a source-built
bottle does not reference, and asserted nothing about the bundle first, so
an empty Frameworks passed it. It now asserts libssl is bundled, then hides
the kegs of every library in the closure.
- Signing verified the bundle after codesign --force --deep had already
re-signed it, repairing the damage it was meant to catch. Per-file
signatures are now verified first.
-Wl,-headerpad_max_install_names moves from the CI job to configure.ac: it
is a precondition of the macos-app target, not a CI preference, and anyone
following the standalone recipe in Makefile.am:22 hit a rebuild-only
failure that ld64's default header slack made intermittent.
The toolchain gate is now unconditional, so a non-Darwin host refuses to
build rather than emitting a bundle nothing verified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* One library reached by two paths is not a basename collision
The bundle depends on both /opt/homebrew/opt/openssl@3/lib/libcrypto.3.dylib
and the Cellar path behind that symlink, so comparing the dependency strings
called Homebrew's own layout a collision and refused to build.
Compare the paths with the directory resolved instead. Two names for one
file now dedupe; two different files under one name still fail, which is the
case that would silently clobber.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Windows job wedges every few dozen runs: the test step never completes, the runner is lost, and the log blob and the `if: always()` artifact uploads go with it. So the wedging test has never been named.
A watchdog now runs beside the suite and ends the step before the runner dies, because a step that fails on its own terms keeps its log and still runs the uploads. It never triggers on elapsed time, since a healthy test and a wedged one look identical by the clock. It watches the progress log instead, where every outcome writes a line, the per-test timeout included: 900 seconds without one means that timeout did not fire, which is the wedge. On the way it names the test in flight as annotations, which is live progress rather than evidence.
That distinction is measured, not assumed. A throwaway workflow emitted notices and then died four ways: a step cancelled by its own timeout keeps its log and every annotation, on Linux and on Windows, from a background subshell as well as the foreground; a runner killed mid-step keeps neither, dropping notices that had been on the wire for twenty seconds.
Refs #795 rather than closing it: this ends the silence, it does not fix the leak behind it. Leftover coverage gap in #949.
Bump src/coucal from 5d2a633 to 0d36322, taking the two commits #945
deliberately skipped.
#31 replaced the log-level #if 0 cascade with COUCAL_LOG_LEVEL, but
defined all five level emitters unconditionally, so the unused ones
warned under -Wall for any consumer lacking coucal's own
-Wno-unused-function. That was the reason #945 pinned behind head.
xroche/coucal#33 tags them with an unused attribute instead and drops
the exemption from coucal's Makefile, so the warnings are gone at the
source rather than masked, and a new upstream CI leg compiles coucal.c
with consumer flags at every verbosity.
The default verbosity is still info, so the compiled-out debug and
trace that #941 measured stay compiled out: the probe from that PR
still reports 0 print handler calls per 20000 inserts. Our build is
clean of compiler diagnostics under gcc and clang. MSVC is unaffected
either way, since it builds at /W3 and the equivalent C4505 is level 4.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* proxytrack: bound the .arc reader and stop the writer trusting a size it has no bytes for
The reader hands back an element carrying a declared size with adr == NULL when
it could not fetch the body, and the .arc writer took the size at face value:
fwrite(NULL, 1, size) faulted inside libc (#931). Both writers now take the
body from a helper that answers 0 when there is nothing to write, and the
record's own length, the Content-length header and the md5 follow it, so what
is written declares what it holds.
A new fuzz-arc harness drives the reader the way --convert does (#929), and
found the rest in seven executions: PT_Delete never freed the indexes it owns,
PT_Index_Delete__Arc never freed its hashtable, and a record could declare a
two-gigabyte body that the reader allocated before the short read failed.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* proxytrack: commit the merged index slot only once the array holds it
PT_IndexMerge() counted the new slot before growing the array, and assigned
realloc's result straight onto indexes->index: a failed allocation dropped the
array it already had and left index_size counting an entry that was never
stored. Harmless while nothing walked the array; PT_Delete() now does. The
array holds pointers, so size it as such rather than as whole PT_Index structs.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* proxytrack: trim the comments added by the .arc hardening
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* proxytrack: initialise the .ndx mutex, keep unsizeable archives loading, and give the bound a positive control
PT_LoadCache__Old() never called MutexInit(), which was survivable while
PT_Index_Delete__Old() was unreachable; PT_Delete() now runs it on every exit.
The readers lock the same handle, so on Windows the .ndx path was serving
unsynchronised.
A .arc past LONG_MAX cannot be sized by a 32-bit ftell, and refusing the whole
archive lost the records that used to load; the bound just stops constraining.
Test 164 asserted only refusals, so a reader that refused every body passed it,
as did an off-by-one on the last record. It now converts an archive whose body
ends on the last byte of the file, and checks the zip writer's own size header.
The truncated.arc seed ran out of file before reaching the bound it is named
for.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Bump src/coucal from a0a9e49 to 5d2a633, clearing the one warning #941
knowingly landed: 7a8198d const-qualified the_empty_string but still
returned it through a plain cast to coucal_key (void*), which httrack
compiles with -Wcast-qual. Upstream keeps the const and routes the
singleton through uintptr_t, so it stays in rodata, and adds
-Wcast-qual to its own mandatory flags so the class cannot return.
Pinned at 5d2a633 rather than upstream head. The next commit (#31)
replaces the log-level #if cascade with COUCAL_LOG_LEVEL and defines
all five level functions unconditionally, which warns twice under
-Wall for any consumer without coucal's own -Wno-unused-function.
Filed as xroche/coucal#32; head follows once that is resolved.
The default level is still info, so the compiled-out debug and trace
that #941 measured stay compiled out: the same probe reports 0 print
handler calls per 20000 inserts at both 5d2a633 and head.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bump src/coucal from 93ec411 to a0a9e49, five upstream fixes.
The one that reaches the engine is the logging change. coucal_trace()
in coucal_add_item_() passes coucal_print_key() as an argument, and the
compiled-out macro expanded to an ordinary variadic call, so the
argument still ran. htshash.c installs key_adrfil_debug_print on
hash->adrfil and hash->former_adrfil, where it snprintf()s the full URL
into a scratch buffer, so every insert into the dedup tables paid for a
URL format that was then discarded. Measured on a probe mirroring
htshash.c's setup: 13262 handler calls per 20000 inserts, now 0, with
the table statistics unchanged.
The other four fixes harden coucal without reaching our call sites: the
custom key free at destruction (our dup handler is an identity pointer
copy and the free handler is empty), the mid-walk delete enumeration
skip (all four enum loops in htsback.c are read-only), the pool-aliased
key use-after-free (no call site passes an item name back as a key),
and the coucal_new() shift width (every call site passes 0).
No ABI change: coucal.h is not installed, struct_coucal is opaque,
struct_coucal_enum is byte-identical, and the 36 coucal symbols
libhttrack exports are unchanged.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
#933 and #934 merged minutes apart and both landed a test numbered 160, so the icon-theme one becomes 163. Only `tests-list.mk` referred to it, and the Windows job picks tests by topic word rather than number, so its coverage is unchanged.
The cache-hdrbounds test also sent stderr to `/dev/null` to hide an expected warning, which threw away the self-test's own diagnostics with it: a failure printed `cache-hdrbounds: FAIL` and nothing about why. stderr now goes to a file both failure paths report.
Four places built or consumed the cache key, each assuming a different maximum URL length. A URL long enough to fill a `lien_back` field is legal and reachable off the wire, so one of them aborted the crawl where it should have missed the cache, and the index load read entry names into a buffer minizip can fill without a terminator.
All four now size off one bound, and the key is built all-or-nothing with the existing `slcatprintfbuff`: too long to store drops the entry with a warning, too long to look up is a miss. Clipping is never right here, because a clipped key is a valid key for some other URL, and that is a cache hit on the wrong content. The new `cache-urlbounds` self-test (`tests/162`) stores at the cap and pins that neither a twin differing only in its last byte nor a decoy sitting on a clip point can be served in its place.
Closes#935Closes#936
* Bound the cache header block instead of trusting the field caps
ZIP_FIELD_STRING and its integer siblings sprintf'd into a fixed 8192-byte
block with no bound, in both the engine cache writer (cache_add) and
ProxyTrack's new.zip writer. The values are remote-controlled -- ETag,
Location, Content-Disposition, the URL itself -- and in cache_add the caps
they are declared with sum past 8192, so the writer could overrun its own
stack buffer.
Route every field through slcatprintfbuff(), a new all-or-nothing bounded
append: a field that does not fit is dropped whole, since a clipped one reads
back as a valid shorter value. X-Save moves ahead of X-Addr/X-Fil so the one
field the reader consumes is not the first casualty of a full block.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Merge origin/master into fix-841-zip-field-bounds
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the bounded ZIP_FIELD_STRING lost in the merge
The merge commit picked up a mutation-testing revert of this macro from the
shared worktree, putting the unbounded sprintf back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the header-bounds test kill the mutants it was walking past
A test audit built seven mutant writers against the new self-test; five
passed. Restoring the legacy field order passed while silently dropping
X-Save, so the reorder this PR relies on was asserted by nothing. An
early-return writer passed vacuously, the check landing on the control
entry because nothing pinned which entry it read. A truncated Location
passed because any non-empty prefix was accepted, and the size bound was
a literal 8192 decoupled from the buffer it was meant to track.
The block is now identified by a field only the maxed entry carries, the
bound comes from a shared CACHE_HEADERS_SIZE, X-Save must survive, and a
still-present X-Fil reports that the entry stopped filling the block
rather than passing quietly. The wrapper consults httrack's exit status,
so an abort after the verdict is no longer a pass.
Adds an .arc to .zip round trip for ProxyTrack's writer, which had no
runtime coverage: every --convert in the suite writes .arc.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tools/HTTrack.icns from the brand master, declare it in Info.plist.in,
copy it into the bundle, and check the plist and the payload agree.
Closes#900
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The application icon is still the pre-brand bitmap set
Replace the 16/32/48 PNGs and the .xpm fallbacks with the HT monogram
generated from the same Jost* master as the masthead wordmark, and extend
the hicolor theme with 64, 128, 256 and a scalable SVG.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the icons on what install and dist emit, not on Makefile.am text
Six mutants survived the first version: an emptied EXTRA_DIST, an empty *dir
variable (automake's install rule exits 0 when it is), a _DATA glob with the
wrong extension, a _DATA entry for a directory the tree does not have, a
missing apps context subdirectory, and a size dropped from install entirely.
All six now fail, and the PNGs are indexed rather than RGBA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* proxytrack cannot re-read the .arc it writes
The version block's declared length counted the blank line closing it, so
the reader consumed the first record's separator and every entry was
skipped: a second --convert over proxytrack's own output loaded nothing.
The bytes on disk are unchanged; only the declared length shrinks by one,
which an older proxytrack reads too. The reader now stops on the last of
the newlines closing the version block, so archives already written the
old way still load.
Closes#834
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bound the version block newline scan and keep rejecting truncated archives
Review of the first commit found two regressions of its own: the newline
run was scanned to its end, so an archive padded with a gigabyte of them
cost a gigabyte of reads where master stopped after two, and tolerating
EOF there turned a length running past the end of the file into a silent
empty load. At most two newlines are read now, and a version block that
does not end on one is rejected as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the version block length against a compensating extra newline
A writer that emits the blank line and still counts it round-trips, so
every assertion passed while the length stayed a byte too long. The
declared block must not end on a blank line either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The tagline no longer sits against the wordmark's baseline
The old GIF was cropped from the cap tops to the baseline, so its box edges were
the letters. The SVG's box is the true ink box, which in Jost also holds the k's
ascender above the caps and the round letters' overshoot below the baseline, and
that shows up as a band of field colour between the wordmark and the tagline.
Neither band can be cropped out of the artwork without cutting ink, so the
masthead takes them back optically. Measured against the old rendering, the
baseline-to-tagline distance is identical and the cap tops land within a third
of a pixel.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cut the comments back to one line each
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Say what the margins actually do
The comment claimed both bands were trimmed. The lower one is, in full; the
upper is trimmed only by what the old bitmap did not already carry, which is why
1.7px is not the 3.56px the artwork measures. Derivation recorded beside the
generators.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ProxyTrack dumps a coucal hashtable stats line on every WebDAV request
coucal logs a per-table statistics summary when a table is deleted, and
with no handler installed it prints that line itself, prefixed with the
table's address. ProxyTrack builds and drops one table per WebDAV
enumeration, so every unauthenticated PROPFIND put one on whatever the
service redirects its output to, heap pointer included.
httrack and htsserver were already covered by hts_init(), which installs
a global coucal handler that drops info-level messages unless HTS_LOG is
set (#416). ProxyTrack does not link libhttrack: it compiles coucal
itself and never calls hts_init(), so it was the last binary on coucal's
built-in sink. It now installs its own handler. Critical and warning go
through proxytrack's log, everything below is dropped unless HTS_LOG is
set, and the summaries then come back without the address.
Unnaming the tables would not have fixed it. coucal_delete() logs the
summary for every table, named or not; the name only decorates the
message.
Closes#918
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Whitelist proxytrack's request-time output instead of naming two absent strings
The absence check pinned two literals, so a reworded leak or one whose
pointer was not at column 0 walked through it. Assert instead that every
line the process writes once serving is an access-log line, which is the
property, and pin the routed form under HTS_LOG whole so the address
cannot creep back between the severity and the message.
Also say why the handler passes "debug" rather than the DEBUG macro: the
macro is NULL outside a debug build, which would quietly make HTS_LOG a
no-op.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Order the whitelist against a request's teardown, not just its access log
proxytrack writes the access line before send(), so waiting on it ordered
nothing that a teardown or keep-alive path writes afterwards: a leak
delayed 400ms past the response survived the check. Send both PROPFINDs
over one connection, since the keep-alive loop does not read the second
request until the first one's teardown has run, and read the capture only
once proxytrack is gone and the pty drainer has marked it complete.
Draining alone was not enough. SIGTERM cuts the work short rather than
truncating a buffer, so a still-pending write is never made at all and
there is nothing left to flush; the ordering is what catches it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A FIFO passes "test -x", so the #920 check ran it: bash gets EACCES from
execve, falls back to reading the file for a shebang, and blocks in open()
with no writer. AS_EXECUTABLE_P is autoconf's own "test -f && test -x", and
AC_PATH_PROGS on the line above already applied it to the PATH search, so the
override path was simply using the weaker predicate.
A regular executable that never returns stays uncovered: no portable timeout
is worth it, and CC= pointing at the same wrapper hangs stock AC_PROG_CC too.
Test 151 gains the FIFO case, and its run() is capped so a regression fails
instead of wedging "make check" with no log.
Closes#922
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* http_xfread1's reserve-only read mode has no caller
The `bufl == -2` branch of `http_xfread1()` allocates the line buffer and
returns without reading. Nothing has ever called it: no call site in the
tree passes -2, and scanning all 2649 revisions in this repository for
`xfread1(` call sites turns up 24 distinct lines, none of them -2. The
branch arrived with the 3.20.2 import commented "force reserve", so it was
probably meant for a preallocate-then-fill pattern that never landed.
Naming it `HTS_XFREAD_RESERVE` in #919 made it read as a supported mode.
No external caller is possible either: `htslib.h` is not installed and the
symbol is hidden, so this is not an API change.
Equivalence checked against the object code. `htsback.o` and
`htsselftest.o` disassemble identically; in `htslib.o` every function
except `http_xfread1` differs only in the `__LINE__` values `htssafe.h`
bakes in, shifted by the seven deleted lines. No new test: nothing changes
for any input a caller can produce, and `01_engine-xfread` plus the chunked
tests still pass.
Closes#923
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Say that any non-positive bufl is line mode, not just the two named
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The masthead wordmark is a 400x34 GIF that blurs on any hi-DPI screen
Replaces it with an outlined SVG across the 38 documentation and WebHTTrack
pages that carry it. The original was set in Futura, so the lockup was refitted
in Jost*, the closest free Futura revival, taking weight from the measured stem
thickness, size from the cap heights and tracking by least squares against the
glyph positions in the old bitmap.
tests/82 now asserts that every image a GUI page names is actually served.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Point the shared chrome generator at the new wordmark
The masthead of the 13 generated pages comes from tools/doc-chrome.py, so
editing the pages alone left the generator disagreeing with its own output and
--check red.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* proxytrack: drop the leftover debug traces on the WebDAV path
Two fprintf(stderr) calls in the PROPFIND path shipped by accident: one dumped
the client-supplied request body, the other the whole generated multistatus
response. Both ran on every PROPFIND with no authentication in front of them,
so any client could write bytes of its choosing into the operator's stderr.
The body is never parsed and the response is derivable from the index, so
neither trace has diagnostic value worth keeping behind a debug level.
Closes#911
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: make the #911 leak check see stdout and the property, not two literals
The absence checks pinned 'DEBUG: DAV-DATA' and '^RESPONSE:', which four
re-added variants walk straight past: a trace without the hyphen, one with no
marker at all, one prefixed so the '^' misses, and one on stdout. The stdout
case is the worst of them: redirected to a file, proxytrack's stdout is fully
buffered and SIGTERM never flushes it, so the leak never reached the log the
test reads.
Give proxytrack a pty instead of a file, so libc line-buffers its output on
Linux and macOS alike, and assert the property: a PROPFIND carries a canary the
index cannot produce, and neither the canary nor a distinctive string from the
generated response may appear in what proxytrack wrote. The two literals stay as
names for the specific regression. The liveness guard now requires a PROPFIND
answered 207, since a depth-rejected one is logged 403 by the shared reply path
without ever reaching the deleted code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Bound the ProxyTrack DAV item buffer against an amplified PROPFIND path
proxytrack_add_DAV_Item() reserved a fixed 1024 bytes and then sprintf'd into
it unbounded. The request path lands in the response twice, once as the href
and once as the displayname, and escapexml() turns each '&' into '&', so
an unauthenticated PROPFIND of roughly 900 ampersands writes about 9000 bytes
off the end of the heap block. No cache entry and no Depth: 1 are needed.
Replace the hand-sized reserve with StringSprintf(), which measures the
formatted output and grows the String to fit, and convert the sibling sprintf
sites in the same file so no unbounded write into a String is left to
re-audit. Sizing beats clipping here: the String already owns a growable
buffer, so nothing has to be dropped.
Closes#836
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bound StringSprintf's pre-C99 retry, and trim the review findings
A genuine vsnprintf conversion error returns -1 just as pre-C99 msvcrt does
for a short buffer, so the doubling search had no way to tell them apart and
grew until realloc aborted. Unreachable from these format strings, which use
only %s and %d, but the helper lives in a shared header and will get more
callers. Cap the search and empty the String past it.
Also: the count assertion piped into wc under pipefail, so a zero count killed
the test through set -e before its diagnostic could print.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Lift the NO_WEBDAV conditional out of a macro argument list
A preprocessor directive inside a macro invocation's arguments is undefined:
it was fine while this was a plain sprintf() call, and MSVC rejected it as
soon as it became StringSprintf(). GCC accepts it, so only the Windows leg
caught it. Compute the DAV header fragment first and pass it as an argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover StringSprintf's exact-fill case and the WebDAV enumeration branch
StringSprintf_ writes the terminator at buffer[ret], so widening its
`ret < capacity` guard by one byte is a heap overflow that only fires when the
formatted output exactly fills the capacity. No crawl test lands on a
capacity boundary, so the mutant survived the suite. The new `strsprintf`
self-test sweeps lengths around 256, 512, 1024 and 2048 with the String's
capacity pinned to each, plus a growing and shrinking sweep on one reused
String, and checks the length, the bytes and the terminator every time.
Test 147 only ever sent Depth: 0, leaving the enumeration branch the same PR
rewrote with no coverage at all. Its fixture gains a child directory, and a
Depth: 1 listing pins the item URLs, including the trailing '/' that
StringPopRight takes back off a directory name.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the String failure paths safe without assert
StringSprintf empties the String when it gives up, but that contract was
only visible in the implementation, and the WebDAV enumeration in
proxytrack pops the trailing '/' straight after it. State it at the
declaration, no-op StringPopRight on an empty String, and skip an
enumerated item the formatter could not name.
StringRoomTotal reported a failed realloc through STRING_ASSERT alone.
The MSVC Release configuration defines NDEBUG, so that check is already
gone from the shipped Windows builds, leaving a NULL buffer under a
capacity bumped before the allocation was known to succeed. Assign both
only on success, and terminate through StringOom_.
Closes#915
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Renumber the String OOM test to 152
151 is taken by the unmerged tests/151_bash-shell-validate.test (PR #920).
The filenames differ, so git would have carried both onto master rather than
conflicting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Declare the new WebDAV test in the Windows skip set
It skips on Windows for the same reason as its two neighbours, MSYS
cannot reap a background listener (#595), and the ratchet fails a skip
it was not told about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the out-of-memory action overridable
STRING_REALLOC and STRING_FREE are #ifndef hooks, and STRING_ASSERT was one
too; replacing it with a hard-wired call took a hook away from downstreams of
this installed header. Route the failure through STRING_OOM instead, with the
print-and-abort default unchanged.
Also flush stderr before aborting: the Windows CRT buffers a redirected
stderr and abort() flushes nothing, which would drop the message the test
matches on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Inject the allocation failure instead of asking for a huge one
The engine self-test pinned a String's capacity so the next doubling asked
for SIZE_MAX/2, on the assumption that no allocator would serve it. Six CI
legs disagreed: i386 has a 3G user space, and the 64-bit runners handed the
request out too, so the test reported "NOT aborted" everywhere but here.
Green that depends on how much memory the machine feels like giving is not a
test.
Drive the path from a standalone helper instead, which defines STRING_REALLOC
to a stub returning NULL before including htsstrings.h. Four cases: growth
with the stub allocating for real, the failure reaching the handler with the
size it asked for, a live buffer surviving a failed realloc, and the shipped
handler printing and aborting. Only the automake build produces the helper,
so the test declares its Windows skip.
The self-test had no portable way to force the failure, so it goes rather
than staying as a handler nobody can rely on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the bytes and the requested size, not just the bookkeeping
An under-allocation survived the helper: shortening the realloc by one byte
while still recording the full capacity left all four cases green, because
only the growth case allocated anything and it checked the capacity number
rather than the memory behind it. Fill the announced capacity to its last
byte and read it back, which the sanitizer legs turn into a hard failure.
The failure cases pinned the initial capacity by asserting 16, so bumping
that policy would have failed a correct tree. Compare the size handed to the
handler against the size the stub was actually asked for instead, which also
catches the under-allocation on legs with no sanitizer.
Drive StringSprintf_ and StringBuffN_ too, the other two places the header
expands STRING_OOM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* configure accepts a BASH_SHELL that is not a usable bash
`./configure BASH_SHELL=/bin/sh` was accepted without a word. `AC_PATH_PROGS`
takes any absolute value verbatim, so the macOS problem #895 fixed (a bash in
POSIX sh-mode driving `make deb` and the test harness) came back, surfacing
much later as a `146_bash-shell.test` failure instead of a configure error. A
relative value never reached the Makefiles at all: it was dropped for whatever
the PATH search turned up.
configure now checks the value it resolved. The shell must be executable,
report a `BASH_VERSION`, and not carry `posix` in `SHELLOPTS`, the
discriminator `146_bash-shell.test` already uses, since an sh-mode bash reports
a version too. A relative or whitespace-carrying override is refused before the
search runs; whitespace would otherwise survive into `$(BASH_SHELL)`, which
nothing in the Makefiles quotes.
Only an explicit override is fatal. When the search itself finds nothing
usable, configure warns and carries on, so a box without bash still builds; it
just cannot run `make check` or `make deb`.
Closes#908
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* An environment in POSIX mode must not be blamed on the bash path
POSIXLY_CORRECT, or an exported SHELLOPTS, puts every bash into POSIX
sh-mode, so `./configure BASH_SHELL=/bin/bash` failed with advice to pass a
path that cannot exist, and a plain configure warned that no usable bash was
found on a box that has one. The probe now runs a second time under `env -u
POSIXLY_CORRECT -u SHELLOPTS`; if the shell is fine once they are cleared, the
message names them and says how to clear them for make as well, since it
inherits the environment. An override stays fatal, the search still only warns.
The bash-ness probe read `BASH_VERSION`, an ordinary variable any shell echoes
back, so `BASH_VERSION=9.9 ./configure BASH_SHELL=/bin/dash` was accepted and
dash landed in `TEST_LOG_COMPILER`. It reads `${BASH_VERSINFO[0]}` instead,
which no environment can fake.
The path guard covered whitespace alone while claiming to cover what make and
the recipe shell split on, so a real bash under a directory named with `;` or
`$` or `#` still reached the Makefile. It now rejects that whole class. Quoting
`$(BASH_SHELL)` at its three uses was the alternative, but make cuts the value
at a `#` and expands a `$` before any shell sees it, so quoting would cover
less than the guard.
The suite now also pins the branch the fatal/warn split rests on: no override,
an unusable bash first in PATH, configure exits 0 with a warning.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Nothing pinned which cause of POSIX mode gets blamed
A shell can be in POSIX mode because it was invoked as sh or because the
environment forces it, and only the second probe can say which. Nothing held
that down, so a version deciding from the environment alone, without
re-probing, passed every case in the suite while telling the user to clear a
variable that would not have helped. Test 151 now runs a bash symlinked as sh
with POSIXLY_CORRECT=1 set as well, and requires the message to name the path.
The comment records why that branch cannot simply read POSIXLY_CORRECT:
autoconf runs "set -o posix" on configure's own shell, so it is set there no
matter what the user's environment holds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A chunked response carrying trailers is discarded as "Invalid chunk"
The chunk automaton expected the line after the terminating zero-length
chunk to be empty. RFC 9112 7.1.2 lets a server put a trailer section
there and asks recipients to discard fields they do not understand;
instead the whole message failed and the resource never reached the
mirror.
The trailer section is now read the way headers are, as a block ending
on a blank line, and thrown away. Reading it as a block also bounds it:
trailers carry no length of their own, so an endless one would hold a
connection slot forever, and the line reader's 8KB buffer caps it. Only
the terminating chunk opens the section, so junk where a data chunk's
own CRLF belongs is still a framing error.
Closes#855
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Renumber the trailer test to 149, 147 is taken by the WebDAV overflow test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Name the line-block bound, and pin the trailer edge cases
Review follow-ups: the 8190-byte cap that bounds a trailer section was an
unnamed literal inside http_xfread1, so raising it for large response
headers would have moved the trailer bound silently. It is now
HTS_LINE_BLOCK_SIZE, named where the reader is declared.
The trailer path also no longer runs the chunk-size parse it then
discards, and eof.html pins the deliberate leniency the change
introduces: past a complete, length-verified body, a trailer section cut
before its blank line still lands. A body cut before the terminating
chunk stays refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Name http_xfread1's read modes instead of passing bare 0, -1 and -2
http_xfread1() selects its read mode from the sign of bufl: a positive
value reads that many bytes, 0 stops at a blank line, -1 at the first LF,
-2 only reserves the buffer. Nothing declared them, so every call site was
an unexplained literal.
Declare HTS_XFREAD_LINE_BLOCK, HTS_XFREAD_LINE and HTS_XFREAD_RESERVE in
htslib.h beside HTS_LINE_BLOCK_SIZE, and use them at each call site. The
selftest keeps its 8192, a byte count rather than a mode.
Behaviour-preserving: the preprocessed output of the changed translation
units is token-identical once the parens around the negative literals and
the __LINE__ digits shifted by the reworded comments are normalized away.
Closes#914
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Note that the reserve-only read mode has no caller
Naming it made it read as a supported mode; it is unreachable (#923).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the reserve-mode note on one line
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Bound the ProxyTrack DAV item buffer against an amplified PROPFIND path
proxytrack_add_DAV_Item() reserved a fixed 1024 bytes and then sprintf'd into
it unbounded. The request path lands in the response twice, once as the href
and once as the displayname, and escapexml() turns each '&' into '&', so
an unauthenticated PROPFIND of roughly 900 ampersands writes about 9000 bytes
off the end of the heap block. No cache entry and no Depth: 1 are needed.
Replace the hand-sized reserve with StringSprintf(), which measures the
formatted output and grows the String to fit, and convert the sibling sprintf
sites in the same file so no unbounded write into a String is left to
re-audit. Sizing beats clipping here: the String already owns a growable
buffer, so nothing has to be dropped.
Closes#836
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bound StringSprintf's pre-C99 retry, and trim the review findings
A genuine vsnprintf conversion error returns -1 just as pre-C99 msvcrt does
for a short buffer, so the doubling search had no way to tell them apart and
grew until realloc aborted. Unreachable from these format strings, which use
only %s and %d, but the helper lives in a shared header and will get more
callers. Cap the search and empty the String past it.
Also: the count assertion piped into wc under pipefail, so a zero count killed
the test through set -e before its diagnostic could print.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Lift the NO_WEBDAV conditional out of a macro argument list
A preprocessor directive inside a macro invocation's arguments is undefined:
it was fine while this was a plain sprintf() call, and MSVC rejected it as
soon as it became StringSprintf(). GCC accepts it, so only the Windows leg
caught it. Compute the DAV header fragment first and pass it as an argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover StringSprintf's exact-fill case and the WebDAV enumeration branch
StringSprintf_ writes the terminator at buffer[ret], so widening its
`ret < capacity` guard by one byte is a heap overflow that only fires when the
formatted output exactly fills the capacity. No crawl test lands on a
capacity boundary, so the mutant survived the suite. The new `strsprintf`
self-test sweeps lengths around 256, 512, 1024 and 2048 with the String's
capacity pinned to each, plus a growing and shrinking sweep on one reused
String, and checks the length, the bytes and the terminator every time.
Test 147 only ever sent Depth: 0, leaving the enumeration branch the same PR
rewrote with no coverage at all. Its fixture gains a child directory, and a
Depth: 1 listing pins the item URLs, including the trailing '/' that
StringPopRight takes back off a directory name.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Declare the new WebDAV test in the Windows skip set
It skips on Windows for the same reason as its two neighbours, MSYS
cannot reap a background listener (#595), and the ratchet fails a skip
it was not told about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A chunked response carrying trailers is discarded as "Invalid chunk"
The chunk automaton expected the line after the terminating zero-length
chunk to be empty. RFC 9112 7.1.2 lets a server put a trailer section
there and asks recipients to discard fields they do not understand;
instead the whole message failed and the resource never reached the
mirror.
The trailer section is now read the way headers are, as a block ending
on a blank line, and thrown away. Reading it as a block also bounds it:
trailers carry no length of their own, so an endless one would hold a
connection slot forever, and the line reader's 8KB buffer caps it. Only
the terminating chunk opens the section, so junk where a data chunk's
own CRLF belongs is still a framing error.
Closes#855
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Renumber the trailer test to 149, 147 is taken by the WebDAV overflow test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
* Name the line-block bound, and pin the trailer edge cases
Review follow-ups: the 8190-byte cap that bounds a trailer section was an
unnamed literal inside http_xfread1, so raising it for large response
headers would have moved the trailer bound silently. It is now
HTS_LINE_BLOCK_SIZE, named where the reader is declared.
The trailer path also no longer runs the chunk-size parse it then
discards, and eof.html pins the deliberate leniency the change
introduces: past a complete, length-verified body, a trailer section cut
before its blank line still lands. A body cut before the terminating
chunk stays refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Spool a frozen backlog slot outside the mirror namespace
back_cleanup_background() named the spool file by appending ".tmp" to the save
name, so it landed beside the mirrored file. That is the shape #774 fixed for
the re-fetch backup: a site serving <path>.tmp has its mirrored copy truncated
by filecreate() and then unlinked when the slot is woken, and the run still
reports success. It is reachable on defaults, not only under a saturated
backlog: a -Z crawl of the bundled bigcrawl site with -c4 logs slots moving to
background.
Route both name shapes through back_spoolname(), which puts them in the
~hts-tmp directory no save name can spell, and drop that directory at the two
sites that unlink a spool. Left out of the #774 PR because these lines also
carried the overflow in #857.
Closes#859
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the -p0 spool relative when no output directory is set
The new name inserted its own separator before ~hts-tmp, but path_html_utf8
already carries one and is empty when -O is absent, so the spool became
/~hts-tmp/tmpfile0.tmp: absolute, in the filesystem root. Master built
"%stmpfile%d.tmp" and stayed relative to the working directory.
create_back_tmpfile() has spelled it the same way since #842. Its empty
path branch looks unreachable from the three call sites, so this side is a
consistency fix with no test behind it, unlike the spool.
The self-test pinned the doubled slash it observed rather than the shape it
wanted; it now asserts the single-separator form and covers the empty
path_html_utf8 case that produced the root path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The panel decoration was drawn as vector and flattened to a 4KB indexed GIF
some time around 2007, with the panel colour baked in as an opaque backdrop.
Refitting its four ellipse boundaries recovers the original geometry, so it
goes back to being what it was: two elliptical annuli, 553 bytes of SVG, with
a transparent background that now composites over the panel instead of having
to match it.
Rasterised at the same size, the only pixels that differ from the GIF are
single-pixel anti-aliasing fringes along the four boundaries. Nothing survives
a 3x3 erosion of that difference, so no edge has moved.
Dark mode still drops the image rather than inverting it, since the ring
lavender is a light-panel tone whichever way the file stores it.
The engine keeps its own embedded copy of this GIF for the backblue.gif it
writes into mirrors. That one is untouched: the filename and byte length are
a contract with pages already on disk.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* configure discards a user-supplied BASH_SHELL
AS_UNSET erased the variable before AC_PATH_PROGS could honour it, so
"./configure BASH_SHELL=/path" had no effect and there was no way to
point the build at a bash other than the first one on PATH. Nothing
presets BASH_SHELL, which was the whole problem with BASH in #895, so
declaring it precious is enough.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Run the nested configure against a symlink farm
An in-tree build leaves a config.status in srcdir, and autoconf then refuses
the out-of-tree run the test needs. Every CI build leg builds in-tree, so the
check failed there while passing on an out-of-tree tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Read the resolved bash from the configure trace, not the Makefile
The nested configure ran without the flags the outer one was given, so on
macOS it died at the openssl check that Homebrew paths satisfy. BASH_SHELL
is resolved long before that, so assert on the trace and let the run fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the value that reaches $(BASH_SHELL), not the macro's decision
Reading the configure trace let a mutant through: resolve the override
correctly, clobber BASH_SHELL one line later, and both assertions passed
while every Makefile got the wrong shell. Prefer the generated Makefile
and keep the trace only as a fallback for a configure that dies early.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The AppStream metainfo still advertises WebHTTrack 3.49.8
The metainfo installs to usr/share/metainfo, so GNOME Software and KDE
Discover read both the version and the "What's new" text out of it. Its
releases block held one entry, 3.49.8, and nothing had moved it since.
It now lists 3.49.8 through 3.49.15, newest first, each with a short
user-facing note taken from history.txt. Dates come from the git tags;
that also corrects 3.49.8's, which carried 3.49.7's date.
01_engine-version-macros.test gains two assertions so the next release
cannot miss this file, or configure.ac: AC_INIT and the top release entry
must both match HTTRACK_VERSIONID, and the entries must descend so the
top one really is the newest.
Closes#884
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Harden the metainfo version check and correct four release notes
The release-version extraction matched <releases ...> as well as <release>,
ignored XML comments and took only the last tag on a shared line, so a parked
or wrapper version= could pose as the newest entry and pass a stale metainfo.
Split tags one per line, drop comments, and anchor the match. Widen the awk
ordering key so a component of 1000 or more cannot borrow into the next.
In the notes: "3.49-2" and "site rules with wildcards" are unreadable in a
software centre, the Windows path bullet does not apply to the Unix WebHTTrack
GUI it ships with, and 3.49.15 listed no web-interface fix at all.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Find the data directory instead of trusting the configure-time one
webhttrack probed a fixed list of prefixes that nothing derived from
--datadir, and the engine baked $(datadir) into the binary with the
argv[0] fallback compiled out. Both fail on any tree that is not where
it was configured.
configure substitutes the real datadir into src/webhttrack, and
hts_resolve_datadir() prefers the compiled-in path but derives one from
argv[0] when it is gone, so a moved install reads its own templates
rather than silently falling back to the built-in defaults.
Closes#887Closes#894
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Resolve the data directory from the executable, not just argv[0]
MSVC broke: HTS_HTTRACKDIR is only defined on non-Windows, so the
argv[0] branch this replaced was live there, not dead. Windows now
passes an empty builtin and falls back to the executable's own
directory, which is what it did before -- except fconcat inserts no
separator, so the old path_bin lacked its trailing slash and never
resolved a template anyway.
Ask the OS for the executable path (/proc/self/exe, _NSGetExecutablePath,
GetModuleFileName) and keep argv[0] as the fallback, so a mirror run
through a PATH lookup resolves too.
The bundle drops the substituted datadir from its copy of webhttrack:
it is a build-machine path there, and the relative entries ahead of it
already find the payload.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Clip the candidate path instead of aborting on a long argv[0]
strlncatbuff() aborts rather than truncates, and appending the layout
suffix to an already-full buffer reaches that: a directory part within
17 bytes of the candidate buffer's size killed the process. Build the
candidate with snprintf and skip it when it does not fit.
The self-test now drives a directory part long enough to trigger it;
without the fix it aborts on "overflow while appending 'layout[i]'".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Ignore the generated src/webhttrack
An in-tree build writes it next to webhttrack.in, where it was untracked
and one "git add -A" away from re-entering the tree with a build
machine's datadir frozen into it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* configure resolves bash to /bin/sh on macOS, and make deb runs a bash script with dash
AC_PATH_PROGS searched into BASH, which bash presets to its own invocation
path. configure re-execs through /bin/sh, and on macOS that shell is a bash, so
the macro honoured the pre-set value and reported "checking for bash...
/bin/sh": a bash in sh-mode that rejects process substitution. Search into
BASH_SHELL instead, a name no shell presets, with AS_UNSET in front so the
environment cannot preset it either.
That makes the obvious fix for the deb target safe. It ran tools/mkdeb.sh with
$(SHELL), which is /bin/sh, so "make deb" died on the first bashism on every
Debian and Ubuntu box. macos-app.sh stays on $(SHELL): it is POSIX sh on
purpose, so shellcheck lints it as sh.
tests/146_bash-shell.test asserts the configured shell exists, sets
BASH_VERSION, is not in POSIX sh-mode, and parses a process substitution.
Closes#895Closes#891
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the comments this branch added, and drop a claim that is no longer true
The four comment blocks the branch added ran two to five lines where one or two
carry the fact. The 146 header also said the macOS shell rejects "the process
substitution the bundle script uses": tools/macos-app.sh has been POSIX sh with
no process substitution since #890, so the gate is there for tests/local-crawl.sh
and tests/webhttrack-smoke.sh, which is what the comment now says.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The test server's bind reverse-resolves 127.0.0.1, which stalls on macOS
http.server's HTTPServer.server_bind() calls getfqdn() on the bind address just
to fill server_name, which nothing in local-server.py reads. On the macos-15
runner that lookup takes ~30s, so the PORT line lands well past every caller's
discovery budget (#870).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* PROBE: macos-15 + startup timing (not for merge)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Move the macOS CI legs to macos-15, and route the last two port waits through the shared helper
84 and 100 kept their own PORT poll loops with 10s budgets, the copies #869 did
not reach; both use discover_server_port now, so there is one implementation
left.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cut the server_bind comment to the why
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* armhf crash reports have no frames, and the empty backtrace fails the build
gcc emits no unwind tables on armhf, so backtrace() comes back empty and the
handler printed a frameless report, which the crash tests read as a failure and
which left 3.49.15-1 stuck at Build-Attempted there. Ask for
-fasynchronous-unwind-tables where the compiler takes it, and say why the report
has no frames when the unwinder still returns nothing.
Closes#892
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Let ASan tolerate the backtrace shim's link order
An LD_PRELOAD library loads ahead of the executable's own libasan, which ASan
refuses by default, so the sanitize leg never reached the crash it was meant to
inspect. Same waiver the other interposer tests carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep an empty backtrace a failure everywhere but 32-bit ARM
Relaxing test 80's skip to the shared message prefix let it swallow the new
"unwinding failed" wording too, so a build that traced nothing anywhere would
have gone green on every architecture. Skip only where nothing can be done
about it: the OS-less case, and 32-bit ARM if its toolchain still refuses to
unwind. Test 143 now pins the exact wording rather than the prefix it shares
with the OS-less notice, and its control run skips the symbolizer it has no
reason to spawn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Declare the new backtrace test's Windows skip
The Windows job compares the skip set exactly, so a test that skips there for a
good reason still fails the gate until it is named.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Ship a macOS app bundle
macOS users install HTTrack through Homebrew and get a working WebHTTrack they
are never told about: the formula installs webhttrack and htsserver, and the
only thing missing is something to double-click. tools/macos-app.sh assembles
HTTrack.app from an installed prefix, with the payload under Contents/Resources
so webhttrack keeps resolving htsserver and its data from its own location, and
a two-line stub in Contents/MacOS for Launch Services.
Nothing about the engine changes. The bundle is possible because webhttrack was
already relocatable and because the data symlink stopped being absolute (#885);
--disable-shared keeps libhttrack inside the binaries so nothing points back at
the staging prefix. That costs no crash diagnostics here, since backtraces are
gated on __linux (src/htsbacktrace.c:50), though it does trip #889 on Linux.
The script verifies what it builds rather than trusting it: no absolute symlink,
the served UI present, no Mach-O still linking the staging prefix, and the
Info.plist version matching the installed binary. configure generates that
plist, so it cannot drift into a fifth hand-maintained version spot of the kind
#884 describes.
CI assembles the bundle, runs the webhttrack smoke through the stub, then moves
the bundle and deletes the prefix it came from and runs it again, which is the
one thing a .app has to survive that a prefix install does not. An ad-hoc
codesign proves it is well formed enough to sign; Gatekeeper needs a Developer
ID and stays out of scope with the DMG.
No custom icon: the largest artwork in the tree is 48x48 and macOS wants 1024,
so that needs a real source asset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* ci: lint the new bundle script, and mark it executable
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Drive the bundle through a make target, and run it with bash
Adds a macos-app target so assembling the bundle goes through the build system
the way make deb does, rather than CI reaching for the script directly.
It runs the script with $(BASH), not $(SHELL). automake's SHELL is /bin/sh, and
the script uses process substitution, so under dash it died partway: the payload
was already copied by then and only the verification was skipped, leaving a
bundle that looked built and had been checked by nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the bundle checks catch what they were missing
The smoke put the bundle's own bin on $PATH, so a launcher that ignored its own
location and just ran "webhttrack" passed as readily as the real one, which is
the most likely way a stub breaks. Dropping $prefix/bin from $PATH kills that:
the browser stub is found through webhttrack's SRCHPATH, not $PATH, so nothing
else needed it.
The bundle also shipped libtool .la files, static archives and include/, none of
them loadable from a static build and the .la files carrying the staging prefix
in libdir=. They are pruned, and a text sweep now fails on any remaining file
that embeds that prefix, which otool cannot see because it reads load commands
only. The prefix is resolved to an absolute path first, or a relative --prefix
made that grep match nothing.
Also: the symlink scan asserts it scanned something, and CFBundleVersion is
compared as well as CFBundleShortVersionString.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assemble the bundle with POSIX sh, not $(BASH)
$(BASH) is not a reliable bash on macOS. /bin/sh there is bash in sh-mode, which
presets $BASH to the path it was invoked as, and AC_PATH_PROGS honours a
pre-set value rather than searching, so configure reports "checking for bash...
/bin/sh". That shell rejects process substitution, and the bundle job died on
it.
Rather than hunt for a real bash, the script no longer needs one: the three
process substitutions become temp-file loops, pipefail goes (not POSIX), and
the target is back on the ordinary $(SHELL) like deb:. shellcheck now reads it
as sh, so a bashism creeping back fails lint instead of macOS CI.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Make the installed html symlink relative
The install-data-hook linked share/httrack/html to an absolute $(htmldir), so
an installed tree only worked at the prefix it was configured for: the link
dangled under DESTDIR staging and in any relocated copy, and webhttrack then
failed its test -d "${DISTPATH}/html" check and exited. It now emits a
relative link when htmldir sits under datadir.
The hook also sat in html/Makefile.am while writing into $(datadir)/httrack,
the directory lang/Makefile.am declares and populates, and it hardcoded
$(prefix)/share instead of following datadir. Moving it to lang/ with
$(langrootdir) drops that cross-subdirectory install-order dependency. A
stale symlink was previously left in place, so an upgrade kept an absolute
one; it is now replaced, a real directory is left alone with a diagnostic,
and a new uninstall-hook removes what install created. A moved datadir still
defeats webhttrack's own data search for an unrelated reason (#887).
The smoke test asserts no installed symlink is absolute and that the link,
resolved inside a copy of the tree at another path, stays inside that copy
and lands on the served UI. It now runs on Linux as well as macOS, since
Linux is where the affected packaging is consumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* ci: match the Linux build job's package list
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: put the stub browser where webhttrack looks first
webhttrack searches its own SRCHPATH (starting at $BINWD, then /usr/local/bin,
/usr/share/bin, /usr/bin) and only appends $PATH after it, so shadowing
x-www-browser through PATH only worked on hosts that have no real one. The
GitHub Linux runners ship /usr/bin/x-www-browser as Edge, which won the search
and then aborted on its SUID sandbox, failing the new Linux job.
Write the stub to $prefix/bin instead, which is SRCHPATH[0] on every platform,
and remove it in the teardown.
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Version bump to 3.49.15 in the three spots that carry it (configure.ac, htsglobal.h, version.rc), plus the release notes and the Debian changelog entry.
VERSION_INFO goes 3:6:0 to 3:7:0, soname still .so.3. The installed htsblk gained a field at its tail and lien_back embeds it by value, so lien_back.is_update and everything after it shifts 8 bytes. An offsetof probe against both tags confirms httrackp's own fields did not move. Same call as 3.49.14: HTTrackQt is the only consumer of these headers, so no libhttrack4 rename.
109 commits since 3.49.14, none of them under debian/, so the changelog entry is a plain new-upstream one with a note that httrack-doc grows about 1 MB from the new GUI screenshots.
The lede led with WARC output and the change report, which are niche features, because those happened to be the last things merged. What matters to someone arriving now is that HTTPS works, that proxies work, that a current page with responsive image sets and lazy loading captures properly, and that an update does not re-fetch the whole site.
The rings the 2007 pages carried behind their content panel come back, and now on every page rather than the handful that had them. The image has the light panel colour baked in, so dark mode drops it rather than showing a pale rectangle.
While in there: the wordmark is black on transparent and all but vanished against the dark field, which every page inherited along with the shared chrome. It is inverted in dark mode now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
addurl.html was 116 words and five screenshots of a 2003 dialog, reached only from a single sentence in the guide. Its substance moves into that sentence: what the Add a URL button does that the address box cannot, including the browser-capture proxy, which lang.def confirms is still in the GUI. The stale screenshots go with it, and a fresh capture of that dialog belongs in the next screenshot walk rather than in a page of its own.
options.html had been an orphan since #661 gave it a legacy-list banner and repointed everything that used to link it. It becomes a stub pointing at the generated option reference and the command-line guide.
plug_330.html documents the callback interface of releases 3.30 to 3.40. Its title said so; now a banner does too, the way Fred Cohen's guide is labelled on the index, because someone arriving from a search engine reads the page and not the title.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Nine of the twenty pages linked nowhere at all and seven had a single inbound link, so a reader landing on cache.html from a search engine got no navigation, no breadcrumb and no way back. Each page also carried its own copy of the same inline stylesheet inside a six-table shell with a 400px floor, which is why sixteen of them scrolled sideways on a phone.
They now share the guide's chrome: masthead, sidebar, footer, and where the headings allow it a generated "On this page" list. file:// has no server includes, so the navigation is real markup in every page, written by tools/doc-chrome.py and verified by CI; tools/doc-links.py resolves every relative link and fragment. Both refuse to pass vacuously, and both were checked against planted defects.
The prose is untouched. For twelve of the fifteen pages the content text is identical word for word; the three exceptions are cmdguide.html losing the contents list the sidebar now carries, contact.html re-encoded to UTF-8, and one font tag whose removal joined two words the browser already ran together. Each page also gains a real title and its own description, replacing the shared blurb whose keyword list still advertised Windows 95 and AIX 4.0.
Image zoom and sidebar highlighting move to doc.js so every page has them, leaving guide.js the platform switcher and the option filter. The guide renders identically at 1100px in light and dark, and thirteen behaviours pass unchanged before and after; at 390px one Proxy paragraph that used to overflow its container by 37px now wraps. Nothing scrolls sideways at 390px or 1100px, and no page throws a script error.
httrack.man.html keeps its own bare styling for now, since reskinning it belongs in the generator that writes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The documentation index is the page WinHTTrack opens for generic Help and the one httrack.com links as "Documentation", and it presented twenty destinations as a flat bullet list with nothing to say which one a newcomer wants. It is now a task-grouped hub in the interface guide's idiom, with overview.html folded in as the lede and refreshed for 3.50, and Fred Cohen's guide under an Archive heading labelled as written for 3.10, keeping its URL.
The chrome the hub needs is split out of guide.css into doc.css; guide.css keeps the platform switcher, tab strip and option entries. guide.html renders pixel-identically before and after the split at 1100px and 390px, in light and dark.
overview.html, start.html and cmddoc.html become meta-refresh stubs, like the retired step pages. start.html was a window.open popup launcher and cmddoc.html has 404ed since #661 deleted it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Test 142 guards against a vacuous run by requiring every asset it types to exist. img/android_spider.png stopped existing when the Android help page was folded into the unified guide and its screenshots were renamed guide-droid-*.png: #874 and #876 were in flight at the same time, so neither run saw the other half, and master has failed this test on every platform since it landed.
The three references now point at guide-droid-opt-spider.png, a PNG in the same directory, so the check still tests what it was written to test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver labels every PNG as image/gif, and serves JPEG as a download
The response header was picked per broad file class, so is_image() matched
both .gif and .png but always emitted Content-type: image/gif, and anything
outside the five hardcoded classes (.jpg, .svg, .ico, .xml) went out as
application/octet-stream, which browsers download instead of render.
Replace the per-class headers with an extension to type table, matched on the
real extension rather than on a substring of the path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the untested paths the content-type table can regress on
The first test named 9 of the 12 extensions and no superstring case, so a
lookup matching on a prefix, or one that dropped the htm or jpeg row, stayed
green. Add those fixtures, a raw-socket check that the header line is still
CRLF-framed, and the no-cache pair that only the HTML branch carries.
Also serve .xml as application/xml (RFC 7303 deprecates text/xml) and add
webp and avif, the two image types a browsed mirror most often needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* One guide for WinHTTrack, WebHTTrack and Android, instead of 18 thin pages
The GUI documentation was 18 files whose bodies were mostly a copy-pasted style
block, illustrated with 1998-era screenshots of WinHTTrack, with the Android app
documented separately and WebHTTrack not shown at all. html/guide.html replaces
them with a single page: one walkthrough and one option reference, and a
switcher that swaps the screenshots and the platform-specific notes between the
three versions. Each option now also names its command-line equivalent.
The screenshots are the three capture sets from httrack-works, prepared by
tools/doc-images.py. Deep links carry the platform, so guide.html#win/opt-limits
opens the Limits section already showing the Windows dialog; the WebHTTrack
option pages use that form. The old filenames stay as redirect stubs for the
forum links that still point at them.
Scripting is an enhancement throughout: without it the page shows every
platform's text and screenshots rather than hiding any of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Document the options the old pages never had
A coverage probe against the WebHTTrack form controls found the reference
inherited the old doc's blind spots: WARC, the CDXJ index and WACZ, --changes,
sitemap seeding, cookies-file, the urlhack opt-outs, strip-query, the proxy type
selector and the random inter-file pause were all exposed in the UI and absent
here. Labels and descriptions taken from lang.def, CLI flags from htsalias.c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tools/screenshot-walk.py, which starts htsserver, drives its pages in
headless Chromium and captures one PNG per documentation screen, plus a
manual screenshots workflow that runs it on a runner. The Android and
Windows front ends already have their equivalent.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Refresh the Android help page from current app screenshots
The ten screenshots dated from an older build and covered only two of the
eleven option tabs. Recaptured all of them from app versionCode 85 (engine
3.49.14) and added Build, Spider and Log/Index/Cache, whose contents the
one-line summaries could not convey.
The new shots also correct three claims: the proxy tab now picks a protocol
(HTTP, SOCKS5, HTTP CONNECT tunnel), Log/Index/Cache gained WARC output and
the change report, and Base path opens a chooser rather than being fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Document every option tab on the Android page
Android users cannot easily flip between help pages, so the page now carries
its own option reference: all eleven tabs, each with its screenshot and every
setting on it, replacing the pointer at the desktop reference.
Six settings the desktop pages never covered are documented here for the
first time: keep-alive, the inter-file pause, the three URL-hack opt-outs on
Links, and Strip query keys on Experts Only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub marks macos-14 deprecated, so both macOS jobs are on an image headed
for brownouts and removal. macos-15 is the same arm64 host and every path in
those jobs is brew --prefix-derived, so nothing else moves.
Also point Dependabot at git submodules: src/coucal is a hard build dependency
that only advances when someone remembers to look.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
local-crawl.sh waits 30s for local-server.py's "PORT <n>" line and matches it
anywhere, because a cold Python start under a parallel make check lags past a
second and a warning merged via 2>&1 can precede the line. Seventeen tests that
launch the server directly carried an older copy of that loop: 5s, and only the
first line of the log. macos-15 exposed it, failing 15 of them where macos-14
passed on margin, but nothing about the bug is macOS-specific.
Move the proven idiom into testlib.sh next to stop_server and call it from all
seventeen, which also gets them the log dump on failure that local-crawl.sh has
and the inline copies did not -- the reason the CI failure said only "could not
discover server port" with no way to tell a slow start from a dead server.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenSSL 4.0 removes what 3.x deprecated. The engine is already clean against
that set, but nothing holds it there: a deprecated call would build fine on
every current leg and only surface when a distro moves to 4.0.
Build with -DOPENSSL_NO_DEPRECATED, which tracks whatever OpenSSL the runner
has rather than pinning a level the headers may reject. Compile-only; the
header contract is what is under test and the matrix already runs the suite.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add a hidden -#c option that crashes on purpose, for crash-handler testing
The crash handlers had no way to be exercised: httrack's own fatal-signal
handler, WinHTTrack's, and the Android app's mirror-recovery path could only
be tested by waiting for a real bug. -#c[=KIND] faults deliberately, in
segv, abort, trap or stack flavours, announcing on stderr that the crash was
asked for so a log never reads as a genuine failure.
Undocumented and unlisted, but always compiled in: the point is to test the
shipped artifact (the APK, the installer, the distro package), which a
build-time flag would exclude.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert a backtrace portably: macOS has no backtrace() at all
The frame check matched glibc's "[0xaddr]" form, which macOS never prints:
USES_BACKTRACE is gated on __linux, so every other OS gets the "No stack
trace available" notice instead. Accept either, and also require the
handler's closing line so the assertion still proves it ran to the end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The webhttrack GUI cannot reach --warc-cdx, --wacz or --warc-max-size
PR #672 exposed --warc and --warc-file on the "Log, index, cache" tab; the
three WARC flags added after it never got a control, so a CDXJ index or a
WACZ package is unreachable from the GUI. Add the two checkboxes and the
size field, and translate the six new lang.def strings into all 30
languages instead of letting them fall back to English.
An unknown ${LANG_*} renders as the empty string, so a key missing from
lang.def or from one lang/*.txt blanks a label instead of failing: test 140
now asserts every key the templates use exists everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Move the lang completeness check into 62_lang-integrity, its owner
62_lang-integrity.test already owns lang.def/lang-file consistency; it
checked that no msgid drifted, but not that every msgid is translated
everywhere, which is the gap that let the WARC options ship untranslated.
Add the completeness pass there and drop the separate 140 test, which
reimplemented the same parser in Python and was scoped narrowly enough to
be blind to the 24 msgids already missing from 26 files. Those are waived
by name against #863, and a waiver that stops being needed now fails.
Also check every ${LANG_*} the templates interpolate resolves, in any
wrapper form: the ${html:html:...} and ${fexist:...:...} spellings were
outside the dropped test's regex.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* url_savename never clamps the final path segment, only the whole path
Inside url_savename's over-ceiling shortening, every directory segment is cut
to MAX_SEG_LEN but the name itself is copied under the whole-path budget alone.
A single component could therefore run to ~226 characters on Windows while its
parent directories were held to 48, and once the directories had eaten the
budget the cut landed mid-name and took the extension with it, leaving a page
saved with no .html at all.
Clamp the name like any other segment, and reserve a plain extension across the
cut the way #623 already reserves the ".delayed" marker.
75_engine-longpath-posix's Windows branch asserted the clamp all along; it had
never run until #847 broadened the CI glob, so this also drops the expected_skip
that filed the failure as #852.
Closes#852
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix 01_engine-savename's Windows expectation for the clamped name
The Windows arm pinned the pre-fix output: 210 characters of name and no
extension at all. It now takes the 48-char segment clamp with its ".html"
intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Percent-escape a query-string character reference the page charset cannot represent
On a page with no declared charset (iso-8859-1 fallback), a reference like
€ was left as literal source text in the query string: its '&' and '#'
then act as delimiters, so the origin parsed more parameters than the document
expressed. The URL Standard writes such a code point %26%23<decimal>%3B.
Adds hts_unescapeEntitiesWithCharsetSpecial() with an opt-in
UNESCAPE_ENTITIES_URL_QUERY flag, set only by the query-string call site; the
plain decoder keeps its text contract and its in-place guarantee. The query is
now decoded out-of-place, since the escape grows the string.
Closes#854
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Detect an unrepresentable code point with the encoder, not a round trip
The round-trip comparison escaped code points the charset does represent:
hts_convertStringToUTF8() short-circuits on an ASCII string and hands the bytes
back undecoded, so every reference on ISO-2022-JP, UTF-7 or HZ compared unequal,
and glibc's CJK tables are not injective (cp932 U+301C came back as U+FF5E).
On POSIX iconv already fails outright on a code point it cannot encode, so the
NULL return is the answer. Windows substitutes instead of failing, and reports
it through WideCharToMultiByte's lpUsedDefaultChar, which we now pass;
hts_convertStringFromUTF8Strict() turns that into a NULL. Test 139 grows a
declared shift_jis and a declared iso-2022-jp page, which the two charset cases
it had could not have caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the ISO-2022-JP query per field, and let the two cp932 tables differ
Windows maps 0x8160 to U+FF5E rather than U+301C and reports the substitution,
so 〜 is genuinely unencodable there and the escape is right: accept either
encoding for the shift_jis page. ISO-2022-JP's trailing shift back to ASCII also
differs between iconv and Windows, so that page is asserted field by field, which
is what the round-trip check would have broken anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Let Windows substitute the euro sign on a stateful codepage
WideCharToMultiByte refuses lpUsedDefaultChar on ISO-2022-JP, so the
substitution cannot be seen and € goes out as '?' there, as it did before this
branch. It splits no query, and the page is in the test for the references the
charset does represent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Broaden Windows CI's engine-test glob and skip-set compare (#844)
The Windows engine-test loop named individual tests (01_engine-*.test plus a
hand-maintained tail) because its glob missed NNN_engine-*.test, so a new
test landed with zero Windows coverage until someone remembered to add it by
hand; two branches doing that at once collided on the same line. Broaden the
glob to *_engine-*.test, matching the *_local-*.test pattern already used
next to it, so a stray test file gets picked up automatically instead of
silently skipped.
That newly covers 66_engine-port80-strip, 67_engine-delayed-truncate,
75_engine-longpath-posix, 80_engine-crash-symbolize and
104_engine-warc-longurl, none of which had ever run on a Windows runner.
80_engine-crash-symbolize exits 77 there by design (no backtrace()), which
would have tripped the old exact-string expected_skips gate and turned the
leg red. Switch that gate to a sorted-set compare (one name per line, so an
append no longer shares a line either) and add the new skip; a mismatch now
prints a diff naming exactly which test's skip state changed instead of a
raw string dump.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Generalize the last two hand-pinned test names in the Windows CI loop
13_crawl_proxy_https.test and 60_crawl-log-salvage.test matched no
glob, so a renumber or a same-named future test would silently drop
Windows coverage again. Tightly globbed instead of using *_crawl*,
which would also sweep in the live-network crawl-*/crawl_https tests
this job never runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* kill_tree's Windows path falls through into a POSIX process-group kill
test-timeout.sh and run_with_timeout skip set -m on Windows (nothing there can
signal a group), so no target of kill_tree ever heads its own process group.
kill_tree's taskkill branch already reaps the Windows tree, but had no return,
so it always fell through to kill -9 -"$pid": on Windows that treats $pid as a
process-group id it never is, and can hit whatever real group its number
collides with under MSYS's own pid churn -- including the harness's own,
which is how PR #847's CI run died silently right after 102_local-ftp-refetch
finished, mid-cleanup, with no FAIL line and a truncated artifact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A chunked response cut at a chunk boundary is stored and cached as complete
back_finalize()'s incomplete-transfer guard is gated on r.totalsize >= 0, so it
only fires for a response that declared a Content-Length. A chunked response
keeps totalsize == -1 until the chunk headers add up, and the sum always equals
what arrived, so a stream that ended before its terminating zero-length chunk
took the success branch: the partial body was mirrored, cached, and on an
--update replaced the previous good copy.
The terminating chunk is part of the framing, so its absence makes the body
truncated by definition. Track it through chunk_blocksize, which only reaches
-1 once that header is parsed, and refuse the transfer in the two places the
short-Content-Length path already covers: back_finalize skips the cache, and
back_wait fails the transfer so the engine retries and leaves the mirrored copy
alone.
An HTTP/1.0 identity body delimited only by connection close has no framing to
check, and is deliberately left alone.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: cache assertions, -%B coverage, disk-write limit
Adds the cache half of the spec the test never asserted (--cache-found /
--cache-not-found in local-crawl.sh, reading hts-cache/new.zip), a -%B leg
covering opt->tolerant on a chunked body, and a direct-to-disk .bin resource.
Gates the back_wait block on r.statuscode > 0 like its back_finalize
counterpart, and drops the chunk_size == 0 hunk: a negative chunk-size header
is already refused by the short-body guard and the Invalid chunk path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the chunk_size == 0 sentinel guard, and pin it
A chunk-size line of 80000000 lands in an int as INT_MIN via sscanf("%x"):
a bare else claimed it as the terminating chunk while totalsize went negative,
so neither the framing check nor the short-body check fired and an --update
deleted the good copy and cached a 0-byte 200. Invalid chunk cannot cover this,
being set after back_finalize has already stored the response.
The update pass of chunktrunc/hostile.html now serves that header, and
chunktrunc/reset.bin aborts its body with an RST so a read error keeps its own
diagnostic instead of the framing check relabelling it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bound the parsed chunk size before it reaches the in-RAM realloc
sscanf("%x") into an int lands a chunk-size line of 80000000 on INT_MIN, which
drives totalsize negative past the INT32_MAX guard and sign-extends into a 16 EB
realloc (#860, pre-existing on master; the new #840 test is what reaches it).
Parse the field with strtoull and drop anything an int cannot hold, so the
existing arithmetic and the in-RAM cap keep working on a value known to be in
[0, INT32_MAX].
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A URL can still be saved onto the engine's own temporary files
The re-fetch backup and the content-coding temporaries built their names
inside the mirror, so a site serving the matching path had its mirrored
copy used as scratch and then deleted. #818 escaped hts-cache and hts-tmp
as reserved path components, but three shapes still got through (#842): a
trailing space that cleanEndingSpaceOrDot() stripped back off after the
check, and the first path component, which the table could not see because
it anchors on a leading '/' that url_savename has already removed. The
latter covers a single-label hostname too, and left the DOS device names
equally exposed in that position.
The temporaries now live in ~hts-tmp/. url_savename maps '~' to '_' in
every name it builds, so nothing a site can serve resolves inside it
whatever the escape does; the escape is still fixed, as defence for
hts-cache/ itself. The frozen-slot spool (<save>.tmp) shared the
collision, plus a heap buffer sized from url_sav that the path_html branch
overran, and moves to the same helper. The .delayed placeholder does not:
it goes through url_savename's collision detection, so a competing link
gets a suffix rather than the file.
Closes#774Closes#842
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Scope the reserved-name escape and fix the review blockers
Restrict the new first-component match so a hostname label is not rewritten:
only a trailing run cleanEndingSpaceOrDot() strips may end the name there,
so aux.example.com keeps its name while a single-label hts-tmp host and a
path-first hts-cache/ are still escaped. Renaming a host directory would
have moved an existing mirror out from under --update --purge-old.
Drop the frozen-slot spool rewrite: it carried a pre-existing heap overflow
that belongs in its own change. Clear back->tmpfile when structcheck() fails
in the named branch, as the unnamed one does. Test 132 now plants a leftover
at the exact temporary path, so both a reverted HTS_TMPDIR and a no-op
back_refetch_backup fail it; 131 pins the hostname labels.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The frozen-slot spool name is written into a buffer sized from a different string
back_cleanup_background() sized the temporary's name buffer from url_sav and
then, under -p0, wrote a path_html-derived name into it. Build the name in a
bounded buffer and duplicate it, so neither form can overrun.
Closes#857
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the spool actually ran, and correct the overflow's blast radius
The -p0 name escapes on a plain build too: _FORTIFY_SOURCE=3 catches the
same write in __sprintf_chk, so the abort is not sanitizer-only. Test 138
now runs under -Z and requires the engine log to show a slot spooled and no
serialize error, so a build with neither fortify nor ASan still proves the
branch was taken. tmpname gains room for the ".tmp" a full-length save name
appends, so no input is refused that the old exact-sized buffer accepted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A 304 during --update leaks the whole previous htsblk
back_wait() handles a 304 by replacing the response struct with the cache
entry, carrying only the socket and keep-alive members across via
back_connxfr(). The struct assignment drops every owned pointer the live
response still held without freeing any of them: the 8 KB header buffer on
every update, plus the two WARC header stashes when --warc-file is on. An
update over a 10k-page site drops roughly 80 MB in one run.
back_clear_entry() already knew how to tear those down, so the frees move into
a helper that both it and the 304 path call.
The new test runs the two-pass mini304 crawl with LeakSanitizer on, which the
sanitized CI job otherwise disables. The fresh first pass is the control: it
has no cache entry to read back and is clean either way. The update pass
reports 16 KB in 2 objects on master, one per unchanged URL, and nothing with
the fix.
Closes#782
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the test header
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: reuse deleteaddr(), and cover the WARC limb
back_free_response() was reimplementing deleteaddr(), which already frees adr
and headers and NULLs both; call it instead so the two cannot drift.
Test 114 never passed --warc-file, so the warc_free_request() limb ran with
both pointers NULL on every path it exercised and deleting it kept the test
green. A third pass turns the archive on, and it now fails with the 835 and 238
byte stashes when that call goes away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Register the new leak test as an expected Windows skip
The Win32/x64 job pins the exact set of tests allowed to skip, so an
all-skipped suite cannot report green. 114_local-update-304-leak needs a
LeakSanitizer build and MSVC has no equivalent, so it skips there and tripped
the gate with fail=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* A 304 revisit archives none of the exchange it stands for
The cache entry overwrites the whole htsblk in back_wait(), taking the
stashed request and response headers with it, so the revisit record gets
a synthesized "HTTP/1.1 200 OK" block and no request record. Both blocks
now move across the swap, but only for a real 304: the engine also forces
HTTP_NOT_MODIFIED by itself, and those have no 304 to carry. Since a 304
declares no Content-Type, warc_write_transaction() takes the resource type
from the caller so revisit CDXJ lines keep their mime.
cache_read_including_broken() returns the entry's response struct after
back_clear_entry() has freed it; adr, headers and location come back NULL
now, as the normal cache_readex() path already returns them.
Closes#826
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Do not expand an empty bodyargs array under macOS bash 3.2
Test 122 asks local-crawl.sh only for the revisit checks, so it is the
first caller to leave bodyargs empty. Bash before 4.4 calls that unbound
under set -u, which passes on Linux and fails the macOS leg.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the opt argument to hts_rename_over call sites
Lost when the master merge took this file wholesale; #816 gave the function an
httrackp parameter and the three call sites here reverted to the old form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* A 304 revisit names no peer IP, and an engine-forced not-modified is archived as one the server never sent
back_connxfr() moves the socket and keep-alive members across the cache-entry
swap in the 304 path of back_wait(), but not the peer address, so every
revisit record came out with no WARC-IP-Address (#838). Carry it across the
same swap.
Separately, several engine hacks (the same-size heuristic, the #176
error-ignore-on-update path, ...) force HTTP_NOT_MODIFIED themselves when the
server actually said something else. That statuscode flows into the same 304
branch, and warc_write_backtransaction() turned it into a revisit record whose
WARC-Profile claims a server-not-modified 304 that was never exchanged (#839).
The fix reuses the existing server_sent_304 signal (already used to gate the
request/response header carry-over): its persisted proxy is
warc_resphdr != NULL, present only when the swap actually moved real 304
headers across. When it's absent, no server exchange happened this run to
archive, and there's no freshly re-fetched body either to dedup against a
same-run predecessor, so the fix skips the record entirely rather than
writing an identical-payload-digest revisit with nothing behind it to refer
to.
tests/134 and tests/135 extend the mini304/errmask fixtures into a two-pass
--warc-file crawl and assert against tests/warc-validate.py, which gained
--expect-ip and --no-record-for for these checks. Both reproduce their bug
when run against the pre-fix code.
Closes#838Closes#839
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix review findings: an engine-forced not-modified must still count as unbacked
Writing no WARC record at all for an engine-forced not-modified (the #839 fix
in the previous commit) dropped w->unbacked_revisits accounting along with it,
so an all-forced pass wrongly looked fully backed: warc_commit's swap guard let
it replace the previous archive with an (almost) empty one, reopening #777's
exact shape, and warc_cdx_flush then left a stale .cdx pointing into the
truncated file. Fix: emit a self-referencing identical-payload-digest revisit
instead of nothing, still counted as unbacked, so the guard keeps protecting
the previous archive exactly as it does for a real 304.
Classifying a notmodified htsblk by warc_resphdr's presence was too broad: it
also matched back_add()'s cache-priority branch, which never sent a request at
all and is unrelated to the back_wait() hacks #839 targets. Replaced it with an
explicit hts_boolean, warc_forced_notmodified, set only inside back_wait()'s
304-swap block from the already-computed server_sent_304, leaving every other
notmodified path (including back_add()'s) at its default and thus at its prior
behavior.
tests/134 now asserts the exact peer IP instead of mere presence, since a
zeroed-but-AF_INET address also satisfies a non-empty check. tests/135 gained
a second scenario: an all-forced-not-modified rerun against the same archive
name, which is the shape that lets the swap guard's bug actually destroy the
previous archive; the mixed-fixture scenario alone couldn't see it, since a
real 304 alongside it already keeps the guard's counter above zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the cache-priority path and gate the digest revisit on OpenSSL
warc_forced_notmodified had exactly one assignment site, inside back_wait()'s
304-swap block. back_add()'s cache-priority branch (-C1, a lock-file resume,
or opt->state.stop) sets notmodified too but never sent a request, so it fell
through to the zero default and got classified as a genuine 304, emitting a
server-not-modified revisit around a synthesized "HTTP/1.1 200 OK" block. Set
the flag there as well: no request there ever means no server 304 to claim.
The identical-payload-digest revisit for an engine-forced not-modified also
assumed a digest was available, but payload_digest_b32() returns nothing
without OpenSSL. Gate the profile on have_pdig; write no record when it's
absent, since there's neither a digest to point a revisit at nor a real
exchange to record as a response, keeping w->unbacked_revisits incremented
either way so the archive-protection guard still holds.
tests/136 drives the cache-priority path with -C1 and checks the same
WARC-Profile assertion as tests/135. Both new branches (135 and 136) split on
HTTPS_SUPPORT so the no-OpenSSL CI leg exercises the no-digest path instead of
skipping it; 135's archive-kept scenario drops --wacz for the same reason, a
package needs OpenSSL for its SHA-256 digests and would otherwise skip the one
thing it exists to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* An empty body looks digest-less too, and the profile tests never checked the real 304
has_payload requires body_len > 0, so a genuinely empty body (a real,
zero-byte file) fell into the same "no digest" branch as a build without
OpenSSL and got silently dropped from the archive instead of a proper
identical-payload-digest revisit. A zero-length body has a well-defined
digest (sha1 of nothing); compute it directly for that case instead of
reusing has_payload's gate.
tests/135 asserted mini304's pages were revisits but never which profile, so
forcing warc_forced_notmodified true unconditionally (a regression that would
mislabel or suppress a real 304) still passed. Added the server-not-modified
assertion for both mini304 pages, and a matching zero-length fixture
(errmask/empty.dat) plus its identical-payload-digest assertion. tests/136
gained a body-hex check on the fresh archive: its update-pass checks only
assert absence, so a warc_write_transaction stub that writes nothing at all
went undetected on the no-openssl leg.
The unbacked-revisits counter still increments even when nothing gets
written (load-bearing: it's what keeps the swap guard from letting an
all-unchanged pass overwrite a good archive with an empty one), so the
"this pass revisited N URL(s)" log line no longer claims those URLs got a
named body; it now says the archive doesn't hold their current content,
true whether a record was written or not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR #794 moved four gmtime() call sites onto a reentrant hts_gmtime()
helper, but a handful of localtime()/gmtime() sites survived: the
progress/log timestamps in htsback.c and htscore.c, ProxyTrack's
timezone lookup in proxy/store.c (which also dereferenced localtime()'s
result with no NULL check), and the RFC822 date helpers in htslib.c.
glibc's gmtime/localtime buffer is process-wide, not per-thread, so two
of the engine's worker threads racing on it can each read back the
other's broken-down time.
Adds hts_localtime() next to hts_gmtime() and converts every remaining
call site.
Separately, time_gmt_rfc822() and get_filetime_rfc822() fell back to
localtime() whenever gmtime() failed, then handed the result to
time_rfc822(), which always appends "GMT" to the formatted string.
Local time labelled GMT is wrong regardless of which thread wins the
race, so both now fail (empty string / return 0) instead of guessing.
gmtime() essentially never fails on a valid time_t, so this path is not
reachable in practice; the new test guards it at the source level
instead of trying to trigger the failure at runtime.
tests/133_engine-reentrant-time.test drives a new -#test=localtime
self-test: a concurrent hts_localtime() stress test against a fixed
reference table (mirroring #794's hts_gmtime test, TZ pinned to a
non-UTC offset so a local/UTC mix-up cannot hide behind a UTC CI
runner), a get_filetime_rfc822() GMT round-trip check, and a source
guard confirming time_gmt_rfc822()/get_filetime_rfc822() never
reference localtime() again. Verified red on two independent mutants:
reverting hts_localtime() to a raw localtime()-and-copy corrupted
17549/400000 concurrent conversions, and reintroducing the local-time
fallback tripped the source guard.
Closes#806
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A PROPFIND on an exact cache entry crashes proxytrack
PT_Enumerate() reports a folder's default document as a zero-length name,
and the WebDAV listing loop read thisUrl[thisUrlLen - 1] on it, four
gigabytes past the string. One unauthenticated request took the whole
listener down.
Closes#828
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the new skip on Windows and tighten the listing assertions
The Windows job runs *_local-*.test and compares the skip list against an
exact string, so an unpinned skip fails the leg with fail=0, which reads
like a flake. Assert the href and the response count too: displayname
alone comes from the href's trailing component, so it cannot see a wrong
path above it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The .bak re-fetch backup destroys a mirrored file of the same name
The re-fetch backup, the .z content-coding spool and its .u decode
target were all named by appending an extension to the local save name,
which is a name the mirror can hold. A site serving <path>.bak had its
copy renamed over as the backup and then unlinked on commit, and the run
reported success. Build them in an hts-tmp directory beside the file
instead, and remove it once the last slot sharing it is done.
Closes#774
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cache repair deletes the old cache before a rename it never checks
Both zip-repair paths removed the damaged cache, moved repair.zip onto
its name without looking at the result, and announced a successful
recovery either way. A refused move left the entries under a name
nothing reads and no cache at all. Go through hts_rename_over(), which
keeps the destination when it cannot replace it, and report the failure
instead of claiming success.
Closes#786
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* A failed filecreate() on an unknown-length re-fetch commits the backup away
The re-fetch backup was dropped on commit without checking that the new
copy exists. filecreate() can fail after the backup was taken (EMFILE,
ENOSPC, EROFS), and back_finalize()'s incomplete-transfer guard is gated
on a known Content-Length, so a chunked response went straight to the
commit and lost both copies. Restore the backup instead when there is
nothing to commit to.
Closes#775
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* hts_rename_over() can lose its destination when the retried rename fails
The unlink-then-rename fallback leaves nothing in place of dst between
the unlink and the retry, so a retry that fails too loses it. Park dst
under a free scratch name instead, drop it once the move succeeded, and
put it back otherwise.
Closes#790
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Give the cache-repair test Windows coverage
The Windows job iterates a fixed glob, so a 106_engine-* test never ran
there; name it in the list. The move it exercises is the one whose
failure mode is Windows-specific.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Give the re-fetch backup test Windows coverage
The Windows job iterates a fixed glob, so a 108_engine-* test never ran
there. The rename semantics this drives are exactly what differs on the
CRT, which reserves EACCES for a locked source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: check the restore, probe in UTF-8, state the honest guarantee
The move back out of the parked name was unchecked, so a retry that
failed for a reason that still applied left dst absent with the content
orphaned under a name nothing reported. Check it, retry once, and name
the parked copy in the log; hts_rename_over() takes an httrackp for that.
The aside probe used fexist(), which is not UTF-8 and consults the ANSI
codepage on Windows while the renames beside it are wide. It also reads
a directory as a free name, so the park now skips a name whose rename
refuses rather than giving up on it.
The header claimed a failure leaves dst as it was, which the crash
window between the two renames does not give.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Reserve hts-tmp and hts-cache so no URL can name a temporary
Moving the temporaries into an hts-tmp directory only pushed the
collision one level down: hts-tmp is a path segment a URL can name, so a
site serving /d/hts-tmp/a.bin.bak still landed on the backup path for
/d/a.bin and was consumed. url_savename now escapes hts-tmp and
hts-cache as whole path components, the way it already escapes the DOS
device names, which also stops a URL from naming hts-cache/new.zip.
The three tests now drive the hostile shape rather than the pre-fix
name: a savename assertion on the escape, a crawl serving
/bakname/hts-tmp/a.bin.bak, and a selftest pinning the temporary's name.
Also: retry the backup once when another slot removed the shared
directory between structcheck and the rename; report a commit that had
to restore, so the caller does not cache the new validators against the
old body; drop the coded body on the too-long branch as its sibling
does; and scope the leftover scan to plants inside hts-tmp only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* The aside fallback parked a directory that stood in the way
Windows refuses every rename onto an existing target, so the fallback is
production code there rather than the rare path it is on POSIX. A
directory at the destination was renamed aside like a file, the move
then succeeded, and UNLINK could not drop the parked directory, so the
call reported success where master had reported failure and left an
orphan behind. 101_local-update-stale-bak plants exactly that shape and
caught it on both Windows legs.
Park a regular file only. A directory in the way is refused, as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A 304 during --update leaks the whole previous htsblk
back_wait() handles a 304 by replacing the response struct with the cache
entry, carrying only the socket and keep-alive members across via
back_connxfr(). The struct assignment drops every owned pointer the live
response still held without freeing any of them: the 8 KB header buffer on
every update, plus the two WARC header stashes when --warc-file is on. An
update over a 10k-page site drops roughly 80 MB in one run.
back_clear_entry() already knew how to tear those down, so the frees move into
a helper that both it and the 304 path call.
The new test runs the two-pass mini304 crawl with LeakSanitizer on, which the
sanitized CI job otherwise disables. The fresh first pass is the control: it
has no cache entry to read back and is clean either way. The update pass
reports 16 KB in 2 objects on master, one per unchanged URL, and nothing with
the fix.
Closes#782
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the test header
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: reuse deleteaddr(), and cover the WARC limb
back_free_response() was reimplementing deleteaddr(), which already frees adr
and headers and NULLs both; call it instead so the two cannot drift.
Test 114 never passed --warc-file, so the warc_free_request() limb ran with
both pointers NULL on every path it exercised and deleting it kept the test
green. A third pass turns the archive on, and it now fails with the 835 and 238
byte stashes when that call goes away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Register the new leak test as an expected Windows skip
The Win32/x64 job pins the exact set of tests allowed to skip, so an
all-skipped suite cannot report green. 114_local-update-304-leak needs a
LeakSanitizer build and MSVC has no equivalent, so it skips there and tripped
the gate with fail=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* A 304 revisit archives none of the exchange it stands for
The cache entry overwrites the whole htsblk in back_wait(), taking the
stashed request and response headers with it, so the revisit record gets
a synthesized "HTTP/1.1 200 OK" block and no request record. Both blocks
now move across the swap, but only for a real 304: the engine also forces
HTTP_NOT_MODIFIED by itself, and those have no 304 to carry. Since a 304
declares no Content-Type, warc_write_transaction() takes the resource type
from the caller so revisit CDXJ lines keep their mime.
cache_read_including_broken() returns the entry's response struct after
back_clear_entry() has freed it; adr, headers and location come back NULL
now, as the normal cache_readex() path already returns them.
Closes#826
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Do not expand an empty bodyargs array under macOS bash 3.2
Test 122 asks local-crawl.sh only for the revisit checks, so it is the
first caller to leave bodyargs empty. Bash before 4.4 calls that unbound
under set -u, which passes on Linux and fails the macOS leg.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the opt argument to hts_rename_over call sites
Lost when the master merge took this file wholesale; #816 gave the function an
httrackp parameter and the three call sites here reverted to the old form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ProxyTrack's .ndx reader trusted an unparsed offset and the length of a path
An .ndx entry whose position field does not parse left "pos" untouched, so it
kept the offset of the entry before it, and the first entry read whatever the
stack held. The cache then resolves that URL to a different record, or to
nothing at all.
The base path and the .dat/.ndx filenames were built with strncat calls
bounded by the source path rather than by their 1024-byte destinations, so a
cache named on the command line with a long enough path overran them: ASan
reports a heap overflow on the filenames, UBSan an out-of-bounds index on the
base path right after. The zip reader carried its own copy of the same base
path code, reachable once the archive opens, and both now share one bounded
helper. A path that cannot fit is dropped rather than clipped, since a
truncated one names a different file.
Closes#825
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Skip the deep-path half where the filesystem will not hold the path
macOS caps a single path at 1024 bytes, so the 1200-byte tree the base-path
cases need cannot be created there and mkdir -p would fail the test outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* hts_newthread() leaves the pthread attributes object undestroyed when thread creation fails
hts_newthread() folds pthread_attr_init(), pthread_attr_setstacksize() and
pthread_create() into one short-circuit condition, and destroys the attributes
object only on the branch where all three succeeded. If setstacksize or create
fails (EAGAIN under thread or memory exhaustion being the realistic case), attr
stays initialised for good. glibc allocates nothing for a default-initialised
attr, so nothing actually leaks on Linux. This is a portability and hygiene fix
rather than a live bug.
The init has to come out of the chain instead of just gaining a destroy call:
the shared condition cannot tell which of the three failed, and destroying an
object pthread_attr_init() never initialised is undefined.
The imbalance is invisible to a leak checker, so the test counts it directly. A
small LD_PRELOAD shim refuses pthread_create() and records which attributes
objects nobody destroyed afterwards: 1 undestroyed on master, 0 with the fix.
Closes#772
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: tighter comments, and only attr-bearing refusals count
Condense the two-line note above pthread_attr_init(), fix the "so that is can
be independent" typo on the line the diff moves, and count a refused spawn as
tested only when it carried an attributes object, so a NULL-attr create from
elsewhere cannot satisfy the guard on its own. Note why the shim's fixed slot
count and unlocked counters are enough.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Backward scans from strlen(s) - 1 walk off the front of the buffer
optinclude_file()'s config-line right trim and url_savename()'s collision
rename both start at the last character and decrement with no lower bound.
An all-space httrackrc line and an all-digit save name walk below their
stack buffers; the second is reachable from a crawl.
Both go through hts_rtrimlen(), which counts down from the end. The
Content-Disposition trim was a third copy and now shares it.
Closes#814
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* x + strlen(x) - 1 points before the buffer on an empty string
The pointer spelling of #770. All 27 occurrences go through
hts_lastcharptr(), which clamps to the terminating NUL, and the two
hand-written ternary guards from #729 and #767 fold into it.
Closes#781
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the collision rename on Windows and from a followed link
116_engine-rtrim ran nowhere on Windows: that job iterates a fixed glob
that 116_engine-* does not match, and the rename is savename logic, which
behaves differently there. Add it by name.
The test only drove the rename through -g, which pins depth to 0 and so
needs both colliding URLs on the command line. Under -N "%n.%t" depth is
unrestricted, so one starting URL is enough and the crawled page picks
both names itself. Also strip CR before the empty-option check, which a
CRLF log would otherwise pass vacuously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* size_t i = strlen(s) - 1: the (i > 0) guard cannot see the underflow
An unsigned index seeded with strlen(s) - 1 becomes SIZE_MAX on an empty
string, and SIZE_MAX passes the (i > 0) the sites carry. Seed from
hts_lastcharoffset() instead, and cover the spelling in the lastchar
self-test and its source guard.
Closes#821
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Drop the historical narration from hts_lastcharptr's contract
The declaration is the API surface, so it keeps the double-evaluation
warning a caller cannot derive from the signature; what the macro
replaced is git's job. Its grep exclusion in the test goes with it,
since that comment was the only htssafe.h line the scan matched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Catch the bare-reassignment spelling in the source guard
The declaration pattern cannot see punycode.c, where input_length is
declared a dozen lines above the subtraction, so reverting that fix alone
left the guard silent. Match an uncast assignment too; the (int) casts in
htscore.c and htstools.c are the safe form and stay out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ProxyTrack's .arc writer overflowed its 8192-byte header block
Converting a cache to .arc builds each entry's response headers with sprintf
into a fixed char headers[8192]. The one bound guarding the cached header
block compared its length against sizeof(headers) - strlen(headers) - 1, a
subtraction that wraps once the string reaches 8192 bytes, so the guard passed
precisely when it should have failed. The Location: write and the closing CRLF
had no bound at all. Every value comes back off a cache, and an .arc whose
entry carries enough unknown header lines is enough: ASan reports a 12 KB
stack write.
Each append now clips to the room left, holding back four bytes for the blank
line that ends the block and for the CRLF a clip landing mid-line still owes
its line. Clipping instead of aborting is deliberate: these are cache reads,
where a short record beats killing the conversion.
Closes#820
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bracket the .arc header sweep around the measured cap
The fixed 7950..8250 window was 300 conversions picked to survive any change
to what the writer emits on its own. Measure that from the control entry
instead and sweep 32 bytes either side of the cap, which is the same coverage
in a fifth of a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Report a clipped .arc header block, and tighten the test around the real cap
Review follow-ups on the header-block bound. Truncating a cached header block
was silent, so a short record downstream had nothing pointing back at it; the
writer now names the URL on stderr, the way the readers already report a
corrupted entry. The test's own cap assertion allowed 8192 where the block
tops out at 8191, so a one-byte overflow would have passed it. runlen() had
been copied from the long-fields test and now lives in testlib.sh, and the
comment on the four reserved bytes says what depends on them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Backward scans from strlen(s) - 1 walk off the front of the buffer
optinclude_file()'s config-line right trim and url_savename()'s collision
rename both start at the last character and decrement with no lower bound.
An all-space httrackrc line and an all-digit save name walk below their
stack buffers; the second is reachable from a crawl.
Both go through hts_rtrimlen(), which counts down from the end. The
Content-Disposition trim was a third copy and now shares it.
Closes#814
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the collision rename on Windows and from a followed link
116_engine-rtrim ran nowhere on Windows: that job iterates a fixed glob
that 116_engine-* does not match, and the rename is savename logic, which
behaves differently there. Add it by name.
The test only drove the rename through -g, which pins depth to 0 and so
needs both colliding URLs on the command line. Under -N "%n.%t" depth is
unrestricted, so one starting URL is enough and the crawled page picks
both names itself. Also strip CR before the empty-option check, which a
CRLF log would otherwise pass vacuously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* x + strlen(x) - 1 points before the buffer on an empty string (#819)
* x + strlen(x) - 1 points before the buffer on an empty string
The pointer spelling of #770. All 27 occurrences go through
hts_lastcharptr(), which clamps to the terminating NUL, and the two
hand-written ternary guards from #729 and #767 fold into it.
Closes#781
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Drop the historical narration from hts_lastcharptr's contract
The declaration is the API surface, so it keeps the double-evaluation
warning a caller cannot derive from the signature; what the macro
replaced is git's job. Its grep exclusion in the test goes with it,
since that comment was the only htssafe.h line the scan matched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cache repair deletes the old cache before a rename it never checks
Both zip-repair paths removed the damaged cache, moved repair.zip onto
its name without looking at the result, and announced a successful
recovery either way. A refused move left the entries under a name
nothing reads and no cache at all. Go through hts_rename_over(), which
keeps the destination when it cannot replace it, and report the failure
instead of claiming success.
Closes#786
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* hts_rename_over() can lose its destination when the retried rename fails
The unlink-then-rename fallback leaves nothing in place of dst between
the unlink and the retry, so a retry that fails too loses it. Park dst
under a free scratch name instead, drop it once the move succeeded, and
put it back otherwise.
Closes#790
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Give the cache-repair test Windows coverage
The Windows job iterates a fixed glob, so a 106_engine-* test never ran
there; name it in the list. The move it exercises is the one whose
failure mode is Windows-specific.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: check the restore, probe in UTF-8, state the honest guarantee
The move back out of the parked name was unchecked, so a retry that
failed for a reason that still applied left dst absent with the content
orphaned under a name nothing reported. Check it, retry once, and name
the parked copy in the log; hts_rename_over() takes an httrackp for that.
The aside probe used fexist(), which is not UTF-8 and consults the ANSI
codepage on Windows while the renames beside it are wide. It also reads
a directory as a free name, so the park now skips a name whose rename
refuses rather than giving up on it.
The header claimed a failure leaves dst as it was, which the crash
window between the two renames does not give.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* The aside fallback parked a directory that stood in the way
Windows refuses every rename onto an existing target, so the fallback is
production code there rather than the rare path it is on POSIX. A
directory at the destination was renamed aside like a file, the move
then succeeded, and UNLINK could not drop the parked directory, so the
call reported success where master had reported failure and left an
orphan behind. 101_local-update-stale-bak plants exactly that shape and
caught it on both Windows legs.
Park a regular file only. A directory in the way is refused, as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* An FTP --update resumed a complete mirror with REST and spliced the old file into the new body
FTP sent REST whenever the mirrored file merely existed, and on an --update
pass every previously mirrored file exists, so a complete copy was treated as
an interrupted download. The server resumed at its length and the mirror ended
up part old body, part new tail, at exactly the remote size, so nothing
downstream noticed. Resuming now follows the decision back_add() already makes
for HTTP, which only marks a copy partial when the cache does not hold it, and
r.size is seeded from the resume offset so a genuine resume is no longer
reported "FTP file incomplete".
Closes#798
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Format the two comment lines the earlier pass missed
git-clang-format only sees the diff present when it runs; the comments were
translated after it, so those lines never went through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* An FTP resume spliced a changed remote when there was no cache entry
With no cache entry to tell a partial from a stale copy, back_add() took
range_req_size from the file size and the transfer resumed with REST, so a
remote that had changed since had its tail appended to the old head at exactly
the remote length.
FTP has no conditional retrieval, so the client has to decide. Resuming now
needs SIZE to report a remote strictly longer than the local copy and MDTM to
report it no newer than that copy. The mirror is stamped from MDTM as the HTTP
path is stamped from Last-Modified, so later passes compare two server-clock
times; a server that does not answer MDTM proves nothing and is re-fetched
whole.
Closes#823
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Clear r.lastmodified before an FTP attempt
run_launch_ftp() already resets msg, statuscode and size for a retry; without
lastmodified in that list a retry whose MDTM fails stamps the mirror with the
previous attempt's date.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Require the copy's date to match MDTM, not merely to lead it
"Remote no newer than the local copy" is one-sided, and it lets through exactly
the mirrors #823 is about: nothing stamped an FTP file before this branch, so
every existing one carries a client-clock mtime ahead of any MDTM, as does a
copy restored from an archive or moved without preserving times. The first pass
over such a tree resumed and spliced.
The date now has to match, which is what the HTTP path gets from the server via
If-Unmodified-Since. A partial written here is stamped from the same MDTM, so
the resume it exists for still compares equal.
Pass 6 plants a stale copy dated by the local clock over a larger changed
remote; it splices under the old comparison. Pass 5 now dates its copy as the
remote so the missing MDTM is the only thing refusing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: fix TESTS list order and backslash after the union merge
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* --single-file dropped the fragment on an inlined reference
sf_resolve cuts a reference at '#' to find the mirrored file, and sf_inline
then replaced the whole reference with the data: URI it built, so the fragment
never came back. On an SVG that changes what renders: "sprite.svg#icon" selects
one element, and without the selector the browser draws the whole sheet.
Re-attach it to both replacements, the data: URI and the rebased path an
over-cap asset falls back to. The query stays dropped: it named the remote
resource, not the mirrored file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Do not re-encode escapes the fragment already carried
The fragment is copied out of the document, so its '%' and '&' are the
document's own escapes; percent-encoding them again turns "#a&b" into a
lookup for "a&b". A mirrored path is the opposite case, a raw filesystem
name that has to be escaped, '#' included now that a fragment can follow it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the test header and drop an unused top_srcdir
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin every member of the fragment escape set
Only ')' and the '%'/'&' pass-through were exercised, so dropping '"', a
quote, a paren, a backslash, '<'/'>', whitespace or the high-byte rule from
sf_append_escaped left the suite green. The two that would be a real
injection are the '"' that ends the attribute the rewriter re-quotes and the
whitespace that ends an unquoted url() token; the fixture now carries one
fragment per class, and each one dies to its own mutant.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Generate the test list instead of hand-maintaining it
Every branch that adds a test edited the same TESTS list in tests/Makefile.am,
so any two in flight collided there, and resolving that by taking one side drops
the other test silently -- nothing downstream notices a name that stopped being
listed. bootstrap now writes tests/tests-list.mk from tests/*.test and
Makefile.am includes it, so a test is registered by existing.
The generated file is untracked and shipped by make dist, the same arrangement
configure and the Makefile.in files already use, so tarball builds are unchanged.
Refs #844
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Track the test list rather than generating it at bootstrap
The first attempt had ./bootstrap generate tests/tests-list.mk, which broke
every build: CI regenerates with autoreconf -fi and Debian with dh_autoreconf,
neither of which runs bootstrap, so automake aborted on the missing include.
Track the fragment instead. One "TESTS += name" per line with no continuation,
and merge=union scoped to that file, so two branches adding a test union cleanly
instead of colliding -- the shape matters, since unioning a backslash-continued
list drops an entry silently, which is why #843 was closed.
EXTRA_DIST does not need the fragment; automake already ships an included file
via am__DIST_COMMON.
Refs #844
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Correct the comment that still described the generated design
tests/Makefile.am claimed the list was generated and that a test registers by
existing. Neither is true after the pivot to a tracked file, and a contributor
who believed it would add a .test and have it silently skipped.
Also record the one hazard union cannot avoid: a deletion merged alongside
another branch's append is restored, so a removal wants a commit of its own.
Refs #844
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Reading an .ndx cache with full-length URL fields reads past the buffer
binput() returned count + 1 unconditionally, assuming it had consumed a
separator. When it stopped on the buffer's terminating NUL, or because the
destination filled, that byte was not a separator and the caller's cursor
moved one past it. In PT_LoadCache__Old the cursor then leaves the heap
buffer holding the .ndx and the next binput() call reads out of bounds; the
loop's a < (use + ndxSize) guard cannot see it, since the drift happens
after the check.
Step over the byte only when it really is a newline. cache_brstr() carried
a caller-side patch for the same drift and had to follow, or its adr[off-1]
probe would read before the buffer once binput() can return 0.
Closes#793
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Broaden test 117 and make its bookkeeping fail closed
The trigger is the walk reaching the buffer's terminating NUL, which any
.ndx not ending in a newline does; full-length fields are one route to it,
not the only one. Add a field that stops short of every bound, and a pair
straddling cache_brstr's own 256-byte field, the second destination the
diff touches.
items() returned 0 whenever the item-count line was missing, so a crawl
that printed nothing passed the zero-key assertion. Signals now clean up on
their own trap line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Rename test 117's zero-byte fixture off the Windows NUL device
Windows resolves NUL.<anything> to the null device, so nul.ndx never
existed there and proxytrack reported an unloadable index. The suite's
Win32 leg caught it once items() stopped treating a missing count as zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A 304 during --update leaks the whole previous htsblk
back_wait() handles a 304 by replacing the response struct with the cache
entry, carrying only the socket and keep-alive members across via
back_connxfr(). The struct assignment drops every owned pointer the live
response still held without freeing any of them: the 8 KB header buffer on
every update, plus the two WARC header stashes when --warc-file is on. An
update over a 10k-page site drops roughly 80 MB in one run.
back_clear_entry() already knew how to tear those down, so the frees move into
a helper that both it and the 304 path call.
The new test runs the two-pass mini304 crawl with LeakSanitizer on, which the
sanitized CI job otherwise disables. The fresh first pass is the control: it
has no cache entry to read back and is clean either way. The update pass
reports 16 KB in 2 objects on master, one per unchanged URL, and nothing with
the fix.
Closes#782
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the test header
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: reuse deleteaddr(), and cover the WARC limb
back_free_response() was reimplementing deleteaddr(), which already frees adr
and headers and NULLs both; call it instead so the two cannot drift.
Test 114 never passed --warc-file, so the warc_free_request() limb ran with
both pointers NULL on every path it exercised and deleting it kept the test
green. A third pass turns the archive on, and it now fails with the 835 and 238
byte stashes when that call goes away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Register the new leak test as an expected Windows skip
The Win32/x64 job pins the exact set of tests allowed to skip, so an
all-skipped suite cannot report green. 114_local-update-304-leak needs a
LeakSanitizer build and MSVC has no equivalent, so it skips there and tripped
the gate with fail=0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* hts_rename_over() can lose its destination when the retried rename fails
The unlink-then-rename fallback leaves nothing in place of dst between
the unlink and the retry, so a retry that fails too loses it. Park dst
under a free scratch name instead, drop it once the move succeeded, and
put it back otherwise.
Closes#790
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Review fixes: check the restore, probe in UTF-8, state the honest guarantee
The move back out of the parked name was unchecked, so a retry that
failed for a reason that still applied left dst absent with the content
orphaned under a name nothing reported. Check it, retry once, and name
the parked copy in the log; hts_rename_over() takes an httrackp for that.
The aside probe used fexist(), which is not UTF-8 and consults the ANSI
codepage on Windows while the renames beside it are wide. It also reads
a directory as a free name, so the park now skips a name whose rename
refuses rather than giving up on it.
The header claimed a failure leaves dst as it was, which the crash
window between the two renames does not give.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* The aside fallback parked a directory that stood in the way
Windows refuses every rename onto an existing target, so the fallback is
production code there rather than the rare path it is on POSIX. A
directory at the destination was renamed aside like a file, the move
then succeeded, and UNLINK could not drop the parked directory, so the
call reported success where master had reported failure and left an
orphan behind. 101_local-update-stale-bak plants exactly that shape and
caught it on both Windows legs.
Park a regular file only. A directory in the way is refused, as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The FTP worker writes url_sav itself, so its slot carries a size but no
in-memory body. Serializing that slot to the on-disk ready table stores no
body, and the read-back took the size from what it stored, leaving zero: the
link writer then saw an empty response and created a 0-byte file over the
bytes already on disk, while the engine logged the transfer as a success.
Test 110 mirrors twelve files at -c8, which is what makes a ready slot wait
long enough to be swapped, and -#test=backswap covers the round-trip directly.
Closes#797
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* PT_GetTime copied gmtime's shared static instead of a reentrant breakdown
On _WIN32 the success path took gmtime()'s pointer and dereferenced it after
the fact, so a concurrent conversion on another thread could change the
breakdown under it. The POSIX branch was already reentrant via gmtime_r, and
the same #ifdef pair had been copy-pasted into hts_now_iso8601() and the WARC
auto-name; fold all of them onto one hts_gmtime() helper, and give ProxyTrack's
WebDAV listing the same treatment, since it read the static's fields well past
the call.
Windows uses Microsoft's gmtime_s (destination first, errno_t return), not the
C11 Annex K function of the same name.
Covered by a new "gmtime" engine self-test: a reference table checks the
breakdown itself, which is what catches a swapped-argument call on the MSVC
leg, and eight threads hammering the helper catch a return to the shared
static (16k of 400k conversions corrupt with that mutant in place).
Closes#794
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Give the gmtime self-test teeth on the failure path and off UTC
Three holes the test-design audit found by running mutants rather than
reading the diff.
A helper that discarded gmtime_r's NULL and always claimed success passed
every phase, yet that boolean is the only failure signal hts_now_iso8601,
warc_open and PT_GetTime have; all three would have formatted an
uninitialised struct tm. A forced-failure row now converts INT64_MAX, gated
on a 64-bit time_t.
The localtime_r mutant only died on a non-UTC box. CI runners are UTC, where
localtime_r and gmtime_r agree on every reference row, so the test exports
TZ=XXX5.
The "first result survives the second call" phase could not fail: both
buffers are caller-owned stack storage no implementation writing through
tmbuf could disturb. Removed rather than left reading as coverage.
expect_ok() was a third byte-identical copy; it moves to tests/testlib.sh
with the two existing callers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Rename the self-test's out-of-range time_t off the "far" keyword
WinDef.h defines "far" away to nothing, so the declaration lost its variable
and MSVC rejected the file. Both Windows legs caught it; the POSIX builds
never see the macro.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(void) does not suppress glibc's warn_unused_result, so gcc warned on
the read-back in st_logcallback. Treat a failed read as the test failure
it is instead of asserting against an empty buffer.
Closes#812
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Match captured output with a here-string, not a pipe into grep -q
grep -q exits on the first match, so whatever the producer still had to
write takes SIGPIPE; under pipefail that becomes the pipeline's status and
an assertion that held reports failure. bash issues one write() per line,
so any match that is not on the last line is exposed.
Converts every test assertion whose producer is a shell builtin or shell
function, including two pipelines used as an if condition where the SIGPIPE
silently flips the branch. Generalizes the AGENTS.md bullet, which only
covered the "&& fail" spelling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Convert the remaining pipe-into-grep -q sites
The self-test drivers survive only because the matched line is the last
thing httrack prints; one added line of output turns a pass into 141.
Nothing in tests/ pipes into grep -q now.
The two zlib drivers claimed the harness might run them under a POSIX
/bin/sh: it does not. configure resolves $(BASH) to bash, test-timeout.sh
execs it, and 01_zlib-warc-wacz.test already uses "set -o pipefail" (which
dash lacks) on the macOS leg. Kept the half that is true, BSD tool flags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The -o help text promises a generated error page the engine never builds
`-o` only decides whether the error page the server sent survives: `store_errpage`
keeps `r.adr` alive so the normal save path writes it, and the `-o0` arm frees it.
Nothing anywhere builds a stand-in body. The one block that would have was dead
since the 3.20.2 import and was removed in #783.
Reword the help line, the man page and fcguide's two `-o` prose blocks to say the
server's error page is saved rather than generated, and extend 23_local-errpage
so the mirrored 404 has to carry the server's own body.
Closes#787
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Condense the -o1 control comment to one line
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* An FTP --update resumed a complete mirror with REST and spliced the old file into the new body
FTP sent REST whenever the mirrored file merely existed, and on an --update
pass every previously mirrored file exists, so a complete copy was treated as
an interrupted download. The server resumed at its length and the mirror ended
up part old body, part new tail, at exactly the remote size, so nothing
downstream noticed. Resuming now follows the decision back_add() already makes
for HTTP, which only marks a copy partial when the cache does not hold it, and
r.size is seeded from the resume offset so a genuine resume is no longer
reported "FTP file incomplete".
Closes#798
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Format the two comment lines the earlier pass missed
git-clang-format only sees the diff present when it runs; the comments were
translated after it, so those lines never went through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The changed-lines clang-format job resolved its base as origin/<base_ref>,
which is master's tip when the job runs. Once master gains a C commit while
a PR is open, the comparison also picks up the reverse of that commit and
the job fails on code the PR never touched.
Use git merge-base instead, and fail loudly if there is none rather than
falling back to a whole-tree comparison.
Closes#800
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
hts_log_vprint() makes a va_copy and then throws it away: the callback is handed
the original args, and vfprintf() writes the log file from that same,
already-consumed list. On x86_64 a va_list is a one-element array, so the callee
moves the caller's cursor; the second traversal reads past the register save
area, and a %s yields a junk pointer that vfprintf() dereferences.
Only an embedder that installs a callback is affected, so the CLI never sees it.
HTTrack Android does: --sitemap is the first option whose LOG_NOTICE lines carry
arguments, and ticking its checkbox segfaults the crawl thread. The same crawl is
clean under the CLI built with ASan+UBSan.
Pass the copy to the callback. -#test=logcallback sends one line with a %d and a
%s through both sinks and compares them; on the unfixed code the log file gets
"0 " and the test fails. It also logs a second line below opt->debug with no
opt->log, pinning that the callback fires above the level filter.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Under `set -e` a failing command in an EXIT trap becomes the script's exit status, so a hiccup while tearing down fixtures fails a test whose assertions all passed. That is what turned `57_local-proxy-connect.test` red on the Windows x64 leg of #765: five OK lines, no FAIL, exit 1. Every EXIT trap in the suite now runs teardown with errexit off, and the signal traps keep their own `trap` line, since sharing `set +e` with HUP/INT/QUIT/PIPE/TERM would leave errexit off for the rest of a signalled run and let a torn-down test still report success.
A `|| true` on the `rm` would have been smaller, but it throws away the only diagnostic, and the evidence does not say which teardown command failed: a blocked `rm -rf` on the Windows runner exits 1 and prints "Device or resource busy", while the log shows exit 1 and nothing at all. The sharing violation in the issue is the plausible mechanism rather than a confirmed one, so whatever it really is now prints its own error.
`99_teardown-status.test` pins the semantics both ways and scans the suite so a new test cannot reintroduce the shape, the leaky combined trap included. The `return 0` that three `cleanup()` bodies ended with never protected anything, since errexit fires at the failing command before it is reached.
Closes#773
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A test that wedges runs until CI cancels the step, and a cancelled step keeps neither its log nor the artifacts its `if: always()` uploads would have produced. That is why nobody has ever been able to say which Windows test hangs, across 19 dead jobs in the last day alone (#795).
Each test now runs under a wall-clock budget at the automake harness level, so an overrun names the test, dumps the surviving process tree and an engine stack, and exits 124. The step then fails rather than being cancelled, which is what keeps the log. Every POSIX `make check` leg gets this; the Windows leg runs its own serial loop and now calls the same wrapper. The budget is 600s, the value the Windows leg already used: it has to clear the 540s a three-pass crawl may legitimately take under `local-crawl.sh`'s own 180s-per-pass watchdogs, against a slowest healthy test that actually measures 39s.
Two things sit on top of the per-test bound. The Windows suite gives up at 25 minutes so it fails on its own terms well before the 45-minute step timeout, and it sweeps leaked engine processes between tests, naming whichever test left them. An orphaned `httrack.exe` starving the runner is the leading theory for the hang, and that sweep is what would confirm it.
This also fixes an unbounded `wait` after `kill_tree` in testlib.sh. When the kill failed to reap, which is exactly the native-Windows case those watchdogs exist for, the watchdog blocked forever and never printed the timeout it was about to report.
Stacks differ by platform, and each branch says which one it took, because a dump that silently produces nothing reads as coverage. Linux sends SIGABRT and lets httrack's own crash handler symbolize itself, verified against a real wedged crawl where it named `back_wait` at htsback.c:2710. macOS cannot do that, since htsbacktrace.c gates the handler on `__linux`, so it uses `sample(1)`. Windows uses `cdb` from the SDK, and that is the one path I could not exercise from here; it probes for the binary, reports when it is absent, and is bounded so a debugger that wedges cannot become the new hang.
Does not fix#795, only makes it diagnosable.
The between-test sweep costs the Windows step about two minutes (506s before, 619s on this run). That buys naming whichever test leaks, which is the whole lead on #795; it can be narrowed or dropped once the leak is found.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A 304 revisit record carried a `WARC-Profile` and nothing else naming what it stood for. Neither replay engine reads that field: pywb resolves in `_load_different_url_payload` on `WARC-Refers-To-Target-URI` and raises `ArchiveLoadFailed` without it, wabac.js reads `warcRefersToTargetURI` and otherwise answers Not Found. So the records were conformant under WARC 1.1 6.7.3, which only recommends the field, and unreplayable in both engines that matter.
For a server-not-modified revisit the referred-to URI is the record's own target URI, so it is free to emit.
No `WARC-Refers-To-Date` alongside it. The cache persists no capture timestamp: the field list in `cache_add()` ends at `Last-Modified`, which is a document property, and the zip entry's own date is set from that same value. Emitting it would assert that a record exists with that `WARC-Date`, which is false and would misdirect pywb's `closest=` lookup. Both engines already handle the field being absent, pywb by falling back to the CDX timestamp and wabac.js to the revisit's own. A real date needs a capture time in the cache, which is worth doing when the segment work lands and a previous index is being read anyway.
The validator now requires every revisit to name a capture, and requires a server-not-modified one to name its own URI. Test 73 already drives it over an archive of revisits; against the pre-fix engine it fails there.
Stacked on #788. Without it the new header line is what pushes a 995 to 1004 byte URL past `wbuf_printf`'s old 1024-byte buffer, and `warc_emit` then drops the record whole; the review caught that before this was pushed. The merge is in the branch so the pair is what got tested.
Closes#778
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every WARC header line is built with `wbuf_printf()`, which formatted into a 1024-byte stack buffer and returned `-1` instead of growing. `"WARC-Target-URI: %s\r\n"` costs 19 fixed bytes, so the line failed once a URL reached 1005 bytes, the `-1` reached the `goto done` in `warc_emit()`, and the record was abandoned. The page still got mirrored, the crawl still exited 0, and nothing was logged, so the only symptom was a URL missing from the archive.
`wbuf` reallocs already, so oversized output now formats straight into it after a `wbuf_reserve()`, which is the growth half of `wbuf_add()` split out. Every field benefits, not only the two carrying URLs. The second pass is bounded against what was reserved rather than trusted: advancing `len` by a return value larger than the reservation would push `len` past `cap` and corrupt the bounds check of every later append.
`-#test=warc-longurl` sweeps 100 to 9000 bytes across the boundary. Against the old formatter it fails at exactly 1005 and up, with 1003 and 1004 passing. Each record carries a distinct payload because identical ones dedupe into revisits, which would otherwise hide the response records the test counts.
One caveat on the sanitizer evidence, since it is easy to over-read. The buffer grows by doubling, so a small off-by-one lands in allocation slack where ASan cannot see it; that is why the second-pass bound is a logic check rather than something left to the sanitizer. The ASan+UBSan run over the sweep is clean, but only after planting a deliberate overflow in `wbuf_reserve` to confirm the probe actually fires. It did not, at first: libtool silently drops `-fsanitize` from the shared-library link, which produced no binary at all and a "clean" result that meant nothing.
Worth knowing for the segment work: `warc_emit()` returning `-1` also sets `w->failed`, which suppresses the archive swap added in #777. Before this fix a single over-long URL would make an `--update` pass throw away its whole archive and keep the previous one.
Closes#785
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An FTP re-fetch called `filecreate()` on the mirrored file before a byte of the transfer had arrived, so a read error, a timeout or a short body destroyed the previous copy. HTTP has moved the good copy aside to a `.bak` since the #77 follow-up and puts it back when the transfer fails; FTP never took that backup. Rather than give it a second copy of the idiom, the backup moves out of `back_wait()`'s direct-to-disk block into `back_refetch_backup()`, which the FTP transfer now calls too. `back_finalize()` already restores it, so FTP inherits that. The REST resume branch appends and needs no backup.
That alone was not enough. `back_cleanup_background()` swaps a ready slot to the on-disk table through `back_clear_entry()`, which unlinks `back->tmpfile`, and a failed re-fetch sits at `STATUS_READY` unfinalized until the parser picks it up. The new `.bak` was being deleted in that window before `back_finalize()` could restore it, about half the time on a loaded box. `slot_can_be_cached_on_disk()` now refuses a slot that still owns a temporary, which closes the same window on the HTTP `.bak` and on the content-coding spool. The crawl-level race needs concurrency the suite cannot pin down, so `-#test=backswap` covers the predicate directly.
The suite had no FTP server. `tests/ftp-server.py` is a minimal one (PASV, SIZE, REST, RETR, LIST) with a mode file the test rewrites between passes, so a path can start failing without the port moving and taking the mirror directory name with it. Test 102 mirrors three files, then re-fetches one cut short, one served empty and one healthy: the first two must come back byte identical and be reported `unchanged`, the third replaced. It runs at `-c1` because a parallel FTP crawl loses whole transfers to #797, which is older and unrelated and would flake it on a loaded runner. #798 came out of the same work: an FTP `--update` sends `REST` over a complete mirror and splices the old file into the new body.
Closes#771
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unlink-then-rename fallback in `hts_rename_over()` is there because Windows' rename() refuses an existing target, but it fired on any failed rename, ENOENT on the source included. A caller moving a temp file it never managed to write lost the destination and got `HTS_FALSE` back, which reads as "nothing happened". #754 unified four hand-rolled copies into this helper, so every call site inherited it.
The unlink now runs only for EEXIST, and only with a source that exists. EEXIST is the value that matters: the CRT maps ERROR_ALREADY_EXISTS there and keeps EACCES for a source another process holds, so accepting EACCES as well would have deleted the destination for a failure it had no part in, and the retry would then fail with the destination already gone. The source check is belt and braces for a CRT that reports neither. `hts_rename_utf8()` preserves errno across its free() calls now, since the gate reads it.
The existing call sites all derive their source's existence from a create that had to succeed first, so the bug was latent rather than live, but three of those destinations are user data: a mirrored file, a finished .wacz, and a rewritten page nothing will re-fetch. What is left of the window after this is #790.
The selftest probes what rename() does to an existing target and asserts against the regime it finds, so it runs three ways on Linux: native, then under an LD_PRELOAD rename() with Windows' shape, then under one reporting a locked source. The harness pins the expected regime per platform, so no leg can pass having exercised the other half. Seven mutants of the gate were each confirmed to fail it.
Closes#779
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three bugs have already come out of the same one-liner: `x[strlen(x) - 1]` indexes one byte before the buffer when the string turns out to be empty (#729, #730, #768). Rather than wait for a fourth, this replaces the idiom everywhere with helpers that cannot underflow.
All 48 occurrences were reclassified by tracing each guard to where it actually lives instead of to the two lines above the index. 44 were already guarded, often a dozen lines up or in the caller, and one sits inside a commented-out block. The rest have no guard at the site, and two of those are reachable from a crawl.
`htscore.c:2194` is the one that matters: a one-byte stack out-of-bounds write in the end-of-mirror purge, confirmed under ASan and UBSan. `linput()` reads `old.lst` in 999-byte chunks, so a 1000-byte line comes back as 999 bytes plus a one-byte tail; for that tail `line + 1` is empty, and without `-O` so is `path_html`, which leaves `file` empty when the index runs. The crawled site controls the save-path length and therefore the line length. It needs a second run over the same project, which is what a re-crawl does. As controls, a 900-byte path and the same path with `-O` both stay clean.
`htsparse.c:2009` is a one-byte overread in the HTML parser, reachable from `<a href=" ">`, where the guard sits on the wrong side of the `&&`. The write on the next line is accidentally safe for the same reason. #768 itself turned out not to be reachable from a crawl, only through the `-#test=savename` hook, so I would not call that one a security fix.
The helpers live in `htssafe.h` beside the other bounded string operations, so no file needed a new include. Guards doing more than an emptiness check are kept, since folding those in would append a separator to an empty string.
`-#test=lastchar` checks the helpers against a poisoned neighbouring byte, including the `/`-before-the-buffer case from #768, and putting the missing length check back makes it fail. It also greps the source, so reverting any converted site fails instead of leaving the self-test green. `97_local-purge-longpath` drives the purge site end to end; on unfixed code the sanitized CI leg aborts at `htscore.c:2194`.
Built from identical source paths, 61 of 72 objects are byte-identical, one differs only in debug info, and the remaining 10 are the files edited.
The pointer spelling `x + strlen(x) - 1` has 25 occurrences and is filed separately as #781.
Closes#770Closes#768
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stand-in body `httpmirror()` builds for an error page has been unreachable since the 3.20.2 import: both guards say the opposite of their comments, so the block runs only when there is no save name and the URL *is* `/robots.txt`. `create_html_warning` is never assigned, so the HTML arm is dead outright. The GIF arm fires only if the user maps `.txt` to `image/gif` with `--assume`, and it then swaps `r.adr` for a 1070-byte buffer while leaving `r.size` at the error body's length, so the `robots_parse()` call below over-reads the heap.
Repair is not the one-character fix the inverted guards suggest. `filesave()` below writes `r.size` bytes, so uninverting the guards without also setting `r.size` moves the same over-read into the user's mirror. A correct repair would then replace every server error body with HTTrack's 2003 template by default, which is what `23_local-errpage.test` asserts is kept. Nobody has asked for the placeholder in twenty years, and #17 asked for the opposite.
`--errpage` itself is untouched: `store_errpage` keeps the server's error body and the normal save writes it. A byte-level differential against a master build over five error shapes under both `-o1` and `-o0` gives identical mirrors, and rebuilt objects differ only in `htscore.o`. The new test drives the one input that reached the block, and fails on master in a plain build rather than only under the sanitizers. Separately, the `-o` help text promises a generated page the engine has never produced; that predates this change and is filed as #787.
Closes#769
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`back_finalize()` moves an existing file aside to `<file>.bak` before truncating it, so an aborted re-fetch can put the previous copy back. That rename was a bare `RENAME`, and Windows refuses to rename onto an existing target, so a `.bak` outliving a killed run made every later re-fetch of that URL fail the rename, take the `tmpfile = NULL` branch with nothing logged, and truncate the live file with no backup at all. The #77 guard was off for that file, silently and for good. `hts_rename_over()` (#754) unlinks and retries, which fixes it.
Worth a look, because clobbering a stale `.bak` is not free. A run killed between the rename-aside and the finalize leaves the previous complete copy in `.bak` and a partial in the live file, and the next re-fetch now overwrites the good one. It still looks like the right trade: POSIX `rename()` has behaved exactly this way since #77 landed, so the alternative is leaving Windows with a guard that stays dead until someone deletes the file by hand, and a leftover `.bak` is engine garbage that nothing advertises or ever restores. The clobber is logged now, so it is at least visible.
Any failure to create the backup is logged too, instead of quietly disabling the safety net. `htscache.c`'s static `hts_rename()` wrapper and the hand-rolled `old.zip` unlink at its only call site go the same way, which removes a copy of the unlink-then-rename idiom rather than adding a fifth.
Test 101 plants both kinds of leftover before the update pass: a stale file, which has to be clobbered so the `-M` abort can still restore the pass-1 copy, and a directory, which cannot be clobbered and has to be reported instead. Only the Windows leg arms the first half, since POSIX clobbers on its own; the second is armed everywhere. Test 37 gains a live update pass so the dead pass rotates onto an existing `old.zip`, which nothing covered.
Two older bugs on the same path came out of the review and are filed separately: #774 (the `.bak` name collides with a mirrored file of the same name, reproduced) and #775 (a failed `filecreate()` on a chunked re-fetch commits the backup away).
Closes#758
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* warc: keep the previous archive when a pass has no bodies to replace it
A second crawl into the same output reopened the WARC with "wb" and
truncated it. On a cache-served pass nearly every URL comes back 304, so
the new file held revisit records whose payloads had just been deleted,
and the regenerated WACZ came out with zero page rows: a package that
replays nothing, in place of one that replayed fine.
The writer now builds into a sibling .tmp whenever an archive is already
there, and only swaps it in at close if the result can stand on its own.
A pass that only revisited URLs it did not re-download keeps the previous
.warc.gz, .cdx and .wacz untouched and says so.
What --update should ultimately mean for WARC output is still open; this
only stops the silent destruction in the meantime.
Closes#759
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* warc: guard the segment swap and cover the rotated archive
A run that lost a record or a segment must not replace a whole archive,
and hts_rename_over unlinks its destination when the source is missing,
so every segment has to be on disk before the first rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* warc: force rotation in the segment test and guard its vacuity
--warc-max-size 2000 never rotated under the harness's --robots=0, so the
segment test was checking a single-file archive; a mutant that renamed
only segment 0 passed it. 600 rotates, and --archive-min-files fails the
test if a shrinking crawl ever stops producing the segments it checks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MD5 comparison the comment promises left the tree in two steps: #467
extracted the function out of htsparse.c's HT_ADD_END macro without the skip
branch, and #512 removed the //[HTML-MD5]// cache entry it read. Drop the
parenthetical; the write is unconditional.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A failed re-fetch overwrote the mirrored file with the aborted read's debris
A transfer that dies before a complete response has no body, yet the save path
still consulted r.adr, which at that point holds whatever the aborted header
read left behind: raw status-line bytes, or an empty buffer that truncated the
file to zero on macOS. Require a successful transfer, as the empty-body half of
the condition already did.
Closes#748
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the failed re-fetch keeps its bytes on every platform
Test 93 filtered reset.bin out of its bucket lists because a connection killed
before the status line surfaced differently per platform. It no longer does, so
assert the resource like any other: unchanged, with the bytes pass 1 mirrored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: make the re-fetch assertions catch a crawl that never re-fetched
The new test passed with no second pass at all, so a regression that stopped
re-fetching would have looked green. Assert the failure the fixture provokes,
give stay.bin a fresh pass-2 body so a fix that stopped overwriting anything
fails, and compare reset.bin by checksum rather than by length.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* --purge-old deleted a file whose re-fetch never got a response
A transfer that dies on the wire leaves the previously mirrored copy in place,
but back_finalize() returned without noting it, so the URL fell out of new.lst
and the end-of-update purge treated the file like a page that had vanished from
the site. Note the surviving copy, the way the incomplete-transfer branch above
already does, for the connection-level failures htsparse.c retries on. A
deliberate skip (too big, MIME-excluded, cancelled) keeps its current fate.
Closes#746
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the copy for every failure, not just the five retryable codes
A malformed status line, an oversized declared length or a mid-flight abort all
land on STATUSCODE_INVALID, outside the retryable set, and the purge still ate
the file. Invert the test: any failure keeps the copy except the codes that
mean the engine passed the resource over on purpose.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the retry exhaustion, not the per-platform failure message
The message a cut connection produces depends on whether any bytes arrived, so
matching it would fail on a runner that sees none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* A failed re-fetch overwrote the mirrored file with the aborted read's debris
A transfer that dies before a complete response has no body, yet the save path
still consulted r.adr, which at that point holds whatever the aborted header
read left behind: raw status-line bytes, or an empty buffer that truncated the
file to zero on macOS. Require a successful transfer, as the empty-body half of
the condition already did.
Closes#748
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the failed re-fetch keeps its bytes on every platform
Test 93 filtered reset.bin out of its bucket lists because a connection killed
before the status line surfaced differently per platform. It no longer does, so
assert the resource like any other: unchanged, with the bytes pass 1 mirrored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: make the re-fetch assertions catch a crawl that never re-fetched
The new test passed with no second pass at all, so a regression that stopped
re-fetching would have looked green. Assert the failure the fixture provokes,
give stay.bin a fresh pass-2 body so a fix that stopped overwriting anything
fails, and compare reset.bin by checksum rather than by length.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the retry exhaustion, not the per-platform failure message
The message a cut connection produces depends on whether any bytes arrived, so
matching it would fail on a runner that sees none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
htsthread_wait_n(background_threads - 1) subtracts one more than the count of
threads that must not be joined. Without --ppid that count is zero, so the wait
asks for a negative number of outstanding threads and the counter never gets
there; with --ppid it waits on the pinger, which by design never returns.
Wait for background_threads instead, which is what the sibling call inside
back_launch_cmd() already passes.
Closes#753
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsAddLink walks back from an empty codebase, one byte before the buffer
Same idiom as the lienrelatif() underflow fixed in #729: the trim that walks
back to the last '/' starts at codebase + strlen(codebase) - 1, which is
codebase - 1 when the string is empty, and the loop dereferences it before
a > codebase stops it.
Unlike #729 there is no reachable empty value. codebase is copied from a
recorded link's fil, and no hts_record_link() call site can supply an empty
one: every fil is either a literal seed or an ident_url_absolute() /
ident_url_relatif() success return, and fil_simplifie() restores "/" or "./"
rather than leaving a path empty. The guard goes in anyway, and -#test=addlink
drives the walk directly: under the sanitize job's ASan+UBSan build the test
fails on the unfixed walk and passes with the guard. Its second case pins the
ordinary trim, which the guard leaves alone.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* review: add the case that actually notices the codebase trim
For an ordinary relative link ident_url_relatif() re-derives the directory
from the path it is handed, so deleting the trim outright left the two
existing cases green. A query-only link ("?x=1") copies that path whole, and
does catch it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* structcheck() builds its rename target with an unbounded sprintf
Both structcheck() and structcheck_utf8() move a regular file sitting where a
directory belongs, and build the "<name>.txt" target with a raw sprintf into a
2048-byte buffer. It stays in bounds only because of a strlen(path) >
HTS_URLMAXSIZE guard dozens of lines above, which nothing at the write site
mentions. Route both through sprintfbuff() and fail with ENAMETOOLONG, so the
bound is local.
Armed the probe: with that distant guard patched out, a 2045-byte path makes
the old sprintf write 2049 bytes into tmpbuf[2048] under ASan; the same build
with sprintfbuff() returns -1 and reports nothing.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* review: tighten the structcheck self-test and its comments
The path builder could end a path with a bare separator when the base
directory's length hit the wrong residue, so the test aborted on a long
$TMPDIR. The refusal case also asserted on a component structcheck never
creates; assert on the outermost one instead.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the max-length rename case, macOS PATH_MAX is 1024
The path the guard admits (HTS_URLMAXSIZE) plus ".txt" is longer than macOS
accepts, so fopen() failed there. The case could not tell a fixed build from an
unfixed one anyway; what is left covers the guard and the rename on both entry
points.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
configure has rejected --without-zlib since #750, and htsglobal.h
#errors on a forced HTS_USEZLIB=0, so the macro can only ever be 1.
Collapse every #if/#ifdef HTS_USEZLIB guard (htsweb.c, htsname.c,
htslib.c, htszlib.c, htsselftest.c, htscodec.c) to its always-taken
branch; htsweb.c's guard used #ifdef where every other site used #if,
an inconsistency that no longer matters once the guard is gone.
htscodec.c's #else arms were a genuine zlib-free content-coding path
(Accept-Encoding: identity, hts_codec_unpack returning -1), not stubs.
Removed for consistency with the other nine guards: the cache
(htscache.c, proxy/store.c) already reaches minizip unconditionally
from ~60 call sites with no null backend, so a zlib-free build is not
actually reachable today regardless of this file.
No new test: this is dead-code removal with no behavior change.
Verified by differential build against master: htscodec.o and
htszlib.o are byte-identical; the other touched objects differ only
in __LINE__ immediates shifted by the removed guard lines (plus one
cosmetic objdump label-annotation artifact each in htsselftest.o and
htsweb.o, from string-literal pool reordering). Exported symbols in
libhttrack.so are unchanged. make check: 166/166 (158 pass, 8 skip).
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
htsselftest.c and tests/Makefile.am were held by #718 while #747 was fixed, so
the thread-counting fix went in without a test. -#test=threadwait covers it
from both sides: a wait placed right after a spawn joins that thread, and
wait_n(n) leaves n running rather than draining them.
One spawn per round is what makes it bite. A batch gives the earlier threads
time to raise the counter themselves, which is why an eight-thread version
passed on the unfixed engine; one thread per round failed 10 runs out of 10.
The changes-race self-test can now drop the counter it kept because
htsthread_wait() could not be trusted to join.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:05:44 +02:00
519 changed files with 21920 additions and 9212 deletions
AC_MSG_ERROR([BASH_SHELL must not contain shell or make metacharacters, got: $BASH_SHELL]) ;;
'' | [[\\/]]* | ?:[[\\/]]*) ;;
*) AC_MSG_ERROR([BASH_SHELL must be an absolute path, got: $BASH_SHELL]) ;;
esac
hts_bash_override=$BASH_SHELL
AC_PATH_PROGS([BASH_SHELL], [bash], [/bin/bash])
# An absolute override is taken verbatim, so BASH_SHELL=/bin/sh would put #895 back and only
# surface at "make check" or "make deb" (#908). What we found ourselves is only a warning:
# a box with no bash still builds, it just cannot run those two.
AC_MSG_CHECKING([whether $BASH_SHELL is a bash outside POSIX mode])
hts_bash_why=
hts_bash_env=no
# AS_EXECUTABLE_P, not "test -x": the PATH search above already demands a regular file, and
# bash blocks forever reading a FIFO it failed to exec, so -x alone hangs configure (#922).
if ! AS_EXECUTABLE_P(["$BASH_SHELL"]); then
hts_bash_why="not an executable regular file"
elif test -z "$("$BASH_SHELL" -c 'echo "${BASH_VERSINFO[[0]]}"' 2>/dev/null)"; then
# Not BASH_VERSION: that is an ordinary variable, so any shell echoes back a spoofed one.
hts_bash_why="not a bash: it reports no BASH_VERSINFO"
else
# sh-mode bash reports a version too, so only SHELLOPTS tells the two apart.
case $("$BASH_SHELL" -c 'echo ":$SHELLOPTS:"' 2>/dev/null) in
*:posix:*)
hts_bash_why="a bash in POSIX sh-mode"
# POSIXLY_CORRECT and an exported SHELLOPTS do that to every bash on the box, so no path
# can pass and blaming this one would send the user hunting for another. Reading them
# here would not do: configure puts its own shell in posix mode, which sets both.
case $(env -u POSIXLY_CORRECT -u SHELLOPTS "$BASH_SHELL" -c 'echo ":$SHELLOPTS:"' 2>/dev/null) in
'' | *:posix:*) ;; # no "env -u", or posix whatever the environment: blame the path
*) hts_bash_env=yes ;;
esac
;;
esac
fi
if test -z "$hts_bash_why"; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
hts_bash_msg="POSIXLY_CORRECT or SHELLOPTS forces every bash into POSIX sh-mode, $BASH_SHELL included. Clear them for configure and for make, which hands them to make check and make deb: env -u POSIXLY_CORRECT -u SHELLOPTS ..."
if test "$hts_bash_env" = yes; then
if test -n "$hts_bash_override"; then
AC_MSG_ERROR([$hts_bash_msg])
fi
AC_MSG_WARN([$hts_bash_msg])
else
if test -n "$hts_bash_override"; then
AC_MSG_ERROR([BASH_SHELL=$BASH_SHELL is $hts_bash_why])
fi
AC_MSG_WARN([no usable bash found: $BASH_SHELL is $hts_bash_why. "make check" and "make deb" need one; pass BASH_SHELL=/path/to/bash])
fi
fi
AC_PROG_CC
AM_PROG_CC_C_O
m4_warn([obsolete],
@@ -48,14 +113,13 @@ m4_warn([obsolete],
# script's behavior did not change. They are probably safe to remove.
AC_CHECK_INCLUDES_DEFAULT
AC_PROG_EGREP
# $(SED) substitutes $(datadir) into src/webhttrack
AC_PROG_SED
LT_INIT
AC_PROG_LN_S
LT_INIT
# bash, used to run the test scripts (see tests/Makefile.am TEST_LOG_COMPILER)
Run one target by hand: `fuzz/fuzz-url -max_total_time=300 corpusdir fuzz/corpus/url`. Seed corpora live in `corpus/<target>/`; a crash reproducer is replayed with `fuzz/fuzz-url crash-file`.
`fuzz-arc` is the odd one out: it drives proxytrack's `.arc` reader the way `--convert` does, through a temp file rather than a buffer, and it compiles `src/proxy/store.c` into the harness because proxytrack does not link libhttrack. Both readers and the writer print to stderr on malformed input, so pass `-close_fd_mask=2` for anything longer than a corpus replay.
This file lists all changes and fixes that have been made for HTTrack
3.49-16
+ New: macOS ships a signed and notarized HTTrack.app with its own icon, bundling the OpenSSL it needs so a downloaded copy launches (#890, #900, #901, #950)
+ New: the desktop icons install into the hicolor theme, scalable SVG included, so Icon=httrack resolves in a launcher (#932, #933)
+ Fixed: an unauthenticated PROPFIND overflowed the ProxyTrack DAV item buffer (#836)
+ Fixed: the cached-headers block was built with an unbounded sprintf (#841)
+ Fixed: a chunked response carrying trailers was rejected as "Invalid chunk" (#855)
+ Fixed: an over-long URL aborted the whole mirror inside the cache instead of reading as a miss (#935, #936)
+ Fixed: ProxyTrack could not re-read the .arc it writes, and crashed on a record whose body it could not read (#834, #929, #931)
+ Fixed: ProxyTrack logged a hashtable stats line and every PROPFIND body to stderr (#911, #918)
+ Fixed: the frozen-slot spool was written inside the mirror namespace, so it landed in the mirrored tree (#859)
+ Fixed: a stack overflow killed httrack with no diagnostic, crash reports named no frame of the executable, and on armhf they had no frames at all (#866, #889, #892)
+ Fixed: an install moved away from its configured prefix could not find its data directory, its shared library, or the html symlink (#885, #887, #894, #906)
+ Fixed: tooltips in the web GUI broke on a translation containing an apostrophe; the escaping now also covers quotes, backslashes, markup and DBCS lead bytes (#864)
+ Fixed: nine strings of the Windows GUI's option dialogs were untranslated in 26 of the 30 language files, and six language files disagreed with their own declared charset (#863, #963)
+ Fixed: the AppStream metainfo still advertised WebHTTrack 3.49.8 (#884)
+ Fixed: the installed development headers did not compile standalone (#943)
+ Fixed: configure discarded a user-supplied BASH_SHELL, resolved bash to /bin/sh on macOS, and hung on one pointing at a FIFO (#891, #895, #908, #922)
+ Changed: the masthead wordmark and the rings background are SVG, so they stay sharp on a hi-DPI screen (#910, #916)
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-15
+ New: --single-file rewrites each saved page with its assets inlined as data: URIs (#713)
+ New: --changes reports what a crawl added, updated or removed against the previous mirror (#714)
+ New: --sitemap and --sitemap-url ingest sitemaps, so pages nothing links to are still found (#712)
+ New: webhttrack exposes --warc-cdx, --wacz and --warc-max-size (#862)
+ Fixed: --update and --purge-old destroyed a good local copy when the re-fetch got no response, was aborted mid-read, or when the backup meant to protect it failed (#746, #748, #758, #775)
+ Fixed: an FTP re-fetch truncated the mirrored file, resumed a complete mirror with REST and spliced the old body into the new one, and a successful transfer was blanked when its backlog slot was swapped out (#771, #797, #798, #823)
+ Fixed: a chunked response cut at a chunk boundary was stored and cached as complete (#840)
+ Fixed: cache repair deleted the old cache before a rename it never checked, and the unlink-then-rename fallback could lose the destination (#779, #786, #790, #824)
+ Fixed: several backward scans from strlen(s) - 1 read before their buffer on an empty string (#730, #768, #770, #814, #821)
+ Fixed: a URL could be saved onto the engine's own temporary files, and the final path segment was never clamped (#774, #842, #852)
+ Fixed: a query-string character reference the page charset cannot represent was left unescaped, changing how the query parses (#854)
+ Fixed: a second --update pass overwrote the previous WARC and regenerated a page-less WACZ (#759)
+ Fixed: WARC output dropped URLs of 1005 bytes or more, archived nothing for a 304 revisit, and marked an engine-forced not-modified as one the server sent (#778, #785, #826, #838, #839)
+ Fixed: ProxyTrack overflowed its .arc header block, walked past the end of an .ndx buffer, trusted an unparsed offset, and crashed on a PROPFIND or on an entry with no usable Last-Modified (#793, #820, #825, #828)
+ Fixed: webhttrack leaked the session id into crawled pages, overflowed its command line so a quoted value could inject flags, let a posted project path repoint the served root, and built its redirect Location in a 256-byte stack buffer (#700, #706, #707, #710)
+ Fixed: in the web GUI, options ticked on by default could not be un-ticked, "max site size" set a per-file cap instead of the HTML one, and the mirror link on the finished page could not be followed (#708, #709, #725)
+ Fixed: htsserver labelled every PNG as image/gif and offered JPEG as a download, and an unauthenticated GET of a directory spun the server forever (#724, #875)
+ Fixed: webhttrack hung at exit when no mirror had been launched (#753)
+ Fixed: oversized cache and header fields aborted the engine or overflowed a neighbouring field instead of being clipped (#701, #715, #717, #722, #732)
+ Fixed: a -%S rules file of 4 GB or more overran the heap (#702)
+ Fixed: the CLI display did not repaint when the terminal was resized (#97)
+ Fixed: a document with no declared charset double-encoded the title lifted for the local index (#848)
+ Fixed: a fragment on an inlined reference was dropped, losing an SVG sprite selector (#766)
+ Fixed: several time helpers handed out libc's shared gmtime/localtime static rather than a reentrant breakdown (#794, #806)
+ Changed: fatal-signal backtraces name engine frames instead of a bare module and offset (#705)
+ Changed: --without-zlib is rejected at configure time rather than failing at link (#735)
+ Changed: the offline documentation gains one GUI guide with screenshots, an Android option reference, and a restructured index
+ Changed: multiple internal hardening, test and CI improvements
3.49-14
+ New: WARC/1.1 archive output (--warc), with a sorted CDXJ index (--warc-cdx) and WACZ packaging (--wacz), also available from webhttrack (#668)
+ New: -%F takes named footer fields such as {url}, {lastmodified}, {mime}, {charset} and {status} instead of a fixed layout (#667)
@@ -98,7 +148,7 @@ This file lists all changes and fixes that have been made for HTTrack
+ Fixed: report why a -%L URL list could not be loaded (#49)
+ Changed: multiple internal hardening, build and CI improvements
.49-9
3.49-9
+ Fixed: file-type detection from the Content-Type header: trust a declared type over a binary URL extension, honor --assume under the delayed type check, and keep a known extension against a bogus or empty Content-Type (#267, #29, #56)
+ Fixed: an uninitialized-buffer read when the Content-Type is empty (#411)
+ Fixed: restored C++ source-compatibility of the installed headers so reverse dependencies (httraqt) build again (#413)
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="How to mirror a website with HTTrack on Android: install the app, create a project, enter the address, run the mirror, and browse the result on your device."/>
<metaname="keywords"content="httrack, HTTrack, android, offline browser, web mirror utility, website mirroring, mobile, Google Play"/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<title>HTTrack Website Copier - Cache format specification</title>
For updating purpose, HTTrack stores original (untouched) HTML data,
references to downloaded files, and other meta-data (especially parts of the HTTP headers) in a cache,
located in the hts-cache directory. Because local html pages are always modified to "fit" the local
filesystem structure, and because meta-data such as the last-Modified date and Etag can not be stored
with the associated files, the cache is absolutely mandatory for reprocessing (update/continue) phases.
<br/><br/>
<h3>The (new) cache.zip format</h3>
The 3.31 release of HTTrack introduces a new cache format, more extensible and efficient than the previous one (ndx/dat format).
The main advantages of this cache are:
<ul>
<li>One single file for a complete website cache archive</li>
<li>Standard <ahref="http://www.pkware.com/products/enterprise/white_papers/appnote.txt"target="_new">ZIP</a> format, that can be easily reused on most platforms and languages</li>
<li>Compressed data with the efficient and opened <ahref="http://www.gzip.org/zlib/"target="_new">zlib</a> format</li>
</ul>
The cache is made of ZIP files entries ; with one ZIP file entry per fetched URL (successfully or not - errors are also stored).<br/>
For each entry:
<ul>
<li>The ZIP file name is the original URL [<small><ahref="#orig">see notes below</a></small>]</li>
<li>The ZIP file contents, <b>if available</b>, is the original (compressed, using the deflate algorythm) data</li>
<li>The ZIP file extra field (in the local file header) contains a list of meta-fields, very similar to the <ahref="http://www.ietf.org/rfc/rfc2616.txt?number=2616"target="new_">HTTP</a> headers fields. See also <ahref="http://www.ietf.org/rfc/rfc2396.txt?number=2396"target="new_">RFC</a>.</li><br/>
<li>The ZIP file timestamp follows the "Last-Modified-Since" field given for this URL, if any</li>
There are also specific issues regarding this format:
<ul>
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
</ul>
<br/>
<b>Meta-data stored in the "extra field" of the local file headers</b><br/>
The extra field is composed of text data, and this text data is composed of distinct lines of headers.
The end of text, <b>or</b> a double CR/LF, mark the end of this zone.
This method allows you to optionally store original HTTP headers just after the "meta-data" headers for informational use.<br/>
<br/>
<b>The status line (the first headers line)</b><br/>
Indicates if the data are present (value=1) in the cache (that is, as ZIP data), or in an external file (value=0).
This field MUST be the first field.
<li>X-StatusCode</li><br>
The modified (by httrack) status code after processing. 304 error codes ("Not modified"), for example, are transformed into "200" codes after processing.
<li>X-StatusMessage</li><br>
The modified (by httrack) status message.
<li>X-Size</li><br>
The stored (either in cache, or in an external file) data size.
<li>X-Charset</li><br>
The original charset.
<li>X-Addr</li><br>
The original URL address part.
<li>X-Fil</li><br>
The original URL path part.
<li>X-Save</li><br>
The local filename, depending on user's "build structure" preferences.
</ul>
<br/>
<b>Standard (RFC 2616) "useful" fields:</b><br/>
<ul>
<li>Content-Type</li>
<li>Last-Modified</li>
<li>Etag</li>
<li>Location</li>
<li>Content-Disposition</li>
</ul>
<br/>
<b>Specific fields in "BNF-like" grammar:</b><br/>
The 3.31 release of HTTrack introduces a new cache format, more extensible and efficient than the previous one (ndx/dat format).
The main advantages of this cache are:
<ul>
<li>One single file for a complete website cache archive</li>
<li>Standard <ahref="http://www.pkware.com/products/enterprise/white_papers/appnote.txt"target="_new">ZIP</a> format, that can be easily reused on most platforms and languages</li>
<li>Compressed data with the efficient and opened <ahref="http://www.gzip.org/zlib/"target="_new">zlib</a> format</li>
</ul>
The cache is made of ZIP files entries ; with one ZIP file entry per fetched URL (successfully or not - errors are also stored).<br/>
For each entry:
<ul>
<li>The ZIP file name is the original URL [<small><ahref="#orig">see notes below</a></small>]</li>
<li>The ZIP file contents, <b>if available</b>, is the original (compressed, using the deflate algorythm) data</li>
<li>The ZIP file extra field (in the local file header) contains a list of meta-fields, very similar to the <ahref="http://www.ietf.org/rfc/rfc2616.txt?number=2616"target="new_">HTTP</a> headers fields. See also <ahref="http://www.ietf.org/rfc/rfc2396.txt?number=2396"target="new_">RFC</a>.</li><br/>
<li>The ZIP file timestamp follows the "Last-Modified-Since" field given for this URL, if any</li>
There are also specific issues regarding this format:
<ul>
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
</ul>
<br/>
<b>Meta-data stored in the "extra field" of the local file headers</b><br/>
The extra field is composed of text data, and this text data is composed of distinct lines of headers.
The end of text, <b>or</b> a double CR/LF, mark the end of this zone.
This method allows you to optionally store original HTTP headers just after the "meta-data" headers for informational use.<br/>
<br/>
<b>The status line (the first headers line)</b><br/>
Indicates if the data are present (value=1) in the cache (that is, as ZIP data), or in an external file (value=0).
This field MUST be the first field.
<li>X-StatusCode</li><br>
The modified (by httrack) status code after processing. 304 error codes ("Not modified"), for example, are transformed into "200" codes after processing.
<li>X-StatusMessage</li><br>
The modified (by httrack) status message.
<li>X-Size</li><br>
The stored (either in cache, or in an external file) data size.
<li>X-Charset</li><br>
The original charset.
<li>X-Addr</li><br>
The original URL address part.
<li>X-Fil</li><br>
The original URL path part.
<li>X-Save</li><br>
The local filename, depending on user's "build structure" preferences.
</ul>
<br/>
<b>Standard (RFC 2616) "useful" fields:</b><br/>
<ul>
<li>Content-Type</li>
<li>Last-Modified</li>
<li>Etag</li>
<li>Location</li>
<li>Content-Disposition</li>
</ul>
<br/>
<b>Specific fields in "BNF-like" grammar:</b><br/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<title>HTTrack Website Copier - Change report format specification</title>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<p>This is a task-oriented guide to the <tt>httrack</tt> command line: how to do the
things people actually ask for, and the handful of defaults that surprise
@@ -112,21 +85,6 @@ the log. Every project writes <tt>hts-log.txt</tt> (and <tt>hts-err.txt</tt>) in
its output directory, and those files name every URL that was refused, redirected,
or filtered out. Read them first.</p>
<h4>On this page</h4>
<ulclass="tblNoBorder">
<li><ahref="#quickstart">1. Quick start</a></li>
<li><ahref="#scope">2. Scope: how far the crawl reaches</a></li>
<li><ahref="#filters">3. Filters and scan rules</a></li>
<li><ahref="#limits">4. Limits and politeness</a></li>
<li><ahref="#names">5. File names and types</a></li>
<li><ahref="#links">6. Links and page building</a></li>
<li><ahref="#identity">7. Identity, cookies and login</a></li>
<li><ahref="#proxy">8. Proxy and network</a></li>
<li><ahref="#update">9. Update and cache</a></li>
<li><ahref="#experts">10. Experts and scripting</a></li>
<li><ahref="#recipes">11. Recipes</a></li>
</ul>
<h3id="quickstart">1. Quick start</h3>
<p>A mirror is one command: a start URL and an output directory.</p>
@@ -156,7 +114,7 @@ you write. Get this right and most "it downloaded too much" or "it only grabbed
the index" problems disappear.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--depth (-r)</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
<tr><td><tt>--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
<tr><td><tt>--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
@@ -242,7 +200,7 @@ options will not help there. That is an
bracket forms match narrower sets. The full table, with size and mime rules, is on
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt>— any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt>— a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
@@ -261,7 +219,7 @@ accident, and the <a href="abuse.html">abuse page</a> is worth a read. The limit
below let you go faster when you own the target, and slower when you do not.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--max-rate (-A)</tt></td><td>Maximum transfer rate in bytes/sec. <b>The default is about 100 KB/s even without this flag.</b> Raise it to go faster.</td></tr>
<tr><td><tt>--sockets (-c)</tt></td><td>Number of parallel connections (default 4). <tt>--tiny</tt>, <tt>--wide</tt> and <tt>--ultrawide</tt> are presets.</td></tr>
<tr><td><tt>--connection-per-second (-%c)</tt></td><td>New connections opened per second (default 5).</td></tr>
@@ -288,7 +246,7 @@ usually about a URL like <tt>/article?id=42</tt> or a <tt>.php</tt> page that is
really HTML.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
<tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
<tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
@@ -312,7 +270,7 @@ naming.</p>
ones it kept so the local copy browses offline. These options tune both halves.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--keep-links (-K)</tt></td><td>How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare <tt>-K</tt> keeps <b>absolute</b> URLs, and <tt>-K0</tt> is the <b>relative</b> default. <tt>-K3</tt> keeps absolute URIs, <tt>-K4</tt> keeps the original links.</td></tr>
<tr><td><tt>--replace-external (-x), --generate-errors (-o)</tt></td><td>Replace external links with an error page, and generate an error page for links that failed.</td></tr>
<tr><td><tt>--preserve (-%p), --disable-passwords (-%x)</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
@@ -338,7 +296,7 @@ server), and obeys robots. Plenty of sites filter exactly that profile. These
options control what HTTrack says about itself.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--user-agent (-F)</tt></td><td>The <tt>User-Agent</tt>. Set a browser string to get past crawler blocks; <tt>--user-agent ""</tt> sends none.</td></tr>
<tr><td><tt>--headers (-%X)</tt></td><td>Add raw header lines to every request.</td></tr>
@@ -369,7 +327,7 @@ across links at the same time.</p>
<h3id="proxy">8. Proxy and network</h3>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
<tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
<tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
@@ -392,7 +350,7 @@ updating possible. It is not a size-limited scratch area you can delete: throw i
away and you lose the ability to continue or update the mirror.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--continue</tt></td><td>Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.</td></tr>
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
<tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
@@ -415,7 +373,7 @@ dynamic pages to look "changed" on every run. See the
mirror behavior and expose the engine.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<trclass="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--get URL</tt></td><td>Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.</td></tr>
<tr><td><tt>--spider --testlinks --skeleton</tt></td><td>Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.</td></tr>
<tr><td><tt>--userdef-cmd (-V)</tt></td><td>Run a shell command on each downloaded file; <tt>$0</tt> is the file path. Good for on-the-fly processing.</td></tr>
@@ -540,24 +498,14 @@ all disabled.</small></p>
<ahref="cache.html">cache</a>; for the change report, see
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, linux, windows, macos, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<li>In case of troubles/problems during transfer, <b><u><fontcolor="red">first check the hts-log.txt (and hts-err.txt) files to figure out what happened</b></u></font>. These log files report all
<li>In case of troubles/problems during transfer, <b><u>first check the hts-log.txt (and hts-err.txt) files to figure out what happened</b></u>. These log files report all
events that may be useful to detect a problem. You can also ajust the debug level of the log files in the option
</li><li>
The tutorial written by Fred Cohen is a very good document to read, to understand how to use the engine,
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="How to mirror a website with the HTTrack graphical interface: a step-by-step walkthrough and a full option reference for WinHTTrack on Windows, WebHTTrack on Linux and Unix, and HTTrack for Android.">
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.