121 Commits

Author SHA1 Message Date
Xavier Roche
de484e0670 Release 3.49.21 (#1226)
* Release 3.49.21

Version bumps in configure.ac, htsglobal.h, version.rc and the AppStream
metainfo, plus the curated release notes.

VERSION_INFO goes to 3:14:0, a revision-only bump: httrackp gained a tail
field (host_alias) and #1202 added an export, so nothing moved and nothing
went away.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Apply the release review: notes and changelog wording

The changelog entry named upstream flags, which the packaging audience does
not want; the notes instantiated example hostnames and quoted a UI label
where their neighbours stay abstract. The copyright bullet cites #1163 as
well, and the new export is named rather than implied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 16:51:35 +02:00
Xavier Roche
b6d15c7bc8 Pin the host-alias bytes the WinHTTrack validator now depends on (#1224)
hts_host_alias_rule_ok accepts 0x80-0xFF and a leading '-', both by
design, and nothing pinned either. httrack-windows#109 dropped its
mirrored grammar for this exported validator, so a tightened reject set
would break it silently.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 14:31:07 +00:00
Xavier Roche
db22036173 The Windows watchdog test calls a reaped ping a survivor (#1225)
* The Windows watchdog test calls a reaped ping a survivor

58_watchdog has failed the Win32 leg on every master push since the suite
went parallel. The grandchild check reads tasklist one second after the
watchdog fires and asks whether the ping's PID is still listed; neither half
survives sixteen concurrent tests. Match the image name as well as the PID,
and poll for it to go rather than sleeping once.

On a non-timeout failure the driver re-runs the test traced and appends to
the same log, then reports the last 25 lines: for an intermittent failure
that is the run that passed. Keep the failing run's tail before the trace can
append, and report both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Control alive(), and pin the report's order and bound

The image-name conjunct is new, so a predicate that never matches would
call every survivor reaped: prove it fires on a live ping first, and fold
case the way proclib.sh's tasklist matchers already do.

172 now pins the two properties the marker alone left open: the failing
tail must come before the label, and both halves must stay tails.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 16:11:55 +02:00
Xavier Roche
50f151e11b freeport hands out a port it has already released (#1221)
`freeport` binds a socket to port 0, records the number and closes it, so the port is free again before its caller does anything with it. Under `make check -j16` two tests can be handed the same one. `start_proxytrack` already retried; the other two callers did not.

The one that matters is `56_local-proxy-noleak`, which wants a port that stays dead for the whole crawl so that connecting to it is refused. A retry cannot help when the exposure lasts the length of the test: if a sibling binds that number, the decoy proxy answers and the test stops testing its own premise while still reporting green. Such a port is now held rather than sampled, bound and never listened on, so connections are still refused. `htsserver_start` retries in the shape `start_proxytrack` already uses, and `freeport`'s comment now says which of the two a caller owes it.

Test 288 asserts the discriminator rather than the symptom, since an unbound port refuses a connection too: a competing bind must fail while the port is held, and succeed once it is released. Extracting the helper also fixed a leak in 276, whose private copy registered its cleanup inside a command substitution and left a python process sleeping 900s behind every run.

Closes #1218
2026-08-12 12:42:29 +00:00
Xavier Roche
729f823db9 A build path containing "assert" or "panic" fails the crawl tests (#1222)
* Read the crawl's own log, not the command line printed above it

hts-log.txt line 2 echoes the engine's whole command line, absolute paths
included, so --log-found and --log-not-found were matching the paths the run
was given. 36_local-bigcrawl searches for "assert", which made it fail for
anyone building under a directory whose name contains that word; TMPDIR, -O and
the source directory reach that line too.

Only that line is skipped. The banner above it names the URL, which
14_local-https does match, and it is left in place if it ever stops looking
like the echo: a "not found" audit hiding a real match is the worse failure.

Closes #1220

Signed-off-by: Xavier Roche <roche@httrack.com>

* Review: pin the banner too, and cut the comments

The pair of cases constrained only "line 2 gone, line 4 present", so the
first-cut mistake this fixes, dropping the banner as well, passed both.
14_local-https catches it but skips where TLS is not built.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 11:42:07 +00:00
Xavier Roche
01274a93f5 The WebHTTrack pages hardcode a copyright year the doc pages now compute (#1217)
* Expand the WebHTTrack footer year instead of hardcoding it (#1165)

The 23 pages under html/server/ carried 1998-2026 as literal markup, which
doc-chrome.py cannot reach, so from next New Year the docs and the GUI would
have disagreed with nothing to catch it.

htsserver already resolves a ${prefix:name} family in the templates it serves,
so the year becomes one more of those. Computed per request, it cannot drift,
and no regeneration or check has to watch it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Classify the date: token in the template escaping audit (#1165)

185 requires every ${...} in the server templates to carry the escaping prefix
its position needs, and a bare key in element body is a finding. The year the
server computes is not runtime data and cannot carry markup, so it joins the
translations and build-time constants instead.

Paired both ways: a settings value in element body must still be a finding, or
the exemption is a hole rather than a classification.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fall back to the clock on an unusable SOURCE_DATE_EPOCH, and harden the test

A garbage or over-range override reached time zero, so the footer rendered
1998-1970 rather than ignoring the value. The parse now has to consume the whole
string, and a value gmtime rejects falls back to the real clock.

The test grew what the review found missing: the tree-wide sweep that a single
converted page cannot satisfy, a pinned epoch an hour before New Year UTC read
from 14 hours east of it, so gmtime and localtime give different years, and the
two unusable overrides. Renumbered off 286, which master took meanwhile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Refuse a SOURCE_DATE_EPOCH whose year no footer can show

glibc's gmtime accepts the whole int range of tm_year, so a parsable
SOURCE_DATE_EPOCH could reach the footer as "-2" or overflow the
tm_year + 1900 add. Both break the digits-only promise test 185's
date: exemption rests on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 11:33:33 +00:00
Xavier Roche
bd765e4ed6 Give the compile-time rules one spelling, HTS_STATIC_ASSERT (#1219)
Three constants were pinned three ways: an enum dividing by zero in htsparse.c,
and two negative-array typedefs in htsarena.h. htsglobal.h already had
HTS_COMPILE_ASSERT for the expression case, so this adds the declaration case
beside it, on the enum form, and moves all three onto it.

Each of the three was checked by breaking what it guards and confirming the
build stops: a non-power-of-two alignment kills both arena rules, and shrinking
SINGLEFILE_MAX_SPAN kills the parse one.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 11:24:34 +00:00
Xavier Roche
17e3d241fa Windows CI runs from different branches starve each other's runners (#1214)
* Windows CI runs from different branches starve each other's runners

windows-build keys its concurrency group on github.ref, so it only cancels
superseded runs of the same branch. Runs on different branches overlap freely,
and at two platforms each, up to six Windows jobs can share the pool. #732 and
#733 were two Win32 jobs starting 2 seconds apart that both wedged the 45-minute
step timeout and both passed on a rerun of the identical commits; #1126 wedged
the same way; PR #1203's x64 leg lost communication with the server while three
runs were in flight.

A job-level group keyed on matrix.platform caps that at one x64 and one Win32
job across the whole repository. It needs queue: max, because the default queue
holds a single pending entry and cancels it when the next job arrives, which
would trade starvation for cancelled checks. The workflow-level per-ref cancel
stays, so a second push to a branch still drops the older run and frees its
queue slot.

A Windows job takes about 34 minutes, so a third concurrent PR now waits instead
of racing. That wait shrinks once the suite step stops running its tests
serially.

Closes #1213

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the comment to two lines

Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 10:23:56 +00:00
Xavier Roche
ee20bfbb2b The Windows suite runs its tests one at a time (#1216)
The Windows suite step takes 23.5 minutes because `ci-windows-suite.sh` runs its 238 tests one at a time, while every `make check` on other platforms uses `-j min(2*nproc,16)`. Measured against the Linux suite test for test, Windows is only 1.61x slower at the same work: the gap is the missing `-j`, not the platform. Scheduling the real per-test times gives 3.3 min at `-j8`.

The pool is the small part. Three assumptions in the shared harness had to go first, because each is only safe while one test runs at a time. `kill_tree` fell back to killing every engine image and every `python.exe` on the host when it could not resolve a Windows PID, which under `-jN` kills the other tests' servers; it now takes a PID captured at fork time, and the host-wide sweep needs an explicit opt-in the driver sets only when it runs serially. `dump_crawl_logs` globbed and deleted every `httrack_local.*` in a shared TMPDIR, so each test now gets its own. The between-test reap taskkills by image name, so it runs once after the pool drains, trading per-test leak attribution for parallelism; a wedge is still bounded by the per-test timeout and its tree kill.

`wait -n` is bash 4.3, and macOS drives this script under 3.2, so the pool falls back to the serial path there. Test 286 runs a stub suite both ways and fails if the parallel leg does not beat the serial one; forcing the driver to one worker makes it red.

Closes #1215
2026-08-12 11:46:51 +02:00
Xavier Roche
bde9eadb05 An early exit from hts_main_internal leaks the rebuilt command line (#1210)
* Release the rebuilt command line on every early exit

hts_main_internal allocated it once and freed it only at the end, so each of
the 88 early returns leaked the token chunks and the slot array. It never
mattered much for the binary, which is exiting anyway, but hts_main2 is an
exported entry point and an embedder pays it per call.

htsmain_free() already runs on every one of those paths, and each caller
returns straight after, so the release belongs there.

Closes #1206

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review: tighten the comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the new leak test in the Windows suite's expected skips

It needs a LeakSanitizer build, so it skips there like 114 does, and that job
fails any change to the skip set rather than let coverage drift silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 09:43:19 +00:00
Xavier Roche
88f0a0174d The Windows build rebuilds OpenSSL from source on every run (#1209)
The vcpkg binary cache has not served a package since 2026-08-04: both legs log `Restored 0 package(s)` and rebuild openssl from source, 7.7 min on x64 and 5.1 min on Win32. The key named only the manifest, so nothing busted it when a runner image orphaned every archive, and actions/cache skips the save on an exact hit. That is why it stayed broken for eight days rather than one run.

Naming every input vcpkg's ABI hash folds in would only make a list that can still come up short, so the restore keys on the run id and prefix-matches instead, the save runs when the archive set changed, and a cache vcpkg matched nothing in is dropped rather than carried forward. Two runs are needed to show it: the first supersedes the dead entry, the second should restore warm and bring the build step to roughly 2 min.

Closes #1208
2026-08-12 09:00:53 +00:00
Xavier Roche
696553b0f9 Three open-coded chunked pools with the same stable-pointer contract (#1212)
* Give the three chunked pools one implementation

htscore.c grew two of them, for link strings and for lien_url records, and
#1205 added a third for the rebuilt command line. All three exist for the same
reason, stated in one of their own comments: "There are references to item
pointers, so we can not just realloc()". All three picked 32768. The growable
primitives the tree already has are no substitute, since String and TypedArray
move their contents and coucal's pool rebases its keys on purpose.

htsarena.h is that pool, once: chunks that are never resized, doubling to a
ceiling, with the alignment and the overflow guards in one place instead of
three. The 32 KB cap on a single link string goes with it, and so does the
assert that used to abort on a longer one.

Closes #1207

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Charge alignment padding where it is needed, not to every allocation

Rounding each size up cost several bytes on every link string a mirror
records, where the pool it replaces packed them. Only the lien_url records
need the alignment, so pad at the allocation that wants it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the arena's alignment assumptions at compile time

Review: the padding takes HTS_ARENA_ALIGN for a power of two and the header
for a multiple of it, and nothing said so. A platform where either stops
holding would have miscompiled quietly; now it does not compile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 08:57:30 +00:00
Xavier Roche
99d494bd39 The resume offer echoes the program path instead of the command line it would run (#1211)
* Quote the command line in the resume offer, instead of the program path

The offer to resume or update a mirror ends with the command line it would
run, but printed argv[0], so it read "OK to Continue httrack /usr/bin/httrack".
It has never worked: the buffer it reads holds the rebuilt command line as
NUL-separated tokens, so the append stopped at the first one.

Printing the rest is not a plain loop over them. HT_PRINT appends through
strcatbuff into a fixed 2 KB buffer and aborts on overflow, which a command
line replayed from doit.log outgrows, so the tokens are clipped to what is
left and closed with an ellipsis.

Closes #1204

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review: tighten the comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the clip mark too, so the append needs nothing of its caller

Review: the ' ...' went in through HT_PRINT, which aborts on overflow, so it
leaned on the preamble the caller had written leaving room. Not reachable
today, since every argv element is bounded, but the function should not
depend on that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 08:37:37 +00:00
Xavier Roche
53eace3cfe A cleared box in the wizard cannot disable the index, and --purge-old=1 kills the run (#1201)
* A long option drops the value its short form takes

--index=0 dropped its value and read back as the bare --index, which enables
the index: the alias table calls -I a "single" option, and that class throws an
attached value away. So clearing "create a top index" in the WebHTTrack wizard
could not turn it off, and neither could --noindex.

Two new classes carry the value the short form already accepts: "onoff" for the
flags whose parser reads a 0 suffix (-I0), "level" for those that parse a number
(-%v2). Everything still "single" now refuses an attached value rather than
dropping it. The bare form of every alias is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the emitted short form against the engine, not the table

Review found the value-taking rows had no test tying them to the parser: the
table loop only checks bare forms, which are identical for all three classes,
so a row wrongly marked onoff would emit -x0 and kill the run unseen. 282 now
sweeps every onoff and level row the engine lists and fails on "invalid
option", and asserts the refusal message and the exact flag set each wizard
box emits rather than a substring.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Three param rows are booleans in disguise, so --purge-old=1 kills the run

purge-old (-X), httpproxy-ftp (-%f) and extended-parsing (-%P) are classified
param, but their parser cases have the onoff shape: set the flag, then consume
a 0 suffix. The class demands a following token, so --purge-old=1 emitted -X1
and died on "invalid option 1", and --purge-old before a URL ate the URL as its
value and mirrored nothing. A profile carrying NoPurgeOldFiles=0 makes the
wizard write exactly that command line.

They were misclassified rather than newly broken, so they move to onoff with
the rest. No fourth row has this shape: every other param row reads a real
value, and every param1 takes a separate string. help() now annotates an
optional value, which regenerates the man page and its rendered HTML.

Closes #1200

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The users guide still promises a numeric value for the three flags

fcguide.html carries a pasted copy of the option list, where -X, -%f and -%P
read [=N]. Only 0 was ever meaningful, and help() now says [=0].

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The class sweep reads the table it is checking

Review found it circular: a row that should take a value but is still single
never appears in the list, which is the #1200 bug exactly. step4.html is an
independent source, so every --name=0 it writes must now resolve to a row whose
class takes one. The sweep also tries =1, the value that emitted -X1 and killed
the run, and a refusal must carry a message.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the digit run before indexing it

CodeQL cpp/offset-use-before-range-check: value[i] was read before i < 16
gated it. In bounds either way, but the wrong order for a parser fed user
input. Same boundary: 16 digits accepted, 17 refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 10:21:54 +02:00
Xavier Roche
1dcfcf491f The Android guide screenshots crawl the emulator's host-loopback alias (#1203)
10.0.2.2 is the Android emulator's fixed alias for the host machine's own
127.0.0.1, so the URL field and the progress line showed a reader an address
they cannot use. A grep for the literal loopback address reports the set clean,
which is why #1133's purge left it behind on this platform alone.

Regenerated from httrack-works 3988c35, where the walk now maps www.example.com
onto that alias inside the emulator. Two images move; the other 17 captures
reproduce byte for byte.

Closes #1199

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 07:45:41 +00:00
Xavier Roche
14e9108049 A --host-alias rule may carry bytes no host can, and no front end can call the validator (#1202)
* Reject a newline inside a --host-alias rule, and export the validator

hts_host_alias_rule_ok() did not reject '\n', which the engine uses to join
the rules it accepted, so --host-alias $'a.com=b.com\nc.com' validated as one
rule and hts_host_alias_match() then read it as two, dropping the '='-less
fragment without a word.

The same function moves to httrack-library.h behind HTSEXT_API, so a front end
can reject a malformed rule in its own field instead of mirroring the grammar
and drifting from it.

Closes #1192
Closes #1172

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Reject any control byte and an unparsable canonical scheme in a rule

Exporting the validator makes its reject set a supported contract, so close
the two holes review found beside the '\n' one. A control byte anywhere in a
rule is refused: '\v' and '\f' are not on the argv filter's CR/LF/TAB list and
reached the host the fold writes. The canonical side is cut with
jump_protocol_const rather than a literal "://", the way every consumer cuts
it, so "b.com=x://a.com" no longer folds http://b.com to a host of x://a.com.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 07:15:01 +00:00
Xavier Roche
5c068a3c65 A large .httrackrc aborts the engine on the rebuilt command line's token block (#1205)
* Grow the rebuilt command line's token block instead of aborting on it

The block was sized once, from the current argv plus fsize("config") plus
32 KB of slack, and never grew. The engine reads .httrackrc, httrackrc or
/etc/httrack.conf, never a file called "config", so nothing in that sum
accounted for what the expansion would add; past the slack cmdl_ins ran out
of room and strlcpybuff aborted the process on a config file the user wrote
themselves.

The block becomes a chain of chunks, so it grows the way the slot array has
since #1196. A chunk is never resized, which is what makes the growth safe:
every argv[] slot points into it and stays valid, as does argv_firsturl in
htscoremain.c, where a realloc would have had to rebase them all.

Closes #1198

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the chunk fields for what they are, and catch a clipped token

Review follow-ups: blk_size/blk_used named a blk field the chunk list replaced,
CMDL_CHUNK_DATA was a macro where its peer cmdl_room is a function, and the
test's two markers would both survive an implementation that clipped a token to
the room left instead of growing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 07:11:45 +00:00
Xavier Roche
e9336b4d87 Nothing checks that guide.html's hardcoded image sizes match the images (#1193)
* Check guide.html's hardcoded image sizes against the files it ships

Every <img> in the guide carries a width/height pair, re-verified by hand
after each recapture. A screenshot regenerated at another geometry ships
stretched and nothing goes red, which is the failure mode a regeneration is
actually exposed to.

tests/guide-image-check.py parses the tags and compares each declared pair
against PIL's, and catches the two neighbours: an <img> whose file is gone,
and a guide-*.png under html/img/ the guide stopped referencing. 279 drives
it against the tree, then mutates a copy to prove each arm fires.

Closes #1187

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the review holes in the guide image check

The distcheck srcdir is read-only and cp -a carried that over, so the first
mutant hit EACCES and leaked its temp tree. Comment stripping, a
case-insensitive tag match and a guard on any src the parse cannot read stop
a defect from being skipped in silence, and a repeated attribute now resolves
the way a browser lays it out.

None of the old mutants touched a PNG, so a checker that never opened a file
passed every one of them. 279 now resizes a shipped image, which is the
defect #1187 is actually about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Read the PNG geometry from the IHDR, so 279 runs on CI

No workflow installs Pillow and no runner image carries it, so the PIL guard
skipped 279 on every leg: a green suite that never ran the check. png-colors.py
already states the rule this broke, that the suite cannot assume Pillow on a
build host.

Twenty-four bytes of header carry what the check needs, so it reads them
itself, verifying the signature and the IHDR type rather than slicing on
faith. A missing checker is now a hard failure: it is in EXTRA_DIST, so its
absence is a packaging bug rather than an environment to tolerate. Three new
arms cover the reader's own refusals, and the image-side arm swaps in another
shipped PNG instead of resizing one through Pillow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 02:35:33 +00:00
Xavier Roche
11c357e73a The installed headers are never compiled standalone or pairwise on MSVC (#1190)
* Compile the installed headers standalone and pairwise on MSVC

205, 206, 207 and 269 all reach the installed header set through automake's
install rule and gate on abs_top_builddir, so all four exit 77 on the Windows
job, which runs the test scripts directly. Nothing else compiles
DevIncludes_DATA the way a consumer does: the DLL build only proves the
engine's own fixed include order, with LIBHTTRACK_EXPORTS set.

Move 269's sweep into tests/install-headers-sweep.sh and drive it from both
legs. 269 hands it what automake installed; a new step in the existing
libhttrack job stages the same list out of the source tree and compiles it with
cl.exe, batching every unit into one response file per language because a spawn
on that runner costs more than the compile. 278 audits the wiring, since no
sweep can notice its own step being dropped.

Closes #1153

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Put the bytecode mode back on the command line, widen the wiring audit

Review found the -U half of the sweep could go vacuous: the mode was
emitted into each unit as a #define, so a -DHTS_INTERNAL_BYTECODE
arriving through TEST_CPPFLAGS made both halves the same compile while
the unit count still read right. Both modes are command-line flags
again, with a synthetic header that must fail in one and compile in the
other.

Also: assert five named headers reach the staged set, so a Makefile.am
reformat the awk mis-parses cannot shrink the sweep quietly; resolve a
../ entry against the build tree alone; and teach 278's audit to reject
a conditional or continue-on-error step or job, a --self-test that never
sweeps, a sweep placed before the build, and an invocation left as a
comment, each with its own control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Find the vcpkg include root instead of spelling it

The MSBuild integration installs under src\vcpkg_installed\<triplet>\<triplet>,
nesting the triplet twice, so the step threw before it reached cl.exe. Search
for it, and name openssl/ssl.h as the thing that has to be found: htsbasenet.h
includes it by default, and it is the only reason the path is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 00:28:34 +00:00
Xavier Roche
94d2d79217 A wizard project reload cannot read back a zero or an empty value (#1184)
* Keep a value of zero when a wizard project is reloaded

The winprofile.ini reader turned "0" into an empty value for every key, so a
rate, depth or timeout the user set to zero came back as the wizard's own
default. Only the checkbox keys, which step4.html writes as ${ztest:x:0:1},
need that inverse; everything else now reads its zero back.

Adds tests/274, the first test to reach the reader at all: it drives a real
project reload over HTTP and covers the escape set as well as the coercion.

Closes #1177
Closes #1171

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Let a present winprofile.ini key overwrite the wizard default

An empty value acquired a NULL String, which coucal_readptr refuses, so
${do:copy:} skipped the key and the startup default survived. Also guards the
truncated percent escape both unescape helpers walked past.

Refs #1186

Signed-off-by: Xavier Roche <roche@httrack.com>

* Guard the truncated percent escape in the two remaining copies

htstools.c reads the category out of the same winprofile.ini and proxystrings.h
carries a third copy of the loop, both walking past the terminator on a value
that ends in a bare %.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Default the cache box at startup, not on every step2 render

${do:set:cache:1} ran on each render, so a reloaded profile that cleared the
box had it ticked again by the next visit to step2. Only a fresh session may
take the default, which is what initOn[] is for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Send the cleared cache and URL-hack boxes to the engine

Both read their state with test:, which emits nothing for an empty value, so
an unchecked box left the engine on its default while the box itself rendered
unticked. ztest: is the form the neighbouring cookies and parse-java lines
already use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert a fresh wizard session still defaults to caching

Nothing covered the startup default itself, so dropping it from initOn[] left
the suite green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 22:52:21 +00:00
Xavier Roche
52ac9ee142 A doit.log or a long command line overflows the rebuilt argv (#1196)
* Grow the rebuilt command line instead of guessing 1024 spare slots

htscoremain.c allocated argc + 1024 argv slots for the command line it
rebuilds and never re-checked that headroom, so both writers could run
off the end: cmdl_add on a command line whose --opt=value arguments each
expand to two tokens, and cmdl_ins replaying hts-cache/doit.log, which is
bounded only by the 8000-byte line the reader accepts.

The token count is not knowable up front (config files and doit.log each
add an input-defined number), so the slot array now grows on demand and
argc <= size is re-established before every write. cmdl_add, cmdl_ins and
optinclude_file take the array, its capacity and the token block as one
struct, which is what lets a growth reach every writer.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the config-file writer and refresh the stale macro references

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Address review: cmdl_init/cmdl_free, and a distinct OOM result for config files

The struct was half-initialized between its brace initializer and the two
fields the caller filled in by hand, and three new out-of-memory paths
returned without freeing it. cmdl_init and cmdl_free own both allocations,
so neither state is reachable.

optinclude_file returned "the file was read" on an allocation failure too,
which let the caller proceed with a half-expanded config. It now reports
that case apart, and the caller panics as its sibling writers already do.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 22:44:23 +00:00
Xavier Roche
d432772e1e A rule that begins with a dash kills the whole run (#1183)
* Take a rule that begins with a dash (#1179)

--strip-query and --host-alias take a user-written rule, and a rule may
legitimately start with '-'. The engine read it as the next option name,
so the whole run aborted before the mirror and the valid rules beside it
were lost.

Closes #1179

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: narrow the exemption, and prove the rule's bytes

Restore the diagnostic when a rule option is followed by a real option name,
and pin byte fidelity plus the short and '=' spellings in the test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: trim the comments added by this change

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Return the computed boolean rather than a ternary over it

House convention: a comparison or && chain is already an hts_boolean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Run the no-parameter probes through an absolute httrack path

noparam cds before invoking it, and make check puts the build's src/ on PATH
relatively, so a bare name found nothing there. An installed /usr/bin/httrack
hid that locally; CI has none and every build leg failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 22:39:05 +00:00
Xavier Roche
997160c98f The guide's WebHTTrack and Android screenshots predate the fields they describe (#1197)
The WebHTTrack set was shot before #1166 and #1176 landed. Its Spider tab showed neither the Host aliases box nor the four-row Strip query keys that replaced a single-line field, though the surrounding prose already described both. That page is 229 px taller now, and `guide.html`'s hardcoded height moves with it. The Android set predated #97, #114 and #116 the same way.

Captures are httrack-works `1af9fed`, the Android half shot by that session at httrack-android `f3123ad`. Regenerated with `tools/doc-images.py png`.

Seven web captures moved and I checked each. The Spider tab is the real change. Progress and finished carry live crawl counters. The project-name shot picks up a fresh random temp suffix in its base path. Three option tabs differ only by antialiasing on their select widgets, visually identical at a mean channel delta of 39. All 19 Android shots moved because that session recaptured its whole set rather than the four stale tabs, which keeps the version line and clock consistent.

The Windows images belong to #1180, now merged, so the two never overlapped. Every image is re-checked against `guide.html`: 55 of them, one dimension changed, no other drift.

Closes #1194
2026-08-11 22:20:33 +00:00
Xavier Roche
c3a5deef57 freeport reserves the ICP port over TCP, not the UDP it will be bound in (#1185)
* Reserve the ICP port in UDP, and let proxytrack name a bind failure

freeport() reserved both proxytrack ports by binding a plain TCP socket, but
proxytrack opens its ICP socket with SOCK_DGRAM. Windows keeps excluded port
ranges per protocol, so a TCP-derived number can land inside a UDP exclusion
and all three retries lose the same way. It now takes the protocol of each
port it hands out and probes in that one.

The failure was hard to read because proxytrack printed strerror(errno) for a
socket error Winsock reports through WSAGetLastError(), so the one line that
should have named the cause said "No error". It now reports the protocol, the
port and the system message for the call that actually failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Report a resolver failure as one, and pin the probe's protocol to its port

Review of the first commit found smallserver_init() reading errno on the
gethost() path, where getaddrinfo() never sets it: an unresolvable host printed
"cannot bind tcp port N ...: Success (0)". It now carries its own code and
proxytrack says "cannot resolve".

Test 276 grew with it. The socket-type spy logs the port beside the kind, so a
probe that opens a UDP socket and hands back a TCP-drawn number no longer
passes; both proxytrack destinations are exercised, and a probe forced to
collide proves the redraw loop reports exhaustion. Renamed smallserver_init's
"family" parameter to socktype, which is what it carries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 21:11:01 +00:00
Xavier Roche
b33ef757e5 Nothing runs black or flake8, so .flake8 binds nothing (#1182)
* Reflow tools/doc-images.py and tools/doc-chrome.py with black

Pure black output, no edits by hand. Settles the 14 E501 that .flake8
already forbade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Gate black and flake8 on the tracked Python files in CI

.flake8 has bound nothing since it arrived: no job ran black, flake8 or
any other Python linter, which is how two tools/ scripts drifted. Fold
both into the existing lint job, over git ls-files rather than a glob
list so a .py in a new directory cannot escape.

Closes #1170

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Split the Python lint into per-tool steps

Review follow-up: shellcheck and shfmt each get their own step so a red
names the tool, and the list moves to a shared env var like SHELL_SCRIPTS.
Guard the empty list too, which flake8 would otherwise accept silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 21:01:27 +00:00
Xavier Roche
9bb989053d A target that is not a keyword shares one tab, and dark mode prints white on white (#1181)
* html/: target="_new" shares one tab, and dark mode prints white on white

`_new` is not a target keyword, so browsers treat it as a named browsing
context called "new" and every one of the ten hand-written links reuses the
same tab. `_blank` is the keyword they meant. #1157 fixed the generated
footer; these predate it.

doc.css's `@media print` forced a white body but left the palette alone, so
in a browser that keeps matching prefers-color-scheme: dark while printing,
`--ink` stayed #e6e6ee on paper. Reset the tokens instead, which also covers
the panel and anything added later that reads them.

Closes #1164
Closes #1158

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Also fix cache.html's two transposed target="new_"

Same defect one page over, found while reviewing: `new_` is no more a
keyword than `_new`. Test 277 now rejects any target value outside the four
keywords rather than grepping for one misspelling, so a transposition is
caught too, and it reads both quote styles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 20:56:00 +00:00
Xavier Roche
c27d55c345 The guide's WinHTTrack screenshots predate the address, rate and host-alias changes (#1180)
Four of the guide's WinHTTrack images predate changes that shipped in httrack-windows. The walkthrough now crawls www.example.com:8099 instead of the loopback address (#104), a new project no longer starts capped at 25 KB/s (#103), and the Spider tab gained a Host aliases row (#101). Captures are httrack-works 8f63cbc, regenerated with `tools/doc-images.py png`.

Only those four move. The other fourteen Windows captures are byte-identical, and no web or Android image changed, so nothing rode in from a sibling capture set. Geometry is unchanged on all four, so `guide.html`'s hardcoded width and height still match.

The prose needs no addition: #1175 already flipped the Host aliases row to `data-for="win web droid"` ahead of the field shipping. The new image does surface something I have left alone, though. WinHTTrack draws Host aliases as a single-line box, the same height as Sitemap address, while the guide says "one rule per line". The engine walks newline-separated rules, so that control holds exactly one, which is what #1168 fixed on the WebHTTrack side. Whether it should be multi-line is httrack-windows' call, and I could not confirm the limit from source here, so I have written no caveat.
2026-08-11 20:52:32 +00:00
Xavier Roche
40e94df2cc The wizard can only hold one --strip-query rule (#1176)
* Take several strip-query rules in the wizard, one flag each

The engine accumulates --strip-query into a newline-separated list, one rule
per flag, but the wizard bound the option to a single-line input and emitted a
single flag, so a project built there held one rule. The field is a textarea
now and the arglist: directive #1166 added emits one flag per rule.

Closes #1168

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: exact assertions, and a scoped rule that matches

The winprofile and reopen checks matched substrings, so a merged pair passed
both; they compare the whole value now. 26's scoped rule matched no host, which
left a mutant dropping every scoped rule green on all four legs, and the control
strips a key the fixture never carries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test the last-match-wins rule the guide now documents, and the escaping

Both legs of the accumulating pair carried the same key, so a union or a
first-match-wins engine passed them; the new leg needs the later rule to
replace the earlier one. 273 posts a rule holding a '<', which raw would close
the box early.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 21:04:30 +02:00
Xavier Roche
c71f37189a Reshoot the Windows progress screenshot on a busy mirror (#1174)
The guide showed a mirror in progress as one connection trickling in a
small file, which is not what the pane looks like in use.
httrack-windows#102 changed the capture walk to serve itself a larger site
over eight connections, so it now shows eight transfers in flight with
their bars part-filled.

Regenerated from httrack-works 82ca70b. guide-win-language moves with it:
its About line carries the engine version, and CI now builds against
3.49-20 where that set was shot at 3.49-19.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 18:14:17 +00:00
Xavier Roche
cb91ecaedc Show the Host aliases row to WinHTTrack readers (#1175)
httrack-windows#101 merged as 2e65f8f, putting the field on WinHTTrack's
Spider page. The row now names all three front ends.

Closes #1173

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 18:13:55 +00:00
Xavier Roche
faa6964d18 The WebHTTrack wizard cannot reach --host-alias (#1166)
* Expose --host-alias in the WebHTTrack wizard and the guide

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: bash 3.2, the reload binding, and the guide wording

mapfile is bash 4 and macOS runs the suite under 3.2, so the argv read becomes
a plain loop. The test now reopens the option page with the rule stored, which
is what a typo in ${attr:hostalias} would break: the field renders empty and
the project loses the rule on every reload.

The guide row takes the CLI help's own wording (hosts fold onto one) and says
what the validator does with a trailing slash.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Take several host-alias rules in the wizard, one flag each

The engine accumulates --host-alias one rule per flag, so a single-line field
could only ever declare one canonical host. The field becomes a textarea and a
new arglist: template directive emits one flag per rule.

Rules separate on any whitespace, the way the URL box separates two URLs,
except beside a ',' or an '=' where the space belongs to the rule the parser
trims it out of: "a.com , b.com = c.com" stays one rule. The profile keeps them
on one ini line with the separator percent-encoded, which unescapeini() turns
back into a line break on reload.

--strip-query has the same shape and is now #1168.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the whitespace shapes the rules box accepts

A table over the runs a user actually types: repeated spaces, a tab, a CRLF
run, leading and trailing whitespace, and the runs beside a ',' or an '=' that
belong to their rule. The empty cases carry the weight, since the engine
refuses an empty rule and a blank line must not reach it as --host-alias "".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Split the rules box on every whitespace the Android app splits on

The wizard's splitter took " \t\r\n" while the Android field uses Java's \s,
so a pasted "a=b\vc=d" was two rules there and one rejected rule here. Widen
it to isspace()'s C-locale set.

The guide's "Strip query keys" row also claimed win and web only, hiding from
the in-app Android guide a field the app has long shown as editStripQuery.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Show the Host aliases row to Android readers

httrack-android#116 merged as b90e8ec, so the app has the field the row
describes. The row travels with the engine the app pins, so a build that
carries this guide also carries the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:48:44 +00:00
Xavier Roche
fce40a5511 The Windows guide screenshots lag the rebaselined captures (#1169)
httrack-works 8ae37ab reshot the Windows set from master, moving two of the
images doc-images.py generates. The mirror-progress counters differ because the
demo site grew from 24 pages at 0.35s to 120 at 0.8s, so the animation capture
added by httrack-windows#100 has a crawl long enough to film. The welcome pane
differs only in the runner's D: volume label; no GUI changed.

Regenerating deliberately, because doc-images.py reads the httrack-works working
tree live: left alone these land in whichever unrelated PR next runs the script.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:14:55 +00:00
Xavier Roche
0f3dbcded4 A long URL overflows the dedup key buffer in htshash.c (#1167)
* A long URL overflows the dedup key buffer in htshash.c

The dedup key is the host followed by the path, built with two unbounded
strcpy() into hash->normfil. Both halves come off the wire and each can reach
HTS_URLMAXSIZE * 2, so a hostile URL wrote up to twice what the buffer holds.
The write landed in the next field of the struct, where neither ASan nor
_FORTIFY_SOURCE can see it.

Size the key scratch for what it actually holds, a host plus a path, and clip
rather than abort on anything longer: the key only feeds the hash, equality is
decided by key_adrfil_equals_generic, so a clipped key costs a collision. The
match string hts_query_strip_keys() builds from the same two halves aborted on
the same input; it clips now too.

Closes #1160

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the strip-query match string and let the test own the canary

Review follow-ups on the key fix: use the tree's own strclipbuff() instead of a
hand-rolled clip, size hts_query_strip_keys()'s own host+path buffer so an
in-range URL is never truncated into matching a rule the whole one would not,
and keep an overlong path out of fil_normalized_filtered_ex()'s smaller
scratch.

hash_url_key() is now the key builder the engine itself calls, so the self-test
poisons its own arena and catches an overrun of any size instead of reading a
private field. normfil2 goes back to its old size: it only ever holds one
normalized path.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 17:00:14 +02:00
Xavier Roche
04c75c430a A site served under several hostnames is mirrored once per name (#1156)
Some sites answer on several hostnames with the same content. HTTrack treated each name as its own site. Links to the other name were out of scope by default, and once a filter let them in, the whole tree came down again under a second directory.

`--host-alias "alias[,alias...]=canonical"` (`-%C`, repeatable) names the hosts of one site. They then count as the same address for the travel rules, and each link is folded onto the canonical host in `url_savename`, which every link passes before it is type-probed, named, recorded or fetched. So one host reaches the wire and the cache whichever name linked there. Aliases match with the +/- filter glob and the last matching rule wins; the right side is a single literal host, checked when the option is parsed. Either side may name a scheme: on the right it replaces the link's, for a host that answers on one scheme only (`legacy.example.com=https://example.com`), and on the left it narrows the rule to that scheme (`https://www.example.com=ftp://files.example.com`). A rule may name anything an address holds and nothing it does not, so a path is refused with a diagnostic rather than ignored: scheme, credentials, host and port live in `adr`, a path lives in `fil`. Chains are followed to their end, since the fold has to give the same host however often a link is named; rules pointing in a circle name no canonical host, so those are left alone and reported.

The interaction with the url hacks is the subtle part. The mapping is decoupled from `-%u`, as `--strip-query` already is, so `-%u0` cannot silently turn it off. But the host is matched in the form the dedup layer is using: under the `www.` collapse a rule naming `example.com` also covers `www.example.com`, which `-%u` would otherwise key onto the bare name and away from the canonical one, and with the collapse off the rule matches the name as the link spells it, so `--no-www-dedup` still keeps those apart. The collapse applies to the rule as well as to the host, so either spelling of an alias works under `-%u`.

There is a `-#test=hostalias` self-test, and a crawl test that reaches the second name once per engine path that folds it: a shared page, a page nothing else links to, a redirect, a page whose type is known only after a probe, and a sitemap entry. A control run keeps the alias URL absolute and fetches none of them, and an HTTPS leg covers the case that made the canonical-side scheme necessary. A misdeclared direction, where the canonical host redirects back to the alias, was probed by hand: the existing same-file-redirect handling absorbs it and the crawl ends normally.

Closes #16
2026-08-11 13:07:15 +00:00
Xavier Roche
0e0072ec03 The footer copyright year is pinned to 2026 and will disagree with the site (#1163)
* Compute the footer copyright year instead of pinning it to 2026

The year was hardcoded because --check diffs the regenerated chrome against
the shipped bytes, so computing it would have turned every 1st of January
into a red CI leg. That left httrack.com, which prints date('Y'), and the
shipped pages disagreeing from next New Year with nothing to catch it.

Compute it, and let --check ignore the digits: a page shipped in an earlier
year stays in sync until something else rewrites it, and the next
regeneration restamps all 15. The mask is scoped to the footer region
because contact.html carries a second 1998-YYYY in its licence text, which
still has to match byte for byte.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test the year tolerance, and stop it accepting non-ASCII digits

The mask was written \d, which in a Python str pattern matches every Unicode
decimal digit, so a footer carrying Arabic-Indic or fullwidth digits passed
--check where it failed before. [0-9] is what was meant.

Nothing exercised the new branch either. The tree ships the year it was
generated in, so on CI the two sides agree and the tolerance is never
reached; it would have run for the first time on a New Year, in production.
Honour SOURCE_DATE_EPOCH as man/makeman.sh already does, which pins the
stamp and gives the test a seam, and add 271 to drive it: a lagging year
passes, a drifted footer and non-ASCII digits still fail. Each assertion is
paired with the mutant that kills it.

Also rename undated() to mask_footer_year(): it masks one field in one
region, and the old name promised a page-wide date strip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 12:26:37 +00:00
Xavier Roche
ceab9c834b Nothing in CI builds our only consumer of the installed headers (#1155)
* Add a CI job building HTTraQt against the branch

HTTraQt is the only consumer of our installed headers, so it is the only
thing exercising the public C++ API surface. Twice an httraqt FTBFS is how
we learned we broke source compat, the second time six days late and by
accident (#1150).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: m4/ in the filter, weekly run, stray-library shapes

m4 macros feed AM_CPPFLAGS for every installed header, so a change there
can break the consumer without touching src/. The weekly run catches Qt
and runner-image drift, since upstream itself is pinned.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fetch HTTraQt from our mirror, shallow, by SHA

No PR should depend on SourceForge being up. xroche/httraqt mirrors the
tree with its full history; fetching the pinned SHA directly also drops
the whole-history clone and the separate pin check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the mirror for what it is, build infrastructure

xroche/httraqt reads as a claim on a project whose author we have just
asked about restarting it. The -ci-mirror suffix does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Compile the installed headers with exceptions and RTTI off too

Qt permits -fno-exceptions/-fno-rtti, so a consumer's translation unit
may see our headers that way and 206 never did. Additive: the
compiler's default mode keeps its own coverage, and the flag pair is
probed like the -std ones so a compiler without it skips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 11:59:06 +00:00
dependabot[bot]
082d87e284 Bump github/codeql-action from 4.37.4 to 4.37.6 (#1159)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.4 to 4.37.6.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.37.4...v4.37.6)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.37.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-11 11:29:44 +00:00
Xavier Roche
58e8aec88f The doc footer drops the patents chicklet, and two pages escape the check (#1157)
* The doc footer drops the patents chicklet, and two pages escape the check

The shipped doc footer had drifted from httrack.com's: no End Software
Patents chicklet, no <small> around the copyright line so it rendered a
size larger than the site's, and a .5rem top margin holding it off the
panel's black edge. doc.css now takes site.css's .sitefooter shape (flex
row, line-height 1.3, no top margin), with the chicklet at the far end.

The chicklet ships as html/images/esp_chicklet.png and is referenced
relatively rather than by the site's absolute src, since these pages are
read off the install directory where that would be a broken image and an
outbound request from a local page. The link points at the canonical
https://endsoftwarepatents.org/innovating-without-patents/, where the
site's http URL lands after two redirects, and uses target="_blank"
because _new is not a target keyword.

The footer also becomes its own doc-chrome region, as the masthead did in
#1154. It used to sit inside the generated bottom region, which index.html
and guide.html do not have: both build their own sidebar and take the
masthead alone, so their footers were hand-maintained and unchecked.
Corrupting index.html's footer text passes --check on master and fails
after this; a page that loses the markers is reported rather than crashing
the generator.

The year stays hardcoded: --check diffs the regenerated chrome against the
shipped bytes, so computing it would red CI every 1st of January.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Report a malformed chrome region per page instead of aborting the run

A page whose region markers are half-present reached region()'s bare
SystemExit, which named no page and stopped the sweep, so a drift on any
later page went unreported. Catch it per page, print the page, and carry
on. A duplicate marker pair is now rejected too: region() substitutes the
first pair only, so a trailing second one shipped unchecked on any region,
the masthead included.

Also add rel="noopener" to the chicklet link, and reword two comments: the
footer's contrast note no longer claims .8em now that <small> nests inside
it, and doc.css points at site.css's .sitefooter as the shape it mirrors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 11:28:05 +00:00
Xavier Roche
ce66c2391a The site menu is injected at a guessed </header>, not a marker (#1154)
* The site menu is injected at a guessed </header>, not a marker

httrack.com serves the html/ pages through a wrapper that injects the site
menu under the masthead, and it locates that spot by finding the first
</header>. Only index.html and guide.html carry a literal
<!-- doc-chrome:masthead -->. The other 13 get their masthead emitted inside
the generated top region, so its content is governed while the marker itself
never reaches the shipped bytes.

doc-chrome.py now emits the markers as part of top, so all 15 pages expose the
same injection point and --check keeps them there. Shipped bytes change by
exactly those two comment lines per page and nothing else, which matters
because the site copies these files verbatim. Two checks make the guarantee
direct rather than resting on "it sits in a generated region": a page carrying
the chrome whose file has lost the marker fails --check, and a page loading
doc.css with no markers at all is now an error instead of dropping out of the
check unnoticed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Reword the three new comments after the review prose pass

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 09:49:17 +00:00
Xavier Roche
49e93b171b Nothing runs the test suite under emulation, so the buildd finds those failures first (#1149)
* Run the suite under qemu-user, where only the buildd does today

cross-arch.yml compiles for the nine ports architectures and stops at make, so
the Debian buildd is the first machine anywhere to execute the suite on one. It
reports by failing the package build, which is how both failures in #1146
reached sid past a green cross leg.

s390x rather than the hppa that reported it: a release architecture, so the
container image and the build-deps come from the main archive instead of
debian-ports, and big-endian at no extra cost. Gated on tests/ changes, since
the harness is where an assumption about the native shape of a process lands.

The script reads /proc/self/cmdline back before building: a runner that stopped
emulating would leave this leg green while covering nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Read the emulation probe off a child, and pin what must pass

qemu answers /proc/self/cmdline from the guest argv it was handed, so a process
asking about itself sees bash and never the interpreter. Only another pid
reaches the host kernel, which is both why the self-check would have failed
forever and why the harness sees the shift at all.

make check exits 0 for an all-SKIP run, and this leg skips the crawl tests by
design, so a container that lost a dependency would report a green covering
nothing. Read the pass count back and pin a floor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* TEMPORARY: revert #1147 to prove this leg reds on what it targets

Backed out in the next commit. Without it the leg cannot be shown to catch
anything, and the issue asks for exactly that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore #1147, the control having shown the leg reds

The reverted run failed 105_suite-timeout under emulation, which is what the
hppa buildd reported. 151 skipped there rather than failing: its pacer fires on
this runner, where an emulated configure is far cheaper than on a buildd, so the
leg reproduces the argv-shift failure deterministically and the wall-clock one
only on a slow enough host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 08:56:18 +00:00
Xavier Roche
3ea7a773fa Installed headers break under some include orders (#1152)
A consumer picks its own include order, and five ordered pairs of our installed headers do not compile. Both causes are a guard placed over a block that a later, better-informed include should still have got to run: htsstrings.h kept a partial copy of htsglobal.h's attribute helpers behind the same `#ifndef HTS_UNUSED`, so including it first starved htssafe.h of `HTS_CHECK_RESULT`; htsbasenet.h tested `HTS_USEOPENSSL` before anything had defined it, and its own `HTS_DEFBASENETH` then locked the OpenSSL block out for good, leaving `SSL` undeclared in htsopt.h. Both now include htsglobal.h rather than guess at it.

The first one is not hypothetical. It shipped in 3.49.15 and broke httraqt, our only reverse dependency, as an RC-severity FTBFS ([Debian #1143660](https://bugs.debian.org/1143660)); Debian worked around it downstream by dropping the include and it never reached us, so it is still live in 3.49.20. The second has never been reported and turned up while sweeping for the first.

205 and 206 could not catch either, because they compile each installed header on its own, and on its own every header passes. 269 sweeps all 182 ordered pairs in C and C++ and in both `HTS_INTERNAL_BYTECODE` states, six installed headers gating declarations on that macro. Its control is a synthetic pair that compiles in one order only: it proves the sweep can see the shape it hunts, without asserting that any real header stays broken. Verified by reverting each fix in turn, each failing 269 alone and naming its own pair, and by making the sweep vacuous, which the control catches.

Closes #1150
Closes #1151
2026-08-11 10:32:43 +02:00
Xavier Roche
5175cf986f A caught signal kills every in-flight transfer, and a stopped FTP transfer waits twice (#1145)
`back_wait()` called `select()` and threw the return away. On EINTR Linux leaves the three fd sets exactly as the caller filled them, so every in-flight socket reads back as an exception and the slot dies with a fabricated "Receive Error": any caught signal costs a transfer, and on master a burst of SIGCHLD against a trickling body truncates the mirror. Reading the return is only half of it, because the prompt `^C` people rely on during an HTTP download exists only through that same accident. Nothing on the receive path ever read `opt->state.stop`, so a correct `select()` on its own would leave HTTP waiting out `--timeout` the way FTP does today. `back_wait()` now sweeps every live non-FTP slot when the stop flag is up, sharing the mirror-limit abort loop, and reports the user's stop instead of a socket failure. FTP slots are left alone: the worker thread owns both socket and slot, and finishing one under a live `fwrite()` is a use-after-free. So are the transfers running under a cap, because `back_checkmirror()` raises the same flag the first time `-M` or `--max-time` is reached: while that grace runs the sweep takes only the slots still waiting to connect, and the limit block that owns those transfers ends them with its own status and their WARC record.

On the FTP side a stopped transfer against a silent server waited out `--timeout` twice, once on the data socket and again on the QUIT reply nobody was going to send. Only the second window goes. The data wait keeps its own, since the first `^C` promises to finish pending transfers and clipping that wait truncates transfers still making progress (#1107). With `--timeout=0` a stop stays unbounded, which is what the user asked for, and a second `^C` already exits at once. No thread drain either: teardown is #1051, and a partial drain would hang the exit against that unbounded wait. Tests 262 and 263 cover both halves against a no-signal control arm, and the backstop self-test now pins the wider sweep with the FTP slot as its control.

Closes #1110
Closes #1096

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 08:31:07 +00:00
Xavier Roche
d25dd436ee cleanup_push traps a SIGPIPE the runners never let it see (#1141)
Settles #1136 as (b): PIPE leaves cleanup_push's trap list.

A non-interactive bash cannot trap a signal it inherited as ignored, and that is how the ubuntu runners hand SIGPIPE down, so `trap -p PIPE` reads back `trap -- '' SIGPIPE` and the entry was inert on the very hosts it was meant to cover. `perl -e '$SIG{PIPE}="IGNORE"; system("bash", ...)'` reproduces the CI red exactly: the subject sits out its full 30s sleep and exits 0. The container and macOS legs passed because their process ancestry resets the disposition. Even where it installed the entry bought nothing, since bash runs the EXIT trap for a fatal PIPE and the stack drains either way; only QUIT needs its own trap, because a non-interactive shell ignores it.

256 now reads the trap list back out of a live subject and checks it against the list it knows, so the next signal added without a leg goes red. Putting PIPE back makes that new case fail.

Closes #1136

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 08:18:36 +00:00
Xavier Roche
0fe1e2ba34 An aborted --update run purges against the files it never reached (#1142)
An interrupted run still ran the end-of-mirror purge, comparing old.lst against the truncated new.lst of what it had managed to fetch. Everything it had not reached was deleted, and `--purge-old` is on by default, so no flag has to be set for this.

The gate now reads `opt->state.stop` and `state.exit_xh` once the crawl loop ends, and skips the purge and the `--changes` report when either is set. Reading the cursor is not enough: a stop that lands while a page is in flight makes the parser drop the links that page carries, so a run can reach the end of its list having never queued half the mirror. Every stop route sets `state.stop` (signals, the GUI, and the `--max-time` and `--max-size` caps through `hts_request_stop`), and nothing sets it on a mirror that ran to the end. The report is suppressed for the reason #1061 gives: a run that never walked the list cannot say what is gone.

The rest of the completion sequence stays. `hts_finish_makeindex` and `index_finish` close files the run was writing, `singlefile_process_mirror` only rewrites pages that arrived whole, and `warc_close_opt` commits records for transfers that really happened, where `warc_abort_opt` would throw them away. The summary line is the other change: it says "mirror aborted after" instead of claiming a complete mirror.

Test 261 covers both halves, a SIGTERM during a stalled FTP `--update` and a `--max-time` cap firing while the page carrying the only links to six leaves is still in flight. The complete crawl in between still purges a file it dropped, so a fix that merely silences the purge fails there.

Closes #1109

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:47:55 +00:00
Xavier Roche
9bc300296f The Windows suite has no wall-clock bound, so a blown budget reaches the workflow timeout (#1144)
The Windows suite has no wall-clock bound. Its heartbeat kills only a progress log that has gone static, and the suite deadline is read between tests, but every path that follows a blown budget writes a progress line of its own, and each of those re-arms the staticness window. A test starting a second under the deadline can spend the per-test budget, then a traced rerun on a budget of its own, and land on the workflow's 45-minute step timeout. A cancelled step keeps neither its log nor its artifacts, so nobody learns what it was doing.

The heartbeat now takes an absolute cap and kills at 2400s whatever the progress log is doing, and the traced rerun is charged to the suite deadline instead of getting a second budget. 226 asserts the arithmetic against the workflow's own timeout, 171 drives the cap on its virtual clock against a suite that never stops reporting, and both assertions were checked by mutation. I could not reproduce any of this locally, since it needs a Windows runner, so the fix rests on reading the two scripts rather than on a repro.

The artifacts of the runs the issue cites are still live, and they say something else. Both Win32 legs finished the suite on their own terms and kept their step log and artifacts. Run 31397002827 ran 1264s and failed on 79_local-proxytrack-webdav-mime, whose proxytrack bound none of three port pairs; run 31405831680 ran 1060s and failed on 226 tripping over the new name in the driver's comment. 260 ran on both Windows legs of the first run and passed, in 11s on Win32 and 13s on x64, so it goes back into the selection through a crawl-harness category. The one real overrun is the x64 leg of 31405831680, which ran past 44 minutes and was cancelled with nothing surviving. 260 was already renamed out of the globs in that tree, so it was not the cause, and nothing survives to say what was.

Closes #1126

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:47:36 +00:00
Xavier Roche
a6315d04a3 Two suite tests fail on Debian's emulated hppa buildd (#1147)
* The hppa buildd kills 151 and hides the engine from 105

Debian's hppa is a qemu-user chroot, and two suite tests fail there in 3.49.20-1.

105 read /proc/<pid>/cmdline's first field as the program name, which binfmt_misc
gives to the interpreter, so its own probe never saw the fake engine start.
proclib already knew that shift; hoist the two regexes it matches on and reach
them from the shell too.

151 was killed at the 600s budget with four cases to go. The pacer reserved 1.5x
the last step for one more step, and its first twelve cases are cheap rejects
while the last four run configure to completion. Reserve for every step left
instead, past the first, which 196 spends warming a shared config.cache.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Order 151's expensive cases first instead of projecting the pacer

Reserving every remaining step at the last one's cost skips a run that fits
whenever one step is slower than its neighbours: a 2x outlier in 196 case 2
projects over four remaining cases and skips a run that ends at 562s of 600.
Keep the one-step-ahead reserve and give 151 the ordering it needs instead --
its four full configures first, so no step left can outrun the reserve the one
before it set.

Also fold proc_program_name onto the awk prologue so the interpreter shift has
one implementation, keep qemu-img-static and friends out of the shift (they feed
kill -9), and pin the reserve's half-step margin, which no fixture covered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:06:51 +00:00
Xavier Roche
893b542953 getrandom() is called undeclared when configure finds the function but not its header (#1143)
configure probes `sys/random.h` and `getrandom()` separately, so a libc that exports the symbol without shipping the header reached the call site with no prototype in scope. gcc 14, clang 16 and C23 reject an implicit declaration, so that build fails instead of falling back to /dev/urandom. #1138 guarded the include on both macros and left the call on one; a single macro derived at the include now gates both.

Nothing reached that cell before. There is now a CI leg that configures with `ac_cv_header_sys_random_h=no`, checks the combination really came out that way, and builds: red on master, green here. With getrandom gone it is also the only leg that runs the /dev/urandom fallback. The `enum { x = 1/(cond) }` pin at the call site covers the rest, so if the two guards split again it is a build error even where the compiler would only warn.

`hts_random_bytes` now answers a zero-length ask before reaching any source. The getrandom path already did; the fallback opened /dev/urandom to read nothing, and Windows left it to RtlGenRandom's undocumented behavior.

The new selftest (`-#test=random`, test 267) covers `hts_random_bytes` itself, not the compile break. It draws eight times with a different filler byte each round, so a byte the fill skipped is distinguishable from one it wrote, and it checks that nothing past the ask is touched. Mutants that fill half the buffer, run one byte past the end, return a constant, or scribble on a zero-length ask all fail it. A counter stub passes, since what it asserts is extent and liveness, not entropy quality.

Closes #1139

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 06:44:37 +00:00
Xavier Roche
3232f81373 Wizard domain answers match beyond the domain (#1140)
Three of the wizard's answers built their host filter as `<host>*`, with nothing between the host and the wildcard, so `+foo.com*` also authorized `foo.com.evil.org`. Answering "mirror all links located on the same domain" on a page linking to such a host pulled it into the mirror. Answer 5's allow-the-domain branch had the same shape, and answer 2's exclude over-excluded for the same reason. All three now emit `<host>/*`, and nothing is lost at the bare-host end because both forms the wizard matches against insert a `/` after the host.

This is a visible behavior change: someone who answered "same domain" and relied on the loose match will now see fewer hosts accepted.

To make the emission testable without a crawl, it moved out of the answer switch into `wizard_answer_filter()`, which `-#test=wizardfilter` drives from tests/264. A differential against the old inline code over answers, hosts and paths came back with only the three intended changes; answer 1's collapse of a doubled slash is preserved and pinned by the test.

Closes #1119

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 06:25:06 +00:00
commitmaniac
4a5854f31d htsrandom: the macOS build fails on an include only getrandom needs (#1138)
macOS's sys/random.h uses u_int without defining it, so a bare include does not compile. Autoconf's header test passes anyway, because it prepends sys/types.h before testing. The only consumer of that header here is getrandom(), which macOS does not have, so the include is dead weight on that platform. Guard it on HAVE_GETRANDOM as well, matching the guard already on the call.

Reported by MacPorts while upgrading to 3.49.19. Tested on macOS 10.7.5 with Xcode 4.6.3.

Signed-off-by: commitmaniac <201806046+commitmaniac@users.noreply.github.com>
2026-08-11 07:08:26 +02:00
Xavier Roche
df0fbd4711 Put the URL pane back as the software-centre screenshot (#1137)
#1132 moved to the scan-rules pane because every wizard pane advertised a
loopback address. #1135 reshot the set with the crawl addressed as
www.example.com, and the site now serves those bytes.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:40:26 +00:00
Xavier Roche
77d03052a6 Release 3.49.20
Two engine fixes are the substance of this one. An off-site link whose name looked like a file was parsed as a page, and then spent the referring page's whole depth budget on the foreign site. And a mirror that was asked to quit while an FTP worker was still receiving printed its exit notice, then waited forever on that slot. The rest of the range is the documentation refresh, the WebHTTrack language menu, and test and CI work.

VERSION_INFO goes to 3:13:0, revision only: no installed struct changed shape, no export came or went. debian/changelog picks up the League Spartan webfont the docs now bundle, and Standards-Version 4.7.4 still matches current policy.
2026-08-10 21:11:46 +00:00
Xavier Roche
628fc70727 The Windows suite runs none of the tests covering the libraries it depends on (#1131)
* The Windows suite runs none of the tests covering the libraries it depends on

ci-windows-suite.sh selects its tests by topic word, and no glob matches the
harness self-tests: 254, 256 and 257 (testlib.sh) and 258 (crawllib.sh) never
run there, though every test the job does run now sources one of those
libraries. Two categories pull them in; 260 keeps its name clear of both until
#1126 explains why a deliberately failing crawl wedges the Win32 leg.

172 stubs one test per category, so it gains two and its tally moves with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* 258's httrack shim never reaches PATH on Windows, and 172 cannot see 260 come back

Two gaps the review found in the tests this PR newly selects there.

The argv shim in 258 lives under TMPDIR, which the Windows job sets to a
cygpath -m drive-letter path, so bash reads "D:/a/_temp/..." on PATH as two
entries and the real engine answers every case. testlib gains posixpath(),
nativepath's inverse, which 226 was already open-coding; 258 also proves the
shim is reachable rather than assuming it, since an unreachable one silently
tests the wrong binary.

172 could not kill a crawllib glob widened to *_crawl-*, which drops 258 and
selects 260: in the stub tree the tally happened to hold. A decoy named the way
260 is, selected by nothing, plus a duplicate-selection check on the progress
log, close both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The decoy and duplicate checks report after the tally that hides them

Every wrong selection also moves ran=, so the tally fired first and the two
named checks never got to say what went wrong. Reordered ahead of it, and the
decoy failure drops the whole suite output it was quoting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* 256 stops at the first signal that breaks, so three of the four go unmeasured

Windows reds on the SIGINT leg: the subject sits out the whole sleep and exits 0,
so the trap never ran. The loop ends there and says nothing about HUP or QUIT,
which is the half needed to tell a missing trap from a signal MSYS will not
deliver to a bash of its own. It now reports every leg with the time each took.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* MSYS never delivers INT or QUIT to a bash its own kill sent them to

Measured on both Windows runners now that 256 reports every leg: the SIGINT and
SIGQUIT subjects sit out the whole 30s sleep and exit 0, where TERM and HUP
return at once with the trap's status. Never delivered, rather than delivered
and ignored, which is why 243 and 255 already skip there. Those two legs are cut
on Windows; the other four sections of 256 keep running.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The 260 exclusion lives in a filename, where a rename would ship the wedge

Review found the #1126 exclusion resting on prose and a glob miss: rename 260
into one of the topic words and the Win32 leg wedges again, with nothing to say
why. The driver now names the file and stops when it is gone, so a rename reds
in the first second rather than after 45 minutes on the runner. 172 drives that
by renaming the staged copy.

Also from the review: 256 tests PIPE, the fifth signal in the trap list it had
never sent, and takes its signal set from an array rather than a split string;
258 keeps $shim native and converts only the PATH entry; two comments that had
grown by accretion are split back to one point each, and 258's overstated the
shim guard, which buys a clearer failure rather than the only one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Sending SIGPIPE was scope creep, and four Linux legs disagreed with my box

The fifth signal in cleanup_push's trap had never been sent, so I added it. On
the runners the subject sat out its whole sleep and exited 0, where bash 5.2.37
here returns at once with the trap's status: a self-sent SIGPIPE reaches a
non-interactive bash differently depending on the build. Reverted to the four
signals this test has always sent, and the gap is filed rather than guessed at.

Refs #1136

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:36:19 +00:00
Xavier Roche
7ff60e9de7 Reshoot the WebHTTrack guide screenshots without the loopback address (#1135)
The seven screens that showed the crawled address or the mirror folder now
read www.example.com.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:23:48 +00:00
Xavier Roche
4588831dcd The doc screenshots advertise 127.0.0.1 and a random port (#1133)
The crawled address is on screen in the wizard panes and names the mirror
folder, so every guide screenshot showing it, and the software-centre image,
carried a loopback artifact.

--site-host and --site-port let the walk address its own server by name; the
workflow maps www.example.com and lowers ip_unprivileged_port_start so it can
answer on 80. A local run keeps the old behaviour.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:44:26 +00:00
Xavier Roche
adbc46a5b8 The software-centre screenshot advertises a loopback address (#1132)
The wizard pane it pointed at has one piece of user content, the URL box, and
the walk crawls a throwaway loopback server, so the hero image read
http://127.0.0.1:43925/. The scan-rules pane shows real capability and no
address. Reverting to a wizard pane wants the walk to crawl a named host.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:40:30 +00:00
Xavier Roche
249b96b975 Advertise WebHTTrack with a screenshot from this decade (#1130)
The metainfo pointed at a 300x238 JPEG of the UI in a Mozilla window on
Windows 98 chrome, unchanged since 3.30.1. It points at a guide asset now, so
future reshoots refresh the software-centre listing on their own.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:54:24 +02:00
Xavier Roche
d8402d7ad2 local-crawl.sh keeps its own copy of the server launch and the crawl backstops (#1123)
* local-crawl.sh keeps its own copy of the server launch and the crawl backstops

crawllib.sh was extracted from the tests that drive local-server.py themselves,
leaving local-crawl.sh -- the launcher the other 62 crawl tests go through --
with a second copy of the launch block and of the --max-time/watchdog pair. It
now sources the library: the server comes up through local_server_start, the two
backstop constants live in one place, and the four near-identical httrack
invocations collapse into run_pass/require_pass.

The engine is still backgrounded here rather than run through local_crawl, which
would put it in its own process group where the suite watchdog's stack dump
cannot reach it (105).

--tls earns its place in the library now that a caller wants it, and 258 covers
it by fetching over TLS rather than reading the scheme back off BASEURL.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* 14_local-https asserts nothing about the scheme it crawled

Its audit (5 files, no errors, the page is there) reads the same whether the
crawl was TLS or plain, so a harness that lost --tls would turn it into a
duplicate of the plain-http tests. The mutation matrix for local-crawl.sh's
--tls passthrough survived on exactly that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the comments added by the crawllib adoption

Signed-off-by: Xavier Roche <roche@httrack.com>

* Nothing tests that local-crawl.sh fails loudly

The 62 tests behind it read nothing but its exit status, so a harness that
swallowed a failed crawl would report every one of them green. 260 covers the
three ways a run must end badly: a crawl the engine refuses, a later pass that
fails while the first pass's mirror stands, and an audit that does not hold.
The middle one is the case the others cannot see, since a missing mirror ends
the run at the host-root check whether or not the pass was required to succeed.

run_pass now returns the crawl's status instead of leaving it in a global, and
require_pass runs the pass itself, so a call site names its log and label once
and cannot check the previous pass's result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the new harness test out of the Windows suite's topic globs

Named 260_local-*, it was selected by ci-windows-suite.sh, and the Win32 leg
then died at its step timeout twice on the same commit while x64 passed. It is
the first Windows-selected test that deliberately fails a crawl, and the suite
kept neither a log nor an artifact, so the cause is unknown; #1126 records what
was measured. The other harness self-tests (254, 256-258) sit outside those
globs too, and widening the Windows selection is its own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Do not name the Windows driver in the new test's comment

226 greps every .test for that filename and demands the credential-scrub line
when it finds one, since sourcing the driver starts an interpreter that can post
statuses. A comment cannot be told from a source line by that grep, so the
mention alone reddened every leg. 226 skips on this host, so only CI saw it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 17:22:47 +00:00
Xavier Roche
b268caa220 The guide still explains the language dropdown that #1086 removed (#1125)
The web note told readers the menu opens blank and that its first entry means
leave the interface alone. That placeholder option is gone; the menu now shows
the language in force and picking another reloads the page in it.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 17:21:23 +00:00
Xavier Roche
fb59dd6aba Reshoot the WebHTTrack screenshots after the masthead fix (#1129)
Every screen carries the masthead, so ef100363 dated all eighteen.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 17:18:19 +00:00
Xavier Roche
ef100363a7 The WebHTTrack tagline bar sits off the wordmark's baseline (#1127)
#927 and #1111 cancelled the SVG's built-in whitespace in html/doc.css, so the
doc pages tightened and the WebHTTrack UI, which renders the same masthead from
its own stylesheet, kept the gap.

Measured with getBoundingClientRect in the same browser: bar top minus image
bottom was 0 on the server pages against -1.09 on a doc page, and is now -1.09
on both.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 16:28:08 +00:00
Xavier Roche
c3e1a202c1 The GUI guide's screenshots are stale, and the palette step greys out their backgrounds (#1124)
* Reshoot the WebHTTrack screenshots in the GUI guide

The set was captured at 3.49-14, before the masthead wordmark and the rings
background became SVGs (#916, #910) and before the welcome page learned to
mark the language in force (#1086).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Quantize the guide images by median cut, not max coverage

Max coverage spreads the 256 entries over the colour cube regardless of how
many pixels each region costs, so the large flat panel and rings backgrounds
came out grey or tinted. Mean channel error drops from 5.3 to 0.5, for 650K.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:38:05 +00:00
Xavier Roche
f55f68c94e The documentation pages ask for Trebuchet before the system font, and have no display face (#1122)
doc.css put "Trebuchet MS" and Verdana ahead of system-ui in the body stack, so only Linux ever reached the platform font: Windows and macOS both stopped at Trebuchet. This reorders to system-ui first, keeps Trebuchet and Verdana behind it, and puts League Spartan 700 on h1 and h2. It goes on headings alone because it is a geometric display face, and it reads worse than the system font over the long prose of the reference pages. The website's stylesheet now does the same.

The font is self-hosted rather than linked from Google Fonts, since the pages are read from an installed directory and over file:// as often as over http. It is the latin subset at weight 700, 12888 bytes, with its OFL beside it, and `doc.css` reaches it by a relative URL that CSS resolves against the stylesheet rather than the document. New test 259 walks every `url()` in `html/*.css` and fails when the target is missing from the installed tree or the tarball, which test 225 cannot do: it drops the htmldir region on purpose, so either install rule could have gone missing with nothing going red.

`debian/copyright` gains an OFL-1.1 stanza, text inlined because OFL-1.1 is not in common-licenses. It previously declared every file GPL-3+, which is wrong for a bundled font. Debian ships `fonts-league-spartan`, but a `@font-face` cannot portably reach a fontconfig family and the tarball and macOS builds have no such package, so the bundled copy stays. Upstream declares no Reserved Font Name, which is what lets the subset keep the family name.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 16:10:25 +02:00
Xavier Roche
749de004c7 Engine self-tests assert a substring, so the value underneath can change unseen (#1121)
* tests: pin the engine self-test assertions that only matched a substring

Five self-tests asserted their result with a substring grep, so a change
in the value underneath went unseen: moving the advertised
Accept-Encoding from identity;q=0.9 to q=0.8 leaves "acceptencoding
self-test OK" in the output and the old test green. assert_selftest pins
the line and the exit status and prints want beside got.

Two more compared the whole line already but through a bare
test ... || exit 1, which reds without naming what differed.

The greps were also pipes into grep -q, which AGENTS.md forbids; at these
output sizes the producer finishes before grep exits, so removing them is
conformance rather than a fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: shorten the watchdog capture comment

Two lines, and "can SIGPIPE" rather than a claim of a live bug: at the
param block's size sed always finishes writing, so the trap is latent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:18:35 +00:00
Xavier Roche
09e3e1f62b The tests driving the local server each rebuild its launch, and most crawl with no time cap (#1120)
* tests: give the local-server cluster a shared launch and crawl helper

Twenty-two tests drive tests/local-server.py themselves rather than
through local-crawl.sh, and each rebuilds the same launch: resolve the
script, find a python, background it, poll stdout for the ephemeral port,
register the reaping, build the base URL. Only three of their crawls
carried --max-time and none a watchdog, so a wedge here ran until CI
cancelled the job at 45 minutes, taking the log with it.

tests/crawllib.sh adds local_server_start and local_crawl, covered by
258_crawllib.test from a subject process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: review fixes on the local-server helpers

Drop local_crawl's --deadline, which no test outside its own used;
CRAWL_DEADLINE already drives the watchdog for local-crawl.sh. Truncate
the crawl log rather than append, matching every redirect it replaced.
Skip the reap when the server is already gone: 240 stops its own server
mid-test, and stop_server answers an unresolvable pid on Windows by
killing every python.exe.

235 keeps its own run_with_timeout: it already had a watchdog, and
local_crawl's failure would preempt the four messages it prints.

258 pins the whole argv per call and truncates it first, so an
assertion cannot pass on the previous crawl's arguments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: let a test disarm its server's teardown

The cleanup registered by local_server_start expands its arguments at
push time, so a test that stops its own server mid-run leaves the frame
holding a dead pid. Signalling a number the system has since recycled
kills an unrelated process, then stalls reap_bounded for its whole grace
period. Track the live servers in SRV_PIDS and register the slot, so
local_server_stop can empty it. 240 is the caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:37:19 +02:00
Xavier Roche
4364e6c93e The engine self-test scripts each carry their own teardown, invocation and silent comparisons (#1118)
* tests: an assert_* vocabulary and an engine self-test façade

assert_eq/assert_match/assert_no_match/assert_file/assert_dir/assert_min_bytes
print what they wanted beside what arrived, and assert_selftest wraps the
httrack -O /dev/null -#test=NAME invocation the tier spells out by hand,
checking the exit status the old `test "$(...)" = ...` form discarded.
httrack_path resolves the binary the ten tests that cd away resolve themselves.

257 exercises each one from a subject process, since an assertion that fires
exits: every helper has a case that must hold and one that must fire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: the engine self-test tier joins the shared library

The 01_engine-* / 01_zlib-* tier sourced nothing: 50 hand-written trap pairs,
a repeated absolute-binary incantation, and comparisons ending in a bare
`|| exit 1`. Those become cleanup_push, httrack_path and the assert_* helpers.

Two latent traps go with them: the mime helper piped into head -1, which
SIGPIPEs the engine under pipefail, and half the signal traps only removed the
temp dir on EXIT.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: review fixes on the assertion vocabulary

Drop assert_no_match, assert_dir and assert_min_bytes: 257 was their only
caller, and a dead helper is what PR #1113 deferred them to avoid. The
duplication they would serve (five copies of size_of, the grep -q <<< cluster)
lives in the tests PR 4 covers.

assert_file and assert_match gain real callers here instead: four artifact
checks and the renameover/cookieimport/direnum output matches.

The -O /dev/null comment claimed the engine refuses to start without an output
path. It does not: htscoremain.c dispatches -#test ahead of the usage gate.

hdr() in 01_engine-header.test kept its capture in an assignment, where errexit
can still see a failing engine, and httrack_path now names a failed cd rather
than returning a bare /httrack.

257 grows the cases that kill a prefix compare, a case-folding matcher, a
dropped label, and a status check satisfied by exit 127.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: retry the proxytrack port pick instead of reding on a lost race

freeport hands out a port it has already closed, so between the pick and
proxytrack's own bind another process can take it: 79 lost that race on the
Windows x64 leg. Factor the pick, the listen wait and a three-attempt retry
into testlib.sh, and have the four proxytrack tests launch through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: give each proxytrack attempt its own log, and fold the port pickers

Review of the retry found the .done marker the pty drainer writes by path
escapes the fresh inode, so a killed attempt could satisfy 153's wait_drained
and let its console-leak check read a half-copied log. Each attempt now writes
its own $ptlog. The two ports come from one freeport call, which cannot hand
back the same number twice, and the three copies of that bind-and-close trick
become one helper. A retry says so on stderr, a proxytrack that dies announcing
neither outcome is named as such, and a launcher that leaves no pid is caught.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 07:59:28 +00:00
Xavier Roche
35319c05be SIGTERM never reaches teardown while an FTP worker is receiving (#1106)
`sig_finish` raises `state.exit_xh` on SIGTERM, but the mirror's wait loop in `hts_mirror_wait_for_next_file` never polled it: it leaves only when the link is done, when `back_checkmirror` fails, or when `hts_loop_tick` says so. Nothing there ends a live FTP worker, and the call that does, `ftp_stop_workers()` from `back_delete_all` and `hts_free_opt`, sits past the loop in teardown. So "Exit requested to engine" printed and the crawl then waited forever on the receiving slot. Polling `exit_xh` there and returning the way the two neighbouring abort checks already do lets the outer loop's own `exit_xh` test end the crawl and reach teardown. The poll fires on any nonzero value, so a fatal `exit_xh = -1` raised while a slot is live no longer wedges either.

`hts_mirror_process_user_interaction` needed the same poll. The loop above is only reached while a socket is free, so with every slot busy the mirror parks one frame higher, in the wait for a free socket, whose guards were the identical pair. Both polls are load-bearing: with only the inner one, three stalled links against two sockets still hang; with only the outer one, the single-link case does.

`^C` is untouched. It sets `state.stop`, not `exit_xh`, and neither poll reads it, so a first `^C` still finishes pending transfers and a second still exits immediately. That half of #1096 is a separate bug on a separate path, `wait_socket_receive` in `htsftp.c` never reading `state.stop`, and it is not fixed here, so the issue stays open. Nothing in the code makes letting a receiving slot finish a deliberate policy, and a stalled HTTP body does not behave like the FTP one: measured on master, SIGINT ends a stalled HTTP transfer in 0.1s and a stalled FTP one only after `--timeout` expires. The HTTP promptness is itself accidental, #1110.

One consequence worth naming. The `exit_xh` route falls through the whole completion sequence, `--purge-old` included, so a SIGTERM during an `--update` run can purge files it never re-fetched. That is pre-existing on that route, but this fix reaches it where the engine used to hang, and a user who reached for SIGKILL kept their mirror. Filed as #1109.

`tests/255_local-ftp-sigterm.test` signals a crawl parked on a stalled FTP body with `--timeout=0 --max-time=0`, so only the fix can end it. It pins the exit path rather than just the exit: reverting the hunk hangs, and returning the caller's error code or the already-done code leaves the wait too but loses the cache or drops the link silently, which the log assertions catch. A saturated leg covers the outer poll, and controls check that a single `^C` still completes an in-flight transfer, that the `^C` actually reached `sig_leave`, and that an unsignalled crawl is unaffected. It skips on Windows, where MSYS cannot deliver the signal to a native exe.

Refs #1096

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 07:40:12 +00:00
Xavier Roche
e4fbe856e5 The chrome check cannot see the masthead on two of the fifteen pages (#1116)
guide.html and index.html build their own sidebar, so they carried no markers at
all and the masthead above it drifted freely: changing the tagline in guide.html
left --check reporting 0 out of sync, where the same edit in faq.html was caught.

The masthead is byte-identical on all fifteen, so it becomes its own region that
a page can take without the generated navigation. Both now follow the tool: a
canary in MASTHEAD reaches 15 pages, where before it reached 13.

Closes #1115

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 08:19:41 +02:00
Xavier Roche
4f49eec86b The docs and the site disagree on what HTTrack is called (#1114)
The site says "Free software offline browser", these pages said "Open Source
offline browser", and the French tagline has always been "Aspirateur de sites
web libre", so one language already made the claim the other did not. Xavier
picked the site's wording for the web surfaces.

The lang/ catalogs keep theirs: they are the GUI, the English value is their
join key, and 17 carry a real translation that changing it would orphan.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 07:41:45 +02:00
Xavier Roche
f8ed3e771f Every test script carries its own copy of fail(), skip() and the teardown traps (#1113)
* tests: split the wedge forensics out of testlib.sh into proclib.sh

testlib.sh is sourced by 74 tests, but 235 of its lines are the process
forensics only the timeout driver and the Windows suite ever call.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* tests: fold the copied fail(), skip() and teardown traps into testlib.sh

62 copies of fail(), 7 of skip() and ~110 hand-written trap pairs go away.
cleanup_push installs the signal half two thirds of them never had, with
errexit off (#773); it takes a command and its arguments, not a shell
snippet, so it needs no eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: fix the review findings on cleanup_push and the proclib split

Re-sourcing testlib.sh reset the cleanup stack, so 172 leaked its temp tree;
the stack is now self-preserving and 256 covers it. Both argv slices carry
the [@]+ guard an empty frame needs on bash 3.2. The engine matchers move to
proclib.sh beside their only consumers, dropping the SC2034 pair that hid the
cross-file reference. 100 and 105 register incrementally instead of rebuilding
an EXIT trap, which is what the LIFO stack is for; 103's scanner now inspects
the libraries, whose traps its errexit gate had stopped reaching. 250's dead
poll stub waits for the child to announce, closing the other half of #1092.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: keep the default $top_srcdir relative

100_local-purge-longpath.test hands it straight to python.exe, which cannot
resolve the absolute MSYS path the library had started defaulting to, so the
fixture server never came up and the test failed where it used to skip. The
Windows suite exports no value of its own, so the default is what it runs on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 21:58:42 +02:00
Xavier Roche
4231a141d3 The documentation footer fails AA contrast in light mode (#1112)
#878 gave the footer color: #eef, which is 3.56:1 on the --field it sits on,
below the 4.5:1 AA wants for text this size. Dropping it inherits body's --ink
at 4.62:1; no lighter colour clears the bar on that field, and --ink-soft is
worse there at 2.38:1. The dark override already passes at 7.76:1.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 18:13:54 +02:00
Xavier Roche
314ee45350 The tagline bar still sits off the wordmark's baseline (#1111)
#927 gave the wordmark negative margins to put the black bar back against it,
but left the .4rem that pushes the bar down, so about 5.3px of gap survived the
fix meant to close it. Dropping it leaves the collapsed sibling margin at the
-1.1px #927 measured for the SVG's overshoot below the baseline.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 18:07:45 +02:00
Xavier Roche
54d93200a4 Two suite tests fail on the hppa, powerpc and ppc64 Debian buildds (#1108)
* Fix two suite tests that fail on the hppa, powerpc and ppc64 buildds

246_engine-connect-stop needs its connects to TEST-NET-1 still in flight, but
probed that the instant connect() returned; a network that rejects 192.0.2.1
rather than dropping it answers milliseconds later, and back_wait() then ends
slots the control asserts it left alone. Settle before re-reading the sockets.

183_altstack-worker pinned the worker's fault to SIGSEGV, which hppa does not
report for a spent thread stack. Accept any fault the handler covers, and print
the report when it is none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the two comments this change adds

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Match the worker's fault by signal name, not number

SIGBUS is 7 on x86 and 10 on hppa, so a numeric set cannot be right on
both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Gate the dead-host fixtures on a network that refuses TEST-NET-1

tools/hostile-net.sh runs a command in a netns whose peer owns 192.0.2.1 and
listens on nothing, behind 30ms of netem so the RST lands after connect()
returns rather than before it. That is the powerpc and ppc64 buildds, and
without the settle probe 246 fails under it with the buildds' exact log.

The script refuses to run when the address is dropped or answered, so a leg
that reproduces neither cannot pass silently. Widen SHELL_SCRIPTS to tools/*.sh
while here: it was an explicit list that had already lost mk-sbuild-chroot.sh.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cut the comments and rules this change adds

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the hostile-network leg skipped on the settle probe

Without it a netem that stops delaying leaves the leg green while 246 skips on
the older check, so the branch the leg guards never runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 16:59:11 +02:00
Xavier Roche
6e98e33495 The htsserver tests copy five helpers per file, and the copies have drifted (#1105)
* The htsserver tests copy five helpers per file, and the copies have drifted

Each of the ~20 tests driving htsserver carried its own freeport, start,
srvpid, portof and request, plus an inline python socket client: eleven
copies of freeport in three variants, eight of srvpid in six. Two of the
divergences were live bugs. Five copies read the announced pid with
"sed ... | head -1", which SIGPIPEs the producer so that under pipefail a
spurious 141 becomes the test's verdict; 91_webhttrack-directory.test wrote
a firstline() to dodge that and nobody copied it. And the socket timeout had
drifted to 15s in one copy and 30s in four, with only the 15s copy catching
socket.timeout and reporting, so the other four wedge where that one fails
cleanly.

tests/webhttracklib.sh and tests/httpclient.py now own picking the port, the
background launch under an ignored SIGTERM, the wait for the URL=
announcement, the pid the server reports (Windows announces none) and the
reaping of both. portof is gone entirely, since the library picks the port
and nothing has to parse it back out of the URL.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqzMu5gKg9FeyhfNrRgupd
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix what the review found in the new htsserver test library

The --write-limit path backgrounded a pipeline, so $! was the reader rather
than the server. A start that never announced then killed only the reader,
orphaned the server on its port, and parked the reaping wait on the whole
job, which left the test immune to SIGTERM. Process substitution instead, so
the background pid is the server.

Centralising the launch also moved two python forks and a blocking wait
inside the one-second window 223 needs to mint two session ids in the same
second: under 8x load the window grew from ~0.28s to ~0.70s and the test
began skipping, which reads green while its assertion stops running.
htsserver_start takes --port and htsserver_freeport hands back several ports
bound together, so 223 picks both before the clock starts.

Also: htsserver_assert_reaped now covers the pid the server announces, and
refuses an empty list rather than passing on one; and firstline drops the
$'..' inside a parameter expansion that Apple's bash 3.2 cannot parse.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqzMu5gKg9FeyhfNrRgupd
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 16:55:17 +02:00
Xavier Roche
67c4668a3b A runner-image apt 403 reds a whole job, and the FTP active-mode branch never compiles (#1103)
Two CI gaps, in one PR because both land in `.github/workflows` and would collide.

`apt-get update` runs in 19 jobs under `set -euo pipefail`, so a 403 from `packages.microsoft.com`, a source the runner image ships and we never install from, takes a job down before a single package arrives. It reads like a real failure too: on #1093 the leg that died was named `msan (MemorySanitizer, clang)`. The update is now advisory in the 16 jobs that run on a GitHub runner, and `apt-get install` still fails loudly if a package we do need is missing. The three container jobs keep the strict form, since every source in a `ubuntu:devel` or `debian:sid` image is one we put there.

`FTP_PASV` is 1 in every build, so the `#if !FTP_PASV` active-mode branch is dead at runtime and never reaches a compiler either, which left #1087 re-verifying it by hand. The `build` job now recompiles `htsftp.c` alone with `-DFTP_PASV=0` and fails on any warning, which is why the define grew an `#ifndef` guard. It goes through make, so the object gets the flags the real one gets, and is dropped afterwards so `make check` links a normal library. The control is an undefined `accept` in that object: only the active branch calls it, so a probe that quietly rebuilt passive mode fails instead of passing. Two mutants kill the probe, a stray warning inside the branch and a revert of the `#ifndef`.

Closes #1091
Closes #1097


Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:42:18 +00:00
Xavier Roche
e6015e29bb Bump src/coucal so an in-tree build stops leaving the submodule dirty (#1102)
coucal's .gitignore now covers the libtool and automake droppings our
in-tree build drops into src/coucal (*.lo, *.la, .libs/, .deps/,
.dirstamp), same class as #905. Also picks up a codeql-action pin from
dependabot. No code change, no ABI impact.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:00:06 +00:00
Xavier Roche
ba53bca32e A foreign-host link that looks like a file is scanned as a page (#1093)
A link to `https://www.dropbox.com/s/xxx/distorted.jpg` looks like a file, so the engine accepts it even though it points off-site. Dropbox answers with `text/html`, and the preview page is then parsed with the referring page's whole depth budget. Every link on it is same-host from there, so one link mirrors the entire foreign site: 372 files in the reproduction against the one page it was entitled to, and wikipedia `File:` pages do the same. The new test drives two loopback servers and pins the mirror of the foreign one, which held 3 files under `-n` and 7 under `+*.jpg` before the fix.

At the parse gate such a link's depth now counts as 0, so the page is still fetched and stored, through the store-without-scan path a depth limit already uses, but nothing on it is followed. The gate is `is_html_mime_type()` rather than the wider hypertext set, or a foreign stylesheet would stop being parsed and its images would go missing; the one cost is a typeless response, which that macro reads as HTML and which now picks up a stray `.readme`. Hosts are compared through `jump_identification_const()`, since `adr` carries the scheme and any `user:pw@` and a raw compare made a site foreign to itself. Nothing about filters enters into it: the only exemption is being a seed, which the user asked for by name. Also fixes `hts_testlinksize`, which built its filter URLs with the separator on the wrong buffer.

Closes #121
Closes #1095


Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 12:45:23 +00:00
Xavier Roche
5b59cdb159 Run 230_local-ftp-userpass on Windows again (#1100)
The skip landed on a 3/3 wedge observed before #1039, #1055, #1071 and #1081 were in the tree. With all four merged, the test was re-run three times on current master with the gate and the `expected_skips` entry removed: six green legs, x64 and Win32, every run.

The runs were sequential on purpose. `windows-build`'s concurrency group is per-ref with `cancel-in-progress`, and overlapping Windows jobs are themselves a known way to lose a runner, so three parallel runs would have risked manufacturing the failure under test.

Detail and run IDs are on #1038.

Closes #1038

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:14:17 +02:00
Xavier Roche
c520d81f6b FTP active mode waits for the data connection with no timeout and no stop check (#1087)
Active mode waited for the server's data connection in a blocking accept(), so it ignored --timeout, --max-time and the stop flag, the way the two connects did before #1071. It now polls on the same 100ms tick, re-reading the stop flags and what is left of --max-time each pass, and puts the accepted socket back into blocking mode because BSD hands it the listener's non-blocking flag.

The branch is compiled out today (FTP_PASV is 1), so there is no runtime path to test. I compiled htsftp.c with FTP_PASV forced to 0 instead: clean, no warnings, and the object's undefined accept symbol confirms the branch was really built. The default is untouched. No build in CI covers that branch, which #1091 tracks, so this check was a one-off by hand.

A review found a second bug in the same code, folded in here: the active-mode listener and the data socket were both closed with a raw close() that left the descriptor in the variable, so the _CHECK_HALT_FTP after the transfer loop closed them again through _HALT_FTP. The data socket one is on the live PASV path. Both now reset to INVALID_SOCKET the way the PASV path already does around its own deletesoc calls. 253_local-ftp-close-once.test cancels a link mid-transfer through the engine's own per-link cancel, the one WebHTTrack's stop button pushes, and an LD_PRELOAD close() interposer reds the test on any EBADF. Reverting the data socket to its raw close gives exactly one.

Closes #1074
Closes #1090

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:14:13 +02:00
Xavier Roche
0748fa44e9 StringRoomTotal doubles its capacity without saturating (#1088)
StringRoomTotal doubled its capacity with no cap, so once past SIZE_MAX/2 the product wrapped to a capacity below the one already held and no further doubling climbed back out. Nothing reaches that today, since the realloc sits inside the loop and an outsized request dies in STRING_OOM several doublings earlier, but the tree had one grower that saturates (TypedArrayEnsureRoom, fixed in #1056) and one that did not. Past the halfway point it now asks for the requested total rather than doubling.

Clamping to SIZE_MAX, the obvious mirror of TypedArrayEnsureRoom, hands GCC a constant `realloc(p, SIZE_MAX)` and lights `-Walloc-size-larger-than` at every String growth site: 24 new warnings in proxytrack alone. The requested total is a runtime value, so there is nothing to fold. #1062 also asked for the realloct/freet wrappers; I left those alone, since they are plain aliases for realloc/free, while `freet` is a statement macro that assigns to its argument, htsstrings.h is installed, and pulling in htssafe.h leaks config.h into a header documented as standalone.

Two stringoom modes cover it. `saturate` forces the capacity past the halfway point, which nothing reaches through the API, and pins the size the allocator is asked for; it dies on its first allocation and so never sees a second step. `growloop` runs the whole loop against an allocator that keeps saying yes, under a step budget, so a clamp that fails to raise the capacity fails instead of spinning. Both go red on the old doubling and on a clamp that hands back the current capacity.

One unrelated fix rides along. `tests/250_timeout-poll-nofork.test` checked that the wedged child was gone with a single `kill -0` that nothing had waited for, and that race reddened this PR's own `build (no python3, Debian buildd)` leg. It now waits for the pid, bounded well under the child's own self-limit so a child expiring on its budget cannot pass for a successful kill.

Closes #1062
Closes #1092

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:14:01 +02:00
Xavier Roche
0ced83ccfc The heartbeat's sleep can inherit the caller's stdout, and no test sees it (#1089)
The `>/dev/null 2>&1` on `ci_suite_heartbeat`'s `sleep` is load-bearing, not noise suppression. The suite's EXIT trap kills the heartbeat shell and not the sleep it forked, so an orphan still holding the step's stdout keeps its reader waiting for EOF, which delays or loses the step the watchdog was there to protect.

171 is blind to it: its `sleep` is a shell function on a virtual clock, so nothing is forked and no descriptor is inherited, and the test stays green with the redirection deleted. The new leg goes in 172, which already drives the driver on real sleeps. Start the heartbeat on a 15s tick with its stdout in a pipe, let it enter the first sleep, kill it the way the trap does, then time how long the reader waits for EOF: 0s today, 13s with the redirection removed. Two things keep a prompt EOF from passing for the wrong reason: a control leg runs first and forks a child that does keep the pipe, and the leg samples the heartbeat pid before it signals, since one that died before forking its sleep would look identical.

Costs 172 about 8 seconds.

Closes #949

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:13:57 +02:00
Xavier Roche
171ffb0754 Tests 13 and 52 mint a cert the tree already ships (#1101)
* Tests 13 and 52 mint a cert the tree already ships (#1099)

Both minted a two-day self-signed cert with openssl req on every run, which
is why both carried openssl in their skip gate. make_tls_pem now joins the
tracked tests/server.crt and tests/server.key, the pair local-crawl.sh
already hands every --tls crawl, so the gate and the keygen go away.

Closes #1099

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the fixture against load_cert_chain, not against the helper

The two byte comparisons only restated what make_tls_pem cats: a
server.key that does not match server.crt passed 254 and reddened 13
and 52. Load the pem the way the python servers do, and trim the
comments to one line each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the write in the fixture helper's failure message

A missing destination directory fails the redirect, not the read, and
the old wording blamed the fixture for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 10:42:24 +00:00
Xavier Roche
e12fe63246 A failing cert fixture leaves tests 13 and 52 with an empty log (#1098)
`13_crawl_proxy_https` and `52_local-socks5` both built their self-signed cert with an `openssl req` whose output went to `/dev/null 2>&1`. Under `set -e` a failure there exits 1 before either test's first `echo`. Automake's log then kept only the `FAIL ... (exit status: 1)` line, with no clue what failed. It happened twice on a macOS 26.5 box.

The fixture was identical in both tests, so it moves to `testlib.sh` as `make_tls_pem`, which keeps openssl's output and prints it when the command fails.

`254_testlib-tls-pem.test` checks the helper from both sides. A working run has to produce a pem holding the key and then the certificate, in the order the callers' `load_cert_chain()` reads. An `openssl` shimmed onto PATH that exits 3 has to make the helper exit 1 with that openssl's reason on stderr; the shim still writes the files it was asked for, so nothing but the helper's own exit can satisfy the status check. Each of those assertions was confirmed against a mutant that breaks it.

Leaving `openssl` out of the picture entirely looks better still: `tests/server.crt` and `tests/server.key` are already tracked and distributed, with a SAN and a 2056 expiry. Moving 13 and 52 onto them would drop `openssl` from their skip gates, but it changes where those two tests run, so it belongs in its own change.

Closes #1094


Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 11:48:09 +02:00
Xavier Roche
ff266a0e61 The macOS Release asset name does not say it is arm64-only (#1085)
* The macOS Release asset name does not say it is arm64-only (#1083)

The DMG is arm64-only, but the file attached to the Release is
HTTrack-<version>.dmg, so nothing in the name warns an Intel user that it
will not launch. Everything else already says arm64: the job title, the
upload artifact, and the copy httrack.com serves.

The rename happens at the Release upload rather than at build time because
httrack-com's stage_macos_dmg.sh asserts the artifact basename is exactly
HTTrack-$VER.dmg, so renaming earlier would break staging in that repo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Shorten the rename comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the DMG at build time, not at the Release upload

Xavier's call: the image stays arm64-only and says so in its name. Naming it
where it is created gives the artifact, the Release asset and the logs one
name, instead of making the workflow a second authority over it.

The arch comes from the bundle's own Mach-O list, and counts only if every
one of them carries it, so a thin dylib cannot get the image called universal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Guard the arch intersection against an empty token

Review found that a trailing space in lipo's output would put an empty
token in the set, which every binary then "carries", so a thin arm64
build would be named universal. Apple's lipo emits none, checked with
od -c against a real bundle on an Apple Silicon machine, so this is
insurance rather than a live bug: without the guard that input yields
universal, with it arm64.

lipo joins the tool preflight too, so a missing one says so instead of
reporting that the binaries share no architecture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 07:31:27 +00:00
Xavier Roche
dfc0efff3a WebHTTrack's language menu never marks the language in force (#1086)
* WebHTTrack's language menu never marks the language in force (#1084)

The welcome page opened its menu blank instead of showing the active
language: index.html used ${list:#iso}, and that form carries no default.
The listid: form does, and the launcher already hands htsserver the index
as "lang".

Switching the template alone was not enough. Format 2 opens its first
option before the loop that applies the default, so index 1 (English, the
commonest case) could never be marked. That opening now checks the default
as well, which also covers the nine other listid: menus whose default is 1;
there the first option was already the one the browser displayed, so
nothing changes on screen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the placeholder option and assert the rendered language

Review findings: the empty <option value=0> could itself post lang=0,
which the n > 0 guard ignores, leaving the menu blank again; and the test
asserted the menu alone, so it passed on a menu that merely echoed the
requested index while the body rendered another language.

The placeholder is gone, and test 251 now checks the page's own greeting
and covers the case where no language is set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the test's server PIDs out of a subshell

serve() printed the URL, so callers ran it as url=$(serve ...) and the
pids it recorded died with that subshell. Cleanup then killed nothing and
the four servers outlived the test, which the macOS runner reports as
orphan processes and fails the job on.

It sets ${url} instead. Verified by count: the old shape leaves four
htsserver processes behind, this one leaves none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 07:14:27 +00:00
Xavier Roche
a0622bcd04 Release 3.49.19
Version bump for 3.49.19. Most of what landed since 3.49.18 is FTP: an over-long user name in an `ftp://` URL logged in as a different account, a crawl ignored `--timeout` and `--max-time` and pinned a slot for five minutes, a dead host took two minutes to give up with no way to shorten it, and the engine freed the transfer slots while an FTP thread was still writing to one. Elsewhere, an unbounded panic message could smash the stack, a crawl that gave up mid-loop lost its WARC archive and its change report, and a resume the server refused could still lose the partial file.

`VERSION_INFO` goes 3:11:0 to 3:12:0, revision only: `httrackp` gained a tail field and `htsopt.h` added two `volatile` qualifiers, so no layout moved and nothing went away. Soname stays `libhttrack.so.3`, exports 167 to 168. Nothing under `debian/` landed since 3.49.18, so that changelog entry is the upstream overview plus the two build failures other distributions reported. `Standards-Version` 4.7.4 still matches debian-policy.
2026-08-08 22:12:35 +02:00
Xavier Roche
f5fbd1a82e The suite watchdogs give up the first time a poll tick cannot start a process (#1081)
* The suite watchdogs give up the first time a poll tick cannot start a process

ci_suite_heartbeat read its clock through a command substitution, so the
watchdog needed a fork to tell the time, and a sleep that returns without
waiting either killed it under the caller's errexit or turned
test-timeout.sh's wait loop into a fork storm. Both fail exactly when the
box is short of processes, which is when the watchdog is the only thing
still reporting.

The clock now assigns hb_time instead of printing, the heartbeat's tick is
guarded, and the wait loop gives up after ten ticks that never waited,
naming the test in the progress log the off-box watchdog reads.

References #1038.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: dump the crawl logs before TMPDIR goes, and pin the guards with mutants

The nofork exit skipped dump_crawl_logs, so the EXIT trap deleted the very
logs that would explain the hang. reap_bounded stays out: it polls on the
same broken tick.

250 now asserts the tenth tick by count, that the wedged child was killed,
and that a tick failing every other call is never read as a wedge. 171's
sleep stub returns 126 once, which is what catches the heartbeat guard
being removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:22:21 +02:00
Xavier Roche
496e85a3ac --single-file writes marks it cannot read back (#1080)
* --single-file writes marks it cannot read back

sf_parse_mark refused a <len> above SF_MAX_REF (4096), but htsparse measures
the escaped bytes it appended: HTS_URLMAXSIZE*2 of savename through
escape_for_html_print_full plus as much query through escape_for_html_print.
A mark past the cap parses as nothing and survives into the page as literal
text, taking its reference with it. Derive the cap from the emitter, and have
singlefile_mark decline a span sf_parse_mark would not read back so the two
cannot drift apart. SF_MAX_REF keeps its value for the resolution buffers.

Also swap singlefile_may_mark's hand-rolled search for hts_memstr, clip the
unresolved-reference warning, and make the charset case in test 94 bite: the
server now answers /sfmark.html with a mark in its Content-Type charset.

Closes #1054
Closes #1069

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Check the span derivation at the emit site, and test what the review found vacuous

SINGLEFILE_MAX_SPAN moves to the header so htsparse can assert it against the
tempo and lien it is derived from: a resize there now breaks the build instead
of silently switching inlining off.

Test gaps the review named: the digit-loop saturation guard had no case (a
2^64+10 length wraps to 10 and eats the padding without it), the spans loop
only checked the reference was still present rather than that nothing but the
mark had gone, and nothing asserted the warning clip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:19:59 +02:00
Xavier Roche
d4b77e53dc Ctrl-C does not shorten an in-flight HTTP connect (#1079)
* Ctrl-C does not shorten an in-flight HTTP connect

The user stop only kept new links from being queued: a slot already
waiting for its connection stayed there until --timeout, so stopping a
crawl against an unreachable host took minutes.

back_wait() now ends every slot whose connection is not established yet
(name resolution, connect, TLS handshake) as soon as the stop flag is
set, the way #1071 did for FTP. Slots already receiving are left alone.

Closes #1073

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cancel a stalled name resolution on stop, and harden the sweep's test

The HTTP resolver was still uncancellable: hts_dns_resolve_all() passed no
cancel flag, so a Ctrl-C during a resolve behind a black hole waited out
--timeout per host, the same stall the connect sweep fixes. Pass the
mirror's stop flag, which is what #1071 added the parameter for.

The backstop self-test now covers the states a sweep must not take (FTP,
chunked), runs a control pass with the mirror still running, checks the
socket was closed instead of only cleared, and repeats after re-arming, so
a one-shot or unconditional sweep fails it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:19:06 +02:00
Xavier Roche
507f3565e8 A crawl on a terminal with stdin at end of file toggles its display mode in a loop (#1078)
* Tell an stdin at end of file from a keypress

select() reporting stdin readable does not mean there is anything to read: at
end of file it stays ready forever, and check_stdin() sold that to the crawl
loop as an ENTER. The display then toggled on every pass, so a terminal run
with stdin closed alternated between the spinner and the full panel.

Peek a byte before answering yes, and drop the fflush(stdin) that would have
discarded it (undefined on an input stream anyway). The Windows branch reads
the console queue rather than stdin, so it has no end of file to confuse.

Closes #1072

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Model the ticket's terminal at EOF, not just a redirection

The EOF leg fed httrack /dev/null, so a fix that simply gave up off a
terminal would have passed it while leaving the reported bug alive. Add a
third crawl that types ^D into the pty, which is the case #1072 describes.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:18:11 +02:00
Xavier Roche
4df1434b56 The ProxyTrack Windows skips outlived the reap that made them necessary (#1077)
* tests: the Windows engine reap named only httrack.exe

kill_tree's by-name fallback and reap_leftover_processes each carried their own
hand-written image list beside ENGINE_EXE_RE, and both had drifted: a proxytrack
orphan survived either one. Derive all three from a single ENGINE_EXES.

With the reap covering it, drop the stale #595 skip from the webdav-mime test.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: anchor the Windows image matcher, un-skip webdav-overflow

Review found a fourth hand-written engine list in dump_windows_stacks, and that
the tasklist greps were unanchored: notepad-httrack-notes.exe read as a leaked
engine. Both now go through ENGINE_IMAGE_RE, derived from ENGINE_EXES.

webdav-overflow joins webdav-mime on Windows. webdav-default and
proxytrack-quiet keep their skip, with the real reason: they read proxytrack's
console through a pty, and Windows has neither Python's pty nor os.fork.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:17:24 +02:00
Xavier Roche
d85d91f8c6 Adding a test conflicts with every other test-adding PR (#1075)
* Adding a test conflicts with every other test-adding PR

tests/tests-list.mk was a single shared file every test-adding PR appended
to. merge=union kept that clean locally, but GitHub never reads
.gitattributes, so on the forge each pair of such PRs conflicted and every
merge re-conflicted the rest of the queue.

configure now globs tests/[0-9]*_*.test into TESTS_LIST and tests/Makefile.am
reads it back as TESTS = @TESTS_LIST@, the one form automake passes through
untouched. A make-level $(wildcard) cannot replace it: automake word-splits
TESTS at automake time and emits a broken rule per word.

check-tests-list.sh becomes check-test-names.sh, guarding the naming
convention the glob depends on rather than a list that no longer exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch a build dir whose TESTS predates the test you just added

The glob moved the silent-skip hazard rather than removing it: TESTS_LIST is a
literal frozen into tests/Makefile by config.status, and nothing regenerates it
when a .test appears, so a new test was simply not run and every other test
still passed. 231_test-names.test now compares tests/ against the TESTS the
running Makefile holds, which turns that into a red test on the next make check
and covers the deletion case too.

The glob also dropped the old list's count oracle, so a test its author forgot
to git add left CI running a smaller suite, green. check-test-names.sh now
compares the glob against git ls-files, skipping that when nothing there is
tracked so tarball builds are unaffected.

It also missed a case-mistaken 01_Engine.TEST, and claimed a name with a space
in it was one make check runs, when configure space-joins the basenames and
make sees two words. Both are rejected now, with a control each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A git that cannot answer read as a clean tracked set

The tracked-set oracle swallowed git's exit status, so git missing from PATH or
a safe.directory refusal looked exactly like a tarball with nothing tracked:
over a checkout holding an untracked test the checker printed a clean line and
exited 0. It now probes whether git can answer at all and says on stderr when it
cannot, leaving the genuine empty-listing skip that tarball builds rely on
untouched. 231 stubs git out to prove the note fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 17:05:06 +00:00
Xavier Roche
b16ffc7be2 The --single-file crawl assertions run on Windows but are reported as a skip (#1066)
* The --single-file crawl assertions run on Windows but are reported as a skip

94_local-single-file.test only reaches its exit 77 after every crawl assertion
has passed, so the Windows suite counts real coverage as a skip. Move the
htsserver block to 241_local-single-file-gui.test and drop 94 from
expected_skips.

Refresh two stale justifications while there: 48 and 71 were pinned on the now
closed #581, and 153 had none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert tokens, not substrings, in the single-file GUI and mirror checks

"--single-file" is a substring of "--single-file-max-size=5000", so the flag
assertion passed with the bare flag gone; the value match had no trailing
delimiter, so 50000 passed for 5000. Both mutants now kill the test.

94 carried three fail-open checks that this PR makes load-bearing on Windows: a
grep whose status was discarded, a mark sweep that read an unsearchable tree as
clean, and a --changes crawl writing to the first mirror's -O, which left its
page absent and every check on it vacuous.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep grep's status in the mark sweep, and size the page the change report names

The file-count floor could not see what the sweep saw: it walked its own find
expression, so mirrored files at mode 000 counted 11, passed, and left grep's
status laundered by the pipe it was meant to guard. Keep grep's own exit status
instead, and run tr as a separate step.

The --changes check named an invariant it did not test. The report stats the
file it lists, so comparing its size for the page against the page on disk is
what proves expansion ran first; moving singlefile_process_mirror() after
hts_changes_close_opt() now fails it at 1037 bytes against 28705.

Also derive the crawl options from one array rather than retyping them at four
call sites, and match the max-size flag as a token: the right-hand anchor let a
template fusion through.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Move the change-report probe out of a command substitution for bash 3.2

macOS ships bash 3.2, which ends $(...) at the ")" and runs the here-document
body as shell commands, so the probe failed with "import: command not found".
Write it to a file at statement level and run that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 16:29:59 +00:00
Xavier Roche
defd6735f3 A WARC abort can orphan a .tmp when teardown reopens the writer (#1068)
* A WARC abort can orphan a .tmp when teardown reopens the writer

warc_close_opt() and warc_abort_opt() left opt->state.warc NULL, which the
emit hook reads as "never opened". XH_uninit then finalizes the slots still
in flight, and back_finalize's warc_write_backtransaction lazily reopens the
archive. The reopened writer nobody closes leaves a .tmp behind, since the
archive it would replace is already in place.

Both now leave the existing WARC_DISABLED sentinel instead, which the hook
already honours: no writer, and none to open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make test 242 tell the close case from the abort case

The size compare after teardown could not fire: a late reopen always writes
the temporary, never the archive, so the two sizes matched on every path. And
nothing pinned which run the archive held, so a close that stopped committing
its own run passed both assertions.

Each run now carries its own body, and the archive is read back: valid
framing, the run that had to survive, and neither of the other two. Abort
keeps run 1, close commits run 2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Say what the framing check covers, and convert warc_slurp's path

The WARC/1.1 compare reads the first record's version line, so it catches a
file that is no longer an archive, not a corrupt one; the late-body leg is an
append guard rather than a discriminator, since nothing can reach the archive
without losing the run that was there. Both said so in one line.

warc_slurp() also opened its path raw where every other selftest path goes
through fconv(), and the test now traps the signals on their own line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Note that copy_htsopt must not copy the embedded state

The rule was written only for sitemap_state and singlefile_state; a copy of
state would clear a WARC writer the live opt had already finished with.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 15:52:40 +00:00
Xavier Roche
178f9156e5 An FTP crawl of a dead host takes two minutes to exit, and Ctrl-C cannot shorten it (#1071)
* An FTP crawl of a dead host takes two minutes to exit

The worker sat in a blocking connect(), so --timeout, --max-time and the
engine's stop flag were all bypassed and the kernel's own connect timeout
was the only bound. It now connects non-blocking and polls on the same
100ms tick wait_socket_receive uses, re-reading the stop flag and what is
left of --max-time each pass; the resolve goes through a new bounded entry
point taking an explicit deadline and a cancel flag.

The PASV data path also gated its connect on server_size > 0, which is
sizeof(SOCaddr) and so always true, rather than on the resolve result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Make the new tests able to tell the fix from an instant failure

The timing assertions had no lower bound, so ftp_connect's two
instant-failure returns passed both legs, and the log they matched is
wording the HTTP backend shares. Each leg now floors its elapsed time,
the interrupt leg proves the crawl was still parked before it signals,
and both anchor the FTP form of the message.

Split so Windows runs the half that needs no signal, poll a debug line
rather than sleeping toward the connect, and cover the cancellable
resolve through the DNS selftest's scripted black-hole backend, which
192.0.2.1 (a literal, resolved instantly) never reaches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Restore an unrelated test's file mode

A chmod glob for the two new tests caught 24_local-resume-overlap.test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Move the FTP timeout test off 244, which #1070 already claims

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Make the two stop flags volatile

The new 100ms poll loops re-read stop_ftp and state.stop with no lock, so
their correctness rests on the compiler not hoisting the load out of the
loop. select() and Sleep() are opaque enough today, but that is the shape
of the calls rather than a guarantee, and the code these loops replaced
sat in a blocking syscall instead.

hts_dns_resolve_all_bounded takes the flag by pointer, so its cancel
parameter becomes const volatile: passing &back->stop_ftp would otherwise
discard the qualifier.

Not an ABI break, though htsopt.h is installed: volatile changes no
offset, size or alignment, only the declared types.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 14:44:04 +00:00
Xavier Roche
ee5db0a9e6 -%v1 overwrites its own output and -%v2 paints escapes into a redirected log (#1070)
Salvages what stands on its own from #362; the progress bar itself is not here.

In `-%v1` each completed file printed a line ending with a CR and a clear-to-EOL, so every line overwrote the one before it. A finished crawl left the terminal holding a single line and no record of what had been fetched, and redirecting that output gave one unreadable line of escapes. Those lines end with a newline now, and the clear-to-EOL goes with them. That part is Matteo Fiorini's, credited in the trailer.

The rest is a defect #362 shared rather than fixed. An explicit `-%v2` kept painting cursor-addressed panels whatever stdout was, because the `isatty` probe in `htscoremain.c` only picks the default verbosity. That probe is also compiled out on Windows, where the spinner wrote its CRs straight into a redirected file. Both are gated on a terminal now, each where it is drawn rather than by rewriting the user's option. The CLI arms its panel in `htsshow_start()`, which the mid-crawl ENTER toggle re-enters, so the gate holds for the whole crawl. Off a terminal `-%v2` paints nothing; `-%v1` is the mode that gives a redirectable log.

`241_local-tty-output.test` crawls the local server three ways into a pipe. It requires that no escape and no CR survive, that each crawl actually ran, and that `-%v1` keeps one line per file. The controls repeat the crawls with stdout on a raw pty and require a repainting panel and a turning spinner, so a fix that merely silenced the display would fail instead of pass. Reverting either completion line, the panel gate or the terminal probe breaks the test, and so does deleting the display outright. One gap: the spinner gate cannot fail on POSIX, where `quiet` is already forced off a tty, so only the Windows leg exercises it.
2026-08-08 16:06:55 +02:00
Xavier Roche
edeada647c A crawl that gives up mid-loop loses its WARC archive and its change report (#1064)
`httpmirror()` has four exits that never reach the teardown at the end of the function: the three crawl-loop bailouts and the no-data rollback. None of them runs `warc_close_opt`, `hts_changes_close_opt` or `hts_sitemap_free`, so any crawl the engine gives up on loses all three. Tripping the `-#L` link limit is the easiest way to get there; a shell or GUI abort does it too.

The WARC is what this costs in practice. Segments are written under `.tmp` names and renamed only once the writer closes, so when a previous archive exists, a bailout strands the whole capture in a `warc-out.warc.gz.tmp` nothing will ever pick up. The previous archive stays in place as if the run had never happened, with a stale `.cdx` beside it.

All four exits now jump to one label, with `retcode` keeping the bailouts' -1 apart from the tail's 1. The rollback still calls `warc_abort_opt` first, which NULLs the writer and makes the shared `warc_close_opt` a no-op there. `hts_cache_reconcile` still runs after `XH_uninit`, which is what closes the cache files it renames.

The change report goes the other way. An abort never reaches `hts_changes_indexed()`, so any report it writes claims the cache is off and no deletions were found, and on the rollback path that false report replaces a true one describing the mirror the rollback just restored. So only a mirror that reached the end of the loop writes `hts-changes.json`. Every other exit frees the accumulator, which is the leak the issue actually named.

Test 240 poisons the archive and snapshots the report, aborts on the link limit, then re-runs against a dead server. Three separate mutants each kill a different assertion. It covers the `-#L` bailout only: the other live one needs the loop callback to refuse, which the CLI cannot drive. The third, a failing `hts_mirror_check_moved`, is unreachable today since every return in that function is 0, and it is routed with the others rather than left as the odd one out.

Also frees `primary` on the exits that leaked it. #1060 is unaffected: its window, a writer lazily reopened during `back_delete_all`, is unchanged.

Closes #1061
2026-08-08 09:27:46 +00:00
Xavier Roche
562f6790ec Two of the Android help screens have no anchor to deep-link at (#1065)
* Two of the Android help screens have no anchor to deep-link at

The Android app is getting per-screen contextual help (httrack-android#113)
and needs somewhere to link. Six of the eight screens already have a target,
since the wizard steps and the option panel kept their ids when the eighteen
GUI pages were folded into guide.html. The first-launch permission note and
the paragraph saying where mirrors land did not, so add an id to each.

Those ids are link API now: the app hardcodes them and an installed release
outlives whatever engine SHA it was built against. The new test resolves
every guide.html# reference in html/ against the file's actual ids, which
also covers the redirect stubs left by the page merge, and names the eight
the Android side depends on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* BSD sed has no \| alternation, so macOS never strips the platform prefix

The scan left "droid/step-run" unstripped on any POSIX BRE sed, and no id
matches that, so the test failed everywhere but GNU.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the reinvented link scan; tools/doc-links.py already resolves them

The generic half duplicated tools/doc-links.py, which the documentation CI job
already runs over the whole doc set. What is left is the part it cannot know:
the eight anchors httrack-android hardcodes, each present once and on an
element the droid platform can see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 07:56:13 +00:00
Xavier Roche
cbb2257847 A rejected resume can still lose the file (#1057)
* A rejected resume can still lose the file

With --retries=0 an unusable Content-Range deleted the partial and never
re-fetched it: the restart-whole recovery was written as an error retry, so it
was both gated on the retry budget and charged to it. Restarting a transfer the
server refused to resume is not a retry of a failed transfer, so it no longer
spends the budget; refetch_whole, which only a previous restart sets, latches it
to one so an always-unusable server cannot loop.

#581 was already fixed by #655, but nothing held that fix in place. Tests 48 and
71 both pass with it disabled and neither reaches the restart path, so test 235
drives it with an LD_PRELOAD interposer that refuses the unlink the way Windows
refuses one on a file it still holds open.

The unlink in the memory-resume branch also ran while a FILE* on that file was
still open, which is what left the partial behind on Windows; it now closes
first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Register 235_local-resume-recovery in the Windows expected-skip set

The Windows suite pins its skip set, so a new test that skips there reds the
job at fail=0. Also drops the stale #581 note on 71, whose skip is now the
same MSYS signal limitation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Carry the whole-file restart latch across a link re-record

The free restart introduced by this branch is bounded only by
lien_url::refetch_whole, but the two sites that re-record a link while
preserving its retry count -- the case-alias redirect replay and the
412/416 re-get -- never copied that latch, and hts_record_link zeroes the
new entry. A server alternating an unusable 206 with a 301 to a
case-different alias of the same path therefore restarts for free at
every hop: 148 restarts and no retries consumed in 60s at both
--retries=0 and --retries=1, where the merge base terminates in one
second.

Copy the latch wherever retry is copied, and say so at the field's
declaration. Test 235 gains a leg driving that alternation; it also
asserts a later genuine failure still had its retry, which the previous
three legs could not see.

The unlink interposer now tells an open handle from a closed one through
/proc/self/fd instead of refusing by basename, so reverting the fclose in
the memory-resume branch turns the last leg red rather than leaving it
green.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:36:01 +02:00
Xavier Roche
0e95c58865 Three unreachable defects from #1040, and a live stack overflow found beside one (#1056)
* Fix three unreachable defects in the array and bounds-check macros

TypedArrayEnsureRoom doubled the capacity until the requested room fit, so a
large enough request wrapped it to zero and the loop never ended; capa * width
could wrap too, into an allocation smaller than the capacity claimed. Growth
now saturates and asserts, and the macro uses the realloct/freet wrappers.

HTS_IS_CHAR_BUFFER cannot be made sound on MSVC, so its comment now names both
sizes that go blind (char[8] on Win64, char[4] on Win32), and the two
pointer-sized destinations in the tree are widened. HTS_CDLMAXSIZE keeps its
value; the *2 comment beside it went stale when 3.30.1 raised HTS_URLMAXSIZE.

Also fixes a live stack overflow found while auditing HTS_CDLMAXSIZE:
htscoremain.c formatted "Missing quote in %s" with a raw sprintf into a bare
HTS_CDLMAXSIZE buffer, overrunning it by 17 bytes on a max-length argument.

Closes #1040

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the array overflow self-test's abort out of the optimizer's reach

Test 237's SIZE_MAX-room mode relied on realloc returning NULL, which nothing
observed: the array is dead after the call, so clang (and gcc from sid) drop
the allocation, assume it succeeded, and fold away the "== NULL" that aborts.
Printing the pointer keeps the allocation live. ASan meanwhile kills the
process itself on an over-large request rather than returning NULL, so that
one run asks for libc's behaviour instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the remaining raw sprintf panic messages in htscoremain.c

The "Unable to create log file %s" pair formats the whole -O path into a
char[HTS_CDLMAXSIZE + 256]. That path is not argv-gated: check_path() runs
expand_home() on it, which admits 2047 bytes, so
  HOME=<2000 chars> httrack -q -O '~/x' <url>
smashes ~800 bytes of stack (fortify aborts at htscoremain.c:2583).

The second "Missing quote in %s" site copies argv[na] into a +256 buffer
first, so its 17-byte prefix can overrun the same way even though the
argv gate keeps today's callers short of it.

Also fixes the HTS_CDLMAXSIZE comment (the gate rejects >= 1024, so 1023
is the longest accepted) and trims the comments added by this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:03:30 +02:00
Xavier Roche
8184b45696 Read the .arc bytes with od instead of piping into head (#1063)
`tail -c +N file | head -c 3` leaves tail writing into a pipe head has already
closed. Under this test's `set -o pipefail` that SIGPIPE becomes the script's
exit status, so 154 dies at 141 having printed nothing. GNU coreutils happens
to survive it, which is why the red only shows on the uutils leg. od seeks to
the offset itself and reads no more than it needs, so no second process is
left holding the other end.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:02:40 +02:00
Xavier Roche
62ca6e087a A repeated crawl can write a byte-identical .cdx, so the rerun test reds (#1046)
* A repeated crawl can write a byte-identical .cdx, so the rerun test reds

--archive-replaced-on-rerun compared bytes to decide the second pass had
rewritten each archive file. WARC segments carry a fresh record UUID, but the
.cdx holds no per-run entropy: only a second-granularity timestamp and
length/offset fields that move with the gzip member sizes. Two identical passes
can produce the same bytes, about 0.5% of runs, and the test read that as a
stale file.

Assert the replacement positively instead, reusing the poison the .wacz check
already used: every snapshotted file is overwritten with a marker between the
passes, and none of it may survive. Also make warc_cdx_flush() say something
when it leaves the previous pass's index beside a freshly swapped archive,
rather than returning silently on an open failure or an empty index.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the WARC writer on the rollback path, and stop warning when nothing was replaced

warc_open's own failure paths call warc_close with cdx_on already set, and
warc_commit returns early when there is no previous archive to protect, so a
failed archive open reached the new zero-count warning and logged a second,
false error line. Gate it on the writer having opened, and on the run not
having failed.

The "no data seems to have been transferred" branch in httrack_main returns
before the teardown that closes the writer, which is the one path where a stale
index can outlive its archive and the warning can never fire. Close it there
too. Closing it outright would be wrong: the temporary holds a warcinfo record,
so warc_commit would swap an almost empty archive over the previous one and
destroy it, measured as 2431 bytes and 2 index lines down to 1509 and 1. The
new entry point marks the run as one that must replace nothing, which drops the
temporaries and keeps the previous archive, while still writing the index for
an archive written in place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the --archive-replaced-on-rerun header

Cause before effect, and the entry fits in three lines again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Stop reading a straggler tick as a live reporter in 172

The stub reporter advances its marker with a forked mv, and that mv outlives
the kill -9 of its parent, so the marker can move once more after the wedge
branch has already ended the reporter. still_ticking calls it alive on the
first differing sample, which reds the suite (#1058). The two negative call
sites now wait for the marker to hold still instead, which a straggler only
rebaselines; the positive one keeps the old predicate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber the WARC rollback test to 238

232 was taken by 232_online-gate-outoftree.test, which landed on master
in #1047 while this branch was open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Gate the empty-index warning on the archive this run actually touched

warc_cdx_flush() warned about a stale .cdx whenever a writer opened cleanly
and indexed nothing, but warc_commit() returns early when there is no previous
archive to protect, so a first run that saves nothing hit the warning with no
index to invalidate. Gate on protect_prev, which reaching the flush proves a
swap happened, and add the mirror case the old gate stayed silent on: a run
written in place and then abandoned, whose archive is clobbered while an index
beside it still describes what used to be there.

Report a write error ahead of fclose's echo of it, too: glibc drops the
unwritten buffer on a failed flush, so the "index is incomplete" arm only ever
fired when the total happened to land on a buffer boundary.

A warc-cdx-errors self-test drives all five outcomes, over /dev/full for the
two write failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Warn about a stale index only when one is on disk

The empty-index gate keyed on protect_prev, which stands in for "this run
replaced the archive", but not for the index the message names: a run written
in place and abandoned, or one swapped over a previous archive, warned that
<base>.cdx "was not rewritten" whether or not that file had ever existed. It
now asks the disk. The self-test's two positive cases left no index behind, so
they passed on a message about a file that was not there; give them one, and
add the two negatives that pin the terms the old cases could not tell apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 21:19:39 +00:00
Xavier Roche
7c6e5c3018 An FTP crawl ignores --timeout and --max-time, pinning a slot for five minutes (#1050)
* An FTP crawl ignores --timeout and --max-time, pinning a slot for five minutes

run_launch_ftp hardcoded a 300 second read timeout and never looked at
opt->timeout, so every control- and data-channel read on an FTP slot
inherited it: a server that accepted the connection and then went quiet
held the slot for five minutes. The timeout now comes from opt->timeout,
which defaults to 120 rather than 300.

--max-time could not end an in-flight FTP transfer either. That slot is
owned by the FTP thread, which back_wait cannot abort without racing it,
so each wait inside the thread is instead clipped to what is left of the
mirror deadline and the thread gives up on its own.

Closes #1039

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the FTP data-channel wait and --timeout=0 in the tests

The two tests passed against a tree that hardcoded the body-transfer wait
back to 300, because one never reached RETR and the other read the outer
timeout rather than the value the wait used. Test 233 now runs a second,
answering server alongside the mute one: a stalled body transfer under
--timeout=5 with no --max-time, so only the data-channel wait can hold
it, and a trickled one under --timeout=0, which must complete rather than
expire at once.

Also trims the comments the review flagged, and documents what the two
FTP helpers do rather than only how opt is used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the premature 3.49-19 release-notes block

history.txt carries a version block only when a release is cut, curated
from the tag diff; twelve releases show no feature PR opening one. Two
open PRs had each created 3.49-19 at the same anchor, and the file has no
merge=union, so the second to land would have conflicted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 21:05:07 +00:00
Xavier Roche
1878595043 The engine frees the transfer slots while an FTP thread is still writing to one (#1055)
* Wait for FTP threads before freeing the slots and options they still read

An FTP transfer runs in a detached thread holding a pointer into sback->lnk,
and nothing waited for it: back_delete_all() wiped the slot and back_free()
released the array while the thread was still writing through them. A mirror
limit makes it trivial to hit, since the engine stops the moment the cap trips.
Against a trickling server, "-M100000" segfaulted in run_launch_ftp writing
back->r.size.

httrack.c drained threads before hts_free_opt(), but hts_main() and htsweb.c
did not, so opt had the same exposure.

htsftp.c now keeps a mutex-guarded list of live workers, registered before the
thread is spawned and released only after the worker's last store to the slot.
ftp_stop_workers() cancels them and waits, and runs at both lifetime
boundaries: back_delete_all() for the slot array, hts_free_opt() for opt.

The wait has to be bounded rather than merely patient, so wait_socket_receive()
and get_ftp_line() now take the slot and leave their poll loop as soon as
stop_ftp is raised. Previously the flag was only read between operations, so a
silent server held teardown for the whole read timeout. A failed thread spawn is
also handled now; it used to leave a slot nobody would ever reap.

Test 236 crawls a trickling server under a size cap, then a stalled one, and
requires teardown well inside the read timeout. Both fail on master, the first
on the segfault and the second by taking 300 seconds.

Closes #1051

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Budget the trickle crawl's wall clock, and trim the review comments

The trickle half of test 236 asserted only that the crawl survived, which a
drain that waits for the transfer instead of cancelling it also satisfies: it
exits 0 with the whole file fetched. Measured at 103s against 3s for a real
cancel, so the pass now carries a 30s budget too. Also records why the crash is
reproducible at all, since it rests on the backlog array being large enough for
free() to unmap.

Comment cleanup: one issue citation instead of five, a one-fact contract for
ftp_stop_workers, and a one-line test header. The registration-timing note keeps
the invariant and drops the issue number. The history entry said --max-size
could not stop an FTP download, which reads as the cap never firing; it fired,
then crashed or took the full read timeout to tear down.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* tests: fail 236 on a sanitizer report, not only on a signal

Both crawl cases only asserted "not killed by a signal", which a sanitized
build never satisfies: ASan catches the #1051 use-after-free and exits 1
unless the job also sets abort_on_error, so the regression test passed on a
buggy binary. Capture the crawl's stderr and reject any ASan/UBSan/MSan
report; leaks at exit stay out of scope. Vacuous on a normal build, where the
signal check still fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the premature 3.49-19 release-notes block

history.txt carries a version block only when a release is cut, curated
from the tag diff; twelve releases show no feature PR opening one. Two
open PRs had each created 3.49-19 at the same anchor, and the file has no
merge=union, so the second to land would have conflicted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 22:06:53 +02:00
Xavier Roche
057648e335 --single-file maintains a second HTML parser instead of reusing htsparse (#1049)
`--single-file` shipped with its own HTML and CSS scanner in `htssinglefile.c`, about 1140 lines doing what `htsparse.c` already does. This replaces it. htsparse now appends an unguessable per-run mark to each reference it saves that the pass may inline, and the end-of-mirror pass becomes a substitution over those marks: no HTML and no CSS is parsed there at all. The scanner goes, and with it the private attribute table that had already drifted from `hts_detect[]` twice. `htssinglefile.c` goes from 1155 to 863 lines; `htsparse.c` gains 31.

The mark is `#!<16-hex secret>.<class>.<len>`. It is a fragment, so a mirror an interrupted run left marked still browses. The secret is CSPRNG bits drawn once per run and never written to disk, which is what makes the mark unforgeable: a site cannot spell one, so no sanitiser has to keep hostile bytes away from it. `<len>` is the length of the reference preceding the mark, so the pass never has to guess where a reference begins. `<class>` is the context htsparse saw, checked against the resolved type so a mismatch fails loudly instead of inlining whatever the walk found.

Behaviour worth knowing about. A fragment comes back byte-identical rather than re-escaped, since the mark covers the reference alone and what follows it is document text nothing touches; the query moved inside the marked span, because glued to base64 it corrupts the payload. Expanding a mark needs the secret, but recognising one well enough to delete it does not, so the end-of-mirror sweep strips by shape. That clears what an interrupted earlier run abandoned, and costs a page any mark-shaped text of its own. A `<link>` carrying no `rel` is no longer inlined, which is what the old scanner did.

`htsserver` links libhttrack but cannot see a hidden symbol inside it, so the CSPRNG that #1011 added privately to `htsweb.c` and this branch added to `htstools.c` now share `htsrandom.c`.

Closes #749
2026-08-07 18:25:28 +00:00
Xavier Roche
7a7206d7f1 Update Turkish translation from issue #306 contribution (#1048)
* Update Turkish translation from issue #306 contribution

Value-merges the epiusu/httrack fork's Turkish.txt into our own: our
key set and order are kept as-is, but each key's value is replaced
with the fork's translation where the fork has that key. 227 of our
508 values changed; the fork's 44 missing keys (translated against an
older English.txt) keep their existing value, and the fork's one key
absent from our English.txt is dropped.

Co-Authored-By: Ümit Solmaz <usncb@yandex.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch a value left as raw UTF-8 under a declared single-byte charset

A single-byte charset maps all 256 byte values, so the existing iconv
decode can never fail on one and cannot see a translation pasted in as
UTF-8 without transcoding. Only the C1-control heuristic stood in the
way, and a UTF-8 sequence whose bytes are all >= 0xA0 decodes to
ordinary printable characters under Latin-1/-9, sailing straight past
it.

Scan the raw bytes instead: a whole line that is itself valid
multi-byte UTF-8 under a file declaring a single-byte charset is the
leak. Scoped to those charsets because the premise fails for BIG5,
gb2312 and shift-jis, whose genuine text matches the same shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Two Polish translations were stored as UTF-8 in a windows-1250 file

The Keep-Alive option label and its tooltip carry raw UTF-8 bytes in a
file that declares windows-1250, so the GUI and htsserver render them
as mojibake ("połączenia"). Transcode just those two values; the rest
of the file was always windows-1250.

Found by the raw-UTF-8 check added in the previous commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The lang charset check uses grep -P, which macOS does not have

The raw-UTF-8 scan reached for a PCRE lookahead, so BSD grep exits with
"invalid option -- P" and the loop that consumes its output sees nothing.
The per-file scan then passes everything, and the negative control fires
instead: "the charset check passed a deliberately mislabelled file".

Express the same grammar as a POSIX ERE over raw bytes and let awk apply
it, which also folds the per-line length probe into the one pass. Checked
against the PCRE on all 30 lang files plus both probes, under gawk and
mawk: identical line sets.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Ümit Solmaz <usncb@yandex.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 17:22:29 +00:00
Xavier Roche
e4df3b387c Out-of-tree builds skip the online crawl tests with a misleading reason (#1047)
* tests: resolve online-gate helper scripts against the script's own directory

The seven online crawl tests invoked check-network.sh and crawl-test.sh by bare
relative name, which the shell resolves against the caller's cwd. make check
runs from the build tree's tests/, so an out-of-tree build finds no copy: the
gate failed to open, the || branch reported "skipping online unit tests", and
the tests skipped whatever --enable-online-unit-tests asked for.

check-network.sh had the same defect in its own auto-probe call to
crawl-test.sh. Resolve all of them against $(dirname "$0"), the idiom
local-crawl.sh already uses. 232 pins the skip reason from an unrelated cwd and
sweeps the tree for the same form. The option's help string advertised
default=yes; the code has always defaulted to no.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* tests: keep 232's comment clear of 226's credential-scrub gate

226 flags any .test naming the Windows suite driver without the scrub line,
comment or not, so the exemption note tripped it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: tighten 232's sweep, which is the only check below the gate

With the flag off check-network.sh exits before a test reaches its own
crawl-test.sh line, so the runtime probe cannot see that class at all and the
static sweep carries it alone. Four spellings of #1016 were getting through: a
leading ./, a quoted bare name, eval, and a variable holding the bare name.

Quoting and ./ change nothing about what the shell opens, so the shape
exemption is gone and the two call sites that mean the cwd are named instead. A
variable operand is resolved back to its assignment, and eval sites are pinned
as a set rather than passed over. Each form is planted as a fixture, with
anchored and commented fixtures for the other direction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: 172 never checked that it ran its own staged suite copy

Pointing it at the srcdir copy passed, dropping the reap_leftover_processes
neuter it appends to the staged testlib.sh and leaving the test to reap sibling
engines under "make check -j", a flake whose symptom points nowhere near its
cause. test-timeout.sh sources testlib.sh from the staging directory too, so a
plain marker file would still be written under the mutant; record the sourcing
script instead and require the driver.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 16:38:48 +00:00
Xavier Roche
58ddc5a6b1 An --enable-fuzzers build cannot link proxytrack (#1045)
* Link the sancov runtime into every program under --enable-fuzzers

--enable-fuzzers appends -fsanitize=fuzzer-no-link to DEFAULT_CFLAGS, so
every object in the tree carries sancov instrumentation, but only fuzz/
linked a runtime defining those hooks. src/proxytrack, the first program
linked, failed on __sancov_lowest_stack and __sanitizer_cov_trace_*.

Append the flag to DEFAULT_LDFLAGS in the same branch, and add a CI leg
that builds the bare configuration: the existing fuzz job links ASan,
whose runtime happens to define the same hooks, so it could not see this.

Closes #1030

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the two new comments

Both opened with setup instead of the point, and the configure.ac one
restated its neighbour two lines above.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 17:39:46 +02:00
Xavier Roche
fb01ad6f65 A resurrected test registration stops make check with no test results (#1044)
* The union merge driver on tests-list.mk resurrects deleted registrations

merge=union on tests/tests-list.mk only applies to a local merge: GitHub
does not run the driver when it computes mergeability, so both appending
PRs go CONFLICTING anyway, and a conflicted PR runs no checks. Where it
does apply it never removes a line, so a rebase past a deletion restores
the old registration, and make check then exits 2 with no test results.

Drop the attribute and check the pairing instead, in the lint job (a
stale registration kills make check before it can report) and in the
suite (an unregistered file is simply never run).

Closes #1037

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the union, and make the registration it resurrects loud

Review of the first commit: the attribute is right on 75 of the 76 test
add/remove events since it landed, and dropping it would not have helped
the conflicted-PR half of #1037, since GitHub never ran the driver. So
keep it, pin the file to LF, and let the checker cover its one failure
mode.

Give 231 four broken fixtures: every branch of the checker is silent on
a healthy tree, so a checker that had stopped comparing passed the old
assertion just as well. Pin LC_ALL=C, since BSD comm need not share the
collation GNU sort used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Count a directory or dangling symlink named NN_x.test as absent

Set equality held for both, so a registration pointing at one passed the
checker while make check would still choke on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 15:57:09 +02:00
Xavier Roche
1a211555d0 3.49.18 fails to build on Ubuntu: two tests assume GNU coreutils (#1043)
* Two tests break under uutils, the coreutils Ubuntu 25.10 made default

105 built its fake engine by symlinking sleep to a file named httrack,
and uutils' multi-call binary refuses an argv[0] it does not own, so
both fakes died before the first assertion and the oracle read an empty
pgid. 153 put env(1) in front of a backgrounded proxytrack; uutils' env
forks instead of exec'ing, so stop_server killed only the env and
proxytrack kept the pty open, leaving the drainer waiting for an EOF
that never came.

The fake is now a copy of the shell, which answers to any name, and 105
checks that it started: one that never ran showed up only as an empty
field in a later assertion, which reads as a bug in the code under test
rather than in the fixture.

The new CI job builds and runs the suite on ubuntu:devel, which is where
the Ubuntu buildds are and no runner here was.

Closes #1042

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the fake engine, and check the argv[0] dispatch rather than the banner

Review found the fifo fake blocks in open() forever, where the sleep it
replaced self-healed in 60s: a test killed outright (kill_tree sends
SIGKILL, and the outsider sits in its own process group) would leave a
process named httrack on the runner for good. Opening read-write lets
the read time out instead.

The uutils check now renames sleep and requires it to be refused, which
is the property #1042 broke on. The banner works today but has already
been reworded once: 0.0.30 prints "sleep 0.0.30" and 0.8.0 "sleep
(uutils coreutils) 0.8.0". Configure also asserts the codecs and TLS the
way the sibling job does, so a renamed dev package cannot turn the tests
that cover them into skips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Re-trigger CI

The pull_request run wedged: all 23 jobs finished or were cancelled while the
run itself stayed 'queued', so the required checks never reported and neither
rerun nor cancel was accepted (CI run 31124085297, windows-build 31124081217).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Re-trigger CI after the Actions outage

The 2026-08-06 Actions incident throttled webhooks, so the previous push never created a workflow run on this head.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 00:37:43 +00:00
Xavier Roche
f2abfc36ef Nothing survives a wedged Windows runner to say which test wedged it (#1033)
* Give the Windows suite a watchdog that does not need to fork

Every recorded #795 event dangles in the suite step, and the bash heartbeat
added for it has never once fired. Each tick costs a fork on sleep, tail, awk
and finally taskkill, while the leading explanation for the wedge is a box on
which process creation has stopped, so the watchdog is disarmed by the very
condition it exists to catch.

Add a native one: a single long-lived PowerShell process that spawns nothing,
reads the progress log, polls memory, handles, process count and free disk
in-process, and posts a commit status every 30 seconds. That status is the only
thing that leaves the VM before it dies, so the last one to land timestamps the
death and names the test in flight. On a static log it walks Win32_Process
downward from the driver's pid and kills that subtree, never upward, since
killing Runner.Worker.exe reproduces this signature exactly.

The bash heartbeat stays as the fallback for a box with no PowerShell; only one
of the two ever runs, or the loser of the race overwrites the winner's verdict.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 224 ran the real watchdog because TMPDIR splits the PATH entry

The Windows runner hands the suite a TMPDIR of D:/a/_temp, so the stub bin
directory the test prepends to PATH is cut at the drive letter's colon and
never searched. The launch then found the runner's own pwsh and drove the
real watchdog with the real status token, and the stub's argv file the
assertions read never appeared.

Map the temp dir back to a POSIX path where cygpath exists, and assert the
stub is what pwsh now resolves to, so a shadow that fails again says so
instead of reporting a launch that did happen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Report the Windows suite step off-box while the runner is still alive

Every recorded #795 event dangles in the suite step and leaves nothing to read:
the log 404s, the annotations come back empty, and the if:always() uploads never
start. The bash heartbeat cannot help, because it reports through the same
channels the dying runner takes with it.

Add a PowerShell process that only observes. It spawns nothing, reads the
progress log, polls memory, handles, process count and free disk in-process, and
posts a commit status every 30 seconds. That status is the only thing that
leaves the VM before it dies, so the last one to land timestamps the death and
names the test in flight. The driver posts its own verdict as it exits, so a
status left pending means the runner really did go.

The bash heartbeat is untouched and still owns the kill. This one has no verdict
to race it over.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the stub shadows PowerShell before driving the launch

An unshadowed pwsh runs the real watchdog, and the argv assertions that follow
then read as a plain launch failure rather than as what happened.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The status backoff must throttle the API, not the artifact

A fork PR's token cannot post, and the artifact log was the only record left of
those runs. Log every status line whatever the backoff decides, and let it skip
the request alone.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the launch arguments in the form a Windows interpreter can open

The driver hands PowerShell the native path, so comparing against the MSYS
one failed both Windows legs. nativepath() is the identity elsewhere, which
is why the Linux run never saw it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Post a verdict a killed step cannot forge, and keep the token out of the tests

bash reaches its EXIT trap with $? = 0 after a signal, so a cancelled or
timed-out suite posted `success`. The verdict now comes from explicit INT, TERM
and HUP handlers, and every exit path that can still run code posts it: the
heartbeat's kill path posts before it kills, since a taskkill runs no trap, and
the final post is retried so one transient 5xx cannot leave a finished suite
reading `pending`.

The step's token is taken out of the environment and handed to the watchdog
process alone, so none of the ~90 test scripts the driver runs can post a
status with it, and the checkout stops persisting it into .git/config.

A failing stdout write no longer kills the loop: the catch handler's own log
write was outside any try, so under ErrorActionPreference=Stop a full disk, the
state a sick runner reaches, ended the telemetry in silence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Grade the telemetry off the API it calls, not off the log it prints

Printing and posting are decoupled, so assertions that counted printed lines
could see neither the request rate nor the backoff: hard-coding the staticness,
posting on every poll and never skipping a post all passed the suite. The legs
now run the real posting path against a local sink that records each request,
and each is required to trip a mutant of its own.

172 copied everything but the .ps1, so the launch helper returned early and the
real driver launching a real interpreter was covered by nothing. It now drives
that path, and proves the token reaches the watchdog and no test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber to 226: master took 225, and the union driver put 224 back

tests-list.mk carries merge=union, which never removes a line, so the rebase
resurrected the registration of a test file that no longer exists. The tell is
"make check" exiting 2 with no FAIL lines and no "# TOTAL:" summary at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the fork-failure claim from the watchdog's header

The 153,280-fork count behind it was an artefact of /proc/stat counting threads
and unrelated system-wide activity: the suite creates 26,158 processes, and the
engine and zlib blocks make 39.2% of them without holding one of the 34 deaths.
Spawning nothing is still worth having, as insurance rather than as a theory.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Let the new watchdog test survive the macOS and Windows legs

Two platform assumptions the Linux leg cannot see. The signal teardown
kills $BASHPID, which macOS's bash 3.2 does not define, so the kill was a
no-op and the suite exited 0 and posted "success" instead of "failure".
And /dev/full only stops writes for a process of the shell's own runtime:
MSYS cannot hand its emulated device to a native PowerShell, which then
writes as if to NUL, so the log-write-fatal mutant survived a leg that
had nothing to grade it with. The old guard also returned 1 under errexit
on a box with no /dev/full at all, macOS included.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cut the watchdog's verdict down to telemetry

GitHub already records the job's conclusion, so the commit status does
not need to carry a second one, and every attempt to make it do so
produced a way for it to lie. Gone: the one-shot -Post mode and its
retry, the driver's INT/TERM/HUP handlers, and the post-before-kill
ordering on the heartbeat path. Every status now goes out as success
carrying elapsed time, staticness, the test in flight and the counters,
and Get-CimInstance goes with them, its connect to a wedged WMI service
being unbounded and able to hang the one reporter left.

The tests around the reduced surface absorb what those parts were
carrying. 172 stubs PowerShell unconditionally, so a shell holding a
real pwsh and a token can no longer drive the real watchdog at
api.github.com (measured: zero outbound connections, against a control
that reaches 140.82.121.5:443), and 226 audits that every test able to
source the driver scrubs the credentials first. The staticness leg
watches long enough to see an unbounded value and grades it against
elapsed time one for one, so a clamped q now dies. Both mutant harnesses
grade on the assertion the mutant must trip rather than on a nonzero
exit. The workflow keys are read from the block that owns them through a
YAML parse, with one control per key and a round-trip control proving
that reformatting moves no verdict. The /dev/full leg runs only where a
write through the same interpreter really fails on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Grade the throttle on a ratio no slow box can move

The staticness leg carried an absolute post-count floor, which a loaded
Windows runner would have failed on its own. Every due tick logs its
line before the throttle decides, so posts against logged lines is the
same ratio at any tick rate; that check moves to its own 8s leg, and the
staticness window widens to 16s so the one-for-one comparison has room
a slow tick cannot shrink.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Kill the reporter with the wedge, and test what nothing was testing

The wedge branch ended the suite with TerminateProcess, which runs no EXIT
trap, so the PowerShell reporter outlived the step it reports on and kept
re-posting a frozen tail over the informative final status. It now goes
first, ahead of the kill.

Nothing covered the teardown either: deleting it left three orphan stubs
and 172 still passed. The stub now ticks a marker, so both the EXIT trap
and the new wedge kill are graded on a reporter that has actually stopped,
and 171 pins the order the kills fire in.

The workflow audit resolved the top-level permissions only, so a job-level
"permissions: {contents: read}" dropping statuses: write passed, and the
env keys were graded on truthiness, so a WATCHDOG_SHA swapped for the PR
head sha passed. It now reads the effective permissions of the job running
the suite and asserts the values, with a control per key. On the .ps1 side,
Get-WatchdogCounters had no assertion anywhere and the production cadence
was never read: the self-test pins both, and MaxSeconds is bounded below as
well as above, since a reporter that stops early leaves the wedge unreported.

The posting legs now assert the log never carries the token, controlled by a
mutant that logs one: tests/*.log rides into an artifact nothing masks.
Smaller: 172 takes its noexec skip before the command -v that reads false
there, and the driver scrubs the token before forking its first child.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 15:32:50 +00:00
Xavier Roche
b2cd2a51d2 An over-long FTP user name in a URL logs in as a different account (#1035)
* FTP: refuse over-long URL userinfo instead of logging in as another account

ftp_split_userpass() clipped a URL's "user[:pass]@" into user[256]/pass[256],
so a URL naming one account put a shortened name on the wire and httrack
mirrored whatever that account served, exiting clean. The split now reports a
field that does not fit and the link fails with STATUSCODE_INVALID, the
refuse-don't-clip contract ftp_command() follows for an over-long path (#1019).
A compile-time assertion pins both buffers to what a "USER <user>" control line
holds, so widening one cannot reintroduce the clip in the command formatter.

Nothing bounded the user field at the '@' either, only at the first ':' or the
end of the string, so ftp://user@host:21/f logged in as "user@host".

The ftp-userpass self-test asserted the truncation as intended behaviour.

Closes #1032

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: exercise the colon-less split, and the credentials on the wire

The ftp-userpass sweep always wrote a ':', so the branch where only the '@'
bounds the user name was seen once, three bytes long, yet that is the one
reaching memcpy() with an unbounded length. Sweep it as a third case, and add
the bare over-long and at-boundary URLs to the crawl loops.

The crawl half probes a dead port, so no login is attempted and a clip applied
after the split went unseen. A new local-ftp test drives the test server and
asserts the exact USER/PASS bytes on the control channel; it needs the server
to answer 331, since httrack sends PASS only on a 3xx.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the fixture-server reap, and drop a fail-open pipe from the FTP login test

stop_server ran a bare `wait` from an EXIT trap, so a fixture the kill never
reached blocked a test that had already passed until the harness timed it out
and called it a failure. reap_bounded gives up after REAP_GRACE, and costs
nothing when the child is already gone.

The new login test read the mirrored file through `find | head -n1`: head
exits on the first line, find takes SIGPIPE, and under pipefail that 141 aborts
the test through set -e with nothing printed at all. The mirror path is
deterministic, so assert it directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: skip the FTP login test on Windows, where it loses the x64 runner

230_local-ftp-userpass drives a real FTP login against the Python fixture, and
the x64 leg of windows-build has died every time it ran: the suite step sits
past its 45-minute timeout and the runner is lost to CPU/memory starvation,
while the Win32 leg of the same commits is green 3/3. Nothing separates the two
legs but matrix.platform, and none of the suite's own guards fired, so the cause
is on the 64-bit engine side and is not pinned yet.

Skip it there and pin the name in the driver's expected-skips ratchet, which
would otherwise red the job on an unannounced skip. The wire-level assertions
keep running on Linux and macOS, where they are what proves the credentials
reach the control channel unclipped.

Refs #1038

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 16:33:03 +02:00
Xavier Roche
562cd1c867 The test harness forks a sleep for every poll tick (#1034)
* Poll the test harness without forking a sleep per tick

Every deadline loop in the suite -- test-timeout.sh, wait_bounded,
run_with_timeout, reap_bounded -- ticked with sleep(1), one process per tick for
as long as the test ran. Measured over a serial run of the 202-test Windows
subset that is 5815 of the 25961 processes the suite creates; poll_wait replaces
them with a read(1) timing out on a fifo nobody writes to, leaving 68.

The tick can then be fine without being expensive, so wait_bounded notices a
finished crawl in 0.1s rather than up to a second, which is where the wall-clock
gain comes from.

Classify the tick by whether fd 9 is still open, not by the read's status: bash
3.2 (macOS) reports a timeout as 1, which is indistinguishable from a closed fd,
and poll_wait would retire its timer on the first whole-second tick while still
reporting the fd tick.

Left on the forked tick under MSYS: its fifos are emulated, and the leg #795 is
about is no place to find out how its select() behaves.

The new test is 225, not 224: #1029 took that number in parallel. The Windows CI
selector keys on the *_watchdog*.test topic glob rather than the number, so the
coverage follows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Put the sleep shim on a PATH MSYS can parse

MSYS hands out a drive-letter TMPDIR, so prepending the shim directory to PATH
produced an entry split at the colon: the shim was never found, every count read
zero, and the Win32 leg failed the one assertion it could still reach. The shim
now lives under a colon-free root, and count_sleeps proves it is reachable before
anything asserts on the counts, since an unreachable shim counts zero and reads
as 'nothing forked'.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the starvation cases real, and a dead fixture server loud

58_watchdog's starvation case went inert once the poll stopped forking: the shim
only ever reached the payload, so the guard measured a wall clock nobody had
stretched. It forces the forked tick like 105 does, and the override is now read
per tick rather than once, since both tests turn it on after testlib has ticked.
A poll-counted deadline calibrated to the 0.1s tick passes the old case and fails
this one.

local-crawl.sh had folded the two port-wait failures into one message, and 72 and
105 grep for it to skip an announce race: a fixture server that died at startup
was silently skipped in both. discover_server_port now returns 2 for that, and
the crawl fails loudly.

Also: a full minute for the port wait, since the loop it replaced self-extended
under the load that makes the wait long and the new one is a hard wall; a unique
fifo name, because two subshells share $$ and the loser opens a vanished path
that bash's <> turns into a never-blocking regular file; `true` over `:`, whose
redirection error kills a POSIX-mode shell; a reachability canary on the sleep
shim, which MSYS drive-letter TMPDIRs leave off PATH; and 227, renumbered off
master's 225.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 15:35:05 +02:00
Xavier Roche
a2a20ef684 Fix what Fedora's build patches around, and pin the installed file set
Fedora ships httrack with no patches, but its spec worked around three things
that were ours to fix, and its build broke twice in three weeks on changes to
our installed file set.

The metainfo declared a stock icon that is not one (stock names come from the
freedesktop Icon Naming Specification), which appstream-util rejects and which
also suppressed the merge from WebHTTrack.desktop that supplies the real icon.
libtest/readme.txt carried the pre-2005 FSF postal address. CI and
210_appstream-metainfo.test now run both validators.

225_install-manifest.test pins the installed file set, the class that broke
their 3.49.13 and 3.49.18 builds. It found a real bug on its first outing: the
.desktop files installed to $(prefix)/share/applications, so any --datadir
override put them where no desktop environment looks.
2026-08-06 10:50:56 +00:00
Xavier Roche
bfe0a4a7f7 The 16 px icon antialiases every stem to grey (#1031)
* Draw the small application icons on the pixel grid

The 16 px icon was the vector master downscaled, so every stem edge landed
on a fraction of a pixel: 74 colours in a 16x16 tile, four of them black.
The small sizes are now built in a viewBox equal to their pixel size, with
the outline coordinates rounded to whole pixels and no downscale after, so
the raster carries the field and the ink and nothing between them.

Still generated from the same Jost* master; the generator change is
httrack-works 90ceb7e2.

Test 228 counts the shipped raster's colours through a stdlib PNG reader,
since the suite cannot assume Pillow or ImageMagick on a build host.

Closes #938

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the new test's comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 11:33:03 +02:00
Xavier Roche
03c30e17ce The installed socket headers still need -D_POSIX_C_SOURCE under -std=c99 (#1028)
* The installed socket headers still need -D_POSIX_C_SOURCE under -std=c99

htsnet.h and htsopt.h were the last installed headers a strict-ISO consumer
had to prefix with -D_POSIX_C_SOURCE, and tests/206 carried an exception for
exactly those two. All of it came from one inline body: SOCaddr_inetntoa_
calls getnameinfo() and NI_NUMERICHOST, and no libc declares either to a
__STRICT_ANSI__ translation unit.

The body moves to a new src/htsnet.c and the exception in 206 goes away, so
all fourteen installed headers now compile under -std=c99 and -std=c11
against the strict-mode libc.

This turns a static inline into an exported symbol, an ABI addition rather
than a break, so VERSION_INFO gets a revision-only bump to 3:11:0 and the
soname and the Debian package stay put. proxytrack does not link libhttrack,
so it compiles htsnet.c in directly under -DHTS_NO_LIBHTTRACK, which leaves
the export marker bare there because MSVC rejects a dllimport definition.

Closes #1001

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin 206's header count to DevIncludes_DATA and unhide the strnlen check

The >= 10 floor let a header vanish from DevIncludes_DATA with 206 still
green, so #1001 could regress to "not installed" unnoticed: derive the
expected count from the list itself and name htsnet.h and htsopt.h beside
htssafe.h, since a dropped entry moves both sides of the count.

The "htssafe_strnlen_ is not libc's" preprocessor check sat inside the
gate asking whether a binary can run in TMPDIR, so a noexec TMPDIR or a
cross build skipped a check that needs no execution. Hoist it out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 08:59:27 +00:00
Xavier Roche
1a45e0237b An over-long FTP path or host name aborts the process instead of failing the link (#1029)
* FTP: refuse an over-long command instead of aborting on the copy

An FTP path or host name that survives the upstream clamps reached an
aborting strcpybuff and killed the process instead of failing the link.
The path case needs 1024 bytes exactly and nobody could reproduce it;
the host case is trivial, since the URL host was copied into a 256-byte
buffer with no check at all.

Every FTP control line now shares one FTP_LINE_SIZE, ftp_command()
reports a command that did not fit, and an over-long path or host fails
the transfer rather than being clipped into a request for another file.
send_line() also dropped the CRLF off a maximal command, and the
SIZE/MDTM probes reused the same builder.

Closes #1019

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the FTP control-line capacity and probe the host gate at its edge

The tests passed the fix without pinning it. Six mutants survived: widening
line_retr past the line[] it is copied into (that is #1019 verbatim), a stray
write one byte past ftp_command()'s bound, a host gate widened to reject every
host, and one moved off _adr[256] so a 256-byte host aborts again.

ftp_command_line() replaces the size argument at every build site with a
compile-time check that the buffer is FTP_LINE_SIZE, and the copy destination
is asserted equal to line_retr. The self-test now compares the whole poisoned
tail instead of one canary byte, and the crawl leg drives 255/256/257-byte
hosts plus a loopback outsider, so a widened gate no longer reads as a pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:51:33 +02:00
Xavier Roche
2a44dc1728 doc: note the --changes example is one captured run (#1027)
generator and date are genuine per-run fields (htschanges.c stamps
HTTRACK_VERSION and the current time on every write), so dropping them would
misrepresent the payload. Label the sample instead, so a reader does not
compare it against their own output as if it should match.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:51:25 +02:00
Xavier Roche
23e182281c The suite watchdog cannot see the engine on an emulated buildd (#1026)
* The suite watchdog cannot see the engine on an emulated buildd

Under qemu-user the process the kernel reports is the binfmt interpreter and
the engine is only its first argument, so the name matchers find nothing to
name or signal. Skip an interpreter before reading the command, gated on the
interpreter's own name: matching the argument unconditionally would take any
wrapper (strace, timeout) for the engine and signal that instead.

request_engine_backtraces only sends SIGABRT on Linux, so 105_suite-timeout
now asserts the death there and the "no stack mechanism known" report on the
platforms that get none, which is what Hurd's "GNU" uname takes.

Closes #1025

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Only shift past a real emulator, and cover the branch Hurd takes

qemu-img and its friends match "qemu-<word>" but take an image, not a
program, so shifting past them read a disk path as the process name --
into a list that is fed to kill. Exclude them, anchor the -binfmt form,
and give each a control row.

No CI leg reaches the no-stack-mechanism branch, which is the one the
Hurd buildds take, so drive it through a uname shim: it has to report
itself and leave the engine unsignalled.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 06:55:55 +00:00
Xavier Roche
3426ea35ea Release 3.49.18 (#1024)
* Release 3.49.18

Source-only release so packagers can pick up the test-suite build fixes:
3.49.17-1 was FTBFS on armhf, powerpc, hppa and loong64, and the suite
failed on a Fedora build root with no ps. The engine is untouched, so
VERSION_INFO moves by revision alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Wrap the 3.49.18 changelog entry under 80 columns

lintian's debian-changelog-line-too-long fires on two lines of the new
entry, and the CI gate runs --fail-on=error,warning. The sid perl
transition that had been masking this gate has cleared, so it is live
again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Sharpen the 3.49-18 release notes

The #1021 line named the symptom and implied the remedy was a louder
failure; the fix is a /proc fallback, so the diagnostics work. Add the
internal catch-all the 3.49-17 block carries, so the cross-arch matrix
appears to the upstream audience and not only in debian/changelog.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:43:50 +00:00
Xavier Roche
0274e016af Four architectures cannot build the altstack test shim, and nothing before the buildd sees it (#1023)
* Fix the four 3.49.17 buildd failures and cross-compile for those arches in CI

The LD_PRELOAD altstack shim did not build on Debian's 32-bit time64
architectures, and 183 read a frame count no unwinder there can produce.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Install the cross libc, and keep a trace-less report a failure

Review found the frame floor took a report naming no frame at all for a
weak unwinder, and the cross jobs had no target headers.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the cross jobs a libssl-free configure and assert what they built

No ports architecture has a cross libssl, and an empty TESTS= would have
passed having built nothing.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Add deb-src in place: a second stanza collides on the keyring spelling

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Extract zlib unpatched: Debian's arch patches need Debian's rules

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Turn off zlib's s390x vector CRC: the .dfsg repack drops its sources

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop zlib's s390x vx object too, not just its define

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the raw_mmap comment

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the raw_mmap comment inside clang-format's width

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:43:51 +00:00
Xavier Roche
22fcf9ffff The suite watchdog dumps nothing on a host without ps (#1022)
* The suite watchdog dumps nothing on a host without ps

Fedora's build root ships no procps, so every process list in the hang
diagnostics came back empty: the guard still fired, but named no process
and could not signal the wedged engine for a stack. Read /proc directly
when ps is missing, and say so when neither source is available.

Closes #1021

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Harden the no-ps fallback and its test

Review follow-ups: a ps that exits 0 while listing nothing (hidepid, a
locked-down container) now falls back too, whitespace inside an argv can
no longer shift the columns the consumers match on, a failed /proc open
stops leaking to stderr, and a bogus CLK_TCK can no longer abort the
shell. The test leg pins the header line, the column order and the group
filter, and drives request_engine_backtraces itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* kill -0 succeeds on a zombie, so the SIGABRT check failed in a container

The fake engine is reparented once its subshell exits, and a container's
pid 1 does not always reap it, so the signalled process stays visible to
kill -0. Read its state from /proc instead: gone or Z both prove the
signal landed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:16:14 +00:00
485 changed files with 19197 additions and 5816 deletions

3
.gitattributes vendored
View File

@@ -6,6 +6,3 @@
# bash die on $'\r' on every line of them.
*.test text eol=lf
*.sh text eol=lf
# Scoped to this file so a union can never touch build logic.
tests/tests-list.mk merge=union

View File

@@ -22,7 +22,7 @@ jobs:
- name: Install appstreamcli
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends appstream
appstreamcli --version

View File

@@ -38,10 +38,11 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
appstream appstream-util
- name: Configure
run: |
@@ -79,6 +80,27 @@ jobs:
- name: Build
run: make -j"$(nproc)"
# FTP_PASV is 1 in every build, so #if !FTP_PASV never reaches a compiler
# and rots unnoticed (#1091). Compiling is all this proves; nothing runs it.
- name: Compile the FTP active-mode branch
run: |
set -euo pipefail
objs="src/libhttrack_la-htsftp.lo src/libhttrack_la-htsftp.o src/.libs/libhttrack_la-htsftp.o"
# accept() marks active-mode code: absent from the object Build made,
# present below. One-sided, it passes on a file that ignored the define.
syms=$(nm --undefined-only src/.libs/libhttrack_la-htsftp.o)
if grep -qw accept <<<"$syms"; then
echo "::error::accept() is reachable with FTP_PASV=1; the marker no longer discriminates"
exit 1
fi
rm -f $objs
# Through CC: overriding CPPFLAGS would drop configure's
# -D_FORTIFY_SOURCE and the object-size warnings that come with it.
make -C src libhttrack_la-htsftp.lo CC="$CC -DFTP_PASV=0 -Werror"
syms=$(nm --undefined-only src/.libs/libhttrack_la-htsftp.o)
grep -qw accept <<<"$syms" || { echo "::error::-DFTP_PASV=0 built no active-mode code"; exit 1; }
rm -f $objs
# A backstop only: tests/test-timeout.sh bounds each test, and a healthy run
# is a minute here, two on macOS. Without it a stall ran to the job's 6h default.
- name: Test
@@ -107,7 +129,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -137,6 +159,108 @@ jobs:
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# Reproduces the buildds that refuse TEST-NET-1 instead of dropping it, where
# every fixture needing a stalled connect must skip rather than fail (#1108).
hostile-network:
name: build (network refuses TEST-NET-1)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev iproute2
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
./configure
- name: Build
run: make -j"$(nproc)"
- name: Test on a network that refuses TEST-NET-1
timeout-minutes: 20
run: |
set -euo pipefail
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
tools/hostile-net.sh make check -j"$jobs"
# A refusal landing before connect() returns skips on the older probe
# instead, leaving the settle this leg exists to guard unrun.
cat tests/246_engine-connect-stop.log
grep -q 'within 500 ms' tests/246_engine-connect-stop.log
- name: Print the test log on failure
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# The Ubuntu buildds run a development release every runner above is well behind.
# That gap is what shipped 3.49.18 broken on every Ubuntu architecture but
# riscv64, when 25.10 made uutils the default coreutils (#1042).
ubuntu-devel:
name: build (Ubuntu devel, uutils coreutils)
runs-on: ubuntu-24.04
container: ubuntu:devel
timeout-minutes: 40
steps:
# Before checkout: the container has no git, so submodules would be lost.
- name: Install build dependencies
run: |
set -euo pipefail
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
git ca-certificates python3 procps
- name: Assert the coreutils dispatch on argv[0]
# The property #1042 broke on, not the version banner, which uutils has
# already reworded once. Without it the job quietly becomes a slower copy
# of the one above.
run: |
set -euo pipefail
ln -s "$(command -v sleep)" /tmp/not-a-coreutil
if /tmp/not-a-coreutil 0; then
echo "::error::coreutils here take any argv[0]; this image no longer guards uutils" >&2
exit 1
fi
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
./configure
# A renamed dev package would disable a codec or TLS here, and the tests
# that cover them would SKIP rather than fail.
grep -q "define HTS_USEBROTLI 1" config.h
grep -q "define HTS_USEZSTD 1" config.h
grep -q "define HTS_USEOPENSSL 1" config.h
- name: Build
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 25
run: |
set -euo pipefail
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
- name: Print the test log on failure
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# Portability: build and test on macOS (Darwin/clang) on a native runner --
# no VM. The tree has no __APPLE__ branches, so Darwin exercises the
# generic-Unix path on a second libc and kernel. brew's openssl@3 is keg-only,
@@ -231,7 +355,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -340,7 +464,7 @@ jobs:
run: |
set -euo pipefail
sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential gcc-multilib autoconf automake libtool \
autoconf-archive zlib1g-dev:i386 libssl-dev:i386 \
@@ -383,7 +507,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -438,7 +562,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev
@@ -487,7 +611,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -510,6 +634,33 @@ jobs:
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
run: bash fuzz/run-fuzzers.sh fuzz check
# Catches #1030: the job above links ASan, whose runtime already defines the
# sancov hooks the instrumentation calls, hiding a program left without one.
fuzz-no-sanitizer:
name: build (fuzzers, no sanitizer runtime, clang)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
- name: Configure (fuzzers, static)
run: |
set -euo pipefail
autoreconf -fi
./configure CC=clang --enable-fuzzers --disable-shared
- name: Build
run: make -j"$(nproc)"
# Optional-dependency build: compile and test with HTTPS/OpenSSL disabled --
# the configuration users on minimal systems build, and one libssl is not even
# installed here so configure cannot silently re-enable it. The matrix above
@@ -526,7 +677,7 @@ jobs:
- name: Install build dependencies (no libssl)
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive zlib1g-dev
@@ -549,6 +700,47 @@ jobs:
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# The cell of #1139: a libc exporting getrandom() without shipping
# <sys/random.h>. Nothing else reaches it, and it is a hard build break when
# the call is not gated on the header. It also runs the /dev/urandom fallback.
no-getrandom:
name: build (getrandom header hidden)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
- name: Configure (sys/random.h hidden from configure)
run: |
set -euo pipefail
autoreconf -fi
./configure ac_cv_header_sys_random_h=no
# A libc without getrandom() at all would pass this job vacuously.
grep -q "define HAVE_GETRANDOM 1" config.h
! grep -q "define HAVE_SYS_RANDOM_H" config.h
- name: Build
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
- name: Print the test log on failure
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# OpenSSL 4.0 removes the API 3.x deprecated. Building with that set hidden
# keeps the engine ready for it: a deprecated call fails here rather than the
# day a distro moves. Compile-only, since the headers are what is under test.
@@ -563,7 +755,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -642,7 +834,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -686,6 +878,7 @@ jobs:
fi
lint:
# The name is a required status check on master; renaming it blocks every PR.
name: lint (shellcheck, shfmt)
runs-on: ubuntu-24.04
# Every tracked shell script; the globs expand at run time. Kept here so the
@@ -702,22 +895,25 @@ jobs:
src/webhttrack.in
tests/*.sh
tests/*.test
tools/macos-app.sh
tools/macos-bundle.sh
tools/macos-release.sh
tools/mkdeb.sh
tools/*.sh
steps:
- uses: actions/checkout@v7
- name: Install linters
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
# noble ships shfmt 3.8.0 (universe), matching the pinned local dev
# version; use it rather than fetching a release binary from github.com.
sudo apt-get install -y --no-install-recommends shellcheck shfmt appstream
sudo apt-get install -y --no-install-recommends shellcheck shfmt appstream \
appstream-util
# Pinned to the local dev versions: black's stable style moves across
# majors, so the runner image's build would reformat on its own schedule.
pip install --quiet black==25.1.0 flake8==7.1.1
shfmt --version
appstreamcli --version
black --version
flake8 --version
- name: shellcheck
run: shellcheck $SHELL_SCRIPTS
@@ -725,6 +921,27 @@ jobs:
- name: shfmt
run: shfmt -d -i 4 $SHELL_SCRIPTS
# Read the list rather than hand-keeping one: a .py added in a new
# directory then gets checked without anyone widening a glob.
- name: List the tracked Python files
run: |
set -euo pipefail
mapfile -t py < <(git ls-files '*.py')
test "${#py[@]}" -gt 0 # flake8 given no paths exits 0, checking nothing
echo "${#py[@]} tracked Python files"
echo "PYTHON_FILES=${py[*]}" >> "$GITHUB_ENV"
- name: black
run: black --check --diff $PYTHON_FILES
- name: flake8
run: flake8 $PYTHON_FILES
# A misnamed test falls outside the glob "make check" runs, so it goes
# missing without failing anything.
- name: test file names
run: bash tests/check-test-names.sh
# MSBuild rejects a malformed .vcxproj with a bare MSB4025 and no build, so
# catch it here in seconds rather than on a Windows runner minutes in.
- name: XML well-formedness (MSBuild project files)
@@ -742,6 +959,8 @@ jobs:
set -euo pipefail
appstreamcli validate --no-net --explain \
html/server/div/com.httrack.WebHTTrack.metainfo.xml
appstream-util validate-relax --nonet \
html/server/div/com.httrack.WebHTTrack.metainfo.xml
# Check clang-format on CHANGED LINES ONLY. The engine predates clang-format
# (it was shaped by an old Visual Studio formatter) and does not round-trip,
@@ -763,7 +982,7 @@ jobs:
| sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc >/dev/null
echo "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-19 main" \
| sudo tee /etc/apt/sources.list.d/llvm-19.list >/dev/null
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends clang-format-19
# The clang-format-19 package ships the git-clang-format driver;
# expose it unsuffixed so "git clang-format" finds it.

View File

@@ -33,12 +33,12 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev
- uses: github/codeql-action/init@v4.37.4
- uses: github/codeql-action/init@v4.37.6
with:
languages: c-cpp
build-mode: manual
@@ -67,6 +67,6 @@ jobs:
./configure
make -j"$(nproc)"
- uses: github/codeql-action/analyze@v4.37.4
- uses: github/codeql-action/analyze@v4.37.6
with:
category: "/language:c-cpp"

116
.github/workflows/cross-arch.yml vendored Normal file
View File

@@ -0,0 +1,116 @@
# Cross-compiles for the Debian architectures no GitHub runner exists for.
# 3.49.17 failed on four of them at once and every failure was a compile or
# assemble error that x86-64 and arm64 cannot produce, so the buildd was the
# first thing to see the code.
name: Cross-arch
on:
push:
branches: [master]
pull_request:
workflow_dispatch:
# Least privilege: the workflow only needs to read the repo.
permissions:
contents: read
# Cancel superseded runs on the same branch or PR.
concurrency:
group: cross-arch-${{ github.ref }}
cancel-in-progress: true
jobs:
cross:
name: cross (${{ matrix.arch }})
runs-on: ubuntu-24.04
# sid, not stable: the point is to predict what the sid buildds will say.
container: debian:sid
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
# arch selects the hardening flags, triplet is the toolchain prefix.
# The first four are 3.49.17's failures.
- { arch: armhf, triplet: arm-linux-gnueabihf }
- { arch: powerpc, triplet: powerpc-linux-gnu }
- { arch: hppa, triplet: hppa-linux-gnu }
- { arch: loong64, triplet: loongarch64-linux-gnu }
- { arch: sh4, triplet: sh4-linux-gnu }
- { arch: m68k, triplet: m68k-linux-gnu }
- { arch: sparc64, triplet: sparc64-linux-gnu }
- { arch: riscv64, triplet: riscv64-linux-gnu }
- { arch: s390x, triplet: s390x-linux-gnu }
env:
DEB_HOST_ARCH: ${{ matrix.arch }}
TRIPLET: ${{ matrix.triplet }}
ZPREFIX: /tmp/zlib-${{ matrix.arch }}
steps:
- name: Install the toolchain
run: |
set -euo pipefail
# deb-src, so zlib below comes from the archive and not a pinned URL.
# In place: a second stanza has to repeat Signed-By, and apt rejects
# the same source carrying two spellings of the keyring.
sed -i 's/^Types: deb$/Types: deb deb-src/' \
/etc/apt/sources.list.d/debian.sources
apt-get update
apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
dpkg-dev git ca-certificates \
"gcc-$TRIPLET" "libc6-dev-$DEB_HOST_ARCH-cross"
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Cross-build zlib
run: |
set -euo pipefail
# Mandatory, and no ports arch has a zlib1g-dev:<arch> to
# multiarch-install. -fPIC because libhttrack.so links it.
mkdir -p /tmp/zsrc && cd /tmp/zsrc
# Unpatched: Debian's arch-specific patches expect its own rules to
# build them (s390x's vector CRC pulls a header they add), and we only
# need something to link -lz against.
apt-get source --download-only zlib1g
dpkg-source --skip-patches -x ./*.dsc src
cd src
CHOST="$TRIPLET" CC="$TRIPLET-gcc" CFLAGS="-O2 -fPIC" \
./configure --prefix="$ZPREFIX" --static
# configure detects s390x's vector CRC, whose sources live in the
# contrib/ the .dfsg repack drops: drop the define and the object it
# would build. Nothing here needs a fast CRC.
sed -i -E 's/(-DHAVE_S390X_VX|crc32_vx\.l?o)//g' Makefile
make -j"$(nproc)"
make install
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
mkdir -p /tmp/bld && cd /tmp/bld
# The buildd's own per-arch flags: -D_FILE_OFFSET_BITS=64 is the shim
# hazard, and -fstack-clash-protection is not offered everywhere.
eval "$(dpkg-buildflags --export=sh)"
# A cross AC_TRY_RUN answers "cross", which would swap in the bundled
# snprintf that no buildd ever builds. Every glibc passes these.
# --disable-https: no ports arch has a cross libssl, so the TLS paths
# are the one part of the tree this matrix does not compile.
"$GITHUB_WORKSPACE/configure" --host="$TRIPLET" \
--build="$(dpkg-architecture -qDEB_BUILD_GNU_TYPE)" --disable-https \
ac_cv_have_working_snprintf=yes ac_cv_have_working_vsnprintf=yes \
CPPFLAGS="${CPPFLAGS:-} -I$ZPREFIX/include" \
LDFLAGS="${LDFLAGS:-} -L$ZPREFIX/lib"
- name: Build
run: make -C /tmp/bld -j"$(nproc)"
- name: Build the test artifacts
# Nothing runs here, but the shims are where three of the four 3.49.17
# failures were: they only build under `check`. Asserted, because an
# empty TESTS= would otherwise pass having built nothing.
run: |
set -euo pipefail
make -C /tmp/bld -j"$(nproc)" check TESTS=
test -f /tmp/bld/tests/.libs/libaltstackprobe.so

78
.github/workflows/emulated-suite.yml vendored Normal file
View File

@@ -0,0 +1,78 @@
# cross-arch.yml stops at compile for the ports architectures, so the Debian
# buildd is the first machine to run the suite on one, and it reports by failing
# the package build (#1146, #1148). This runs it under qemu-user, where a process
# is not the shape a native one is and every step costs far more.
name: Emulated suite
on:
# The harness is where a native-shape assumption lands, so gate on it rather
# than on every PR: this leg is slow and is not a required check.
pull_request:
paths:
- "tests/**"
- "tools/emulated-suite.sh"
- ".github/workflows/emulated-suite.yml"
push:
branches: [master]
schedule:
- cron: "41 5 * * 1"
workflow_dispatch:
# Least privilege: the workflow only needs to read the repo.
permissions:
contents: read
concurrency:
group: emulated-suite-${{ github.ref }}
cancel-in-progress: true
jobs:
emulated:
name: emulated suite (${{ matrix.arch }})
runs-on: ubuntu-24.04
# The emulated apt, configure and build take the bulk of it, well under the
# 6h ceiling: a wedge should not run that long.
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
# Emulation is what is under test, not parisc: the hppa that reported
# #1146 lives in debian-ports, with no container image and a far more
# fragile archive. s390x is in main, and big-endian, which nothing else
# here covers.
arch: [s390x]
steps:
- name: Register the binfmt handlers
run: |
set -euo pipefail
sudo apt-get update
# Registered with the F flag, so the kernel holds the interpreter open
# and the container needs no copy of it.
sudo apt-get install -y --no-install-recommends qemu-user-static binfmt-support
test -e "/proc/sys/fs/binfmt_misc/qemu-${{ matrix.arch }}" ||
{ ls /proc/sys/fs/binfmt_misc; exit 1; }
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Build and run the suite emulated
run: |
set -euo pipefail
# The build dir is a mounted volume, so the suite log outlives the
# container: a failure here has no detail otherwise.
mkdir -p "$RUNNER_TEMP/emul-bld"
docker run --rm --platform "linux/${{ matrix.arch }}" \
-v "$GITHUB_WORKSPACE:$GITHUB_WORKSPACE" -w "$GITHUB_WORKSPACE" \
-v "$RUNNER_TEMP/emul-bld:/bld" \
-e "GITHUB_WORKSPACE=$GITHUB_WORKSPACE" \
-e "EMULATED_ARCH=${{ matrix.arch }}" \
debian:sid bash ./tools/emulated-suite.sh
- name: Collect the suite log
if: failure()
uses: actions/upload-artifact@v7
with:
name: emulated-suite-${{ matrix.arch }}-log
path: ${{ runner.temp }}/emul-bld/tests/test-suite.log
if-no-files-found: warn

106
.github/workflows/httraqt.yml vendored Normal file
View File

@@ -0,0 +1,106 @@
# Build HTTraQt, the only consumer of our installed headers, against this branch.
name: HTTraQt
# Filtered, so it must never join the required set: a check that a path filter
# skips stays "Expected" forever and blocks the merge.
on:
push:
branches: [master]
paths: ['src/**', 'm4/**', 'configure.ac', '.github/workflows/httraqt.yml']
pull_request:
paths: ['src/**', 'm4/**', 'configure.ac', '.github/workflows/httraqt.yml']
schedule:
- cron: '37 05 * * 1' # upstream is pinned; this catches drift in Qt and the runner image
workflow_dispatch:
permissions:
contents: read
concurrency:
group: httraqt-${{ github.ref }}
cancel-in-progress: true
env:
# Unofficial mirror of the SourceForge tree, so no PR depends on a third-party host.
HTTRAQT_REPO: https://github.com/xroche/httraqt-ci-mirror.git
HTTRAQT_REF: a1175565c61ab9afd7bda2dc96ed1182c2d44ec3 # 1.4.11 + cmake patch, upstream HEAD since 2023-01-06
PREFIX: /usr/local
jobs:
build:
name: build HTTraQt against this branch
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
cmake qt6-base-dev qt6-multimedia-dev
- name: Build and install libhttrack
run: |
set -euo pipefail
autoreconf -fi
bld="$RUNNER_TEMP/bld"
mkdir -p "$bld"
(cd "$bld" && "$GITHUB_WORKSPACE/configure" --prefix="$PREFIX" --disable-static)
make -C "$bld" -j"$(nproc)"
sudo make -C "$bld" install
sudo ldconfig
- name: Assert no distribution libhttrack can shadow ours
run: |
set -euo pipefail
# FindHttrack.cmake hardcodes /usr/include/httrack first, so a stray libhttrack-dev would test Debian's headers.
for stray in /usr/include/httrack /usr/lib/libhttrack.so /usr/lib/*/libhttrack.so; do
if [ -e "$stray" ]; then
echo "::error::$stray exists; HTTraQt would build against it, not this branch"
exit 1
fi
done
test -f "$PREFIX/include/httrack/htsglobal.h"
- name: Fetch HTTraQt
run: |
set -euo pipefail
# Fetching the SHA itself needs no pin check afterwards, and keeps the
# tree out of the workspace so our own build cannot pick it up.
rm -rf "$RUNNER_TEMP/httraqt"
git init -q "$RUNNER_TEMP/httraqt"
ok=
for attempt in 1 2 3; do
if git -C "$RUNNER_TEMP/httraqt" fetch -q --depth 1 "$HTTRAQT_REPO" "$HTTRAQT_REF"; then ok=1; break; fi
echo "fetch failed (attempt $attempt)"
[ "$attempt" -eq 3 ] || sleep $((attempt * 15))
done
[ -n "$ok" ] || { echo "::error::cannot reach $HTTRAQT_REPO"; exit 1; }
git -C "$RUNNER_TEMP/httraqt" checkout -q --detach FETCH_HEAD
- name: Configure HTTraQt
run: |
set -euo pipefail
# CMAKE_MINIMUM_REQUIRED is 2.8 upstream, which CMake 4 rejects outright.
# The redirect is load-bearing: a bare MESSAGE() is NOTICE mode, which goes to stderr.
cmake -S "$RUNNER_TEMP/httraqt" -B "$RUNNER_TEMP/httraqt-bld" \
-DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
2>&1 | tee "$RUNNER_TEMP/cmake.log"
# A configure that resolved elsewhere would compile someone else's headers.
grep -q "httrack header directory found: $PREFIX/include/httrack/" "$RUNNER_TEMP/cmake.log"
- name: Build HTTraQt
run: cmake --build "$RUNNER_TEMP/httraqt-bld" -j"$(nproc)"
- name: Assert it linked against the libhttrack we just built
run: |
set -euo pipefail
# Not bin/: CMAKE_RUNTIME_OUTPUT_DIRECTORY is set after ADD_EXECUTABLE, too late.
bin="$RUNNER_TEMP/httraqt-bld/httraqt"
ldd "$bin" | grep -E "libhttrack\.so.* => $PREFIX/lib/" \
|| { ldd "$bin"; exit 1; }

View File

@@ -34,7 +34,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -52,6 +52,15 @@ jobs:
pip install --quiet playwright
playwright install --with-deps chromium
# The address is on screen in the wizard panes and names the mirror folder,
# so the crawl needs a presentable host rather than 127.0.0.1 and a random
# port. example.com is reserved for documentation (RFC 2606).
- name: Let the crawl address a named site on port 80
run: |
set -euo pipefail
echo "127.0.0.1 www.example.com" | sudo tee -a /etc/hosts
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80 # bind 80 unprivileged
- name: Capture
env:
LANGN: ${{ inputs.lang || '1' }}
@@ -61,6 +70,7 @@ jobs:
# --root is the dist root; the source tree already holds every piece
# htsserver reads, so this needs no install.
python3 tools/screenshot-walk.py --htsserver src/htsserver --root . \
--site-host www.example.com --site-port 80 \
--lang "$LANGN" --scale "$SCALE" --out shots
- name: Upload the screens

View File

@@ -14,6 +14,9 @@ on:
permissions:
contents: read
# The suite watchdog reports as a commit status, the only channel that
# outlives the runner it is reporting on (#795).
statuses: write
# Cancel superseded runs on the same branch or PR.
concurrency:
@@ -28,6 +31,11 @@ jobs:
matrix:
platform: [x64, Win32]
configuration: [Release]
# Windows runs from other branches starve this one's runner (#1213), so cap
# them repo-wide at one per platform. queue: max: the default cancels a waiter.
concurrency:
group: windows-build-${{ matrix.platform }}
queue: max
# Redirect vcpkg's default `files` binary cache into the workspace so
# actions/cache can persist it. vcpkg builds openssl/brotli/zlib/zstd from
# source otherwise, several minutes every run.
@@ -37,6 +45,9 @@ jobs:
- uses: actions/checkout@v7
with:
submodules: recursive # coucal lives in src/coucal
# Or the job token stays in .git/config, where every test the suite
# runs can read it.
persist-credentials: false
# Located through vswhere rather than microsoft/setup-msbuild: the repo
# only allows GitHub-owned actions.
@@ -61,17 +72,30 @@ jobs:
run: New-Item -ItemType Directory -Force -Path $env:VCPKG_DEFAULT_BINARY_CACHE | Out-Null
# x-gha is gone (vcpkg-tool #1662 dropped it after GitHub changed the cache
# API), so cache the binary archives directly. Keyed on the manifest, which
# carries the builtin-baseline, so a Dependabot bump busts it; restore-keys
# still seeds the unchanged ports' archives, so only the bumped one rebuilds.
- name: Cache vcpkg binary archives
uses: actions/cache@v6
# API), so cache the binary archives directly. An exact key hit skips the save,
# so a cache orphaned by a toolchain bump pins itself, and naming every input
# vcpkg's ABI hash folds in is a list that can come up short: key on the run
# id instead, and restore by prefix.
- name: Restore the vcpkg binary archives
uses: actions/cache/restore@v6
with:
path: ${{ github.workspace }}\vcpkg_cache
key: vcpkg-${{ matrix.platform }}-${{ hashFiles('src/vcpkg.json') }}
key: vcpkg-${{ matrix.platform }}-${{ hashFiles('src/vcpkg.json') }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: |
vcpkg-${{ matrix.platform }}-${{ hashFiles('src/vcpkg.json') }}-
vcpkg-${{ matrix.platform }}-
# Counted before the build, or vcpkg's own writes during it make a cold cache
# look warm.
- name: Count the restored vcpkg archives
shell: pwsh
run: |
$a = @(Get-ChildItem -Recurse -File -Filter *.zip `
$env:VCPKG_DEFAULT_BINARY_CACHE -EA SilentlyContinue)
Write-Host "restored $($a.Count) archive(s)"
$a.FullName | Out-File -FilePath vcpkg-restored.txt
"VCPKG_ARCHIVES_BEFORE=$($a.Count)" | Out-File -FilePath $env:GITHUB_ENV -Append
# The runner image's vcpkg checkout is pinned to some commit; our manifest's
# builtin-baseline is usually newer, so `git show <baseline>:versions/...`
# fails until that commit is local. Fetch exactly the pinned baseline (read
@@ -101,6 +125,50 @@ jobs:
if ($LASTEXITCODE -ne 0) { throw "$proj failed" }
}
# A stale cache and a warm one both restore and rebuild; only archives
# restored and unmatched prove staleness.
- name: Check whether vcpkg could use the restored archives
id: vcpkg
if: ${{ !cancelled() }}
shell: pwsh
run: |
$before = [int]$env:VCPKG_ARCHIVES_BEFORE
$stale = $false
# The log is missing when the build never reached MSBuild.
if (Test-Path msbuild-libhttrack.log) {
$m = Select-String -Path msbuild-libhttrack.log -Pattern 'Restored (\d+) package' |
Select-Object -First 1
if ($m) {
$restored = [int]$m.Matches[0].Groups[1].Value
Write-Host "vcpkg restored $restored of $before archive(s)"
$stale = ($restored -eq 0 -and $before -gt 0)
} else { Write-Host "no vcpkg restore line in the build log" }
} else { Write-Host "no build log" }
if ($stale) {
Write-Host "::warning::stale vcpkg cache: $before archives, none matching this toolchain; every port rebuilt from source"
# Provably dead: drop them rather than carry every past toolchain
# generation in the saved entry forever. -LiteralPath because a piped
# string binds -Path, which globs on a name carrying [ ].
Get-Content vcpkg-restored.txt -EA SilentlyContinue | Where-Object { $_ } |
ForEach-Object { Remove-Item -LiteralPath $_ -Force -EA SilentlyContinue }
}
$after = @(Get-ChildItem -Recurse -File -Filter *.zip `
$env:VCPKG_DEFAULT_BINARY_CACHE -EA SilentlyContinue).Count
Write-Host "$after archive(s) to cache"
# Pruning can leave the count unmoved while replacing every archive, so the
# stale case saves on its own account rather than on growth.
"save=$($stale -or ($after -gt $before))".ToLower() |
Out-File -FilePath $env:GITHUB_OUTPUT -Append
# Only when the archive set changed, so a warm run adds no entry and the
# eviction budget stays for the runs that rebuilt something.
- name: Save the vcpkg binary archives
if: ${{ !cancelled() && steps.vcpkg.outputs.save == 'true' }}
uses: actions/cache/save@v6
with:
path: ${{ github.workspace }}\vcpkg_cache
key: vcpkg-${{ matrix.platform }}-${{ hashFiles('src/vcpkg.json') }}-${{ github.run_id }}-${{ github.run_attempt }}
# WinHTTrack links src\$(Platform)\$(Configuration)\libhttrack.lib, so the
# import lib and the DLL both have to land there under that exact name.
- name: Check the binaries landed where they are expected
@@ -161,6 +229,33 @@ jobs:
throw "CRT mismatch across the boundary: libhttrack.dll=$a httrack.exe=$b"
}
# The DLL build only proves the engine's own fixed include order, the one
# order a consumer need not use, and 205/206/207/269 all gate on an automake
# environment this job has not (#1153).
- name: Compile the installed headers standalone and pairwise
shell: pwsh
timeout-minutes: 20
run: |
$ErrorActionPreference = "Stop"
$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
$vs = & $vswhere -latest -products * -property installationPath | Select-Object -First 1
if (-not $vs) { throw "Visual Studio not found" }
Import-Module "$vs\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"
$arch = if ("${{ matrix.platform }}" -eq "x64") { "x64" } else { "x86" }
Enter-VsDevShell -VsInstallPath $vs -SkipAutomaticLocation `
-DevCmdArguments "-arch=$arch -host_arch=x64"
# htsbasenet.h reaches for <openssl/ssl.h> unless a consumer turns it
# off, so the sweep needs what vcpkg installed for the build. Searched,
# not spelled: the install root nests the triplet twice.
$dirs = @(Get-ChildItem -Path "src\vcpkg_installed" -Recurse -Depth 3 `
-Directory -Filter include -ErrorAction SilentlyContinue)
if (-not ($dirs | Where-Object { Test-Path (Join-Path $_.FullName "openssl\ssl.h") })) {
throw "no vcpkg include directory under src\vcpkg_installed holds openssl/ssl.h"
}
$inc = @($dirs | ForEach-Object { "-I" + $_.FullName })
bash tests/install-headers-sweep.sh --backend cl --srcdir . -- @inc
if ($LASTEXITCODE -ne 0) { throw "the installed headers do not compile standalone and pairwise" }
# The engine ships ~90 tests and none had ever run on Windows: "make check"
# is Linux/macOS only. These are the offline ones, driven from Git Bash
# against the native httrack.exe. They subsume the self-tests this step
@@ -171,6 +266,14 @@ jobs:
shell: bash
working-directory: tests
timeout-minutes: 45
env:
# Through the environment, never argv, which the process list exposes.
WATCHDOG_TOKEN: ${{ secrets.GITHUB_TOKEN }}
WATCHDOG_REPO: ${{ github.repository }}
# github.sha here is the PR's merge commit, so statuses posted against it stay out of the PR's checks UI.
WATCHDOG_SHA: ${{ github.sha }}
WATCHDOG_CONTEXT: windows-suite (${{ matrix.platform }}, ${{ matrix.configuration }})
WATCHDOG_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
bash ./ci-windows-suite.sh \
"$(cygpath -u "$GITHUB_WORKSPACE")/src/${{ matrix.platform }}/${{ matrix.configuration }}"

View File

@@ -19,8 +19,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
(`request_queue_size`) so macOS/BSD don't drop connections under a parallel
`-c16` bigcrawl the way Python's default backlog of 5 did.
Or run `sh build.sh` to do bootstrap + configure + make in one shot.
- A `tests/NN_*.test` runs only if listed in `tests/tests-list.mk`; an
unregistered file is silently skipped.
- `configure` globs `TESTS` from `tests/[0-9]*_*.test`, so a new test needs no
registration, but an existing build dir keeps the list it was configured with:
`231_test-names.test` goes red until you reconfigure. It also compares the glob
against `git ls-files`, so a test you forgot to `git add` fails there instead of
quietly shrinking CI's suite. Name one outside the pattern and it never runs;
`tests/check-test-names.sh` (also a CI lint) rejects that.
- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
@@ -38,6 +42,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
match that is not on the last line is exposed. Capture the reply, assert the
status line it must carry (an empty, truncated or redirected one is
marker-free too), then match with a here-string: `grep -q M <<<"$reply"`.
`head -c N` and `head -n N` end the same way, and there the reds are
platform-specific: GNU `tail | head -c 3` survives, the uutils coreutils leg
turns the same line into exit 141 with no output at all. Let the reader seek
instead: `od -An -c -j <skip> -N <len> file`.
- Never assert a fault by signal *number*: SIGBUS is 7 on x86, arm, powerpc and
s390x but 10 on hppa, alpha, mips and sparc. Map it with `kill -l "$n"`.
- A fixture that needs a host to stay silent must settle (500ms), re-probe every
socket, and SKIP when one answered: the powerpc and ppc64 buildds refuse
TEST-NET-1 milliseconds after `connect()` where runners drop it.
`tools/hostile-net.sh` runs a command on such a network.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every

View File

@@ -1,6 +1,6 @@
AC_PREREQ([2.71])
AC_INIT([httrack], [3.49.17], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_INIT([httrack], [3.49.21], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_COPYRIGHT([
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998-2015 Xavier Roche and other contributors
@@ -29,6 +29,15 @@ AC_CONFIG_SRCDIR(src/httrack.c)
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_HEADERS(config.h)
AM_INIT_AUTOMAKE([subdir-objects])
# 3:14:0: revision-only bump. httrackp gained a tail field (host_alias) and #1202
# added an export (hts_host_alias_rule_ok); no layout moved, nothing went away.
# 3:13:0: revision-only bump. htsstrings.h's StringRoomTotal grew saturation in
# its macro body; no installed struct moved and no export came or went.
# 3:12:0: revision-only bump. httrackp gained a tail field (singlefile_state) and
# htsopt.h added volatile qualifiers; no layout moved, nothing went away.
# 3:11:0: revision-only bump. #1001 moved SOCaddr_inetntoa_ out of htsnet.h, adding
# an export where callers had an inline; nothing changed or went away.
# 3:10:0: revision-only bump; only the version macro moved, the engine is untouched.
# 3:9:0: revision-only bump. #991 and #1005 each added an export
# (hts_set_thread_hooks, escape_control_url); nothing changed or went away.
# 3:8:0: revision-only bump, no ABI change.
@@ -37,7 +46,7 @@ AM_INIT_AUTOMAKE([subdir-objects])
# moves nothing). Soname stays .so.3: HTTrackQt is the only consumer of the installed
# headers, so a libhttrack4 rename isn't worth it.
# (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
VERSION_INFO="3:9:0"
VERSION_INFO="3:14:0"
AM_MAINTAINER_MODE
AC_USE_SYSTEM_EXTENSIONS
@@ -392,7 +401,10 @@ AC_CHECK_TYPE(sa_family_t, [], [AC_DEFINE([sa_family_t], [uint16_t], [sa_family_
AX_CHECK_ALIGNED_ACCESS_REQUIRED
# check for various headers
AC_CHECK_HEADERS([execinfo.h sys/ioctl.h])
AC_CHECK_HEADERS([execinfo.h sys/ioctl.h sys/random.h])
## CSPRNG for the --single-file mark; /dev/urandom is the fallback
AC_CHECK_FUNCS([getrandom])
### zlib (mandatory)
CHECK_ZLIB()
@@ -547,7 +559,7 @@ AC_DEFINE(HAVE_STRNLEN, 1,[Check for strnlen])], AC_MSG_RESULT([not found]))
## Online unit tests
AC_MSG_CHECKING(whether to enable online unit tests)
AC_ARG_ENABLE([online-unit-tests],
[AS_HELP_STRING([--enable-online-unit-tests=@<:@yes/no/auto@:>@],[Enable online-unit-tests @<:@default=yes@:>@])],
[AS_HELP_STRING([--enable-online-unit-tests=@<:@yes/no/auto@:>@],[Enable online-unit-tests @<:@default=no@:>@])],
[
case "${enableval}" in
no|yes|auto)
@@ -589,6 +601,10 @@ if test x"$fuzzers" = x"yes"; then
AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer-no-link],
[DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fsanitize=fuzzer-no-link"],
[AC_MSG_ERROR([--enable-fuzzers requires libFuzzer support (clang)])])
# Ordinary programs are instrumented too, so they need that runtime at link.
AX_CHECK_LINK_FLAG([-fsanitize=fuzzer-no-link],
[DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -fsanitize=fuzzer-no-link"],
[AC_MSG_ERROR([--enable-fuzzers requires libFuzzer support (clang)])])
# clang's static sanitizer runtimes clash with -Wl,--no-undefined on the .so.
if test x"$enable_shared" != x"no"; then
AC_MSG_ERROR([--enable-fuzzers requires --disable-shared])
@@ -596,6 +612,20 @@ if test x"$fuzzers" = x"yes"; then
fi
AM_CONDITIONAL([FUZZERS], [test x"$fuzzers" = x"yes"])
# tests/Makefile.am takes its TESTS from here instead of a checked-in list,
# which made every pair of test-adding PRs conflict on the forge (#844).
AC_MSG_CHECKING([for test scripts])
TESTS_LIST=
hts_ntests=0
for hts_t in "$srcdir"/tests/[[0-9]]*_*.test; do
test -f "$hts_t" || continue
TESTS_LIST="$TESTS_LIST `echo "$hts_t" | sed 's|.*/||'`"
hts_ntests=`expr $hts_ntests + 1`
done
AC_SUBST([TESTS_LIST])
AC_MSG_RESULT([$hts_ntests])
test "$hts_ntests" -gt 0 || AC_MSG_ERROR([no test script matched tests/[[0-9]]*_*.test])
# Final output
AC_CONFIG_FILES([
Makefile

41
debian/changelog vendored
View File

@@ -1,3 +1,44 @@
httrack (3.49.21-1) unstable; urgency=medium
* New upstream release: a site answering to several hostnames can now be
mirrored once rather than once per name, plus fixes to mirror
interruption, to an update pass that purged files it had never reached,
and to several buffer overflows; full list in history.txt.
-- Xavier Roche <xavier@debian.org> Wed, 12 Aug 2026 15:05:15 +0200
httrack (3.49.20-1) unstable; urgency=medium
* New upstream release: an off-site link that looked like a file could pull
in the whole foreign site, and a mirror asked to stop while receiving over
FTP hung; full list in history.txt. The documentation was also refreshed.
* debian/copyright: record the League Spartan webfont the documentation now
bundles (OFL-1.1).
-- Xavier Roche <xavier@debian.org> Mon, 10 Aug 2026 21:08:40 +0200
httrack (3.49.19-1) unstable; urgency=medium
* New upstream release: mostly FTP fixes, plus stack-overflow, mirror
interruption and --update data-loss fixes; full list in history.txt. Two
of them matter to the build: the test suite no longer assumes GNU
coreutils, and the AppStream metainfo no longer declares a non-stock icon
name that appstream-util rejects.
-- Xavier Roche <xavier@debian.org> Sat, 08 Aug 2026 21:29:56 +0200
httrack (3.49.18-1) unstable; urgency=medium
* New upstream release, fixing the 3.49.17-1 build failures on armhf,
powerpc, hppa and loong64: the test suite's mmap interposer did not
compile on the first three, and the alternate-stack test insisted on a
longer backtrace than loong64's unwinder produces. Nothing in the engine
changed. Upstream now cross-builds the test shims for the ports
architectures on every push, so the next such break surfaces before the
buildds do.
-- Xavier Roche <xavier@debian.org> Wed, 05 Aug 2026 23:00:24 +0200
httrack (3.49.17-1) unstable; urgency=medium
* New upstream release: security fixes in the FTP and HTTP request paths and

95
debian/copyright vendored
View File

@@ -44,6 +44,13 @@ License: FSFAP
permitted in any medium without royalty provided the copyright notice and
this notice are preserved. This file is offered as-is, without any warranty.
Files: html/fonts/*
Copyright: 2020 The League Spartan Project Authors
License: OFL-1.1
Comment:
The woff2 is the latin subset at weight 700, as served by Google Fonts.
Upstream declares no Reserved Font Name, so the subset keeps the family name.
Files: debian/*
Copyright: 2002-2026 Xavier Roche <xavier@debian.org>
License: GPL-3+
@@ -106,3 +113,91 @@ License: BSD-3-clause
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
License: OFL-1.1
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
.
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
.
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
.
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
.
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.

View File

@@ -58,7 +58,9 @@ EXTRA_DIST = README.md run-fuzzers.sh \
corpus/singlefile/img-src.html corpus/singlefile/link-rel.html \
corpus/singlefile/style-block.html corpus/singlefile/style-attr.html \
corpus/singlefile/srcset.html corpus/singlefile/rawtext.html \
corpus/singlefile/malformed.html corpus/singlefile/many-attrs.html \
corpus/singlefile/malformed.html \
corpus/singlefile/mark-at-eof.html corpus/singlefile/mark-only.html \
corpus/singlefile/mark-degenerate.html corpus/singlefile/traversal.html \
corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz \
corpus/arc/roundtrip.arc corpus/arc/truncated.arc \

View File

@@ -1,3 +1,4 @@
<img src="a.png">
<img src=big.png alt=over-cap>
<img src="../escape.png"><img src="/abs.png"><img src="data:,x">
<img src="a.png">
<img src=big.png alt=over-cap>
<img src="../escape.png"><img src="/abs.png">
<img src="data:,x">

View File

@@ -1,4 +1,3 @@
<link rel="stylesheet" href="s.css">
<link rel=icon href=a.png>
<link rel="next" href="p2.html">
<link rel="preload" href="j.js">
<link rel="stylesheet" href="s.css">
<link rel="icon" href="a.png">
<link rel="canonical" href="other.html">

View File

@@ -1,5 +1,3 @@
<img src="unterminated.png
<div style="background:url(a.png">
<style>@import url(
<!-- unterminated comment
<a href=
<p a=1 b=2 c=3 d=4 e=5 f=6 g=7 h=8 i=9 j=10 title="> <img src=a.png>">x</p>
<!--><img src="a.png">
<img src="a.png" >

View File

@@ -1 +0,0 @@
<img src="a.png" a0="v" a1="v" a2="v" a3="v" a4="v" a5="v" a6="v" a7="v" a8="v" a9="v" a10="v" a11="v" a12="v" a13="v" a14="v" a15="v" a16="v" a17="v" a18="v" a19="v" a20="v" a21="v" a22="v" a23="v" a24="v" a25="v" a26="v" a27="v" a28="v" a29="v" a30="v" a31="v" a32="v" a33="v" a34="v" a35="v" a36="v" a37="v" a38="v" a39="v" a40="v" a41="v" a42="v" a43="v" a44="v" a45="v" a46="v" a47="v" a48="v" a49="v" a50="v" a51="v" a52="v" a53="v" a54="v" a55="v" a56="v" a57="v" a58="v" a59="v" a60="v" a61="v" a62="v" a63="v" a64="v" a65="v" a66="v" a67="v" a68="v" a69="v">

View File

@@ -0,0 +1 @@
a.png

View File

@@ -0,0 +1,5 @@

==
""
a.png#!htsinlin
#!htsinlin

View File

@@ -0,0 +1 @@


View File

@@ -1,4 +1,3 @@
<script>var s="</scripting>"; if(a</b) x=1;</script>
<script src="j.js"></script>
<textarea></textareas></textarea>
<title></titles></title>
<script src="j.js"></script>
<script>var u="a.png";</script>
<textarea>a.png</textarea>

View File

@@ -1,2 +1 @@
<img srcset="a.png 1x, big.png 2x, a.png 100w">
<source srcset="a.png,, a.png 2x," src="a.png">
<img srcset="a.png 1x, big.png 2x, a.png 480w">

View File

@@ -1,2 +1,2 @@
<div style="background:url(a.png);list-style:url('a.png')"></div>
<p style='background:url("a.png")'>x</p>
<div style="background:url(a.png)"></div>
<div style='background:url("a.png")'></div>

View File

@@ -1,5 +1,4 @@
<style>@import "s.css";
@import url(sub/b.css);
div{background:url(a.png)}
/* url(a.png) */ p:after{content:"url(a.png)"}
<style>@import "s.css";
@import url(sub/b.css);
div{background:url(a.png)}
</style>

View File

@@ -0,0 +1,4 @@
<img src="../../../../etc/passwd">
<img src="a.png#frag?q=1">
<img src="sub/../a.png">
<img src="./a.png">

View File

@@ -60,6 +60,10 @@ static void sf_cleanup(void) {
(void) remove(sf_root);
}
/* One httrackp for the whole run: the mark secret lives on it, so a fresh one
per input would make every mark in the corpus unrecognisable. */
static httrackp *sf_opt = NULL;
/* A missing asset would silently reduce the target to its parser half. */
static void sf_write(const char *name, const char *data, size_t len) {
char path[700];
@@ -76,6 +80,48 @@ static void sf_text(const char *name, const char *data) {
sf_write(name, data, strlen(data));
}
/* Append ref plus a real mark for it. The secret is drawn per httrackp, so a
fixture cannot spell one; every mark the target sees is built here. */
static void sf_marked(String *out, const char *ref, char cls) {
char mark[SINGLEFILE_MARK_MAX];
StringCat(*out, ref);
StringCat(*out,
singlefile_mark(sf_opt, mark, sizeof(mark), cls, strlen(ref)));
}
/* \001<ref>\002 in an input becomes <ref> plus its mark, which is the only
way a corpus file can reach the mark parser at all. */
static void sf_expand_input(const char *in, size_t len, String *out) {
size_t i, start = 0;
StringClear(*out);
for (i = 0; i < len; i++) {
if (in[i] == '\001') {
start = StringLength(*out);
} else if (in[i] == '\002') {
char mark[SINGLEFILE_MARK_MAX];
StringCat(*out, singlefile_mark(sf_opt, mark, sizeof(mark),
SINGLEFILE_CLASS_ANY,
StringLength(*out) - start));
} else {
StringAddchar(*out, in[i]);
}
}
}
static void sf_css(const char *name, const char *pre, const char *ref,
const char *post) {
String body = STRING_EMPTY;
StringCopy(body, pre);
sf_marked(&body, ref, SINGLEFILE_CLASS_ANY);
StringCat(body, post);
sf_write(name, StringBuff(body), StringLength(body));
StringFree(body);
}
static void sf_init(void) {
static const char png[] = "\x89PNG\r\n\x1a\n";
static const char big[4096] = "\x89PNG";
@@ -83,6 +129,9 @@ static void sf_init(void) {
char path[700];
hts_init();
sf_opt = hts_create_opt();
sf_opt->log = sf_opt->errlog = NULL;
sf_opt->single_file_max_size = FUZZ_SF_CAP;
snprintf(sf_root, sizeof(sf_root), "%s/httrack-fuzz-sf-XXXXXX",
tmp != NULL && tmp[0] != '\0' ? tmp : "/tmp");
if (mkdtemp(sf_root) == NULL)
@@ -95,37 +144,50 @@ static void sf_init(void) {
sf_write("a.png", png, sizeof(png) - 1);
sf_write("big.png", big, sizeof(big));
sf_text("j.js", "var x=1;\n");
/* @import plus a url(), so an inlined stylesheet recurses and its own
relative reference is rebased. */
sf_text("s.css", "@import url(sub/b.css);\ndiv{background:url(a.png)}\n");
sf_text("sub/b.css", "p{background:url(../a.png)}\n");
/* Marked, so an inlined stylesheet recurses into its own marks and its
un-inlinable reference is rebased; unmarked assets leave the target as a
bare scan that reaches nothing. */
{
String css = STRING_EMPTY;
StringCopy(css, "@import url(");
sf_marked(&css, "sub/b.css", SINGLEFILE_CLASS_CSS);
StringCat(css, ");\ndiv{background:url(");
sf_marked(&css, "a.png", SINGLEFILE_CLASS_ANY);
StringCat(css, ")}\np{background:url(");
sf_marked(&css, "big.png", SINGLEFILE_CLASS_ANY);
StringCat(css, ")}\n");
sf_write("s.css", StringBuff(css), StringLength(css));
StringFree(css);
}
sf_css("sub/b.css", "p{background:url(", "../a.png", ")}\n");
}
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
static int inited = 0;
String out = STRING_EMPTY;
httrackp *opt;
/* Exact-length, unterminated: the rewriter is span-based, so ASan bounds a
read past html_len instead of it landing on a terminator. */
char *html = malloct(size != 0 ? size : 1);
String in = STRING_EMPTY;
char *html;
size_t html_len;
if (!inited) {
sf_init();
inited = 1;
}
memcpy(html, data, size);
opt = hts_create_opt();
opt->log = opt->errlog = NULL;
opt->single_file_max_size = FUZZ_SF_CAP;
sf_expand_input((const char *) data, size, &in);
html_len = StringLength(in);
/* Exact-length, unterminated: the rewriter is span-based, so ASan bounds a
read past html_len instead of it landing on a terminator. */
html = malloct(html_len != 0 ? html_len : 1);
memcpy(html, StringBuff(in), html_len);
StringFree(in);
StringClear(out);
(void) singlefile_rewrite_html(opt, sf_root, sf_page, html, size,
(void) singlefile_rewrite_html(sf_opt, sf_root, sf_page, html, html_len,
SINGLEFILE_MAX_PAGE_SIZE, &out);
StringFree(out);
freet(html);
hts_free_opt(opt);
return 0;
}

View File

@@ -4,6 +4,64 @@ HTTrack Website Copier release history:
This file lists all changes and fixes that have been made for HTTrack
3.49-21
+ New: --host-alias (-%C) folds the several hostnames of one site onto a single canonical host, so a site that answers under more than one hostname is fetched once instead of once per name (#16)
+ New: the WebHTTrack wizard can set the host-alias rules and the guide documents them for every front end; a malformed rule is refused where it can still be corrected, through a newly exported hts_host_alias_rule_ok() (#1161, #1172, #1192)
+ Fixed: any signal caught during a mirror killed every transfer in flight, so a burst of them silently truncated the mirror; a stop now ends an HTTP transfer as the user's stop rather than a fabricated receive error (#1110)
+ Fixed: stopping an FTP transfer against a silent server waited out --timeout twice, once on the data socket and again on a QUIT reply nobody was going to send (#1096)
+ Fixed: a crawl that ended early ran --update's purge against the files it had never reached, deleting good ones (#1109)
+ Fixed: a long URL overflowed the duplicate-detection key buffer (#1160)
+ Fixed: a long command line, a large doit.log or a large .httrackrc aborted the engine while it rebuilt its arguments (#1189, #1198)
+ Fixed: a scan rule beginning with a dash ended the whole run (#1179)
+ Fixed: --index=0 and --noindex enabled the index rather than disabling it, and --purge-old=1 killed the run; an option that takes no value now refuses one instead of silently dropping it (#1195, #1200)
+ Fixed: reloading a WebHTTrack project lost every zero and every cleared field, so a rate limit of 0 came back as 100 KB/s and a cleared user-agent refilled itself (#1177, #1171, #1186)
+ Fixed: the wizard kept only the last of several --strip-query rules, and its domain answers matched beyond the domain they named (#1168, #1119)
+ Fixed: the offer to resume an interrupted mirror printed the program path instead of the command line it would rerun (#1204)
+ Fixed: the build failed where getrandom() has no declaration, and the installed headers did not compile under some include orders (#1139, #1150, #1151)
+ Changed: the documentation and the WebHTTrack pages compute their copyright year instead of carrying a fixed one (#1163, #1165)
+ Changed: refreshed the guide screenshots for all three interfaces, and repaired the documentation's dark mode and its search-target tabs (#1164, #1158)
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-20
+ Fixed: a link to another host whose name looked like a file was scanned as a page, so one link could mirror the whole foreign site on the referring page's depth budget (#121)
+ Fixed: a size-bounded filter built both of its filter URLs wrong when the path had no leading slash (#1095)
+ Fixed: a mirror told to quit while an FTP file was being received announced the exit and then waited forever on that slot (#1106)
+ Fixed: stopping an FTP transfer closed its data socket twice, so a threaded crawl could lose another worker's connection (#1090)
+ Fixed: WebHTTrack's welcome page opened its language menu blank instead of marking the language in use (#1084)
+ Fixed: two suite tests failed on the hppa, powerpc and ppc64 Debian buildds, holding the package back on those ports (#1108)
+ Fixed: a growable string could wrap its capacity to zero instead of saturating (#1062)
+ Changed: the macOS disk image carries its architecture in its name, so an Intel user can tell before downloading that the build is for Apple Silicon (#1083)
+ Changed: refreshed the GUI guide, its screenshots, the documentation typography and the footer contrast
+ Changed: settled on one spelling of the product name across the docs and the site
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-19
+ Fixed: an over-long user name in an ftp:// URL was truncated at the credential boundary, so the login went to a different account (#1032)
+ Fixed: an over-long FTP path or host name aborted the process instead of failing that one link (#1019)
+ Fixed: an FTP crawl ignored --timeout and --max-time, so a server that went quiet held a slot for five minutes (#1039)
+ Fixed: an FTP crawl of a dead host took two minutes to give up, and no signal could shorten it (#1059)
+ Fixed: the engine released the transfer slots while an FTP thread was still writing to one, crashing a mirror that hit its size or time limit (#1051)
+ Fixed: Ctrl-C did not shorten a connection or a name lookup already in flight, so ending a crawl against an unreachable host took minutes (#1073)
+ Fixed: a crawl that gave up mid-loop, on the link limit or an abort, lost its WARC archive and its change report, and a WARC written past teardown was stranded in a .tmp file (#1061, #1060)
+ Fixed: a crawl that rolled back for want of data still replaced the previous WARC archive, stranding its temporary and never writing its index; a stale or unwritable index is now reported rather than left silently in place (#1053)
+ Fixed: a resume the server answered with an unusable Content-Range deleted the partial file and never fetched it again; the restart no longer spends the retry budget, and is latched to one per link (#581, #1052)
+ Fixed: a long argument to -n or -O smashed the stack through an unbounded panic message, and the growable arrays could wrap their capacity to zero (#1040)
+ Fixed: --single-file could write a reference mark longer than it reads back, leaving it in the delivered page as visible text (#1054, #1069)
+ Fixed: on a terminal with no input left, a crawl toggled between the spinner and the full panel on every pass (#1072)
+ Fixed: -%v1 ended each line with a carriage return, so the lines overwrote each other and a redirected log was unreadable; -%v2 now paints its panel only on a terminal
+ Fixed: the installed socket headers did not compile under -std=c99 without _POSIX_C_SOURCE (#1001)
+ Fixed: the 16 px application icon rendered every stem as grey (#938)
+ Fixed: the desktop files installed outside the data directory, and the AppStream metainfo declared an icon name that is not a stock one
+ Fixed: the build failed on distributions whose coreutils are not GNU's (#1042)
+ Changed: --single-file drops its own HTML and CSS scanner and reuses the crawler's parser, so it follows the same references the mirror does (#749)
+ Changed: updated Turkish translation, and repaired two mangled Polish strings (#306)
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-18
+ Fixed: the 3.49.17 package build failed on four architectures: the test suite's mmap interposer did not compile on armhf, powerpc or hppa, and the alternate-stack test rejected loong64's shorter backtrace (#1023)
+ Fixed: the test suite failed on a host with no ps command, such as a Fedora build root; its hang diagnostics now read /proc directly (#1021)
+ Changed: multiple internal test and CI improvements, including a cross-compile matrix covering the Debian ports architectures
3.49-17
+ New: a pkg-config file, libhttrack.pc, ships with the development headers (#1018)
+ Fixed: a crawled page could read the WebHTTrack session id and drive the control panel; the id is now unguessable and cross-origin commands are refused (#877)

View File

@@ -4,6 +4,7 @@ HelpHtmldir = $(htmldir)
HelpHtmlimgdir = $(HelpHtmldir)/img
HelpHtmldivdir = $(HelpHtmldir)/div
HelpHtmlimagesdir = $(HelpHtmldir)/images
HelpHtmlfontsdir = $(HelpHtmldir)/fonts
HelpHtmlTxtdir = $(HelpHtmldir)
WebHtmldir = $(HelpHtmldir)/server
WebHtmlimagesdir = $(HelpHtmldir)/server/images
@@ -15,7 +16,7 @@ WebIcon64x64dir = $(datadir)/icons/hicolor/64x64/apps
WebIcon128x128dir = $(datadir)/icons/hicolor/128x128/apps
WebIcon256x256dir = $(datadir)/icons/hicolor/256x256/apps
WebIconScalabledir = $(datadir)/icons/hicolor/scalable/apps
VFolderEntrydir = $(prefix)/share/applications
VFolderEntrydir = $(datadir)/applications
MetaInfodir = $(datadir)/metainfo
# Wildcards are globbed against $(srcdir): a bare "*.html" is resolved against
@@ -27,6 +28,8 @@ HelpHtml_DATA = $(srcdir)/*.html $(srcdir)/*.js $(srcdir)/*.css
HelpHtmldiv_DATA = div/search.sh
HelpHtmlimg_DATA = $(srcdir)/img/*
HelpHtmlimages_DATA = $(srcdir)/images/*
# doc.css reaches the woff2 by a relative URL, so it installs beside the pages.
HelpHtmlfonts_DATA = $(srcdir)/fonts/*
HelpHtmlTxt_DATA = ../greetings.txt ../history.txt ../license.txt
WebHtml_DATA = $(srcdir)/server/*.html $(srcdir)/server/*.js $(srcdir)/server/*.css
WebHtmlimages_DATA = $(srcdir)/server/images/*
@@ -45,6 +48,7 @@ VFolderEntry_DATA = $(srcdir)/server/div/*.desktop
MetaInfo_DATA = $(srcdir)/server/div/*.metainfo.xml
EXTRA_DIST = $(HelpHtml_DATA) $(HelpHtmlimg_DATA) $(HelpHtmlimages_DATA) \
$(HelpHtmlfonts_DATA) \
$(HelpHtmldiv_DATA) $(WebHtml_DATA) $(WebHtmlimages_DATA) \
$(WebPixmap_DATA) $(WebIcon16x16_DATA) $(WebIcon32x32_DATA) $(WebIcon48x48_DATA) \
$(WebIcon64x64_DATA) $(WebIcon128x128_DATA) $(WebIcon256x256_DATA) \

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -534,9 +536,14 @@ And then, put the email address in your pages through:
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -75,8 +77,8 @@ The main advantages of this cache are:
<ul>
<li>One single file for a complete website cache archive</li>
<li>Standard <a href="http://www.pkware.com/products/enterprise/white_papers/appnote.txt" target="_new">ZIP</a> format, that can be easily reused on most platforms and languages</li>
<li>Compressed data with the efficient and opened <a href="http://www.gzip.org/zlib/" target="_new">zlib</a> format</li>
<li>Standard <a href="http://www.pkware.com/products/enterprise/white_papers/appnote.txt" target="_blank">ZIP</a> format, that can be easily reused on most platforms and languages</li>
<li>Compressed data with the efficient and opened <a href="http://www.gzip.org/zlib/" target="_blank">zlib</a> format</li>
</ul>
The cache is made of ZIP files entries ; with one ZIP file entry per fetched URL (successfully or not - errors are also stored).<br />
@@ -84,7 +86,7 @@ For each entry:
<ul>
<li>The ZIP file name is the original URL [<small><a href="#orig">see notes below</a></small>]</li>
<li>The ZIP file contents, <b>if available</b>, is the original (compressed, using the deflate algorythm) data</li>
<li>The ZIP file extra field (in the local file header) contains a list of meta-fields, very similar to the <a href="http://www.ietf.org/rfc/rfc2616.txt?number=2616" target="new_">HTTP</a> headers fields. See also <a href="http://www.ietf.org/rfc/rfc2396.txt?number=2396" target="new_">RFC</a>.</li><br />
<li>The ZIP file extra field (in the local file header) contains a list of meta-fields, very similar to the <a href="http://www.ietf.org/rfc/rfc2616.txt?number=2616" target="_blank">HTTP</a> headers fields. See also <a href="http://www.ietf.org/rfc/rfc2396.txt?number=2396" target="_blank">RFC</a>.</li><br />
<li>The ZIP file timestamp follows the "Last-Modified-Since" field given for this URL, if any</li>
</ul>
@@ -230,9 +232,14 @@ Libraries should generally handle this peculiar format, however.
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -155,6 +157,8 @@ rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
<h3 id="example">Example</h3>
One captured run; <tt>generator</tt> and <tt>date</tt> will read differently in yours.
<pre>
{
"schema": 1,
@@ -209,9 +213,14 @@ scope: both change what a mirror directory contains.</li>
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -117,6 +119,7 @@ the index" problems disappear.</p>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--depth (-r)</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
<tr><td><tt>--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
<tr><td><tt>--host-alias (-%C)</tt></td><td>Declares that several hostnames serve the same site, for example <tt>--host-alias "www2.example.com,m.example.com=example.com"</tt>. Those names then count as the starting host: their links travel under the same-address rules, are fetched from the canonical host, and are saved once rather than once per name. The left side takes filter-style globs and comma-separated names, the right side one literal host, port included. Either side may name a scheme (<tt>legacy.example.com=https://example.com</tt> reaches a host that only answers over HTTPS, and <tt>https://www.example.com=ftp://files.example.com</tt> matches only the HTTPS form). A path is refused: this maps hosts, not URLs.</td></tr>
<tr><td><tt>--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
<tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
<tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
@@ -146,7 +149,9 @@ usually not a scope option at all: it is an off-host redirect. A start URL of
<tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
lands you on a different host, and same-host scope stops the crawl there. Start
from the final URL, or add a filter that re-admits the real host (see
<a href="#filters">Filters</a>). The log will show the redirect.</p>
<a href="#filters">Filters</a>). The log will show the redirect. When both names
really are one site, <tt>--host-alias</tt> says so once and the mirror stops
depending on which one you started from.</p>
<p><tt>-n</tt> is the fix for pages that render locally without their images or
stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note
@@ -503,9 +508,14 @@ all disabled.</small></p>
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -23,12 +23,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -206,9 +208,14 @@ roche at httrack dot com (Xavier ROCHE)<br>
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -103,9 +105,14 @@ compared to the previous mirror. This page describes that file.
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -32,6 +32,20 @@
}
}
/* Additions stay out of the palette block above: it is kept byte-identical with the website's. */
:root {
--body-font: system-ui, -apple-system, "Segoe UI", Roboto, "Trebuchet MS", Verdana, sans-serif;
}
/* Relative URL: the docs are read from an installed directory too. */
@font-face {
font-family: "League Spartan";
src: url(fonts/league-spartan-700-latin.woff2) format("woff2");
font-weight: 700;
font-style: normal;
font-display: swap;
}
* { box-sizing: border-box; }
/* The plain declaration before each var() is what IE11 gets: it drops the
@@ -44,6 +58,7 @@ body {
color: #111;
color: var(--ink);
font: 1em/1.6 "Trebuchet MS", Verdana, system-ui, -apple-system, "Segoe UI", sans-serif;
font-family: var(--body-font);
}
/* ---- page frame ---- */
@@ -72,7 +87,6 @@ body {
/* A one-line label has no use for the body's prose leading. */
line-height: 1;
padding: .3rem .4rem;
margin-top: .4rem;
}
.wrap {
@@ -106,11 +120,18 @@ body {
.wrap.solo { display: block; }
}
/* Colour inherited: nothing lighter than --ink reaches 4.5:1 on the --field
this sits on, and at this size it is small text. */
/* Mirrors site.css's .sitefooter: flush on the panel's black edge, chicklet at the far end. */
footer {
max-width: 68rem;
margin: .5rem auto 0;
color: #eef;
margin: 0 auto;
font-size: .8em;
line-height: 1.3;
display: flex;
justify-content: space-between;
align-items: center;
gap: 1rem;
}
@media (prefers-color-scheme: dark) {
@@ -121,6 +142,9 @@ footer {
h2, h3 { scroll-margin-top: 1rem; }
/* Top two levels only: a display face reads worse than the system font over prose. */
h1, h2 { font-family: "League Spartan", var(--body-font); font-weight: 700; }
h1 { font-size: 1.9em; margin: 0 0 .3em; line-height: 1.2; }
h2 { font-size: 1.45em; margin: 2.5em 0 .5em; padding-bottom: .2em; border-bottom: 2px solid var(--rule); }
h3 { font-size: 1.15em; margin: 2em 0 .4em; }
@@ -342,8 +366,26 @@ dialog#zoom::backdrop { background: rgba(0, 0, 0, .8); }
z-index: 10;
}
/* Some browsers still apply prefers-color-scheme: dark when printing, so the ink
comes out #e6e6ee on white paper. Reset the palette, not each rule reading it. */
@media print {
:root {
--field: #77b;
--panel: #ccd;
--accent: #99c;
--ink: #111;
--ink-soft: #444;
--rule: #aab;
--link: #00c;
--link-hover: #c00;
--code-bg: #e8e8f2;
--chip: #dde;
--shadow: rgba(0, 0, 0, .25);
}
body { background: #fff; }
.toc, .filter { display: none; }
.wrap { display: block; border: 0; }
/* Not a token, same paper: the dark scheme inverts the wordmark to white. */
.masthead img { filter: none; }
}

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -318,7 +320,7 @@ However, be sure to always download HTTrack from a <b>trusted source</b> (prefer
<br><br><a NAME="QG0c">Q: <strong>This software is 'free', but I bought it from an authorized reseller . What's going on?</strong></a><br>
A: <em>
HTTrack is free (free as in 'freedom') as it is covered by the <a href="http://www.gnu.org/licenses/gpl.txt" target="_new">GNU General Public License (GPL)</a>.
HTTrack is free (free as in 'freedom') as it is covered by the <a href="http://www.gnu.org/licenses/gpl.txt" target="_blank">GNU General Public License (GPL)</a>.
You can freely download it, without paying any fees, copy it to your friends, and modify it if you respect the license.
There are NO official/authorized resellers, because HTTrack is <b>NOT</b> a commercial product.
But you can be charged for duplication fees, or any other services (example: software CDroms or shareware collections, or fees for maintenance),
@@ -881,9 +883,14 @@ A: <em>Feel free to <a href="contact.html">contact us</a>!
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -174,7 +176,7 @@ Action options:
Proxy options:
P proxy use (-P proxy:port or -P user:pass@proxy:port) (--proxy &lt;param&gt;)
%f *use proxy for ftp (f0 don't use) (--httpproxy-ftp[=N])
%f *use proxy for ftp (f0 don't use) (--httpproxy-ftp[=0])
Limits options:
rN set the mirror depth to N (* r9999) (--depth[=N])
@@ -195,7 +197,7 @@ Flow control:
HN host is abandoned if: 0=never, 1=timeout, 2=slow, 3=timeout or slow (--host-control[=N])
Links options:
%P *extended parsing, attempt to parse all links, even in unknown tags or Javascript (%P0 don't use) (--extended-parsing[=N])
%P *extended parsing, attempt to parse all links, even in unknown tags or Javascript (%P0 don't use) (--extended-parsing[=0])
n get non-html files 'near' an html file (ex: an image located outside) (--near)
t test all URLs (even forbidden ones) (--test)
%L &lt;file&gt; add all URL located in this text file (one URL per line) (--list &lt;param&gt;)
@@ -209,7 +211,7 @@ Build options:
%x do not include any password for external password protected websites (%x0 include) (--no-passwords)
%q *include query string for local files (useless, for information purpose only) (%q0 don't include) (--include-query-string)
o *save the server's error pages (404..) (o0 discard them) (--generate-errors)
X *purge old files after update (X0 keep delete) (--purge-old[=N])
X *purge old files after update (X0 keep delete) (--purge-old[=0])
Spider options:
bN accept cookies in cookies.txt (0=do not accept,* 1=accept) (--cookies[=N])
@@ -2669,9 +2671,14 @@ for only 2 simultaneous sesions.
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -13,12 +13,14 @@
<body>
<!-- doc-chrome:top -->
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -416,9 +418,14 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

93
html/fonts/OFL.txt Normal file
View File

@@ -0,0 +1,93 @@
Copyright 2020 The League Spartan Project Authors (https://github.com/theleagueof/league-spartan)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.

Binary file not shown.

View File

@@ -13,12 +13,14 @@
</head>
<body>
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -58,6 +60,9 @@
<main id="main">
<!-- Section ids are link API: the redirect stubs and the Android app's contextual help
hardcode them, and an app release outlives its engine pin. -->
<h1>The HTTrack interface, step by step</h1>
<p class="lede">HTTrack copies a website to your disk so you can read it offline. The same
@@ -88,7 +93,7 @@ itself down once those pings stop.</p>
<p>Each option also lists its command-line equivalent, so anything you set here can later be
scripted. The <a href="cmdguide.html">command-line guide</a> covers that side.</p>
<div class="note" data-for="droid">
<div class="note" data-for="droid" id="droid-first-launch">
<p><b>First launch.</b> Android asks for permission to store mirrors on your device. Without
it the app cannot save anything, so tap <b>Allow</b>.</p>
<figure>
@@ -117,13 +122,13 @@ data-for="droid">Tap <b>Next</b></span> to create a project, or open one you alr
<figure>
<img data-for="win" src="img/guide-win-start.png" alt="WinHTTrack welcome pane" width="1040" height="744">
<img data-for="web" src="img/guide-web-start.png" alt="WebHTTrack welcome page" width="1024" height="388">
<img data-for="web" src="img/guide-web-start.png" alt="WebHTTrack welcome page" width="1024" height="387">
<img data-for="droid" src="img/guide-droid-start.png" alt="HTTrack for Android welcome screen" width="600" height="1300">
<figcaption>The welcome screen.</figcaption>
</figure>
<p class="note" data-for="web">The language dropdown starts blank on purpose: its first entry
means "leave the interface as it is". Pick a language only if you want to change it.</p>
<p class="note" data-for="web">The dropdown shows the language the interface is already in.
Picking another one reloads the page in it.</p>
<p class="note" data-for="web">On macOS, <code>brew install httrack</code> installs WebHTTrack
alongside the command line tool. Open <b>HTTrack.app</b> if you have it, or run
@@ -138,7 +143,7 @@ together in the list.</p>
<figure>
<img data-for="win" src="img/guide-win-project-name.png" alt="WinHTTrack project name, category and base path" width="1040" height="744">
<img data-for="web" src="img/guide-web-project-name.png" alt="WebHTTrack project name, category and base path" width="1024" height="346">
<img data-for="web" src="img/guide-web-project-name.png" alt="WebHTTrack project name, category and base path" width="1024" height="345">
<img data-for="droid" src="img/guide-droid-project-name.png" alt="Android project name, category and base path" width="600" height="1300">
<figcaption>Project name, category, and where the files go.</figcaption>
</figure>
@@ -157,7 +162,7 @@ keep their links to each other.</p>
<figure>
<img data-for="win" src="img/guide-win-project-setup.png" alt="WinHTTrack action and web addresses" width="1040" height="744">
<img data-for="web" src="img/guide-web-project-setup.png" alt="WebHTTrack action and web addresses" width="1024" height="496">
<img data-for="web" src="img/guide-web-project-setup.png" alt="WebHTTrack action and web addresses" width="1024" height="495">
<img data-for="droid" src="img/guide-droid-project-setup.png" alt="Android web address and action" width="600" height="1300">
<figcaption>The addresses to copy, and what to do with them.</figcaption>
</figure>
@@ -212,7 +217,7 @@ port, you point your browser at it as a proxy and click the link or the submit b
usual, and the address the browser asks for is added to the project.</p>
<figure data-for="web">
<img data-for="web" src="img/guide-web-add-url.png" alt="WebHTTrack add-a-URL dialog" width="1024" height="634">
<img data-for="web" src="img/guide-web-add-url.png" alt="WebHTTrack add-a-URL dialog" width="1024" height="600">
<figcaption>Adding a single address, with an optional login.</figcaption>
</figure>
@@ -241,7 +246,7 @@ mirror straight away.</p>
<figure data-for="win web">
<img data-for="win" src="img/guide-win-ready.png" alt="WinHTTrack connection settings before starting" width="1040" height="744">
<img data-for="web" src="img/guide-web-ready.png" alt="WebHTTrack ready to start" width="1024" height="508">
<img data-for="web" src="img/guide-web-ready.png" alt="WebHTTrack ready to start" width="1024" height="489">
<figcaption>The last screen before the mirror runs.</figcaption>
</figure>
@@ -274,7 +279,7 @@ mirrored pages are rewritten to point at each other, so it browses offline in an
<figure>
<img data-for="win" src="img/guide-win-finished.png" alt="WinHTTrack mirror complete" width="1040" height="744">
<img data-for="web" src="img/guide-web-finished.png" alt="WebHTTrack mirror complete" width="1024" height="487">
<img data-for="web" src="img/guide-web-finished.png" alt="WebHTTrack mirror complete" width="1024" height="486">
<img data-for="droid" src="img/guide-droid-finished.png" alt="Android mirror complete" width="600" height="1300">
<figcaption>Done.</figcaption>
</figure>
@@ -287,7 +292,7 @@ usual causes.</p>
<p data-for="win web">The files sit under the base path you chose, in a folder named after the
project. Opening <code>index.html</code> there browses the mirror without HTTrack.</p>
<p data-for="droid">Mirrors are written to
<p data-for="droid" id="droid-storage-location">Mirrors are written to
<code>/storage/emulated/0/HTTrack/Websites</code>, in a folder named after the project. That is
shared storage, so a file manager or a USB cable can reach it, and opening
<code>index.html</code> there browses the mirror without the app.</p>
@@ -438,7 +443,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-flow-control.png" alt="WinHTTrack Flow Control tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-flow-control.png" alt="WebHTTrack Flow Control tab" width="1024" height="910">
<img data-for="web" src="img/guide-web-opt-flow-control.png" alt="WebHTTrack Flow Control tab" width="1024" height="862">
<img data-for="droid" src="img/guide-droid-opt-flow-control.png" alt="Android Flow Control tab" width="600" height="1300">
</figure>
@@ -481,7 +486,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-links.png" alt="WinHTTrack Links tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-links.png" alt="WebHTTrack Links tab" width="1024" height="781">
<img data-for="web" src="img/guide-web-opt-links.png" alt="WebHTTrack Links tab" width="1024" height="749">
<img data-for="droid" src="img/guide-droid-opt-links.png" alt="Android Links tab" width="600" height="1300">
</figure>
@@ -518,7 +523,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-build.png" alt="WinHTTrack Build tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-build.png" alt="WebHTTrack Build tab" width="1024" height="986">
<img data-for="web" src="img/guide-web-opt-build.png" alt="WebHTTrack Build tab" width="1024" height="976">
<img data-for="droid" src="img/guide-droid-opt-build.png" alt="Android Build tab" width="600" height="1300">
</figure>
@@ -584,7 +589,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-browser-id.png" alt="WinHTTrack Browser ID tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-browser-id.png" alt="WebHTTrack Browser ID tab" width="1024" height="706">
<img data-for="web" src="img/guide-web-opt-browser-id.png" alt="WebHTTrack Browser ID tab" width="1024" height="654">
<img data-for="droid" src="img/guide-droid-opt-browser-id.png" alt="Android Browser ID tab" width="600" height="1300">
</figure>
@@ -624,7 +629,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-spider.png" alt="WinHTTrack Spider tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-spider.png" alt="WebHTTrack Spider tab" width="1024" height="1557">
<img data-for="web" src="img/guide-web-opt-spider.png" alt="WebHTTrack Spider tab" width="1024" height="1770">
<img data-for="droid" src="img/guide-droid-opt-spider.png" alt="Android Spider tab" width="600" height="1300">
</figure>
@@ -698,11 +703,23 @@ bottom are the ones that keep you welcome.</p>
mirror something that needs you logged in, by exporting the session from your browser.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt" data-for="win web droid">
<div class="opt-head"><span class="opt-name">Strip query keys</span><span class="cli">-%g, --strip-query</span></div>
<p>Comma-separated query keys to drop when naming saved files, such as
<code>sid,utm_source</code>. Keeps one page from being saved several times under tracking
parameters that change nothing.</p>
parameters that change nothing. You can scope a rule to the addresses it covers, for example
<code>shop.example.com/*=sid</code>.</p>
<p data-for="web">The box takes one rule per line. Only the last matching rule applies, so keep
the keys for one scope together on a line rather than on lines of their own.</p>
</div>
<div class="opt" data-for="win web droid">
<div class="opt-head"><span class="opt-name">Host aliases</span><span class="cli">-%C, --host-alias</span></div>
<p>Names the other hostnames one site answers to, one rule per line, written
<code>www2.example.com,m.example.com=example.com</code>. They fold onto that host: their links
travel under the same rules and each page is saved once instead of once per name. Either side may
carry a scheme, as in <code>legacy.example.com=https://example.com</code>. A trailing slash is
ignored, but a path is refused: this maps hosts, not addresses.</p>
</div>
</section>
@@ -711,7 +728,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-proxy.png" alt="WinHTTrack Proxy tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-proxy.png" alt="WebHTTrack Proxy tab" width="1024" height="762">
<img data-for="web" src="img/guide-web-opt-proxy.png" alt="WebHTTrack Proxy tab" width="1024" height="760">
<img data-for="droid" src="img/guide-droid-opt-proxy.png" alt="Android Proxy tab" width="600" height="1300">
</figure>
@@ -748,7 +765,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-log-index-cache.png" alt="WinHTTrack Log, Index, Cache tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-log-index-cache.png" alt="WebHTTrack Log, Index, Cache tab" width="1024" height="1278">
<img data-for="web" src="img/guide-web-opt-log-index-cache.png" alt="WebHTTrack Log, Index, Cache tab" width="1024" height="1266">
<img data-for="droid" src="img/guide-droid-opt-log-index-cache.png" alt="Android Log, Index, Cache tab" width="600" height="1300">
</figure>
@@ -847,7 +864,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-experts-only.png" alt="WinHTTrack Experts Only tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-experts-only.png" alt="WebHTTrack Experts Only tab" width="1024" height="1088">
<img data-for="web" src="img/guide-web-opt-experts-only.png" alt="WebHTTrack Experts Only tab" width="1024" height="1085">
<img data-for="droid" src="img/guide-droid-opt-experts-only.png" alt="Android Experts Only tab" width="600" height="1300">
</figure>
@@ -910,7 +927,12 @@ are how you widen a crawl deliberately.</p>
<dialog id="zoom" aria-label="Enlarged screenshot"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

View File

@@ -76,7 +76,7 @@ offline browser : copy websites to a local directory</p>
[ <b>-w, --mirror</b> ] [ <b>-W, --mirror-wizard</b> ] [
<b>-g, --get-files</b> ] [ <b>-i, --continue</b> ] [ <b>-Y,
--mirrorlinks</b> ] [ <b>-P, --proxy</b> ] [ <b>-%f,
--httpproxy-ftp[=N]</b> ] [ <b>-%b, --bind</b> ] [ <b>-rN,
--httpproxy-ftp[=0]</b> ] [ <b>-%b, --bind</b> ] [ <b>-rN,
--depth[=N]</b> ] [ <b>-%eN, --ext-depth[=N]</b> ] [ <b>-mN,
--max-files[=N]</b> ] [ <b>-MN, --max-size[=N]</b> ] [
<b>-EN, --max-time[=N]</b> ] [ <b>-AN, --max-rate[=N]</b> ]
@@ -85,48 +85,50 @@ offline browser : copy websites to a local directory</p>
--sockets[=N]</b> ] [ <b>-TN, --timeout[=N]</b> ] [ <b>-RN,
--retries[=N]</b> ] [ <b>-JN, --min-rate[=N]</b> ] [ <b>-HN,
--host-control[=N]</b> ] [ <b>-%P,
--extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
--extended-parsing[=0]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
--test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
--cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
] [ <b>-%Z, --single-file</b> ] [ <b>-LN,
--long-names[=N]</b> ] [ <b>-KN, --keep-links[=N]</b> ] [
<b>-x, --replace-external</b> ] [ <b>-%x,
--disable-passwords</b> ] [ <b>-%q,
--include-query-string</b> ] [ <b>-%g, --strip-query</b> ] [
<b>-o, --generate-errors</b> ] [ <b>-X, --purge-old[=N]</b>
] [ <b>-%p, --preserve</b> ] [ <b>-%T, --utf8-conversion</b>
[ <b>-%N, --delayed-type-check[=N]</b> ] [ <b>-%D,
--cached-delayed-type-check[=0]</b> ] [ <b>-%M,
--mime-html[=0]</b> ] [ <b>-%Z, --single-file[=0]</b> ] [
<b>-LN, --long-names[=N]</b> ] [ <b>-KN,
--keep-links[=N]</b> ] [ <b>-x, --replace-external</b> ] [
<b>-%x, --disable-passwords[=0]</b> ] [ <b>-%q,
--include-query-string[=0]</b> ] [ <b>-%g, --strip-query</b>
] [ <b>-%C, --host-alias</b> ] [ <b>-o,
--generate-errors[=N]</b> ] [ <b>-X, --purge-old[=0]</b> ] [
<b>-%p, --preserve</b> ] [ <b>-%T, --utf8-conversion[=0]</b>
] [ <b>-bN, --cookies[=N]</b> ] [ <b>-%K, --cookies-file</b>
] [ <b>-%Y, --why</b> ] [ <b>-u, --check-type[=N]</b> ] [
<b>-j, --parse-java[=N]</b> ] [ <b>-sN, --robots[=N]</b> ] [
<b>-%h, --http-10</b> ] [ <b>-%k, --keep-alive</b> ] [
<b>-%z, --disable-compression</b> ] [ <b>-%B, --tolerant</b>
] [ <b>-%s, --updatehack</b> ] [ <b>-%u, --urlhack</b> ] [
<b>-%A, --assume</b> ] [ <b>-@iN, --protocol[=N]</b> ] [
<b>-%w, --disable-module</b> ] [ <b>-F, --user-agent</b> ] [
<b>-%R, --referer</b> ] [ <b>-%E, --from</b> ] [ <b>-%F,
--footer</b> ] [ <b>-%l, --language</b> ] [ <b>-%a,
--accept</b> ] [ <b>-%X, --headers</b> ] [ <b>-C,
--cache[=N]</b> ] [ <b>-k, --store-all-in-cache</b> ] [
<b>-%r, --warc</b> ] [ <b>-%d, --changes</b> ] [ <b>-%n,
--do-not-recatch</b> ] [ <b>-%v, --display</b> ] [ <b>-Q,
--do-not-log</b> ] [ <b>-q, --quiet</b> ] [ <b>-z,
--extra-log</b> ] [ <b>-Z, --debug-log</b> ] [ <b>-v,
--verbose</b> ] [ <b>-f, --file-log</b> ] [ <b>-f2,
--single-log</b> ] [ <b>-I, --index</b> ] [ <b>-%i,
--build-top-index</b> ] [ <b>-%I, --search-index</b> ] [
<b>-pN, --priority[=N]</b> ] [ <b>-S, --stay-on-same-dir</b>
] [ <b>-D, --can-go-down</b> ] [ <b>-U, --can-go-up</b> ] [
<b>-%h, --http-10[=0]</b> ] [ <b>-%k, --keep-alive[=0]</b> ]
[ <b>-%z, --disable-compression[=0]</b> ] [ <b>-%B,
--tolerant[=0]</b> ] [ <b>-%s, --updatehack[=0]</b> ] [
<b>-%u, --urlhack[=0]</b> ] [ <b>-%A, --assume</b> ] [
<b>-@iN, --protocol[=N]</b> ] [ <b>-%w, --disable-module</b>
] [ <b>-F, --user-agent</b> ] [ <b>-%R, --referer</b> ] [
<b>-%E, --from</b> ] [ <b>-%F, --footer</b> ] [ <b>-%l,
--language</b> ] [ <b>-%a, --accept</b> ] [ <b>-%X,
--headers</b> ] [ <b>-C, --cache[=N]</b> ] [ <b>-k,
--store-all-in-cache</b> ] [ <b>-%r, --warc</b> ] [ <b>-%d,
--changes[=0]</b> ] [ <b>-%n, --do-not-recatch[=0]</b> ] [
<b>-%v, --display[=N]</b> ] [ <b>-Q, --do-not-log</b> ] [
<b>-q, --quiet</b> ] [ <b>-z, --extra-log</b> ] [ <b>-Z,
--debug-log</b> ] [ <b>-v, --verbose</b> ] [ <b>-f,
--file-log</b> ] [ <b>-f2, --single-log</b> ] [ <b>-I,
--index[=0]</b> ] [ <b>-%i, --build-top-index[=0]</b> ] [
<b>-%I, --search-index[=N]</b> ] [ <b>-pN,
--priority[=N]</b> ] [ <b>-S, --stay-on-same-dir</b> ] [
<b>-D, --can-go-down</b> ] [ <b>-U, --can-go-up</b> ] [
<b>-B, --can-go-up-and-down</b> ] [ <b>-a,
--stay-on-same-address</b> ] [ <b>-d,
--stay-on-same-domain</b> ] [ <b>-l, --stay-on-same-tld</b>
] [ <b>-e, --go-everywhere</b> ] [ <b>-%H,
--debug-headers</b> ] [ <b>-%!,
--disable-security-limits</b> ] [ <b>-V, --userdef-cmd</b> ]
[ <b>-%W, --callback</b> ] [ <b>-y,
--background-on-suspend</b> ] [ <b>-K, --keep-links[=N]</b>
]</p>
--disable-security-limits[=0]</b> ] [ <b>-V,
--userdef-cmd</b> ] [ <b>-%W, --callback</b> ] [ <b>-y,
--background-on-suspend[=0]</b> ] [ <b>-K,
--keep-links[=N]</b> ]</p>
<h2>DESCRIPTION
<a name="DESCRIPTION"></a>
@@ -297,7 +299,7 @@ httrack --continue</b></p>
<p>*use proxy for ftp (f0 don&rsquo;t use)
(--httpproxy-ftp[=N])</p> </td></tr>
(--httpproxy-ftp[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -529,7 +531,7 @@ errors (*R1) (--retries[=N])</p></td></tr>
<p style="margin-top: 1em">*extended parsing, attempt to
parse all links, even in unknown tags or Javascript (%P0
don&rsquo;t use) (--extended-parsing[=N])</p></td></tr>
don&rsquo;t use) (--extended-parsing[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -635,7 +637,7 @@ structure, 1+: see below) (--structure[=N])</p></td></tr>
make any link test but wait for files download to start
instead (experimental) (%N0 don&rsquo;t use, %N1 use for
unknown extensions, * %N2 always use)
(--delayed-type-check)</p> </td></tr>
(--delayed-type-check[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -648,7 +650,7 @@ unknown extensions, * %N2 always use)
<p>cached delayed type check, don&rsquo;t wait for remote
type during updates, to speedup them (%D0 wait, * %D1
don&rsquo;t wait) (--cached-delayed-type-check)</p></td></tr>
don&rsquo;t wait) (--cached-delayed-type-check[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -660,7 +662,7 @@ don&rsquo;t wait) (--cached-delayed-type-check)</p></td></tr>
<p>generate a RFC MIME-encapsulated full-archive (.mht)
(--mime-html)</p> </td></tr>
(--mime-html[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -678,7 +680,7 @@ between pages stay relative; audio and video stay links);
--single-file-max-size N caps each asset (default 10485760
bytes). %M is the better container where a Chromium-family
browser is a given: one archive, no base64 tax on text, a
shared asset stored once (--single-file)</p></td></tr>
shared asset stored once (--single-file[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -740,7 +742,8 @@ absolute URI links, K5 transparent proxy link)
<p>do not include any password for external password
protected websites (%x0 include) (--disable-passwords)</p></td></tr>
protected websites (%x0 include)
(--disable-passwords[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -753,7 +756,7 @@ protected websites (%x0 include) (--disable-passwords)</p></td></tr>
<p>*include query string for local files (useless, for
information purpose only) (%q0 don&rsquo;t include)
(--include-query-string)</p> </td></tr>
(--include-query-string[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -772,13 +775,26 @@ information purpose only) (%q0 don&rsquo;t include)
<td width="4%">
<p>-%C</p></td>
<td width="5%"></td>
<td width="82%">
<p>fold other hostnames of one site onto it
([scheme://]alias[,...]=[scheme://]host) (--host-alias
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-o</p></td>
<td width="5%"></td>
<td width="82%">
<p>*save the server&rsquo;s error pages (404..) (o0 discard
them) (--generate-errors)</p></td></tr>
them) (--generate-errors[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -790,7 +806,7 @@ them) (--generate-errors)</p></td></tr>
<p>*purge old files after update (X0 keep delete)
(--purge-old[=N])</p> </td></tr>
(--purge-old[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -813,7 +829,7 @@ them) (--generate-errors)</p></td></tr>
<td width="82%">
<p>links conversion to UTF-8 (--utf8-conversion)</p></td></tr>
<p>links conversion to UTF-8 (--utf8-conversion[=0])</p></td></tr>
</table>
<h3>Spider options:
@@ -909,7 +925,7 @@ rules)) (--robots[=N])</p></td></tr>
<p>force HTTP/1.0 requests (reduce update features, only
for old servers or proxies) (--http-10)</p></td></tr>
for old servers or proxies) (--http-10[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -922,7 +938,7 @@ for old servers or proxies) (--http-10)</p></td></tr>
<p>use keep-alive if possible, greately reducing latency
for small files and test requests (%k0 don&rsquo;t use)
(--keep-alive)</p> </td></tr>
(--keep-alive[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -934,7 +950,7 @@ for small files and test requests (%k0 don&rsquo;t use)
<p>do not request compressed content (%z0 request)
(--disable-compression)</p> </td></tr>
(--disable-compression[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -946,7 +962,7 @@ for small files and test requests (%k0 don&rsquo;t use)
<p>tolerant requests (accept bogus responses on some
servers, but not standard!) (--tolerant)</p></td></tr>
servers, but not standard!) (--tolerant[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -959,7 +975,7 @@ servers, but not standard!) (--tolerant)</p></td></tr>
<p>update hacks: various hacks to limit re-transfers when
updating (identical size, bogus response..)
(--updatehack)</p> </td></tr>
(--updatehack[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -971,7 +987,7 @@ updating (identical size, bogus response..)
<p>url hacks: various hacks to limit duplicate URLs (strip
//, www.foo.com==foo.com..) (--urlhack)</p></td></tr>
//, www.foo.com==foo.com..) (--urlhack[=0])</p></td></tr>
</table>
<p style="margin-left:18%;">opt out of one url-hack part:
@@ -1186,7 +1202,7 @@ past N bytes, --warc-cdx also writes a sorted CDXJ index,
<p>write hts-changes.json listing what this crawl left new,
changed, unchanged and gone compared to the previous mirror
(--changes)</p> </td></tr>
(--changes[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -1198,7 +1214,7 @@ changed, unchanged and gone compared to the previous mirror
<p>do not re-download locally erased files
(--do-not-recatch)</p> </td></tr>
(--do-not-recatch[=0])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -1210,7 +1226,7 @@ changed, unchanged and gone compared to the previous mirror
<p>display on screen filenames downloaded (in realtime) - *
%v1 short version - %v2 full animation (--display)</p></td></tr>
%v1 short version - %v2 full animation (--display[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -1298,7 +1314,7 @@ changed, unchanged and gone compared to the previous mirror
<td width="82%">
<p>*make an index (I0 don&rsquo;t make) (--index)</p></td></tr>
<p>*make an index (I0 don&rsquo;t make) (--index[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -1310,7 +1326,7 @@ changed, unchanged and gone compared to the previous mirror
<p>make a top index for a project folder (* %i0 don&rsquo;t
make) (--build-top-index)</p></td></tr>
make) (--build-top-index[=0])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
@@ -1322,7 +1338,7 @@ make) (--build-top-index)</p></td></tr>
<p>make an searchable index for this mirror (* %I0
don&rsquo;t make) (--search-index)</p></td></tr>
don&rsquo;t make) (--search-index[=N])</p></td></tr>
</table>
<h3>Expert options:
@@ -1627,7 +1643,7 @@ with -#test=NAME [args])</p></td></tr>
<td width="82%">
<p>wait time (--advanced-wait)</p></td></tr>
<p>wait time (--advanced-wait[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
@@ -1661,7 +1677,7 @@ with -#test=NAME [args])</p></td></tr>
<p style="margin-top: 1em">bypass built-in security limits
aimed to avoid bandwidth abuses (bandwidth, simultaneous
connections) (--disable-security-limits)</p></td></tr>
connections) (--disable-security-limits[=0])</p></td></tr>
</table>
<p style="margin-left:18%;">IMPORTANT NOTE: DANGEROUS
@@ -1711,7 +1727,7 @@ myfoo.so[,myparameters]) (--callback &lt;param&gt;)</p></td></tr>
<p>go to background when suspended (y0 don&rsquo;t)
(--background-on-suspend)</p> </td></tr>
(--background-on-suspend[=0])</p> </td></tr>
</table>
<h3>Details: Option N

Binary file not shown.

After

Width:  |  Height:  |  Size: 842 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 25 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 103 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 83 KiB

After

Width:  |  Height:  |  Size: 97 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 95 KiB

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 38 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.8 KiB

After

Width:  |  Height:  |  Size: 8.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.1 KiB

After

Width:  |  Height:  |  Size: 7.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.3 KiB

After

Width:  |  Height:  |  Size: 7.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.9 KiB

After

Width:  |  Height:  |  Size: 9.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.8 KiB

After

Width:  |  Height:  |  Size: 7.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.7 KiB

After

Width:  |  Height:  |  Size: 7.7 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.8 KiB

After

Width:  |  Height:  |  Size: 8.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.6 KiB

After

Width:  |  Height:  |  Size: 8.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.4 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.2 KiB

After

Width:  |  Height:  |  Size: 5.3 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.3 KiB

After

Width:  |  Height:  |  Size: 9.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.5 KiB

After

Width:  |  Height:  |  Size: 9.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 KiB

After

Width:  |  Height:  |  Size: 13 KiB

View File

@@ -10,12 +10,14 @@
</head>
<body>
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap solo">
<main id="main">
@@ -108,7 +110,12 @@ not own.</p>
</main>
</div>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- doc-chrome:footer -->
<footer>
<small>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</small>
<a href="https://endsoftwarepatents.org/innovating-without-patents/" target="_blank" rel="noopener" title="This site is innovating without patents!"><img src="images/esp_chicklet.png" width="91" height="17" alt="End Software Patents"></a>
</footer>
<!-- /doc-chrome:footer -->
</body>
</html>

Some files were not shown because too many files have changed in this diff Show More