145 Commits

Author SHA1 Message Date
Xavier Roche
77d03052a6 Release 3.49.20
Two engine fixes are the substance of this one. An off-site link whose name looked like a file was parsed as a page, and then spent the referring page's whole depth budget on the foreign site. And a mirror that was asked to quit while an FTP worker was still receiving printed its exit notice, then waited forever on that slot. The rest of the range is the documentation refresh, the WebHTTrack language menu, and test and CI work.

VERSION_INFO goes to 3:13:0, revision only: no installed struct changed shape, no export came or went. debian/changelog picks up the League Spartan webfont the docs now bundle, and Standards-Version 4.7.4 still matches current policy.
2026-08-10 21:11:46 +00:00
Xavier Roche
628fc70727 The Windows suite runs none of the tests covering the libraries it depends on (#1131)
* The Windows suite runs none of the tests covering the libraries it depends on

ci-windows-suite.sh selects its tests by topic word, and no glob matches the
harness self-tests: 254, 256 and 257 (testlib.sh) and 258 (crawllib.sh) never
run there, though every test the job does run now sources one of those
libraries. Two categories pull them in; 260 keeps its name clear of both until
#1126 explains why a deliberately failing crawl wedges the Win32 leg.

172 stubs one test per category, so it gains two and its tally moves with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* 258's httrack shim never reaches PATH on Windows, and 172 cannot see 260 come back

Two gaps the review found in the tests this PR newly selects there.

The argv shim in 258 lives under TMPDIR, which the Windows job sets to a
cygpath -m drive-letter path, so bash reads "D:/a/_temp/..." on PATH as two
entries and the real engine answers every case. testlib gains posixpath(),
nativepath's inverse, which 226 was already open-coding; 258 also proves the
shim is reachable rather than assuming it, since an unreachable one silently
tests the wrong binary.

172 could not kill a crawllib glob widened to *_crawl-*, which drops 258 and
selects 260: in the stub tree the tally happened to hold. A decoy named the way
260 is, selected by nothing, plus a duplicate-selection check on the progress
log, close both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The decoy and duplicate checks report after the tally that hides them

Every wrong selection also moves ran=, so the tally fired first and the two
named checks never got to say what went wrong. Reordered ahead of it, and the
decoy failure drops the whole suite output it was quoting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* 256 stops at the first signal that breaks, so three of the four go unmeasured

Windows reds on the SIGINT leg: the subject sits out the whole sleep and exits 0,
so the trap never ran. The loop ends there and says nothing about HUP or QUIT,
which is the half needed to tell a missing trap from a signal MSYS will not
deliver to a bash of its own. It now reports every leg with the time each took.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* MSYS never delivers INT or QUIT to a bash its own kill sent them to

Measured on both Windows runners now that 256 reports every leg: the SIGINT and
SIGQUIT subjects sit out the whole 30s sleep and exit 0, where TERM and HUP
return at once with the trap's status. Never delivered, rather than delivered
and ignored, which is why 243 and 255 already skip there. Those two legs are cut
on Windows; the other four sections of 256 keep running.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The 260 exclusion lives in a filename, where a rename would ship the wedge

Review found the #1126 exclusion resting on prose and a glob miss: rename 260
into one of the topic words and the Win32 leg wedges again, with nothing to say
why. The driver now names the file and stops when it is gone, so a rename reds
in the first second rather than after 45 minutes on the runner. 172 drives that
by renaming the staged copy.

Also from the review: 256 tests PIPE, the fifth signal in the trap list it had
never sent, and takes its signal set from an array rather than a split string;
258 keeps $shim native and converts only the PATH entry; two comments that had
grown by accretion are split back to one point each, and 258's overstated the
shim guard, which buys a clearer failure rather than the only one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Sending SIGPIPE was scope creep, and four Linux legs disagreed with my box

The fifth signal in cleanup_push's trap had never been sent, so I added it. On
the runners the subject sat out its whole sleep and exited 0, where bash 5.2.37
here returns at once with the trap's status: a self-sent SIGPIPE reaches a
non-interactive bash differently depending on the build. Reverted to the four
signals this test has always sent, and the gap is filed rather than guessed at.

Refs #1136

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:36:19 +00:00
Xavier Roche
7ff60e9de7 Reshoot the WebHTTrack guide screenshots without the loopback address (#1135)
The seven screens that showed the crawled address or the mirror folder now
read www.example.com.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:23:48 +00:00
Xavier Roche
4588831dcd The doc screenshots advertise 127.0.0.1 and a random port (#1133)
The crawled address is on screen in the wizard panes and names the mirror
folder, so every guide screenshot showing it, and the software-centre image,
carried a loopback artifact.

--site-host and --site-port let the walk address its own server by name; the
workflow maps www.example.com and lowers ip_unprivileged_port_start so it can
answer on 80. A local run keeps the old behaviour.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:44:26 +00:00
Xavier Roche
adbc46a5b8 The software-centre screenshot advertises a loopback address (#1132)
The wizard pane it pointed at has one piece of user content, the URL box, and
the walk crawls a throwaway loopback server, so the hero image read
http://127.0.0.1:43925/. The scan-rules pane shows real capability and no
address. Reverting to a wizard pane wants the walk to crawl a named host.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:40:30 +00:00
Xavier Roche
249b96b975 Advertise WebHTTrack with a screenshot from this decade (#1130)
The metainfo pointed at a 300x238 JPEG of the UI in a Mozilla window on
Windows 98 chrome, unchanged since 3.30.1. It points at a guide asset now, so
future reshoots refresh the software-centre listing on their own.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:54:24 +02:00
Xavier Roche
d8402d7ad2 local-crawl.sh keeps its own copy of the server launch and the crawl backstops (#1123)
* local-crawl.sh keeps its own copy of the server launch and the crawl backstops

crawllib.sh was extracted from the tests that drive local-server.py themselves,
leaving local-crawl.sh -- the launcher the other 62 crawl tests go through --
with a second copy of the launch block and of the --max-time/watchdog pair. It
now sources the library: the server comes up through local_server_start, the two
backstop constants live in one place, and the four near-identical httrack
invocations collapse into run_pass/require_pass.

The engine is still backgrounded here rather than run through local_crawl, which
would put it in its own process group where the suite watchdog's stack dump
cannot reach it (105).

--tls earns its place in the library now that a caller wants it, and 258 covers
it by fetching over TLS rather than reading the scheme back off BASEURL.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* 14_local-https asserts nothing about the scheme it crawled

Its audit (5 files, no errors, the page is there) reads the same whether the
crawl was TLS or plain, so a harness that lost --tls would turn it into a
duplicate of the plain-http tests. The mutation matrix for local-crawl.sh's
--tls passthrough survived on exactly that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the comments added by the crawllib adoption

Signed-off-by: Xavier Roche <roche@httrack.com>

* Nothing tests that local-crawl.sh fails loudly

The 62 tests behind it read nothing but its exit status, so a harness that
swallowed a failed crawl would report every one of them green. 260 covers the
three ways a run must end badly: a crawl the engine refuses, a later pass that
fails while the first pass's mirror stands, and an audit that does not hold.
The middle one is the case the others cannot see, since a missing mirror ends
the run at the host-root check whether or not the pass was required to succeed.

run_pass now returns the crawl's status instead of leaving it in a global, and
require_pass runs the pass itself, so a call site names its log and label once
and cannot check the previous pass's result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the new harness test out of the Windows suite's topic globs

Named 260_local-*, it was selected by ci-windows-suite.sh, and the Win32 leg
then died at its step timeout twice on the same commit while x64 passed. It is
the first Windows-selected test that deliberately fails a crawl, and the suite
kept neither a log nor an artifact, so the cause is unknown; #1126 records what
was measured. The other harness self-tests (254, 256-258) sit outside those
globs too, and widening the Windows selection is its own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Do not name the Windows driver in the new test's comment

226 greps every .test for that filename and demands the credential-scrub line
when it finds one, since sourcing the driver starts an interpreter that can post
statuses. A comment cannot be told from a source line by that grep, so the
mention alone reddened every leg. 226 skips on this host, so only CI saw it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 17:22:47 +00:00
Xavier Roche
b268caa220 The guide still explains the language dropdown that #1086 removed (#1125)
The web note told readers the menu opens blank and that its first entry means
leave the interface alone. That placeholder option is gone; the menu now shows
the language in force and picking another reloads the page in it.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 17:21:23 +00:00
Xavier Roche
fb59dd6aba Reshoot the WebHTTrack screenshots after the masthead fix (#1129)
Every screen carries the masthead, so ef100363 dated all eighteen.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 17:18:19 +00:00
Xavier Roche
ef100363a7 The WebHTTrack tagline bar sits off the wordmark's baseline (#1127)
#927 and #1111 cancelled the SVG's built-in whitespace in html/doc.css, so the
doc pages tightened and the WebHTTrack UI, which renders the same masthead from
its own stylesheet, kept the gap.

Measured with getBoundingClientRect in the same browser: bar top minus image
bottom was 0 on the server pages against -1.09 on a doc page, and is now -1.09
on both.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 16:28:08 +00:00
Xavier Roche
c3e1a202c1 The GUI guide's screenshots are stale, and the palette step greys out their backgrounds (#1124)
* Reshoot the WebHTTrack screenshots in the GUI guide

The set was captured at 3.49-14, before the masthead wordmark and the rings
background became SVGs (#916, #910) and before the welcome page learned to
mark the language in force (#1086).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Quantize the guide images by median cut, not max coverage

Max coverage spreads the 256 entries over the colour cube regardless of how
many pixels each region costs, so the large flat panel and rings backgrounds
came out grey or tinted. Mean channel error drops from 5.3 to 0.5, for 650K.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:38:05 +00:00
Xavier Roche
f55f68c94e The documentation pages ask for Trebuchet before the system font, and have no display face (#1122)
doc.css put "Trebuchet MS" and Verdana ahead of system-ui in the body stack, so only Linux ever reached the platform font: Windows and macOS both stopped at Trebuchet. This reorders to system-ui first, keeps Trebuchet and Verdana behind it, and puts League Spartan 700 on h1 and h2. It goes on headings alone because it is a geometric display face, and it reads worse than the system font over the long prose of the reference pages. The website's stylesheet now does the same.

The font is self-hosted rather than linked from Google Fonts, since the pages are read from an installed directory and over file:// as often as over http. It is the latin subset at weight 700, 12888 bytes, with its OFL beside it, and `doc.css` reaches it by a relative URL that CSS resolves against the stylesheet rather than the document. New test 259 walks every `url()` in `html/*.css` and fails when the target is missing from the installed tree or the tarball, which test 225 cannot do: it drops the htmldir region on purpose, so either install rule could have gone missing with nothing going red.

`debian/copyright` gains an OFL-1.1 stanza, text inlined because OFL-1.1 is not in common-licenses. It previously declared every file GPL-3+, which is wrong for a bundled font. Debian ships `fonts-league-spartan`, but a `@font-face` cannot portably reach a fontconfig family and the tarball and macOS builds have no such package, so the bundled copy stays. Upstream declares no Reserved Font Name, which is what lets the subset keep the family name.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 16:10:25 +02:00
Xavier Roche
749de004c7 Engine self-tests assert a substring, so the value underneath can change unseen (#1121)
* tests: pin the engine self-test assertions that only matched a substring

Five self-tests asserted their result with a substring grep, so a change
in the value underneath went unseen: moving the advertised
Accept-Encoding from identity;q=0.9 to q=0.8 leaves "acceptencoding
self-test OK" in the output and the old test green. assert_selftest pins
the line and the exit status and prints want beside got.

Two more compared the whole line already but through a bare
test ... || exit 1, which reds without naming what differed.

The greps were also pipes into grep -q, which AGENTS.md forbids; at these
output sizes the producer finishes before grep exits, so removing them is
conformance rather than a fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: shorten the watchdog capture comment

Two lines, and "can SIGPIPE" rather than a claim of a live bug: at the
param block's size sed always finishes writing, so the trap is latent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:18:35 +00:00
Xavier Roche
09e3e1f62b The tests driving the local server each rebuild its launch, and most crawl with no time cap (#1120)
* tests: give the local-server cluster a shared launch and crawl helper

Twenty-two tests drive tests/local-server.py themselves rather than
through local-crawl.sh, and each rebuilds the same launch: resolve the
script, find a python, background it, poll stdout for the ephemeral port,
register the reaping, build the base URL. Only three of their crawls
carried --max-time and none a watchdog, so a wedge here ran until CI
cancelled the job at 45 minutes, taking the log with it.

tests/crawllib.sh adds local_server_start and local_crawl, covered by
258_crawllib.test from a subject process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: review fixes on the local-server helpers

Drop local_crawl's --deadline, which no test outside its own used;
CRAWL_DEADLINE already drives the watchdog for local-crawl.sh. Truncate
the crawl log rather than append, matching every redirect it replaced.
Skip the reap when the server is already gone: 240 stops its own server
mid-test, and stop_server answers an unresolvable pid on Windows by
killing every python.exe.

235 keeps its own run_with_timeout: it already had a watchdog, and
local_crawl's failure would preempt the four messages it prints.

258 pins the whole argv per call and truncates it first, so an
assertion cannot pass on the previous crawl's arguments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: let a test disarm its server's teardown

The cleanup registered by local_server_start expands its arguments at
push time, so a test that stops its own server mid-run leaves the frame
holding a dead pid. Signalling a number the system has since recycled
kills an unrelated process, then stalls reap_bounded for its whole grace
period. Track the live servers in SRV_PIDS and register the slot, so
local_server_stop can empty it. 240 is the caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:37:19 +02:00
Xavier Roche
4364e6c93e The engine self-test scripts each carry their own teardown, invocation and silent comparisons (#1118)
* tests: an assert_* vocabulary and an engine self-test façade

assert_eq/assert_match/assert_no_match/assert_file/assert_dir/assert_min_bytes
print what they wanted beside what arrived, and assert_selftest wraps the
httrack -O /dev/null -#test=NAME invocation the tier spells out by hand,
checking the exit status the old `test "$(...)" = ...` form discarded.
httrack_path resolves the binary the ten tests that cd away resolve themselves.

257 exercises each one from a subject process, since an assertion that fires
exits: every helper has a case that must hold and one that must fire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: the engine self-test tier joins the shared library

The 01_engine-* / 01_zlib-* tier sourced nothing: 50 hand-written trap pairs,
a repeated absolute-binary incantation, and comparisons ending in a bare
`|| exit 1`. Those become cleanup_push, httrack_path and the assert_* helpers.

Two latent traps go with them: the mime helper piped into head -1, which
SIGPIPEs the engine under pipefail, and half the signal traps only removed the
temp dir on EXIT.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: review fixes on the assertion vocabulary

Drop assert_no_match, assert_dir and assert_min_bytes: 257 was their only
caller, and a dead helper is what PR #1113 deferred them to avoid. The
duplication they would serve (five copies of size_of, the grep -q <<< cluster)
lives in the tests PR 4 covers.

assert_file and assert_match gain real callers here instead: four artifact
checks and the renameover/cookieimport/direnum output matches.

The -O /dev/null comment claimed the engine refuses to start without an output
path. It does not: htscoremain.c dispatches -#test ahead of the usage gate.

hdr() in 01_engine-header.test kept its capture in an assignment, where errexit
can still see a failing engine, and httrack_path now names a failed cd rather
than returning a bare /httrack.

257 grows the cases that kill a prefix compare, a case-folding matcher, a
dropped label, and a status check satisfied by exit 127.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: retry the proxytrack port pick instead of reding on a lost race

freeport hands out a port it has already closed, so between the pick and
proxytrack's own bind another process can take it: 79 lost that race on the
Windows x64 leg. Factor the pick, the listen wait and a three-attempt retry
into testlib.sh, and have the four proxytrack tests launch through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: give each proxytrack attempt its own log, and fold the port pickers

Review of the retry found the .done marker the pty drainer writes by path
escapes the fresh inode, so a killed attempt could satisfy 153's wait_drained
and let its console-leak check read a half-copied log. Each attempt now writes
its own $ptlog. The two ports come from one freeport call, which cannot hand
back the same number twice, and the three copies of that bind-and-close trick
become one helper. A retry says so on stderr, a proxytrack that dies announcing
neither outcome is named as such, and a launcher that leaves no pid is caught.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 07:59:28 +00:00
Xavier Roche
35319c05be SIGTERM never reaches teardown while an FTP worker is receiving (#1106)
`sig_finish` raises `state.exit_xh` on SIGTERM, but the mirror's wait loop in `hts_mirror_wait_for_next_file` never polled it: it leaves only when the link is done, when `back_checkmirror` fails, or when `hts_loop_tick` says so. Nothing there ends a live FTP worker, and the call that does, `ftp_stop_workers()` from `back_delete_all` and `hts_free_opt`, sits past the loop in teardown. So "Exit requested to engine" printed and the crawl then waited forever on the receiving slot. Polling `exit_xh` there and returning the way the two neighbouring abort checks already do lets the outer loop's own `exit_xh` test end the crawl and reach teardown. The poll fires on any nonzero value, so a fatal `exit_xh = -1` raised while a slot is live no longer wedges either.

`hts_mirror_process_user_interaction` needed the same poll. The loop above is only reached while a socket is free, so with every slot busy the mirror parks one frame higher, in the wait for a free socket, whose guards were the identical pair. Both polls are load-bearing: with only the inner one, three stalled links against two sockets still hang; with only the outer one, the single-link case does.

`^C` is untouched. It sets `state.stop`, not `exit_xh`, and neither poll reads it, so a first `^C` still finishes pending transfers and a second still exits immediately. That half of #1096 is a separate bug on a separate path, `wait_socket_receive` in `htsftp.c` never reading `state.stop`, and it is not fixed here, so the issue stays open. Nothing in the code makes letting a receiving slot finish a deliberate policy, and a stalled HTTP body does not behave like the FTP one: measured on master, SIGINT ends a stalled HTTP transfer in 0.1s and a stalled FTP one only after `--timeout` expires. The HTTP promptness is itself accidental, #1110.

One consequence worth naming. The `exit_xh` route falls through the whole completion sequence, `--purge-old` included, so a SIGTERM during an `--update` run can purge files it never re-fetched. That is pre-existing on that route, but this fix reaches it where the engine used to hang, and a user who reached for SIGKILL kept their mirror. Filed as #1109.

`tests/255_local-ftp-sigterm.test` signals a crawl parked on a stalled FTP body with `--timeout=0 --max-time=0`, so only the fix can end it. It pins the exit path rather than just the exit: reverting the hunk hangs, and returning the caller's error code or the already-done code leaves the wait too but loses the cache or drops the link silently, which the log assertions catch. A saturated leg covers the outer poll, and controls check that a single `^C` still completes an in-flight transfer, that the `^C` actually reached `sig_leave`, and that an unsignalled crawl is unaffected. It skips on Windows, where MSYS cannot deliver the signal to a native exe.

Refs #1096

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 07:40:12 +00:00
Xavier Roche
e4fbe856e5 The chrome check cannot see the masthead on two of the fifteen pages (#1116)
guide.html and index.html build their own sidebar, so they carried no markers at
all and the masthead above it drifted freely: changing the tagline in guide.html
left --check reporting 0 out of sync, where the same edit in faq.html was caught.

The masthead is byte-identical on all fifteen, so it becomes its own region that
a page can take without the generated navigation. Both now follow the tool: a
canary in MASTHEAD reaches 15 pages, where before it reached 13.

Closes #1115

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 08:19:41 +02:00
Xavier Roche
4f49eec86b The docs and the site disagree on what HTTrack is called (#1114)
The site says "Free software offline browser", these pages said "Open Source
offline browser", and the French tagline has always been "Aspirateur de sites
web libre", so one language already made the claim the other did not. Xavier
picked the site's wording for the web surfaces.

The lang/ catalogs keep theirs: they are the GUI, the English value is their
join key, and 17 carry a real translation that changing it would orphan.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 07:41:45 +02:00
Xavier Roche
f8ed3e771f Every test script carries its own copy of fail(), skip() and the teardown traps (#1113)
* tests: split the wedge forensics out of testlib.sh into proclib.sh

testlib.sh is sourced by 74 tests, but 235 of its lines are the process
forensics only the timeout driver and the Windows suite ever call.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* tests: fold the copied fail(), skip() and teardown traps into testlib.sh

62 copies of fail(), 7 of skip() and ~110 hand-written trap pairs go away.
cleanup_push installs the signal half two thirds of them never had, with
errexit off (#773); it takes a command and its arguments, not a shell
snippet, so it needs no eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: fix the review findings on cleanup_push and the proclib split

Re-sourcing testlib.sh reset the cleanup stack, so 172 leaked its temp tree;
the stack is now self-preserving and 256 covers it. Both argv slices carry
the [@]+ guard an empty frame needs on bash 3.2. The engine matchers move to
proclib.sh beside their only consumers, dropping the SC2034 pair that hid the
cross-file reference. 100 and 105 register incrementally instead of rebuilding
an EXIT trap, which is what the LIFO stack is for; 103's scanner now inspects
the libraries, whose traps its errexit gate had stopped reaching. 250's dead
poll stub waits for the child to announce, closing the other half of #1092.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: keep the default $top_srcdir relative

100_local-purge-longpath.test hands it straight to python.exe, which cannot
resolve the absolute MSYS path the library had started defaulting to, so the
fixture server never came up and the test failed where it used to skip. The
Windows suite exports no value of its own, so the default is what it runs on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 21:58:42 +02:00
Xavier Roche
4231a141d3 The documentation footer fails AA contrast in light mode (#1112)
#878 gave the footer color: #eef, which is 3.56:1 on the --field it sits on,
below the 4.5:1 AA wants for text this size. Dropping it inherits body's --ink
at 4.62:1; no lighter colour clears the bar on that field, and --ink-soft is
worse there at 2.38:1. The dark override already passes at 7.76:1.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 18:13:54 +02:00
Xavier Roche
314ee45350 The tagline bar still sits off the wordmark's baseline (#1111)
#927 gave the wordmark negative margins to put the black bar back against it,
but left the .4rem that pushes the bar down, so about 5.3px of gap survived the
fix meant to close it. Dropping it leaves the collapsed sibling margin at the
-1.1px #927 measured for the SVG's overshoot below the baseline.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 18:07:45 +02:00
Xavier Roche
54d93200a4 Two suite tests fail on the hppa, powerpc and ppc64 Debian buildds (#1108)
* Fix two suite tests that fail on the hppa, powerpc and ppc64 buildds

246_engine-connect-stop needs its connects to TEST-NET-1 still in flight, but
probed that the instant connect() returned; a network that rejects 192.0.2.1
rather than dropping it answers milliseconds later, and back_wait() then ends
slots the control asserts it left alone. Settle before re-reading the sockets.

183_altstack-worker pinned the worker's fault to SIGSEGV, which hppa does not
report for a spent thread stack. Accept any fault the handler covers, and print
the report when it is none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the two comments this change adds

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Match the worker's fault by signal name, not number

SIGBUS is 7 on x86 and 10 on hppa, so a numeric set cannot be right on
both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Gate the dead-host fixtures on a network that refuses TEST-NET-1

tools/hostile-net.sh runs a command in a netns whose peer owns 192.0.2.1 and
listens on nothing, behind 30ms of netem so the RST lands after connect()
returns rather than before it. That is the powerpc and ppc64 buildds, and
without the settle probe 246 fails under it with the buildds' exact log.

The script refuses to run when the address is dropped or answered, so a leg
that reproduces neither cannot pass silently. Widen SHELL_SCRIPTS to tools/*.sh
while here: it was an explicit list that had already lost mk-sbuild-chroot.sh.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cut the comments and rules this change adds

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the hostile-network leg skipped on the settle probe

Without it a netem that stops delaying leaves the leg green while 246 skips on
the older check, so the branch the leg guards never runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 16:59:11 +02:00
Xavier Roche
6e98e33495 The htsserver tests copy five helpers per file, and the copies have drifted (#1105)
* The htsserver tests copy five helpers per file, and the copies have drifted

Each of the ~20 tests driving htsserver carried its own freeport, start,
srvpid, portof and request, plus an inline python socket client: eleven
copies of freeport in three variants, eight of srvpid in six. Two of the
divergences were live bugs. Five copies read the announced pid with
"sed ... | head -1", which SIGPIPEs the producer so that under pipefail a
spurious 141 becomes the test's verdict; 91_webhttrack-directory.test wrote
a firstline() to dodge that and nobody copied it. And the socket timeout had
drifted to 15s in one copy and 30s in four, with only the 15s copy catching
socket.timeout and reporting, so the other four wedge where that one fails
cleanly.

tests/webhttracklib.sh and tests/httpclient.py now own picking the port, the
background launch under an ignored SIGTERM, the wait for the URL=
announcement, the pid the server reports (Windows announces none) and the
reaping of both. portof is gone entirely, since the library picks the port
and nothing has to parse it back out of the URL.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqzMu5gKg9FeyhfNrRgupd
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix what the review found in the new htsserver test library

The --write-limit path backgrounded a pipeline, so $! was the reader rather
than the server. A start that never announced then killed only the reader,
orphaned the server on its port, and parked the reaping wait on the whole
job, which left the test immune to SIGTERM. Process substitution instead, so
the background pid is the server.

Centralising the launch also moved two python forks and a blocking wait
inside the one-second window 223 needs to mint two session ids in the same
second: under 8x load the window grew from ~0.28s to ~0.70s and the test
began skipping, which reads green while its assertion stops running.
htsserver_start takes --port and htsserver_freeport hands back several ports
bound together, so 223 picks both before the clock starts.

Also: htsserver_assert_reaped now covers the pid the server announces, and
refuses an empty list rather than passing on one; and firstline drops the
$'..' inside a parameter expansion that Apple's bash 3.2 cannot parse.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqzMu5gKg9FeyhfNrRgupd
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 16:55:17 +02:00
Xavier Roche
67c4668a3b A runner-image apt 403 reds a whole job, and the FTP active-mode branch never compiles (#1103)
Two CI gaps, in one PR because both land in `.github/workflows` and would collide.

`apt-get update` runs in 19 jobs under `set -euo pipefail`, so a 403 from `packages.microsoft.com`, a source the runner image ships and we never install from, takes a job down before a single package arrives. It reads like a real failure too: on #1093 the leg that died was named `msan (MemorySanitizer, clang)`. The update is now advisory in the 16 jobs that run on a GitHub runner, and `apt-get install` still fails loudly if a package we do need is missing. The three container jobs keep the strict form, since every source in a `ubuntu:devel` or `debian:sid` image is one we put there.

`FTP_PASV` is 1 in every build, so the `#if !FTP_PASV` active-mode branch is dead at runtime and never reaches a compiler either, which left #1087 re-verifying it by hand. The `build` job now recompiles `htsftp.c` alone with `-DFTP_PASV=0` and fails on any warning, which is why the define grew an `#ifndef` guard. It goes through make, so the object gets the flags the real one gets, and is dropped afterwards so `make check` links a normal library. The control is an undefined `accept` in that object: only the active branch calls it, so a probe that quietly rebuilt passive mode fails instead of passing. Two mutants kill the probe, a stray warning inside the branch and a revert of the `#ifndef`.

Closes #1091
Closes #1097


Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:42:18 +00:00
Xavier Roche
e6015e29bb Bump src/coucal so an in-tree build stops leaving the submodule dirty (#1102)
coucal's .gitignore now covers the libtool and automake droppings our
in-tree build drops into src/coucal (*.lo, *.la, .libs/, .deps/,
.dirstamp), same class as #905. Also picks up a codeql-action pin from
dependabot. No code change, no ABI impact.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:00:06 +00:00
Xavier Roche
ba53bca32e A foreign-host link that looks like a file is scanned as a page (#1093)
A link to `https://www.dropbox.com/s/xxx/distorted.jpg` looks like a file, so the engine accepts it even though it points off-site. Dropbox answers with `text/html`, and the preview page is then parsed with the referring page's whole depth budget. Every link on it is same-host from there, so one link mirrors the entire foreign site: 372 files in the reproduction against the one page it was entitled to, and wikipedia `File:` pages do the same. The new test drives two loopback servers and pins the mirror of the foreign one, which held 3 files under `-n` and 7 under `+*.jpg` before the fix.

At the parse gate such a link's depth now counts as 0, so the page is still fetched and stored, through the store-without-scan path a depth limit already uses, but nothing on it is followed. The gate is `is_html_mime_type()` rather than the wider hypertext set, or a foreign stylesheet would stop being parsed and its images would go missing; the one cost is a typeless response, which that macro reads as HTML and which now picks up a stray `.readme`. Hosts are compared through `jump_identification_const()`, since `adr` carries the scheme and any `user:pw@` and a raw compare made a site foreign to itself. Nothing about filters enters into it: the only exemption is being a seed, which the user asked for by name. Also fixes `hts_testlinksize`, which built its filter URLs with the separator on the wrong buffer.

Closes #121
Closes #1095


Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 12:45:23 +00:00
Xavier Roche
5b59cdb159 Run 230_local-ftp-userpass on Windows again (#1100)
The skip landed on a 3/3 wedge observed before #1039, #1055, #1071 and #1081 were in the tree. With all four merged, the test was re-run three times on current master with the gate and the `expected_skips` entry removed: six green legs, x64 and Win32, every run.

The runs were sequential on purpose. `windows-build`'s concurrency group is per-ref with `cancel-in-progress`, and overlapping Windows jobs are themselves a known way to lose a runner, so three parallel runs would have risked manufacturing the failure under test.

Detail and run IDs are on #1038.

Closes #1038

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:14:17 +02:00
Xavier Roche
c520d81f6b FTP active mode waits for the data connection with no timeout and no stop check (#1087)
Active mode waited for the server's data connection in a blocking accept(), so it ignored --timeout, --max-time and the stop flag, the way the two connects did before #1071. It now polls on the same 100ms tick, re-reading the stop flags and what is left of --max-time each pass, and puts the accepted socket back into blocking mode because BSD hands it the listener's non-blocking flag.

The branch is compiled out today (FTP_PASV is 1), so there is no runtime path to test. I compiled htsftp.c with FTP_PASV forced to 0 instead: clean, no warnings, and the object's undefined accept symbol confirms the branch was really built. The default is untouched. No build in CI covers that branch, which #1091 tracks, so this check was a one-off by hand.

A review found a second bug in the same code, folded in here: the active-mode listener and the data socket were both closed with a raw close() that left the descriptor in the variable, so the _CHECK_HALT_FTP after the transfer loop closed them again through _HALT_FTP. The data socket one is on the live PASV path. Both now reset to INVALID_SOCKET the way the PASV path already does around its own deletesoc calls. 253_local-ftp-close-once.test cancels a link mid-transfer through the engine's own per-link cancel, the one WebHTTrack's stop button pushes, and an LD_PRELOAD close() interposer reds the test on any EBADF. Reverting the data socket to its raw close gives exactly one.

Closes #1074
Closes #1090

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:14:13 +02:00
Xavier Roche
0748fa44e9 StringRoomTotal doubles its capacity without saturating (#1088)
StringRoomTotal doubled its capacity with no cap, so once past SIZE_MAX/2 the product wrapped to a capacity below the one already held and no further doubling climbed back out. Nothing reaches that today, since the realloc sits inside the loop and an outsized request dies in STRING_OOM several doublings earlier, but the tree had one grower that saturates (TypedArrayEnsureRoom, fixed in #1056) and one that did not. Past the halfway point it now asks for the requested total rather than doubling.

Clamping to SIZE_MAX, the obvious mirror of TypedArrayEnsureRoom, hands GCC a constant `realloc(p, SIZE_MAX)` and lights `-Walloc-size-larger-than` at every String growth site: 24 new warnings in proxytrack alone. The requested total is a runtime value, so there is nothing to fold. #1062 also asked for the realloct/freet wrappers; I left those alone, since they are plain aliases for realloc/free, while `freet` is a statement macro that assigns to its argument, htsstrings.h is installed, and pulling in htssafe.h leaks config.h into a header documented as standalone.

Two stringoom modes cover it. `saturate` forces the capacity past the halfway point, which nothing reaches through the API, and pins the size the allocator is asked for; it dies on its first allocation and so never sees a second step. `growloop` runs the whole loop against an allocator that keeps saying yes, under a step budget, so a clamp that fails to raise the capacity fails instead of spinning. Both go red on the old doubling and on a clamp that hands back the current capacity.

One unrelated fix rides along. `tests/250_timeout-poll-nofork.test` checked that the wedged child was gone with a single `kill -0` that nothing had waited for, and that race reddened this PR's own `build (no python3, Debian buildd)` leg. It now waits for the pid, bounded well under the child's own self-limit so a child expiring on its budget cannot pass for a successful kill.

Closes #1062
Closes #1092

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:14:01 +02:00
Xavier Roche
0ced83ccfc The heartbeat's sleep can inherit the caller's stdout, and no test sees it (#1089)
The `>/dev/null 2>&1` on `ci_suite_heartbeat`'s `sleep` is load-bearing, not noise suppression. The suite's EXIT trap kills the heartbeat shell and not the sleep it forked, so an orphan still holding the step's stdout keeps its reader waiting for EOF, which delays or loses the step the watchdog was there to protect.

171 is blind to it: its `sleep` is a shell function on a virtual clock, so nothing is forked and no descriptor is inherited, and the test stays green with the redirection deleted. The new leg goes in 172, which already drives the driver on real sleeps. Start the heartbeat on a 15s tick with its stdout in a pipe, let it enter the first sleep, kill it the way the trap does, then time how long the reader waits for EOF: 0s today, 13s with the redirection removed. Two things keep a prompt EOF from passing for the wrong reason: a control leg runs first and forks a child that does keep the pipe, and the leg samples the heartbeat pid before it signals, since one that died before forking its sleep would look identical.

Costs 172 about 8 seconds.

Closes #949

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:13:57 +02:00
Xavier Roche
171ffb0754 Tests 13 and 52 mint a cert the tree already ships (#1101)
* Tests 13 and 52 mint a cert the tree already ships (#1099)

Both minted a two-day self-signed cert with openssl req on every run, which
is why both carried openssl in their skip gate. make_tls_pem now joins the
tracked tests/server.crt and tests/server.key, the pair local-crawl.sh
already hands every --tls crawl, so the gate and the keygen go away.

Closes #1099

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the fixture against load_cert_chain, not against the helper

The two byte comparisons only restated what make_tls_pem cats: a
server.key that does not match server.crt passed 254 and reddened 13
and 52. Load the pem the way the python servers do, and trim the
comments to one line each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the write in the fixture helper's failure message

A missing destination directory fails the redirect, not the read, and
the old wording blamed the fixture for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 10:42:24 +00:00
Xavier Roche
e12fe63246 A failing cert fixture leaves tests 13 and 52 with an empty log (#1098)
`13_crawl_proxy_https` and `52_local-socks5` both built their self-signed cert with an `openssl req` whose output went to `/dev/null 2>&1`. Under `set -e` a failure there exits 1 before either test's first `echo`. Automake's log then kept only the `FAIL ... (exit status: 1)` line, with no clue what failed. It happened twice on a macOS 26.5 box.

The fixture was identical in both tests, so it moves to `testlib.sh` as `make_tls_pem`, which keeps openssl's output and prints it when the command fails.

`254_testlib-tls-pem.test` checks the helper from both sides. A working run has to produce a pem holding the key and then the certificate, in the order the callers' `load_cert_chain()` reads. An `openssl` shimmed onto PATH that exits 3 has to make the helper exit 1 with that openssl's reason on stderr; the shim still writes the files it was asked for, so nothing but the helper's own exit can satisfy the status check. Each of those assertions was confirmed against a mutant that breaks it.

Leaving `openssl` out of the picture entirely looks better still: `tests/server.crt` and `tests/server.key` are already tracked and distributed, with a SAN and a 2056 expiry. Moving 13 and 52 onto them would drop `openssl` from their skip gates, but it changes where those two tests run, so it belongs in its own change.

Closes #1094


Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 11:48:09 +02:00
Xavier Roche
ff266a0e61 The macOS Release asset name does not say it is arm64-only (#1085)
* The macOS Release asset name does not say it is arm64-only (#1083)

The DMG is arm64-only, but the file attached to the Release is
HTTrack-<version>.dmg, so nothing in the name warns an Intel user that it
will not launch. Everything else already says arm64: the job title, the
upload artifact, and the copy httrack.com serves.

The rename happens at the Release upload rather than at build time because
httrack-com's stage_macos_dmg.sh asserts the artifact basename is exactly
HTTrack-$VER.dmg, so renaming earlier would break staging in that repo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Shorten the rename comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the DMG at build time, not at the Release upload

Xavier's call: the image stays arm64-only and says so in its name. Naming it
where it is created gives the artifact, the Release asset and the logs one
name, instead of making the workflow a second authority over it.

The arch comes from the bundle's own Mach-O list, and counts only if every
one of them carries it, so a thin dylib cannot get the image called universal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Guard the arch intersection against an empty token

Review found that a trailing space in lipo's output would put an empty
token in the set, which every binary then "carries", so a thin arm64
build would be named universal. Apple's lipo emits none, checked with
od -c against a real bundle on an Apple Silicon machine, so this is
insurance rather than a live bug: without the guard that input yields
universal, with it arm64.

lipo joins the tool preflight too, so a missing one says so instead of
reporting that the binaries share no architecture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 07:31:27 +00:00
Xavier Roche
dfc0efff3a WebHTTrack's language menu never marks the language in force (#1086)
* WebHTTrack's language menu never marks the language in force (#1084)

The welcome page opened its menu blank instead of showing the active
language: index.html used ${list:#iso}, and that form carries no default.
The listid: form does, and the launcher already hands htsserver the index
as "lang".

Switching the template alone was not enough. Format 2 opens its first
option before the loop that applies the default, so index 1 (English, the
commonest case) could never be marked. That opening now checks the default
as well, which also covers the nine other listid: menus whose default is 1;
there the first option was already the one the browser displayed, so
nothing changes on screen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the placeholder option and assert the rendered language

Review findings: the empty <option value=0> could itself post lang=0,
which the n > 0 guard ignores, leaving the menu blank again; and the test
asserted the menu alone, so it passed on a menu that merely echoed the
requested index while the body rendered another language.

The placeholder is gone, and test 251 now checks the page's own greeting
and covers the case where no language is set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the test's server PIDs out of a subshell

serve() printed the URL, so callers ran it as url=$(serve ...) and the
pids it recorded died with that subshell. Cleanup then killed nothing and
the four servers outlived the test, which the macOS runner reports as
orphan processes and fails the job on.

It sets ${url} instead. Verified by count: the old shape leaves four
htsserver processes behind, this one leaves none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 07:14:27 +00:00
Xavier Roche
a0622bcd04 Release 3.49.19
Version bump for 3.49.19. Most of what landed since 3.49.18 is FTP: an over-long user name in an `ftp://` URL logged in as a different account, a crawl ignored `--timeout` and `--max-time` and pinned a slot for five minutes, a dead host took two minutes to give up with no way to shorten it, and the engine freed the transfer slots while an FTP thread was still writing to one. Elsewhere, an unbounded panic message could smash the stack, a crawl that gave up mid-loop lost its WARC archive and its change report, and a resume the server refused could still lose the partial file.

`VERSION_INFO` goes 3:11:0 to 3:12:0, revision only: `httrackp` gained a tail field and `htsopt.h` added two `volatile` qualifiers, so no layout moved and nothing went away. Soname stays `libhttrack.so.3`, exports 167 to 168. Nothing under `debian/` landed since 3.49.18, so that changelog entry is the upstream overview plus the two build failures other distributions reported. `Standards-Version` 4.7.4 still matches debian-policy.
2026-08-08 22:12:35 +02:00
Xavier Roche
f5fbd1a82e The suite watchdogs give up the first time a poll tick cannot start a process (#1081)
* The suite watchdogs give up the first time a poll tick cannot start a process

ci_suite_heartbeat read its clock through a command substitution, so the
watchdog needed a fork to tell the time, and a sleep that returns without
waiting either killed it under the caller's errexit or turned
test-timeout.sh's wait loop into a fork storm. Both fail exactly when the
box is short of processes, which is when the watchdog is the only thing
still reporting.

The clock now assigns hb_time instead of printing, the heartbeat's tick is
guarded, and the wait loop gives up after ten ticks that never waited,
naming the test in the progress log the off-box watchdog reads.

References #1038.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: dump the crawl logs before TMPDIR goes, and pin the guards with mutants

The nofork exit skipped dump_crawl_logs, so the EXIT trap deleted the very
logs that would explain the hang. reap_bounded stays out: it polls on the
same broken tick.

250 now asserts the tenth tick by count, that the wedged child was killed,
and that a tick failing every other call is never read as a wedge. 171's
sleep stub returns 126 once, which is what catches the heartbeat guard
being removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:22:21 +02:00
Xavier Roche
496e85a3ac --single-file writes marks it cannot read back (#1080)
* --single-file writes marks it cannot read back

sf_parse_mark refused a <len> above SF_MAX_REF (4096), but htsparse measures
the escaped bytes it appended: HTS_URLMAXSIZE*2 of savename through
escape_for_html_print_full plus as much query through escape_for_html_print.
A mark past the cap parses as nothing and survives into the page as literal
text, taking its reference with it. Derive the cap from the emitter, and have
singlefile_mark decline a span sf_parse_mark would not read back so the two
cannot drift apart. SF_MAX_REF keeps its value for the resolution buffers.

Also swap singlefile_may_mark's hand-rolled search for hts_memstr, clip the
unresolved-reference warning, and make the charset case in test 94 bite: the
server now answers /sfmark.html with a mark in its Content-Type charset.

Closes #1054
Closes #1069

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Check the span derivation at the emit site, and test what the review found vacuous

SINGLEFILE_MAX_SPAN moves to the header so htsparse can assert it against the
tempo and lien it is derived from: a resize there now breaks the build instead
of silently switching inlining off.

Test gaps the review named: the digit-loop saturation guard had no case (a
2^64+10 length wraps to 10 and eats the padding without it), the spans loop
only checked the reference was still present rather than that nothing but the
mark had gone, and nothing asserted the warning clip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:19:59 +02:00
Xavier Roche
d4b77e53dc Ctrl-C does not shorten an in-flight HTTP connect (#1079)
* Ctrl-C does not shorten an in-flight HTTP connect

The user stop only kept new links from being queued: a slot already
waiting for its connection stayed there until --timeout, so stopping a
crawl against an unreachable host took minutes.

back_wait() now ends every slot whose connection is not established yet
(name resolution, connect, TLS handshake) as soon as the stop flag is
set, the way #1071 did for FTP. Slots already receiving are left alone.

Closes #1073

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cancel a stalled name resolution on stop, and harden the sweep's test

The HTTP resolver was still uncancellable: hts_dns_resolve_all() passed no
cancel flag, so a Ctrl-C during a resolve behind a black hole waited out
--timeout per host, the same stall the connect sweep fixes. Pass the
mirror's stop flag, which is what #1071 added the parameter for.

The backstop self-test now covers the states a sweep must not take (FTP,
chunked), runs a control pass with the mirror still running, checks the
socket was closed instead of only cleared, and repeats after re-arming, so
a one-shot or unconditional sweep fails it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:19:06 +02:00
Xavier Roche
507f3565e8 A crawl on a terminal with stdin at end of file toggles its display mode in a loop (#1078)
* Tell an stdin at end of file from a keypress

select() reporting stdin readable does not mean there is anything to read: at
end of file it stays ready forever, and check_stdin() sold that to the crawl
loop as an ENTER. The display then toggled on every pass, so a terminal run
with stdin closed alternated between the spinner and the full panel.

Peek a byte before answering yes, and drop the fflush(stdin) that would have
discarded it (undefined on an input stream anyway). The Windows branch reads
the console queue rather than stdin, so it has no end of file to confuse.

Closes #1072

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Model the ticket's terminal at EOF, not just a redirection

The EOF leg fed httrack /dev/null, so a fix that simply gave up off a
terminal would have passed it while leaving the reported bug alive. Add a
third crawl that types ^D into the pty, which is the case #1072 describes.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:18:11 +02:00
Xavier Roche
4df1434b56 The ProxyTrack Windows skips outlived the reap that made them necessary (#1077)
* tests: the Windows engine reap named only httrack.exe

kill_tree's by-name fallback and reap_leftover_processes each carried their own
hand-written image list beside ENGINE_EXE_RE, and both had drifted: a proxytrack
orphan survived either one. Derive all three from a single ENGINE_EXES.

With the reap covering it, drop the stale #595 skip from the webdav-mime test.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: anchor the Windows image matcher, un-skip webdav-overflow

Review found a fourth hand-written engine list in dump_windows_stacks, and that
the tasklist greps were unanchored: notepad-httrack-notes.exe read as a leaked
engine. Both now go through ENGINE_IMAGE_RE, derived from ENGINE_EXES.

webdav-overflow joins webdav-mime on Windows. webdav-default and
proxytrack-quiet keep their skip, with the real reason: they read proxytrack's
console through a pty, and Windows has neither Python's pty nor os.fork.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 21:17:24 +02:00
Xavier Roche
d85d91f8c6 Adding a test conflicts with every other test-adding PR (#1075)
* Adding a test conflicts with every other test-adding PR

tests/tests-list.mk was a single shared file every test-adding PR appended
to. merge=union kept that clean locally, but GitHub never reads
.gitattributes, so on the forge each pair of such PRs conflicted and every
merge re-conflicted the rest of the queue.

configure now globs tests/[0-9]*_*.test into TESTS_LIST and tests/Makefile.am
reads it back as TESTS = @TESTS_LIST@, the one form automake passes through
untouched. A make-level $(wildcard) cannot replace it: automake word-splits
TESTS at automake time and emits a broken rule per word.

check-tests-list.sh becomes check-test-names.sh, guarding the naming
convention the glob depends on rather than a list that no longer exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch a build dir whose TESTS predates the test you just added

The glob moved the silent-skip hazard rather than removing it: TESTS_LIST is a
literal frozen into tests/Makefile by config.status, and nothing regenerates it
when a .test appears, so a new test was simply not run and every other test
still passed. 231_test-names.test now compares tests/ against the TESTS the
running Makefile holds, which turns that into a red test on the next make check
and covers the deletion case too.

The glob also dropped the old list's count oracle, so a test its author forgot
to git add left CI running a smaller suite, green. check-test-names.sh now
compares the glob against git ls-files, skipping that when nothing there is
tracked so tarball builds are unaffected.

It also missed a case-mistaken 01_Engine.TEST, and claimed a name with a space
in it was one make check runs, when configure space-joins the basenames and
make sees two words. Both are rejected now, with a control each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A git that cannot answer read as a clean tracked set

The tracked-set oracle swallowed git's exit status, so git missing from PATH or
a safe.directory refusal looked exactly like a tarball with nothing tracked:
over a checkout holding an untracked test the checker printed a clean line and
exited 0. It now probes whether git can answer at all and says on stderr when it
cannot, leaving the genuine empty-listing skip that tarball builds rely on
untouched. 231 stubs git out to prove the note fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 17:05:06 +00:00
Xavier Roche
b16ffc7be2 The --single-file crawl assertions run on Windows but are reported as a skip (#1066)
* The --single-file crawl assertions run on Windows but are reported as a skip

94_local-single-file.test only reaches its exit 77 after every crawl assertion
has passed, so the Windows suite counts real coverage as a skip. Move the
htsserver block to 241_local-single-file-gui.test and drop 94 from
expected_skips.

Refresh two stale justifications while there: 48 and 71 were pinned on the now
closed #581, and 153 had none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert tokens, not substrings, in the single-file GUI and mirror checks

"--single-file" is a substring of "--single-file-max-size=5000", so the flag
assertion passed with the bare flag gone; the value match had no trailing
delimiter, so 50000 passed for 5000. Both mutants now kill the test.

94 carried three fail-open checks that this PR makes load-bearing on Windows: a
grep whose status was discarded, a mark sweep that read an unsearchable tree as
clean, and a --changes crawl writing to the first mirror's -O, which left its
page absent and every check on it vacuous.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep grep's status in the mark sweep, and size the page the change report names

The file-count floor could not see what the sweep saw: it walked its own find
expression, so mirrored files at mode 000 counted 11, passed, and left grep's
status laundered by the pipe it was meant to guard. Keep grep's own exit status
instead, and run tr as a separate step.

The --changes check named an invariant it did not test. The report stats the
file it lists, so comparing its size for the page against the page on disk is
what proves expansion ran first; moving singlefile_process_mirror() after
hts_changes_close_opt() now fails it at 1037 bytes against 28705.

Also derive the crawl options from one array rather than retyping them at four
call sites, and match the max-size flag as a token: the right-hand anchor let a
template fusion through.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Move the change-report probe out of a command substitution for bash 3.2

macOS ships bash 3.2, which ends $(...) at the ")" and runs the here-document
body as shell commands, so the probe failed with "import: command not found".
Write it to a file at statement level and run that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 16:29:59 +00:00
Xavier Roche
defd6735f3 A WARC abort can orphan a .tmp when teardown reopens the writer (#1068)
* A WARC abort can orphan a .tmp when teardown reopens the writer

warc_close_opt() and warc_abort_opt() left opt->state.warc NULL, which the
emit hook reads as "never opened". XH_uninit then finalizes the slots still
in flight, and back_finalize's warc_write_backtransaction lazily reopens the
archive. The reopened writer nobody closes leaves a .tmp behind, since the
archive it would replace is already in place.

Both now leave the existing WARC_DISABLED sentinel instead, which the hook
already honours: no writer, and none to open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make test 242 tell the close case from the abort case

The size compare after teardown could not fire: a late reopen always writes
the temporary, never the archive, so the two sizes matched on every path. And
nothing pinned which run the archive held, so a close that stopped committing
its own run passed both assertions.

Each run now carries its own body, and the archive is read back: valid
framing, the run that had to survive, and neither of the other two. Abort
keeps run 1, close commits run 2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Say what the framing check covers, and convert warc_slurp's path

The WARC/1.1 compare reads the first record's version line, so it catches a
file that is no longer an archive, not a corrupt one; the late-body leg is an
append guard rather than a discriminator, since nothing can reach the archive
without losing the run that was there. Both said so in one line.

warc_slurp() also opened its path raw where every other selftest path goes
through fconv(), and the test now traps the signals on their own line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Note that copy_htsopt must not copy the embedded state

The rule was written only for sitemap_state and singlefile_state; a copy of
state would clear a WARC writer the live opt had already finished with.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 15:52:40 +00:00
Xavier Roche
178f9156e5 An FTP crawl of a dead host takes two minutes to exit, and Ctrl-C cannot shorten it (#1071)
* An FTP crawl of a dead host takes two minutes to exit

The worker sat in a blocking connect(), so --timeout, --max-time and the
engine's stop flag were all bypassed and the kernel's own connect timeout
was the only bound. It now connects non-blocking and polls on the same
100ms tick wait_socket_receive uses, re-reading the stop flag and what is
left of --max-time each pass; the resolve goes through a new bounded entry
point taking an explicit deadline and a cancel flag.

The PASV data path also gated its connect on server_size > 0, which is
sizeof(SOCaddr) and so always true, rather than on the resolve result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Make the new tests able to tell the fix from an instant failure

The timing assertions had no lower bound, so ftp_connect's two
instant-failure returns passed both legs, and the log they matched is
wording the HTTP backend shares. Each leg now floors its elapsed time,
the interrupt leg proves the crawl was still parked before it signals,
and both anchor the FTP form of the message.

Split so Windows runs the half that needs no signal, poll a debug line
rather than sleeping toward the connect, and cover the cancellable
resolve through the DNS selftest's scripted black-hole backend, which
192.0.2.1 (a literal, resolved instantly) never reaches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Restore an unrelated test's file mode

A chmod glob for the two new tests caught 24_local-resume-overlap.test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Move the FTP timeout test off 244, which #1070 already claims

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Make the two stop flags volatile

The new 100ms poll loops re-read stop_ftp and state.stop with no lock, so
their correctness rests on the compiler not hoisting the load out of the
loop. select() and Sleep() are opaque enough today, but that is the shape
of the calls rather than a guarantee, and the code these loops replaced
sat in a blocking syscall instead.

hts_dns_resolve_all_bounded takes the flag by pointer, so its cancel
parameter becomes const volatile: passing &back->stop_ftp would otherwise
discard the qualifier.

Not an ABI break, though htsopt.h is installed: volatile changes no
offset, size or alignment, only the declared types.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 14:44:04 +00:00
Xavier Roche
ee5db0a9e6 -%v1 overwrites its own output and -%v2 paints escapes into a redirected log (#1070)
Salvages what stands on its own from #362; the progress bar itself is not here.

In `-%v1` each completed file printed a line ending with a CR and a clear-to-EOL, so every line overwrote the one before it. A finished crawl left the terminal holding a single line and no record of what had been fetched, and redirecting that output gave one unreadable line of escapes. Those lines end with a newline now, and the clear-to-EOL goes with them. That part is Matteo Fiorini's, credited in the trailer.

The rest is a defect #362 shared rather than fixed. An explicit `-%v2` kept painting cursor-addressed panels whatever stdout was, because the `isatty` probe in `htscoremain.c` only picks the default verbosity. That probe is also compiled out on Windows, where the spinner wrote its CRs straight into a redirected file. Both are gated on a terminal now, each where it is drawn rather than by rewriting the user's option. The CLI arms its panel in `htsshow_start()`, which the mid-crawl ENTER toggle re-enters, so the gate holds for the whole crawl. Off a terminal `-%v2` paints nothing; `-%v1` is the mode that gives a redirectable log.

`241_local-tty-output.test` crawls the local server three ways into a pipe. It requires that no escape and no CR survive, that each crawl actually ran, and that `-%v1` keeps one line per file. The controls repeat the crawls with stdout on a raw pty and require a repainting panel and a turning spinner, so a fix that merely silenced the display would fail instead of pass. Reverting either completion line, the panel gate or the terminal probe breaks the test, and so does deleting the display outright. One gap: the spinner gate cannot fail on POSIX, where `quiet` is already forced off a tty, so only the Windows leg exercises it.
2026-08-08 16:06:55 +02:00
Xavier Roche
edeada647c A crawl that gives up mid-loop loses its WARC archive and its change report (#1064)
`httpmirror()` has four exits that never reach the teardown at the end of the function: the three crawl-loop bailouts and the no-data rollback. None of them runs `warc_close_opt`, `hts_changes_close_opt` or `hts_sitemap_free`, so any crawl the engine gives up on loses all three. Tripping the `-#L` link limit is the easiest way to get there; a shell or GUI abort does it too.

The WARC is what this costs in practice. Segments are written under `.tmp` names and renamed only once the writer closes, so when a previous archive exists, a bailout strands the whole capture in a `warc-out.warc.gz.tmp` nothing will ever pick up. The previous archive stays in place as if the run had never happened, with a stale `.cdx` beside it.

All four exits now jump to one label, with `retcode` keeping the bailouts' -1 apart from the tail's 1. The rollback still calls `warc_abort_opt` first, which NULLs the writer and makes the shared `warc_close_opt` a no-op there. `hts_cache_reconcile` still runs after `XH_uninit`, which is what closes the cache files it renames.

The change report goes the other way. An abort never reaches `hts_changes_indexed()`, so any report it writes claims the cache is off and no deletions were found, and on the rollback path that false report replaces a true one describing the mirror the rollback just restored. So only a mirror that reached the end of the loop writes `hts-changes.json`. Every other exit frees the accumulator, which is the leak the issue actually named.

Test 240 poisons the archive and snapshots the report, aborts on the link limit, then re-runs against a dead server. Three separate mutants each kill a different assertion. It covers the `-#L` bailout only: the other live one needs the loop callback to refuse, which the CLI cannot drive. The third, a failing `hts_mirror_check_moved`, is unreachable today since every return in that function is 0, and it is routed with the others rather than left as the odd one out.

Also frees `primary` on the exits that leaked it. #1060 is unaffected: its window, a writer lazily reopened during `back_delete_all`, is unchanged.

Closes #1061
2026-08-08 09:27:46 +00:00
Xavier Roche
562f6790ec Two of the Android help screens have no anchor to deep-link at (#1065)
* Two of the Android help screens have no anchor to deep-link at

The Android app is getting per-screen contextual help (httrack-android#113)
and needs somewhere to link. Six of the eight screens already have a target,
since the wizard steps and the option panel kept their ids when the eighteen
GUI pages were folded into guide.html. The first-launch permission note and
the paragraph saying where mirrors land did not, so add an id to each.

Those ids are link API now: the app hardcodes them and an installed release
outlives whatever engine SHA it was built against. The new test resolves
every guide.html# reference in html/ against the file's actual ids, which
also covers the redirect stubs left by the page merge, and names the eight
the Android side depends on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* BSD sed has no \| alternation, so macOS never strips the platform prefix

The scan left "droid/step-run" unstripped on any POSIX BRE sed, and no id
matches that, so the test failed everywhere but GNU.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the reinvented link scan; tools/doc-links.py already resolves them

The generic half duplicated tools/doc-links.py, which the documentation CI job
already runs over the whole doc set. What is left is the part it cannot know:
the eight anchors httrack-android hardcodes, each present once and on an
element the droid platform can see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 07:56:13 +00:00
Xavier Roche
cbb2257847 A rejected resume can still lose the file (#1057)
* A rejected resume can still lose the file

With --retries=0 an unusable Content-Range deleted the partial and never
re-fetched it: the restart-whole recovery was written as an error retry, so it
was both gated on the retry budget and charged to it. Restarting a transfer the
server refused to resume is not a retry of a failed transfer, so it no longer
spends the budget; refetch_whole, which only a previous restart sets, latches it
to one so an always-unusable server cannot loop.

#581 was already fixed by #655, but nothing held that fix in place. Tests 48 and
71 both pass with it disabled and neither reaches the restart path, so test 235
drives it with an LD_PRELOAD interposer that refuses the unlink the way Windows
refuses one on a file it still holds open.

The unlink in the memory-resume branch also ran while a FILE* on that file was
still open, which is what left the partial behind on Windows; it now closes
first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Register 235_local-resume-recovery in the Windows expected-skip set

The Windows suite pins its skip set, so a new test that skips there reds the
job at fail=0. Also drops the stale #581 note on 71, whose skip is now the
same MSYS signal limitation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Carry the whole-file restart latch across a link re-record

The free restart introduced by this branch is bounded only by
lien_url::refetch_whole, but the two sites that re-record a link while
preserving its retry count -- the case-alias redirect replay and the
412/416 re-get -- never copied that latch, and hts_record_link zeroes the
new entry. A server alternating an unusable 206 with a 301 to a
case-different alias of the same path therefore restarts for free at
every hop: 148 restarts and no retries consumed in 60s at both
--retries=0 and --retries=1, where the merge base terminates in one
second.

Copy the latch wherever retry is copied, and say so at the field's
declaration. Test 235 gains a leg driving that alternation; it also
asserts a later genuine failure still had its retry, which the previous
three legs could not see.

The unlink interposer now tells an open handle from a closed one through
/proc/self/fd instead of refusing by basename, so reverting the fclose in
the memory-resume branch turns the last leg red rather than leaving it
green.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:36:01 +02:00
Xavier Roche
0e95c58865 Three unreachable defects from #1040, and a live stack overflow found beside one (#1056)
* Fix three unreachable defects in the array and bounds-check macros

TypedArrayEnsureRoom doubled the capacity until the requested room fit, so a
large enough request wrapped it to zero and the loop never ended; capa * width
could wrap too, into an allocation smaller than the capacity claimed. Growth
now saturates and asserts, and the macro uses the realloct/freet wrappers.

HTS_IS_CHAR_BUFFER cannot be made sound on MSVC, so its comment now names both
sizes that go blind (char[8] on Win64, char[4] on Win32), and the two
pointer-sized destinations in the tree are widened. HTS_CDLMAXSIZE keeps its
value; the *2 comment beside it went stale when 3.30.1 raised HTS_URLMAXSIZE.

Also fixes a live stack overflow found while auditing HTS_CDLMAXSIZE:
htscoremain.c formatted "Missing quote in %s" with a raw sprintf into a bare
HTS_CDLMAXSIZE buffer, overrunning it by 17 bytes on a max-length argument.

Closes #1040

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the array overflow self-test's abort out of the optimizer's reach

Test 237's SIZE_MAX-room mode relied on realloc returning NULL, which nothing
observed: the array is dead after the call, so clang (and gcc from sid) drop
the allocation, assume it succeeded, and fold away the "== NULL" that aborts.
Printing the pointer keeps the allocation live. ASan meanwhile kills the
process itself on an over-large request rather than returning NULL, so that
one run asks for libc's behaviour instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the remaining raw sprintf panic messages in htscoremain.c

The "Unable to create log file %s" pair formats the whole -O path into a
char[HTS_CDLMAXSIZE + 256]. That path is not argv-gated: check_path() runs
expand_home() on it, which admits 2047 bytes, so
  HOME=<2000 chars> httrack -q -O '~/x' <url>
smashes ~800 bytes of stack (fortify aborts at htscoremain.c:2583).

The second "Missing quote in %s" site copies argv[na] into a +256 buffer
first, so its 17-byte prefix can overrun the same way even though the
argv gate keeps today's callers short of it.

Also fixes the HTS_CDLMAXSIZE comment (the gate rejects >= 1024, so 1023
is the longest accepted) and trims the comments added by this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:03:30 +02:00
Xavier Roche
8184b45696 Read the .arc bytes with od instead of piping into head (#1063)
`tail -c +N file | head -c 3` leaves tail writing into a pipe head has already
closed. Under this test's `set -o pipefail` that SIGPIPE becomes the script's
exit status, so 154 dies at 141 having printed nothing. GNU coreutils happens
to survive it, which is why the red only shows on the uutils leg. od seeks to
the offset itself and reads no more than it needs, so no second process is
left holding the other end.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:02:40 +02:00
Xavier Roche
62ca6e087a A repeated crawl can write a byte-identical .cdx, so the rerun test reds (#1046)
* A repeated crawl can write a byte-identical .cdx, so the rerun test reds

--archive-replaced-on-rerun compared bytes to decide the second pass had
rewritten each archive file. WARC segments carry a fresh record UUID, but the
.cdx holds no per-run entropy: only a second-granularity timestamp and
length/offset fields that move with the gzip member sizes. Two identical passes
can produce the same bytes, about 0.5% of runs, and the test read that as a
stale file.

Assert the replacement positively instead, reusing the poison the .wacz check
already used: every snapshotted file is overwritten with a marker between the
passes, and none of it may survive. Also make warc_cdx_flush() say something
when it leaves the previous pass's index beside a freshly swapped archive,
rather than returning silently on an open failure or an empty index.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the WARC writer on the rollback path, and stop warning when nothing was replaced

warc_open's own failure paths call warc_close with cdx_on already set, and
warc_commit returns early when there is no previous archive to protect, so a
failed archive open reached the new zero-count warning and logged a second,
false error line. Gate it on the writer having opened, and on the run not
having failed.

The "no data seems to have been transferred" branch in httrack_main returns
before the teardown that closes the writer, which is the one path where a stale
index can outlive its archive and the warning can never fire. Close it there
too. Closing it outright would be wrong: the temporary holds a warcinfo record,
so warc_commit would swap an almost empty archive over the previous one and
destroy it, measured as 2431 bytes and 2 index lines down to 1509 and 1. The
new entry point marks the run as one that must replace nothing, which drops the
temporaries and keeps the previous archive, while still writing the index for
an archive written in place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the --archive-replaced-on-rerun header

Cause before effect, and the entry fits in three lines again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Stop reading a straggler tick as a live reporter in 172

The stub reporter advances its marker with a forked mv, and that mv outlives
the kill -9 of its parent, so the marker can move once more after the wedge
branch has already ended the reporter. still_ticking calls it alive on the
first differing sample, which reds the suite (#1058). The two negative call
sites now wait for the marker to hold still instead, which a straggler only
rebaselines; the positive one keeps the old predicate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber the WARC rollback test to 238

232 was taken by 232_online-gate-outoftree.test, which landed on master
in #1047 while this branch was open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Gate the empty-index warning on the archive this run actually touched

warc_cdx_flush() warned about a stale .cdx whenever a writer opened cleanly
and indexed nothing, but warc_commit() returns early when there is no previous
archive to protect, so a first run that saves nothing hit the warning with no
index to invalidate. Gate on protect_prev, which reaching the flush proves a
swap happened, and add the mirror case the old gate stayed silent on: a run
written in place and then abandoned, whose archive is clobbered while an index
beside it still describes what used to be there.

Report a write error ahead of fclose's echo of it, too: glibc drops the
unwritten buffer on a failed flush, so the "index is incomplete" arm only ever
fired when the total happened to land on a buffer boundary.

A warc-cdx-errors self-test drives all five outcomes, over /dev/full for the
two write failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Warn about a stale index only when one is on disk

The empty-index gate keyed on protect_prev, which stands in for "this run
replaced the archive", but not for the index the message names: a run written
in place and abandoned, or one swapped over a previous archive, warned that
<base>.cdx "was not rewritten" whether or not that file had ever existed. It
now asks the disk. The self-test's two positive cases left no index behind, so
they passed on a message about a file that was not there; give them one, and
add the two negatives that pin the terms the old cases could not tell apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 21:19:39 +00:00
Xavier Roche
7c6e5c3018 An FTP crawl ignores --timeout and --max-time, pinning a slot for five minutes (#1050)
* An FTP crawl ignores --timeout and --max-time, pinning a slot for five minutes

run_launch_ftp hardcoded a 300 second read timeout and never looked at
opt->timeout, so every control- and data-channel read on an FTP slot
inherited it: a server that accepted the connection and then went quiet
held the slot for five minutes. The timeout now comes from opt->timeout,
which defaults to 120 rather than 300.

--max-time could not end an in-flight FTP transfer either. That slot is
owned by the FTP thread, which back_wait cannot abort without racing it,
so each wait inside the thread is instead clipped to what is left of the
mirror deadline and the thread gives up on its own.

Closes #1039

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the FTP data-channel wait and --timeout=0 in the tests

The two tests passed against a tree that hardcoded the body-transfer wait
back to 300, because one never reached RETR and the other read the outer
timeout rather than the value the wait used. Test 233 now runs a second,
answering server alongside the mute one: a stalled body transfer under
--timeout=5 with no --max-time, so only the data-channel wait can hold
it, and a trickled one under --timeout=0, which must complete rather than
expire at once.

Also trims the comments the review flagged, and documents what the two
FTP helpers do rather than only how opt is used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the premature 3.49-19 release-notes block

history.txt carries a version block only when a release is cut, curated
from the tag diff; twelve releases show no feature PR opening one. Two
open PRs had each created 3.49-19 at the same anchor, and the file has no
merge=union, so the second to land would have conflicted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 21:05:07 +00:00
Xavier Roche
1878595043 The engine frees the transfer slots while an FTP thread is still writing to one (#1055)
* Wait for FTP threads before freeing the slots and options they still read

An FTP transfer runs in a detached thread holding a pointer into sback->lnk,
and nothing waited for it: back_delete_all() wiped the slot and back_free()
released the array while the thread was still writing through them. A mirror
limit makes it trivial to hit, since the engine stops the moment the cap trips.
Against a trickling server, "-M100000" segfaulted in run_launch_ftp writing
back->r.size.

httrack.c drained threads before hts_free_opt(), but hts_main() and htsweb.c
did not, so opt had the same exposure.

htsftp.c now keeps a mutex-guarded list of live workers, registered before the
thread is spawned and released only after the worker's last store to the slot.
ftp_stop_workers() cancels them and waits, and runs at both lifetime
boundaries: back_delete_all() for the slot array, hts_free_opt() for opt.

The wait has to be bounded rather than merely patient, so wait_socket_receive()
and get_ftp_line() now take the slot and leave their poll loop as soon as
stop_ftp is raised. Previously the flag was only read between operations, so a
silent server held teardown for the whole read timeout. A failed thread spawn is
also handled now; it used to leave a slot nobody would ever reap.

Test 236 crawls a trickling server under a size cap, then a stalled one, and
requires teardown well inside the read timeout. Both fail on master, the first
on the segfault and the second by taking 300 seconds.

Closes #1051

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Budget the trickle crawl's wall clock, and trim the review comments

The trickle half of test 236 asserted only that the crawl survived, which a
drain that waits for the transfer instead of cancelling it also satisfies: it
exits 0 with the whole file fetched. Measured at 103s against 3s for a real
cancel, so the pass now carries a 30s budget too. Also records why the crash is
reproducible at all, since it rests on the backlog array being large enough for
free() to unmap.

Comment cleanup: one issue citation instead of five, a one-fact contract for
ftp_stop_workers, and a one-line test header. The registration-timing note keeps
the invariant and drops the issue number. The history entry said --max-size
could not stop an FTP download, which reads as the cap never firing; it fired,
then crashed or took the full read timeout to tear down.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* tests: fail 236 on a sanitizer report, not only on a signal

Both crawl cases only asserted "not killed by a signal", which a sanitized
build never satisfies: ASan catches the #1051 use-after-free and exits 1
unless the job also sets abort_on_error, so the regression test passed on a
buggy binary. Capture the crawl's stderr and reject any ASan/UBSan/MSan
report; leaks at exit stay out of scope. Vacuous on a normal build, where the
signal check still fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the premature 3.49-19 release-notes block

history.txt carries a version block only when a release is cut, curated
from the tag diff; twelve releases show no feature PR opening one. Two
open PRs had each created 3.49-19 at the same anchor, and the file has no
merge=union, so the second to land would have conflicted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 22:06:53 +02:00
Xavier Roche
057648e335 --single-file maintains a second HTML parser instead of reusing htsparse (#1049)
`--single-file` shipped with its own HTML and CSS scanner in `htssinglefile.c`, about 1140 lines doing what `htsparse.c` already does. This replaces it. htsparse now appends an unguessable per-run mark to each reference it saves that the pass may inline, and the end-of-mirror pass becomes a substitution over those marks: no HTML and no CSS is parsed there at all. The scanner goes, and with it the private attribute table that had already drifted from `hts_detect[]` twice. `htssinglefile.c` goes from 1155 to 863 lines; `htsparse.c` gains 31.

The mark is `#!<16-hex secret>.<class>.<len>`. It is a fragment, so a mirror an interrupted run left marked still browses. The secret is CSPRNG bits drawn once per run and never written to disk, which is what makes the mark unforgeable: a site cannot spell one, so no sanitiser has to keep hostile bytes away from it. `<len>` is the length of the reference preceding the mark, so the pass never has to guess where a reference begins. `<class>` is the context htsparse saw, checked against the resolved type so a mismatch fails loudly instead of inlining whatever the walk found.

Behaviour worth knowing about. A fragment comes back byte-identical rather than re-escaped, since the mark covers the reference alone and what follows it is document text nothing touches; the query moved inside the marked span, because glued to base64 it corrupts the payload. Expanding a mark needs the secret, but recognising one well enough to delete it does not, so the end-of-mirror sweep strips by shape. That clears what an interrupted earlier run abandoned, and costs a page any mark-shaped text of its own. A `<link>` carrying no `rel` is no longer inlined, which is what the old scanner did.

`htsserver` links libhttrack but cannot see a hidden symbol inside it, so the CSPRNG that #1011 added privately to `htsweb.c` and this branch added to `htstools.c` now share `htsrandom.c`.

Closes #749
2026-08-07 18:25:28 +00:00
Xavier Roche
7a7206d7f1 Update Turkish translation from issue #306 contribution (#1048)
* Update Turkish translation from issue #306 contribution

Value-merges the epiusu/httrack fork's Turkish.txt into our own: our
key set and order are kept as-is, but each key's value is replaced
with the fork's translation where the fork has that key. 227 of our
508 values changed; the fork's 44 missing keys (translated against an
older English.txt) keep their existing value, and the fork's one key
absent from our English.txt is dropped.

Co-Authored-By: Ümit Solmaz <usncb@yandex.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch a value left as raw UTF-8 under a declared single-byte charset

A single-byte charset maps all 256 byte values, so the existing iconv
decode can never fail on one and cannot see a translation pasted in as
UTF-8 without transcoding. Only the C1-control heuristic stood in the
way, and a UTF-8 sequence whose bytes are all >= 0xA0 decodes to
ordinary printable characters under Latin-1/-9, sailing straight past
it.

Scan the raw bytes instead: a whole line that is itself valid
multi-byte UTF-8 under a file declaring a single-byte charset is the
leak. Scoped to those charsets because the premise fails for BIG5,
gb2312 and shift-jis, whose genuine text matches the same shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Two Polish translations were stored as UTF-8 in a windows-1250 file

The Keep-Alive option label and its tooltip carry raw UTF-8 bytes in a
file that declares windows-1250, so the GUI and htsserver render them
as mojibake ("połączenia"). Transcode just those two values; the rest
of the file was always windows-1250.

Found by the raw-UTF-8 check added in the previous commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The lang charset check uses grep -P, which macOS does not have

The raw-UTF-8 scan reached for a PCRE lookahead, so BSD grep exits with
"invalid option -- P" and the loop that consumes its output sees nothing.
The per-file scan then passes everything, and the negative control fires
instead: "the charset check passed a deliberately mislabelled file".

Express the same grammar as a POSIX ERE over raw bytes and let awk apply
it, which also folds the per-line length probe into the one pass. Checked
against the PCRE on all 30 lang files plus both probes, under gawk and
mawk: identical line sets.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Ümit Solmaz <usncb@yandex.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 17:22:29 +00:00
Xavier Roche
e4df3b387c Out-of-tree builds skip the online crawl tests with a misleading reason (#1047)
* tests: resolve online-gate helper scripts against the script's own directory

The seven online crawl tests invoked check-network.sh and crawl-test.sh by bare
relative name, which the shell resolves against the caller's cwd. make check
runs from the build tree's tests/, so an out-of-tree build finds no copy: the
gate failed to open, the || branch reported "skipping online unit tests", and
the tests skipped whatever --enable-online-unit-tests asked for.

check-network.sh had the same defect in its own auto-probe call to
crawl-test.sh. Resolve all of them against $(dirname "$0"), the idiom
local-crawl.sh already uses. 232 pins the skip reason from an unrelated cwd and
sweeps the tree for the same form. The option's help string advertised
default=yes; the code has always defaulted to no.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* tests: keep 232's comment clear of 226's credential-scrub gate

226 flags any .test naming the Windows suite driver without the scrub line,
comment or not, so the exemption note tripped it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: tighten 232's sweep, which is the only check below the gate

With the flag off check-network.sh exits before a test reaches its own
crawl-test.sh line, so the runtime probe cannot see that class at all and the
static sweep carries it alone. Four spellings of #1016 were getting through: a
leading ./, a quoted bare name, eval, and a variable holding the bare name.

Quoting and ./ change nothing about what the shell opens, so the shape
exemption is gone and the two call sites that mean the cwd are named instead. A
variable operand is resolved back to its assignment, and eval sites are pinned
as a set rather than passed over. Each form is planted as a fixture, with
anchored and commented fixtures for the other direction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: 172 never checked that it ran its own staged suite copy

Pointing it at the srcdir copy passed, dropping the reap_leftover_processes
neuter it appends to the staged testlib.sh and leaving the test to reap sibling
engines under "make check -j", a flake whose symptom points nowhere near its
cause. test-timeout.sh sources testlib.sh from the staging directory too, so a
plain marker file would still be written under the mutant; record the sourcing
script instead and require the driver.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 16:38:48 +00:00
Xavier Roche
58ddc5a6b1 An --enable-fuzzers build cannot link proxytrack (#1045)
* Link the sancov runtime into every program under --enable-fuzzers

--enable-fuzzers appends -fsanitize=fuzzer-no-link to DEFAULT_CFLAGS, so
every object in the tree carries sancov instrumentation, but only fuzz/
linked a runtime defining those hooks. src/proxytrack, the first program
linked, failed on __sancov_lowest_stack and __sanitizer_cov_trace_*.

Append the flag to DEFAULT_LDFLAGS in the same branch, and add a CI leg
that builds the bare configuration: the existing fuzz job links ASan,
whose runtime happens to define the same hooks, so it could not see this.

Closes #1030

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the two new comments

Both opened with setup instead of the point, and the configure.ac one
restated its neighbour two lines above.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 17:39:46 +02:00
Xavier Roche
fb01ad6f65 A resurrected test registration stops make check with no test results (#1044)
* The union merge driver on tests-list.mk resurrects deleted registrations

merge=union on tests/tests-list.mk only applies to a local merge: GitHub
does not run the driver when it computes mergeability, so both appending
PRs go CONFLICTING anyway, and a conflicted PR runs no checks. Where it
does apply it never removes a line, so a rebase past a deletion restores
the old registration, and make check then exits 2 with no test results.

Drop the attribute and check the pairing instead, in the lint job (a
stale registration kills make check before it can report) and in the
suite (an unregistered file is simply never run).

Closes #1037

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the union, and make the registration it resurrects loud

Review of the first commit: the attribute is right on 75 of the 76 test
add/remove events since it landed, and dropping it would not have helped
the conflicted-PR half of #1037, since GitHub never ran the driver. So
keep it, pin the file to LF, and let the checker cover its one failure
mode.

Give 231 four broken fixtures: every branch of the checker is silent on
a healthy tree, so a checker that had stopped comparing passed the old
assertion just as well. Pin LC_ALL=C, since BSD comm need not share the
collation GNU sort used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Count a directory or dangling symlink named NN_x.test as absent

Set equality held for both, so a registration pointing at one passed the
checker while make check would still choke on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 15:57:09 +02:00
Xavier Roche
1a211555d0 3.49.18 fails to build on Ubuntu: two tests assume GNU coreutils (#1043)
* Two tests break under uutils, the coreutils Ubuntu 25.10 made default

105 built its fake engine by symlinking sleep to a file named httrack,
and uutils' multi-call binary refuses an argv[0] it does not own, so
both fakes died before the first assertion and the oracle read an empty
pgid. 153 put env(1) in front of a backgrounded proxytrack; uutils' env
forks instead of exec'ing, so stop_server killed only the env and
proxytrack kept the pty open, leaving the drainer waiting for an EOF
that never came.

The fake is now a copy of the shell, which answers to any name, and 105
checks that it started: one that never ran showed up only as an empty
field in a later assertion, which reads as a bug in the code under test
rather than in the fixture.

The new CI job builds and runs the suite on ubuntu:devel, which is where
the Ubuntu buildds are and no runner here was.

Closes #1042

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the fake engine, and check the argv[0] dispatch rather than the banner

Review found the fifo fake blocks in open() forever, where the sleep it
replaced self-healed in 60s: a test killed outright (kill_tree sends
SIGKILL, and the outsider sits in its own process group) would leave a
process named httrack on the runner for good. Opening read-write lets
the read time out instead.

The uutils check now renames sleep and requires it to be refused, which
is the property #1042 broke on. The banner works today but has already
been reworded once: 0.0.30 prints "sleep 0.0.30" and 0.8.0 "sleep
(uutils coreutils) 0.8.0". Configure also asserts the codecs and TLS the
way the sibling job does, so a renamed dev package cannot turn the tests
that cover them into skips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Re-trigger CI

The pull_request run wedged: all 23 jobs finished or were cancelled while the
run itself stayed 'queued', so the required checks never reported and neither
rerun nor cancel was accepted (CI run 31124085297, windows-build 31124081217).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Re-trigger CI after the Actions outage

The 2026-08-06 Actions incident throttled webhooks, so the previous push never created a workflow run on this head.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 00:37:43 +00:00
Xavier Roche
f2abfc36ef Nothing survives a wedged Windows runner to say which test wedged it (#1033)
* Give the Windows suite a watchdog that does not need to fork

Every recorded #795 event dangles in the suite step, and the bash heartbeat
added for it has never once fired. Each tick costs a fork on sleep, tail, awk
and finally taskkill, while the leading explanation for the wedge is a box on
which process creation has stopped, so the watchdog is disarmed by the very
condition it exists to catch.

Add a native one: a single long-lived PowerShell process that spawns nothing,
reads the progress log, polls memory, handles, process count and free disk
in-process, and posts a commit status every 30 seconds. That status is the only
thing that leaves the VM before it dies, so the last one to land timestamps the
death and names the test in flight. On a static log it walks Win32_Process
downward from the driver's pid and kills that subtree, never upward, since
killing Runner.Worker.exe reproduces this signature exactly.

The bash heartbeat stays as the fallback for a box with no PowerShell; only one
of the two ever runs, or the loser of the race overwrites the winner's verdict.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 224 ran the real watchdog because TMPDIR splits the PATH entry

The Windows runner hands the suite a TMPDIR of D:/a/_temp, so the stub bin
directory the test prepends to PATH is cut at the drive letter's colon and
never searched. The launch then found the runner's own pwsh and drove the
real watchdog with the real status token, and the stub's argv file the
assertions read never appeared.

Map the temp dir back to a POSIX path where cygpath exists, and assert the
stub is what pwsh now resolves to, so a shadow that fails again says so
instead of reporting a launch that did happen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Report the Windows suite step off-box while the runner is still alive

Every recorded #795 event dangles in the suite step and leaves nothing to read:
the log 404s, the annotations come back empty, and the if:always() uploads never
start. The bash heartbeat cannot help, because it reports through the same
channels the dying runner takes with it.

Add a PowerShell process that only observes. It spawns nothing, reads the
progress log, polls memory, handles, process count and free disk in-process, and
posts a commit status every 30 seconds. That status is the only thing that
leaves the VM before it dies, so the last one to land timestamps the death and
names the test in flight. The driver posts its own verdict as it exits, so a
status left pending means the runner really did go.

The bash heartbeat is untouched and still owns the kill. This one has no verdict
to race it over.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the stub shadows PowerShell before driving the launch

An unshadowed pwsh runs the real watchdog, and the argv assertions that follow
then read as a plain launch failure rather than as what happened.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The status backoff must throttle the API, not the artifact

A fork PR's token cannot post, and the artifact log was the only record left of
those runs. Log every status line whatever the backoff decides, and let it skip
the request alone.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the launch arguments in the form a Windows interpreter can open

The driver hands PowerShell the native path, so comparing against the MSYS
one failed both Windows legs. nativepath() is the identity elsewhere, which
is why the Linux run never saw it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Post a verdict a killed step cannot forge, and keep the token out of the tests

bash reaches its EXIT trap with $? = 0 after a signal, so a cancelled or
timed-out suite posted `success`. The verdict now comes from explicit INT, TERM
and HUP handlers, and every exit path that can still run code posts it: the
heartbeat's kill path posts before it kills, since a taskkill runs no trap, and
the final post is retried so one transient 5xx cannot leave a finished suite
reading `pending`.

The step's token is taken out of the environment and handed to the watchdog
process alone, so none of the ~90 test scripts the driver runs can post a
status with it, and the checkout stops persisting it into .git/config.

A failing stdout write no longer kills the loop: the catch handler's own log
write was outside any try, so under ErrorActionPreference=Stop a full disk, the
state a sick runner reaches, ended the telemetry in silence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Grade the telemetry off the API it calls, not off the log it prints

Printing and posting are decoupled, so assertions that counted printed lines
could see neither the request rate nor the backoff: hard-coding the staticness,
posting on every poll and never skipping a post all passed the suite. The legs
now run the real posting path against a local sink that records each request,
and each is required to trip a mutant of its own.

172 copied everything but the .ps1, so the launch helper returned early and the
real driver launching a real interpreter was covered by nothing. It now drives
that path, and proves the token reaches the watchdog and no test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber to 226: master took 225, and the union driver put 224 back

tests-list.mk carries merge=union, which never removes a line, so the rebase
resurrected the registration of a test file that no longer exists. The tell is
"make check" exiting 2 with no FAIL lines and no "# TOTAL:" summary at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the fork-failure claim from the watchdog's header

The 153,280-fork count behind it was an artefact of /proc/stat counting threads
and unrelated system-wide activity: the suite creates 26,158 processes, and the
engine and zlib blocks make 39.2% of them without holding one of the 34 deaths.
Spawning nothing is still worth having, as insurance rather than as a theory.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Let the new watchdog test survive the macOS and Windows legs

Two platform assumptions the Linux leg cannot see. The signal teardown
kills $BASHPID, which macOS's bash 3.2 does not define, so the kill was a
no-op and the suite exited 0 and posted "success" instead of "failure".
And /dev/full only stops writes for a process of the shell's own runtime:
MSYS cannot hand its emulated device to a native PowerShell, which then
writes as if to NUL, so the log-write-fatal mutant survived a leg that
had nothing to grade it with. The old guard also returned 1 under errexit
on a box with no /dev/full at all, macOS included.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cut the watchdog's verdict down to telemetry

GitHub already records the job's conclusion, so the commit status does
not need to carry a second one, and every attempt to make it do so
produced a way for it to lie. Gone: the one-shot -Post mode and its
retry, the driver's INT/TERM/HUP handlers, and the post-before-kill
ordering on the heartbeat path. Every status now goes out as success
carrying elapsed time, staticness, the test in flight and the counters,
and Get-CimInstance goes with them, its connect to a wedged WMI service
being unbounded and able to hang the one reporter left.

The tests around the reduced surface absorb what those parts were
carrying. 172 stubs PowerShell unconditionally, so a shell holding a
real pwsh and a token can no longer drive the real watchdog at
api.github.com (measured: zero outbound connections, against a control
that reaches 140.82.121.5:443), and 226 audits that every test able to
source the driver scrubs the credentials first. The staticness leg
watches long enough to see an unbounded value and grades it against
elapsed time one for one, so a clamped q now dies. Both mutant harnesses
grade on the assertion the mutant must trip rather than on a nonzero
exit. The workflow keys are read from the block that owns them through a
YAML parse, with one control per key and a round-trip control proving
that reformatting moves no verdict. The /dev/full leg runs only where a
write through the same interpreter really fails on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Grade the throttle on a ratio no slow box can move

The staticness leg carried an absolute post-count floor, which a loaded
Windows runner would have failed on its own. Every due tick logs its
line before the throttle decides, so posts against logged lines is the
same ratio at any tick rate; that check moves to its own 8s leg, and the
staticness window widens to 16s so the one-for-one comparison has room
a slow tick cannot shrink.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Kill the reporter with the wedge, and test what nothing was testing

The wedge branch ended the suite with TerminateProcess, which runs no EXIT
trap, so the PowerShell reporter outlived the step it reports on and kept
re-posting a frozen tail over the informative final status. It now goes
first, ahead of the kill.

Nothing covered the teardown either: deleting it left three orphan stubs
and 172 still passed. The stub now ticks a marker, so both the EXIT trap
and the new wedge kill are graded on a reporter that has actually stopped,
and 171 pins the order the kills fire in.

The workflow audit resolved the top-level permissions only, so a job-level
"permissions: {contents: read}" dropping statuses: write passed, and the
env keys were graded on truthiness, so a WATCHDOG_SHA swapped for the PR
head sha passed. It now reads the effective permissions of the job running
the suite and asserts the values, with a control per key. On the .ps1 side,
Get-WatchdogCounters had no assertion anywhere and the production cadence
was never read: the self-test pins both, and MaxSeconds is bounded below as
well as above, since a reporter that stops early leaves the wedge unreported.

The posting legs now assert the log never carries the token, controlled by a
mutant that logs one: tests/*.log rides into an artifact nothing masks.
Smaller: 172 takes its noexec skip before the command -v that reads false
there, and the driver scrubs the token before forking its first child.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 15:32:50 +00:00
Xavier Roche
b2cd2a51d2 An over-long FTP user name in a URL logs in as a different account (#1035)
* FTP: refuse over-long URL userinfo instead of logging in as another account

ftp_split_userpass() clipped a URL's "user[:pass]@" into user[256]/pass[256],
so a URL naming one account put a shortened name on the wire and httrack
mirrored whatever that account served, exiting clean. The split now reports a
field that does not fit and the link fails with STATUSCODE_INVALID, the
refuse-don't-clip contract ftp_command() follows for an over-long path (#1019).
A compile-time assertion pins both buffers to what a "USER <user>" control line
holds, so widening one cannot reintroduce the clip in the command formatter.

Nothing bounded the user field at the '@' either, only at the first ':' or the
end of the string, so ftp://user@host:21/f logged in as "user@host".

The ftp-userpass self-test asserted the truncation as intended behaviour.

Closes #1032

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: exercise the colon-less split, and the credentials on the wire

The ftp-userpass sweep always wrote a ':', so the branch where only the '@'
bounds the user name was seen once, three bytes long, yet that is the one
reaching memcpy() with an unbounded length. Sweep it as a third case, and add
the bare over-long and at-boundary URLs to the crawl loops.

The crawl half probes a dead port, so no login is attempted and a clip applied
after the split went unseen. A new local-ftp test drives the test server and
asserts the exact USER/PASS bytes on the control channel; it needs the server
to answer 331, since httrack sends PASS only on a 3xx.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the fixture-server reap, and drop a fail-open pipe from the FTP login test

stop_server ran a bare `wait` from an EXIT trap, so a fixture the kill never
reached blocked a test that had already passed until the harness timed it out
and called it a failure. reap_bounded gives up after REAP_GRACE, and costs
nothing when the child is already gone.

The new login test read the mirrored file through `find | head -n1`: head
exits on the first line, find takes SIGPIPE, and under pipefail that 141 aborts
the test through set -e with nothing printed at all. The mirror path is
deterministic, so assert it directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: skip the FTP login test on Windows, where it loses the x64 runner

230_local-ftp-userpass drives a real FTP login against the Python fixture, and
the x64 leg of windows-build has died every time it ran: the suite step sits
past its 45-minute timeout and the runner is lost to CPU/memory starvation,
while the Win32 leg of the same commits is green 3/3. Nothing separates the two
legs but matrix.platform, and none of the suite's own guards fired, so the cause
is on the 64-bit engine side and is not pinned yet.

Skip it there and pin the name in the driver's expected-skips ratchet, which
would otherwise red the job on an unannounced skip. The wire-level assertions
keep running on Linux and macOS, where they are what proves the credentials
reach the control channel unclipped.

Refs #1038

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 16:33:03 +02:00
Xavier Roche
562cd1c867 The test harness forks a sleep for every poll tick (#1034)
* Poll the test harness without forking a sleep per tick

Every deadline loop in the suite -- test-timeout.sh, wait_bounded,
run_with_timeout, reap_bounded -- ticked with sleep(1), one process per tick for
as long as the test ran. Measured over a serial run of the 202-test Windows
subset that is 5815 of the 25961 processes the suite creates; poll_wait replaces
them with a read(1) timing out on a fifo nobody writes to, leaving 68.

The tick can then be fine without being expensive, so wait_bounded notices a
finished crawl in 0.1s rather than up to a second, which is where the wall-clock
gain comes from.

Classify the tick by whether fd 9 is still open, not by the read's status: bash
3.2 (macOS) reports a timeout as 1, which is indistinguishable from a closed fd,
and poll_wait would retire its timer on the first whole-second tick while still
reporting the fd tick.

Left on the forked tick under MSYS: its fifos are emulated, and the leg #795 is
about is no place to find out how its select() behaves.

The new test is 225, not 224: #1029 took that number in parallel. The Windows CI
selector keys on the *_watchdog*.test topic glob rather than the number, so the
coverage follows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Put the sleep shim on a PATH MSYS can parse

MSYS hands out a drive-letter TMPDIR, so prepending the shim directory to PATH
produced an entry split at the colon: the shim was never found, every count read
zero, and the Win32 leg failed the one assertion it could still reach. The shim
now lives under a colon-free root, and count_sleeps proves it is reachable before
anything asserts on the counts, since an unreachable shim counts zero and reads
as 'nothing forked'.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the starvation cases real, and a dead fixture server loud

58_watchdog's starvation case went inert once the poll stopped forking: the shim
only ever reached the payload, so the guard measured a wall clock nobody had
stretched. It forces the forked tick like 105 does, and the override is now read
per tick rather than once, since both tests turn it on after testlib has ticked.
A poll-counted deadline calibrated to the 0.1s tick passes the old case and fails
this one.

local-crawl.sh had folded the two port-wait failures into one message, and 72 and
105 grep for it to skip an announce race: a fixture server that died at startup
was silently skipped in both. discover_server_port now returns 2 for that, and
the crawl fails loudly.

Also: a full minute for the port wait, since the loop it replaced self-extended
under the load that makes the wait long and the new one is a hard wall; a unique
fifo name, because two subshells share $$ and the loser opens a vanished path
that bash's <> turns into a never-blocking regular file; `true` over `:`, whose
redirection error kills a POSIX-mode shell; a reachability canary on the sleep
shim, which MSYS drive-letter TMPDIRs leave off PATH; and 227, renumbered off
master's 225.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 15:35:05 +02:00
Xavier Roche
a2a20ef684 Fix what Fedora's build patches around, and pin the installed file set
Fedora ships httrack with no patches, but its spec worked around three things
that were ours to fix, and its build broke twice in three weeks on changes to
our installed file set.

The metainfo declared a stock icon that is not one (stock names come from the
freedesktop Icon Naming Specification), which appstream-util rejects and which
also suppressed the merge from WebHTTrack.desktop that supplies the real icon.
libtest/readme.txt carried the pre-2005 FSF postal address. CI and
210_appstream-metainfo.test now run both validators.

225_install-manifest.test pins the installed file set, the class that broke
their 3.49.13 and 3.49.18 builds. It found a real bug on its first outing: the
.desktop files installed to $(prefix)/share/applications, so any --datadir
override put them where no desktop environment looks.
2026-08-06 10:50:56 +00:00
Xavier Roche
bfe0a4a7f7 The 16 px icon antialiases every stem to grey (#1031)
* Draw the small application icons on the pixel grid

The 16 px icon was the vector master downscaled, so every stem edge landed
on a fraction of a pixel: 74 colours in a 16x16 tile, four of them black.
The small sizes are now built in a viewBox equal to their pixel size, with
the outline coordinates rounded to whole pixels and no downscale after, so
the raster carries the field and the ink and nothing between them.

Still generated from the same Jost* master; the generator change is
httrack-works 90ceb7e2.

Test 228 counts the shipped raster's colours through a stdlib PNG reader,
since the suite cannot assume Pillow or ImageMagick on a build host.

Closes #938

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the new test's comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 11:33:03 +02:00
Xavier Roche
03c30e17ce The installed socket headers still need -D_POSIX_C_SOURCE under -std=c99 (#1028)
* The installed socket headers still need -D_POSIX_C_SOURCE under -std=c99

htsnet.h and htsopt.h were the last installed headers a strict-ISO consumer
had to prefix with -D_POSIX_C_SOURCE, and tests/206 carried an exception for
exactly those two. All of it came from one inline body: SOCaddr_inetntoa_
calls getnameinfo() and NI_NUMERICHOST, and no libc declares either to a
__STRICT_ANSI__ translation unit.

The body moves to a new src/htsnet.c and the exception in 206 goes away, so
all fourteen installed headers now compile under -std=c99 and -std=c11
against the strict-mode libc.

This turns a static inline into an exported symbol, an ABI addition rather
than a break, so VERSION_INFO gets a revision-only bump to 3:11:0 and the
soname and the Debian package stay put. proxytrack does not link libhttrack,
so it compiles htsnet.c in directly under -DHTS_NO_LIBHTTRACK, which leaves
the export marker bare there because MSVC rejects a dllimport definition.

Closes #1001

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin 206's header count to DevIncludes_DATA and unhide the strnlen check

The >= 10 floor let a header vanish from DevIncludes_DATA with 206 still
green, so #1001 could regress to "not installed" unnoticed: derive the
expected count from the list itself and name htsnet.h and htsopt.h beside
htssafe.h, since a dropped entry moves both sides of the count.

The "htssafe_strnlen_ is not libc's" preprocessor check sat inside the
gate asking whether a binary can run in TMPDIR, so a noexec TMPDIR or a
cross build skipped a check that needs no execution. Hoist it out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 08:59:27 +00:00
Xavier Roche
1a45e0237b An over-long FTP path or host name aborts the process instead of failing the link (#1029)
* FTP: refuse an over-long command instead of aborting on the copy

An FTP path or host name that survives the upstream clamps reached an
aborting strcpybuff and killed the process instead of failing the link.
The path case needs 1024 bytes exactly and nobody could reproduce it;
the host case is trivial, since the URL host was copied into a 256-byte
buffer with no check at all.

Every FTP control line now shares one FTP_LINE_SIZE, ftp_command()
reports a command that did not fit, and an over-long path or host fails
the transfer rather than being clipped into a request for another file.
send_line() also dropped the CRLF off a maximal command, and the
SIZE/MDTM probes reused the same builder.

Closes #1019

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the FTP control-line capacity and probe the host gate at its edge

The tests passed the fix without pinning it. Six mutants survived: widening
line_retr past the line[] it is copied into (that is #1019 verbatim), a stray
write one byte past ftp_command()'s bound, a host gate widened to reject every
host, and one moved off _adr[256] so a 256-byte host aborts again.

ftp_command_line() replaces the size argument at every build site with a
compile-time check that the buffer is FTP_LINE_SIZE, and the copy destination
is asserted equal to line_retr. The self-test now compares the whole poisoned
tail instead of one canary byte, and the crawl leg drives 255/256/257-byte
hosts plus a loopback outsider, so a widened gate no longer reads as a pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:51:33 +02:00
Xavier Roche
2a44dc1728 doc: note the --changes example is one captured run (#1027)
generator and date are genuine per-run fields (htschanges.c stamps
HTTRACK_VERSION and the current time on every write), so dropping them would
misrepresent the payload. Label the sample instead, so a reader does not
compare it against their own output as if it should match.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:51:25 +02:00
Xavier Roche
23e182281c The suite watchdog cannot see the engine on an emulated buildd (#1026)
* The suite watchdog cannot see the engine on an emulated buildd

Under qemu-user the process the kernel reports is the binfmt interpreter and
the engine is only its first argument, so the name matchers find nothing to
name or signal. Skip an interpreter before reading the command, gated on the
interpreter's own name: matching the argument unconditionally would take any
wrapper (strace, timeout) for the engine and signal that instead.

request_engine_backtraces only sends SIGABRT on Linux, so 105_suite-timeout
now asserts the death there and the "no stack mechanism known" report on the
platforms that get none, which is what Hurd's "GNU" uname takes.

Closes #1025

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Only shift past a real emulator, and cover the branch Hurd takes

qemu-img and its friends match "qemu-<word>" but take an image, not a
program, so shifting past them read a disk path as the process name --
into a list that is fed to kill. Exclude them, anchor the -binfmt form,
and give each a control row.

No CI leg reaches the no-stack-mechanism branch, which is the one the
Hurd buildds take, so drive it through a uname shim: it has to report
itself and leave the engine unsignalled.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 06:55:55 +00:00
Xavier Roche
3426ea35ea Release 3.49.18 (#1024)
* Release 3.49.18

Source-only release so packagers can pick up the test-suite build fixes:
3.49.17-1 was FTBFS on armhf, powerpc, hppa and loong64, and the suite
failed on a Fedora build root with no ps. The engine is untouched, so
VERSION_INFO moves by revision alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Wrap the 3.49.18 changelog entry under 80 columns

lintian's debian-changelog-line-too-long fires on two lines of the new
entry, and the CI gate runs --fail-on=error,warning. The sid perl
transition that had been masking this gate has cleared, so it is live
again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Sharpen the 3.49-18 release notes

The #1021 line named the symptom and implied the remedy was a louder
failure; the fix is a /proc fallback, so the diagnostics work. Add the
internal catch-all the 3.49-17 block carries, so the cross-arch matrix
appears to the upstream audience and not only in debian/changelog.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:43:50 +00:00
Xavier Roche
0274e016af Four architectures cannot build the altstack test shim, and nothing before the buildd sees it (#1023)
* Fix the four 3.49.17 buildd failures and cross-compile for those arches in CI

The LD_PRELOAD altstack shim did not build on Debian's 32-bit time64
architectures, and 183 read a frame count no unwinder there can produce.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Install the cross libc, and keep a trace-less report a failure

Review found the frame floor took a report naming no frame at all for a
weak unwinder, and the cross jobs had no target headers.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the cross jobs a libssl-free configure and assert what they built

No ports architecture has a cross libssl, and an empty TESTS= would have
passed having built nothing.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Add deb-src in place: a second stanza collides on the keyring spelling

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Extract zlib unpatched: Debian's arch patches need Debian's rules

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Turn off zlib's s390x vector CRC: the .dfsg repack drops its sources

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop zlib's s390x vx object too, not just its define

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the raw_mmap comment

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the raw_mmap comment inside clang-format's width

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:43:51 +00:00
Xavier Roche
22fcf9ffff The suite watchdog dumps nothing on a host without ps (#1022)
* The suite watchdog dumps nothing on a host without ps

Fedora's build root ships no procps, so every process list in the hang
diagnostics came back empty: the guard still fired, but named no process
and could not signal the wedged engine for a stack. Read /proc directly
when ps is missing, and say so when neither source is available.

Closes #1021

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Harden the no-ps fallback and its test

Review follow-ups: a ps that exits 0 while listing nothing (hidepid, a
locked-down container) now falls back too, whitespace inside an argv can
no longer shift the columns the consumers match on, a failed /proc open
stops leaking to stderr, and a bogus CLK_TCK can no longer abort the
shell. The test leg pins the header line, the column order and the group
filter, and drives request_engine_backtraces itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* kill -0 succeeds on a zombie, so the SIGABRT check failed in a container

The fake engine is reparented once its subshell exits, and a container's
pid 1 does not always reap it, so the signalled process stays visible to
kill -0. Read its state from /proc instead: gone or Z both prove the
signal landed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:16:14 +00:00
Xavier Roche
8d7c9caf61 Release 3.49.17 (#1020)
* Release 3.49.17

Bumps the four version spots, the AppStream release block and VERSION_INFO
(3:8:0 -> 3:9:0, revision only: #991 added an exported symbol and #1018 changed
no POSIX ABI, so the soname stands and there is no Debian package rename).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cite issues rather than PRs in the release notes, and fill three gaps

House style cites the issue a change closes; five lines named the PR instead.
The pkg-config line cited #978, which is the macOS relocation report and says
nothing about pkg-config.

Adds the command-box escaping fix (#998), the struct addrinfo scoping (#987)
and the four-architecture build failure (#1015), all user-facing and none
covered by a neighbouring line. The crash-report line now names the loader
case it was citing #996 for, and drops #997, which has no user-visible symptom.
AppStream said C:\ab was mangled as C:\folder, but the decode needs two hex
digits after the backslash, so \fo never matched.

VERSION_INFO's rationale named only #991; #1005 exported escape_control_url
too. The bump stays revision-only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:05:18 +02:00
Xavier Roche
a7201b5d22 A relocated install still cannot find libhttrack on macOS (#1018)
* A relocated install still cannot find libhttrack on macOS

#964 gave httrack and htsserver a loader-relative rpath but opted Darwin
out: dyld expands @rpath only for a load path spelled that way, and
libtool stamps the dylib with the absolute libdir, so the entry would
never be consulted.

The id comes out of libtool's own archive_cmds template rather than from
LDFLAGS, so configure rewrites that template when the gate is on, and the
dylib links as @rpath/libhttrack.3.dylib from the start. The Darwin probe
now builds a library carrying such an id and runs a program that needs
it, so a dead LC_RPATH cannot pass for a live one; an unrecognised
template leaves the whole thing off.

Anything linked against libhttrack after this records the @rpath id and
must supply an rpath of its own. That is not an soname break, so
VERSION_INFO and the Debian package name are untouched.

Closes #978

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Tighten the Darwin rpath gate and what 219 proves

Arm the install_name rewrite only when both libtool templates take it, so a
future divergence between archive_cmds and archive_expsym_cmds disarms the gate
rather than letting the -export-symbols path stamp an absolute id.

219 now reads the executables' LC_RPATH too, which catches an id moved to @rpath
while the binaries carry no entry answering it, and counts the libhttrack
dependency per binary instead of totalling across both.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Ship a pkg-config file for libhttrack

The @rpath install_name is consumer-visible: anything linking against
libhttrack afterwards records @rpath/libhttrack.3.dylib and needs its own
LC_RPATH. libhttrack.pc is the standard pairing, handing out the include path
and, for a non-system libdir, -Wl,-rpath,${libdir}. The gate is
configure's existing hts_origin_sysdir, so an rpath into /usr/lib stays out.

222_pkgconfig-consumer.test stages an install, queries the staged .pc, then
builds and runs a consumer off it; the run is what proves the id and the rpath
resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the pkg-config file out of the macOS bundle

An .app ships runtime, not development files, and the .pc names the staging
prefix the bundle check rejects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the comments added with libhttrack.pc

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Let 222 run against a sanitized libhttrack

The consumer is built from the .pc alone, so ASan's runtime is not first in its
library list; the interposer tests already turn that check off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make libhttrack.pc link statically, and 222 able to see it

--static --libs handed out -lcrypto before -lssl, with -lz and -lzstd ahead
of the libcrypto that needs them, so an archive link left 76 symbols
undefined. Name the dependencies as pkg-config modules in Requires.private,
which orders them and carries any -L an out-of-the-way prefix needs, falling
back to the probed link line where a dependency ships no .pc. libhttrack-dev
gains the matching -dev dependencies: pkg-config fails outright on a module
whose own .pc is missing.

Test 222 could not see any of this. --define-variable overrides an
assignment the .pc already carries, so its relocation overwrote prefix,
libdir and includedir before anything was asserted: a hard-coded
libdir=${exec_prefix}/lib, which breaks Debian multiarch, passed unchanged.
It also only grepped --static --libs and then linked with the dynamic ones.
Assert the un-relocated values, and build and run a real static consumer.

The rpath the .pc hands out now answers to the same gates the executables'
own rpath respects, so --disable-origin-rpath and an archive-only build stay
rpath-free; 196 covers all six cases. On Darwin 222 skips where configure
left the dylib an absolute install name, and 219 skips on an unset
abs_top_builddir rather than erroring, as 222 already did.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim test 222 to the assertions that carry a mutant

The pkg-config consumer test had grown to 225 lines, much of it belt and
braces around checks that already fire elsewhere: the -I/-L extraction and
its stage-relocation loop, the -lz grep on --static --libs, and the rpath
gate check that test 196 now reads straight out of the generated .pc in all
six of its configure cases.

Each of the eight mutants the test was written for still dies: a wrong
libdir, includedir or prefix in the .pc, a multiarch build hard-coding
libdir=${exec_prefix}/lib, the rpath flag dropped from Libs:, the old
Libs.private order with no Requires.private, Cflags one level too high, and
a .pc generated but never installed. The dropped rpath is now caught by the
run itself, since the consumer is launched with LD_LIBRARY_PATH unset
wherever the .pc promises an rpath.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 10:25:32 +00:00
Xavier Roche
77182813a5 A crawled page can read the WebHTTrack session id and drive the control panel (#1011)
* Sandbox mirrored pages so their script cannot reach the control panel

htsserver serves the browsed mirror under /website/ on the same origin as
the panel under /server/, where an unauthenticated GET renders the session
id that gates commands. Send a CSP sandbox without allow-same-origin on
mirrored responses only, so a crawled page's script lands in an opaque
origin.

Closes #877

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the blind-POST channel and make the session id unguessable

The CSP sandbox added in this branch narrows #877 but does not close it: a
sandboxed page can still fire a CORS-safelisted no-cors POST at /server/, and
the command runs even though the reply is opaque. Refuse any POST whose Origin
names something other than the panel itself; a missing Origin stays allowed, as
non-browser clients send none.

The session id that authenticates those commands was an md5 of time(NULL) and
rand() seeded from the same second, guessable from the "Mirrored from" stamp
every mirrored page carries. Seed it from /dev/urandom, or RtlGenRandom on
Windows, and refuse to start when neither answers.

Also add allow-downloads to the sandbox token list, without which Chrome blocks
a mirrored .zip or .pdf link.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the origin refusal, the unguessable id and a subdirectory mirror

Test 220 grew a mirrored page one level down and a post-save panel probe: both
a gate keyed on server state and one that only sandboxes mirror-root files
passed the earlier version. It now also drives the Origin refusal end to end,
asserting the refused save binds no mirror. Test 221 starts two servers inside
one second and requires their ids to differ, which the clock-seeded version
cannot do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the two servers distinct ports up front

Two sequential ephemeral-port picks can return the same number; binding both
sockets at once cannot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber the session-id test clear of 221

221 landed on master as the FTP control-byte crawl test (#1012).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 09:33:55 +00:00
Xavier Roche
e2a57d6c07 WebHTTrack settings pages interpolate values into double-quoted attributes unescaped (#1013)
* WebHTTrack settings pages interpolate values into double-quoted attributes unescaped

#988 escaped the templates' element bodies and event handlers but left
attribute values alone, because cat_html_escaped() keeps the double quote
raw on purpose: the tooltips are written title='${html:LANG_x}'. The
settings pages put about seventy values inside double-quoted attributes,
where a " opens a new attribute.

Add a fourth interpolation mode, attr:, escaping < > & ' " as entities and
nothing else, and point the 230 double-quoted attribute positions at it.
file.html's file:// href stays on the html-urlescaped mode, which now emits
%22 for the quote the way unquoted: already did, and step2.html's in-script
literal moves to js:. Test 185 now requires attr: in a double-quoted
attribute and js: inside a script block; 217 drives hostile values through
three templates and a translation.

Closes #989

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the two attribute sites the template sweep could not reach

The liststr: format builds its own <option value="..."> in C, so no
template edit covers it, and it escaped with the helper that leaves the
double quote raw. Its values are the project directory names and the
Category= line of each project's winprofile.ini, both of which can hold
a quote.

The save side kept hex-decoding a backslash plus two hex digits, so a
path typed as C:\ab displayed correctly but was written to ~/.httrack.ini
as C: plus byte 0xAB. The ini writer has no inverse for that escape; the
one caller that needs it is a template literal (${test:proxytype:\3A//}),
so the decode now applies only there.

Test 185 classified attributes with a regex that missed a spaced =, an
unquoted or single-quoted value, a template outside html/server/*.html,
and an output-mode region the file never closes; the counting floors it
asserted also survived a count-preserving rewrite. It now walks tags,
asserts the property on every site it classifies, and proves it can see
each of those forms on a synthetic template first.

Test 217 gains a Latin-1 byte in a translation, the POST-render-POST
round trip, and the two fixes above.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the comments the last commit added

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 08:20:14 +00:00
Xavier Roche
502ce78413 FTP URL bytes reach the control channel unchecked (#1012)
* FTP commands are built from percent-decoded URL bytes with no control-character check

A path or userinfo carrying %0d%0a decodes back to raw CR and LF that reach the
FTP control channel unchecked, so a crawled ftp:// link can append commands of
its own to the ones HTTrack sends.

Screen the command in send_line(), the choke point every command goes through,
the way htsproxy.c already screens the CONNECT authority, and fail the transfer
up front in run_launch_ftp() so the caller reports the error instead of waiting
out a reply to a command that was never sent.

Closes #1010

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Share the control-byte predicate, quote the LIST argument, cover the driver

The check landed as a third copy of a rule htsproxy.c already implements twice
(CONNECT authority, SOCKS5 host); move it to hts_is_control_free() in htslib.c
and call it from all three. The FTP path is now decoded once into ftp_path and
screened there, so SIZE and MDTM stop relying on their argument happening to be
a prefix of the screened RETR line.

"LIST -A %s" pasted the path in bare, so a space in it gave the server a second
token: ftp://host/x%20-la/ went out as LIST -A /x -la/. Both listing and
retrieval now go through ftp_command(), which also quotes a leading '-'.

The self-test drove send_line() directly and never entered run_launch_ftp(),
so deleting the pre-screen passed every assertion while a real crawl stalled on
a 300s read. 221_local-ftp-ctrlchars.test crawls the injected URL against
tests/ftp-server.py and requires a prompt, logged failure; the self-test now
builds each command from its URL path instead of asserting strings by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Dump the mirror when the quoted listing assertion fails

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 07:32:26 +00:00
Xavier Roche
a3e1d3dd40 3.49.16-1 fails to build on four Debian architectures (#1017)
* Fix the four 3.49.16-1 Debian buildd failures

armhf and loong64 cannot unwind out of the frame that faulted on the guard
page, so 180_crash-stack-overflow saw no repeated address and read that as
"never recursed". Judge the recursion only where the report is deeper than
the ordinary-fault control, which keeps the mutant that stopped recursing
failing on x86-64.

hppa needs ~150s per configure run, so 151_bash-shell-validate and
196_install-rpath-gates blew the 600s per-test budget meant to catch a
wedge. They now pace themselves against that budget and skip out rather
than take the build down with them.

hurd-i386 has a network, so the online crawl probe said yes and the crawls
then failed on DNS. Debian Policy 4.9 forbids network access during a build
anyway, so stop asking for it. While there, install crawl-test.sh's traps
one at a time: SIGSTKFLT is Linux-only and its absence made the whole trap
command complain on every Hurd run.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close two vacuity holes the review found in the new gates

Test 180 compared the stack trace against the segv control's depth, but the
two call chains differ by a frame that only -O2 tail-calls away: at -O0, -O1
and -Og the control is 12 frames and a crash_stack that faults in its own
body is 11, so that mutant slipped through the gate. Use an absolute floor
instead. The handler contributes 3 frames and any non-recursing crash_stack
still unwinds 11, so 6 sits clear of both.

skip_if_out_of_budget projected the step that just ran across every step
left. Test 196 shares one config.cache, so its first step costs several
times the rest and the projection over-estimated by ~2.5x, skipping runs
that fit -- including the hppa case this is for. Look one step ahead
instead, at 1.5x the last step's cost.

Both gates now have coverage in 105_suite-timeout.test: dropping the export
or the skip fails it.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 07:19:09 +00:00
Xavier Roche
8fcc9d8cae A non-PIE build's crash report still names no frame of the executable (#1014)
* Crash reports from a non-PIE build name no frame of the executable

dladdr() fills dli_fbase with the mapping start and dl_iterate_phdr() fills
dlpi_addr with the relocation bias; the two agree only for a PIE. On an
ET_EXEC the /proc/self/exe override added by #984 never fires, and the
offsets handed to addr2line are the load address too low. Match the main
program by load range, and subtract the bias rather than the base.

Also reject /proc/self/exe when it names the loader rather than the program
(#996), and point the two copies of that resolution at each other (#997).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Reject /proc/self/exe by identity, not by argv[0]'s basename

dladdr() returns argv[0] verbatim for the main program, so the #996 guard
compared /proc/self/exe against a caller-controlled string. Whenever the two
basenames differ and argv[0] carries no slash -- update-alternatives, a Nix or
snap wrapper, a busybox-style dispatcher, a cosmetic argv[0] -- main_path was
dropped, access() then failed on the bare name, and the executable's whole
module vanished from the report.

Reject /proc/self/exe when it stats to a NAMED link-map entry instead: the main
program's entry is the unnamed one, so that catches the loader and nothing else,
and keeps every symlink and rename. Test 218 gains the renamed-symlink launch.

Also reset main_hi with main_lo, and match the mktemp path literally rather than
as an ERE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 07:00:43 +00:00
Xavier Roche
38eb901cea A worker thread that exhausts its stack still dies with no diagnostic (#991)
* A worker thread that exhausts its stack still dies with no diagnostic

sigaltstack() state is per-thread, so the alternate stack #866 gave the main
thread does nothing for the engine's workers, and the crawl recurses there.
A worker running out of stack leaves the kernel no room for a signal frame,
and it is killed outright with no report.

htsthread gains a pair of hooks the thread start routine runs around the
worker body, and httrack registers the altstack installer and its release.
Releasing matters now that it is per-thread: 64kB of mapping per worker,
never given back, adds up over a crawl.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the release half of the worker alternate-stack hooks

183 could only see the install: crash_stack_thread never returns, so
hts_entry_point never reaches the leave hook and every release-side bug was
structurally invisible. altstackprobe grows an LD_PRELOAD tracer for
sigaltstack(), mmap() and munmap(), and a second leg drives -#test=threadwait,
whose workers do return: 25 stacks installed, 24 handed back, each unmap right
behind its own SS_DISABLE, and the main thread's kept.

Also: hts_set_thread_hooks refuses half a pair, crash_threadstack aborts on a
spawn failure instead of leaving the test to blame the handler, and the -#c
kinds list clips rather than aborts now the table has outgrown its 64 bytes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Gate the alt-stack tracer on Linux: mmap64 and off64_t are glibc-only

The macOS leg builds tests/altstackprobe.c like every other platform, and it
has neither off64_t nor mmap64. SYS_gettid is Linux-only too, so gating just
the glibc half would still not compile there.

The whole tracer now lives behind __linux__, with mmap64 behind __GLIBC__
inside it: musl has no LFS split and its plain mmap() is already the one the
engine calls. Nothing outside the gate changed, so on macOS the file is the
one that was already building, plus four headers it has. The trace is what
183's second leg reads, and that leg is Linux-only by the same reasoning.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 06:12:20 +00:00
Xavier Roche
fbd825d4dd A crawled link can inject header lines into the requests HTTrack sends (#1008)
* Percent-escape network-derived URL pieces in outgoing request headers

A crawled href carrying &#13;&#10; keeps raw CR and LF in the engine's link
buffers: htsparse.c escapes control bytes before hts_unescapeEntities() decodes
the character references, so the escape runs too early to see them. The request
line survived that because it re-escapes 'fil' at emission, but four other
interpolations in http_sendhead() emitted the same data raw, letting a crawled
page add header lines of its choosing to the requests HTTrack then sends.

Escape at emission with escape_check_url(), the same escaper the request line
already uses, so a poisoned URL now goes out percent-encoded in every field
that carries it. The scratch buffer is sized like the url_adr/url_fil buffers
it mirrors, so escaping can only shrink what reaches the request buffer.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Cover the Referer host escape, and state the buffer bound accurately

The proxied leaf pages had no outbound links, so a poisoned host never became
anyone's referer and the Referer host escape went ungraded: reverting it alone
still passed. Give those pages a link so the host reaches a Referer, which also
adds a positive control on the plain proxied host.

The scratch buffer comment claimed escaping only shrinks its input, which is
backwards: percent-escaping expands up to 3x. What holds is that the result is
capped at the source buffer's own size, so the worst case emitted is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Grade the two escapes the probe could not reach, and the bare-LF split

The checker split each captured request on CRLF only, so an injection
that used a bare LF stayed glued inside one line. It reported "none
carrying an injected header" on a wire that carried one; real servers
and proxies do terminate a header line on a bare LF. Split on LF and
strip the CR, the way a tolerant parser reads it.

The proxy leg asserted nothing about the Referer, and no leg reached
the ftp-through-proxy request line at all, so those two escapes were
graded only by the catch-all. A crawl change that stopped following
either link would have un-covered them silently, which is how the
referer host went ungraded the first time round. Give the proxied root
an ftp:// link, and assert that both requests happened and carry the
escaped authority.

The crawl status was discarded, so an assertf abort would have been
graded on whatever bytes it managed to send first. Fail on the deadline
and on death by signal, and add the missing signal trap so the probe
server cannot outlive an interrupted run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Do not name a signal for an exit status that may be a panic

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 20:09:55 +00:00
Xavier Roche
c36a5428d7 Release notes understate the #841 cache-header overflow (#1009)
The 3.49.16 entries call it an unbounded sprintf and count it as one of "two
ProxyTrack buffer overflows", but #934 bounded that block in the engine's own
cache_add too, where the field caps sum past the 8192-byte stack buffer and the
values come off the wire. Say so, so a reader does not conclude only proxytrack
was exposed.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 22:04:59 +02:00
Xavier Roche
2945f0a5e4 The WebHTTrack progress panel renders crawled URLs into HTML and JavaScript unescaped (#988)
* Escape crawled URLs in the WebHTTrack progress panel

refresh.html interpolated ${info.state}, ${info.name} and ${info.file} as
raw HTML and ${info.url_sav} into a single-quoted JS literal, so a link
path holding < > or ' ran as script in the panel origin. Point them at
htsserver.c's html: and js: modes, and sweep the other templates for the
same shape. Widen 185's scan to any unescaped handler or element-body
interpolation, and add 186 to probe the rendered panel during a crawl.

Closes #973

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the panel's name column in the escaping test

186 asserted only on the leaf of the crawled path, so info.name could go
back to raw and still pass: 185 cannot see it either, because ${unquoted:}
sits in the directive regex it skips. The link now carries metacharacters
in a directory component too, short enough to survive the engine's 40-char
elision of that column, and the poll breaks on either form so the positive
control stays honest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make test 186 executable, like its siblings

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Re-arm the output mode that ${do:end-if} clears

${do:end-if} sets outputmode to 0 rather than restoring what it replaced
(htsserver.c), so the if-not-empty block partway down step4.html's command
textarea drops the ${do:output-mode:html} armed above it, and the 17
${unquoted:} settings below render raw. Posting maxtime or url2 as
A</textarea><img src=x onerror=alert(1)>B breaks out of the textarea.
finished.html's <pre> has the same shape; those two are the only ones.

--sitemap-url moves from html: to arg: like its quoted siblings, since
cat_html_escaped() escapes neither " nor \.

Test 185 had unquoted: on its directive skip list, which is how all 17
sailed past it, and now also rejects an html: value on the command line.
Test 83 reads back the escaped forms at runtime.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 19:42:37 +00:00
Xavier Roche
2312d711e6 A control byte in a link makes the crawler fetch a URL the page never wrote (#1005)
* A vertical tab or form feed in a link is deleted, and the crawler fetches a URL the page never wrote

The link scan treated any byte below 32 as the end of a URL unless is_space()
accepted it, and is_space() counts VT and FF. Neither is among the CR, LF and
TAB stripped from the URL further down, so both survived into
escape_remove_control(), which deletes them: href="/a<VT>bc" resolved to /abc.
The other 28 control bytes ended the scan where they stood, so the crawler
fetched the truncated prefix instead.

A control byte the existing end-of-URL tests did not already consume now costs
the link. CR, LF and TAB keep their exemption because the parser strips them a
few lines later, and the check sits below that end-of-URL logic rather than
above it, so a VT or FF ending an unquoted attribute value still ends it.

Closes #982

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Percent-encode a link's control bytes instead of truncating or deleting them

A control byte inside a link ended the URL where it stood, so
href="/s<SOH>oh.html" fetched /s. VT and FF came out worse: is_space()
counts them, so the scan carried them into the URL, and neither is among
the CR, LF and TAB stripped further down, so escape_remove_control()
deleted them and href="/a<VT>bc" resolved to /abc.

Both now percent-encode as a browser does. The encoding runs on the
extracted link, where the destination capacity is visible, because a byte
grows to three; a link whose encoded form no longer fits is dropped rather
than clipped to a URL nobody wrote.

Closes #982

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the wording of the encode-site comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert that a link's CR, LF and TAB are still stripped

Test 158 covered the encoding half of the fix but not the stripping half,
on a line this branch rewrote. Making the strip unconditional, so CR, LF
and TAB percent-encode like every other control byte, left the whole suite
green. The new href carries all three and only its stripped target exists
on the server, so that mutant now fails on the 404 it provokes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Strip a link's end control bytes, and bound the query's growth too

Review of the encoding change found two holes.

A control byte at either end of a link used to end the scan there, which
threw away nothing and gave the right URL; encoding it instead turned
href="page.html<SOH>" into a 404. Both ends now strip every C0 control or
space, as the WHATWG URL parser does, so only interior bytes are encoded.
The leading loop stops on the terminator the trailing loop just wrote,
since '\0' is <= ' ' and it would otherwise read on into the previous
link's leftovers and fetch those.

The path bound missed the query, which is escaped separately and appended
back by a helper that clips silently. An absolute link whose path and
query together outgrow the buffer was landing just under the URL-length
gate and being fetched truncated; it is now dropped like the path case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 19:37:14 +00:00
Xavier Roche
7bff8133bb Two installed headers declare symbols the library does not export (#990)
* Two installed headers declare symbols the library does not export

htsbasenet.h declares openssl_ctx and htsnet.h declares
hts_dns_set_resolver_backend, both hidden by -fvisibility=hidden and both
installed into $(includedir)/httrack. A consumer that includes either header
and uses the name compiles, then fails to link.

Move both behind HTS_INTERNAL_BYTECODE, the guard the other internal
declarations in the installed set already use. Every in-tree caller defines
it, so the engine build is unaffected.

Test 207 installs the headers, derives the hidden set from the library's
symbol table minus its dynamic table, and links a probe for every hidden name
a consumer can reach through an installed header.

Closes #977

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 207's floor against a vacuous run counts only header-static symbols

Every one of the 31 symbols that reaches the link probe is a static function
defined in the installed header itself (abortf_ and the rest of htssafe.h,
StringOom_ from htsstrings.h). The probe object compiles its own copy, so the
link cannot fail whatever the library exports, and "probed >= 5" holds without
the harvest ever seeing an extern declaration. Cutting the header loop down to
htssafe.h alone leaves probed at 7, and the test still passes with 13 of the 14
headers unchecked.

So plant a leak: a canary header declaring the hidden symbol the negative
control has already shown cannot link. The loop has to report it, which puts
the install, the harvest, the intersection and the link on the same path a real
leak takes. The cut-down loop now fails.

The HTTRACK_SHLIB comment also blamed the skip on static-only builds. macOS
skips as well, where libtool names the library .dylib.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim test 207's comments to the house one-line default

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Declare struct addrinfo at file scope in htsnet.h

Without it the tag in the resolver-backend prototypes is a fresh type scoped
to its own declaration wherever <netdb.h> has not already declared it.

Closes #987

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 19:10:38 +00:00
Xavier Roche
e5985903e3 The crash report can hang: the symbolizer forks from the signal handler (#985)
* Crash report can hang: the symbolizer forks from the signal handler

fork() runs the pthread_atfork prepare handlers before it forks, and glibc's
malloc registers one that takes every arena lock. A signal raised inside
malloc (glibc's own heap-corruption detector aborts from exactly there, and
SIGABRT is wired to sig_fatal) then leaves the handler blocked on a lock its
own thread holds, and the report is lost.

vfork() bypasses the atfork handlers, at the price of a child that may only
issue syscalls: the PATH search execvp() performs allocates, so the symbolizer
is resolved once at startup and the child execv()s an absolute path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 182 pins a 64-bit address width, so it fails the i386 leg

addr2line -a zero-pads the address to the target's pointer size: 8 nibbles
against a 32-bit ELF, 16 against a 64-bit one. The gate is "is addr2line
installed", not the architecture, so the {16} match fails a correct build on
the gcc -m32 leg and on Debian's 32-bit buildds. Test 80 already matches the
same line unanchored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Spawn the symbolizer with posix_spawnp() instead of fork()

posix_spawnp() reaches the same child through clone(CLONE_VM|CLONE_VFORK),
which runs no pthread_atfork handler, so the arena lock a signal raised inside
malloc already holds cannot deadlock the report away.

It also does the PATH search itself and reports an exec failure through its
return value, so the hand-rolled find_on_path(), the two statics caching its
answer and the BT_NO_SYMBOLIZER exit protocol all go. Building the file actions
in hts_backtrace_init() keeps the only allocation off the crash path, and with
the rewrite happening in the parent the addr2line to llvm-symbolizer fallback
comes back.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the symbolizer output lands on the report fd

The child's stdout redirect is now a file actions object built at startup, so a
wrong target sends symbolized frames to the crawler's own stdout instead. The
existing checks merge both streams and cannot see that.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Take the report fd out of hts_print_backtrace()'s signature

The child's stdout redirect is prebuilt at init against stderr, so passing any
other fd quietly skipped symbolization on a runtime check. The one caller
passed stderr, and htsbacktrace.h ships with the program and not with the
installed dev headers, so there is no ABI cost to making the wrong value
impossible to write.

Test 182 also picks up test 80's ARM gate. A build with no unwind tables
prints "unwinding failed" instead of the OS notice, which 182's skip check
misses, so it would have gone red on armhf rather than skipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert a symbol name in test 182, not just the frame shape

An all-"??" symbolization regression still emits the address lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 18:52:04 +00:00
Xavier Roche
ed8a376278 escape_remove_control() leaves the original tail glued to the result (#981)
* escape_remove_control() leaves the original tail glued to the result

It compacted the non-control bytes downward in place but never wrote the
new terminator, so any input it actually shortened came back as the
compacted head plus whatever the old tail left behind: "/a\013bc" came
out "/abcc". The function is HTSEXT_API and declared in the installed
httrack-library.h, so the broken contract is a public one.

The parser can reach it: VT and FF pass the link scan as is_space()
members and the later strip only removes CR, LF and TAB, so a page with
href="/a<VT>bc" fetched /abcc before this.

Closes #974

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* escape-control self-test lets a shifted cut and a stray write through

Grading mutants against the new test found two survivors. No vector held
a space, so moving the loop's cut to `c > 32` stripped it and the suite
stayed green. Nothing reached above 0x0c either, so `c >= 13` passed too.
The added vectors pin both sides of the cut, and the second drops two
bytes instead of one.

The canary read one byte at `inlen + 1`, so a stray write two or more
past the compacted end went unseen. The buffer is already poisoned in
full, so scanning the rest of it costs nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 18:19:30 +00:00
Xavier Roche
10fa9315b3 Release 3.49.16 (#1006)
* Release 3.49.16

Version bump across configure.ac, htsglobal.h, version.rc and the AppStream
metainfo, plus the release notes for history.txt and debian/changelog.

VERSION_INFO 3:7:0 -> 3:8:0: no installed struct or exported signature moved
this cycle, so the soname stays .so.3 and Debian needs no package rename.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Correct the 3.49.16 release notes

The AppStream metainfo describes WebHTTrack and is read by Linux software
centres, so it now lists only WebHTTrack-visible changes: the macOS bullet and
the nine option-dialog strings (WinHTTrack-only, no hits outside lang/) are out,
the charset re-encoding is in. Drop the task-switcher claim, which webhttrack
cannot make: it is a script that launches a browser.

Fix two over-claims in history.txt. Only the trailer section after the
terminating chunk was rejected, not any chunked page. The cache aborted the
mirror on an over-long URL rather than merely refusing one it had accepted.

Cite #901 rather than the PR that closed it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 17:22:33 +00:00
Xavier Roche
67afcef6c1 Installed headers do not compile under -std=c99 (#986)
* Installed headers do not compile under -std=c99

htssafe.h spells typeof without the underscores in three GNU-only
branches and calls POSIX strnlen from its inline bodies. __GNUC__
survives -std=c99 but neither the keyword nor the declaration does, so
htssafe.h and the three installed headers including it stop a strict-ISO
consumer.

Use __typeof__, and route the inline helpers through a private wrapper
that calls the libc strnlen wherever it is actually declared. tests/206
compiles every installed header under -std=c99 and -std=c11.

Closes #972

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 206 asks for more POSIX than it needs

The two socket headers get -D_POSIX_C_SOURCE so getnameinfo() and
NI_NUMERICHOST resolve. At 200809L that also un-hides strnlen, so
htsnet.h and htsopt.h stopped covering the strnlen half of #972: delete
the __STRICT_ANSI__ guard from htssafe_strnlen_ and both still compile
clean. 200112L exposes the networking surface and leaves strnlen hidden,
so both catch that mutant again.

The level is probed rather than hard-coded, because only glibc is
verified here. A libc that needs 200809L still passes, and the test
reports the lost coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 206 never expands the macros it is meant to protect

Two of the three __typeof__ sites this PR fixes sit in
htsbuff_must_be_array_, which is a macro. A macro body only reaches the
compiler where it expands, and 206 compiles a bare #include of each
header, so reverting either of those two lines to plain typeof left the
test green. Only HTS_IS_CHAR_BUFFER was covered, and then only because
htsarrays.h happens to expand it.

Add a translation unit that instantiates htsbuff_array(), htsbuff_catn()
and strcpybuff(). Reverting any of the three sites now reds 206; before
this, only one of them did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 206 never runs the byte loop it protects

206 is -fsyntax-only throughout, so htssafe_strnlen_'s fallback was covered
for parsing and nothing else: changing its bound to `i <= maxlen` or its
result to `i - 1` left the test green. That loop is the strict-ISO
consumer's strnlen, and this header bounds copies from the wire.

Compile and run one strict-ISO TU that differentials htssafe_strnlen_
against memchr over every content/bound pair up to 8 bytes, on a poisoned
buffer so an overshot bound cannot land on a NUL and read as correct. The
gate is asserted first, or a build that fell back to libc would grade libc.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fail rather than silently drop strnlen coverage in test 206

The 200809L fallback un-hides strnlen, so it would report PASS while covering
less than it claims. No CI libc reaches it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Mark hts_record_assert_memory_failed HTS_UNUSED (#1002)

A C++ consumer of the installed headers hits -Wunused-function on this
static function: it is only called from a TypedArray macro expansion,
so a translation unit that just includes htsarrays.h leaves it unused.
g++ only runs that check on a real compile, not -fsyntax-only, which
is why 206's C loop (all -fsyntax-only) missed it; the new C++ leg
uses a real -c compile for exactly this reason. htssafe.h already
tags the same shape HTS_UNUSED.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 16:34:18 +00:00
Xavier Roche
8f453a4597 Fix missing letter in Slovak translation of "search" (#1004)
lang/Slovak.txt line 166 read Hľdanie where the windows-1250-encoded
Slovak word is Hľadanie, missing the "a". Pre-existing typo left over
from #963's charset-declaration move, unrelated to that PR's changes.

Closes #994

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 16:33:30 +00:00
Xavier Roche
8f0489accd Pin addr2line to the C locale in test 157 (#1000)
The frame match keys on addr2line's " at " separator, which binutils
translates: under fr_FR.UTF-8 it reads " a" and the test fails on a
report whose frames are all present.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 16:09:39 +00:00
Xavier Roche
5ab645e370 Nothing checks that the data directory comes from the OS path, not argv[0] (#992)
* Nothing checks that the data directory comes from the OS path, not argv[0]

Reverting the startup lines to argv[0] alone left the whole suite green: the
part -#test=datadir cannot see is which path gets handed to
hts_resolve_datadir(). 215_engine-datadir-ospath runs a copy of the engine from
a fabricated bin/ under two lying argv[0]s, one naming a decoy tree that has
templates of its own and one a bare name as a PATH lookup leaves, and asserts
the templates the run resolved are the copy's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fall back to src/httrack when the build needed no libtool wrapper

--disable-shared leaves the program in src/ with no .libs copy, and the test
then failed on a missing HTTRACK_BIN instead of running.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Plant a binary at the decoy path so an existence-checked argv[0] is caught

The decoy tree had templates but no program, so an engine that prefers argv[0] only when it names an existing file reached the OS path by accident and passed all three runs. Gating that preference on access(argv[0], F_OK) reproduces it: the test as merged passes that engine, this one fails it on the decoy run.

The decoy also goes first on PATH for the bare-name run, so an engine resolving argv[0] by lookup has something wrong to find.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 15:45:20 +00:00
Xavier Roche
a8712be127 Make six lang/*.txt files agree with their own LANGUAGE_CHARSET (#983)
* Make six lang/*.txt files agree with their own LANGUAGE_CHARSET

Croatian, Slovak, Polski and Slovenian are stored in CP1250 and Macedonian in
CP1251, but they declared an ISO-8859 charset. html/server/*.html copies
LANGUAGE_CHARSET into its <meta> and htsserver.c decodes the POST body with it,
so every s-caron, z-caron and Cyrillic letter came out wrong in webhttrack.
Strings added since #588 followed the declaration instead of the surrounding
bytes, leaving the files mixed.

The declaration moves to the codepage the bytes are actually in, and the minority
lines are transcoded to match. Going the other way was not an option: Latin-1
cannot spell Slovenian at all, and ISO-8859-2 would lose the typographic quotes
and ellipsis Polski uses. windows-1250 and windows-1251 are already in the
engine's codepage table and already declared by Cesky, Russian, Bulgarian,
Ukrainian and Uzbek.

Svenska declared ISO-8859-2 over Latin-1 bytes, which agree on everything it uses
except a-ring; only the declaration changes there.

62_lang-integrity now decodes every file through its declared charset and rejects
both a byte the charset has no mapping for and a decode landing in the C1 range,
which is the tell of a Windows codepage wearing an ISO-8859 label. Two
deliberately mislabelled copies guard the check itself, one per branch.

Closes #963

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the charset guard honest on macOS

BSD sed does not read \r in an s/// replacement, so the C1 probe would have
been labelled ISO-8859-2r there and failed on the unknown charset name instead
of on the C1 branch it exists to cover. The extractor strips CR anyway, so the
escape bought nothing. Also default n to 1, since a grep that errors leaves it
empty and the comparison then reads as a pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 15:15:22 +00:00
Xavier Roche
c5cd0cc4c2 A crash report names no frame of the httrack executable (#984)
* Name the executable's own frames in a crash report

dladdr() reports the main program's path as argv[0], which is a bare name
when the binary was found on PATH. The symbolizer's access() check then
fails and it drops the whole module, so every frame of the executable stays
a raw offset. A --disable-shared build puts the entire engine there, which
is why 105_suite-timeout failed and 80_engine-crash-symbolize skipped.

Resolve /proc/self/exe once at init and match frames against the main
program's load base, so the symbolizer gets a path it can open.

Not a link-flag problem: relinking with -rdynamic exports none of these
symbols, since -fvisibility=hidden and file-scope statics make them
STB_LOCAL and --export-dynamic only promotes global ones.

Closes #889

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Reject a clipped /proc/self/exe path

readlink() cannot report truncation, so an executable path of BT_PATH_SIZE
bytes or more came back as a prefix that find_main_object() kept. The prefix
fits copy_bounded(), and access() accepts it whenever it names a readable
directory, so the crash report gained a 1023-byte header pointing at that
directory and addr2line's "is a directory" complaint in place of the
executable's frames. A full buffer now reads as unresolved, which falls back
to argv[0] as before.

Checked by hand: an httrack installed 1024 bytes deep and run off PATH prints
the raw trace only, and 1023 still symbolizes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin test 157 to one executable block and the symbol column

A second block for the executable means library frames were misattributed to
it, and the bare symbol match could hit a path component named main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 15:10:16 +00:00
Xavier Roche
c553099b6d Add a weekly networked AppStream metainfo check (#979)
* Add a weekly networked AppStream metainfo check

The per-PR lint job and tests/210 validate with --no-net so an httrack.com
outage never reds an unrelated PR, but that also hides the one failure a
software centre user would notice: a dead screenshot URL. Add a scheduled
workflow that runs appstreamcli validate without --no-net weekly, and on
demand via workflow_dispatch; it does not touch the existing --no-net gates.

Closes #971

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the appstream-network comments to one line each

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 14:23:45 +00:00
Xavier Roche
2d582348da Ignore the 13 fuzzer binaries an in-tree --enable-fuzzers build drops in fuzz/ (#980)
fuzz/Makefile.am's noinst_PROGRAMS only builds under --enable-fuzzers, so #960
left fuzz/ alone. An anchored per-binary list keeps the tracked fuzz-*.c
sources out of the pattern, unlike a fuzz-* glob that would swallow them too.

Closes #970

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 14:19:36 +00:00
Xavier Roche
f937b4a4ec A relocated install cannot find its own shared library (#964)
* A relocated install cannot find its own shared library

httrack and htsserver now link with a loader-relative rpath,
$ORIGIN/../lib (@loader_path/../lib on macOS), beside the absolute
libdir libtool records, so a tree configured for one prefix and
unpacked somewhere else finds libhttrack instead of failing before
main.

Not every loader expands that token, so configure probes it instead of
trusting the linker: it builds a small shared library and an executable
that needs it, then runs the executable from an unrelated directory
with the library path cleared. Solaris and the older BSDs get a second
attempt with -Wl,-z,origin. Nothing is added when libdir is already on
the loader's search path, which covers every distribution package.

Closes #906

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Put the configured libdir ahead of the binary-relative rpath

Emitting $ORIGIN/../lib first shadowed the configured libdir whenever
$(bindir)/../lib is a different directory, which a --libdir=$prefix/lib64
or a split-bindir layout makes routine: a stale library sitting there
won the lookup on an ordinary, non-relocated install. Naming $(libdir)
ourselves before the token puts it back in front, so the token only
answers once the configured libdir is gone.

The system-libdir gate no longer reads libtool's
sys_lib_dlsearch_path_spec, which is only augmented when /etc/ld.so.conf
exists and would let a distribution libdir through in a sysroot or a
minimal container; it pattern-matches /lib, /lib64, /usr/lib and
/usr/lib64 and their subdirectories instead. /usr/local is no longer
treated as a system libdir, so a default configure now enables the
rpath and the tests exercise it.

Cross compiling no longer enables the feature unprobed. It defaults to
off with --enable-origin-rpath as the override, and that override takes
the -Wl,-z,origin spelling Solaris and the older BSDs need, since
nothing can run to tell the two apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the rpath ordering and the cases that must not get one

195_install-relocate.test asserted only that the token was present, so
it could not see it landing ahead of the configured libdir, and it
skipped outright wherever the rpath was suppressed. It now reads the
rpath entries in order and requires the libdir to come first, requires
proxytrack to carry no token at all, and in a suppressed build asserts
the absence rather than skipping.

Proving the relocated copy started was also not enough: a libhttrack
elsewhere on the loader path would have answered. The copy's own library
is now moved away and the binary has to stop working, which replaces the
ldconfig guard (found by `command -v ldconfig`, which fails for a user
without /sbin on PATH, and matching any soname including a .so.2 that
could never satisfy .so.3) and works on macOS, where there is no ldd.
Staging through DESTDIR rather than overriding the paths keeps libtool
from rewriting the build tree while the rest of the suite runs.

196_install-rpath-gates.test configures a system prefix, a multiarch
libdir, --disable-origin-rpath and --disable-shared and asserts each
one emits no rpath, with a private prefix first so a probe that never
succeeds cannot make the rest vacuous. One shared cache keeps the five
runs to a few seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Suppress the rpath for the 32-bit multilib libdirs too

/lib32, /libx32 and their /usr counterparts are on the loader path, and
libtool's sys_lib_dlsearch_path_spec used to cover them, so replacing
that read with an enumerated list handed a 32-bit distribution build an
rpath it never had and lintian's binary-or-shlib-defines-rpath with it.
Still enumerated rather than globbed as /lib*, which would swallow
/libfoo, /usr/library and /usr/libexec.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Configure a symlink farm, not the already-configured srcdir

196_install-rpath-gates.test configures out of $abs_top_srcdir, which
automake rejects when a config.status sits there. CI builds in-tree, so
every leg that runs the full suite failed; the out-of-tree local runs and
the msan leg (restricted TESTS) never saw it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Refresh mergeability after master moved

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Turn the binary-relative rpath off on Darwin, and read Mach-O with otool

dyld consults LC_RPATH only for an @rpath/ load path, and libtool stamps
libhttrack with an absolute -install_name, so the @loader_path entry the
Darwin arm emitted was never read. The copied tree stayed just as broken.
Gate the feature off there instead of shipping a dead load command.

That left the two bugs the macOS leg was actually failing on. Test 195
picked its object dumper by whatever happened to be installed, and an ELF
reader handed a Mach-O prints no rpath rather than an error, so a present
rpath read as an absent one; it now picks by platform. Test 196's nested
configures inherit none of the parent's CPPFLAGS/LDFLAGS, so brew's
keg-only openssl turned the default --enable-https into a hard configure
error. The gate has nothing to do with TLS, so they ask for =auto.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 13:51:48 +00:00
Xavier Roche
e810fc79d0 A stack-overflow SIGSEGV kills httrack with no diagnostic at all (#962)
* A stack-overflow SIGSEGV kills httrack with no diagnostic at all

sig_fatal went in with plain signal(), so it ran on the faulting thread's
own stack. When the fault is stack exhaustion there is no room left for a
signal frame, the kernel falls back to the default action, and the process
dies at 139 having printed nothing. Register the four fatal signals through
sigaction() with SA_ONSTACK, over a 64 kB alternate stack allocated at
startup, and prime backtrace() at init so glibc's lazy libgcc_s dlopen()
does not happen inside the handler.

Only the main thread is covered: htsbacktrace.c is linked into the httrack
binary rather than into libhttrack, so the engine's worker threads cannot
reach the installer. They behave as before, SA_ONSTACK being ignored where
no alternate stack exists.

Closes #866

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test 180 could not tell a stack overflow from an ordinary fault

Three mutants passed it: a crash_stack() that only dereferences NULL, a
backtrace() truncated to one frame, and a backtrace that returns nothing at
all, since the frame assertion accepted "No stack trace available" as a
match. The reports now have to differ: a runaway recursion repeats the same
return address ~250 times where the segv control repeats none, and the
control asserts the threshold still separates them.

Also stop skipping on macOS and the BSDs, where sigaltstack() exists and the
fix is live; only the trace text is gated on Linux. Each assertion now names
what failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The alt stack leaked, and took over the sanitizer's

LeakSanitizer called the 64 kB alt stack a direct leak: nothing retains the
pointer once sigaltstack() owns it. Map it instead of allocating it, which
also keeps the handler's own stack out of the heap whose corruption it may
be reporting.

The leak was the tell for the worse bug. ASan already has an alt stack
installed by the time signal_handlers() runs, 32 kB of it, and the old size
floor rejected it and installed ours over the top, moving ASan's handlers
onto our mapping. Defer to any alt stack already installed regardless of
size: whoever put it there sized it for their own handler, and ours needs
about 10 kB of the 32 that the smallest of them offers.

Test 181 pins both directions through an LD_PRELOAD observer that compares
the mapping the process ends up with against the one it was handed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Refresh mergeability after master moved

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 13:06:27 +00:00
Xavier Roche
c13c56d8bb Installed dev headers do not compile standalone (#958)
* Installed dev headers do not compile standalone

Three headers in DevIncludes_DATA fail on their own from $(includedir)/httrack
once HTS_INTERNAL_BYTECODE is defined: htswrap.h includes coucal.h, which is
never installed and which the file does not use; htsmodules.h uses HTSEXT_API
without htsglobal.h; htsdefines.h uses size_t without <stddef.h>.

205_install-headers.test installs the list into a temp dir and compiles a
one-line TU per header in both preprocessor states.

Closes #943

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Split CC so the test runs on the linux-i386 leg

That job configures with CC="gcc -m32", and looking the whole string up as one
executable made the test exit 77. A skip reads as success, so the one leg with a
different data model was checking nothing. Same for CC="ccache gcc".

The compiler now also has to compile a bare <stdio.h> first, or a box without
gcc-multilib blames every installed header instead of reporting its own toolchain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pass the build's CPPFLAGS to the installed-header compile

htsbasenet.h includes <openssl/ssl.h> under HTS_USEOPENSSL, so the test needs
OpenSSL's include path like any consumer does. Linux resolves it from
/usr/include, which is why only the macOS leg failed: brew's openssl@3 is
keg-only and reachable only through the CPPFLAGS given to configure.

Reproduced by hiding /usr/include/openssl behind a -nostdinc mirror and exposing
it from a private prefix, which reds htsnet.h and htsopt.h exactly as CI did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 12:44:39 +00:00
Xavier Roche
73c99ac3a0 Tooltip JavaScript breaks on any translation containing an apostrophe (#967)
* Tooltip JavaScript breaks on any translation containing an apostrophe

The onMouseOver handlers interpolate through ${html:...}, which escapes
for HTML text. The browser decodes an attribute value before compiling
it as JavaScript, so &#39; arrives as a bare quote and closes the string
literal; a translation carrying a double quote ends the attribute
outright. Add a js: template mode escaping for both layers and move all
304 handler sites onto it.

Closes #864

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the step2.html and title-attribute gaps in the js: escaping test

The static check could not match ${html:html:...}, the very shape it was
added to catch, and the runtime probe only fetched option1.html: reverting
all eight step2.html handlers left the test green. Nothing asserted that a
title= attribute stays on html: outside option1.html either.

Walk every server template instead, pairing each ${...} with the attribute
holding it, and fetch step2.html as well. Also feed the fixture a newline
and a tab, which had no coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Emit the backslash as \x5c so a DBCS trail byte cannot swallow it

shift-jis, BIG5 and gb2312 all accept 0x5c as a trail byte, and those are
the declared charsets of three shipped lang files, so the browser really
does run a DBCS decoder over the page. An orphan lead byte followed by the
emitted \\ pair decodes as one glyph plus a stray backslash, which escapes
whatever comes next. Emitting every escape as a \xNN group leaves the
decoder nothing to pair with: the only bytes cat_js_escaped adds are '\',
'x' and hex digits.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 12:25:36 +00:00
Xavier Roche
5bd0bba4e0 Nine lang.def strings are untranslated in 26 of the 30 language files (#961)
* Nine lang.def strings are untranslated in 26 of the 30 language files

LANG_F15b, LANG_I6c, LANG_I23c/d/e, LANG_I35c and LANG_I43c/d/e were
translated only in English, Francais, Dansk and Portugues-Brasil, so the
Windows GUI's option dialogs fell back to English everywhere else. Fill
them in for the other 26 files, each encoded in that file's declared
LANGUAGE_CHARSET and inserted where English.txt orders it.

Svenska.txt declares ISO-8859-2 but holds Latin-1, and Chinese-BIG5.txt
needs cp950 rather than strict big5. Romanian.txt and Slovenian.txt
declare ISO-8859-1, which cannot spell either language, so both stay
unaccented ASCII the way #862 left them.

Test 62 waived these nine msgids by name and fails once a waiver is no
longer needed, so drop them from that list: it reports 231 untranslated
msgids against master.

Closes #863

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Romanian: restore the two circumflexes ISO-8859-1 can spell

Only the-breve, s-comma and t-comma are unrepresentable in the declared
charset; a- and i-circumflex are 0xE2 and 0xEE, and the rest of the file
already spells them that way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 13:31:12 +02:00
dependabot[bot]
fa490d96da Bump github/codeql-action from 4 to 4.37.4 (#976)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4 to 4.37.4.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4...v4.37.4)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.37.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-04 13:26:16 +02:00
dependabot[bot]
17128490f9 Bump github.com/microsoft/vcpkg from 2026.06.24 to 2026.07.29 in /src (#975)
Bumps [github.com/microsoft/vcpkg](https://github.com/microsoft/vcpkg) from 2026.06.24 to 2026.07.29.
- [Release notes](https://github.com/microsoft/vcpkg/releases)
- [Commits](https://github.com/microsoft/vcpkg/compare/master...9e593bb18ea69cc5095e012465dcd675a822ed0d)

---
updated-dependencies:
- dependency-name: github.com/microsoft/vcpkg
  dependency-version: 2026.07.29
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-04 13:26:08 +02:00
Xavier Roche
cf830dd5d3 Icon=httrack never resolves through the share/pixmaps fallback (#966)
* Icon=httrack never resolves through the share/pixmaps fallback

share/pixmaps only ever held the sized names, so the bare-basename lookup a
desktop falls back on when no icon theme is installed never matched. Install
httrack.xpm (a 32x32 copy) alongside them rather than renaming one: Debian
globs the whole directory and a rename would strand the old names on upgrade.

Closes #932

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the pixmaps fallback to the icon, not to any 32x32 image

The geometry check passed on a solid-red square, so compare the installed
fallback against the installed 32x32 copy, declaration line aside. Also drop
the banned pipe into grep -q while here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 09:37:28 +00:00
Xavier Roche
1e3d51276f The suite watchdog may kill its own taskkill, and an empty category reads as a failing test (#965)
* The suite watchdog may kill its own taskkill, and an empty category reads as a failing test

ci_suite_heartbeat runs in a subshell of the process it targets, and kill_tree
on Windows is taskkill /F /T /PID, so taskkill is a grandchild of its own
target. A leaves-first /T would reap the watchdog before reaching the root: the
annotation is printed, nothing dies, and the step runs on to the 45-minute
cancel with no log. Signal the target directly first, then through the tree, so
the kill no longer depends on taskkill outliving its own ancestors.

The driver's category globs also expanded to themselves when they matched
nothing, handing test-timeout.sh a literal pattern that exits 127 and lands in
the tally as a failing test named after the glob. Categories now carry a label
and expand under nullglob; an empty one names itself and stops the suite before
any test runs.

Closes #952
Closes #953

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make the empty-category gate cover the one category that names a single test

runnable:00_runnable.test carries no metacharacter, so nullglob cannot empty it
and the gate never fired for it: the name reached test-timeout.sh unexpanded and
was counted as a test failing 127, exactly what the gate exists to prevent. Give
it a pattern like every other category, and empty it in the driver test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 08:23:22 +00:00
Xavier Roche
f1c3e1f5ef Ignore the files an in-tree build drops in the checkout (#960)
tools/Info.plist is the one that matters: configure generates it from
Info.plist.in, so a stray "git add -A" would commit a snapshot that
shadows it and pins CFBundleShortVersionString to whatever version built
it (#884, one directory over).

Closes #905

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 08:03:42 +00:00
Xavier Roche
57b22af27a Validate the AppStream metainfo in CI and under make check (#959)
#897 pinned the metainfo version against htsglobal.h, but nothing checked
the file was valid AppStream: a broken one passes the suite and a software
centre then drops WebHTTrack without saying so.

The lint job gains an appstreamcli step beside the .vcxproj well-formedness
pass, and 210_appstream-metainfo.test runs the same check under make check,
skipping when appstreamcli is absent. --no-net keeps the gate off the
screenshot host's reachability; errors and warnings fail, hints do not.

The test validates a copy with <id> stripped as a positive control, so a
validator that never fails cannot make it vacuous.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 07:55:00 +00:00
Xavier Roche
3db75afde4 Cancel superseded windows-build runs (#957)
windows-build.yml was the only workflow without a concurrency group, so
every push left the runs before it going. Match ci.yml and codeql.yml.

Closes #942

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 09:33:15 +02:00
Xavier Roche
103874dcd8 The macOS bundle declares a minimum OS that nothing checks (#956)
* Derive the bundle's minimum macOS from its payload

tools/Info.plist.in hardcoded LSMinimumSystemVersion 11.0, but nothing
built the payload for 11.0: with no MACOSX_DEPLOYMENT_TARGET set, clang
on the macos-15 runner targets the host, and all nine Mach-Os in the
released DMG report minos 15.0. Finder believes the plist and launches
on macOS 11 through 14, then dyld refuses the binaries.

macos-app.sh now takes the highest minos in the assembled bundle and
writes it to LSMinimumSystemVersion before the ad-hoc signature seals
Info.plist, so the two cannot drift again. The floor has to be the
maximum rather than the main executable's: five of those nine are
Homebrew bottles whose minos was fixed when the bottle was built, so a
deployment target on our own compile would not move them.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the declared minimum macOS instead of rewriting it

Review pushed back on deriving LSMinimumSystemVersion from the payload:
it makes the shipped minimum follow whatever runner image built the
bundle, so bumping a CI label would move HTTrack's advertised floor with
nobody deciding it. Declare 15.0, which is what the binaries need today,
and fail the build when the payload disagrees.

macho_floor also swallowed an otool failure: the call sat non-final in a
pipeline, so a Mach-O otool could not map contributed nothing and the
floor came from the rest, which is the same wrong-floor bug this fixes.
deps() already guards its otool the same way.

Dropping the plist rewrite drops plist_set with it, and with it the
awk-on-XML edge cases and the read-back check that was only meaningful
while the derived value differed from the template's.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the comments the review flagged

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 07:13:10 +00:00
Xavier Roche
f546fc4ddf CI watchdog deadlines stretch under the starvation they exist to catch (#955)
* The CI watchdog deadlines stretch under the starvation they exist to catch

Every deadline in the test harness counts poll iterations and assumes each
costs exactly the tick it asked for. Under the CPU and fork starvation that
wedges a runner, an iteration costs several times that, so the budget arrives
late in wall-clock terms or never: the 45-minute step timeout cancels first,
and a cancelled step keeps neither its log nor its artifacts. That is why the
wedged Windows jobs in #795 name no test.

Measure $SECONDS instead, in test-timeout.sh and in run_with_timeout,
wait_bounded and reap_bounded, which restores the wall-clock contract
run_with_timeout's comment already described.

The guards are checked against a reverted copy via a slow-sleep shim: with
each poll stretched fourfold, the counting code takes 44s to honour a 1s
budget and 16s a 3s one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Compare the deadlines strictly, and make the guards kill their mutants

$SECONDS is floored, so a reading equal to the budget can be a fraction
under it: -ge fired up to a second early, which kills healthy work. Compare
strictly instead, at the cost of at most a second late.

The 58_watchdog guard also passed on a watchdog that killed everything on
sight, because one stretched reap poll lifted the elapsed time over its lower
bound, and its control raced (the child exited before the first poll). Give
the control a command that outlives several polls, and widen the timing bound
to sit a full poll quantum either side of 8s healthy and 16s counted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 00:28:46 +02:00
Xavier Roche
420cfdfd1d Gatekeeper blocks the macOS bundle: nothing signs or notarizes it (#954)
tools/macos-release.sh signs every Mach-O with the hardened runtime, notarizes and staples the app, packs a DMG and notarizes that too. A new macos-release workflow runs it on a tag or on demand, with a Developer ID imported into a throwaway keychain from repo secrets. Signing is per-Mach-O rather than --deep, which reaches nested code but never applies the runtime, and notarytool's status line is read rather than its exit code.

The bundle stub is now a Mach-O (tools/httrack-launcher.c) rather than a shell script, which Apple treats as a resource and TCC cannot attribute a prompt to. The Mach-O walk and the Info.plist reader are shared with macos-app.sh via tools/macos-bundle.sh, and ci.yml's macos-app job runs the same script ad-hoc on every PR.

Closes #901
2026-08-03 21:35:10 +00:00
Xavier Roche
895049bfe1 The Windows suite driver is 140 lines of shell inlined in YAML (#951)
* The Windows suite driver is 140 lines of shell inlined in YAML

Move the "Run the engine test suite" step body to tests/ci-windows-suite.sh,
where shellcheck and shfmt reach it and it can be run by hand. The two
deliberate word-splits in the skip-set compare needed a directive; nothing
else changed, verified by diffing the shfmt-normalized old body against the
new file.

ci_annotate and ci_suite_heartbeat move with it, out of the test library
every test sources on every platform. 171_watchdog-heartbeat.test sources
the driver, which returns early unless run directly.

Closes #948

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test the driver on this platform, and ask the shell whether it was sourced

The guard read "${BASH_SOURCE[0]}" = "$0", which is only false by accident of
what the caller put in $0: bash -c '. "$0"' <driver> makes the two equal and
falls through into the suite. Ask the shell instead.

172_ci-windows-driver.test pins that case, the bindir contract and the loop's
accounting against a stub bindir, none of which a Windows-only leg proves
before merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Silence two shellcheck findings in the new driver test

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make the driver test survive distcheck's read-only srcdir

cp carries the source's mode over, so under distcheck the neutered testlib
copy came out read-only and the append failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-03 18:14:46 +00:00
Xavier Roche
9308adc16e A downloaded HTTrack.app cannot launch: Homebrew's OpenSSL never travels with it (#950)
* Bundle the non-system dylibs, or a downloaded HTTrack.app cannot launch

src/Makefile.am links $(OPENSSL_LIBS) into libhttrack, which on a build
machine resolves to Homebrew. The bundle check only rejected the staging
prefix, so /opt/homebrew paths sailed through: fine for someone who ran
brew install, fatal for anyone who mounts a DMG.

macos-app.sh now copies the transitive closure of non-system dylibs into
Contents/Frameworks, rewrites the load commands to @rpath, and adds a
depth-correct @loader_path rpath to every Mach-O. install_name_tool
invalidates the signature and arm64 refuses to run an unsigned binary, so
each rewritten file is re-signed ad-hoc.

The check is now that every load command resolves to /usr/lib,
/System/Library, or a file present in Frameworks, and it fails if it
scanned no Mach-O at all. Because a static check cannot fail while the
loader is quietly falling back to Homebrew, CI also hides Homebrew's
openssl@3 and re-runs the smoke against the moved bundle.

Part of #901.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the linker header room for the @rpath rewrite

install_name_tool refuses when the new load commands do not fit the
existing header: "changing install names or rpaths can't be redone".
The bundle build now passes -Wl,-headerpad_max_install_names, and the
script says so when the rewrite fails rather than surfacing the
toolchain's own message.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix two bundling bugs the review found, and three checks that could not fail

machos() ended with a while loop, so its status was the last iteration's
file(1) test. The bundle always holds shell scripts, and find walks APFS in
directory-hash order, so whenever a non-Mach-O came last the script died
under set -e with no diagnostic and a half-populated Frameworks. Green CI
here was luck, not evidence.

Two dependencies sharing a basename were both copied in the same pass, the
second clobbering the first, and relink() then pointed both references at
the one survivor. Frameworks is flat and cannot express the difference, so
this now fails with both paths named.

The checks that could not fail:

- machos() prefiltered on mode and suffix, and the same list drove the copy,
  the relink and the audit, so a non-executable Mach-O was unbundled AND
  unchecked. It now enumerates every file.
- deps() piped otool into awk, so a failing otool yielded no dependencies and
  the binary was recorded as clean. Its status is now checked. Reading the
  indented lines rather than tail -n +2 also stops a universal binary's
  per-slice headers from parsing as dependencies.
- The CI probe hid only the openssl@3 opt symlink, which a source-built
  bottle does not reference, and asserted nothing about the bundle first, so
  an empty Frameworks passed it. It now asserts libssl is bundled, then hides
  the kegs of every library in the closure.
- Signing verified the bundle after codesign --force --deep had already
  re-signed it, repairing the damage it was meant to catch. Per-file
  signatures are now verified first.

-Wl,-headerpad_max_install_names moves from the CI job to configure.ac: it
is a precondition of the macos-app target, not a CI preference, and anyone
following the standalone recipe in Makefile.am:22 hit a rebuild-only
failure that ld64's default header slack made intermittent.

The toolchain gate is now unconditional, so a non-Darwin host refuses to
build rather than emitting a bundle nothing verified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* One library reached by two paths is not a basename collision

The bundle depends on both /opt/homebrew/opt/openssl@3/lib/libcrypto.3.dylib
and the Cellar path behind that symlink, so comparing the dependency strings
called Homebrew's own layout a collision and refused to build.

Compare the paths with the directory resolved instead. Two names for one
file now dedupe; two different files under one name still fail, which is the
case that would silently clobber.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 16:18:28 +00:00
Xavier Roche
ab3ccc4d66 A wedged Windows suite takes the log that would name it (#947)
The Windows job wedges every few dozen runs: the test step never completes, the runner is lost, and the log blob and the `if: always()` artifact uploads go with it. So the wedging test has never been named.

A watchdog now runs beside the suite and ends the step before the runner dies, because a step that fails on its own terms keeps its log and still runs the uploads. It never triggers on elapsed time, since a healthy test and a wedged one look identical by the clock. It watches the progress log instead, where every outcome writes a line, the per-test timeout included: 900 seconds without one means that timeout did not fire, which is the wedge. On the way it names the test in flight as annotations, which is live progress rather than evidence.

That distinction is measured, not assumed. A throwaway workflow emitted notices and then died four ways: a step cancelled by its own timeout keeps its log and every annotation, on Linux and on Windows, from a background subshell as well as the foreground; a runner killed mid-step keeps neither, dropping notices that had been on the wire for twenty seconds.

Refs #795 rather than closing it: this ends the silence, it does not fix the leak behind it. Leftover coverage gap in #949.
2026-08-03 16:33:35 +02:00
Xavier Roche
12a1629bab Catch up to coucal head, now that its log rework is consumer-clean (#946)
Bump src/coucal from 5d2a633 to 0d36322, taking the two commits #945
deliberately skipped.

#31 replaced the log-level #if 0 cascade with COUCAL_LOG_LEVEL, but
defined all five level emitters unconditionally, so the unused ones
warned under -Wall for any consumer lacking coucal's own
-Wno-unused-function. That was the reason #945 pinned behind head.
xroche/coucal#33 tags them with an unused attribute instead and drops
the exemption from coucal's Makefile, so the warnings are gone at the
source rather than masked, and a new upstream CI leg compiles coucal.c
with consumer flags at every verbosity.

The default verbosity is still info, so the compiled-out debug and
trace that #941 measured stay compiled out: the probe from that PR
still reports 0 print handler calls per 20000 inserts. Our build is
clean of compiler diagnostics under gcc and clang. MSVC is unaffected
either way, since it builds at /W3 and the equivalent C4505 is level 4.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 10:53:33 +00:00
Xavier Roche
602bdb0065 proxytrack crashes on an .arc record whose body it could not read (#944)
* proxytrack: bound the .arc reader and stop the writer trusting a size it has no bytes for

The reader hands back an element carrying a declared size with adr == NULL when
it could not fetch the body, and the .arc writer took the size at face value:
fwrite(NULL, 1, size) faulted inside libc (#931). Both writers now take the
body from a helper that answers 0 when there is nothing to write, and the
record's own length, the Content-length header and the md5 follow it, so what
is written declares what it holds.

A new fuzz-arc harness drives the reader the way --convert does (#929), and
found the rest in seven executions: PT_Delete never freed the indexes it owns,
PT_Index_Delete__Arc never freed its hashtable, and a record could declare a
two-gigabyte body that the reader allocated before the short read failed.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* proxytrack: commit the merged index slot only once the array holds it

PT_IndexMerge() counted the new slot before growing the array, and assigned
realloc's result straight onto indexes->index: a failed allocation dropped the
array it already had and left index_size counting an entry that was never
stored. Harmless while nothing walked the array; PT_Delete() now does. The
array holds pointers, so size it as such rather than as whole PT_Index structs.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* proxytrack: trim the comments added by the .arc hardening

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* proxytrack: initialise the .ndx mutex, keep unsizeable archives loading, and give the bound a positive control

PT_LoadCache__Old() never called MutexInit(), which was survivable while
PT_Index_Delete__Old() was unreachable; PT_Delete() now runs it on every exit.
The readers lock the same handle, so on Windows the .ndx path was serving
unsynchronised.

A .arc past LONG_MAX cannot be sized by a 32-bit ftell, and refusing the whole
archive lost the records that used to load; the bound just stops constraining.

Test 164 asserted only refusals, so a reader that refused every body passed it,
as did an off-by-one on the last record. It now converts an archive whose body
ends on the last byte of the file, and checks the zip writer's own size header.
The truncated.arc seed ran out of file before reaching the bound it is named
for.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-03 10:33:24 +00:00
Xavier Roche
191b35106a The empty-key singleton is handed back with its const cast away (#945)
Bump src/coucal from a0a9e49 to 5d2a633, clearing the one warning #941
knowingly landed: 7a8198d const-qualified the_empty_string but still
returned it through a plain cast to coucal_key (void*), which httrack
compiles with -Wcast-qual. Upstream keeps the const and routes the
singleton through uintptr_t, so it stays in rodata, and adds
-Wcast-qual to its own mandatory flags so the class cannot return.

Pinned at 5d2a633 rather than upstream head. The next commit (#31)
replaces the log-level #if cascade with COUCAL_LOG_LEVEL and defines
all five level functions unconditionally, which warns twice under
-Wall for any consumer without coucal's own -Wno-unused-function.
Filed as xroche/coucal#32; head follows once that is resolved.

The default level is still info, so the compiled-out debug and trace
that #941 measured stay compiled out: the same probe reports 0 print
handler calls per 20000 inserts at both 5d2a633 and head.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 10:27:11 +00:00
Xavier Roche
fb786b0501 Coucal's compiled-out trace still formats every URL it inserts (#941)
Bump src/coucal from 93ec411 to a0a9e49, five upstream fixes.

The one that reaches the engine is the logging change. coucal_trace()
in coucal_add_item_() passes coucal_print_key() as an argument, and the
compiled-out macro expanded to an ordinary variadic call, so the
argument still ran. htshash.c installs key_adrfil_debug_print on
hash->adrfil and hash->former_adrfil, where it snprintf()s the full URL
into a scratch buffer, so every insert into the dedup tables paid for a
URL format that was then discarded. Measured on a probe mirroring
htshash.c's setup: 13262 handler calls per 20000 inserts, now 0, with
the table statistics unchanged.

The other four fixes harden coucal without reaching our call sites: the
custom key free at destruction (our dup handler is an identity pointer
copy and the free handler is empty), the mid-walk delete enumeration
skip (all four enum loops in htsback.c are read-only), the pool-aliased
key use-after-free (no call site passes an item name back as a key),
and the coucal_new() shift width (every call site passes 0).

No ABI change: coucal.h is not installed, struct_coucal is opaque,
struct_coucal_enum is byte-identical, and the 36 coucal symbols
libhttrack exports are unchanged.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 09:41:43 +00:00
Xavier Roche
50282e52fa Two tests share the number 160, and one hides its own failure (#940)
#933 and #934 merged minutes apart and both landed a test numbered 160, so the icon-theme one becomes 163. Only `tests-list.mk` referred to it, and the Windows job picks tests by topic word rather than number, so its coverage is unchanged.

The cache-hdrbounds test also sent stderr to `/dev/null` to hide an expected warning, which threw away the self-test's own diagnostics with it: a failure printed `cache-hdrbounds: FAIL` and nothing about why. stderr now goes to a file both failure paths report.
2026-08-03 09:38:43 +00:00
Xavier Roche
fd745e3f23 The cache aborts the crawl on URLs it accepts (#939)
Four places built or consumed the cache key, each assuming a different maximum URL length. A URL long enough to fill a `lien_back` field is legal and reachable off the wire, so one of them aborted the crawl where it should have missed the cache, and the index load read entry names into a buffer minizip can fill without a terminator.

All four now size off one bound, and the key is built all-or-nothing with the existing `slcatprintfbuff`: too long to store drops the entry with a warning, too long to look up is a miss. Clipping is never right here, because a clipped key is a valid key for some other URL, and that is a cache hit on the wrong content. The new `cache-urlbounds` self-test (`tests/162`) stores at the cap and pins that neither a twin differing only in its last byte nor a decoy sitting on a clip point can be served in its place.

Closes #935
Closes #936
2026-08-03 10:37:22 +02:00
Xavier Roche
df11ef6bf3 The cached-headers block is built with an unbounded sprintf (#934)
* Bound the cache header block instead of trusting the field caps

ZIP_FIELD_STRING and its integer siblings sprintf'd into a fixed 8192-byte
block with no bound, in both the engine cache writer (cache_add) and
ProxyTrack's new.zip writer. The values are remote-controlled -- ETag,
Location, Content-Disposition, the URL itself -- and in cache_add the caps
they are declared with sum past 8192, so the writer could overrun its own
stack buffer.

Route every field through slcatprintfbuff(), a new all-or-nothing bounded
append: a field that does not fit is dropped whole, since a clipped one reads
back as a valid shorter value. X-Save moves ahead of X-Addr/X-Fil so the one
field the reader consumes is not the first casualty of a full block.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Merge origin/master into fix-841-zip-field-bounds

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore the bounded ZIP_FIELD_STRING lost in the merge

The merge commit picked up a mutation-testing revert of this macro from the
shared worktree, putting the unbounded sprintf back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make the header-bounds test kill the mutants it was walking past

A test audit built seven mutant writers against the new self-test; five
passed. Restoring the legacy field order passed while silently dropping
X-Save, so the reorder this PR relies on was asserted by nothing. An
early-return writer passed vacuously, the check landing on the control
entry because nothing pinned which entry it read. A truncated Location
passed because any non-empty prefix was accepted, and the size bound was
a literal 8192 decoupled from the buffer it was meant to track.

The block is now identified by a field only the maxed entry carries, the
bound comes from a shared CACHE_HEADERS_SIZE, X-Save must survive, and a
still-present X-Fil reports that the entry stopped filling the block
rather than passing quietly. The wrapper consults httrack's exit status,
so an abort after the verdict is no longer a pass.

Adds an .arc to .zip round trip for ProxyTrack's writer, which had no
runtime coverage: every --convert in the suite writes .arc.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 07:38:30 +00:00
Xavier Roche
4f4699b12b HTTrack.app ships without an icon (#937)
Add tools/HTTrack.icns from the brand master, declare it in Info.plist.in,
copy it into the bundle, and check the plist and the payload agree.

Closes #900

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 09:37:53 +02:00
Xavier Roche
6de2187ee6 The application icon is still the pre-brand bitmap set (#933)
* The application icon is still the pre-brand bitmap set

Replace the 16/32/48 PNGs and the .xpm fallbacks with the HT monogram
generated from the same Jost* master as the masthead wordmark, and extend
the hicolor theme with 64, 128, 256 and a scalable SVG.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the icons on what install and dist emit, not on Makefile.am text

Six mutants survived the first version: an emptied EXTRA_DIST, an empty *dir
variable (automake's install rule exits 0 when it is), a _DATA glob with the
wrong extension, a _DATA entry for a directory the tree does not have, a
missing apps context subdirectory, and a size dropped from install entirely.
All six now fail, and the PNGs are indexed rather than RGBA.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 09:37:50 +02:00
Xavier Roche
69c8069573 proxytrack cannot re-read the .arc it writes (#928)
* proxytrack cannot re-read the .arc it writes

The version block's declared length counted the blank line closing it, so
the reader consumed the first record's separator and every entry was
skipped: a second --convert over proxytrack's own output loaded nothing.

The bytes on disk are unchanged; only the declared length shrinks by one,
which an older proxytrack reads too. The reader now stops on the last of
the newlines closing the version block, so archives already written the
old way still load.

Closes #834

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the version block newline scan and keep rejecting truncated archives

Review of the first commit found two regressions of its own: the newline
run was scanned to its end, so an archive padded with a gigabyte of them
cost a gigabyte of reads where master stopped after two, and tolerating
EOF there turned a length running past the end of the file into a silent
empty load. At most two newlines are read now, and a version block that
does not end on one is rejected as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the version block length against a compensating extra newline

A writer that emits the blank line and still counts it round-trips, so
every assertion passed while the length stayed a byte too long. The
declared block must not end on a blank line either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 06:52:27 +00:00
Xavier Roche
5ec25f3713 The tagline bar carries prose leading it has no use for (#930)
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 06:42:02 +00:00
Xavier Roche
12039c54d4 The tagline no longer sits against the wordmark's baseline (#927)
* The tagline no longer sits against the wordmark's baseline

The old GIF was cropped from the cap tops to the baseline, so its box edges were
the letters. The SVG's box is the true ink box, which in Jost also holds the k's
ascender above the caps and the round letters' overshoot below the baseline, and
that shows up as a band of field colour between the wordmark and the tagline.

Neither band can be cropped out of the artwork without cutting ink, so the
masthead takes them back optically. Measured against the old rendering, the
baseline-to-tagline distance is identical and the cap tops land within a third
of a pixel.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cut the comments back to one line each

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Say what the margins actually do

The comment claimed both bands were trimmed. The lower one is, in full; the
upper is trimmed only by what the old bitmap did not already carry, which is why
1.7px is not the 3.56px the artwork measures. Derivation recorded beside the
generators.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 06:17:16 +00:00
Xavier Roche
c7767990cd ProxyTrack dumps a coucal hashtable stats line on every WebDAV request (#924)
* ProxyTrack dumps a coucal hashtable stats line on every WebDAV request

coucal logs a per-table statistics summary when a table is deleted, and
with no handler installed it prints that line itself, prefixed with the
table's address. ProxyTrack builds and drops one table per WebDAV
enumeration, so every unauthenticated PROPFIND put one on whatever the
service redirects its output to, heap pointer included.

httrack and htsserver were already covered by hts_init(), which installs
a global coucal handler that drops info-level messages unless HTS_LOG is
set (#416). ProxyTrack does not link libhttrack: it compiles coucal
itself and never calls hts_init(), so it was the last binary on coucal's
built-in sink. It now installs its own handler. Critical and warning go
through proxytrack's log, everything below is dropped unless HTS_LOG is
set, and the summaries then come back without the address.

Unnaming the tables would not have fixed it. coucal_delete() logs the
summary for every table, named or not; the name only decorates the
message.

Closes #918

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Whitelist proxytrack's request-time output instead of naming two absent strings

The absence check pinned two literals, so a reworded leak or one whose
pointer was not at column 0 walked through it. Assert instead that every
line the process writes once serving is an access-log line, which is the
property, and pin the routed form under HTS_LOG whole so the address
cannot creep back between the severity and the message.

Also say why the handler passes "debug" rather than the DEBUG macro: the
macro is NULL outside a debug build, which would quietly make HTS_LOG a
no-op.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Order the whitelist against a request's teardown, not just its access log

proxytrack writes the access line before send(), so waiting on it ordered
nothing that a teardown or keep-alive path writes afterwards: a leak
delayed 400ms past the response survived the check. Send both PROPFINDs
over one connection, since the keep-alive loop does not read the second
request until the first one's teardown has run, and read the capture only
once proxytrack is gone and the pty drainer has marked it complete.

Draining alone was not enough. SIGTERM cuts the work short rather than
truncating a buffer, so a still-pending write is never made at all and
there is nothing left to flush; the ordering is what catches it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 18:20:23 +00:00
Xavier Roche
0a1a4379c2 configure hangs forever if BASH_SHELL points at a FIFO (#926)
A FIFO passes "test -x", so the #920 check ran it: bash gets EACCES from
execve, falls back to reading the file for a shebang, and blocks in open()
with no writer. AS_EXECUTABLE_P is autoconf's own "test -f && test -x", and
AC_PATH_PROGS on the line above already applied it to the PATH search, so the
override path was simply using the weaker predicate.

A regular executable that never returns stays uncovered: no portable timeout
is worth it, and CC= pointing at the same wrapper hangs stock AC_PROG_CC too.

Test 151 gains the FIFO case, and its run() is capped so a regression fails
instead of wedging "make check" with no log.

Closes #922

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 20:16:12 +02:00
Xavier Roche
9479913232 http_xfread1's reserve-only read mode has no caller (#925)
* http_xfread1's reserve-only read mode has no caller

The `bufl == -2` branch of `http_xfread1()` allocates the line buffer and
returns without reading. Nothing has ever called it: no call site in the
tree passes -2, and scanning all 2649 revisions in this repository for
`xfread1(` call sites turns up 24 distinct lines, none of them -2. The
branch arrived with the 3.20.2 import commented "force reserve", so it was
probably meant for a preallocate-then-fill pattern that never landed.
Naming it `HTS_XFREAD_RESERVE` in #919 made it read as a supported mode.

No external caller is possible either: `htslib.h` is not installed and the
symbol is hidden, so this is not an API change.

Equivalence checked against the object code. `htsback.o` and
`htsselftest.o` disassemble identically; in `htslib.o` every function
except `http_xfread1` differs only in the `__LINE__` values `htssafe.h`
bakes in, shifted by the seven deleted lines. No new test: nothing changes
for any input a caller can produce, and `01_engine-xfread` plus the chunked
tests still pass.

Closes #923

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Say that any non-positive bufl is line mode, not just the two named

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 18:04:31 +00:00
Xavier Roche
8168aa3a45 The masthead wordmark is a 400x34 GIF that blurs on any hi-DPI screen (#916)
* The masthead wordmark is a 400x34 GIF that blurs on any hi-DPI screen

Replaces it with an outlined SVG across the 38 documentation and WebHTTrack
pages that carry it. The original was set in Futura, so the lockup was refitted
in Jost*, the closest free Futura revival, taking weight from the measured stem
thickness, size from the cap heights and tracking by least squares against the
glyph positions in the old bitmap.

tests/82 now asserts that every image a GUI page names is actually served.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Point the shared chrome generator at the new wordmark

The masthead of the 13 generated pages comes from tools/doc-chrome.py, so
editing the pages alone left the generator disagreeing with its own output and
--check red.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 19:17:01 +02:00
Xavier Roche
72fa732f7e A debug fprintf dumps every PROPFIND body to stderr (#917)
* proxytrack: drop the leftover debug traces on the WebDAV path

Two fprintf(stderr) calls in the PROPFIND path shipped by accident: one dumped
the client-supplied request body, the other the whole generated multistatus
response. Both ran on every PROPFIND with no authentication in front of them,
so any client could write bytes of its choosing into the operator's stderr.
The body is never parsed and the response is derivable from the index, so
neither trace has diagnostic value worth keeping behind a debug level.

Closes #911

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: make the #911 leak check see stdout and the property, not two literals

The absence checks pinned 'DEBUG: DAV-DATA' and '^RESPONSE:', which four
re-added variants walk straight past: a trace without the hyphen, one with no
marker at all, one prefixed so the '^' misses, and one on stdout. The stdout
case is the worst of them: redirected to a file, proxytrack's stdout is fully
buffered and SIGTERM never flushes it, so the leak never reached the log the
test reads.

Give proxytrack a pty instead of a file, so libc line-buffers its output on
Linux and macOS alike, and assert the property: a PROPFIND carries a canary the
index cannot produce, and neither the canary nor a distinctive string from the
generated response may appear in what proxytrack wrote. The two literals stay as
names for the specific regression. The liveness guard now requires a PROPFIND
answered 207, since a depth-rejected one is logged 403 by the shared reply path
without ever reaching the deleted code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 11:31:53 +00:00
Xavier Roche
5a38473343 A failed String allocation is only caught by assert, and the empty-on-failure contract is unstated (#921)
* Bound the ProxyTrack DAV item buffer against an amplified PROPFIND path

proxytrack_add_DAV_Item() reserved a fixed 1024 bytes and then sprintf'd into
it unbounded. The request path lands in the response twice, once as the href
and once as the displayname, and escapexml() turns each '&' into '&amp;', so
an unauthenticated PROPFIND of roughly 900 ampersands writes about 9000 bytes
off the end of the heap block. No cache entry and no Depth: 1 are needed.

Replace the hand-sized reserve with StringSprintf(), which measures the
formatted output and grows the String to fit, and convert the sibling sprintf
sites in the same file so no unbounded write into a String is left to
re-audit. Sizing beats clipping here: the String already owns a growable
buffer, so nothing has to be dropped.

Closes #836

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound StringSprintf's pre-C99 retry, and trim the review findings

A genuine vsnprintf conversion error returns -1 just as pre-C99 msvcrt does
for a short buffer, so the doubling search had no way to tell them apart and
grew until realloc aborted. Unreachable from these format strings, which use
only %s and %d, but the helper lives in a shared header and will get more
callers. Cap the search and empty the String past it.

Also: the count assertion piped into wc under pipefail, so a zero count killed
the test through set -e before its diagnostic could print.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Lift the NO_WEBDAV conditional out of a macro argument list

A preprocessor directive inside a macro invocation's arguments is undefined:
it was fine while this was a plain sprintf() call, and MSVC rejected it as
soon as it became StringSprintf(). GCC accepts it, so only the Windows leg
caught it. Compute the DAV header fragment first and pass it as an argument.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover StringSprintf's exact-fill case and the WebDAV enumeration branch

StringSprintf_ writes the terminator at buffer[ret], so widening its
`ret < capacity` guard by one byte is a heap overflow that only fires when the
formatted output exactly fills the capacity. No crawl test lands on a
capacity boundary, so the mutant survived the suite. The new `strsprintf`
self-test sweeps lengths around 256, 512, 1024 and 2048 with the String's
capacity pinned to each, plus a growing and shrinking sweep on one reused
String, and checks the length, the bytes and the terminator every time.

Test 147 only ever sent Depth: 0, leaving the enumeration branch the same PR
rewrote with no coverage at all. Its fixture gains a child directory, and a
Depth: 1 listing pins the item URLs, including the trailing '/' that
StringPopRight takes back off a directory name.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make the String failure paths safe without assert

StringSprintf empties the String when it gives up, but that contract was
only visible in the implementation, and the WebDAV enumeration in
proxytrack pops the trailing '/' straight after it. State it at the
declaration, no-op StringPopRight on an empty String, and skip an
enumerated item the formatter could not name.

StringRoomTotal reported a failed realloc through STRING_ASSERT alone.
The MSVC Release configuration defines NDEBUG, so that check is already
gone from the shipped Windows builds, leaving a NULL buffer under a
capacity bumped before the allocation was known to succeed. Assign both
only on success, and terminate through StringOom_.

Closes #915

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber the String OOM test to 152

151 is taken by the unmerged tests/151_bash-shell-validate.test (PR #920).
The filenames differ, so git would have carried both onto master rather than
conflicting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Declare the new WebDAV test in the Windows skip set

It skips on Windows for the same reason as its two neighbours, MSYS
cannot reap a background listener (#595), and the ratchet fails a skip
it was not told about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the out-of-memory action overridable

STRING_REALLOC and STRING_FREE are #ifndef hooks, and STRING_ASSERT was one
too; replacing it with a hard-wired call took a hook away from downstreams of
this installed header. Route the failure through STRING_OOM instead, with the
print-and-abort default unchanged.

Also flush stderr before aborting: the Windows CRT buffers a redirected
stderr and abort() flushes nothing, which would drop the message the test
matches on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Inject the allocation failure instead of asking for a huge one

The engine self-test pinned a String's capacity so the next doubling asked
for SIZE_MAX/2, on the assumption that no allocator would serve it. Six CI
legs disagreed: i386 has a 3G user space, and the 64-bit runners handed the
request out too, so the test reported "NOT aborted" everywhere but here.
Green that depends on how much memory the machine feels like giving is not a
test.

Drive the path from a standalone helper instead, which defines STRING_REALLOC
to a stub returning NULL before including htsstrings.h. Four cases: growth
with the stub allocating for real, the failure reaching the handler with the
size it asked for, a live buffer surviving a failed realloc, and the shipped
handler printing and aborting. Only the automake build produces the helper,
so the test declares its Windows skip.

The self-test had no portable way to force the failure, so it goes rather
than staying as a handler nobody can rely on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the bytes and the requested size, not just the bookkeeping

An under-allocation survived the helper: shortening the realloc by one byte
while still recording the full capacity left all four cases green, because
only the growth case allocated anything and it checked the capacity number
rather than the memory behind it. Fill the announced capacity to its last
byte and read it back, which the sanitizer legs turn into a hard failure.

The failure cases pinned the initial capacity by asserting 16, so bumping
that policy would have failed a correct tree. Compare the size handed to the
handler against the size the stub was actually asked for instead, which also
catches the under-allocation on legs with no sanitizer.

Drive StringSprintf_ and StringBuffN_ too, the other two places the header
expands STRING_OOM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 11:29:24 +00:00
Xavier Roche
6745ec9842 configure accepts a BASH_SHELL that is not a usable bash (#920)
* configure accepts a BASH_SHELL that is not a usable bash

`./configure BASH_SHELL=/bin/sh` was accepted without a word. `AC_PATH_PROGS`
takes any absolute value verbatim, so the macOS problem #895 fixed (a bash in
POSIX sh-mode driving `make deb` and the test harness) came back, surfacing
much later as a `146_bash-shell.test` failure instead of a configure error. A
relative value never reached the Makefiles at all: it was dropped for whatever
the PATH search turned up.

configure now checks the value it resolved. The shell must be executable,
report a `BASH_VERSION`, and not carry `posix` in `SHELLOPTS`, the
discriminator `146_bash-shell.test` already uses, since an sh-mode bash reports
a version too. A relative or whitespace-carrying override is refused before the
search runs; whitespace would otherwise survive into `$(BASH_SHELL)`, which
nothing in the Makefiles quotes.

Only an explicit override is fatal. When the search itself finds nothing
usable, configure warns and carries on, so a box without bash still builds; it
just cannot run `make check` or `make deb`.

Closes #908

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* An environment in POSIX mode must not be blamed on the bash path

POSIXLY_CORRECT, or an exported SHELLOPTS, puts every bash into POSIX
sh-mode, so `./configure BASH_SHELL=/bin/bash` failed with advice to pass a
path that cannot exist, and a plain configure warned that no usable bash was
found on a box that has one. The probe now runs a second time under `env -u
POSIXLY_CORRECT -u SHELLOPTS`; if the shell is fine once they are cleared, the
message names them and says how to clear them for make as well, since it
inherits the environment. An override stays fatal, the search still only warns.

The bash-ness probe read `BASH_VERSION`, an ordinary variable any shell echoes
back, so `BASH_VERSION=9.9 ./configure BASH_SHELL=/bin/dash` was accepted and
dash landed in `TEST_LOG_COMPILER`. It reads `${BASH_VERSINFO[0]}` instead,
which no environment can fake.

The path guard covered whitespace alone while claiming to cover what make and
the recipe shell split on, so a real bash under a directory named with `;` or
`$` or `#` still reached the Makefile. It now rejects that whole class. Quoting
`$(BASH_SHELL)` at its three uses was the alternative, but make cuts the value
at a `#` and expands a `$` before any shell sees it, so quoting would cover
less than the guard.

The suite now also pins the branch the fatal/warn split rests on: no override,
an unusable bash first in PATH, configure exits 0 with a warning.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Nothing pinned which cause of POSIX mode gets blamed

A shell can be in POSIX mode because it was invoked as sh or because the
environment forces it, and only the second probe can say which. Nothing held
that down, so a version deciding from the environment alone, without
re-probing, passed every case in the suite while telling the user to clear a
variable that would not have helped. Test 151 now runs a bash symlinked as sh
with POSIXLY_CORRECT=1 set as well, and requires the message to name the path.

The comment records why that branch cannot simply read POSIXLY_CORRECT:
autoconf runs "set -o posix" on configure's own shell, so it is set there no
matter what the user's environment holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 11:08:48 +00:00
Xavier Roche
eb4c1fe64c http_xfread1's read mode is an unnamed integer at every call site (#919)
* A chunked response carrying trailers is discarded as "Invalid chunk"

The chunk automaton expected the line after the terminating zero-length
chunk to be empty. RFC 9112 7.1.2 lets a server put a trailer section
there and asks recipients to discard fields they do not understand;
instead the whole message failed and the resource never reached the
mirror.

The trailer section is now read the way headers are, as a block ending
on a blank line, and thrown away. Reading it as a block also bounds it:
trailers carry no length of their own, so an endless one would hold a
connection slot forever, and the line reader's 8KB buffer caps it. Only
the terminating chunk opens the section, so junk where a data chunk's
own CRLF belongs is still a framing error.

Closes #855

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber the trailer test to 149, 147 is taken by the WebDAV overflow test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the line-block bound, and pin the trailer edge cases

Review follow-ups: the 8190-byte cap that bounds a trailer section was an
unnamed literal inside http_xfread1, so raising it for large response
headers would have moved the trailer bound silently. It is now
HTS_LINE_BLOCK_SIZE, named where the reader is declared.

The trailer path also no longer runs the chunk-size parse it then
discards, and eof.html pins the deliberate leniency the change
introduces: past a complete, length-verified body, a trailer section cut
before its blank line still lands. A body cut before the terminating
chunk stays refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name http_xfread1's read modes instead of passing bare 0, -1 and -2

http_xfread1() selects its read mode from the sign of bufl: a positive
value reads that many bytes, 0 stops at a blank line, -1 at the first LF,
-2 only reserves the buffer. Nothing declared them, so every call site was
an unexplained literal.

Declare HTS_XFREAD_LINE_BLOCK, HTS_XFREAD_LINE and HTS_XFREAD_RESERVE in
htslib.h beside HTS_LINE_BLOCK_SIZE, and use them at each call site. The
selftest keeps its 8192, a byte count rather than a mode.

Behaviour-preserving: the preprocessed output of the changed translation
units is token-identical once the parens around the negative literals and
the __LINE__ digits shifted by the reworded comments are normalized away.

Closes #914

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Note that the reserve-only read mode has no caller

Naming it made it read as a supported mode; it is unreachable (#923).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the reserve-mode note on one line

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 10:53:02 +00:00
Xavier Roche
2d041866ed An unauthenticated PROPFIND overflows the ProxyTrack DAV item buffer (#909)
* Bound the ProxyTrack DAV item buffer against an amplified PROPFIND path

proxytrack_add_DAV_Item() reserved a fixed 1024 bytes and then sprintf'd into
it unbounded. The request path lands in the response twice, once as the href
and once as the displayname, and escapexml() turns each '&' into '&amp;', so
an unauthenticated PROPFIND of roughly 900 ampersands writes about 9000 bytes
off the end of the heap block. No cache entry and no Depth: 1 are needed.

Replace the hand-sized reserve with StringSprintf(), which measures the
formatted output and grows the String to fit, and convert the sibling sprintf
sites in the same file so no unbounded write into a String is left to
re-audit. Sizing beats clipping here: the String already owns a growable
buffer, so nothing has to be dropped.

Closes #836

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound StringSprintf's pre-C99 retry, and trim the review findings

A genuine vsnprintf conversion error returns -1 just as pre-C99 msvcrt does
for a short buffer, so the doubling search had no way to tell them apart and
grew until realloc aborted. Unreachable from these format strings, which use
only %s and %d, but the helper lives in a shared header and will get more
callers. Cap the search and empty the String past it.

Also: the count assertion piped into wc under pipefail, so a zero count killed
the test through set -e before its diagnostic could print.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Lift the NO_WEBDAV conditional out of a macro argument list

A preprocessor directive inside a macro invocation's arguments is undefined:
it was fine while this was a plain sprintf() call, and MSVC rejected it as
soon as it became StringSprintf(). GCC accepts it, so only the Windows leg
caught it. Compute the DAV header fragment first and pass it as an argument.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover StringSprintf's exact-fill case and the WebDAV enumeration branch

StringSprintf_ writes the terminator at buffer[ret], so widening its
`ret < capacity` guard by one byte is a heap overflow that only fires when the
formatted output exactly fills the capacity. No crawl test lands on a
capacity boundary, so the mutant survived the suite. The new `strsprintf`
self-test sweeps lengths around 256, 512, 1024 and 2048 with the String's
capacity pinned to each, plus a growing and shrinking sweep on one reused
String, and checks the length, the bytes and the terminator every time.

Test 147 only ever sent Depth: 0, leaving the enumeration branch the same PR
rewrote with no coverage at all. Its fixture gains a child directory, and a
Depth: 1 listing pins the item URLs, including the trailing '/' that
StringPopRight takes back off a directory name.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Declare the new WebDAV test in the Windows skip set

It skips on Windows for the same reason as its two neighbours, MSYS
cannot reap a background listener (#595), and the ratchet fails a skip
it was not told about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 10:37:22 +00:00
Xavier Roche
5f52c3d942 A chunked response carrying trailers is discarded as "Invalid chunk" (#913)
* A chunked response carrying trailers is discarded as "Invalid chunk"

The chunk automaton expected the line after the terminating zero-length
chunk to be empty. RFC 9112 7.1.2 lets a server put a trailer section
there and asks recipients to discard fields they do not understand;
instead the whole message failed and the resource never reached the
mirror.

The trailer section is now read the way headers are, as a block ending
on a blank line, and thrown away. Reading it as a block also bounds it:
trailers carry no length of their own, so an endless one would hold a
connection slot forever, and the line reader's 8KB buffer caps it. Only
the terminating chunk opens the section, so junk where a data chunk's
own CRLF belongs is still a framing error.

Closes #855

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Renumber the trailer test to 149, 147 is taken by the WebDAV overflow test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the line-block bound, and pin the trailer edge cases

Review follow-ups: the 8190-byte cap that bounds a trailer section was an
unnamed literal inside http_xfread1, so raising it for large response
headers would have moved the trailer bound silently. It is now
HTS_LINE_BLOCK_SIZE, named where the reader is declared.

The trailer path also no longer runs the chunk-size parse it then
discards, and eof.html pins the deliberate leniency the change
introduces: past a complete, length-verified body, a trailer section cut
before its blank line still lands. A body cut before the terminating
chunk stays refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199wAkSVZNBNp51mpRkxMvv
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 11:56:22 +02:00
Xavier Roche
592f3dd154 The frozen-slot spool is written inside the mirror namespace (#912)
* Spool a frozen backlog slot outside the mirror namespace

back_cleanup_background() named the spool file by appending ".tmp" to the save
name, so it landed beside the mirrored file. That is the shape #774 fixed for
the re-fetch backup: a site serving <path>.tmp has its mirrored copy truncated
by filecreate() and then unlinked when the slot is woken, and the run still
reports success. It is reachable on defaults, not only under a saturated
backlog: a -Z crawl of the bundled bigcrawl site with -c4 logs slots moving to
background.

Route both name shapes through back_spoolname(), which puts them in the
~hts-tmp directory no save name can spell, and drop that directory at the two
sites that unlink a spool. Left out of the #774 PR because these lines also
carried the overflow in #857.

Closes #859

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the -p0 spool relative when no output directory is set

The new name inserted its own separator before ~hts-tmp, but path_html_utf8
already carries one and is empty when -O is absent, so the spool became
/~hts-tmp/tmpfile0.tmp: absolute, in the filesystem root. Master built
"%stmpfile%d.tmp" and stayed relative to the working directory.

create_back_tmpfile() has spelled it the same way since #842. Its empty
path branch looks unreachable from the three call sites, so this side is a
consistency fix with no test behind it, unlike the spool.

The self-test pinned the doubled slash it observed rather than the shape it
wanted; it now asserts the single-separator form and covers the empty
path_html_utf8 case that produced the root path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 08:44:06 +00:00
Xavier Roche
620418a9c6 The rings background is a 501x456 GIF that blurs on any hi-DPI screen (#910)
The panel decoration was drawn as vector and flattened to a 4KB indexed GIF
some time around 2007, with the panel colour baked in as an opaque backdrop.
Refitting its four ellipse boundaries recovers the original geometry, so it
goes back to being what it was: two elliptical annuli, 553 bytes of SVG, with
a transparent background that now composites over the panel instead of having
to match it.

Rasterised at the same size, the only pixels that differ from the GIF are
single-pixel anti-aliasing fringes along the four boundaries. Nothing survives
a 3x3 erosion of that difference, so no edge has moved.

Dark mode still drops the image rather than inverting it, since the ring
lavender is a light-panel tone whichever way the file stores it.

The engine keeps its own embedded copy of this GIF for the backblue.gif it
writes into mirrors. That one is untouched: the filename and byte length are
a contract with pages already on disk.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 08:24:21 +00:00
Xavier Roche
f3fa3a8b98 configure discards a user-supplied BASH_SHELL (#907)
* configure discards a user-supplied BASH_SHELL

AS_UNSET erased the variable before AC_PATH_PROGS could honour it, so
"./configure BASH_SHELL=/path" had no effect and there was no way to
point the build at a bash other than the first one on PATH. Nothing
presets BASH_SHELL, which was the whole problem with BASH in #895, so
declaring it precious is enough.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Run the nested configure against a symlink farm

An in-tree build leaves a config.status in srcdir, and autoconf then refuses
the out-of-tree run the test needs. Every CI build leg builds in-tree, so the
check failed there while passing on an out-of-tree tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Read the resolved bash from the configure trace, not the Makefile

The nested configure ran without the flags the outer one was given, so on
macOS it died at the openssl check that Homebrew paths satisfy. BASH_SHELL
is resolved long before that, so assert on the trace and let the run fail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the value that reaches $(BASH_SHELL), not the macro's decision

Reading the configure trace let a mutant through: resolve the override
correctly, clobber BASH_SHELL one line later, and both assertions passed
while every Makefile got the wrong shell. Prefer the generated Makefile
and keep the trace only as a fallback for a configure that dies early.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 12:53:15 +02:00
Xavier Roche
311b99afd4 The AppStream metainfo still advertises WebHTTrack 3.49.8 (#897)
* The AppStream metainfo still advertises WebHTTrack 3.49.8

The metainfo installs to usr/share/metainfo, so GNOME Software and KDE
Discover read both the version and the "What's new" text out of it. Its
releases block held one entry, 3.49.8, and nothing had moved it since.

It now lists 3.49.8 through 3.49.15, newest first, each with a short
user-facing note taken from history.txt. Dates come from the git tags;
that also corrects 3.49.8's, which carried 3.49.7's date.

01_engine-version-macros.test gains two assertions so the next release
cannot miss this file, or configure.ac: AC_INIT and the top release entry
must both match HTTRACK_VERSIONID, and the entries must descend so the
top one really is the newest.

Closes #884

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Harden the metainfo version check and correct four release notes

The release-version extraction matched <releases ...> as well as <release>,
ignored XML comments and took only the last tag on a shared line, so a parked
or wrapper version= could pose as the newest entry and pass a stale metainfo.
Split tags one per line, drop comments, and anchor the match. Widen the awk
ordering key so a component of 1000 or more cannot borrow into the next.

In the notes: "3.49-2" and "site rules with wildcards" are unreadable in a
software centre, the Windows path bullet does not apply to the Unix WebHTTrack
GUI it ships with, and 3.49.15 listed no web-interface fix at all.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 08:02:00 +00:00
Xavier Roche
ef81f5b488 An install that is not where configure put it cannot find its own data (#899)
* Find the data directory instead of trusting the configure-time one

webhttrack probed a fixed list of prefixes that nothing derived from
--datadir, and the engine baked $(datadir) into the binary with the
argv[0] fallback compiled out. Both fail on any tree that is not where
it was configured.

configure substitutes the real datadir into src/webhttrack, and
hts_resolve_datadir() prefers the compiled-in path but derives one from
argv[0] when it is gone, so a moved install reads its own templates
rather than silently falling back to the built-in defaults.

Closes #887
Closes #894

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Resolve the data directory from the executable, not just argv[0]

MSVC broke: HTS_HTTRACKDIR is only defined on non-Windows, so the
argv[0] branch this replaced was live there, not dead. Windows now
passes an empty builtin and falls back to the executable's own
directory, which is what it did before -- except fconcat inserts no
separator, so the old path_bin lacked its trailing slash and never
resolved a template anyway.

Ask the OS for the executable path (/proc/self/exe, _NSGetExecutablePath,
GetModuleFileName) and keep argv[0] as the fallback, so a mirror run
through a PATH lookup resolves too.

The bundle drops the substituted datadir from its copy of webhttrack:
it is a build-machine path there, and the relative entries ahead of it
already find the payload.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Clip the candidate path instead of aborting on a long argv[0]

strlncatbuff() aborts rather than truncates, and appending the layout
suffix to an already-full buffer reaches that: a directory part within
17 bytes of the candidate buffer's size killed the process. Build the
candidate with snprintf and skip it when it does not fit.

The self-test now drives a directory part long enough to trigger it;
without the fix it aborts on "overflow while appending 'layout[i]'".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Ignore the generated src/webhttrack

An in-tree build writes it next to webhttrack.in, where it was untracked
and one "git add -A" away from re-entering the tree with a build
machine's datadir frozen into it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 07:43:50 +00:00
Xavier Roche
22c506944d configure resolves bash to /bin/sh on macOS, and make deb runs a bash script with dash (#898)
* configure resolves bash to /bin/sh on macOS, and make deb runs a bash script with dash

AC_PATH_PROGS searched into BASH, which bash presets to its own invocation
path. configure re-execs through /bin/sh, and on macOS that shell is a bash, so
the macro honoured the pre-set value and reported "checking for bash...
/bin/sh": a bash in sh-mode that rejects process substitution. Search into
BASH_SHELL instead, a name no shell presets, with AS_UNSET in front so the
environment cannot preset it either.

That makes the obvious fix for the deb target safe. It ran tools/mkdeb.sh with
$(SHELL), which is /bin/sh, so "make deb" died on the first bashism on every
Debian and Ubuntu box. macos-app.sh stays on $(SHELL): it is POSIX sh on
purpose, so shellcheck lints it as sh.

tests/146_bash-shell.test asserts the configured shell exists, sets
BASH_VERSION, is not in POSIX sh-mode, and parses a process substitution.

Closes #895
Closes #891

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the comments this branch added, and drop a claim that is no longer true

The four comment blocks the branch added ran two to five lines where one or two
carry the fact. The 146 header also said the macOS shell rejects "the process
substitution the bundle script uses": tools/macos-app.sh has been POSIX sh with
no process substitution since #890, so the gate is there for tests/local-crawl.sh
and tests/webhttrack-smoke.sh, which is what the comment now says.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 07:05:44 +00:00
Xavier Roche
b127323a68 The macOS test server stalls 35s on a reverse DNS lookup nothing reads (#896)
* The test server's bind reverse-resolves 127.0.0.1, which stalls on macOS

http.server's HTTPServer.server_bind() calls getfqdn() on the bind address just
to fill server_name, which nothing in local-server.py reads. On the macos-15
runner that lookup takes ~30s, so the PORT line lands well past every caller's
discovery budget (#870).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* PROBE: macos-15 + startup timing (not for merge)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Move the macOS CI legs to macos-15, and route the last two port waits through the shared helper

84 and 100 kept their own PORT poll loops with 10s budgets, the copies #869 did
not reach; both use discover_server_port now, so there is one implementation
left.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cut the server_bind comment to the why

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 06:21:39 +00:00
Xavier Roche
fac60dff4a armhf crash reports have no frames, and the empty backtrace fails the build (#893)
* armhf crash reports have no frames, and the empty backtrace fails the build

gcc emits no unwind tables on armhf, so backtrace() comes back empty and the
handler printed a frameless report, which the crash tests read as a failure and
which left 3.49.15-1 stuck at Build-Attempted there. Ask for
-fasynchronous-unwind-tables where the compiler takes it, and say why the report
has no frames when the unwinder still returns nothing.

Closes #892

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Let ASan tolerate the backtrace shim's link order

An LD_PRELOAD library loads ahead of the executable's own libasan, which ASan
refuses by default, so the sanitize leg never reached the crash it was meant to
inspect. Same waiver the other interposer tests carry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep an empty backtrace a failure everywhere but 32-bit ARM

Relaxing test 80's skip to the shared message prefix let it swallow the new
"unwinding failed" wording too, so a build that traced nothing anywhere would
have gone green on every architecture. Skip only where nothing can be done
about it: the OS-less case, and 32-bit ARM if its toolchain still refuses to
unwind. Test 143 now pins the exact wording rather than the prefix it shares
with the OS-less notice, and its control run skips the symbolizer it has no
reason to spawn.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Declare the new backtrace test's Windows skip

The Windows job compares the skip set exactly, so a test that skips there for a
good reason still fails the gate until it is named.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 18:22:08 +00:00
Xavier Roche
264478c846 HTTrack ships a working GUI on macOS that no Mac user can find (#890)
* Ship a macOS app bundle

macOS users install HTTrack through Homebrew and get a working WebHTTrack they
are never told about: the formula installs webhttrack and htsserver, and the
only thing missing is something to double-click. tools/macos-app.sh assembles
HTTrack.app from an installed prefix, with the payload under Contents/Resources
so webhttrack keeps resolving htsserver and its data from its own location, and
a two-line stub in Contents/MacOS for Launch Services.

Nothing about the engine changes. The bundle is possible because webhttrack was
already relocatable and because the data symlink stopped being absolute (#885);
--disable-shared keeps libhttrack inside the binaries so nothing points back at
the staging prefix. That costs no crash diagnostics here, since backtraces are
gated on __linux (src/htsbacktrace.c:50), though it does trip #889 on Linux.

The script verifies what it builds rather than trusting it: no absolute symlink,
the served UI present, no Mach-O still linking the staging prefix, and the
Info.plist version matching the installed binary. configure generates that
plist, so it cannot drift into a fifth hand-maintained version spot of the kind
#884 describes.

CI assembles the bundle, runs the webhttrack smoke through the stub, then moves
the bundle and deletes the prefix it came from and runs it again, which is the
one thing a .app has to survive that a prefix install does not. An ad-hoc
codesign proves it is well formed enough to sign; Gatekeeper needs a Developer
ID and stays out of scope with the DMG.

No custom icon: the largest artwork in the tree is 48x48 and macOS wants 1024,
so that needs a real source asset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* ci: lint the new bundle script, and mark it executable

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drive the bundle through a make target, and run it with bash

Adds a macos-app target so assembling the bundle goes through the build system
the way make deb does, rather than CI reaching for the script directly.

It runs the script with $(BASH), not $(SHELL). automake's SHELL is /bin/sh, and
the script uses process substitution, so under dash it died partway: the payload
was already copied by then and only the verification was skipped, leaving a
bundle that looked built and had been checked by nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make the bundle checks catch what they were missing

The smoke put the bundle's own bin on $PATH, so a launcher that ignored its own
location and just ran "webhttrack" passed as readily as the real one, which is
the most likely way a stub breaks. Dropping $prefix/bin from $PATH kills that:
the browser stub is found through webhttrack's SRCHPATH, not $PATH, so nothing
else needed it.

The bundle also shipped libtool .la files, static archives and include/, none of
them loadable from a static build and the .la files carrying the staging prefix
in libdir=. They are pruned, and a text sweep now fails on any remaining file
that embeds that prefix, which otool cannot see because it reads load commands
only. The prefix is resolved to an absolute path first, or a relative --prefix
made that grep match nothing.

Also: the symlink scan asserts it scanned something, and CFBundleVersion is
compared as well as CFBundleShortVersionString.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assemble the bundle with POSIX sh, not $(BASH)

$(BASH) is not a reliable bash on macOS. /bin/sh there is bash in sh-mode, which
presets $BASH to the path it was invoked as, and AC_PATH_PROGS honours a
pre-set value rather than searching, so configure reports "checking for bash...
/bin/sh". That shell rejects process substitution, and the bundle job died on
it.

Rather than hunt for a real bash, the script no longer needs one: the three
process substitutions become temp-file loops, pipefail goes (not POSIX), and
the target is back on the ordinary $(SHELL) like deb:. shellcheck now reads it
as sh, so a bashism creeping back fails lint instead of macOS CI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 20:05:20 +02:00
Xavier Roche
8f383f4caf The installed html symlink is absolute, so an install tree cannot be moved (#888)
* Make the installed html symlink relative

The install-data-hook linked share/httrack/html to an absolute $(htmldir), so
an installed tree only worked at the prefix it was configured for: the link
dangled under DESTDIR staging and in any relocated copy, and webhttrack then
failed its test -d "${DISTPATH}/html" check and exited. It now emits a
relative link when htmldir sits under datadir.

The hook also sat in html/Makefile.am while writing into $(datadir)/httrack,
the directory lang/Makefile.am declares and populates, and it hardcoded
$(prefix)/share instead of following datadir. Moving it to lang/ with
$(langrootdir) drops that cross-subdirectory install-order dependency. A
stale symlink was previously left in place, so an upgrade kept an absolute
one; it is now replaced, a real directory is left alone with a diagnostic,
and a new uninstall-hook removes what install created. A moved datadir still
defeats webhttrack's own data search for an unrelated reason (#887).

The smoke test asserts no installed symlink is absolute and that the link,
resolved inside a copy of the tree at another path, stays inside that copy
and lands on the served UI. It now runs on Linux as well as macOS, since
Linux is where the affected packaging is consumed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* ci: match the Linux build job's package list

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: put the stub browser where webhttrack looks first

webhttrack searches its own SRCHPATH (starting at $BINWD, then /usr/local/bin,
/usr/share/bin, /usr/bin) and only appends $PATH after it, so shadowing
x-www-browser through PATH only worked on hosts that have no real one. The
GitHub Linux runners ship /usr/bin/x-www-browser as Edge, which won the search
and then aborted on its SUID sandbox, failing the new Linux job.

Write the stub to $prefix/bin instead, which is SRCHPATH[0] on every platform,
and remove it in the teardown.

Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 14:52:28 +00:00
499 changed files with 22717 additions and 5718 deletions

3
.gitattributes vendored
View File

@@ -6,6 +6,3 @@
# bash die on $'\r' on every line of them.
*.test text eol=lf
*.sh text eol=lf
# Scoped to this file so a union can never touch build logic.
tests/tests-list.mk merge=union

34
.github/workflows/appstream-network.yml vendored Normal file
View File

@@ -0,0 +1,34 @@
# The per-PR lint runs --no-net so an httrack.com outage can't red an unrelated
# PR; that also blinds it to a broken screenshot URL, which this job catches.
name: appstream metainfo (networked)
on:
schedule:
- cron: "12 6 * * 1"
workflow_dispatch:
permissions:
contents: read
jobs:
validate:
name: appstreamcli validate (network)
# Scheduled workflows run per-fork independently; skip anywhere but upstream.
if: github.repository == 'xroche/httrack'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- name: Install appstreamcli
run: |
set -euo pipefail
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends appstream
appstreamcli --version
# appstreamcli exits 3 on a warning, so a dead screenshot fails the job.
- name: AppStream metainfo validation (network)
run: |
set -euo pipefail
appstreamcli validate --explain \
html/server/div/com.httrack.WebHTTrack.metainfo.xml

View File

@@ -38,10 +38,11 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
appstream appstream-util
- name: Configure
run: |
@@ -79,6 +80,27 @@ jobs:
- name: Build
run: make -j"$(nproc)"
# FTP_PASV is 1 in every build, so #if !FTP_PASV never reaches a compiler
# and rots unnoticed (#1091). Compiling is all this proves; nothing runs it.
- name: Compile the FTP active-mode branch
run: |
set -euo pipefail
objs="src/libhttrack_la-htsftp.lo src/libhttrack_la-htsftp.o src/.libs/libhttrack_la-htsftp.o"
# accept() marks active-mode code: absent from the object Build made,
# present below. One-sided, it passes on a file that ignored the define.
syms=$(nm --undefined-only src/.libs/libhttrack_la-htsftp.o)
if grep -qw accept <<<"$syms"; then
echo "::error::accept() is reachable with FTP_PASV=1; the marker no longer discriminates"
exit 1
fi
rm -f $objs
# Through CC: overriding CPPFLAGS would drop configure's
# -D_FORTIFY_SOURCE and the object-size warnings that come with it.
make -C src libhttrack_la-htsftp.lo CC="$CC -DFTP_PASV=0 -Werror"
syms=$(nm --undefined-only src/.libs/libhttrack_la-htsftp.o)
grep -qw accept <<<"$syms" || { echo "::error::-DFTP_PASV=0 built no active-mode code"; exit 1; }
rm -f $objs
# A backstop only: tests/test-timeout.sh bounds each test, and a healthy run
# is a minute here, two on macOS. Without it a stall ran to the job's 6h default.
- name: Test
@@ -107,7 +129,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -137,13 +159,115 @@ jobs:
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# Reproduces the buildds that refuse TEST-NET-1 instead of dropping it, where
# every fixture needing a stalled connect must skip rather than fail (#1108).
hostile-network:
name: build (network refuses TEST-NET-1)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev iproute2
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
./configure
- name: Build
run: make -j"$(nproc)"
- name: Test on a network that refuses TEST-NET-1
timeout-minutes: 20
run: |
set -euo pipefail
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
tools/hostile-net.sh make check -j"$jobs"
# A refusal landing before connect() returns skips on the older probe
# instead, leaving the settle this leg exists to guard unrun.
cat tests/246_engine-connect-stop.log
grep -q 'within 500 ms' tests/246_engine-connect-stop.log
- name: Print the test log on failure
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# The Ubuntu buildds run a development release every runner above is well behind.
# That gap is what shipped 3.49.18 broken on every Ubuntu architecture but
# riscv64, when 25.10 made uutils the default coreutils (#1042).
ubuntu-devel:
name: build (Ubuntu devel, uutils coreutils)
runs-on: ubuntu-24.04
container: ubuntu:devel
timeout-minutes: 40
steps:
# Before checkout: the container has no git, so submodules would be lost.
- name: Install build dependencies
run: |
set -euo pipefail
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
git ca-certificates python3 procps
- name: Assert the coreutils dispatch on argv[0]
# The property #1042 broke on, not the version banner, which uutils has
# already reworded once. Without it the job quietly becomes a slower copy
# of the one above.
run: |
set -euo pipefail
ln -s "$(command -v sleep)" /tmp/not-a-coreutil
if /tmp/not-a-coreutil 0; then
echo "::error::coreutils here take any argv[0]; this image no longer guards uutils" >&2
exit 1
fi
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
./configure
# A renamed dev package would disable a codec or TLS here, and the tests
# that cover them would SKIP rather than fail.
grep -q "define HTS_USEBROTLI 1" config.h
grep -q "define HTS_USEZSTD 1" config.h
grep -q "define HTS_USEOPENSSL 1" config.h
- name: Build
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 25
run: |
set -euo pipefail
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
- name: Print the test log on failure
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# Portability: build and test on macOS (Darwin/clang) on a native runner --
# no VM. The tree has no __APPLE__ branches, so Darwin exercises the
# generic-Unix path on a second libc and kernel. brew's openssl@3 is keg-only,
# so point configure at it; everything else is in the SDK or default paths.
macos:
name: build (macOS arm64, clang)
runs-on: macos-14
runs-on: macos-15
steps:
- uses: actions/checkout@v7
with:
@@ -191,7 +315,7 @@ jobs:
# temp prefix, then check webhttrack brings up htsserver and serves the UI.
webhttrack-macos:
name: webhttrack smoke (macOS arm64)
runs-on: macos-14
runs-on: macos-15
steps:
- uses: actions/checkout@v7
with:
@@ -217,6 +341,112 @@ jobs:
- name: Smoke-test webhttrack
run: bash tests/webhttrack-smoke.sh "$RUNNER_TEMP/inst"
# Same smoke on the platform that ships webhttrack as a package: the install layout
# it asserts (a relative data symlink, #885) is what Debian consumes, and the macOS
# job alone left that untested on Linux.
webhttrack-linux:
name: webhttrack smoke (Linux x86-64)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
- name: Build and install into a temp prefix
run: |
set -euo pipefail
./bootstrap
./configure --prefix="$RUNNER_TEMP/inst"
make -j"$(nproc)"
make install
- name: Smoke-test webhttrack
run: bash tests/webhttrack-smoke.sh "$RUNNER_TEMP/inst"
# The macOS app bundle (#886): assemble it, then prove it still works after being
# moved, which is the only thing a .app has to survive that a prefix install does not.
macos-app:
name: macOS app bundle (arm64)
runs-on: macos-15
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
brew install autoconf automake libtool autoconf-archive brotli zstd
# --disable-shared keeps libhttrack inside the binaries, so the bundle carries no
# absolute dylib path back to the staging prefix.
- name: Build and install into a temp prefix
run: |
set -euo pipefail
ssl="$(brew --prefix openssl@3)"
brewp="$(brew --prefix)"
./bootstrap
./configure CPPFLAGS="-I${ssl}/include -I${brewp}/include" \
LDFLAGS="-L${ssl}/lib -L${brewp}/lib" \
--prefix="$RUNNER_TEMP/inst" --disable-shared
make -j"$(sysctl -n hw.ncpu)"
make install
- name: Assemble HTTrack.app
run: |
set -euo pipefail
make macos-app APP_FLAGS="--prefix $RUNNER_TEMP/inst --out $RUNNER_TEMP"
plutil -lint "$RUNNER_TEMP/HTTrack.app/Contents/Info.plist"
- name: Run it through the bundle stub
run: |
set -euo pipefail
app="$RUNNER_TEMP/HTTrack.app"
bash tests/webhttrack-smoke.sh "$app/Contents/Resources" "$app/Contents/MacOS/HTTrack"
- name: Run it again after moving it
run: |
set -euo pipefail
mkdir -p "$RUNNER_TEMP/moved"
mv "$RUNNER_TEMP/HTTrack.app" "$RUNNER_TEMP/moved/"
rm -rf "$RUNNER_TEMP/inst"
app="$RUNNER_TEMP/moved/HTTrack.app"
bash tests/webhttrack-smoke.sh "$app/Contents/Resources" "$app/Contents/MacOS/HTTrack"
# The load-command check cannot fail while the loader can still reach Homebrew (#901).
- name: Run it with Homebrew's libraries out of reach
run: |
set -euo pipefail
app="$RUNNER_TEMP/moved/HTTrack.app"
# Without this the hiding below proves nothing: an empty Frameworks passes it.
ls "$app/Contents/Frameworks"/libssl.*.dylib >/dev/null
# The kegs, not just the opt symlinks: a source build records the Cellar path.
hidden=""
for keg in openssl@3 brotli zstd; do
for d in "$(brew --prefix)/opt/$keg" "$(brew --prefix)/Cellar/$keg"; do
if [ -e "$d" ]; then sudo mv "$d" "$d.hidden"; hidden="$hidden $d"; fi
done
done
trap 'for d in $hidden; do sudo mv "$d.hidden" "$d"; done' EXIT
test -n "$hidden"
bash tests/webhttrack-smoke.sh "$app/Contents/Resources" "$app/Contents/MacOS/HTTrack"
# The release path (macos-release.yml) minus Apple, so a bundle that cannot be
# signed or packed fails on the PR rather than on release day.
- name: Sign and pack the bundle, ad hoc
run: |
set -euo pipefail
bash tools/macos-release.sh --app "$RUNNER_TEMP/moved/HTTrack.app" \
--identity - --skip-notarize
# Portability/hardening: 32-bit (i386) build on the x86-64 runner via multilib
# -- no extra hardware. Exercises the 32-bit size_t/pointer ABI, where size
# and bounds math can truncate or wrap in ways 64-bit never reveals (the axis
@@ -234,7 +464,7 @@ jobs:
run: |
set -euo pipefail
sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential gcc-multilib autoconf automake libtool \
autoconf-archive zlib1g-dev:i386 libssl-dev:i386 \
@@ -277,7 +507,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -332,7 +562,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev
@@ -381,7 +611,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -404,6 +634,33 @@ jobs:
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
run: bash fuzz/run-fuzzers.sh fuzz check
# Catches #1030: the job above links ASan, whose runtime already defines the
# sancov hooks the instrumentation calls, hiding a program left without one.
fuzz-no-sanitizer:
name: build (fuzzers, no sanitizer runtime, clang)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
- name: Configure (fuzzers, static)
run: |
set -euo pipefail
autoreconf -fi
./configure CC=clang --enable-fuzzers --disable-shared
- name: Build
run: make -j"$(nproc)"
# Optional-dependency build: compile and test with HTTPS/OpenSSL disabled --
# the configuration users on minimal systems build, and one libssl is not even
# installed here so configure cannot silently re-enable it. The matrix above
@@ -420,7 +677,7 @@ jobs:
- name: Install build dependencies (no libssl)
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive zlib1g-dev
@@ -457,7 +714,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -536,7 +793,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -593,21 +850,23 @@ jobs:
man/makeman.sh
src/htsbasiccharsets.sh
src/htsentities.sh
src/webhttrack
src/webhttrack.in
tests/*.sh
tests/*.test
tools/mkdeb.sh
tools/*.sh
steps:
- uses: actions/checkout@v7
- name: Install linters
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
# noble ships shfmt 3.8.0 (universe), matching the pinned local dev
# version; use it rather than fetching a release binary from github.com.
sudo apt-get install -y --no-install-recommends shellcheck shfmt
sudo apt-get install -y --no-install-recommends shellcheck shfmt appstream \
appstream-util
shfmt --version
appstreamcli --version
- name: shellcheck
run: shellcheck $SHELL_SCRIPTS
@@ -615,6 +874,11 @@ jobs:
- name: shfmt
run: shfmt -d -i 4 $SHELL_SCRIPTS
# A misnamed test falls outside the glob "make check" runs, so it goes
# missing without failing anything.
- name: test file names
run: bash tests/check-test-names.sh
# MSBuild rejects a malformed .vcxproj with a bare MSB4025 and no build, so
# catch it here in seconds rather than on a Windows runner minutes in.
- name: XML well-formedness (MSBuild project files)
@@ -625,6 +889,16 @@ jobs:
echo "ok $f"
done
# A software centre quietly ignores a metainfo it cannot parse. Errors and
# warnings fail; --no-net so the gate is on the file, not httrack.com's uptime.
- name: AppStream metainfo validation
run: |
set -euo pipefail
appstreamcli validate --no-net --explain \
html/server/div/com.httrack.WebHTTrack.metainfo.xml
appstream-util validate-relax --nonet \
html/server/div/com.httrack.WebHTTrack.metainfo.xml
# Check clang-format on CHANGED LINES ONLY. The engine predates clang-format
# (it was shaped by an old Visual Studio formatter) and does not round-trip,
# so we never reformat the whole tree -- only the lines a PR touches.
@@ -645,7 +919,7 @@ jobs:
| sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc >/dev/null
echo "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-19 main" \
| sudo tee /etc/apt/sources.list.d/llvm-19.list >/dev/null
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends clang-format-19
# The clang-format-19 package ships the git-clang-format driver;
# expose it unsuffixed so "git clang-format" finds it.

View File

@@ -33,12 +33,12 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev
- uses: github/codeql-action/init@v4
- uses: github/codeql-action/init@v4.37.4
with:
languages: c-cpp
build-mode: manual
@@ -67,6 +67,6 @@ jobs:
./configure
make -j"$(nproc)"
- uses: github/codeql-action/analyze@v4
- uses: github/codeql-action/analyze@v4.37.4
with:
category: "/language:c-cpp"

116
.github/workflows/cross-arch.yml vendored Normal file
View File

@@ -0,0 +1,116 @@
# Cross-compiles for the Debian architectures no GitHub runner exists for.
# 3.49.17 failed on four of them at once and every failure was a compile or
# assemble error that x86-64 and arm64 cannot produce, so the buildd was the
# first thing to see the code.
name: Cross-arch
on:
push:
branches: [master]
pull_request:
workflow_dispatch:
# Least privilege: the workflow only needs to read the repo.
permissions:
contents: read
# Cancel superseded runs on the same branch or PR.
concurrency:
group: cross-arch-${{ github.ref }}
cancel-in-progress: true
jobs:
cross:
name: cross (${{ matrix.arch }})
runs-on: ubuntu-24.04
# sid, not stable: the point is to predict what the sid buildds will say.
container: debian:sid
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
# arch selects the hardening flags, triplet is the toolchain prefix.
# The first four are 3.49.17's failures.
- { arch: armhf, triplet: arm-linux-gnueabihf }
- { arch: powerpc, triplet: powerpc-linux-gnu }
- { arch: hppa, triplet: hppa-linux-gnu }
- { arch: loong64, triplet: loongarch64-linux-gnu }
- { arch: sh4, triplet: sh4-linux-gnu }
- { arch: m68k, triplet: m68k-linux-gnu }
- { arch: sparc64, triplet: sparc64-linux-gnu }
- { arch: riscv64, triplet: riscv64-linux-gnu }
- { arch: s390x, triplet: s390x-linux-gnu }
env:
DEB_HOST_ARCH: ${{ matrix.arch }}
TRIPLET: ${{ matrix.triplet }}
ZPREFIX: /tmp/zlib-${{ matrix.arch }}
steps:
- name: Install the toolchain
run: |
set -euo pipefail
# deb-src, so zlib below comes from the archive and not a pinned URL.
# In place: a second stanza has to repeat Signed-By, and apt rejects
# the same source carrying two spellings of the keyring.
sed -i 's/^Types: deb$/Types: deb deb-src/' \
/etc/apt/sources.list.d/debian.sources
apt-get update
apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
dpkg-dev git ca-certificates \
"gcc-$TRIPLET" "libc6-dev-$DEB_HOST_ARCH-cross"
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Cross-build zlib
run: |
set -euo pipefail
# Mandatory, and no ports arch has a zlib1g-dev:<arch> to
# multiarch-install. -fPIC because libhttrack.so links it.
mkdir -p /tmp/zsrc && cd /tmp/zsrc
# Unpatched: Debian's arch-specific patches expect its own rules to
# build them (s390x's vector CRC pulls a header they add), and we only
# need something to link -lz against.
apt-get source --download-only zlib1g
dpkg-source --skip-patches -x ./*.dsc src
cd src
CHOST="$TRIPLET" CC="$TRIPLET-gcc" CFLAGS="-O2 -fPIC" \
./configure --prefix="$ZPREFIX" --static
# configure detects s390x's vector CRC, whose sources live in the
# contrib/ the .dfsg repack drops: drop the define and the object it
# would build. Nothing here needs a fast CRC.
sed -i -E 's/(-DHAVE_S390X_VX|crc32_vx\.l?o)//g' Makefile
make -j"$(nproc)"
make install
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
mkdir -p /tmp/bld && cd /tmp/bld
# The buildd's own per-arch flags: -D_FILE_OFFSET_BITS=64 is the shim
# hazard, and -fstack-clash-protection is not offered everywhere.
eval "$(dpkg-buildflags --export=sh)"
# A cross AC_TRY_RUN answers "cross", which would swap in the bundled
# snprintf that no buildd ever builds. Every glibc passes these.
# --disable-https: no ports arch has a cross libssl, so the TLS paths
# are the one part of the tree this matrix does not compile.
"$GITHUB_WORKSPACE/configure" --host="$TRIPLET" \
--build="$(dpkg-architecture -qDEB_BUILD_GNU_TYPE)" --disable-https \
ac_cv_have_working_snprintf=yes ac_cv_have_working_vsnprintf=yes \
CPPFLAGS="${CPPFLAGS:-} -I$ZPREFIX/include" \
LDFLAGS="${LDFLAGS:-} -L$ZPREFIX/lib"
- name: Build
run: make -C /tmp/bld -j"$(nproc)"
- name: Build the test artifacts
# Nothing runs here, but the shims are where three of the four 3.49.17
# failures were: they only build under `check`. Asserted, because an
# empty TESTS= would otherwise pass having built nothing.
run: |
set -euo pipefail
make -C /tmp/bld -j"$(nproc)" check TESTS=
test -f /tmp/bld/tests/.libs/libaltstackprobe.so

162
.github/workflows/macos-release.yml vendored Normal file
View File

@@ -0,0 +1,162 @@
# Signed, notarized HTTrack.dmg (#901). Its own workflow because a fork's pull request
# never gets the Developer ID secrets, and each run spends two notarization submissions.
name: macOS release
on:
push:
tags: ["[0-9]*"]
workflow_dispatch:
permissions:
contents: read
jobs:
dmg:
name: signed DMG (macOS arm64)
runs-on: macos-15
permissions:
contents: write # attach the DMG to the release the tag belongs to
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
# Notarization is minutes in; a missing secret should not surface there.
- name: Check the signing secrets are present
env:
MACOS_CERT_P12: ${{ secrets.MACOS_CERT_P12 }}
MACOS_CERT_PASSWORD: ${{ secrets.MACOS_CERT_PASSWORD }}
MACOS_NOTARY_KEY: ${{ secrets.MACOS_NOTARY_KEY }}
MACOS_NOTARY_KEY_ID: ${{ secrets.MACOS_NOTARY_KEY_ID }}
MACOS_NOTARY_ISSUER_ID: ${{ secrets.MACOS_NOTARY_ISSUER_ID }}
run: |
set -euo pipefail
hint="see secrets/apple/README.md in httrack-works"
: "${MACOS_CERT_P12:?unset repository secret ($hint)}"
: "${MACOS_CERT_PASSWORD:?unset repository secret ($hint)}"
: "${MACOS_NOTARY_KEY:?unset repository secret ($hint)}"
: "${MACOS_NOTARY_KEY_ID:?unset repository secret ($hint)}"
: "${MACOS_NOTARY_ISSUER_ID:?unset repository secret ($hint)}"
- name: Install build dependencies
run: |
set -euo pipefail
brew install autoconf automake libtool autoconf-archive brotli zstd
# --disable-shared keeps libhttrack inside the binaries.
- name: Build and install into a temp prefix
run: |
set -euo pipefail
ssl="$(brew --prefix openssl@3)"
brewp="$(brew --prefix)"
./bootstrap
./configure CPPFLAGS="-I${ssl}/include -I${brewp}/include" \
LDFLAGS="-L${ssl}/lib -L${brewp}/lib" \
--prefix="$RUNNER_TEMP/inst" --disable-shared
make -j"$(sysctl -n hw.ncpu)"
make install
- name: Assemble HTTrack.app
run: |
set -euo pipefail
make macos-app APP_FLAGS="--prefix $RUNNER_TEMP/inst --out $RUNNER_TEMP"
plutil -lint "$RUNNER_TEMP/HTTrack.app/Contents/Info.plist"
- name: Import the Developer ID certificate
env:
CERT_P12: ${{ secrets.MACOS_CERT_P12 }}
CERT_PASSWORD: ${{ secrets.MACOS_CERT_PASSWORD }}
run: |
set -euo pipefail
kc="$RUNNER_TEMP/signing.keychain-db"
kcpass="$(openssl rand -base64 24)"
echo "::add-mask::$kcpass"
security create-keychain -p "$kcpass" "$kc"
# Notarization holds the job well past the five-minute default lock.
security set-keychain-settings -lut 21600 "$kc"
security unlock-keychain -p "$kcpass" "$kc"
# Armed before the file exists: a failed import must not leave the key on a
# disk that later steps run repository code against.
trap 'rm -f "$RUNNER_TEMP/cert.p12"' EXIT HUP INT TERM
# openssl, not base64: macOS spells the decode flag -D and GNU spells it -d.
printf '%s' "$CERT_P12" | openssl base64 -d -A -out "$RUNNER_TEMP/cert.p12"
security import "$RUNNER_TEMP/cert.p12" -k "$kc" -P "$CERT_PASSWORD" \
-T /usr/bin/codesign
# Without this codesign blocks on a UI prompt no runner can answer.
security set-key-partition-list -S apple-tool:,apple:,codesign: \
-s -k "$kcpass" "$kc" >/dev/null
security list-keychains -d user -s "$kc"
id="$(security find-identity -v -p codesigning "$kc" |
awk '/Developer ID Application/ {print $2; exit}')"
test -n "$id" || {
security find-identity -v -p codesigning "$kc" >&2
echo "the .p12 holds no Developer ID Application identity" >&2
exit 1
}
echo "SIGN_IDENTITY=$id" >>"$GITHUB_ENV"
- name: Sign, notarize and pack the DMG
env:
NOTARY_KEY: ${{ secrets.MACOS_NOTARY_KEY }}
NOTARY_KEY_ID: ${{ secrets.MACOS_NOTARY_KEY_ID }}
NOTARY_ISSUER_ID: ${{ secrets.MACOS_NOTARY_ISSUER_ID }}
run: |
set -euo pipefail
key="$RUNNER_TEMP/notary.p8"
trap 'rm -f "$key"' EXIT HUP INT TERM
printf '%s' "$NOTARY_KEY" | openssl base64 -d -A -out "$key"
bash tools/macos-release.sh --app "$RUNNER_TEMP/HTTrack.app" \
--identity "$SIGN_IDENTITY" --out "$RUNNER_TEMP" \
--notary-key "$key" --notary-key-id "$NOTARY_KEY_ID" \
--notary-issuer "$NOTARY_ISSUER_ID"
# The DMG round-trip is the one thing that can strip the stapled ticket, so assess
# the copy that came out of it rather than the app the signing step still holds.
- name: Assess a quarantined copy out of the DMG
run: |
set -euo pipefail
dmg="$(echo "$RUNNER_TEMP"/HTTrack-*.dmg)"
test -f "$dmg"
hdiutil attach -nobrowse -readonly -mountpoint "$RUNNER_TEMP/mnt" "$dmg"
trap 'hdiutil detach "$RUNNER_TEMP/mnt" >/dev/null || true' EXIT
app="$RUNNER_TEMP/quarantined/HTTrack.app"
ditto "$RUNNER_TEMP/mnt/HTTrack.app" "$app"
# On the root only, as a real download lands it; -r would also chase symlinks.
xattr -w com.apple.quarantine "0083;00000000;Safari;" "$app"
# --no-cache, or the verdict already cached for this cdhash answers instead.
log="$RUNNER_TEMP/assess.log"
spctl --assess --type exec --ignore-cache --no-cache -vv "$app" >"$log" 2>&1 || {
cat "$log" >&2
exit 1
}
cat "$log"
grep -q "source=Notarized Developer ID" "$log"
# spctl alone can be answered online; this is the offline-launch proof.
xcrun stapler validate "$app"
# Not a Gatekeeper check: the stub is a script, so nothing here goes through
# LaunchServices. It proves the hardened runtime did not break the dylib loads.
bash tests/webhttrack-smoke.sh "$app/Contents/Resources" "$app/Contents/MacOS/HTTrack"
- name: Upload the DMG
uses: actions/upload-artifact@v7
with:
name: HTTrack-macos-arm64-dmg
path: ${{ runner.temp }}/HTTrack-*.dmg
if-no-files-found: error
# Releases are cut by hand, so a tag can land before one exists.
- name: Attach the DMG to the release
if: github.ref_type == 'tag'
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
if gh release view "$GITHUB_REF_NAME" >/dev/null 2>&1; then
gh release upload "$GITHUB_REF_NAME" "$RUNNER_TEMP"/HTTrack-*.dmg --clobber
else
echo "no release $GITHUB_REF_NAME yet -- the DMG is on this run's artifacts"
fi
- name: Delete the signing keychain
if: always()
run: security delete-keychain "$RUNNER_TEMP/signing.keychain-db" || true

View File

@@ -34,7 +34,7 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get update || true # a third-party source in the runner image can 403 (#1097)
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
@@ -52,6 +52,15 @@ jobs:
pip install --quiet playwright
playwright install --with-deps chromium
# The address is on screen in the wizard panes and names the mirror folder,
# so the crawl needs a presentable host rather than 127.0.0.1 and a random
# port. example.com is reserved for documentation (RFC 2606).
- name: Let the crawl address a named site on port 80
run: |
set -euo pipefail
echo "127.0.0.1 www.example.com" | sudo tee -a /etc/hosts
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80 # bind 80 unprivileged
- name: Capture
env:
LANGN: ${{ inputs.lang || '1' }}
@@ -61,6 +70,7 @@ jobs:
# --root is the dist root; the source tree already holds every piece
# htsserver reads, so this needs no install.
python3 tools/screenshot-walk.py --htsserver src/htsserver --root . \
--site-host www.example.com --site-port 80 \
--lang "$LANGN" --scale "$SCALE" --out shots
- name: Upload the screens

View File

@@ -14,6 +14,14 @@ on:
permissions:
contents: read
# The suite watchdog reports as a commit status, the only channel that
# outlives the runner it is reporting on (#795).
statuses: write
# Cancel superseded runs on the same branch or PR.
concurrency:
group: windows-build-${{ github.ref }}
cancel-in-progress: true
jobs:
libhttrack:
@@ -32,6 +40,9 @@ jobs:
- uses: actions/checkout@v7
with:
submodules: recursive # coucal lives in src/coucal
# Or the job token stays in .git/config, where every test the suite
# runs can read it.
persist-credentials: false
# Located through vswhere rather than microsoft/setup-msbuild: the repo
# only allows GitHub-owned actions.
@@ -166,133 +177,17 @@ jobs:
shell: bash
working-directory: tests
timeout-minutes: 45
env:
# Through the environment, never argv, which the process list exposes.
WATCHDOG_TOKEN: ${{ secrets.GITHUB_TOKEN }}
WATCHDOG_REPO: ${{ github.repository }}
# github.sha here is the PR's merge commit, so statuses posted against it stay out of the PR's checks UI.
WATCHDOG_SHA: ${{ github.sha }}
WATCHDOG_CONTEXT: windows-suite (${{ matrix.platform }}, ${{ matrix.configuration }})
WATCHDOG_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -u
bin="$(cygpath -u "$GITHUB_WORKSPACE")/src/${{ matrix.platform }}/${{ matrix.configuration }}"
export PATH="$bin:$PATH"
command -v httrack >/dev/null || { echo "::error::no httrack.exe in $bin"; exit 1; }
# httrack.exe is native, so MSYS rewrites any argument shaped like a
# POSIX path, and a URL path is shaped exactly like one: "/a/b.html"
# reached the engine as "C:/Program Files/Git/a/b.html". Switch that
# off, and hand the tests a TMPDIR that is already a Windows path.
export MSYS_NO_PATHCONV=1
export MSYS2_ARG_CONV_EXCL='*'
TMPDIR="$(cygpath -m "$RUNNER_TEMP")"
export TMPDIR
# Mirror what configure hands the suite. LC_ALL sets the codeset MSYS maps
# a UTF-8 mirror name onto UTF-16 with, which the intl crawls "test -f".
export HTTPS_SUPPORT=yes BROTLI_ENABLED=yes ZSTD_ENABLED=yes
export LC_ALL=C.UTF-8
# A wedged crawl must not eat the job's timeout budget. timeout(1)'s
# signals can't reap a native httrack.exe (MSYS signals don't reach it),
# so a hang orphaned processes that starved the runner; run_with_timeout
# TerminateProcess-es the whole tree. 600s is unchanged: it clears the
# 540s a three-pass crawl may legitimately take under local-crawl.sh's
# own watchdogs, against a slowest healthy test here of 39s.
. ./testlib.sh
per_test=600
# The whole suite must give up before the step timeout above. A cancelled
# step keeps neither its log nor the artifacts the later if:always()
# steps would upload, so an overrun that ends in a cancel tells us
# nothing; failing on our own terms keeps both. Healthy runs take 8-9
# min. The check sits between tests, so the step can still reach 25 min
# plus one per-test budget, and that worst case stays inside the 45.
suite_deadline=1500
started=$SECONDS
# Survives into the artifact even if the tail of the step log does not.
progress=suite-progress.log
: >"$progress"
pass=0 fail=0 skip=0 failed="" skipped="" deadline=0
# Globbed, not enumerated: a new NNN_engine-*.test or NNN_local-*.test
# is picked up automatically instead of silently getting zero coverage.
for t in 00_runnable.test *_engine-*.test *_zlib-*.test \
*_local-*.test *_watchdog*.test *_crawl_proxy_https.test \
*_crawl-log-salvage.test; do
elapsed=$((SECONDS - started))
if [ "$elapsed" -ge "$suite_deadline" ]; then
echo "::error::suite deadline: ${elapsed}s elapsed, stopping before $t"
echo "DEADLINE before $t after ${elapsed}s" >>"$progress"
# Per-test start times, so the slow ones are named rather than guessed.
sed 's/^/ /' "$progress"
deadline=1
break
fi
echo "RUN $t at ${elapsed}s" >>"$progress"
rc=0
# Same guard "make check" uses on POSIX, so a wedge is diagnosed the
# same way on every platform. It dumps before it kills, which a bare
# run_with_timeout cannot: by the time that returns, the tree whose
# stack we wanted is already gone.
HTTRACK_TEST_TIMEOUT=$per_test bash ./test-timeout.sh "$t" >"$t.log" 2>&1 || rc=$?
case "$rc" in
0) pass=$((pass + 1)); echo "PASS $t" ;;
77) skip=$((skip + 1)) skipped="$skipped $t"; echo "SKIP $t" ;;
124)
fail=$((fail + 1)) failed="$failed $t"
# test-timeout.sh has already written the process list, the stacks
# and the killed crawl's own logs into $t.log.
echo "FAIL $t (timed out, tree killed)"
tail -n 25 "$t.log" | sed 's/^/ /'
;;
*)
fail=$((fail + 1)) failed="$failed $t"
echo "FAIL $t (exit $rc)"
# These assert with `test "$(...)" == "..." || exit 1`, which
# says nothing at all on failure. Re-run traced, still bounded.
run_with_timeout "$per_test" bash -x "$t" >>"$t.log" 2>&1 || true
tail -n 25 "$t.log" | sed 's/^/ /'
;;
esac
echo "$rc $t" >>"$progress"
# An orphaned native httrack.exe spins and starves the runner, which
# is how this job dies with "lost communication" rather than a plain
# timeout. Clear them between tests and name whoever leaked them.
reap_leftover_processes "$t" | tee -a "$progress"
done
echo "ran=$((pass + fail + skip)) pass=$pass fail=$fail skip=$skip" |
tee -a "$GITHUB_STEP_SUMMARY"
# Every gate here exits 77, so an all-skipped suite would report green having
# tested nothing: pin the skips, and floor the passes in case the glob empties.
# One name per line, so two branches each appending one don't collide on the
# same line; compared as a sorted set below, so glob discovery order can't
# cause a false mismatch either.
# footer-overflow and purge-longpath skip on Windows (need a path past MAX_PATH);
# crange pending #581;
# webdav-default and webdav-mime need a reapable background listener, which MSYS cannot give them;
# badmtime needs a filesystem that stores an mtime past gmtime's range;
# single-file ends on a GUI half needing htsserver, which this job does not build;
# update-304-leak needs a LeakSanitizer build, which MSVC has no equivalent of;
# crash-symbolize needs backtrace(), which Windows has no equivalent of.
expected_skips="01_engine-footer-overflow.test
100_local-purge-longpath.test
114_local-update-304-leak.test
120_local-proxytrack-webdav-default.test
48_local-crange-memresume.test
71_local-crange-repaircache.test
79_local-proxytrack-webdav-mime.test
80_engine-crash-symbolize.test
88_local-proxytrack-badmtime.test
94_local-single-file.test"
# First, or the deadline reads as an unexplained shortfall in the gates below.
[ "$deadline" -eq 0 ] || { echo "::error::suite did not finish within ${suite_deadline}s"; exit 1; }
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
# Word-split on whitespace (space-joined $skipped, newline-joined
# expected_skips both work) and sort, so the compare is a set, not a string.
got=$(printf '%s\n' $skipped | sort)
want=$(printf '%s\n' $expected_skips | sort)
if [ "$got" != "$want" ]; then
echo "::error::skip set changed from expected; - missing, + newly skipped"
diff -u <(echo "$want") <(echo "$got") | tail -n +3 | sed 's/^/ /'
exit 1
fi
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }
bash ./ci-windows-suite.sh \
"$(cygpath -u "$GITHUB_WORKSPACE")/src/${{ matrix.platform }}/${{ matrix.configuration }}"
- name: Upload the test logs
if: always()

13
.gitignore vendored
View File

@@ -24,8 +24,12 @@ Makefile.in
/config.log
/config.status
/stamp-h1
# src/webhttrack.in is the source; an in-tree build generates this one (#887).
/src/webhttrack
/src/libhttrack.pc
Makefile
.deps/
.dirstamp
.libs/
*.o
*.lo
@@ -34,6 +38,15 @@ Makefile
*.so.*
*.a
# Built into the checkout by an in-tree "make" / "make check".
/src/httrack
/src/htsserver
/src/proxytrack
/tools/Info.plist
/tests/stringoom
/tests/*.log
/tests/*.trs
# make dist output; dist/ holds httrack-gh-release staging artifacts.
/httrack-*.tar.gz
/dist/

View File

@@ -19,8 +19,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
(`request_queue_size`) so macOS/BSD don't drop connections under a parallel
`-c16` bigcrawl the way Python's default backlog of 5 did.
Or run `sh build.sh` to do bootstrap + configure + make in one shot.
- A `tests/NN_*.test` runs only if listed in `tests/tests-list.mk`; an
unregistered file is silently skipped.
- `configure` globs `TESTS` from `tests/[0-9]*_*.test`, so a new test needs no
registration, but an existing build dir keeps the list it was configured with:
`231_test-names.test` goes red until you reconfigure. It also compares the glob
against `git ls-files`, so a test you forgot to `git add` fails there instead of
quietly shrinking CI's suite. Name one outside the pattern and it never runs;
`tests/check-test-names.sh` (also a CI lint) rejects that.
- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
@@ -38,6 +42,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
match that is not on the last line is exposed. Capture the reply, assert the
status line it must carry (an empty, truncated or redirected one is
marker-free too), then match with a here-string: `grep -q M <<<"$reply"`.
`head -c N` and `head -n N` end the same way, and there the reds are
platform-specific: GNU `tail | head -c 3` survives, the uutils coreutils leg
turns the same line into exit 141 with no output at all. Let the reader seek
instead: `od -An -c -j <skip> -N <len> file`.
- Never assert a fault by signal *number*: SIGBUS is 7 on x86, arm, powerpc and
s390x but 10 on hppa, alpha, mips and sparc. Map it with `kill -l "$n"`.
- A fixture that needs a host to stay silent must settle (500ms), re-probe every
socket, and SKIP when one answered: the powerpc and ppc64 buildds refuse
TEST-NET-1 milliseconds after `connect()` where runners drop it.
`tools/hostile-net.sh` runs a command on such a network.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every

View File

@@ -6,13 +6,33 @@ ACLOCAL_AMFLAGS = -I m4
EXTRA_DIST = INSTALL.Linux \
gpl-fr.txt license.txt greetings.txt history.txt \
httrack-doc.html lang.def README.md tools/mkdeb.sh \
tools/macos-app.sh tools/macos-bundle.sh tools/macos-release.sh \
tools/httrack-launcher.c \
tools/Info.plist.in tools/HTTrack.icns \
bootstrap build.sh
# Build the signed Debian packages from a clean source export. Pass the signing
# key and other options through DEB_FLAGS, e.g.:
# make deb DEB_FLAGS="--key BB71C7E6CB1AD8FAF53FE42A60C3AA7180598EFB"
# See tools/mkdeb.sh --help for all options.
# See tools/mkdeb.sh --help for all options. Not $(SHELL): mkdeb.sh is bash, and dash chokes (#891).
DEB_FLAGS =
deb:
$(SHELL) $(top_srcdir)/tools/mkdeb.sh $(DEB_FLAGS)
$(BASH_SHELL) $(top_srcdir)/tools/mkdeb.sh $(DEB_FLAGS)
.PHONY: deb
# Assemble the macOS application bundle from an installed prefix, e.g.
# make macos-app APP_FLAGS="--prefix /tmp/inst --out /tmp"
APP_FLAGS =
macos-app:
$(SHELL) $(top_srcdir)/tools/macos-app.sh --plist tools/Info.plist \
--icon $(top_srcdir)/tools/HTTrack.icns $(APP_FLAGS)
.PHONY: macos-app
# Sign, notarize and pack that bundle into the DMG we publish, e.g.
# make macos-release RELEASE_FLAGS="--app /tmp/HTTrack.app --identity ABCD1234 \
# --notary-key AuthKey.p8 --notary-key-id KEYID --notary-issuer ISSUERID"
# See tools/macos-release.sh for the rest; .github/workflows/macos-release.yml runs it.
RELEASE_FLAGS =
macos-release:
$(SHELL) $(top_srcdir)/tools/macos-release.sh $(RELEASE_FLAGS)
.PHONY: macos-release

View File

@@ -1,6 +1,6 @@
AC_PREREQ([2.71])
AC_INIT([httrack], [3.49.15], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_INIT([httrack], [3.49.20], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_COPYRIGHT([
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998-2015 Xavier Roche and other contributors
@@ -29,15 +29,89 @@ AC_CONFIG_SRCDIR(src/httrack.c)
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_HEADERS(config.h)
AM_INIT_AUTOMAKE([subdir-objects])
# 3:13:0: revision-only bump. htsstrings.h's StringRoomTotal grew saturation in
# its macro body; no installed struct moved and no export came or went.
# 3:12:0: revision-only bump. httrackp gained a tail field (singlefile_state) and
# htsopt.h added volatile qualifiers; no layout moved, nothing went away.
# 3:11:0: revision-only bump. #1001 moved SOCaddr_inetntoa_ out of htsnet.h, adding
# an export where callers had an inline; nothing changed or went away.
# 3:10:0: revision-only bump; only the version macro moved, the engine is untouched.
# 3:9:0: revision-only bump. #991 and #1005 each added an export
# (hts_set_thread_hooks, escape_control_url); nothing changed or went away.
# 3:8:0: revision-only bump, no ABI change.
# 3:7:0: htsblk gained a tail field and lien_back embeds it by value, so
# lien_back.is_update and everything after it shift +8 (httrackp's own tail growth
# moves nothing). Soname stays .so.3: HTTrackQt is the only consumer of the installed
# headers, so a libhttrack4 rename isn't worth it.
# (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
VERSION_INFO="3:7:0"
VERSION_INFO="3:13:0"
AM_MAINTAINER_MODE
AC_USE_SYSTEM_EXTENSIONS
# A real bash, for "make deb" and the test harness. Not searched into BASH: bash presets
# that to its own path, and macOS /bin/sh is a bash, so the macro never searches (#895).
# BASH_SHELL isn't preset the way BASH is, so AC_ARG_VAR needs no guard. Kept ahead of the
# compiler probes so a bad override dies before them.
AC_ARG_VAR([BASH_SHELL], [path to a real (non-POSIX-mode) bash])
# AC_PATH_PROGS drops a relative override and searches instead, which loses the user's intent.
# Nothing quotes $(BASH_SHELL) in the Makefiles, and quoting could not save it anyway: make
# splits on whitespace, expands '$' and treats '#' as a comment before any shell sees it.
case $BASH_SHELL in
*[[[:space:]]]* | *'#'* | *'$'* | *'`'* | *'\'* | *'"'* | *"'"* | *';'* | *'&'* | *'|'* | \
*'<'* | *'>'* | *'('* | *')'* | *'*'* | *'?'* | *'@<:@'* | *'@:>@'* | *'{'* | *'}'*)
AC_MSG_ERROR([BASH_SHELL must not contain shell or make metacharacters, got: $BASH_SHELL]) ;;
'' | [[\\/]]* | ?:[[\\/]]*) ;;
*) AC_MSG_ERROR([BASH_SHELL must be an absolute path, got: $BASH_SHELL]) ;;
esac
hts_bash_override=$BASH_SHELL
AC_PATH_PROGS([BASH_SHELL], [bash], [/bin/bash])
# An absolute override is taken verbatim, so BASH_SHELL=/bin/sh would put #895 back and only
# surface at "make check" or "make deb" (#908). What we found ourselves is only a warning:
# a box with no bash still builds, it just cannot run those two.
AC_MSG_CHECKING([whether $BASH_SHELL is a bash outside POSIX mode])
hts_bash_why=
hts_bash_env=no
# AS_EXECUTABLE_P, not "test -x": the PATH search above already demands a regular file, and
# bash blocks forever reading a FIFO it failed to exec, so -x alone hangs configure (#922).
if ! AS_EXECUTABLE_P(["$BASH_SHELL"]); then
hts_bash_why="not an executable regular file"
elif test -z "$("$BASH_SHELL" -c 'echo "${BASH_VERSINFO[[0]]}"' 2>/dev/null)"; then
# Not BASH_VERSION: that is an ordinary variable, so any shell echoes back a spoofed one.
hts_bash_why="not a bash: it reports no BASH_VERSINFO"
else
# sh-mode bash reports a version too, so only SHELLOPTS tells the two apart.
case $("$BASH_SHELL" -c 'echo ":$SHELLOPTS:"' 2>/dev/null) in
*:posix:*)
hts_bash_why="a bash in POSIX sh-mode"
# POSIXLY_CORRECT and an exported SHELLOPTS do that to every bash on the box, so no path
# can pass and blaming this one would send the user hunting for another. Reading them
# here would not do: configure puts its own shell in posix mode, which sets both.
case $(env -u POSIXLY_CORRECT -u SHELLOPTS "$BASH_SHELL" -c 'echo ":$SHELLOPTS:"' 2>/dev/null) in
'' | *:posix:*) ;; # no "env -u", or posix whatever the environment: blame the path
*) hts_bash_env=yes ;;
esac
;;
esac
fi
if test -z "$hts_bash_why"; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
hts_bash_msg="POSIXLY_CORRECT or SHELLOPTS forces every bash into POSIX sh-mode, $BASH_SHELL included. Clear them for configure and for make, which hands them to make check and make deb: env -u POSIXLY_CORRECT -u SHELLOPTS ..."
if test "$hts_bash_env" = yes; then
if test -n "$hts_bash_override"; then
AC_MSG_ERROR([$hts_bash_msg])
fi
AC_MSG_WARN([$hts_bash_msg])
else
if test -n "$hts_bash_override"; then
AC_MSG_ERROR([BASH_SHELL=$BASH_SHELL is $hts_bash_why])
fi
AC_MSG_WARN([no usable bash found: $BASH_SHELL is $hts_bash_why. "make check" and "make deb" need one; pass BASH_SHELL=/path/to/bash])
fi
fi
AC_PROG_CC
AM_PROG_CC_C_O
m4_warn([obsolete],
@@ -48,14 +122,13 @@ m4_warn([obsolete],
# script's behavior did not change. They are probably safe to remove.
AC_CHECK_INCLUDES_DEFAULT
AC_PROG_EGREP
# $(SED) substitutes $(datadir) into src/webhttrack
AC_PROG_SED
LT_INIT
AC_PROG_LN_S
LT_INIT
# bash, used to run the test scripts (see tests/Makefile.am TEST_LOG_COMPILER)
AC_PATH_PROGS([BASH], [bash], [/bin/bash])
# Export LD_LIBRARY_PATH name or equivalent.
AC_SUBST(SHLIBPATH_VAR,$shlibpath_var)
@@ -100,6 +173,9 @@ AX_CHECK_COMPILE_FLAG([-fstack-protector-strong], [DEFAULT_CFLAGS="$DEFAULT_CFLA
[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"], [], [-Werror])], [-Werror])
AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"], [], [-Werror])
AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"], [], [-Werror])
# backtrace() unwinds through these; armhf, unlike amd64/arm64, defaults them off
# and printed frameless crash reports.
AX_CHECK_COMPILE_FLAG([-fasynchronous-unwind-tables], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fasynchronous-unwind-tables"], [], [-Werror])
# No --discard-all: it drops the local symbols naming every static function, so
# a trace misattributes them to the nearest surviving global. Costs 0.6% size.
AX_CHECK_LINK_FLAG([-Wl,--no-undefined], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--no-undefined"])
@@ -131,6 +207,156 @@ AC_SUBST([LDFLAGS_PIE])
# Ties a crash trace from a stripped build back to its separate debug symbols.
AX_CHECK_LINK_FLAG([-Wl,--build-id], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--build-id"])
# tools/macos-app.sh rewrites load commands to @rpath, which needs header room ld64 does
# not leave by default; the probe fails on GNU ld, so this self-gates on Darwin.
AX_CHECK_LINK_FLAG([-Wl,-headerpad_max_install_names],
[DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-headerpad_max_install_names"])
### Relocatable install (#906, #978)
# libtool records the configured libdir as an absolute rpath, so a copied tree
# cannot find libhttrack. A binary-relative rpath fixes that where the loader
# expands the token, which the BSDs disagree on: probe by running one.
AC_ARG_ENABLE([origin-rpath],
AS_HELP_STRING([--disable-origin-rpath],
[link the binaries without a binary-relative rpath]),
[hts_origin_asked=yes], [enable_origin_rpath=yes; hts_origin_asked=no])
# hts_origin_ref is the raw token the loader expands; hts_origin_ref_make is the
# spelling that survives make's expansion and libtool's eval of the link line.
AS_CASE([$host_os],
[darwin*], [hts_origin_darwin=yes
hts_origin_ref='@loader_path'
hts_origin_ref_make='@loader_path'],
[hts_origin_darwin=no
hts_origin_ref='$ORIGIN'
hts_origin_ref_make='\$$ORIGIN'])
hts_save_prefix=$prefix
hts_save_exec_prefix=$exec_prefix
test "x$prefix" = xNONE && prefix=$ac_default_prefix
test "x$exec_prefix" = xNONE && exec_prefix=$prefix
eval hts_libdir=\"$libdir\"
eval hts_libdir=\"$hts_libdir\"
prefix=$hts_save_prefix
exec_prefix=$hts_save_exec_prefix
# Enumerated, not read from libtool's sys_lib_dlsearch_path_spec: that one is
# only augmented when /etc/ld.so.conf exists, so a sysroot or minimal container
# would let a distribution libdir through. Spelled out rather than globbed as
# /lib*, which would swallow /libfoo and /usr/libexec.
AS_CASE([$hts_libdir],
[/lib | /lib/* | /lib32 | /lib32/* | /lib64 | /lib64/* | /libx32 | /libx32/*], [hts_origin_sysdir=yes],
[/usr/lib | /usr/lib/* | /usr/lib32 | /usr/lib32/* | /usr/lib64 | /usr/lib64/* | /usr/libx32 | /usr/libx32/*], [hts_origin_sysdir=yes],
[hts_origin_sysdir=no])
# dyld follows LC_RPATH only for a load path spelled @rpath/..., so Darwin also
# needs libhttrack's own id. That comes from libtool's archive_cmds template and
# not from LDFLAGS, so rewrite it; an unrecognised template leaves the gate off.
hts_origin_id=no
if test "x$hts_origin_darwin" = xyes; then
hts_origin_subst='s|-install_name $rpath/|-install_name @rpath/|'
hts_origin_archive=`printf '%s\n' "$archive_cmds" | $SED "$hts_origin_subst"`
hts_origin_expsym=`printf '%s\n' "$archive_expsym_cmds" | $SED "$hts_origin_subst"`
# Both templates or neither: should they ever diverge, the -export-symbols
# path would keep stamping the absolute id the rpath cannot answer.
if test "x$hts_origin_archive" != "x$archive_cmds" &&
test "x$hts_origin_expsym" != "x$archive_expsym_cmds"; then
hts_origin_id=yes
fi
fi
ORIGIN_RPATH=no
RPATH_ORIGIN_LDFLAGS=""
if test "x$enable_origin_rpath" = xno; then
AC_MSG_NOTICE([binary-relative rpath disabled by request])
elif test "x$hts_origin_darwin" = xyes && test "x$hts_origin_id" = xno; then
AC_MSG_NOTICE([binary-relative rpath not used: unrecognised install_name in libtool archive_cmds])
elif test "x$enable_shared" = xno; then
AC_MSG_NOTICE([binary-relative rpath not needed: no shared libraries])
elif test "x$hts_origin_sysdir" = xyes; then
# A distribution libdir is already searched, and an rpath there is a lintian tag.
AC_MSG_NOTICE([binary-relative rpath not used: $hts_libdir is a system libdir])
elif test "x$cross_compiling" = xyes && test "x$hts_origin_asked" = xno; then
AC_MSG_NOTICE([binary-relative rpath not probed: cross compiling, pass --enable-origin-rpath to force it])
else
AC_MSG_CHECKING([whether the loader honours a binary-relative rpath])
AC_CACHE_VAL([hts_cv_origin_rpath], [
hts_cv_origin_rpath=no
# Cross compiling cannot run the probe, so take the stricter spelling first.
hts_origin_cands="none origin"
test "x$cross_compiling" = xyes && hts_origin_cands="origin none"
test "x$hts_origin_darwin" = xyes && hts_origin_cands=loader_path
rm -rf conftest.origin
AS_MKDIR_P([conftest.origin/bin])
AS_MKDIR_P([conftest.origin/lib])
AS_MKDIR_P([conftest.origin/run])
cat >conftest.origin/lib.c <<_ACEOF
int hts_origin_probe(void);
int hts_origin_probe(void) { return 42; }
_ACEOF
cat >conftest.origin/main.c <<_ACEOF
int hts_origin_probe(void);
int main(void) { return hts_origin_probe() == 42 ? 0 : 1; }
_ACEOF
if test "x$hts_origin_darwin" = xyes; then
# The same @rpath id, so a dead LC_RPATH cannot pass for a live one.
hts_origin_mklib="$CC $CFLAGS -dynamiclib -install_name @rpath/libhtsorigin.dylib -o conftest.origin/lib/libhtsorigin.dylib conftest.origin/lib.c"
else
hts_origin_mklib="$CC $CFLAGS -fPIC -shared -o conftest.origin/lib/libhtsorigin.so conftest.origin/lib.c"
fi
hts_origin_link="-Lconftest.origin/lib -lhtsorigin"
if $hts_origin_mklib >&AS_MESSAGE_LOG_FD 2>&1; then
for hts_origin_cand in $hts_origin_cands; do
# Solaris and the older BSDs expand the token only with -z origin.
hts_origin_try=
test "x$hts_origin_cand" = xorigin && hts_origin_try=-Wl,-z,origin
$CC $CFLAGS $LDFLAGS -o conftest.origin/bin/probe conftest.origin/main.c \
$hts_origin_link $hts_origin_try "-Wl,-rpath,$hts_origin_ref/../lib" \
>&AS_MESSAGE_LOG_FD 2>&1 || continue
# From an unrelated cwd, so a loader taking the token literally fails.
if test "x$cross_compiling" = xyes ||
(unset LD_LIBRARY_PATH DYLD_LIBRARY_PATH
cd conftest.origin/run && ../bin/probe) >&AS_MESSAGE_LOG_FD 2>&1; then
hts_cv_origin_rpath=$hts_origin_cand
break
fi
done
fi
rm -rf conftest.origin
])
AS_CASE([$hts_cv_origin_rpath],
[no], [AC_MSG_RESULT([no])],
[
# $(libdir) first, so the token only answers once the configured
# libdir is gone: $(bindir)/../lib is a different directory under a
# lib64 or split-bindir layout and must not shadow it.
RPATH_ORIGIN_LDFLAGS='-Wl,-rpath,$(libdir)'
test "x$hts_cv_origin_rpath" = xorigin && RPATH_ORIGIN_LDFLAGS="$RPATH_ORIGIN_LDFLAGS -Wl,-z,origin"
RPATH_ORIGIN_LDFLAGS="$RPATH_ORIGIN_LDFLAGS -Wl,-rpath,$hts_origin_ref_make/../lib"
if test "x$hts_origin_darwin" = xyes; then
archive_cmds=$hts_origin_archive
archive_expsym_cmds=$hts_origin_expsym
fi
ORIGIN_RPATH=yes
AC_MSG_RESULT([yes ($hts_origin_ref/../lib)])
])
fi
AC_SUBST([ORIGIN_RPATH])
AC_SUBST([RPATH_ORIGIN_LDFLAGS])
# The @rpath id is only safe for a consumer carrying its own LC_RPATH, so
# libhttrack.pc hands one out -- under the gates the executables' own rpath
# respects, so that --disable-origin-rpath and an archive-only build stay
# rpath-free too. ${libdir} is pkg-config's own variable, expanded on query.
PKGCONFIG_RPATH=no
PKGCONFIG_RPATH_LDFLAG=
AS_IF([test "x$enable_origin_rpath" != xno && test "x$enable_shared" != xno &&
test "x$hts_origin_sysdir" = xno],
[PKGCONFIG_RPATH=yes
PKGCONFIG_RPATH_LDFLAG=' -Wl,-rpath,${libdir}'])
AC_SUBST([PKGCONFIG_RPATH])
AC_SUBST([PKGCONFIG_RPATH_LDFLAG])
### Check for -fvisibility=hidden support
gl_VISIBILITY
AM_CFLAGS="$AM_CFLAGS $CFLAG_VISIBILITY"
@@ -173,7 +399,10 @@ AC_CHECK_TYPE(sa_family_t, [], [AC_DEFINE([sa_family_t], [uint16_t], [sa_family_
AX_CHECK_ALIGNED_ACCESS_REQUIRED
# check for various headers
AC_CHECK_HEADERS([execinfo.h sys/ioctl.h])
AC_CHECK_HEADERS([execinfo.h sys/ioctl.h sys/random.h])
## CSPRNG for the --single-file mark; /dev/urandom is the fallback
AC_CHECK_FUNCS([getrandom])
### zlib (mandatory)
CHECK_ZLIB()
@@ -217,7 +446,8 @@ AC_CHECK_LIB(crypto, EVP_get_digestbyname,
LIBS="$LIBS $OPENSSL_LIBS"
AC_CHECK_LIB(ssl, SSL_CTX_new,
[
OPENSSL_LIBS="$OPENSSL_LIBS -lssl"
# -lssl first: an archive link needs it ahead of the -lcrypto it uses.
OPENSSL_LIBS="-lssl $OPENSSL_LIBS"
AC_DEFINE(HTS_USEOPENSSL, 1, [Check for OpenSSL])
],
[
@@ -285,6 +515,34 @@ AC_DEFINE(DLLIB, 1,[Check for dlopen in c])], AC_MSG_WARN([*** not found * this
AC_SUBST(DL_LIBS)
fi
### What libhttrack.pc tells a consumer about its private dependencies
# A static link needs each library ahead of the ones it uses, which only a
# dependency's own .pc gets right (and which alone carries the -L an
# out-of-the-way prefix needs), so name the module wherever one is installed.
# Dependents first: pkg-config emits Requires.private after Libs.private.
AC_ARG_VAR([PKG_CONFIG], [path to the pkg-config utility])
AC_PATH_TOOL([PKG_CONFIG], [pkg-config], [false])
PKGCONFIG_REQUIRES_PRIVATE=
PKGCONFIG_LIBS_PRIVATE=
hts_pc_dep() { # hts_pc_dep <module or empty> <link line>
test -n "$2" || return 0
if test -n "$1" && $PKG_CONFIG --exists "$1" 2>/dev/null; then
PKGCONFIG_REQUIRES_PRIVATE="${PKGCONFIG_REQUIRES_PRIVATE:+$PKGCONFIG_REQUIRES_PRIVATE }$1"
else
PKGCONFIG_LIBS_PRIVATE="${PKGCONFIG_LIBS_PRIVATE:+$PKGCONFIG_LIBS_PRIVATE }$2"
fi
}
# Whichever half of OpenSSL was actually found; https=auto can settle for libcrypto.
AS_CASE([" $OPENSSL_LIBS "],
[*" -lssl "*], [hts_pc_openssl=openssl],
[*" -lcrypto "*], [hts_pc_openssl=libcrypto],
[hts_pc_openssl=])
hts_pc_dep "$hts_pc_openssl" "$OPENSSL_LIBS"
hts_pc_dep libbrotlidec "$BROTLI_LIBS"
hts_pc_dep libzstd "$ZSTD_LIBS"
AC_SUBST([PKGCONFIG_REQUIRES_PRIVATE])
AC_SUBST([PKGCONFIG_LIBS_PRIVATE])
### Check for setuid
AC_CHECK_LIB(c, setuid, [
AC_DEFINE(SETUID, 1,[Check for setuid])], AC_MSG_RESULT([not found]))
@@ -299,7 +557,7 @@ AC_DEFINE(HAVE_STRNLEN, 1,[Check for strnlen])], AC_MSG_RESULT([not found]))
## Online unit tests
AC_MSG_CHECKING(whether to enable online unit tests)
AC_ARG_ENABLE([online-unit-tests],
[AS_HELP_STRING([--enable-online-unit-tests=@<:@yes/no/auto@:>@],[Enable online-unit-tests @<:@default=yes@:>@])],
[AS_HELP_STRING([--enable-online-unit-tests=@<:@yes/no/auto@:>@],[Enable online-unit-tests @<:@default=no@:>@])],
[
case "${enableval}" in
no|yes|auto)
@@ -341,6 +599,10 @@ if test x"$fuzzers" = x"yes"; then
AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer-no-link],
[DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fsanitize=fuzzer-no-link"],
[AC_MSG_ERROR([--enable-fuzzers requires libFuzzer support (clang)])])
# Ordinary programs are instrumented too, so they need that runtime at link.
AX_CHECK_LINK_FLAG([-fsanitize=fuzzer-no-link],
[DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -fsanitize=fuzzer-no-link"],
[AC_MSG_ERROR([--enable-fuzzers requires libFuzzer support (clang)])])
# clang's static sanitizer runtimes clash with -Wl,--no-undefined on the .so.
if test x"$enable_shared" != x"no"; then
AC_MSG_ERROR([--enable-fuzzers requires --disable-shared])
@@ -348,10 +610,25 @@ if test x"$fuzzers" = x"yes"; then
fi
AM_CONDITIONAL([FUZZERS], [test x"$fuzzers" = x"yes"])
# tests/Makefile.am takes its TESTS from here instead of a checked-in list,
# which made every pair of test-adding PRs conflict on the forge (#844).
AC_MSG_CHECKING([for test scripts])
TESTS_LIST=
hts_ntests=0
for hts_t in "$srcdir"/tests/[[0-9]]*_*.test; do
test -f "$hts_t" || continue
TESTS_LIST="$TESTS_LIST `echo "$hts_t" | sed 's|.*/||'`"
hts_ntests=`expr $hts_ntests + 1`
done
AC_SUBST([TESTS_LIST])
AC_MSG_RESULT([$hts_ntests])
test "$hts_ntests" -gt 0 || AC_MSG_ERROR([no test script matched tests/[[0-9]]*_*.test])
# Final output
AC_CONFIG_FILES([
Makefile
src/Makefile
src/libhttrack.pc
man/Makefile
m4/Makefile
templates/Makefile
@@ -360,5 +637,6 @@ html/Makefile
libtest/Makefile
tests/Makefile
fuzz/Makefile
tools/Info.plist
])
AC_OUTPUT

59
debian/changelog vendored
View File

@@ -1,3 +1,62 @@
httrack (3.49.20-1) unstable; urgency=medium
* New upstream release: an off-site link that looked like a file could pull
in the whole foreign site, and a mirror asked to stop while receiving over
FTP hung; full list in history.txt. The documentation was also refreshed.
* debian/copyright: record the League Spartan webfont the documentation now
bundles (OFL-1.1).
-- Xavier Roche <xavier@debian.org> Mon, 10 Aug 2026 21:08:40 +0200
httrack (3.49.19-1) unstable; urgency=medium
* New upstream release: mostly FTP fixes, plus stack-overflow, mirror
interruption and --update data-loss fixes; full list in history.txt. Two
of them matter to the build: the test suite no longer assumes GNU
coreutils, and the AppStream metainfo no longer declares a non-stock icon
name that appstream-util rejects.
-- Xavier Roche <xavier@debian.org> Sat, 08 Aug 2026 21:29:56 +0200
httrack (3.49.18-1) unstable; urgency=medium
* New upstream release, fixing the 3.49.17-1 build failures on armhf,
powerpc, hppa and loong64: the test suite's mmap interposer did not
compile on the first three, and the alternate-stack test insisted on a
longer backtrace than loong64's unwinder produces. Nothing in the engine
changed. Upstream now cross-builds the test shims for the ports
architectures on every push, so the next such break surfaces before the
buildds do.
-- Xavier Roche <xavier@debian.org> Wed, 05 Aug 2026 23:00:24 +0200
httrack (3.49.17-1) unstable; urgency=medium
* New upstream release: security fixes in the FTP and HTTP request paths and
in both WebHTTrack interfaces, where a crawled page could reach the control
panel, plus crash-report and stack-diagnostic fixes; full list in
history.txt. Also fixes the 3.49.16-1 build failures on armhf, loong64 and
hppa, both in the crash-report test's unwinding assumptions and in the
per-test time budget.
* libhttrack-dev ships $(libdir)/pkgconfig/libhttrack.pc, and so now depends
on libhttrack3 and on the -dev packages whose modules that file names:
libssl-dev, libbrotli-dev and libzstd-dev.
* Build with --disable-online-unit-tests. Policy 4.9 forbids network access
during a build, and the previous "auto" probe found one on hurd-i386, whose
crawl tests then failed on DNS.
-- Xavier Roche <xavier@debian.org> Wed, 05 Aug 2026 12:28:23 +0200
httrack (3.49.16-1) unstable; urgency=medium
* New upstream release: a stack overflow in the cache header writer, which a
server's response headers could trigger in httrack and libhttrack3 as well
as in proxytrack, and a ProxyTrack DAV buffer overflow; chunked trailer
handling, relocatable installs, crash-report backtraces on armhf, a hicolor
icon theme and translation fixes; full list in history.txt.
-- Xavier Roche <xavier@debian.org> Tue, 04 Aug 2026 18:32:40 +0200
httrack (3.49.15-1) unstable; urgency=medium
* New upstream release: single-file mirrors with inlined assets, sitemap

5
debian/control vendored
View File

@@ -72,7 +72,10 @@ Package: libhttrack-dev
Architecture: any
Multi-Arch: same
Section: libdevel
Depends: ${misc:Depends}, ${shlibs:Depends}, zlib1g-dev
# libhttrack3: what the .so symlink and libhttrack.pc's -lhttrack resolve onto. The
# other -dev packages: libhttrack.pc names their modules in Requires.private, and
# pkg-config fails outright when one of those .pc files is missing.
Depends: ${misc:Depends}, ${shlibs:Depends}, libhttrack3 (= ${binary:Version}), zlib1g-dev, libssl-dev, libbrotli-dev, libzstd-dev
Description: Httrack website copier includes and development files
This package adds supplemental files for using the httrack website copier
library

95
debian/copyright vendored
View File

@@ -44,6 +44,13 @@ License: FSFAP
permitted in any medium without royalty provided the copyright notice and
this notice are preserved. This file is offered as-is, without any warranty.
Files: html/fonts/*
Copyright: 2020 The League Spartan Project Authors
License: OFL-1.1
Comment:
The woff2 is the latin subset at weight 700, as served by Google Fonts.
Upstream declares no Reserved Font Name, so the subset keeps the family name.
Files: debian/*
Copyright: 2002-2026 Xavier Roche <xavier@debian.org>
License: GPL-3+
@@ -106,3 +113,91 @@ License: BSD-3-clause
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
License: OFL-1.1
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
.
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
.
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
.
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
.
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.

View File

@@ -1,4 +1,5 @@
usr/include/httrack
usr/lib/*/libhttrack.{so}
usr/lib/*/pkgconfig/libhttrack.pc
usr/share/httrack/libtest
usr/lib/*/httrack/libtest

4
debian/rules vendored
View File

@@ -34,7 +34,9 @@ configure-stamp:
dh_autoreconf
# note: dpkg-buildflags to be removed when compat=9
dh_auto_configure -- --enable-online-unit-tests=auto $(shell dpkg-buildflags --export=configure)
# Policy 4.9 forbids network access during a build, and "auto" made it a lottery:
# hurd-i386 FTBFS'd on a crawl whose DNS went away after the probe had said yes.
dh_auto_configure -- --disable-online-unit-tests $(shell dpkg-buildflags --export=configure)
touch configure-stamp

14
fuzz/.gitignore vendored Normal file
View File

@@ -0,0 +1,14 @@
# Anchored: a glob like /fuzz-* would also swallow the tracked fuzz-*.c sources.
/fuzz-charset
/fuzz-meta
/fuzz-idna
/fuzz-entities
/fuzz-unescape
/fuzz-filters
/fuzz-url
/fuzz-header
/fuzz-cachendx
/fuzz-htsparse
/fuzz-singlefile
/fuzz-sitemap
/fuzz-arc

View File

@@ -2,7 +2,7 @@
if FUZZERS
noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \
fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \
fuzz-htsparse fuzz-singlefile fuzz-sitemap
fuzz-htsparse fuzz-singlefile fuzz-sitemap fuzz-arc
endif
AM_CPPFLAGS = \
@@ -30,6 +30,11 @@ fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h
fuzz_singlefile_SOURCES = fuzz-singlefile.c fuzz.h
fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h
# proxytrack does not link libhttrack, so its store compiles into the harness;
# coucal, minizip and md5 still come from the static libhttrack above.
fuzz_arc_SOURCES = fuzz-arc.c fuzz.h $(top_srcdir)/src/proxy/store.c
fuzz_arc_CPPFLAGS = $(AM_CPPFLAGS) -DZLIB_CONST
# List corpus files explicitly: automake does not expand EXTRA_DIST globs.
EXTRA_DIST = README.md run-fuzzers.sh \
corpus/charset/utf8.txt corpus/charset/latin1.txt corpus/charset/sjis.txt \
@@ -53,6 +58,10 @@ EXTRA_DIST = README.md run-fuzzers.sh \
corpus/singlefile/img-src.html corpus/singlefile/link-rel.html \
corpus/singlefile/style-block.html corpus/singlefile/style-attr.html \
corpus/singlefile/srcset.html corpus/singlefile/rawtext.html \
corpus/singlefile/malformed.html corpus/singlefile/many-attrs.html \
corpus/singlefile/malformed.html \
corpus/singlefile/mark-at-eof.html corpus/singlefile/mark-only.html \
corpus/singlefile/mark-degenerate.html corpus/singlefile/traversal.html \
corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz
corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz \
corpus/arc/roundtrip.arc corpus/arc/truncated.arc \
corpus/arc/regress-null-body.arc

View File

@@ -13,3 +13,5 @@ bash /path/to/httrack/fuzz/run-fuzzers.sh fuzz 60 # 60s per target
```
Run one target by hand: `fuzz/fuzz-url -max_total_time=300 corpusdir fuzz/corpus/url`. Seed corpora live in `corpus/<target>/`; a crash reproducer is replayed with `fuzz/fuzz-url crash-file`.
`fuzz-arc` is the odd one out: it drives proxytrack's `.arc` reader the way `--convert` does, through a temp file rather than a buffer, and it compiles `src/proxy/store.c` into the harness because proxytrack does not link libhttrack. Both readers and the writer print to stderr on malformed input, so pass `-close_fd_mask=2` for anything longer than a corpus replay.

View File

@@ -0,0 +1,9 @@
filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9
2 0 test
http://example.com/p.html 0.0.0.0 20250101000000 text/html -1 - - 0 t.arc 77
HTTP/1.1 -1 Broken
Content-Type: text/html
Content-Length: 10

View File

@@ -0,0 +1,11 @@
filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9
2 0 test
http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc 120
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT
Content-Length: 10
BODYMARKER

View File

@@ -0,0 +1,9 @@
filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9
2 0 test
http://example.com/q.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc 2000000000
HTTP/1.1 200 OK
Content-Type: text/html
HI

View File

@@ -1,3 +1,4 @@
<img src="a.png">
<img src=big.png alt=over-cap>
<img src="../escape.png"><img src="/abs.png"><img src="data:,x">
<img src="a.png">
<img src=big.png alt=over-cap>
<img src="../escape.png"><img src="/abs.png">
<img src="data:,x">

View File

@@ -1,4 +1,3 @@
<link rel="stylesheet" href="s.css">
<link rel=icon href=a.png>
<link rel="next" href="p2.html">
<link rel="preload" href="j.js">
<link rel="stylesheet" href="s.css">
<link rel="icon" href="a.png">
<link rel="canonical" href="other.html">

View File

@@ -1,5 +1,3 @@
<img src="unterminated.png
<div style="background:url(a.png">
<style>@import url(
<!-- unterminated comment
<a href=
<p a=1 b=2 c=3 d=4 e=5 f=6 g=7 h=8 i=9 j=10 title="> <img src=a.png>">x</p>
<!--><img src="a.png">
<img src="a.png" >

View File

@@ -1 +0,0 @@
<img src="a.png" a0="v" a1="v" a2="v" a3="v" a4="v" a5="v" a6="v" a7="v" a8="v" a9="v" a10="v" a11="v" a12="v" a13="v" a14="v" a15="v" a16="v" a17="v" a18="v" a19="v" a20="v" a21="v" a22="v" a23="v" a24="v" a25="v" a26="v" a27="v" a28="v" a29="v" a30="v" a31="v" a32="v" a33="v" a34="v" a35="v" a36="v" a37="v" a38="v" a39="v" a40="v" a41="v" a42="v" a43="v" a44="v" a45="v" a46="v" a47="v" a48="v" a49="v" a50="v" a51="v" a52="v" a53="v" a54="v" a55="v" a56="v" a57="v" a58="v" a59="v" a60="v" a61="v" a62="v" a63="v" a64="v" a65="v" a66="v" a67="v" a68="v" a69="v">

View File

@@ -0,0 +1 @@
a.png

View File

@@ -0,0 +1,5 @@

==
""
a.png#!htsinlin
#!htsinlin

View File

@@ -0,0 +1 @@


View File

@@ -1,4 +1,3 @@
<script>var s="</scripting>"; if(a</b) x=1;</script>
<script src="j.js"></script>
<textarea></textareas></textarea>
<title></titles></title>
<script src="j.js"></script>
<script>var u="a.png";</script>
<textarea>a.png</textarea>

View File

@@ -1,2 +1 @@
<img srcset="a.png 1x, big.png 2x, a.png 100w">
<source srcset="a.png,, a.png 2x," src="a.png">
<img srcset="a.png 1x, big.png 2x, a.png 480w">

View File

@@ -1,2 +1,2 @@
<div style="background:url(a.png);list-style:url('a.png')"></div>
<p style='background:url("a.png")'>x</p>
<div style="background:url(a.png)"></div>
<div style='background:url("a.png")'></div>

View File

@@ -1,5 +1,4 @@
<style>@import "s.css";
@import url(sub/b.css);
div{background:url(a.png)}
/* url(a.png) */ p:after{content:"url(a.png)"}
<style>@import "s.css";
@import url(sub/b.css);
div{background:url(a.png)}
</style>

View File

@@ -0,0 +1,4 @@
<img src="../../../../etc/passwd">
<img src="a.png#frag?q=1">
<img src="sub/../a.png">
<img src="./a.png">

106
fuzz/fuzz-arc.c Normal file
View File

@@ -0,0 +1,106 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* Fuzz proxytrack's .arc reader the way `--convert` drives it: the record loop
seeks on lengths read from the file, and every entry reaches a writer. */
#include "fuzz.h"
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "coucal.h"
#include "proxy/store.h"
/* An .arc past this size says nothing a smaller one cannot. */
#define FUZZ_ARC_MAXSIZE (1024 * 1024)
static char arc_dir[256];
static char arc_in[sizeof(arc_dir) + sizeof("/in.arc")];
static char arc_out[sizeof(arc_dir) + sizeof("/out.arc")];
static void fuzz_arc_cleanup(void) {
(void) unlink(arc_in);
(void) unlink(arc_out);
(void) rmdir(arc_dir);
}
/* proxytrack's main() installs one; without it coucal logs stats per free */
static void fuzz_arc_coucal_log(coucal_opaque arg, coucal_loglevel level,
const char *format, va_list args) {
(void) arg;
(void) level;
(void) format;
(void) args;
}
/* PT_GetType() picks the format from the extension, so the names end in .arc */
static int fuzz_arc_setup(void) {
if (arc_in[0] == '\0') {
const char *const tmp = getenv("TMPDIR");
coucal_set_global_assert_handler(fuzz_arc_coucal_log, NULL);
snprintf(arc_dir, sizeof(arc_dir), "%s/fuzz-arc-XXXXXX",
tmp != NULL && *tmp != '\0' ? tmp : "/tmp");
if (mkdtemp(arc_dir) == NULL) {
return -1;
}
snprintf(arc_out, sizeof(arc_out), "%s/out.arc", arc_dir);
snprintf(arc_in, sizeof(arc_in), "%s/in.arc", arc_dir);
atexit(fuzz_arc_cleanup);
}
return 0;
}
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
PT_Indexes indexes;
FILE *fp;
hts_boolean written;
if (size > FUZZ_ARC_MAXSIZE || fuzz_arc_setup() != 0) {
return 0;
}
if ((fp = fopen(arc_in, "wb")) == NULL) {
return 0;
}
written = fwrite(data, 1, size, fp) == size ? HTS_TRUE : HTS_FALSE;
if (fclose(fp) != 0 || !written) {
return 0;
}
indexes = PT_New();
if (indexes != NULL) {
if (PT_AddIndex(indexes, arc_in) > 0) {
/* the writer reads back every entry the loader indexed */
(void) PT_SaveCache(indexes, arc_out);
}
PT_Delete(indexes);
}
return 0;
}

View File

@@ -60,6 +60,10 @@ static void sf_cleanup(void) {
(void) remove(sf_root);
}
/* One httrackp for the whole run: the mark secret lives on it, so a fresh one
per input would make every mark in the corpus unrecognisable. */
static httrackp *sf_opt = NULL;
/* A missing asset would silently reduce the target to its parser half. */
static void sf_write(const char *name, const char *data, size_t len) {
char path[700];
@@ -76,6 +80,48 @@ static void sf_text(const char *name, const char *data) {
sf_write(name, data, strlen(data));
}
/* Append ref plus a real mark for it. The secret is drawn per httrackp, so a
fixture cannot spell one; every mark the target sees is built here. */
static void sf_marked(String *out, const char *ref, char cls) {
char mark[SINGLEFILE_MARK_MAX];
StringCat(*out, ref);
StringCat(*out,
singlefile_mark(sf_opt, mark, sizeof(mark), cls, strlen(ref)));
}
/* \001<ref>\002 in an input becomes <ref> plus its mark, which is the only
way a corpus file can reach the mark parser at all. */
static void sf_expand_input(const char *in, size_t len, String *out) {
size_t i, start = 0;
StringClear(*out);
for (i = 0; i < len; i++) {
if (in[i] == '\001') {
start = StringLength(*out);
} else if (in[i] == '\002') {
char mark[SINGLEFILE_MARK_MAX];
StringCat(*out, singlefile_mark(sf_opt, mark, sizeof(mark),
SINGLEFILE_CLASS_ANY,
StringLength(*out) - start));
} else {
StringAddchar(*out, in[i]);
}
}
}
static void sf_css(const char *name, const char *pre, const char *ref,
const char *post) {
String body = STRING_EMPTY;
StringCopy(body, pre);
sf_marked(&body, ref, SINGLEFILE_CLASS_ANY);
StringCat(body, post);
sf_write(name, StringBuff(body), StringLength(body));
StringFree(body);
}
static void sf_init(void) {
static const char png[] = "\x89PNG\r\n\x1a\n";
static const char big[4096] = "\x89PNG";
@@ -83,6 +129,9 @@ static void sf_init(void) {
char path[700];
hts_init();
sf_opt = hts_create_opt();
sf_opt->log = sf_opt->errlog = NULL;
sf_opt->single_file_max_size = FUZZ_SF_CAP;
snprintf(sf_root, sizeof(sf_root), "%s/httrack-fuzz-sf-XXXXXX",
tmp != NULL && tmp[0] != '\0' ? tmp : "/tmp");
if (mkdtemp(sf_root) == NULL)
@@ -95,37 +144,50 @@ static void sf_init(void) {
sf_write("a.png", png, sizeof(png) - 1);
sf_write("big.png", big, sizeof(big));
sf_text("j.js", "var x=1;\n");
/* @import plus a url(), so an inlined stylesheet recurses and its own
relative reference is rebased. */
sf_text("s.css", "@import url(sub/b.css);\ndiv{background:url(a.png)}\n");
sf_text("sub/b.css", "p{background:url(../a.png)}\n");
/* Marked, so an inlined stylesheet recurses into its own marks and its
un-inlinable reference is rebased; unmarked assets leave the target as a
bare scan that reaches nothing. */
{
String css = STRING_EMPTY;
StringCopy(css, "@import url(");
sf_marked(&css, "sub/b.css", SINGLEFILE_CLASS_CSS);
StringCat(css, ");\ndiv{background:url(");
sf_marked(&css, "a.png", SINGLEFILE_CLASS_ANY);
StringCat(css, ")}\np{background:url(");
sf_marked(&css, "big.png", SINGLEFILE_CLASS_ANY);
StringCat(css, ")}\n");
sf_write("s.css", StringBuff(css), StringLength(css));
StringFree(css);
}
sf_css("sub/b.css", "p{background:url(", "../a.png", ")}\n");
}
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
static int inited = 0;
String out = STRING_EMPTY;
httrackp *opt;
/* Exact-length, unterminated: the rewriter is span-based, so ASan bounds a
read past html_len instead of it landing on a terminator. */
char *html = malloct(size != 0 ? size : 1);
String in = STRING_EMPTY;
char *html;
size_t html_len;
if (!inited) {
sf_init();
inited = 1;
}
memcpy(html, data, size);
opt = hts_create_opt();
opt->log = opt->errlog = NULL;
opt->single_file_max_size = FUZZ_SF_CAP;
sf_expand_input((const char *) data, size, &in);
html_len = StringLength(in);
/* Exact-length, unterminated: the rewriter is span-based, so ASan bounds a
read past html_len instead of it landing on a terminator. */
html = malloct(html_len != 0 ? html_len : 1);
memcpy(html, StringBuff(in), html_len);
StringFree(in);
StringClear(out);
(void) singlefile_rewrite_html(opt, sf_root, sf_page, html, size,
(void) singlefile_rewrite_html(sf_opt, sf_root, sf_page, html, html_len,
SINGLEFILE_MAX_PAGE_SIZE, &out);
StringFree(out);
freet(html);
hts_free_opt(opt);
return 0;
}

View File

@@ -38,7 +38,7 @@ Please visit our Website: http://www.httrack.com
#include "htsbase.h"
/* Heap NUL-terminated copy of the fuzzer input, so ASan bounds every read. */
static char *fuzz_strdup(const uint8_t *data, size_t size) {
HTS_UNUSED static char *fuzz_strdup(const uint8_t *data, size_t size) {
char *s = malloct(size + 1);
memcpy(s, data, size);

View File

@@ -4,6 +4,87 @@ HTTrack Website Copier release history:
This file lists all changes and fixes that have been made for HTTrack
3.49-20
+ Fixed: a link to another host whose name looked like a file was scanned as a page, so one link could mirror the whole foreign site on the referring page's depth budget (#121)
+ Fixed: a size-bounded filter built both of its filter URLs wrong when the path had no leading slash (#1095)
+ Fixed: a mirror told to quit while an FTP file was being received announced the exit and then waited forever on that slot (#1106)
+ Fixed: stopping an FTP transfer closed its data socket twice, so a threaded crawl could lose another worker's connection (#1090)
+ Fixed: WebHTTrack's welcome page opened its language menu blank instead of marking the language in use (#1084)
+ Fixed: two suite tests failed on the hppa, powerpc and ppc64 Debian buildds, holding the package back on those ports (#1108)
+ Fixed: a growable string could wrap its capacity to zero instead of saturating (#1062)
+ Changed: the macOS disk image carries its architecture in its name, so an Intel user can tell before downloading that the build is for Apple Silicon (#1083)
+ Changed: refreshed the GUI guide, its screenshots, the documentation typography and the footer contrast
+ Changed: settled on one spelling of the product name across the docs and the site
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-19
+ Fixed: an over-long user name in an ftp:// URL was truncated at the credential boundary, so the login went to a different account (#1032)
+ Fixed: an over-long FTP path or host name aborted the process instead of failing that one link (#1019)
+ Fixed: an FTP crawl ignored --timeout and --max-time, so a server that went quiet held a slot for five minutes (#1039)
+ Fixed: an FTP crawl of a dead host took two minutes to give up, and no signal could shorten it (#1059)
+ Fixed: the engine released the transfer slots while an FTP thread was still writing to one, crashing a mirror that hit its size or time limit (#1051)
+ Fixed: Ctrl-C did not shorten a connection or a name lookup already in flight, so ending a crawl against an unreachable host took minutes (#1073)
+ Fixed: a crawl that gave up mid-loop, on the link limit or an abort, lost its WARC archive and its change report, and a WARC written past teardown was stranded in a .tmp file (#1061, #1060)
+ Fixed: a crawl that rolled back for want of data still replaced the previous WARC archive, stranding its temporary and never writing its index; a stale or unwritable index is now reported rather than left silently in place (#1053)
+ Fixed: a resume the server answered with an unusable Content-Range deleted the partial file and never fetched it again; the restart no longer spends the retry budget, and is latched to one per link (#581, #1052)
+ Fixed: a long argument to -n or -O smashed the stack through an unbounded panic message, and the growable arrays could wrap their capacity to zero (#1040)
+ Fixed: --single-file could write a reference mark longer than it reads back, leaving it in the delivered page as visible text (#1054, #1069)
+ Fixed: on a terminal with no input left, a crawl toggled between the spinner and the full panel on every pass (#1072)
+ Fixed: -%v1 ended each line with a carriage return, so the lines overwrote each other and a redirected log was unreadable; -%v2 now paints its panel only on a terminal
+ Fixed: the installed socket headers did not compile under -std=c99 without _POSIX_C_SOURCE (#1001)
+ Fixed: the 16 px application icon rendered every stem as grey (#938)
+ Fixed: the desktop files installed outside the data directory, and the AppStream metainfo declared an icon name that is not a stock one
+ Fixed: the build failed on distributions whose coreutils are not GNU's (#1042)
+ Changed: --single-file drops its own HTML and CSS scanner and reuses the crawler's parser, so it follows the same references the mirror does (#749)
+ Changed: updated Turkish translation, and repaired two mangled Polish strings (#306)
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-18
+ Fixed: the 3.49.17 package build failed on four architectures: the test suite's mmap interposer did not compile on armhf, powerpc or hppa, and the alternate-stack test rejected loong64's shorter backtrace (#1023)
+ Fixed: the test suite failed on a host with no ps command, such as a Fedora build root; its hang diagnostics now read /proc directly (#1021)
+ Changed: multiple internal test and CI improvements, including a cross-compile matrix covering the Debian ports architectures
3.49-17
+ New: a pkg-config file, libhttrack.pc, ships with the development headers (#1018)
+ Fixed: a crawled page could read the WebHTTrack session id and drive the control panel; the id is now unguessable and cross-origin commands are refused (#877)
+ Fixed: an ftp:// link carrying encoded line breaks could append commands to the FTP control channel, and an unquoted LIST argument let a path holding a space list something else (#1010)
+ Fixed: a crawled link could inject header lines into the requests HTTrack sends (#1008)
+ Fixed: a control byte in a link made the crawler fetch a URL the page never wrote, and stripping one left the original tail glued to the result (#982, #974)
+ Fixed: the WebHTTrack progress panel rendered crawled URLs into HTML and JavaScript unescaped (#973)
+ Fixed: the WebHTTrack command box rendered its settings unescaped, so a posted value could break out of the box it was shown in (#998)
+ Fixed: the WebHTTrack settings pages interpolated values into attributes unescaped, lost a value containing a quote, and mangled a Windows path such as C:\ab (#989)
+ Fixed: a worker thread exhausting its stack died with no diagnostic (#969)
+ Fixed: a crash report named no frame of the executable on a non-PIE build, and named the dynamic loader rather than the program when launched through it (#995, #996)
+ Fixed: the crash report could hang, its symbolizer being forked from the signal handler (#968)
+ Fixed: the installed headers declared symbols the library does not export, and scoped struct addrinfo inside a parameter list (#977, #987)
+ Fixed: an install moved away from its configured prefix could not find libhttrack on macOS (#978)
+ Fixed: 3.49.16 did not build on armhf, loong64, hppa or hurd-i386 (#1015)
+ Changed: mirrored pages are served sandboxed, so a saved page can no longer script the WebHTTrack panel, and loses cookies, storage and cross-frame access (#877)
+ Changed: htsserver exits with a diagnostic rather than fall back to a guessable session id when no entropy source answers (#877)
+ Changed: an FTP file whose name contains a tab is no longer fetched (#1010)
+ Changed: on macOS, libhttrack records @rpath/libhttrack.3.dylib, so anything linked against it afterwards supplies its own rpath (#978)
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-16
+ New: macOS ships a signed and notarized HTTrack.app with its own icon, bundling the OpenSSL it needs so a downloaded copy launches (#890, #900, #901, #950)
+ New: the desktop icons install into the hicolor theme, scalable SVG included, so Icon=httrack resolves in a launcher (#932, #933)
+ Fixed: an unauthenticated PROPFIND overflowed the ProxyTrack DAV item buffer (#836)
+ Fixed: response headers from the server overflowed the stack buffer holding the cache header block, in both httrack and ProxyTrack (#841)
+ Fixed: a chunked response carrying trailers was rejected as "Invalid chunk" (#855)
+ Fixed: an over-long URL aborted the whole mirror inside the cache instead of reading as a miss (#935, #936)
+ Fixed: ProxyTrack could not re-read the .arc it writes, and crashed on a record whose body it could not read (#834, #929, #931)
+ Fixed: ProxyTrack logged a hashtable stats line and every PROPFIND body to stderr (#911, #918)
+ Fixed: the frozen-slot spool was written inside the mirror namespace, so it landed in the mirrored tree (#859)
+ Fixed: a stack overflow killed httrack with no diagnostic, crash reports named no frame of the executable, and on armhf they had no frames at all (#866, #889, #892)
+ Fixed: an install moved away from its configured prefix could not find its data directory, its shared library, or the html symlink (#885, #887, #894, #906)
+ Fixed: tooltips in the web GUI broke on a translation containing an apostrophe; the escaping now also covers quotes, backslashes, markup and DBCS lead bytes (#864)
+ Fixed: nine strings of the Windows GUI's option dialogs were untranslated in 26 of the 30 language files, and six language files disagreed with their own declared charset (#863, #963)
+ Fixed: the AppStream metainfo still advertised WebHTTrack 3.49.8 (#884)
+ Fixed: the installed development headers did not compile standalone (#943)
+ Fixed: configure discarded a user-supplied BASH_SHELL, resolved bash to /bin/sh on macOS, and hung on one pointing at a FIFO (#891, #895, #908, #922)
+ Changed: the masthead wordmark and the rings background are SVG, so they stay sharp on a hi-DPI screen (#910, #916)
+ Changed: multiple internal hardening, build, test and CI improvements
3.49-15
+ New: --single-file rewrites each saved page with its assets inlined as data: URIs (#713)
+ New: --changes reports what a crawl added, updated or removed against the previous mirror (#714)
@@ -128,7 +209,7 @@ This file lists all changes and fixes that have been made for HTTrack
+ Fixed: report why a -%L URL list could not be loaded (#49)
+ Changed: multiple internal hardening, build and CI improvements
.49-9
3.49-9
+ Fixed: file-type detection from the Content-Type header: trust a declared type over a binary URL extension, honor --assume under the delayed type check, and keep a known extension against a bogus or empty Content-Type (#267, #29, #56)
+ Fixed: an uninitialized-buffer read when the Content-Type is empty (#411)
+ Fixed: restored C++ source-compatibility of the installed headers so reverse dependencies (httraqt) build again (#413)

View File

@@ -4,6 +4,7 @@ HelpHtmldir = $(htmldir)
HelpHtmlimgdir = $(HelpHtmldir)/img
HelpHtmldivdir = $(HelpHtmldir)/div
HelpHtmlimagesdir = $(HelpHtmldir)/images
HelpHtmlfontsdir = $(HelpHtmldir)/fonts
HelpHtmlTxtdir = $(HelpHtmldir)
WebHtmldir = $(HelpHtmldir)/server
WebHtmlimagesdir = $(HelpHtmldir)/server/images
@@ -11,7 +12,11 @@ WebPixmapdir = $(datadir)/pixmaps
WebIcon16x16dir = $(datadir)/icons/hicolor/16x16/apps
WebIcon32x32dir = $(datadir)/icons/hicolor/32x32/apps
WebIcon48x48dir = $(datadir)/icons/hicolor/48x48/apps
VFolderEntrydir = $(prefix)/share/applications
WebIcon64x64dir = $(datadir)/icons/hicolor/64x64/apps
WebIcon128x128dir = $(datadir)/icons/hicolor/128x128/apps
WebIcon256x256dir = $(datadir)/icons/hicolor/256x256/apps
WebIconScalabledir = $(datadir)/icons/hicolor/scalable/apps
VFolderEntrydir = $(datadir)/applications
MetaInfodir = $(datadir)/metainfo
# Wildcards are globbed against $(srcdir): a bare "*.html" is resolved against
@@ -23,27 +28,28 @@ HelpHtml_DATA = $(srcdir)/*.html $(srcdir)/*.js $(srcdir)/*.css
HelpHtmldiv_DATA = div/search.sh
HelpHtmlimg_DATA = $(srcdir)/img/*
HelpHtmlimages_DATA = $(srcdir)/images/*
# doc.css reaches the woff2 by a relative URL, so it installs beside the pages.
HelpHtmlfonts_DATA = $(srcdir)/fonts/*
HelpHtmlTxt_DATA = ../greetings.txt ../history.txt ../license.txt
WebHtml_DATA = $(srcdir)/server/*.html $(srcdir)/server/*.js $(srcdir)/server/*.css
WebHtmlimages_DATA = $(srcdir)/server/images/*
# note: converted & normalized by
# ico2xpm favicon.ico -o httrack.xpm
# mogrify -format xpm -map /usr/share/doc/menu/examples/cmap.xpm httrack.xpm
# Generated from the brand master by gen/export.py. Asserted by tests/160.
# httrack.xpm duplicates the 32x32 one under the bare basename the .desktop
# Icon= key falls back to here when no icon theme is installed (#932).
WebPixmap_DATA = $(srcdir)/server/div/*.xpm
WebIcon16x16_DATA = $(srcdir)/server/div/16x16/*.png
WebIcon32x32_DATA = $(srcdir)/server/div/32x32/*.png
WebIcon48x48_DATA = $(srcdir)/server/div/48x48/*.png
WebIcon64x64_DATA = $(srcdir)/server/div/64x64/*.png
WebIcon128x128_DATA = $(srcdir)/server/div/128x128/*.png
WebIcon256x256_DATA = $(srcdir)/server/div/256x256/*.png
WebIconScalable_DATA = $(srcdir)/server/div/scalable/*.svg
VFolderEntry_DATA = $(srcdir)/server/div/*.desktop
MetaInfo_DATA = $(srcdir)/server/div/*.metainfo.xml
EXTRA_DIST = $(HelpHtml_DATA) $(HelpHtmlimg_DATA) $(HelpHtmlimages_DATA) \
$(HelpHtmlfonts_DATA) \
$(HelpHtmldiv_DATA) $(WebHtml_DATA) $(WebHtmlimages_DATA) \
$(WebPixmap_DATA) $(WebIcon16x16_DATA) $(WebIcon32x32_DATA) $(WebIcon48x48_DATA) \
$(VFolderEntry_DATA) $(MetaInfo_DATA)
install-data-hook:
if test ! -L $(DESTDIR)$(prefix)/share/httrack/html ; then \
( cd $(DESTDIR)$(prefix)/share/httrack \
&& $(LN_S) $(htmldir) html \
) \
fi
$(WebIcon64x64_DATA) $(WebIcon128x128_DATA) $(WebIcon256x256_DATA) \
$(WebIconScalable_DATA) $(VFolderEntry_DATA) $(MetaInfo_DATA)

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">
@@ -155,6 +155,8 @@ rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
<h3 id="example">Example</h3>
One captured run; <tt>generator</tt> and <tt>date</tt> will read differently in yours.
<pre>
{
"schema": 1,

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -26,8 +26,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -32,6 +32,20 @@
}
}
/* Additions stay out of the palette block above: it is kept byte-identical with the website's. */
:root {
--body-font: system-ui, -apple-system, "Segoe UI", Roboto, "Trebuchet MS", Verdana, sans-serif;
}
/* Relative URL: the docs are read from an installed directory too. */
@font-face {
font-family: "League Spartan";
src: url(fonts/league-spartan-700-latin.woff2) format("woff2");
font-weight: 700;
font-style: normal;
font-display: swap;
}
* { box-sizing: border-box; }
/* The plain declaration before each var() is what IE11 gets: it drops the
@@ -44,6 +58,7 @@ body {
color: #111;
color: var(--ink);
font: 1em/1.6 "Trebuchet MS", Verdana, system-ui, -apple-system, "Segoe UI", sans-serif;
font-family: var(--body-font);
}
/* ---- page frame ---- */
@@ -57,14 +72,21 @@ body {
padding: .6rem 0 0;
}
.masthead img { display: block; max-width: 100%; height: auto; }
/* Take back the whitespace the SVG carries around the letters, restoring the old bitmap's spacing. */
.masthead img {
display: block;
max-width: 100%;
height: auto;
margin: -1.7px 0 -1.1px;
}
.masthead .tagline {
background: #000;
color: #fff;
font-weight: bold;
padding: .25rem .4rem;
margin-top: .4rem;
/* A one-line label has no use for the body's prose leading. */
line-height: 1;
padding: .3rem .4rem;
}
.wrap {
@@ -72,14 +94,14 @@ body {
background: var(--panel);
border-bottom: 6px solid #000;
padding: 1.5rem;
/* The rings the 2007 pages carried. The image has the light panel colour baked
into it, so it is dropped rather than inverted in dark mode. */
background-image: url(images/bg_rings.gif);
/* The rings the 2007 pages carried. Transparent, so it rides on --panel. */
background-image: url(images/bg_rings.svg);
background-repeat: no-repeat;
background-position: top right;
}
@media (prefers-color-scheme: dark) {
/* Rings dropped, not inverted: their lavender is a light-panel tone. */
.wrap { background-image: none; }
/* The wordmark is black on transparent, and all but vanishes on the dark field. */
@@ -98,10 +120,11 @@ body {
.wrap.solo { display: block; }
}
/* Colour inherited: nothing lighter than --ink reaches 4.5:1 on the --field
this sits on, and at .8em it is small text. */
footer {
max-width: 68rem;
margin: .5rem auto 0;
color: #eef;
font-size: .8em;
}
@@ -113,6 +136,9 @@ footer {
h2, h3 { scroll-margin-top: 1rem; }
/* Top two levels only: a display face reads worse than the system font over prose. */
h1, h2 { font-family: "League Spartan", var(--body-font); font-weight: 700; }
h1 { font-size: 1.9em; margin: 0 0 .3em; line-height: 1.2; }
h2 { font-size: 1.45em; margin: 2.5em 0 .5em; padding-bottom: .2em; border-bottom: 2px solid var(--rule); }
h3 { font-size: 1.15em; margin: 2em 0 .4em; }

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

View File

@@ -16,8 +16,8 @@
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<div class="wrap">

93
html/fonts/OFL.txt Normal file
View File

@@ -0,0 +1,93 @@
Copyright 2020 The League Spartan Project Authors (https://github.com/theleagueof/league-spartan)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.

Binary file not shown.

View File

@@ -13,12 +13,14 @@
</head>
<body>
<!-- doc-chrome:masthead -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
<img src="images/wordmark.svg" width="400" height="36" alt="HTTrack Website Copier">
<div class="tagline">Free software offline browser</div>
</header>
<!-- /doc-chrome:masthead -->
<div class="wrap">
@@ -58,6 +60,9 @@
<main id="main">
<!-- Section ids are link API: the redirect stubs and the Android app's contextual help
hardcode them, and an app release outlives its engine pin. -->
<h1>The HTTrack interface, step by step</h1>
<p class="lede">HTTrack copies a website to your disk so you can read it offline. The same
@@ -66,7 +71,7 @@ the screenshots and notes follow along.</p>
<div class="platforms" role="group" aria-label="Choose your version">
<button type="button" data-platform="win" aria-pressed="false">WinHTTrack <small>Windows</small></button>
<button type="button" data-platform="web" aria-pressed="false">WebHTTrack <small>Linux and Unix</small></button>
<button type="button" data-platform="web" aria-pressed="false">WebHTTrack <small>Linux, macOS and Unix</small></button>
<button type="button" data-platform="droid" aria-pressed="false">HTTrack <small>Android</small></button>
</div>
@@ -88,7 +93,7 @@ itself down once those pings stop.</p>
<p>Each option also lists its command-line equivalent, so anything you set here can later be
scripted. The <a href="cmdguide.html">command-line guide</a> covers that side.</p>
<div class="note" data-for="droid">
<div class="note" data-for="droid" id="droid-first-launch">
<p><b>First launch.</b> Android asks for permission to store mirrors on your device. Without
it the app cannot save anything, so tap <b>Allow</b>.</p>
<figure>
@@ -117,13 +122,18 @@ data-for="droid">Tap <b>Next</b></span> to create a project, or open one you alr
<figure>
<img data-for="win" src="img/guide-win-start.png" alt="WinHTTrack welcome pane" width="1040" height="744">
<img data-for="web" src="img/guide-web-start.png" alt="WebHTTrack welcome page" width="1024" height="388">
<img data-for="web" src="img/guide-web-start.png" alt="WebHTTrack welcome page" width="1024" height="387">
<img data-for="droid" src="img/guide-droid-start.png" alt="HTTrack for Android welcome screen" width="600" height="1300">
<figcaption>The welcome screen.</figcaption>
</figure>
<p class="note" data-for="web">The language dropdown starts blank on purpose: its first entry
means "leave the interface as it is". Pick a language only if you want to change it.</p>
<p class="note" data-for="web">The dropdown shows the language the interface is already in.
Picking another one reloads the page in it.</p>
<p class="note" data-for="web">On macOS, <code>brew install httrack</code> installs WebHTTrack
alongside the command line tool. Open <b>HTTrack.app</b> if you have it, or run
<code>webhttrack</code> in a terminal; either way the interface opens in your browser and
behaves exactly as described here.</p>
<h2 id="step-project">2. Name the project</h2>
@@ -133,7 +143,7 @@ together in the list.</p>
<figure>
<img data-for="win" src="img/guide-win-project-name.png" alt="WinHTTrack project name, category and base path" width="1040" height="744">
<img data-for="web" src="img/guide-web-project-name.png" alt="WebHTTrack project name, category and base path" width="1024" height="346">
<img data-for="web" src="img/guide-web-project-name.png" alt="WebHTTrack project name, category and base path" width="1024" height="345">
<img data-for="droid" src="img/guide-droid-project-name.png" alt="Android project name, category and base path" width="600" height="1300">
<figcaption>Project name, category, and where the files go.</figcaption>
</figure>
@@ -152,7 +162,7 @@ keep their links to each other.</p>
<figure>
<img data-for="win" src="img/guide-win-project-setup.png" alt="WinHTTrack action and web addresses" width="1040" height="744">
<img data-for="web" src="img/guide-web-project-setup.png" alt="WebHTTrack action and web addresses" width="1024" height="496">
<img data-for="web" src="img/guide-web-project-setup.png" alt="WebHTTrack action and web addresses" width="1024" height="495">
<img data-for="droid" src="img/guide-droid-project-setup.png" alt="Android web address and action" width="600" height="1300">
<figcaption>The addresses to copy, and what to do with them.</figcaption>
</figure>
@@ -207,7 +217,7 @@ port, you point your browser at it as a proxy and click the link or the submit b
usual, and the address the browser asks for is added to the project.</p>
<figure data-for="web">
<img data-for="web" src="img/guide-web-add-url.png" alt="WebHTTrack add-a-URL dialog" width="1024" height="634">
<img data-for="web" src="img/guide-web-add-url.png" alt="WebHTTrack add-a-URL dialog" width="1024" height="600">
<figcaption>Adding a single address, with an optional login.</figcaption>
</figure>
@@ -236,7 +246,7 @@ mirror straight away.</p>
<figure data-for="win web">
<img data-for="win" src="img/guide-win-ready.png" alt="WinHTTrack connection settings before starting" width="1040" height="744">
<img data-for="web" src="img/guide-web-ready.png" alt="WebHTTrack ready to start" width="1024" height="508">
<img data-for="web" src="img/guide-web-ready.png" alt="WebHTTrack ready to start" width="1024" height="489">
<figcaption>The last screen before the mirror runs.</figcaption>
</figure>
@@ -269,7 +279,7 @@ mirrored pages are rewritten to point at each other, so it browses offline in an
<figure>
<img data-for="win" src="img/guide-win-finished.png" alt="WinHTTrack mirror complete" width="1040" height="744">
<img data-for="web" src="img/guide-web-finished.png" alt="WebHTTrack mirror complete" width="1024" height="487">
<img data-for="web" src="img/guide-web-finished.png" alt="WebHTTrack mirror complete" width="1024" height="486">
<img data-for="droid" src="img/guide-droid-finished.png" alt="Android mirror complete" width="600" height="1300">
<figcaption>Done.</figcaption>
</figure>
@@ -282,7 +292,7 @@ usual causes.</p>
<p data-for="win web">The files sit under the base path you chose, in a folder named after the
project. Opening <code>index.html</code> there browses the mirror without HTTrack.</p>
<p data-for="droid">Mirrors are written to
<p data-for="droid" id="droid-storage-location">Mirrors are written to
<code>/storage/emulated/0/HTTrack/Websites</code>, in a folder named after the project. That is
shared storage, so a file manager or a USB cable can reach it, and opening
<code>index.html</code> there browses the mirror without the app.</p>
@@ -433,7 +443,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-flow-control.png" alt="WinHTTrack Flow Control tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-flow-control.png" alt="WebHTTrack Flow Control tab" width="1024" height="910">
<img data-for="web" src="img/guide-web-opt-flow-control.png" alt="WebHTTrack Flow Control tab" width="1024" height="862">
<img data-for="droid" src="img/guide-droid-opt-flow-control.png" alt="Android Flow Control tab" width="600" height="1300">
</figure>
@@ -476,7 +486,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-links.png" alt="WinHTTrack Links tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-links.png" alt="WebHTTrack Links tab" width="1024" height="781">
<img data-for="web" src="img/guide-web-opt-links.png" alt="WebHTTrack Links tab" width="1024" height="749">
<img data-for="droid" src="img/guide-droid-opt-links.png" alt="Android Links tab" width="600" height="1300">
</figure>
@@ -513,7 +523,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-build.png" alt="WinHTTrack Build tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-build.png" alt="WebHTTrack Build tab" width="1024" height="986">
<img data-for="web" src="img/guide-web-opt-build.png" alt="WebHTTrack Build tab" width="1024" height="976">
<img data-for="droid" src="img/guide-droid-opt-build.png" alt="Android Build tab" width="600" height="1300">
</figure>
@@ -579,7 +589,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-browser-id.png" alt="WinHTTrack Browser ID tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-browser-id.png" alt="WebHTTrack Browser ID tab" width="1024" height="706">
<img data-for="web" src="img/guide-web-opt-browser-id.png" alt="WebHTTrack Browser ID tab" width="1024" height="654">
<img data-for="droid" src="img/guide-droid-opt-browser-id.png" alt="Android Browser ID tab" width="600" height="1300">
</figure>
@@ -619,7 +629,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-spider.png" alt="WinHTTrack Spider tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-spider.png" alt="WebHTTrack Spider tab" width="1024" height="1557">
<img data-for="web" src="img/guide-web-opt-spider.png" alt="WebHTTrack Spider tab" width="1024" height="1541">
<img data-for="droid" src="img/guide-droid-opt-spider.png" alt="Android Spider tab" width="600" height="1300">
</figure>
@@ -706,7 +716,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-proxy.png" alt="WinHTTrack Proxy tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-proxy.png" alt="WebHTTrack Proxy tab" width="1024" height="762">
<img data-for="web" src="img/guide-web-opt-proxy.png" alt="WebHTTrack Proxy tab" width="1024" height="760">
<img data-for="droid" src="img/guide-droid-opt-proxy.png" alt="Android Proxy tab" width="600" height="1300">
</figure>
@@ -743,7 +753,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-log-index-cache.png" alt="WinHTTrack Log, Index, Cache tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-log-index-cache.png" alt="WebHTTrack Log, Index, Cache tab" width="1024" height="1278">
<img data-for="web" src="img/guide-web-opt-log-index-cache.png" alt="WebHTTrack Log, Index, Cache tab" width="1024" height="1266">
<img data-for="droid" src="img/guide-droid-opt-log-index-cache.png" alt="Android Log, Index, Cache tab" width="600" height="1300">
</figure>
@@ -842,7 +852,7 @@ bottom are the ones that keep you welcome.</p>
<figure>
<img data-for="win" src="img/guide-win-opt-experts-only.png" alt="WinHTTrack Experts Only tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-experts-only.png" alt="WebHTTrack Experts Only tab" width="1024" height="1088">
<img data-for="web" src="img/guide-web-opt-experts-only.png" alt="WebHTTrack Experts Only tab" width="1024" height="1085">
<img data-for="droid" src="img/guide-droid-opt-experts-only.png" alt="Android Experts Only tab" width="600" height="1300">
</figure>

View File

@@ -37,7 +37,7 @@ a:active { text-decoration: underline; }
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-image: url(images/bg_rings.svg);
background-repeat: no-repeat;
background-position: top right;
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.1 KiB

8
html/images/bg_rings.svg Normal file
View File

@@ -0,0 +1,8 @@
<svg xmlns="http://www.w3.org/2000/svg" width="501" height="456" viewBox="0 0 501 456">
<title>HTTrack rings</title>
<!-- Transparent, unlike the GIF this replaces: the panel's own #ccd shows through. -->
<g fill="#b9b9d0" fill-rule="evenodd">
<path d="M614.59 346.3A323.36 197.55 21.16 1 0 11.46 112.89A323.36 197.55 21.16 1 0 614.59 346.3 ZM590.57 315.66A262.58 160.99 22.18 1 0 104.26 117.43A262.58 160.99 22.18 1 0 590.57 315.66 Z"/>
<path d="M578.08 292.16A185.6 113.57 21.23 1 0 232.08 157.73A185.6 113.57 21.23 1 0 578.08 292.16 ZM564.4 274.56A150.77 92.55 22.26 1 0 285.34 160.32A150.77 92.55 22.26 1 0 564.4 274.56 Z"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 641 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.9 KiB

1
html/images/wordmark.svg Normal file

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 7.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 25 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 94 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 83 KiB

After

Width:  |  Height:  |  Size: 97 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 95 KiB

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 38 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.8 KiB

After

Width:  |  Height:  |  Size: 8.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.1 KiB

After

Width:  |  Height:  |  Size: 7.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.3 KiB

After

Width:  |  Height:  |  Size: 7.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.9 KiB

After

Width:  |  Height:  |  Size: 9.0 KiB

Some files were not shown because too many files have changed in this diff Show More