Fedora ships httrack with no patches, but its spec worked around three things
that were ours to fix, and its build broke twice in three weeks on changes to
our installed file set.
The metainfo declared a stock icon that is not one (stock names come from the
freedesktop Icon Naming Specification), which appstream-util rejects and which
also suppressed the merge from WebHTTrack.desktop that supplies the real icon.
libtest/readme.txt carried the pre-2005 FSF postal address. CI and
210_appstream-metainfo.test now run both validators.
225_install-manifest.test pins the installed file set, the class that broke
their 3.49.13 and 3.49.18 builds. It found a real bug on its first outing: the
.desktop files installed to $(prefix)/share/applications, so any --datadir
override put them where no desktop environment looks.
* Draw the small application icons on the pixel grid
The 16 px icon was the vector master downscaled, so every stem edge landed
on a fraction of a pixel: 74 colours in a 16x16 tile, four of them black.
The small sizes are now built in a viewBox equal to their pixel size, with
the outline coordinates rounded to whole pixels and no downscale after, so
the raster carries the field and the ink and nothing between them.
Still generated from the same Jost* master; the generator change is
httrack-works 90ceb7e2.
Test 228 counts the shipped raster's colours through a stdlib PNG reader,
since the suite cannot assume Pillow or ImageMagick on a build host.
Closes#938
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten the new test's comments
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Release 3.49.18
Source-only release so packagers can pick up the test-suite build fixes:
3.49.17-1 was FTBFS on armhf, powerpc, hppa and loong64, and the suite
failed on a Fedora build root with no ps. The engine is untouched, so
VERSION_INFO moves by revision alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Wrap the 3.49.18 changelog entry under 80 columns
lintian's debian-changelog-line-too-long fires on two lines of the new
entry, and the CI gate runs --fail-on=error,warning. The sid perl
transition that had been masking this gate has cleared, so it is live
again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Sharpen the 3.49-18 release notes
The #1021 line named the symptom and implied the remedy was a louder
failure; the fix is a /proc fallback, so the diagnostics work. Add the
internal catch-all the 3.49-17 block carries, so the cross-arch matrix
appears to the upstream audience and not only in debian/changelog.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Release 3.49.17
Bumps the four version spots, the AppStream release block and VERSION_INFO
(3:8:0 -> 3:9:0, revision only: #991 added an exported symbol and #1018 changed
no POSIX ABI, so the soname stands and there is no Debian package rename).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cite issues rather than PRs in the release notes, and fill three gaps
House style cites the issue a change closes; five lines named the PR instead.
The pkg-config line cited #978, which is the macOS relocation report and says
nothing about pkg-config.
Adds the command-box escaping fix (#998), the struct addrinfo scoping (#987)
and the four-architecture build failure (#1015), all user-facing and none
covered by a neighbouring line. The crash-report line now names the loader
case it was citing #996 for, and drops #997, which has no user-visible symptom.
AppStream said C:\ab was mangled as C:\folder, but the decode needs two hex
digits after the backslash, so \fo never matched.
VERSION_INFO's rationale named only #991; #1005 exported escape_control_url
too. The bump stays revision-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* WebHTTrack settings pages interpolate values into double-quoted attributes unescaped
#988 escaped the templates' element bodies and event handlers but left
attribute values alone, because cat_html_escaped() keeps the double quote
raw on purpose: the tooltips are written title='${html:LANG_x}'. The
settings pages put about seventy values inside double-quoted attributes,
where a " opens a new attribute.
Add a fourth interpolation mode, attr:, escaping < > & ' " as entities and
nothing else, and point the 230 double-quoted attribute positions at it.
file.html's file:// href stays on the html-urlescaped mode, which now emits
%22 for the quote the way unquoted: already did, and step2.html's in-script
literal moves to js:. Test 185 now requires attr: in a double-quoted
attribute and js: inside a script block; 217 drives hostile values through
three templates and a translation.
Closes#989
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Close the two attribute sites the template sweep could not reach
The liststr: format builds its own <option value="..."> in C, so no
template edit covers it, and it escaped with the helper that leaves the
double quote raw. Its values are the project directory names and the
Category= line of each project's winprofile.ini, both of which can hold
a quote.
The save side kept hex-decoding a backslash plus two hex digits, so a
path typed as C:\ab displayed correctly but was written to ~/.httrack.ini
as C: plus byte 0xAB. The ini writer has no inverse for that escape; the
one caller that needs it is a template literal (${test:proxytype:\3A//}),
so the decode now applies only there.
Test 185 classified attributes with a regex that missed a spaced =, an
unquoted or single-quoted value, a template outside html/server/*.html,
and an output-mode region the file never closes; the counting floors it
asserted also survived a count-preserving rewrite. It now walks tags,
asserts the property on every site it classifies, and proves it can see
each of those forms on a synthetic template first.
Test 217 gains a Latin-1 byte in a translation, the POST-render-POST
round trip, and the two fixes above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the comments the last commit added
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Escape crawled URLs in the WebHTTrack progress panel
refresh.html interpolated ${info.state}, ${info.name} and ${info.file} as
raw HTML and ${info.url_sav} into a single-quoted JS literal, so a link
path holding < > or ' ran as script in the panel origin. Point them at
htsserver.c's html: and js: modes, and sweep the other templates for the
same shape. Widen 185's scan to any unescaped handler or element-body
interpolation, and add 186 to probe the rendered panel during a crawl.
Closes#973
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the panel's name column in the escaping test
186 asserted only on the leaf of the crawled path, so info.name could go
back to raw and still pass: 185 cannot see it either, because ${unquoted:}
sits in the directive regex it skips. The link now carries metacharacters
in a directory component too, short enough to survive the engine's 40-char
elision of that column, and the poll breaks on either form so the positive
control stays honest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make test 186 executable, like its siblings
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Re-arm the output mode that ${do:end-if} clears
${do:end-if} sets outputmode to 0 rather than restoring what it replaced
(htsserver.c), so the if-not-empty block partway down step4.html's command
textarea drops the ${do:output-mode:html} armed above it, and the 17
${unquoted:} settings below render raw. Posting maxtime or url2 as
A</textarea><img src=x onerror=alert(1)>B breaks out of the textarea.
finished.html's <pre> has the same shape; those two are the only ones.
--sitemap-url moves from html: to arg: like its quoted siblings, since
cat_html_escaped() escapes neither " nor \.
Test 185 had unquoted: on its directive skip list, which is how all 17
sailed past it, and now also rejects an html: value on the command line.
Test 83 reads back the escaped forms at runtime.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Release 3.49.16
Version bump across configure.ac, htsglobal.h, version.rc and the AppStream
metainfo, plus the release notes for history.txt and debian/changelog.
VERSION_INFO 3:7:0 -> 3:8:0: no installed struct or exported signature moved
this cycle, so the soname stays .so.3 and Debian needs no package rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Correct the 3.49.16 release notes
The AppStream metainfo describes WebHTTrack and is read by Linux software
centres, so it now lists only WebHTTrack-visible changes: the macOS bullet and
the nine option-dialog strings (WinHTTrack-only, no hits outside lang/) are out,
the charset re-encoding is in. Drop the task-switcher claim, which webhttrack
cannot make: it is a script that launches a browser.
Fix two over-claims in history.txt. Only the trailer section after the
terminating chunk was rejected, not any chunked page. The cache aborted the
mirror on an over-long URL rather than merely refusing one it had accepted.
Cite #901 rather than the PR that closed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Tooltip JavaScript breaks on any translation containing an apostrophe
The onMouseOver handlers interpolate through ${html:...}, which escapes
for HTML text. The browser decodes an attribute value before compiling
it as JavaScript, so ' arrives as a bare quote and closes the string
literal; a translation carrying a double quote ends the attribute
outright. Add a js: template mode escaping for both layers and move all
304 handler sites onto it.
Closes#864
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Close the step2.html and title-attribute gaps in the js: escaping test
The static check could not match ${html:html:...}, the very shape it was
added to catch, and the runtime probe only fetched option1.html: reverting
all eight step2.html handlers left the test green. Nothing asserted that a
title= attribute stays on html: outside option1.html either.
Walk every server template instead, pairing each ${...} with the attribute
holding it, and fetch step2.html as well. Also feed the fixture a newline
and a tab, which had no coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Emit the backslash as \x5c so a DBCS trail byte cannot swallow it
shift-jis, BIG5 and gb2312 all accept 0x5c as a trail byte, and those are
the declared charsets of three shipped lang files, so the browser really
does run a DBCS decoder over the page. An orphan lead byte followed by the
emitted \\ pair decodes as one glyph plus a stray backslash, which escapes
whatever comes next. Emitting every escape as a \xNN group leaves the
decoder nothing to pair with: the only bytes cat_js_escaped adds are '\',
'x' and hex digits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Icon=httrack never resolves through the share/pixmaps fallback
share/pixmaps only ever held the sized names, so the bare-basename lookup a
desktop falls back on when no icon theme is installed never matched. Install
httrack.xpm (a 32x32 copy) alongside them rather than renaming one: Debian
globs the whole directory and a rename would strand the old names on upgrade.
Closes#932
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the pixmaps fallback to the icon, not to any 32x32 image
The geometry check passed on a solid-red square, so compare the installed
fallback against the installed 32x32 copy, declaration line aside. Also drop
the banned pipe into grep -q while here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The application icon is still the pre-brand bitmap set
Replace the 16/32/48 PNGs and the .xpm fallbacks with the HT monogram
generated from the same Jost* master as the masthead wordmark, and extend
the hicolor theme with 64, 128, 256 and a scalable SVG.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the icons on what install and dist emit, not on Makefile.am text
Six mutants survived the first version: an emptied EXTRA_DIST, an empty *dir
variable (automake's install rule exits 0 when it is), a _DATA glob with the
wrong extension, a _DATA entry for a directory the tree does not have, a
missing apps context subdirectory, and a size dropped from install entirely.
All six now fail, and the PNGs are indexed rather than RGBA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The masthead wordmark is a 400x34 GIF that blurs on any hi-DPI screen
Replaces it with an outlined SVG across the 38 documentation and WebHTTrack
pages that carry it. The original was set in Futura, so the lockup was refitted
in Jost*, the closest free Futura revival, taking weight from the measured stem
thickness, size from the cap heights and tracking by least squares against the
glyph positions in the old bitmap.
tests/82 now asserts that every image a GUI page names is actually served.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Point the shared chrome generator at the new wordmark
The masthead of the 13 generated pages comes from tools/doc-chrome.py, so
editing the pages alone left the generator disagreeing with its own output and
--check red.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The panel decoration was drawn as vector and flattened to a 4KB indexed GIF
some time around 2007, with the panel colour baked in as an opaque backdrop.
Refitting its four ellipse boundaries recovers the original geometry, so it
goes back to being what it was: two elliptical annuli, 553 bytes of SVG, with
a transparent background that now composites over the panel instead of having
to match it.
Rasterised at the same size, the only pixels that differ from the GIF are
single-pixel anti-aliasing fringes along the four boundaries. Nothing survives
a 3x3 erosion of that difference, so no edge has moved.
Dark mode still drops the image rather than inverting it, since the ring
lavender is a light-panel tone whichever way the file stores it.
The engine keeps its own embedded copy of this GIF for the backblue.gif it
writes into mirrors. That one is untouched: the filename and byte length are
a contract with pages already on disk.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The AppStream metainfo still advertises WebHTTrack 3.49.8
The metainfo installs to usr/share/metainfo, so GNOME Software and KDE
Discover read both the version and the "What's new" text out of it. Its
releases block held one entry, 3.49.8, and nothing had moved it since.
It now lists 3.49.8 through 3.49.15, newest first, each with a short
user-facing note taken from history.txt. Dates come from the git tags;
that also corrects 3.49.8's, which carried 3.49.7's date.
01_engine-version-macros.test gains two assertions so the next release
cannot miss this file, or configure.ac: AC_INIT and the top release entry
must both match HTTRACK_VERSIONID, and the entries must descend so the
top one really is the newest.
Closes#884
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Harden the metainfo version check and correct four release notes
The release-version extraction matched <releases ...> as well as <release>,
ignored XML comments and took only the last tag on a shared line, so a parked
or wrapper version= could pose as the newest entry and pass a stale metainfo.
Split tags one per line, drop comments, and anchor the match. Widen the awk
ordering key so a component of 1000 or more cannot borrow into the next.
In the notes: "3.49-2" and "site rules with wildcards" are unreadable in a
software centre, the Windows path bullet does not apply to the Unix WebHTTrack
GUI it ships with, and 3.49.15 listed no web-interface fix at all.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* One guide for WinHTTrack, WebHTTrack and Android, instead of 18 thin pages
The GUI documentation was 18 files whose bodies were mostly a copy-pasted style
block, illustrated with 1998-era screenshots of WinHTTrack, with the Android app
documented separately and WebHTTrack not shown at all. html/guide.html replaces
them with a single page: one walkthrough and one option reference, and a
switcher that swaps the screenshots and the platform-specific notes between the
three versions. Each option now also names its command-line equivalent.
The screenshots are the three capture sets from httrack-works, prepared by
tools/doc-images.py. Deep links carry the platform, so guide.html#win/opt-limits
opens the Limits section already showing the Windows dialog; the WebHTTrack
option pages use that form. The old filenames stay as redirect stubs for the
forum links that still point at them.
Scripting is an enhancement throughout: without it the page shows every
platform's text and screenshots rather than hiding any of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Document the options the old pages never had
A coverage probe against the WebHTTrack form controls found the reference
inherited the old doc's blind spots: WARC, the CDXJ index and WACZ, --changes,
sitemap seeding, cookies-file, the urlhack opt-outs, strip-query, the proxy type
selector and the random inter-file pause were all exposed in the UI and absent
here. Labels and descriptions taken from lang.def, CLI flags from htsalias.c.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* The webhttrack GUI cannot reach --warc-cdx, --wacz or --warc-max-size
PR #672 exposed --warc and --warc-file on the "Log, index, cache" tab; the
three WARC flags added after it never got a control, so a CDXJ index or a
WACZ package is unreachable from the GUI. Add the two checkboxes and the
size field, and translate the six new lang.def strings into all 30
languages instead of letting them fall back to English.
An unknown ${LANG_*} renders as the empty string, so a key missing from
lang.def or from one lang/*.txt blanks a label instead of failing: test 140
now asserts every key the templates use exists everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Move the lang completeness check into 62_lang-integrity, its owner
62_lang-integrity.test already owns lang.def/lang-file consistency; it
checked that no msgid drifted, but not that every msgid is translated
everywhere, which is the gap that let the WARC options ship untranslated.
Add the completeness pass there and drop the separate 140 test, which
reimplemented the same parser in Python and was scoped narrowly enough to
be blind to the 24 msgids already missing from 26 files. Those are waived
by name against #863, and a waiver that stops being needed now fails.
Also check every ${LANG_*} the templates interpolate resolves, in any
wrapper form: the ${html:html:...} and ${fexist:...:...} spellings were
outside the dropped test's regex.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Read sitemap files so URLs nothing links to are found
HTTrack finds URLs only by parsing links, so anything a site publishes solely
in its sitemap stayed invisible: robots.txt was already parsed, but its
Sitemap: lines were ignored and nothing else in the tree touched sitemaps.
Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and
falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document.
Handles <urlset> and nested <sitemapindex>, plain or gzipped. Discovered URLs
enter with the full depth budget but still go through the wizard, so filters
and scope rules decide; a sitemap is not a filter bypass.
The parser reads attacker-controlled XML off the network, so it is capped on
URL count, index nesting, decompressed size and decompression ratio, and child
sitemaps must stay on the host that named them.
Closes#712
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: distinguish the sitemapindex log line from a urlset one
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: fix an out-of-bounds read, tighten the parser and the tests
lienrelatif() walked back from the last character of its current-path
argument without checking the path was non-empty, reading one byte before
the stack buffer. htsAddLink is the first caller to pass an empty savename,
which sitemap documents have because they are ingested rather than mirrored,
so ASan caught it on the new crawl test.
The parser drops a value whose numeric character reference decodes outside
printable ASCII, rather than leaving the reference verbatim and seeding a URL
the site never published, and classifies a document by its real root element,
so a comment naming the other one no longer flips urlset and sitemapindex.
The robots.txt line reader is bounded by the body size instead of relying on
a NUL terminator.
The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only,
because an uninstrumented libz floods it with false positives.
Tests gain the assertions the earlier ones were missing: which of the
robots.txt route and the /sitemap.xml fallback was taken, that the sitemap
documents stay out of the mirror, that the off-host child sitemap is refused,
the sitemapindex nesting cap, the per-document URL cap at its production
value, and copy_htsopt coverage for the two new fields.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: state what the decompression cap actually binds on
deflate tops out near 1032:1, so hts_codec_maxout never binds before the
64 MiB cap; the old comment implied a ratio guard that cannot fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: a child sitemap is a fetch, so filters and robots.txt must gate it
An adversarial review found that a <sitemapindex> <loc>, and a robots.txt
Sitemap: line, went straight to hts_record_link: the request went out even
when a -* rule or robots.txt Disallow covered it. Only the <urlset> half ran
through the wizard. Gate the document itself on the filters and on
robots.txt, which is all that can apply: the wizard proper wants a referring
link, and its up/down travel rules would judge a child sitemap against the
parent sitemap's own directory. The robots.txt probe is exempt, being the
request that fetches the rules.
A 301 also used to end ingestion silently, since the engine re-queues the
target as a fresh link that carried no sitemap marking. That hit any site
redirecting http to https. The marking now follows the redirect.
The "N URL(s) added" counter reported what the scanner emitted rather than
what was taken, which hid both of the above; it now reads "N of M". The
fallback to /sitemap.xml keys on the same corrected count, so a robots.txt
whose only Sitemap: line is off-host or filtered still falls back. Root
classification skips a UTF-8 BOM and an XML namespace prefix, and the doc
list is cleared when a mirror starts rather than only when it ends.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: gate each fetch by who asked for it, and anchor travel on the start URL
A nine-agent review found a scope escape: the sitemap document was its own
`premier`, so the wizard measured travel from wherever the site chose to put
its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the
whole host. The ingester now points the wizard at the crawl's own start link
and lets each seeded URL become its own anchor, which is what a command-line
seed gets.
Robots handling was both mistimed and undifferentiated. The Sitemap: lines are
now collected by robots_parse, on the same body in the same fetch, and acted on
after the parsed rules are installed rather than before; and the decision comes
from a new hts_robots_forbids extracted out of the wizard, so the sitemap path
inherits the -s1 filters-win override instead of a stricter hand-rolled check.
The four fetches are no longer treated alike: a sitemap the user names is user
intent, one the site declares invites the fetch, only the guessed /sitemap.xml
obeys a Disallow, and the URLs listed inside stay fully gated.
Also: hts_unescapeEntities replaces the private entity decoder, whose guard
tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead,
which is only defined under HTS_USEZLIB; the composed URL buffer now fits two
maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the
bounded search is promoted to htstools as hts_memstr; and the live state moves
from httrackp into htsoptstate, leaving two installed fields rather than three.
Tests gain the scope escape, the three robots cases, a cap-boundary control,
the handler invocation count and a compression-bomb decode. Every one was
checked against a deliberately broken build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: add the fuzz harness the parser was missing, and drop truncated Sitemap: lines
The file header called the scanner fuzzable while fuzz/ registered ten
harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies
and truncated streams off a heap copy of exactly the input size, so an overread
is an ASan report rather than a quiet pass, with a four-file seed corpus.
60000 runs clean under ASan+UBSan.
robots_parse now drops a Sitemap: line that filled its scratch buffer instead
of handing on the half URL it was truncated to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* fuzz: keep only the four sitemap seed inputs
A libFuzzer run writes its finds into the first corpus directory, and 191 of
them were committed with the harness.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* selftest: bound the sitemap document builders' snprintf accumulation
snprintf returns the length it wanted to write, so accumulating it blind
lets the next offset and size argument walk past the buffer. Guard each
step the way the argv builder above already does, and give the per-URL
loop a real remaining-space bound instead of a fixed 33.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: date the new files 2026
The headers were copied from an existing file and kept its 1998 year.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: keep the ingestion state out of htsoptstate
htsoptstate is embedded by value as httrackp.state, so a field at its tail
shifts every httrackp member declared after it: an offsetof probe put
warc_file at 141752 on master and 141760 on the branch. Move the pointer to
httrackp's own tail, where every existing offset holds and copy_htsopt still
ignores it.
Also renumber the crawl test to 89, master having taken 87 and 90, and give
the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear
requires, plus its row in that test's table.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: renumber the sitemap crawl test to 95
Master took 88 through 93 and #720 claims 94.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: realign the htsopt.h comments after the single-file merge
Master's longer LLint declarator moved the block's comment column.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* sitemap: translate the new GUI strings into the remaining 28 locales (#738)
The feature PR added the four LANG_SITEMAP* entries to lang.def with English
and Francais only; every other locale fell back to English in the WebHTTrack
form. Each file is written in its own declared charset.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Single-file output is MHT, which browsers no longer open
Add --single-file (-%Z): after the mirror completes, rewrite every saved
page in place with its stylesheets, scripts, images and fonts embedded as
data: URIs, while links between pages stay relative. The mirror remains a
browsable tree and each page also stands alone.
This cannot reuse the -%M path: MHT streams a MIME part per file as it is
saved, but a data: URI needs the asset's bytes when the page is written,
and pages are normally saved before their assets are fetched. The new pass
runs over the finished tree at the tail of httpmirror(), after the update
purge.
Audio, video, page-to-page links and anything over --single-file-max-size
(10 MB default) keep an ordinary link. References carrying a scheme are
skipped, which covers data: and makes a second --update run a no-op.
Resolution is clamped to the mirror root: the HTML is hostile input.
htsopt.h gains two tail-appended fields; VERSION_INFO is untouched.
Closes#713
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* tests: renumber the single-file test to 87, run the self-test last
Master took 82 through 85 while this branch was open. The engine self-test
also moves to the end of the script: it and the crawl assertions cover
different ground, and failing first hid the crawl half.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Fix the parser and path bugs review found, and cover them
sf_relative_from() indexed one byte past from_dir's terminator whenever the
page's own directory was a prefix of the asset path, which is the ordinary
mirror layout: an out-of-bounds read that also miscounted the ../ prefix and
emitted a broken link. Two review agents hit the same ASan trace; the branch
had no test because every nested asset in the fixture was under the cap.
Also from review: an escaped quote no longer ends a CSS string early (which
exposed its contents to the url()/@import scanner), @import url(...) now
inlines like the quoted form, a raw-text element ends only on a real end tag
rather than any prefix of one, an over-wide tag is copied through by the same
quote-aware scan instead of a second quote-blind one, <!--> is an empty
comment, whitespace before a tag's > survives, and a page cannot inline more
than SINGLEFILE_MAX_PAGE_SIZE, which bounds the multiplicative @import
fan-out. A failed encode no longer leaves a payload-less data: prefix behind,
and base_dir is matched against the root on a component boundary.
sf_readfile now wraps a new readfile2_utf8() rather than being a fifth copy
of the readfile family. Docs place --single-file beside -%M instead of
implying it supersedes it: MHT stays the better container, this wins on
opening anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* singlefile: tighten two comments
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* tests: cover the option plumbing review found untested
copy_htsopt's two new fields, including the >0 guard that must not let an
unset source clear the target's default; -%Z0; and a rejected
--single-file-max-size argument falling back to the built-in cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* tests: stand the single-file GUI check down where htsserver is absent
The Windows job builds httrack.exe only, and reaches this file through its
*_local-*.test glob, so requiring htsserver failed the whole test there even
though every crawl assertion had passed. Skip that half instead, and run the
engine self-test before it so the skip cannot swallow it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* tests: check the sprintfbuff master just made warn_unused_result
#722 gave slprintfbuff the attribute, so the over-wide-tag fixture's call
became the one warning this branch adds over master's baseline. Handle it the
way that PR's own self-test code does; the buffer cannot truncate here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* single-file: translate the new GUI strings into the remaining 28 locales (#739)
The feature PR added the four LANG_SINGLEFILE* entries to lang.def with English
and Francais only; every other locale fell back to English in the WebHTTrack
form. Each file is written in its own declared charset.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* single-file: restore the mirror file mode, and give the guards real coverage
The rewrite spools to a .sfnew temp and renames it over the page, which
bypasses filecreate() and with it the HTS_ACCESS_FILE chmod the engine puts on
every other mirrored file. Under a restrictive umask the pages came out 0600
while their assets stayed 0644, so a mirror served by a webserver or shared
with a group lost read access on exactly the pages. chmod the spool before the
rename.
Two guards had no coverage: deleting the scheme/data: check in sf_resolve left
both the self-test and the crawl test passing, because the fixtures resolved to
paths that were absent either way, and the per-page inline budget was never
exercised. The fixtures now plant a file where each guard's removal would land
the walk, and a self-importing stylesheet measures the budget against a
large-budget control. Charging that budget after the nested rewrite instead of
before let an @import chain spend what its ancestors had already claimed and
drove it negative; charge it up front and refund on failure.
The attribute table missed the lazy-loading attributes hts_detect[] already
downloads, so a modern page inlined almost nothing: add data-src, data-srcset,
lowsrc, object@data and embed@src, and record why the rest stay links.
The new web GUI checkbox had no hidden companion input, so it could be ticked
but never cleared, which is what #725 fixed for every other box. Master's test
90 catches it once the branch merges.
Adds a libFuzzer harness over the rewriter, since it re-serializes hostile
HTML, and renames the crawl test to 91 now that master holds 87 and 90.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* single-file: document the inlined-stylesheet limitation in the CLI guide
Recorded in htssinglefile.h already; the user-facing guide is where someone
raising --single-file-max-size will look.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: move the Windows note down to the gate it explains
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* ci: register the single-file test's Windows skip
Its GUI half needs htsserver, which the Windows job does not build, so the test
now exits 77 there instead of reporting a pass for assertions it never ran. The
skip list is pinned, so it has to be declared. Both Windows jobs reported
fail=0; only the list check was red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Report what a crawl changed against the previous mirror (--changes)
--update already knows which resources were new, which changed and which the
server called unchanged, and throws it away: the flags reach file_notify() and
go no further than a log line, while deletions exist only as a side effect of
purging. --changes (-%d) keeps all of it and writes hts-changes.json plus a
one-line summary in the log.
"Changed" means the bytes differ, not that the server re-sent the resource.
Comparing the mirrored files directly would not work: HTTrack stamps every
parsed page with the crawl date via the footer, so those bytes differ on every
run. Payloads are compared instead, the previous one coming from the cache for
parsed pages and from the local copy sampled just before it is overwritten for
everything else.
The mirror-relative path, not the URL, is the accumulator's key, so a redirect
and its target that share a save name are one entry; and only the first notify
for a file samples its pre-run state, so a retried transfer is not counted
twice. What counts as already mirrored comes from the previous run's file
index rather than from the file's presence on disk: a partial left by this
crawl's own failed attempt is on disk but was never part of the previous
mirror.
The deleted set is now computed whether or not purging is enabled; unlinking
still happens only under --purge-old.
Closes#714
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* changes: never leave a stale report behind
A crawl that mirrored nothing created no accumulator, so hts_changes_close_opt
returned without writing and the previous run's report stayed on disk as if it
described this one. Write it whenever --changes is on. The no-data rollback is
the deliberate exception, and is now documented: it restores the previous cache
generation, so leaving the matching report alone is the consistent behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* changes: drop em dashes from the format page
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* changes: fix the review's three findings
The size shortcut compared rendered on-disk sizes even for parsed pages, whose
payload digests describe something else entirely, so it decided the outcome
before the payload comparison could run: a page whose payload never changed but
whose rewritten links moved read as changed. It now only applies when both
digests describe the file on disk.
file_notify() reaches the accumulator from the FTP download thread as well as
the main one, and the lazy allocation, the coucal write and the entries realloc
were all unguarded. Every entry point now takes a mutex, and the HTML hook does
its cache read before taking it, since that read can itself re-enter
file_notify() and move the array.
Two fixtures cover what nothing did: a gzipped direct-to-disk body that changes
at constant length (without the pre-sample before the decoded temp is renamed,
it reads as unchanged), and a page with a fixed payload behind a redirect whose
target is renamed, so its file on disk changes length while its bytes do not.
Both were checked against builds with the respective fix reverted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* changes: document the renamed-file case in the format page
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* changes: refresh two stale test comments
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Merge origin/master into feat/change-report
Both sides appended to tests/Makefile.am's TESTS; kept master's
86_local-proxytrack-cache-longfields.test alongside 88_local-changes.test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* changes: translate the new GUI strings into the remaining 28 locales (#740)
The feature PR added the two LANG_CHANGES* entries to lang.def with English and
Francais only; every other locale fell back to English in the WebHTTrack form.
Each file is written in its own declared charset.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* changes: fix the review's blocking findings
Lock the report path against the FTP thread the crawl never joins, seal the
accumulator once the report is written, key entries off the project directory
so the report survives --cache=0, stop calling a file gone when the crawl only
failed to re-fetch it, and skip the hook's work entirely when --changes is off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* changes: prove the fixes, and give the report a cache-off mode
Adds a changes-race self-test (the FTP shape: notifier threads against the
report path), fixtures for a transfer the crawl never completes, for a leftover
file at a name the crawl mirrors fresh, and for a cache-off mirror, plus a pass
with purging on. Registers the web GUI's --changes box with the clearing
companion master's #725 now requires, and documents the degraded mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* changes: keep the failed-transfer case portable
A connection killed before the status line surfaces differently on macOS, where
it truncates the mirrored file to zero (#748). Assert only what holds on both:
it is never reported gone, and its file survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Options ticked on by default cannot be un-ticked in the web GUI
An unchecked HTML checkbox posts nothing, so htsserver never overwrites the
value it already holds. Every box in the wizard is one-way: once the stored
value is "1", whether htsserver seeded it at startup, a loaded profile set it,
or the user ticked it earlier in the session, un-ticking and submitting leaves
the option on and draws the box ticked again. Only four boxes, all in
option1.html, carried the companion hidden field that guards against this.
Add it to every remaining bare checkbox, and switch cookies and parsejava to
${ztest:...} so a cleared box emits --cookies=0 / --parse-java=0; ${test:...}
renders nothing at all when the value is empty, which is not "off" for an
option the engine turns on by default.
index, urlhack and keep-alive are deliberately left alone: their long options
are declared "single" in htsalias.c and optalias_check drops the =value, so
--index=0 resolves to -I and turns the option back on. That parser bug is
pre-existing and needs its own fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: pin last-write-wins and cover every checkbox
The runtime leg posted each name once, so a first-wins body parser would
have passed while the fix did nothing in a real browser: post the
duplicated name in both orders and assert the last value wins. Replace the
four hand-written option cases with a table covering all 28 non-skipped
boxes, asserting the command-line token and the Windows-profile key each
state emits, plus a completeness check so a new box cannot slip through
unexercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: rename the loop variable shadowing the scanned page
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* WebHTTrack's mirror links are dead file:// URLs
The GUI is served from http://127.0.0.1:PORT/, and browsers refuse to navigate
from an http: page to a file: URL, so the "browse the mirror" links in
finished.html and file.html did nothing when clicked. They now point at the
server's own /website/ route, which already serves the project directory. The
desktop entry's browse mode still works, because there the shell hands the
file: URL to the browser instead of navigating from a page.
The per-project picker in file.html goes too: /website/ is bound to the running
project, and reaching the other projects over HTTP would mean serving the whole
mirror tree from the control origin.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* finished.html: link the served mirror, not a dead file:// URL
The page is served over http:, so its file:// link is a cross-scheme
navigation that Chrome and Firefox refuse. Nothing happened on click. The
mirror is already reachable at /website/, which the two list entries just
below were using all along.
file.html keeps its file:// links. It is the fresh-session entry point for
sites mirrored earlier, and /website/ is bound to one project, so pointing
it there would trade a dead link for a 404. Serving an arbitrary past
project needs a route that does not exist yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: tighten the browse-link assertion and trim its comments
The runtime check for href="/website/index.html" also matched the list
entry below the anchor, so it passed on the unfixed page; match the
anchor by its mirror-path label instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* webhttrack: "max site size" set a per-file cap instead of the overall one
step4.html mapped all three size fields of the wizard onto --max-files (-m),
so "Max site size" emitted a per-file limit rather than --max-size (-M), and
landed a second -m on the command line. That second -m also clobbered the HTML
per-file limit: a bare -m<n> resets maxfile_html, so whichever of the two came
last won. Point sizemax at --max-size and emit the bare -m before the -m,<n>
form so both per-file caps survive.
Two template typos in the same family, where a malformed ${...} renders as
nothing or as its own key instead of erroring: the winprofile.ini writer's
Dos=${dos was missing its closing brace, and option2b.html's OK button read
${LANG_OK] with a bracket.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* tests: carry the session id in test 81's POST
The gate that landed with #700 refuses a body without one, so the wizard
POST came back refused and the option audit had nothing to read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: close the confirmation-biased gaps in test 81
Post the size fields empty too, so a step4.html that lost its ${test:} guard
and rendered a valueless --max-files= is caught; assert the winprofile.ini
MaxHtml/MaxOther/MaxAll keys the header claimed to audit; and pin the OK button
label, since an unknown ${LANG_} key renders empty and passed the absence check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsserver builds the redirect Location header in a 256-byte stack buffer
The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.
Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.
The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: satisfy shellcheck and shfmt in the new server test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the pre-fix narration from the oversized-value comment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the bound socket, not the announced URL
The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.
Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: bound the response read and the empty --bind run
The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver: the session id must gate the request body, not the reply
Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.
Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.
A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.
Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.
Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* webhttrack: bound the argv vector and escape quotes in the wizard command line
The wizard hands its httrack command line to the engine as one string, which
back_launch_cmd() split back into argv. Two things were wrong with that split.
It wrote into a fixed 1024-pointer vector with no bound, and every unquoted
space in the posted string yields an entry, so an ordinary mirror with a few
hundred URLs walked off the allocation. The split now lives in htscmdline.c as
hts_split_cmdline(), which sizes the vector from the separator count before
filling it, and the engine self-tests can reach it.
Quotes were also purely advisory: they toggled the "inside an argument" state
but nothing escaped them, so a double quote typed into a wizard field (user
agent, footer, path, project name) closed the argument early and the rest of
the value was parsed as fresh options -- among them -V, which reaches system().
Escaping has to happen where the argument boundary is known, so the template
gets its own ${arg:} filter for that context; ${html:} keeps its meaning for
the HTML attributes it is used in everywhere else, and HTML escaping would not
help anyway since the browser undoes it when it posts the command line back.
${arg:} backslash-escapes a quote and a backslash, and the splitter reads those
inside a quoted run, the same convention next_token() already implements for
doit.log. A value containing a quote now survives it intact instead of turning
into options.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* webhttrack: keep a stray quote in an unquoted field out of the split
The url and wildcard-filter fields go into the command line outside quotes,
where no backslash can escape anything: a single quote there flips the parity
of every quote after it, so a later escaped value ends up split as flags and
the escaping buys nothing. Emit %22 for those fields instead.
NULL-terminate the argv vector while here, matching the convention the tree
documents in htscharset.c, and fold the third copy of the entity table into
one helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: do not feed snprintf's return value back as its size argument
snprintf returns the length it wanted to write, so accumulating it blind
lets the next size argument wrap. The buffer is sized well past what the
loop needs, but the pattern is the one the project forbids.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Expose --warc / --warc-file in the webhttrack GUI
Add a WARC output checkbox and an optional archive-name field to the
"Log, Index, Cache" option tab, beside store-all-in-cache. The checkbox
emits --warc (auto-named archive); the text field emits --warc-file NAME.
Wiring mirrors how #589 added cookies-file and strip-query: the option9.html
form fields, the generated httrack command and winprofile.ini in step4.html,
and the reload remap in step2.html.
New LANG_WARC / LANG_WARCFILE strings and tooltips land in lang.def,
English.txt and Francais.txt; the remaining 28 language files fall back to
French for now and are a follow-up.
The webhttrack smoke test now also fetches option9.html and requires the
WARC control to render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Translate the webhttrack WARC strings into the remaining languages
PR #672 added a WARC toggle to the webhttrack GUI with 4 new LANG keys,
translated only in English and French. Append the same 4 msgid/translation
pairs to the other 28 lang/*.txt so those locales stop falling back to
English. Each translation is encoded in the file's declared LANGUAGE_CHARSET
(the charset the server serves the page and parses the form as) with a strict
lossless round-trip, matching each file's existing CRLF/LF line ending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Select URLs" and "Start" section headings on the webhttrack server
wizard were literal English, bypassing the ${LANG_...} substitution, so
they stayed English in all 30 locales. Reuse existing, already-translated
keys: step3 -> ${LANG_G44} ("Web Addresses: (URL)"), step4 -> ${LANG_J9}
("Start"). No lang/*.txt changes needed.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps the copyright year in the html/ documentation footers (and the inline notice in contact.html) from 2007 to 1998-2026. Footer text only, no content changes; fcguide.html (Fred Cohen, upstream) is left untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
HTTPS already reaches the origin through a CONNECT tunnel (#85), but plain HTTP still used the absolute-URI request form, which a CONNECT-only proxy like Tor's HTTPTunnelPort rejects. Adds a `connect://` proxy scheme (`-P connect://host:port`) that drives the proxy with CONNECT for every request: httrack opens a tunnel to the origin (port 80 for http, 443 for https), then sends the request origin-form with no Proxy-Authorization, the same way https does.
It can't be the default (CONNECT-to-:80 breaks proxies that only allow CONNECT on 443) and auto-detection is ruled out, so the mode rides the proxy scheme like socks5:// does. That keeps it ABI- and cache-safe (the scheme lives in proxy.name, no new struct field) and drops into the webhttrack GUI as one more entry beside HTTP and SOCKS5. The tooltip was reworded and retranslated across the lang/*.txt files.
Test 57 fronts a plain-http origin with a CONNECT-only proxy (it 501s anything but CONNECT) and checks the request arrives origin-form, that dropping connect:// gets the classic form refused, that credentials ride the CONNECT and never reach the origin, that a portless origin tunnels to :80, and that a flooding proxy does not hang.
Closes#564.
The web GUI carried per-control help via onMouseOver="info(...)", which
only wrote window.status. Browsers stopped honoring scripted window.status
writes years ago, so those hints render nowhere.
Emit each hint as a native title='...' attribute alongside the existing
handler, reusing the same ${html:LANG_*} string. Single-quoted so the
filter's ' -> ' (and <>&) escaping keeps the attribute well-formed; a
literal " is harmless inside single quotes. The window.status calls stay
(inert) to keep the diff to added attributes.
The webhttrack smoke test now also fetches an option page and requires a
rendered title=' tooltip.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The web GUI was frozen before the engine gained SOCKS5 proxying, --cookies-file,
--pause, --strip-query, and the three --keep-* URL-hack opt-outs. Wire each into
the option tabs, the generated httrack command, the winprofile.ini persistence,
and the reload remap so they reach parity with WinHTTrack (xroche/httrack-windows#31).
The SOCKS5 selector reuses the single proxy field and prepends socks5:// to -P;
it is gated on a non-empty host so a SOCKS5-selected-but-blank config can't emit a
stray scheme token. winprofile.ini key names match WinHTTrack so projects round-trip
across both GUIs. Labels/tooltips reuse the LANG_ strings already added in #588.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Debian AppStream generator flagged both webhttrack desktop entries as
no-metainfo: with no MetaInfo file, the catalog entry is synthesized from
the .desktop file and the package description, which is deprecated and risks
the app being dropped from the metadata catalog.
Add com.httrack.WebHTTrack.metainfo.xml (installed to share/metainfo) for the
main app, launching WebHTTrack.desktop. Mark the secondary "Browse Mirrored
Websites" launcher with X-AppStream-Ignore=true so it doesn't produce a
duplicate, metadata-less catalog entry.
Validated with appstreamcli validate and desktop-file-validate.
Signed-off-by: Xavier Roche <roche@httrack.com>
* closes:#53
Also fixed HTML-escaping issues inside webhttrack
Rationale: The webhttrack script made the wrong assumption that once the "browse" command returned, it meant the user killed the navigation window, and it had to kill the server itself. However, modern browsers tend to "attach" to an existing session (creating a new tab, for example, within an existing window), causing the browsing command to return immediately, thus causing the server to be killed immediately by the webhttrack script. I have rewritten the logic behind, and now the server is able to kill himself if the parent script dies, AND if the browsing client did not make any activity for two minutes. The "activity" can be any browser/refreshed page, or the internal "ping" iframe (which pings the server every 30 seconds). With this model, we *should* be compatible with old browsers, and modern ones.
