66 Commits

Author SHA1 Message Date
Xavier Roche
46294200ed Release 3.49.15 (#883)
Version bump to 3.49.15 in the three spots that carry it (configure.ac, htsglobal.h, version.rc), plus the release notes and the Debian changelog entry.

VERSION_INFO goes 3:6:0 to 3:7:0, soname still .so.3. The installed htsblk gained a field at its tail and lien_back embeds it by value, so lien_back.is_update and everything after it shifts 8 bytes. An offsetof probe against both tags confirms httrackp's own fields did not move. Same call as 3.49.14: HTTrackQt is the only consumer of these headers, so no libhttrack4 rename.

109 commits since 3.49.14, none of them under debian/, so the changelog entry is a plain new-upstream one with a note that httrack-doc grows about 1 MB from the new GUI screenshots.
2026-07-30 09:03:13 +00:00
Xavier Roche
3414596efe The hub lede reads like a changelog, and the rings are gone (#882)
The lede led with WARC output and the change report, which are niche features, because those happened to be the last things merged. What matters to someone arriving now is that HTTPS works, that proxies work, that a current page with responsive image sets and lazy loading captures properly, and that an update does not re-fetch the whole site.

The rings the 2007 pages carried behind their content panel come back, and now on every page rather than the handful that had them. The image has the light panel colour baked in, so dark mode drops it rather than showing a pale rectangle.

While in there: the wordmark is black on transparent and all but vanished against the dark field, which every page inherited along with the shared chrome. It is inverted in dark mode now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-30 08:18:52 +00:00
Xavier Roche
3c96b0f209 Three pages that should not be pages of their own (#880)
addurl.html was 116 words and five screenshots of a 2003 dialog, reached only from a single sentence in the guide. Its substance moves into that sentence: what the Add a URL button does that the address box cannot, including the browser-capture proxy, which lang.def confirms is still in the GUI. The stale screenshots go with it, and a fresh capture of that dialog belongs in the next screenshot walk rather than in a page of its own.

options.html had been an orphan since #661 gave it a legacy-list banner and repointed everything that used to link it. It becomes a stub pointing at the generated option reference and the command-line guide.

plug_330.html documents the callback interface of releases 3.30 to 3.40. Its title said so; now a banner does too, the way Fred Cohen's guide is labelled on the index, because someone arriving from a search engine reads the page and not the title.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-30 08:00:57 +00:00
Xavier Roche
f1ace40d47 Fifteen documentation pages are dead ends in a 2007 table layout (#879)
Nine of the twenty pages linked nowhere at all and seven had a single inbound link, so a reader landing on cache.html from a search engine got no navigation, no breadcrumb and no way back. Each page also carried its own copy of the same inline stylesheet inside a six-table shell with a 400px floor, which is why sixteen of them scrolled sideways on a phone.

They now share the guide's chrome: masthead, sidebar, footer, and where the headings allow it a generated "On this page" list. file:// has no server includes, so the navigation is real markup in every page, written by tools/doc-chrome.py and verified by CI; tools/doc-links.py resolves every relative link and fragment. Both refuse to pass vacuously, and both were checked against planted defects.

The prose is untouched. For twelve of the fifteen pages the content text is identical word for word; the three exceptions are cmdguide.html losing the contents list the sidebar now carries, contact.html re-encoded to UTF-8, and one font tag whose removal joined two words the browser already ran together. Each page also gains a real title and its own description, replacing the shared blurb whose keyword list still advertised Windows 95 and AIX 4.0.

Image zoom and sidebar highlighting move to doc.js so every page has them, leaving guide.js the platform switcher and the option filter. The guide renders identically at 1100px in light and dark, and thirteen behaviours pass unchanged before and after; at 390px one Proxy paragraph that used to overflow its container by 37px now wraps. Nothing scrolls sideways at 390px or 1100px, and no page throws a script error.

httrack.man.html keeps its own bare styling for now, since reskinning it belongs in the generator that writes it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-30 07:42:27 +00:00
Xavier Roche
6b9740dbc2 The documentation index is a 1998 bullet list, and the overview is one paragraph (#878)
The documentation index is the page WinHTTrack opens for generic Help and the one httrack.com links as "Documentation", and it presented twenty destinations as a flat bullet list with nothing to say which one a newcomer wants. It is now a task-grouped hub in the interface guide's idiom, with overview.html folded in as the lede and refreshed for 3.50, and Fred Cohen's guide under an Archive heading labelled as written for 3.10, keeping its URL.

The chrome the hub needs is split out of guide.css into doc.css; guide.css keeps the platform switcher, tab strip and option entries. guide.html renders pixel-identically before and after the split at 1100px and 390px, in light and dark.

overview.html, start.html and cmddoc.html become meta-refresh stubs, like the retired step pages. start.html was a window.open popup launcher and cmddoc.html has 404ed since #661 deleted it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-30 07:23:05 +00:00
Xavier Roche
3e480bf990 Test 142 asserts an Android screenshot that #874 renamed (#881)
Test 142 guards against a vacuous run by requiring every asset it types to exist. img/android_spider.png stopped existing when the Android help page was folded into the unified guide and its screenshots were renamed guide-droid-*.png: #874 and #876 were in flight at the same time, so neither run saw the other half, and master has failed this test on every platform since it landed.

The three references now point at guide-droid-opt-spider.png, a PNG in the same directory, so the check still tests what it was written to test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-30 07:07:45 +00:00
Xavier Roche
401323ff7e htsserver labels every PNG as image/gif, and serves JPEG as a download (#876)
* htsserver labels every PNG as image/gif, and serves JPEG as a download

The response header was picked per broad file class, so is_image() matched
both .gif and .png but always emitted Content-type: image/gif, and anything
outside the five hardcoded classes (.jpg, .svg, .ico, .xml) went out as
application/octet-stream, which browsers download instead of render.

Replace the per-class headers with an extension to type table, matched on the
real extension rather than on a substring of the path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the untested paths the content-type table can regress on

The first test named 9 of the 12 extensions and no superstring case, so a
lookup matching on a prefix, or one that dropped the htm or jpeg row, stayed
green. Add those fixtures, a raw-socket check that the header line is still
CRLF-framed, and the no-cache pair that only the HTML branch carries.

Also serve .xml as application/xml (RFC 7303 deprecates text/xml) and add
webp and avif, the two image types a browsed mirror most often needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 16:08:02 +00:00
Xavier Roche
a21f84ee37 The GUI documentation is eighteen thin pages, one of them for a version nobody can see screenshots of (#874)
* One guide for WinHTTrack, WebHTTrack and Android, instead of 18 thin pages

The GUI documentation was 18 files whose bodies were mostly a copy-pasted style
block, illustrated with 1998-era screenshots of WinHTTrack, with the Android app
documented separately and WebHTTrack not shown at all. html/guide.html replaces
them with a single page: one walkthrough and one option reference, and a
switcher that swaps the screenshots and the platform-specific notes between the
three versions. Each option now also names its command-line equivalent.

The screenshots are the three capture sets from httrack-works, prepared by
tools/doc-images.py. Deep links carry the platform, so guide.html#win/opt-limits
opens the Limits section already showing the Windows dialog; the WebHTTrack
option pages use that form. The old filenames stay as redirect stubs for the
forum links that still point at them.

Scripting is an enhancement throughout: without it the page shows every
platform's text and screenshots rather than hiding any of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Document the options the old pages never had

A coverage probe against the WebHTTrack form controls found the reference
inherited the old doc's blind spots: WARC, the CDXJ index and WACZ, --changes,
sitemap seeding, cookies-file, the urlhack opt-outs, strip-query, the proxy type
selector and the random inter-file pause were all exposed in the UI and absent
here. Labels and descriptions taken from lang.def, CLI flags from htsalias.c.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 16:05:06 +00:00
Xavier Roche
8d51e11c6e WebHTTrack has no documentation screenshots, and no way to produce them (#873)
Add tools/screenshot-walk.py, which starts htsserver, drives its pages in
headless Chromium and captures one PNG per documentation screen, plus a
manual screenshots workflow that runs it on a runner. The Android and
Windows front ends already have their equivalent.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 16:54:19 +02:00
Xavier Roche
51b0142599 Android users have to leave the help page to find out what any option does (#872)
* Refresh the Android help page from current app screenshots

The ten screenshots dated from an older build and covered only two of the
eleven option tabs. Recaptured all of them from app versionCode 85 (engine
3.49.14) and added Build, Spider and Log/Index/Cache, whose contents the
one-line summaries could not convey.

The new shots also correct three claims: the proxy tab now picks a protocol
(HTTP, SOCKS5, HTTP CONNECT tunnel), Log/Index/Cache gained WARC output and
the change report, and Base path opens a chooser rather than being fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Document every option tab on the Android page

Android users cannot easily flip between help pages, so the page now carries
its own option reference: all eleven tabs, each with its screenshot and every
setting on it, replacing the pointer at the desktop reference.

Six settings the desktop pages never covered are documented here for the
first time: keep-alive, the inter-file pause, the three URL-hack opt-outs on
Links, and Strip query keys on Experts Only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 14:42:32 +02:00
dependabot[bot]
dc28c80ad8 Bump src/coucal from ab59c6a to 93ec411 (#871)
Bumps [src/coucal](https://github.com/xroche/coucal) from `ab59c6a` to `93ec411`.
- [Commits](ab59c6a7a0...93ec411c55)

---
updated-dependencies:
- dependency-name: src/coucal
  dependency-version: 93ec411c55667050965c7bb8826c22adc3521dec
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 12:50:54 +00:00
Xavier Roche
21de744be6 CI: move the macOS legs off the deprecated macos-14 image (#867)
GitHub marks macos-14 deprecated, so both macOS jobs are on an image headed
for brownouts and removal. macos-15 is the same arm64 host and every path in
those jobs is brew --prefix-derived, so nothing else moves.

Also point Dependabot at git submodules: src/coucal is a hard build dependency
that only advances when someone remembers to look.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 09:19:57 +00:00
Xavier Roche
3e4b131456 tests: share one port-discovery helper instead of 17 fragile copies (#869)
local-crawl.sh waits 30s for local-server.py's "PORT <n>" line and matches it
anywhere, because a cold Python start under a parallel make check lags past a
second and a warning merged via 2>&1 can precede the line. Seventeen tests that
launch the server directly carried an older copy of that loop: 5s, and only the
first line of the log. macos-15 exposed it, failing 15 of them where macos-14
passed on margin, but nothing about the bug is macOS-specific.

Move the proven idiom into testlib.sh next to stop_server and call it from all
seventeen, which also gets them the log dump on failure that local-crawl.sh has
and the inline copies did not -- the reason the CI failure said only "could not
discover server port" with no way to tell a slow start from a dead server.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 08:07:08 +00:00
Xavier Roche
3dac180aac CI: build once with the deprecated OpenSSL API hidden (#868)
OpenSSL 4.0 removes what 3.x deprecated. The engine is already clean against
that set, but nothing holds it there: a deprecated call would build fine on
every current leg and only surface when a distro moves to 4.0.

Build with -DOPENSSL_NO_DEPRECATED, which tracks whatever OpenSSL the runner
has rather than pinning a level the headers may reject. Compile-only; the
header contract is what is under test and the matrix already runs the suite.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 07:25:19 +00:00
Xavier Roche
21d9289945 No way to exercise the crash handlers without waiting for a real crash (#865)
* Add a hidden -#c option that crashes on purpose, for crash-handler testing

The crash handlers had no way to be exercised: httrack's own fatal-signal
handler, WinHTTrack's, and the Android app's mirror-recovery path could only
be tested by waiting for a real bug. -#c[=KIND] faults deliberately, in
segv, abort, trap or stack flavours, announcing on stderr that the crash was
asked for so a log never reads as a genuine failure.

Undocumented and unlisted, but always compiled in: the point is to test the
shipped artifact (the APK, the installer, the distro package), which a
build-time flag would exclude.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert a backtrace portably: macOS has no backtrace() at all

The frame check matched glibc's "[0xaddr]" form, which macOS never prints:
USES_BACKTRACE is gated on __linux, so every other OS gets the "No stack
trace available" notice instead. Accept either, and also require the
handler's closing line so the assertion still proves it ran to the end.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 06:49:16 +00:00
Xavier Roche
c4879cd25c The webhttrack GUI cannot reach --warc-cdx, --wacz or --warc-max-size (#862)
* The webhttrack GUI cannot reach --warc-cdx, --wacz or --warc-max-size

PR #672 exposed --warc and --warc-file on the "Log, index, cache" tab; the
three WARC flags added after it never got a control, so a CDXJ index or a
WACZ package is unreachable from the GUI. Add the two checkboxes and the
size field, and translate the six new lang.def strings into all 30
languages instead of letting them fall back to English.

An unknown ${LANG_*} renders as the empty string, so a key missing from
lang.def or from one lang/*.txt blanks a label instead of failing: test 140
now asserts every key the templates use exists everywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Move the lang completeness check into 62_lang-integrity, its owner

62_lang-integrity.test already owns lang.def/lang-file consistency; it
checked that no msgid drifted, but not that every msgid is translated
everywhere, which is the gap that let the WARC options ship untranslated.
Add the completeness pass there and drop the separate 140 test, which
reimplemented the same parser in Python and was scoped narrowly enough to
be blind to the 24 msgids already missing from 26 files. Those are waived
by name against #863, and a waiver that stops being needed now fails.

Also check every ${LANG_*} the templates interpolate resolves, in any
wrapper form: the ${html:html:...} and ${fexist:...:...} spellings were
outside the dropped test's regex.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 06:28:09 +00:00
Xavier Roche
cd3051ca1a url_savename never clamps the final path segment, only the whole path (#861)
* url_savename never clamps the final path segment, only the whole path

Inside url_savename's over-ceiling shortening, every directory segment is cut
to MAX_SEG_LEN but the name itself is copied under the whole-path budget alone.
A single component could therefore run to ~226 characters on Windows while its
parent directories were held to 48, and once the directories had eaten the
budget the cut landed mid-name and took the extension with it, leaving a page
saved with no .html at all.

Clamp the name like any other segment, and reserve a plain extension across the
cut the way #623 already reserves the ".delayed" marker.

75_engine-longpath-posix's Windows branch asserted the clamp all along; it had
never run until #847 broadened the CI glob, so this also drops the expected_skip
that filed the failure as #852.

Closes #852

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix 01_engine-savename's Windows expectation for the clamped name

The Windows arm pinned the pre-fix output: 210 characters of name and no
extension at all. It now takes the 48-char segment clamp with its ".html"
intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 22:47:41 +00:00
Xavier Roche
16bbf2ad27 A query-string character reference the page charset cannot represent is left unescaped, changing the query's parse (#856)
* Percent-escape a query-string character reference the page charset cannot represent

On a page with no declared charset (iso-8859-1 fallback), a reference like
&euro; was left as literal source text in the query string: its '&' and '#'
then act as delimiters, so the origin parsed more parameters than the document
expressed. The URL Standard writes such a code point %26%23<decimal>%3B.

Adds hts_unescapeEntitiesWithCharsetSpecial() with an opt-in
UNESCAPE_ENTITIES_URL_QUERY flag, set only by the query-string call site; the
plain decoder keeps its text contract and its in-place guarantee. The query is
now decoded out-of-place, since the escape grows the string.

Closes #854

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Detect an unrepresentable code point with the encoder, not a round trip

The round-trip comparison escaped code points the charset does represent:
hts_convertStringToUTF8() short-circuits on an ASCII string and hands the bytes
back undecoded, so every reference on ISO-2022-JP, UTF-7 or HZ compared unequal,
and glibc's CJK tables are not injective (cp932 U+301C came back as U+FF5E).

On POSIX iconv already fails outright on a code point it cannot encode, so the
NULL return is the answer. Windows substitutes instead of failing, and reports
it through WideCharToMultiByte's lpUsedDefaultChar, which we now pass;
hts_convertStringFromUTF8Strict() turns that into a NULL. Test 139 grows a
declared shift_jis and a declared iso-2022-jp page, which the two charset cases
it had could not have caught.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the ISO-2022-JP query per field, and let the two cp932 tables differ

Windows maps 0x8160 to U+FF5E rather than U+301C and reports the substitution,
so 〜 is genuinely unencodable there and the escape is right: accept either
encoding for the shift_jis page. ISO-2022-JP's trailing shift back to ASCII also
differs between iconv and Windows, so that page is asserted field by field, which
is what the round-trip check would have broken anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Let Windows substitute the euro sign on a stateful codepage

WideCharToMultiByte refuses lpUsedDefaultChar on ISO-2022-JP, so the
substitution cannot be seen and € goes out as '?' there, as it did before this
branch. It splits no query, and the page is in the test for the references the
charset does represent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 00:14:19 +02:00
Xavier Roche
ed0906a4d4 Windows CI's engine-test glob and expected_skips collide the same way tests-list.mk did (#847)
* Broaden Windows CI's engine-test glob and skip-set compare (#844)

The Windows engine-test loop named individual tests (01_engine-*.test plus a
hand-maintained tail) because its glob missed NNN_engine-*.test, so a new
test landed with zero Windows coverage until someone remembered to add it by
hand; two branches doing that at once collided on the same line. Broaden the
glob to *_engine-*.test, matching the *_local-*.test pattern already used
next to it, so a stray test file gets picked up automatically instead of
silently skipped.

That newly covers 66_engine-port80-strip, 67_engine-delayed-truncate,
75_engine-longpath-posix, 80_engine-crash-symbolize and
104_engine-warc-longurl, none of which had ever run on a Windows runner.
80_engine-crash-symbolize exits 77 there by design (no backtrace()), which
would have tripped the old exact-string expected_skips gate and turned the
leg red. Switch that gate to a sorted-set compare (one name per line, so an
append no longer shares a line either) and add the new skip; a mismatch now
prints a diff naming exactly which test's skip state changed instead of a
raw string dump.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Generalize the last two hand-pinned test names in the Windows CI loop

13_crawl_proxy_https.test and 60_crawl-log-salvage.test matched no
glob, so a renumber or a same-named future test would silently drop
Windows coverage again. Tightly globbed instead of using *_crawl*,
which would also sweep in the live-network crawl-*/crawl_https tests
this job never runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* kill_tree's Windows path falls through into a POSIX process-group kill

test-timeout.sh and run_with_timeout skip set -m on Windows (nothing there can
signal a group), so no target of kill_tree ever heads its own process group.
kill_tree's taskkill branch already reaps the Windows tree, but had no return,
so it always fell through to kill -9 -"$pid": on Windows that treats $pid as a
process-group id it never is, and can hit whatever real group its number
collides with under MSYS's own pid churn -- including the harness's own,
which is how PR #847's CI run died silently right after 102_local-ftp-refetch
finished, mid-cleanup, with no FAIL line and a truncated artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 22:07:53 +00:00
Xavier Roche
b3cba2ba5a A chunked response cut at a chunk boundary is stored and cached as complete (#851)
* A chunked response cut at a chunk boundary is stored and cached as complete

back_finalize()'s incomplete-transfer guard is gated on r.totalsize >= 0, so it
only fires for a response that declared a Content-Length. A chunked response
keeps totalsize == -1 until the chunk headers add up, and the sum always equals
what arrived, so a stream that ended before its terminating zero-length chunk
took the success branch: the partial body was mirrored, cached, and on an
--update replaced the previous good copy.

The terminating chunk is part of the framing, so its absence makes the body
truncated by definition. Track it through chunk_blocksize, which only reaches
-1 once that header is parsed, and refuse the transfer in the two places the
short-Content-Length path already covers: back_finalize skips the cache, and
back_wait fails the transfer so the engine retries and leaves the mirrored copy
alone.

An HTTP/1.0 identity body delimited only by connection close has no framing to
check, and is deliberately left alone.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: cache assertions, -%B coverage, disk-write limit

Adds the cache half of the spec the test never asserted (--cache-found /
--cache-not-found in local-crawl.sh, reading hts-cache/new.zip), a -%B leg
covering opt->tolerant on a chunked body, and a direct-to-disk .bin resource.
Gates the back_wait block on r.statuscode > 0 like its back_finalize
counterpart, and drops the chunk_size == 0 hunk: a negative chunk-size header
is already refused by the short-body guard and the Invalid chunk path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore the chunk_size == 0 sentinel guard, and pin it

A chunk-size line of 80000000 lands in an int as INT_MIN via sscanf("%x"):
a bare else claimed it as the terminating chunk while totalsize went negative,
so neither the framing check nor the short-body check fired and an --update
deleted the good copy and cached a 0-byte 200. Invalid chunk cannot cover this,
being set after back_finalize has already stored the response.

The update pass of chunktrunc/hostile.html now serves that header, and
chunktrunc/reset.bin aborts its body with an RST so a read error keeps its own
diagnostic instead of the framing check relabelling it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the parsed chunk size before it reaches the in-RAM realloc

sscanf("%x") into an int lands a chunk-size line of 80000000 on INT_MIN, which
drives totalsize negative past the INT32_MAX guard and sign-extends into a 16 EB
realloc (#860, pre-existing on master; the new #840 test is what reaches it).
Parse the field with strtoull and drop anything an int cannot hold, so the
existing arithmetic and the in-RAM cap keep working on a value known to be in
[0, INT32_MAX].

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:58:27 +02:00
Xavier Roche
90b98e740e A URL can still be saved onto the engine's own temporary files (#853)
* A URL can still be saved onto the engine's own temporary files

The re-fetch backup and the content-coding temporaries built their names
inside the mirror, so a site serving the matching path had its mirrored
copy used as scratch and then deleted. #818 escaped hts-cache and hts-tmp
as reserved path components, but three shapes still got through (#842): a
trailing space that cleanEndingSpaceOrDot() stripped back off after the
check, and the first path component, which the table could not see because
it anchors on a leading '/' that url_savename has already removed. The
latter covers a single-label hostname too, and left the DOS device names
equally exposed in that position.

The temporaries now live in ~hts-tmp/. url_savename maps '~' to '_' in
every name it builds, so nothing a site can serve resolves inside it
whatever the escape does; the escape is still fixed, as defence for
hts-cache/ itself. The frozen-slot spool (<save>.tmp) shared the
collision, plus a heap buffer sized from url_sav that the path_html branch
overran, and moves to the same helper. The .delayed placeholder does not:
it goes through url_savename's collision detection, so a competing link
gets a suffix rather than the file.

Closes #774
Closes #842

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Scope the reserved-name escape and fix the review blockers

Restrict the new first-component match so a hostname label is not rewritten:
only a trailing run cleanEndingSpaceOrDot() strips may end the name there,
so aux.example.com keeps its name while a single-label hts-tmp host and a
path-first hts-cache/ are still escaped. Renaming a host directory would
have moved an existing mirror out from under --update --purge-old.

Drop the frozen-slot spool rewrite: it carried a pre-existing heap overflow
that belongs in its own change. Clear back->tmpfile when structcheck() fails
in the named branch, as the unnamed one does. Test 132 now plants a leftover
at the exact temporary path, so both a reverted HTS_TMPDIR and a no-op
back_refetch_backup fail it; 131 pins the hostname labels.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:44:15 +02:00
Xavier Roche
836c630f34 The frozen-slot spool name is written into a buffer sized from a different string (#858)
* The frozen-slot spool name is written into a buffer sized from a different string

back_cleanup_background() sized the temporary's name buffer from url_sav and
then, under -p0, wrote a path_html-derived name into it. Build the name in a
bounded buffer and duplicate it, so neither form can overrun.

Closes #857

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the spool actually ran, and correct the overflow's blast radius

The -p0 name escapes on a plain build too: _FORTIFY_SOURCE=3 catches the
same write in __sprintf_chk, so the abort is not sanitizer-only. Test 138
now runs under -Z and requires the engine log to show a slot spooled and no
serialize error, so a build with neither fortify nor ASan still proves the
branch was taken. tmpname gains room for the ".tmp" a full-length save name
appends, so no input is refused that the old exact-sized buffer accepted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:43:21 +02:00
Xavier Roche
f0cd4b5089 A 304 revisit names no peer IP, and an engine-forced not-modified is archived as one the server never sent (#849)
* A 304 during --update leaks the whole previous htsblk

back_wait() handles a 304 by replacing the response struct with the cache
entry, carrying only the socket and keep-alive members across via
back_connxfr(). The struct assignment drops every owned pointer the live
response still held without freeing any of them: the 8 KB header buffer on
every update, plus the two WARC header stashes when --warc-file is on. An
update over a 10k-page site drops roughly 80 MB in one run.

back_clear_entry() already knew how to tear those down, so the frees move into
a helper that both it and the 304 path call.

The new test runs the two-pass mini304 crawl with LeakSanitizer on, which the
sanitized CI job otherwise disables. The fresh first pass is the control: it
has no cache entry to read back and is clean either way. The update pass
reports 16 KB in 2 objects on master, one per unchanged URL, and nothing with
the fix.

Closes #782

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the test header

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: reuse deleteaddr(), and cover the WARC limb

back_free_response() was reimplementing deleteaddr(), which already frees adr
and headers and NULLs both; call it instead so the two cannot drift.

Test 114 never passed --warc-file, so the warc_free_request() limb ran with
both pointers NULL on every path it exercised and deleting it kept the test
green. A third pass turns the archive on, and it now fails with the 835 and 238
byte stashes when that call goes away.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Register the new leak test as an expected Windows skip

The Win32/x64 job pins the exact set of tests allowed to skip, so an
all-skipped suite cannot report green. 114_local-update-304-leak needs a
LeakSanitizer build and MSVC has no equivalent, so it skips there and tripped
the gate with fail=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A 304 revisit archives none of the exchange it stands for

The cache entry overwrites the whole htsblk in back_wait(), taking the
stashed request and response headers with it, so the revisit record gets
a synthesized "HTTP/1.1 200 OK" block and no request record. Both blocks
now move across the swap, but only for a real 304: the engine also forces
HTTP_NOT_MODIFIED by itself, and those have no 304 to carry. Since a 304
declares no Content-Type, warc_write_transaction() takes the resource type
from the caller so revisit CDXJ lines keep their mime.

cache_read_including_broken() returns the entry's response struct after
back_clear_entry() has freed it; adr, headers and location come back NULL
now, as the normal cache_readex() path already returns them.

Closes #826

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Do not expand an empty bodyargs array under macOS bash 3.2

Test 122 asks local-crawl.sh only for the revisit checks, so it is the
first caller to leave bodyargs empty. Bash before 4.4 calls that unbound
under set -u, which passes on Linux and fails the macOS leg.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore the opt argument to hts_rename_over call sites

Lost when the master merge took this file wholesale; #816 gave the function an
httrackp parameter and the three call sites here reverted to the old form.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A 304 revisit names no peer IP, and an engine-forced not-modified is archived as one the server never sent

back_connxfr() moves the socket and keep-alive members across the cache-entry
swap in the 304 path of back_wait(), but not the peer address, so every
revisit record came out with no WARC-IP-Address (#838). Carry it across the
same swap.

Separately, several engine hacks (the same-size heuristic, the #176
error-ignore-on-update path, ...) force HTTP_NOT_MODIFIED themselves when the
server actually said something else. That statuscode flows into the same 304
branch, and warc_write_backtransaction() turned it into a revisit record whose
WARC-Profile claims a server-not-modified 304 that was never exchanged (#839).
The fix reuses the existing server_sent_304 signal (already used to gate the
request/response header carry-over): its persisted proxy is
warc_resphdr != NULL, present only when the swap actually moved real 304
headers across. When it's absent, no server exchange happened this run to
archive, and there's no freshly re-fetched body either to dedup against a
same-run predecessor, so the fix skips the record entirely rather than
writing an identical-payload-digest revisit with nothing behind it to refer
to.

tests/134 and tests/135 extend the mini304/errmask fixtures into a two-pass
--warc-file crawl and assert against tests/warc-validate.py, which gained
--expect-ip and --no-record-for for these checks. Both reproduce their bug
when run against the pre-fix code.

Closes #838
Closes #839

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix review findings: an engine-forced not-modified must still count as unbacked

Writing no WARC record at all for an engine-forced not-modified (the #839 fix
in the previous commit) dropped w->unbacked_revisits accounting along with it,
so an all-forced pass wrongly looked fully backed: warc_commit's swap guard let
it replace the previous archive with an (almost) empty one, reopening #777's
exact shape, and warc_cdx_flush then left a stale .cdx pointing into the
truncated file. Fix: emit a self-referencing identical-payload-digest revisit
instead of nothing, still counted as unbacked, so the guard keeps protecting
the previous archive exactly as it does for a real 304.

Classifying a notmodified htsblk by warc_resphdr's presence was too broad: it
also matched back_add()'s cache-priority branch, which never sent a request at
all and is unrelated to the back_wait() hacks #839 targets. Replaced it with an
explicit hts_boolean, warc_forced_notmodified, set only inside back_wait()'s
304-swap block from the already-computed server_sent_304, leaving every other
notmodified path (including back_add()'s) at its default and thus at its prior
behavior.

tests/134 now asserts the exact peer IP instead of mere presence, since a
zeroed-but-AF_INET address also satisfies a non-empty check. tests/135 gained
a second scenario: an all-forced-not-modified rerun against the same archive
name, which is the shape that lets the swap guard's bug actually destroy the
previous archive; the mixed-fixture scenario alone couldn't see it, since a
real 304 alongside it already keeps the guard's counter above zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the cache-priority path and gate the digest revisit on OpenSSL

warc_forced_notmodified had exactly one assignment site, inside back_wait()'s
304-swap block. back_add()'s cache-priority branch (-C1, a lock-file resume,
or opt->state.stop) sets notmodified too but never sent a request, so it fell
through to the zero default and got classified as a genuine 304, emitting a
server-not-modified revisit around a synthesized "HTTP/1.1 200 OK" block. Set
the flag there as well: no request there ever means no server 304 to claim.

The identical-payload-digest revisit for an engine-forced not-modified also
assumed a digest was available, but payload_digest_b32() returns nothing
without OpenSSL. Gate the profile on have_pdig; write no record when it's
absent, since there's neither a digest to point a revisit at nor a real
exchange to record as a response, keeping w->unbacked_revisits incremented
either way so the archive-protection guard still holds.

tests/136 drives the cache-priority path with -C1 and checks the same
WARC-Profile assertion as tests/135. Both new branches (135 and 136) split on
HTTPS_SUPPORT so the no-OpenSSL CI leg exercises the no-digest path instead of
skipping it; 135's archive-kept scenario drops --wacz for the same reason, a
package needs OpenSSL for its SHA-256 digests and would otherwise skip the one
thing it exists to catch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* An empty body looks digest-less too, and the profile tests never checked the real 304

has_payload requires body_len > 0, so a genuinely empty body (a real,
zero-byte file) fell into the same "no digest" branch as a build without
OpenSSL and got silently dropped from the archive instead of a proper
identical-payload-digest revisit. A zero-length body has a well-defined
digest (sha1 of nothing); compute it directly for that case instead of
reusing has_payload's gate.

tests/135 asserted mini304's pages were revisits but never which profile, so
forcing warc_forced_notmodified true unconditionally (a regression that would
mislabel or suppress a real 304) still passed. Added the server-not-modified
assertion for both mini304 pages, and a matching zero-length fixture
(errmask/empty.dat) plus its identical-payload-digest assertion. tests/136
gained a body-hex check on the fresh archive: its update-pass checks only
assert absence, so a warc_write_transaction stub that writes nothing at all
went undetected on the no-openssl leg.

The unbacked-revisits counter still increments even when nothing gets
written (load-bearing: it's what keeps the swap guard from letting an
all-unchanged pass overwrite a good archive with an empty one), so the
"this pass revisited N URL(s)" log line no longer claims those URLs got a
named body; it now says the archive doesn't hold their current content,
true whether a record was written or not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 21:38:58 +00:00
Xavier Roche
22e9520636 A charset-less document double-encodes the title lifted for the local index (#848)
* A charset-less document double-encodes the title lifted for the local index

A page declaring no charset falls back to iso-8859-1, which the parser then
applies to text that may already be UTF-8. The link path has guarded against
that since #180; the title harvested for the generated top-level index had no
such guard, so "café" reached the index as "café". Apply the same guard.

Test 137 crawls a charset-less UTF-8 page, a charset-less Latin-1 page and a
page declaring utf-8; the Latin-1 case is the control that fails if the
conversion is deleted rather than guarded.

Closes #833

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Scope the charset-less escape hatch to documents that declare nothing

The first cut skipped the title conversion whenever the bytes already parsed
as UTF-8, on every page. A declared charset can produce those bytes too: a
gb2312 title of 图片 is CD BC C6 AC, valid UTF-8 by accident, and came out of
the index as raw GBK mojibake.

Carry the declared/guessed distinction instead. htscore.c leaves page_charset_
empty when the document named none, and the parser resolves that to the
iso-8859-1 guess it already applied to links; only the guess now steps aside
for text that is already UTF-8.

Test 137 gains a declared-gb2312 and a declared-iso-8859-1 page whose titles
are valid UTF-8 as bytes, which is what the first gate regressed.

Refs #833

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 20:17:07 +00:00
Xavier Roche
a0a2bee019 Four more time helpers still read libc's shared gmtime/localtime static (#850)
PR #794 moved four gmtime() call sites onto a reentrant hts_gmtime()
helper, but a handful of localtime()/gmtime() sites survived: the
progress/log timestamps in htsback.c and htscore.c, ProxyTrack's
timezone lookup in proxy/store.c (which also dereferenced localtime()'s
result with no NULL check), and the RFC822 date helpers in htslib.c.
glibc's gmtime/localtime buffer is process-wide, not per-thread, so two
of the engine's worker threads racing on it can each read back the
other's broken-down time.

Adds hts_localtime() next to hts_gmtime() and converts every remaining
call site.

Separately, time_gmt_rfc822() and get_filetime_rfc822() fell back to
localtime() whenever gmtime() failed, then handed the result to
time_rfc822(), which always appends "GMT" to the formatted string.
Local time labelled GMT is wrong regardless of which thread wins the
race, so both now fail (empty string / return 0) instead of guessing.
gmtime() essentially never fails on a valid time_t, so this path is not
reachable in practice; the new test guards it at the source level
instead of trying to trigger the failure at runtime.

tests/133_engine-reentrant-time.test drives a new -#test=localtime
self-test: a concurrent hts_localtime() stress test against a fixed
reference table (mirroring #794's hts_gmtime test, TZ pinned to a
non-UTC offset so a local/UTC mix-up cannot hide behind a UTC CI
runner), a get_filetime_rfc822() GMT round-trip check, and a source
guard confirming time_gmt_rfc822()/get_filetime_rfc822() never
reference localtime() again. Verified red on two independent mutants:
reverting hts_localtime() to a raw localtime()-and-copy corrupted
17549/400000 concurrent conversions, and reintroducing the local-time
fallback tripped the source guard.

Closes #806

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:53:40 +00:00
Xavier Roche
457682b5df A PROPFIND on an exact cache entry crashes proxytrack (#829)
* A PROPFIND on an exact cache entry crashes proxytrack

PT_Enumerate() reports a folder's default document as a zero-length name,
and the WebDAV listing loop read thisUrl[thisUrlLen - 1] on it, four
gigabytes past the string. One unauthenticated request took the whole
listener down.

Closes #828

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the new skip on Windows and tighten the listing assertions

The Windows job runs *_local-*.test and compares the skip list against an
exact string, so an unpinned skip fails the leg with fail=0, which reads
like a flake. Assert the href and the response count too: displayname
alone comes from the href's trailing component, so it cannot see a wrong
path above it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:52:26 +00:00
Xavier Roche
91d12d4040 The re-fetch backup can destroy the copy it was taken to protect (#818)
* The .bak re-fetch backup destroys a mirrored file of the same name

The re-fetch backup, the .z content-coding spool and its .u decode
target were all named by appending an extension to the local save name,
which is a name the mirror can hold. A site serving <path>.bak had its
copy renamed over as the backup and then unlinked on commit, and the run
reported success. Build them in an hts-tmp directory beside the file
instead, and remove it once the last slot sharing it is done.

Closes #774

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cache repair deletes the old cache before a rename it never checks

Both zip-repair paths removed the damaged cache, moved repair.zip onto
its name without looking at the result, and announced a successful
recovery either way. A refused move left the entries under a name
nothing reads and no cache at all. Go through hts_rename_over(), which
keeps the destination when it cannot replace it, and report the failure
instead of claiming success.

Closes #786

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A failed filecreate() on an unknown-length re-fetch commits the backup away

The re-fetch backup was dropped on commit without checking that the new
copy exists. filecreate() can fail after the backup was taken (EMFILE,
ENOSPC, EROFS), and back_finalize()'s incomplete-transfer guard is gated
on a known Content-Length, so a chunked response went straight to the
commit and lost both copies. Restore the backup instead when there is
nothing to commit to.

Closes #775

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* hts_rename_over() can lose its destination when the retried rename fails

The unlink-then-rename fallback leaves nothing in place of dst between
the unlink and the retry, so a retry that fails too loses it. Park dst
under a free scratch name instead, drop it once the move succeeded, and
put it back otherwise.

Closes #790

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the cache-repair test Windows coverage

The Windows job iterates a fixed glob, so a 106_engine-* test never ran
there; name it in the list. The move it exercises is the one whose
failure mode is Windows-specific.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the re-fetch backup test Windows coverage

The Windows job iterates a fixed glob, so a 108_engine-* test never ran
there. The rename semantics this drives are exactly what differs on the
CRT, which reserves EACCES for a locked source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: check the restore, probe in UTF-8, state the honest guarantee

The move back out of the parked name was unchecked, so a retry that
failed for a reason that still applied left dst absent with the content
orphaned under a name nothing reported. Check it, retry once, and name
the parked copy in the log; hts_rename_over() takes an httrackp for that.

The aside probe used fexist(), which is not UTF-8 and consults the ANSI
codepage on Windows while the renames beside it are wide. It also reads
a directory as a free name, so the park now skips a name whose rename
refuses rather than giving up on it.

The header claimed a failure leaves dst as it was, which the crash
window between the two renames does not give.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Reserve hts-tmp and hts-cache so no URL can name a temporary

Moving the temporaries into an hts-tmp directory only pushed the
collision one level down: hts-tmp is a path segment a URL can name, so a
site serving /d/hts-tmp/a.bin.bak still landed on the backup path for
/d/a.bin and was consumed. url_savename now escapes hts-tmp and
hts-cache as whole path components, the way it already escapes the DOS
device names, which also stops a URL from naming hts-cache/new.zip.

The three tests now drive the hostile shape rather than the pre-fix
name: a savename assertion on the escape, a crawl serving
/bakname/hts-tmp/a.bin.bak, and a selftest pinning the temporary's name.

Also: retry the backup once when another slot removed the shared
directory between structcheck and the rename; report a commit that had
to restore, so the caller does not cache the new validators against the
old body; drop the coded body on the too-long branch as its sibling
does; and scope the leftover scan to plants inside hts-tmp only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The aside fallback parked a directory that stood in the way

Windows refuses every rename onto an existing target, so the fallback is
production code there rather than the rare path it is on POSIX. A
directory at the destination was renamed aside like a file, the move
then succeeded, and UNLINK could not drop the parked directory, so the
call reported success where master had reported failure and left an
orphan behind. 101_local-update-stale-bak plants exactly that shape and
caught it on both Windows legs.

Park a regular file only. A directory in the way is refused, as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:25:14 +00:00
Xavier Roche
5c0918f328 A 304 revisit archives none of the exchange it stands for (#837)
* A 304 during --update leaks the whole previous htsblk

back_wait() handles a 304 by replacing the response struct with the cache
entry, carrying only the socket and keep-alive members across via
back_connxfr(). The struct assignment drops every owned pointer the live
response still held without freeing any of them: the 8 KB header buffer on
every update, plus the two WARC header stashes when --warc-file is on. An
update over a 10k-page site drops roughly 80 MB in one run.

back_clear_entry() already knew how to tear those down, so the frees move into
a helper that both it and the 304 path call.

The new test runs the two-pass mini304 crawl with LeakSanitizer on, which the
sanitized CI job otherwise disables. The fresh first pass is the control: it
has no cache entry to read back and is clean either way. The update pass
reports 16 KB in 2 objects on master, one per unchanged URL, and nothing with
the fix.

Closes #782

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the test header

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: reuse deleteaddr(), and cover the WARC limb

back_free_response() was reimplementing deleteaddr(), which already frees adr
and headers and NULLs both; call it instead so the two cannot drift.

Test 114 never passed --warc-file, so the warc_free_request() limb ran with
both pointers NULL on every path it exercised and deleting it kept the test
green. A third pass turns the archive on, and it now fails with the 835 and 238
byte stashes when that call goes away.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Register the new leak test as an expected Windows skip

The Win32/x64 job pins the exact set of tests allowed to skip, so an
all-skipped suite cannot report green. 114_local-update-304-leak needs a
LeakSanitizer build and MSVC has no equivalent, so it skips there and tripped
the gate with fail=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A 304 revisit archives none of the exchange it stands for

The cache entry overwrites the whole htsblk in back_wait(), taking the
stashed request and response headers with it, so the revisit record gets
a synthesized "HTTP/1.1 200 OK" block and no request record. Both blocks
now move across the swap, but only for a real 304: the engine also forces
HTTP_NOT_MODIFIED by itself, and those have no 304 to carry. Since a 304
declares no Content-Type, warc_write_transaction() takes the resource type
from the caller so revisit CDXJ lines keep their mime.

cache_read_including_broken() returns the entry's response struct after
back_clear_entry() has freed it; adr, headers and location come back NULL
now, as the normal cache_readex() path already returns them.

Closes #826

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Do not expand an empty bodyargs array under macOS bash 3.2

Test 122 asks local-crawl.sh only for the revisit checks, so it is the
first caller to leave bodyargs empty. Bash before 4.4 calls that unbound
under set -u, which passes on Linux and fails the macOS leg.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore the opt argument to hts_rename_over call sites

Lost when the master merge took this file wholesale; #816 gave the function an
httrackp parameter and the three call sites here reverted to the old form.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:07:05 +00:00
Xavier Roche
be9d9927a6 ProxyTrack's .ndx reader trusted an unparsed offset and the length of a path (#835)
* ProxyTrack's .ndx reader trusted an unparsed offset and the length of a path

An .ndx entry whose position field does not parse left "pos" untouched, so it
kept the offset of the entry before it, and the first entry read whatever the
stack held. The cache then resolves that URL to a different record, or to
nothing at all.

The base path and the .dat/.ndx filenames were built with strncat calls
bounded by the source path rather than by their 1024-byte destinations, so a
cache named on the command line with a long enough path overran them: ASan
reports a heap overflow on the filenames, UBSan an out-of-bounds index on the
base path right after. The zip reader carried its own copy of the same base
path code, reachable once the archive opens, and both now share one bounded
helper. A path that cannot fit is dropped rather than clipped, since a
truncated one names a different file.

Closes #825

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Skip the deep-path half where the filesystem will not hold the path

macOS caps a single path at 1024 bytes, so the 1200-byte tree the base-path
cases need cannot be created there and mkdir -p would fail the test outright.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:51:25 +00:00
Xavier Roche
97d7da5f19 hts_newthread() leaves the pthread attributes object undestroyed when thread creation fails (#807)
* hts_newthread() leaves the pthread attributes object undestroyed when thread creation fails

hts_newthread() folds pthread_attr_init(), pthread_attr_setstacksize() and
pthread_create() into one short-circuit condition, and destroys the attributes
object only on the branch where all three succeeded. If setstacksize or create
fails (EAGAIN under thread or memory exhaustion being the realistic case), attr
stays initialised for good. glibc allocates nothing for a default-initialised
attr, so nothing actually leaks on Linux. This is a portability and hygiene fix
rather than a live bug.

The init has to come out of the chain instead of just gaining a destroy call:
the shared condition cannot tell which of the three failed, and destroying an
object pthread_attr_init() never initialised is undefined.

The imbalance is invisible to a leak checker, so the test counts it directly. A
small LD_PRELOAD shim refuses pthread_create() and records which attributes
objects nobody destroyed afterwards: 1 undestroyed on master, 0 with the fix.

Closes #772

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: tighter comments, and only attr-bearing refusals count

Condense the two-line note above pthread_attr_init(), fix the "so that is can
be independent" typo on the line the diff moves, and count a refused spawn as
tested only when it carried an attributes object, so a NULL-attr create from
elsewhere cannot satisfy the guard on its own. Note why the shim's fixed slot
count and unlocked counters are enough.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:36:53 +00:00
Xavier Roche
15c66557b9 size_t i = strlen(s) - 1: the (i > 0) guard cannot see the underflow (#846)
* Backward scans from strlen(s) - 1 walk off the front of the buffer

optinclude_file()'s config-line right trim and url_savename()'s collision
rename both start at the last character and decrement with no lower bound.
An all-space httrackrc line and an all-digit save name walk below their
stack buffers; the second is reachable from a crawl.

Both go through hts_rtrimlen(), which counts down from the end. The
Content-Disposition trim was a third copy and now shares it.

Closes #814

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* x + strlen(x) - 1 points before the buffer on an empty string

The pointer spelling of #770. All 27 occurrences go through
hts_lastcharptr(), which clamps to the terminating NUL, and the two
hand-written ternary guards from #729 and #767 fold into it.

Closes #781

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the collision rename on Windows and from a followed link

116_engine-rtrim ran nowhere on Windows: that job iterates a fixed glob
that 116_engine-* does not match, and the rename is savename logic, which
behaves differently there. Add it by name.

The test only drove the rename through -g, which pins depth to 0 and so
needs both colliding URLs on the command line. Under -N "%n.%t" depth is
unrestricted, so one starting URL is enough and the crawled page picks
both names itself. Also strip CR before the empty-option check, which a
CRLF log would otherwise pass vacuously.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* size_t i = strlen(s) - 1: the (i > 0) guard cannot see the underflow

An unsigned index seeded with strlen(s) - 1 becomes SIZE_MAX on an empty
string, and SIZE_MAX passes the (i > 0) the sites carry. Seed from
hts_lastcharoffset() instead, and cover the spelling in the lastchar
self-test and its source guard.

Closes #821

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the historical narration from hts_lastcharptr's contract

The declaration is the API surface, so it keeps the double-evaluation
warning a caller cannot derive from the signature; what the macro
replaced is git's job. Its grep exclusion in the test goes with it,
since that comment was the only htssafe.h line the scan matched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch the bare-reassignment spelling in the source guard

The declaration pattern cannot see punycode.c, where input_length is
declared a dozen lines above the subtraction, so reverting that fix alone
left the guard silent. Match an uncast assignment too; the (int) casts in
htscore.c and htstools.c are the safe form and stay out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:34:21 +00:00
Xavier Roche
db4d429468 ProxyTrack's .arc writer overflowed its 8192-byte header block (#831)
* ProxyTrack's .arc writer overflowed its 8192-byte header block

Converting a cache to .arc builds each entry's response headers with sprintf
into a fixed char headers[8192]. The one bound guarding the cached header
block compared its length against sizeof(headers) - strlen(headers) - 1, a
subtraction that wraps once the string reaches 8192 bytes, so the guard passed
precisely when it should have failed. The Location: write and the closing CRLF
had no bound at all. Every value comes back off a cache, and an .arc whose
entry carries enough unknown header lines is enough: ASan reports a 12 KB
stack write.

Each append now clips to the room left, holding back four bytes for the blank
line that ends the block and for the CRLF a clip landing mid-line still owes
its line. Clipping instead of aborting is deliberate: these are cache reads,
where a short record beats killing the conversion.

Closes #820

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bracket the .arc header sweep around the measured cap

The fixed 7950..8250 window was 300 conversions picked to survive any change
to what the writer emits on its own. Measure that from the control entry
instead and sweep 32 bytes either side of the cap, which is the same coverage
in a fifth of a second.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Report a clipped .arc header block, and tighten the test around the real cap

Review follow-ups on the header-block bound. Truncating a cached header block
was silent, so a short record downstream had nothing pointing back at it; the
writer now names the URL on stderr, the way the readers already report a
corrupted entry. The test's own cap assertion allowed 8192 where the block
tops out at 8191, so a one-byte overflow would have passed it. runlen() had
been copied from the long-fields test and now lives in testlib.sh, and the
comment on the four reserved bytes says what depends on them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:27:02 +00:00
Xavier Roche
1b53b43176 Backward scans from strlen(s) - 1 walk off the front of the buffer (#815)
* Backward scans from strlen(s) - 1 walk off the front of the buffer

optinclude_file()'s config-line right trim and url_savename()'s collision
rename both start at the last character and decrement with no lower bound.
An all-space httrackrc line and an all-digit save name walk below their
stack buffers; the second is reachable from a crawl.

Both go through hts_rtrimlen(), which counts down from the end. The
Content-Disposition trim was a third copy and now shares it.

Closes #814

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the collision rename on Windows and from a followed link

116_engine-rtrim ran nowhere on Windows: that job iterates a fixed glob
that 116_engine-* does not match, and the rename is savename logic, which
behaves differently there. Add it by name.

The test only drove the rename through -g, which pins depth to 0 and so
needs both colliding URLs on the command line. Under -N "%n.%t" depth is
unrestricted, so one starting URL is enough and the crawled page picks
both names itself. Also strip CR before the empty-option check, which a
CRLF log would otherwise pass vacuously.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* x + strlen(x) - 1 points before the buffer on an empty string (#819)

* x + strlen(x) - 1 points before the buffer on an empty string

The pointer spelling of #770. All 27 occurrences go through
hts_lastcharptr(), which clamps to the terminating NUL, and the two
hand-written ternary guards from #729 and #767 fold into it.

Closes #781

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop the historical narration from hts_lastcharptr's contract

The declaration is the API surface, so it keeps the double-evaluation
warning a caller cannot derive from the signature; what the macro
replaced is git's job. Its grep exclusion in the test goes with it,
since that comment was the only htssafe.h line the scan matched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:09:31 +00:00
Xavier Roche
2e67587ec7 Cache repair deletes the old cache before a rename it never checks (#817)
* Cache repair deletes the old cache before a rename it never checks

Both zip-repair paths removed the damaged cache, moved repair.zip onto
its name without looking at the result, and announced a successful
recovery either way. A refused move left the entries under a name
nothing reads and no cache at all. Go through hts_rename_over(), which
keeps the destination when it cannot replace it, and report the failure
instead of claiming success.

Closes #786

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* hts_rename_over() can lose its destination when the retried rename fails

The unlink-then-rename fallback leaves nothing in place of dst between
the unlink and the retry, so a retry that fails too loses it. Park dst
under a free scratch name instead, drop it once the move succeeded, and
put it back otherwise.

Closes #790

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the cache-repair test Windows coverage

The Windows job iterates a fixed glob, so a 106_engine-* test never ran
there; name it in the list. The move it exercises is the one whose
failure mode is Windows-specific.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: check the restore, probe in UTF-8, state the honest guarantee

The move back out of the parked name was unchecked, so a retry that
failed for a reason that still applied left dst absent with the content
orphaned under a name nothing reported. Check it, retry once, and name
the parked copy in the log; hts_rename_over() takes an httrackp for that.

The aside probe used fexist(), which is not UTF-8 and consults the ANSI
codepage on Windows while the renames beside it are wide. It also reads
a directory as a free name, so the park now skips a name whose rename
refuses rather than giving up on it.

The header claimed a failure leaves dst as it was, which the crash
window between the two renames does not give.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The aside fallback parked a directory that stood in the way

Windows refuses every rename onto an existing target, so the fallback is
production code there rather than the rare path it is on POSIX. A
directory at the destination was renamed aside like a file, the move
then succeeded, and UNLINK could not drop the parked directory, so the
call reported success where master had reported failure and left an
orphan behind. 101_local-update-stale-bak plants exactly that shape and
caught it on both Windows legs.

Park a regular file only. A directory in the way is refused, as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:50:29 +02:00
Xavier Roche
45557b3dd6 An FTP resume spliced a changed remote when there was no cache entry (#830)
* An FTP --update resumed a complete mirror with REST and spliced the old file into the new body

FTP sent REST whenever the mirrored file merely existed, and on an --update
pass every previously mirrored file exists, so a complete copy was treated as
an interrupted download. The server resumed at its length and the mirror ended
up part old body, part new tail, at exactly the remote size, so nothing
downstream noticed. Resuming now follows the decision back_add() already makes
for HTTP, which only marks a copy partial when the cache does not hold it, and
r.size is seeded from the resume offset so a genuine resume is no longer
reported "FTP file incomplete".

Closes #798

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Format the two comment lines the earlier pass missed

git-clang-format only sees the diff present when it runs; the comments were
translated after it, so those lines never went through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* An FTP resume spliced a changed remote when there was no cache entry

With no cache entry to tell a partial from a stale copy, back_add() took
range_req_size from the file size and the transfer resumed with REST, so a
remote that had changed since had its tail appended to the old head at exactly
the remote length.

FTP has no conditional retrieval, so the client has to decide. Resuming now
needs SIZE to report a remote strictly longer than the local copy and MDTM to
report it no newer than that copy. The mirror is stamped from MDTM as the HTTP
path is stamped from Last-Modified, so later passes compare two server-clock
times; a server that does not answer MDTM proves nothing and is re-fetched
whole.

Closes #823

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Clear r.lastmodified before an FTP attempt

run_launch_ftp() already resets msg, statuscode and size for a retry; without
lastmodified in that list a retry whose MDTM fails stamps the mirror with the
previous attempt's date.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Require the copy's date to match MDTM, not merely to lead it

"Remote no newer than the local copy" is one-sided, and it lets through exactly
the mirrors #823 is about: nothing stamped an FTP file before this branch, so
every existing one carries a client-clock mtime ahead of any MDTM, as does a
copy restored from an archive or moved without preserving times. The first pass
over such a tree resumed and spliced.

The date now has to match, which is what the HTTP path gets from the server via
If-Unmodified-Since. A partial written here is stamped from the same MDTM, so
the resume it exists for still compares equal.

Pass 6 plants a stale copy dated by the local clock over a larger changed
remote; it splices under the old comparison. Pass 5 now dates its copy as the
remote so the missing MDTM is the only thing refusing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: fix TESTS list order and backslash after the union merge

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:49:09 +02:00
Xavier Roche
63af0dc4a2 A fragment on an inlined reference is dropped, so an SVG sprite selector is lost (#804)
* --single-file dropped the fragment on an inlined reference

sf_resolve cuts a reference at '#' to find the mirrored file, and sf_inline
then replaced the whole reference with the data: URI it built, so the fragment
never came back. On an SVG that changes what renders: "sprite.svg#icon" selects
one element, and without the selector the browser draws the whole sheet.

Re-attach it to both replacements, the data: URI and the rebased path an
over-cap asset falls back to. The query stays dropped: it named the remote
resource, not the mirrored file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Do not re-encode escapes the fragment already carried

The fragment is copied out of the document, so its '%' and '&' are the
document's own escapes; percent-encoding them again turns "#a&amp;b" into a
lookup for "a&amp;b". A mirrored path is the opposite case, a raw filesystem
name that has to be escaped, '#' included now that a fragment can follow it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the test header and drop an unused top_srcdir

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin every member of the fragment escape set

Only ')' and the '%'/'&' pass-through were exercised, so dropping '"', a
quote, a paren, a backslash, '<'/'>', whitespace or the high-byte rule from
sf_append_escaped left the suite green. The two that would be a real
injection are the '"' that ends the attribute the rewriter re-quotes and the
whitespace that ends an unquoted url() token; the fixture now carries one
fragment per class, and each one dies to its own mutant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:49:06 +02:00
Xavier Roche
e5c0918d5d Two branches adding a test always collide on the same list (#845)
* Generate the test list instead of hand-maintaining it

Every branch that adds a test edited the same TESTS list in tests/Makefile.am,
so any two in flight collided there, and resolving that by taking one side drops
the other test silently -- nothing downstream notices a name that stopped being
listed. bootstrap now writes tests/tests-list.mk from tests/*.test and
Makefile.am includes it, so a test is registered by existing.

The generated file is untracked and shipped by make dist, the same arrangement
configure and the Makefile.in files already use, so tarball builds are unchanged.

Refs #844

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Track the test list rather than generating it at bootstrap

The first attempt had ./bootstrap generate tests/tests-list.mk, which broke
every build: CI regenerates with autoreconf -fi and Debian with dh_autoreconf,
neither of which runs bootstrap, so automake aborted on the missing include.

Track the fragment instead. One "TESTS += name" per line with no continuation,
and merge=union scoped to that file, so two branches adding a test union cleanly
instead of colliding -- the shape matters, since unioning a backslash-continued
list drops an entry silently, which is why #843 was closed.

EXTRA_DIST does not need the fragment; automake already ships an included file
via am__DIST_COMMON.

Refs #844

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Correct the comment that still described the generated design

tests/Makefile.am claimed the list was generated and that a test registers by
existing. Neither is true after the pivot to a tracked file, and a contributor
who believed it would add a .test and have it silently skipped.

Also record the one hazard union cannot avoid: a deletion merged alongside
another branch's append is restored, so a removal wants a commit of its own.

Refs #844

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:02:39 +02:00
Xavier Roche
55dbca8beb Reading an .ndx cache walks the parser past the end of its buffer (#811)
* Reading an .ndx cache with full-length URL fields reads past the buffer

binput() returned count + 1 unconditionally, assuming it had consumed a
separator. When it stopped on the buffer's terminating NUL, or because the
destination filled, that byte was not a separator and the caller's cursor
moved one past it. In PT_LoadCache__Old the cursor then leaves the heap
buffer holding the .ndx and the next binput() call reads out of bounds; the
loop's a < (use + ndxSize) guard cannot see it, since the drift happens
after the check.

Step over the byte only when it really is a newline. cache_brstr() carried
a caller-side patch for the same drift and had to follow, or its adr[off-1]
probe would read before the buffer once binput() can return 0.

Closes #793

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Broaden test 117 and make its bookkeeping fail closed

The trigger is the walk reaching the buffer's terminating NUL, which any
.ndx not ending in a newline does; full-length fields are one route to it,
not the only one. Add a field that stops short of every bound, and a pair
straddling cache_brstr's own 256-byte field, the second destination the
diff touches.

items() returned 0 whenever the item-count line was missing, so a crawl
that printed nothing passed the zero-key assertion. Signals now clean up on
their own trap line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Rename test 117's zero-byte fixture off the Windows NUL device

Windows resolves NUL.<anything> to the null device, so nul.ndx never
existed there and proxytrack reported an unloadable index. The suite's
Win32 leg caught it once items() stopped treating a missing count as zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:20:23 +02:00
Xavier Roche
8c5b208fac A 304 during --update leaks the response's header buffer (#808)
* A 304 during --update leaks the whole previous htsblk

back_wait() handles a 304 by replacing the response struct with the cache
entry, carrying only the socket and keep-alive members across via
back_connxfr(). The struct assignment drops every owned pointer the live
response still held without freeing any of them: the 8 KB header buffer on
every update, plus the two WARC header stashes when --warc-file is on. An
update over a 10k-page site drops roughly 80 MB in one run.

back_clear_entry() already knew how to tear those down, so the frees move into
a helper that both it and the 304 path call.

The new test runs the two-pass mini304 crawl with LeakSanitizer on, which the
sanitized CI job otherwise disables. The fresh first pass is the control: it
has no cache entry to read back and is clean either way. The update pass
reports 16 KB in 2 objects on master, one per unchanged URL, and nothing with
the fix.

Closes #782

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the test header

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: reuse deleteaddr(), and cover the WARC limb

back_free_response() was reimplementing deleteaddr(), which already frees adr
and headers and NULLs both; call it instead so the two cannot drift.

Test 114 never passed --warc-file, so the warc_free_request() limb ran with
both pointers NULL on every path it exercised and deleting it kept the test
green. A third pass turns the archive on, and it now fails with the 835 and 238
byte stashes when that call goes away.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Register the new leak test as an expected Windows skip

The Win32/x64 job pins the exact set of tests allowed to skip, so an
all-skipped suite cannot report green. 114_local-update-304-leak needs a
LeakSanitizer build and MSVC has no equivalent, so it skips there and tripped
the gate with fail=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 17:18:44 +02:00
Xavier Roche
b7cff7aeee hts_rename_over() can lose its destination when the retried rename fails (#816)
* hts_rename_over() can lose its destination when the retried rename fails

The unlink-then-rename fallback leaves nothing in place of dst between
the unlink and the retry, so a retry that fails too loses it. Park dst
under a free scratch name instead, drop it once the move succeeded, and
put it back otherwise.

Closes #790

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Review fixes: check the restore, probe in UTF-8, state the honest guarantee

The move back out of the parked name was unchecked, so a retry that
failed for a reason that still applied left dst absent with the content
orphaned under a name nothing reported. Check it, retry once, and name
the parked copy in the log; hts_rename_over() takes an httrackp for that.

The aside probe used fexist(), which is not UTF-8 and consults the ANSI
codepage on Windows while the renames beside it are wide. It also reads
a directory as a free name, so the park now skips a name whose rename
refuses rather than giving up on it.

The header claimed a failure leaves dst as it was, which the crash
window between the two renames does not give.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The aside fallback parked a directory that stood in the way

Windows refuses every rename onto an existing target, so the fallback is
production code there rather than the rare path it is on POSIX. A
directory at the destination was renamed aside like a file, the move
then succeeded, and UNLINK could not drop the parked directory, so the
call reported success where master had reported failure and left an
orphan behind. 101_local-update-stale-bak plants exactly that shape and
caught it on both Windows legs.

Park a regular file only. A directory in the way is refused, as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 16:51:19 +02:00
Xavier Roche
bc8b6cfa00 A successful FTP transfer was blanked when its backlog slot was swapped out (#809)
The FTP worker writes url_sav itself, so its slot carries a size but no
in-memory body. Serializing that slot to the on-disk ready table stores no
body, and the read-back took the size from what it stored, leaving zero: the
link writer then saw an empty response and created a 0-byte file over the
bytes already on disk, while the engine logged the transfer as a success.

Test 110 mirrors twelve files at -c8, which is what makes a ready slot wait
long enough to be swapped, and -#test=backswap covers the round-trip directly.

Closes #797

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 16:43:43 +02:00
Xavier Roche
260cef25fe PT_GetTime hands out gmtime's shared static instead of a reentrant breakdown (#805)
* PT_GetTime copied gmtime's shared static instead of a reentrant breakdown

On _WIN32 the success path took gmtime()'s pointer and dereferenced it after
the fact, so a concurrent conversion on another thread could change the
breakdown under it. The POSIX branch was already reentrant via gmtime_r, and
the same #ifdef pair had been copy-pasted into hts_now_iso8601() and the WARC
auto-name; fold all of them onto one hts_gmtime() helper, and give ProxyTrack's
WebDAV listing the same treatment, since it read the static's fields well past
the call.

Windows uses Microsoft's gmtime_s (destination first, errno_t return), not the
C11 Annex K function of the same name.

Covered by a new "gmtime" engine self-test: a reference table checks the
breakdown itself, which is what catches a swapped-argument call on the MSVC
leg, and eight threads hammering the helper catch a return to the shared
static (16k of 400k conversions corrupt with that mutant in place).

Closes #794

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the gmtime self-test teeth on the failure path and off UTC

Three holes the test-design audit found by running mutants rather than
reading the diff.

A helper that discarded gmtime_r's NULL and always claimed success passed
every phase, yet that boolean is the only failure signal hts_now_iso8601,
warc_open and PT_GetTime have; all three would have formatted an
uninitialised struct tm. A forced-failure row now converts INT64_MAX, gated
on a 64-bit time_t.

The localtime_r mutant only died on a non-UTC box. CI runners are UTC, where
localtime_r and gmtime_r agree on every reference row, so the test exports
TZ=XXX5.

The "first result survives the second call" phase could not fail: both
buffers are caller-owned stack storage no implementation writing through
tmbuf could disturb. Removed rather than left reading as coverage.

expect_ok() was a third byte-identical copy; it moves to tests/testlib.sh
with the two existing callers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Rename the self-test's out-of-range time_t off the "far" keyword

WinDef.h defines "far" away to nothing, so the declaration lost its variable
and MSVC rejected the file. Both Windows legs caught it; the POSIX builds
never see the macro.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 16:21:40 +02:00
Xavier Roche
bd60dac914 Handle the fgets return in the log-callback self-test (#827)
(void) does not suppress glibc's warn_unused_result, so gcc warned on
the read-back in st_logcallback. Treat a failed read as the test failure
it is instead of asserting against an empty buffer.

Closes #812

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 16:06:24 +02:00
Xavier Roche
0984aa2530 grep -q SIGPIPEs the producer feeding it, so an assertion that held reports failure (#822)
* Match captured output with a here-string, not a pipe into grep -q

grep -q exits on the first match, so whatever the producer still had to
write takes SIGPIPE; under pipefail that becomes the pipeline's status and
an assertion that held reports failure. bash issues one write() per line,
so any match that is not on the last line is exposed.

Converts every test assertion whose producer is a shell builtin or shell
function, including two pipelines used as an if condition where the SIGPIPE
silently flips the branch. Generalizes the AGENTS.md bullet, which only
covered the "&& fail" spelling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Convert the remaining pipe-into-grep -q sites

The self-test drivers survive only because the matched line is the last
thing httrack prints; one added line of output turns a pass into 141.
Nothing in tests/ pipes into grep -q now.

The two zlib drivers claimed the harness might run them under a POSIX
/bin/sh: it does not. configure resolves $(BASH) to bash, test-timeout.sh
execs it, and 01_zlib-warc-wacz.test already uses "set -o pipefail" (which
dash lacks) on the macOS leg. Kept the half that is true, BSD tool flags.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 15:49:00 +02:00
Xavier Roche
bc6c53598c The -o help text promises a generated error page the engine never builds (#803)
* The -o help text promises a generated error page the engine never builds

`-o` only decides whether the error page the server sent survives: `store_errpage`
keeps `r.adr` alive so the normal save path writes it, and the `-o0` arm frees it.
Nothing anywhere builds a stand-in body. The one block that would have was dead
since the 3.20.2 import and was removed in #783.

Reword the help line, the man page and fcguide's two `-o` prose blocks to say the
server's error page is saved rather than generated, and extend 23_local-errpage
so the mirrored 404 has to carry the server's own body.

Closes #787

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Condense the -o1 control comment to one line

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 15:26:04 +02:00
Xavier Roche
809d5b6ffc An FTP --update resumed a complete mirror with REST and spliced the old file into the new body (#810)
* An FTP --update resumed a complete mirror with REST and spliced the old file into the new body

FTP sent REST whenever the mirrored file merely existed, and on an --update
pass every previously mirrored file exists, so a complete copy was treated as
an interrupted download. The server resumed at its length and the mirror ended
up part old body, part new tail, at exactly the remote size, so nothing
downstream noticed. Resuming now follows the decision back_add() already makes
for HTTP, which only marks a copy partial when the cache does not hold it, and
r.size is seeded from the resume offset so a genuine resume is no longer
reported "FTP file incomplete".

Closes #798

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Format the two comment lines the earlier pass missed

git-clang-format only sees the diff present when it runs; the comments were
translated after it, so those lines never went through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 15:21:50 +02:00
Xavier Roche
536d7515ed CI format check compares against master's tip, not the branch's merge base (#802)
The changed-lines clang-format job resolved its base as origin/<base_ref>,
which is master's tip when the job runs. Once master gains a C commit while
a PR is open, the comparison also picks up the reverse of that commit and
the job fails on code the PR never touched.

Use git merge-base instead, and fail loudly if there is none rather than
falling back to a whole-tree comparison.

Closes #800

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 14:07:45 +02:00
Xavier Roche
374087a81c A log callback consumes the va_list the log file still needs (#801)
hts_log_vprint() makes a va_copy and then throws it away: the callback is handed
the original args, and vfprintf() writes the log file from that same,
already-consumed list. On x86_64 a va_list is a one-element array, so the callee
moves the caller's cursor; the second traversal reads past the register save
area, and a %s yields a junk pointer that vfprintf() dereferences.

Only an embedder that installs a callback is affected, so the CLI never sees it.
HTTrack Android does: --sitemap is the first option whose LOG_NOTICE lines carry
arguments, and ticking its checkbox segfaults the crawl thread. The same crawl is
clean under the CLI built with ASan+UBSan.

Pass the copy to the callback. -#test=logcallback sends one line with a %d and a
%s through both sinks and compares them; on the unfixed code the log file gets
"0 " and the test fails. It also logs a second line below opt->debug with no
opt->log, pinning that the callback fires above the level filter.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 13:48:00 +02:00
Xavier Roche
83079b788d Teardown must not decide a test's verdict (#792)
Under `set -e` a failing command in an EXIT trap becomes the script's exit status, so a hiccup while tearing down fixtures fails a test whose assertions all passed. That is what turned `57_local-proxy-connect.test` red on the Windows x64 leg of #765: five OK lines, no FAIL, exit 1. Every EXIT trap in the suite now runs teardown with errexit off, and the signal traps keep their own `trap` line, since sharing `set +e` with HUP/INT/QUIT/PIPE/TERM would leave errexit off for the rest of a signalled run and let a torn-down test still report success.

A `|| true` on the `rm` would have been smaller, but it throws away the only diagnostic, and the evidence does not say which teardown command failed: a blocked `rm -rf` on the Windows runner exits 1 and prints "Device or resource busy", while the log shows exit 1 and nothing at all. The sharing violation in the issue is the plausible mechanism rather than a confirmed one, so whatever it really is now prints its own error.

`99_teardown-status.test` pins the semantics both ways and scans the suite so a new test cannot reintroduce the shape, the leaky combined trap included. The `return 0` that three `cleanup()` bodies ended with never protected anything, since errexit fires at the failing command before it is reached.

Closes #773

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 13:24:23 +02:00
Xavier Roche
1fa3cb4e74 A wedged test hangs the job until CI cancels it, discarding the log that would name it (#796)
A test that wedges runs until CI cancels the step, and a cancelled step keeps neither its log nor the artifacts its `if: always()` uploads would have produced. That is why nobody has ever been able to say which Windows test hangs, across 19 dead jobs in the last day alone (#795).

Each test now runs under a wall-clock budget at the automake harness level, so an overrun names the test, dumps the surviving process tree and an engine stack, and exits 124. The step then fails rather than being cancelled, which is what keeps the log. Every POSIX `make check` leg gets this; the Windows leg runs its own serial loop and now calls the same wrapper. The budget is 600s, the value the Windows leg already used: it has to clear the 540s a three-pass crawl may legitimately take under `local-crawl.sh`'s own 180s-per-pass watchdogs, against a slowest healthy test that actually measures 39s.

Two things sit on top of the per-test bound. The Windows suite gives up at 25 minutes so it fails on its own terms well before the 45-minute step timeout, and it sweeps leaked engine processes between tests, naming whichever test left them. An orphaned `httrack.exe` starving the runner is the leading theory for the hang, and that sweep is what would confirm it.

This also fixes an unbounded `wait` after `kill_tree` in testlib.sh. When the kill failed to reap, which is exactly the native-Windows case those watchdogs exist for, the watchdog blocked forever and never printed the timeout it was about to report.

Stacks differ by platform, and each branch says which one it took, because a dump that silently produces nothing reads as coverage. Linux sends SIGABRT and lets httrack's own crash handler symbolize itself, verified against a real wedged crawl where it named `back_wait` at htsback.c:2710. macOS cannot do that, since htsbacktrace.c gates the handler on `__linux`, so it uses `sample(1)`. Windows uses `cdb` from the SDK, and that is the one path I could not exercise from here; it probes for the binary, reports when it is absent, and is bounded so a debugger that wedges cannot become the new hang.

Does not fix #795, only makes it diagnosable.

The between-test sweep costs the Windows step about two minutes (506s before, 619s on this run). That buys naming whichever test leaks, which is the whole lead on #795; it can be narrowed or dropped once the leak is found.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 13:06:51 +02:00
Xavier Roche
bab4b70a71 server-not-modified revisit records name no capture, so replay cannot resolve them (#789)
A 304 revisit record carried a `WARC-Profile` and nothing else naming what it stood for. Neither replay engine reads that field: pywb resolves in `_load_different_url_payload` on `WARC-Refers-To-Target-URI` and raises `ArchiveLoadFailed` without it, wabac.js reads `warcRefersToTargetURI` and otherwise answers Not Found. So the records were conformant under WARC 1.1 6.7.3, which only recommends the field, and unreplayable in both engines that matter.

For a server-not-modified revisit the referred-to URI is the record's own target URI, so it is free to emit.

No `WARC-Refers-To-Date` alongside it. The cache persists no capture timestamp: the field list in `cache_add()` ends at `Last-Modified`, which is a document property, and the zip entry's own date is set from that same value. Emitting it would assert that a record exists with that `WARC-Date`, which is false and would misdirect pywb's `closest=` lookup. Both engines already handle the field being absent, pywb by falling back to the CDX timestamp and wabac.js to the revisit's own. A real date needs a capture time in the cache, which is worth doing when the segment work lands and a previous index is being read anyway.

The validator now requires every revisit to name a capture, and requires a server-not-modified one to name its own URI. Test 73 already drives it over an archive of revisits; against the pre-fix engine it fails there.

Stacked on #788. Without it the new header line is what pushes a 995 to 1004 byte URL past `wbuf_printf`'s old 1024-byte buffer, and `warc_emit` then drops the record whole; the review caught that before this was pushed. The merge is in the branch so the pair is what got tested.

Closes #778

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 12:50:52 +02:00
Xavier Roche
dcdcc8745c A URL of 1005 bytes or more is silently dropped from the WARC archive (#788)
Every WARC header line is built with `wbuf_printf()`, which formatted into a 1024-byte stack buffer and returned `-1` instead of growing. `"WARC-Target-URI: %s\r\n"` costs 19 fixed bytes, so the line failed once a URL reached 1005 bytes, the `-1` reached the `goto done` in `warc_emit()`, and the record was abandoned. The page still got mirrored, the crawl still exited 0, and nothing was logged, so the only symptom was a URL missing from the archive.

`wbuf` reallocs already, so oversized output now formats straight into it after a `wbuf_reserve()`, which is the growth half of `wbuf_add()` split out. Every field benefits, not only the two carrying URLs. The second pass is bounded against what was reserved rather than trusted: advancing `len` by a return value larger than the reservation would push `len` past `cap` and corrupt the bounds check of every later append.

`-#test=warc-longurl` sweeps 100 to 9000 bytes across the boundary. Against the old formatter it fails at exactly 1005 and up, with 1003 and 1004 passing. Each record carries a distinct payload because identical ones dedupe into revisits, which would otherwise hide the response records the test counts.

One caveat on the sanitizer evidence, since it is easy to over-read. The buffer grows by doubling, so a small off-by-one lands in allocation slack where ASan cannot see it; that is why the second-pass bound is a logic check rather than something left to the sanitizer. The ASan+UBSan run over the sweep is clean, but only after planting a deliberate overflow in `wbuf_reserve` to confirm the probe actually fires. It did not, at first: libtool silently drops `-fsanitize` from the shared-library link, which produced no binary at all and a "clean" result that meant nothing.

Worth knowing for the segment work: `warc_emit()` returning `-1` also sets `w->failed`, which suppresses the archive swap added in #777. Before this fix a single over-long URL would make an `--update` pass throw away its whole archive and keep the previous one.

Closes #785

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 12:49:36 +02:00
Xavier Roche
c65d1c43c7 An FTP re-fetch truncated the mirrored file before the transfer (#799)
An FTP re-fetch called `filecreate()` on the mirrored file before a byte of the transfer had arrived, so a read error, a timeout or a short body destroyed the previous copy. HTTP has moved the good copy aside to a `.bak` since the #77 follow-up and puts it back when the transfer fails; FTP never took that backup. Rather than give it a second copy of the idiom, the backup moves out of `back_wait()`'s direct-to-disk block into `back_refetch_backup()`, which the FTP transfer now calls too. `back_finalize()` already restores it, so FTP inherits that. The REST resume branch appends and needs no backup.

That alone was not enough. `back_cleanup_background()` swaps a ready slot to the on-disk table through `back_clear_entry()`, which unlinks `back->tmpfile`, and a failed re-fetch sits at `STATUS_READY` unfinalized until the parser picks it up. The new `.bak` was being deleted in that window before `back_finalize()` could restore it, about half the time on a loaded box. `slot_can_be_cached_on_disk()` now refuses a slot that still owns a temporary, which closes the same window on the HTTP `.bak` and on the content-coding spool. The crawl-level race needs concurrency the suite cannot pin down, so `-#test=backswap` covers the predicate directly.

The suite had no FTP server. `tests/ftp-server.py` is a minimal one (PASV, SIZE, REST, RETR, LIST) with a mode file the test rewrites between passes, so a path can start failing without the port moving and taking the mirror directory name with it. Test 102 mirrors three files, then re-fetches one cut short, one served empty and one healthy: the first two must come back byte identical and be reported `unchanged`, the third replaced. It runs at `-c1` because a parallel FTP crawl loses whole transfers to #797, which is older and unrelated and would flake it on a loaded runner. #798 came out of the same work: an FTP `--update` sends `REST` over a complete mirror and splices the old file into the new body.

Closes #771

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 12:29:55 +02:00
Xavier Roche
95f8ebaa44 hts_rename_over() deletes its destination when the source is missing (#784)
The unlink-then-rename fallback in `hts_rename_over()` is there because Windows' rename() refuses an existing target, but it fired on any failed rename, ENOENT on the source included. A caller moving a temp file it never managed to write lost the destination and got `HTS_FALSE` back, which reads as "nothing happened". #754 unified four hand-rolled copies into this helper, so every call site inherited it.

The unlink now runs only for EEXIST, and only with a source that exists. EEXIST is the value that matters: the CRT maps ERROR_ALREADY_EXISTS there and keeps EACCES for a source another process holds, so accepting EACCES as well would have deleted the destination for a failure it had no part in, and the retry would then fail with the destination already gone. The source check is belt and braces for a CRT that reports neither. `hts_rename_utf8()` preserves errno across its free() calls now, since the gate reads it.

The existing call sites all derive their source's existence from a create that had to succeed first, so the bug was latent rather than live, but three of those destinations are user data: a mirrored file, a finished .wacz, and a rewritten page nothing will re-fetch. What is left of the window after this is #790.

The selftest probes what rename() does to an existing target and asserts against the regime it finds, so it runs three ways on Linux: native, then under an LD_PRELOAD rename() with Windows' shape, then under one reporting a locked source. The harness pins the expected regime per platform, so no leg can pass having exercised the other half. Seven mutants of the gate were each confirmed to fail it.

Closes #779

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 12:07:12 +02:00
Xavier Roche
3a096c589a x[strlen(x) - 1] indexes before the buffer on an empty string (#780)
Three bugs have already come out of the same one-liner: `x[strlen(x) - 1]` indexes one byte before the buffer when the string turns out to be empty (#729, #730, #768). Rather than wait for a fourth, this replaces the idiom everywhere with helpers that cannot underflow.

All 48 occurrences were reclassified by tracing each guard to where it actually lives instead of to the two lines above the index. 44 were already guarded, often a dozen lines up or in the caller, and one sits inside a commented-out block. The rest have no guard at the site, and two of those are reachable from a crawl.

`htscore.c:2194` is the one that matters: a one-byte stack out-of-bounds write in the end-of-mirror purge, confirmed under ASan and UBSan. `linput()` reads `old.lst` in 999-byte chunks, so a 1000-byte line comes back as 999 bytes plus a one-byte tail; for that tail `line + 1` is empty, and without `-O` so is `path_html`, which leaves `file` empty when the index runs. The crawled site controls the save-path length and therefore the line length. It needs a second run over the same project, which is what a re-crawl does. As controls, a 900-byte path and the same path with `-O` both stay clean.

`htsparse.c:2009` is a one-byte overread in the HTML parser, reachable from `<a href="   ">`, where the guard sits on the wrong side of the `&&`. The write on the next line is accidentally safe for the same reason. #768 itself turned out not to be reachable from a crawl, only through the `-#test=savename` hook, so I would not call that one a security fix.

The helpers live in `htssafe.h` beside the other bounded string operations, so no file needed a new include. Guards doing more than an emptiness check are kept, since folding those in would append a separator to an empty string.

`-#test=lastchar` checks the helpers against a poisoned neighbouring byte, including the `/`-before-the-buffer case from #768, and putting the missing length check back makes it fail. It also greps the source, so reverting any converted site fails instead of leaving the self-test green. `97_local-purge-longpath` drives the purge site end to end; on unfixed code the sanitized CI leg aborts at `htscore.c:2194`.

Built from identical source paths, 61 of 72 objects are byte-identical, one differs only in debug info, and the remaining 10 are the files edited.

The pointer spelling `x + strlen(x) - 1` has 25 occurrences and is filed separately as #781.

Closes #770
Closes #768

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 11:44:43 +02:00
Xavier Roche
1a4a5082b7 The --errpage placeholder block has been unreachable since 3.20.2 (#783)
The stand-in body `httpmirror()` builds for an error page has been unreachable since the 3.20.2 import: both guards say the opposite of their comments, so the block runs only when there is no save name and the URL *is* `/robots.txt`. `create_html_warning` is never assigned, so the HTML arm is dead outright. The GIF arm fires only if the user maps `.txt` to `image/gif` with `--assume`, and it then swaps `r.adr` for a 1070-byte buffer while leaving `r.size` at the error body's length, so the `robots_parse()` call below over-reads the heap.

Repair is not the one-character fix the inverted guards suggest. `filesave()` below writes `r.size` bytes, so uninverting the guards without also setting `r.size` moves the same over-read into the user's mirror. A correct repair would then replace every server error body with HTTrack's 2003 template by default, which is what `23_local-errpage.test` asserts is kept. Nobody has asked for the placeholder in twenty years, and #17 asked for the opposite.

`--errpage` itself is untouched: `store_errpage` keeps the server's error body and the normal save writes it. A byte-level differential against a master build over five error shapes under both `-o1` and `-o0` gives identical mirrors, and rebuilt objects differ only in `htscore.o`. The new test drives the one input that reached the block, and fails on master in a plain build rather than only under the sanitizers. Separately, the `-o` help text promises a generated page the engine has never produced; that predates this change and is filed as #787.

Closes #769

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 11:31:47 +02:00
Xavier Roche
4fd6767a8f A stale .bak silently disables the re-fetch backup on Windows (#776)
`back_finalize()` moves an existing file aside to `<file>.bak` before truncating it, so an aborted re-fetch can put the previous copy back. That rename was a bare `RENAME`, and Windows refuses to rename onto an existing target, so a `.bak` outliving a killed run made every later re-fetch of that URL fail the rename, take the `tmpfile = NULL` branch with nothing logged, and truncate the live file with no backup at all. The #77 guard was off for that file, silently and for good. `hts_rename_over()` (#754) unlinks and retries, which fixes it.

Worth a look, because clobbering a stale `.bak` is not free. A run killed between the rename-aside and the finalize leaves the previous complete copy in `.bak` and a partial in the live file, and the next re-fetch now overwrites the good one. It still looks like the right trade: POSIX `rename()` has behaved exactly this way since #77 landed, so the alternative is leaving Windows with a guard that stays dead until someone deletes the file by hand, and a leftover `.bak` is engine garbage that nothing advertises or ever restores. The clobber is logged now, so it is at least visible.

Any failure to create the backup is logged too, instead of quietly disabling the safety net. `htscache.c`'s static `hts_rename()` wrapper and the hand-rolled `old.zip` unlink at its only call site go the same way, which removes a copy of the unlink-then-rename idiom rather than adding a fifth.

Test 101 plants both kinds of leftover before the update pass: a stale file, which has to be clobbered so the `-M` abort can still restore the pass-1 copy, and a directory, which cannot be clobbered and has to be reported instead. Only the Windows leg arms the first half, since POSIX clobbers on its own; the second is armed everywhere. Test 37 gains a live update pass so the dead pass rotates onto an existing `old.zip`, which nothing covered.

Two older bugs on the same path came out of the review and are filed separately: #774 (the `.bak` name collides with a mirrored file of the same name, reproduced) and #775 (a failed `filecreate()` on a chunked re-fetch commits the backup away).

Closes #758

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 11:10:17 +02:00
Xavier Roche
56389103bf An --update pass overwrites the previous WARC and leaves a page-less WACZ (#777)
* warc: keep the previous archive when a pass has no bodies to replace it

A second crawl into the same output reopened the WARC with "wb" and
truncated it. On a cache-served pass nearly every URL comes back 304, so
the new file held revisit records whose payloads had just been deleted,
and the regenerated WACZ came out with zero page rows: a package that
replays nothing, in place of one that replayed fine.

The writer now builds into a sibling .tmp whenever an archive is already
there, and only swaps it in at close if the result can stand on its own.
A pass that only revisited URLs it did not re-download keeps the previous
.warc.gz, .cdx and .wacz untouched and says so.

What --update should ultimately mean for WARC output is still open; this
only stops the silent destruction in the meantime.

Closes #759

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* warc: guard the segment swap and cover the rotated archive

A run that lost a record or a segment must not replace a whole archive,
and hts_rename_over unlinks its destination when the source is missing,
so every segment has to be on disk before the first rename.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* warc: force rotation in the segment test and guard its vacuity

--warc-max-size 2000 never rotated under the harness's --robots=0, so the
segment test was checking a single-file archive; a mutant that renamed
only segment 0 passed it. 600 rotates, and --archive-min-files fails the
test if a shrinking crawl ever stops producing the segments it checks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 10:24:06 +02:00
Xavier Roche
a4db58f7b7 hts_finish_html_file is documented as skipping unchanged writes, and does not (#764)
The MD5 comparison the comment promises left the tree in two steps: #467
extracted the function out of htsparse.c's HT_ADD_END macro without the skip
branch, and #512 removed the //[HTML-MD5]// cache entry it read. Drop the
parenthetical; the write is unconditional.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 10:11:11 +02:00
Xavier Roche
1f9728d816 --purge-old deleted a file whose re-fetch never got a response (#765)
* A failed re-fetch overwrote the mirrored file with the aborted read's debris

A transfer that dies before a complete response has no body, yet the save path
still consulted r.adr, which at that point holds whatever the aborted header
read left behind: raw status-line bytes, or an empty buffer that truncated the
file to zero on macOS. Require a successful transfer, as the empty-body half of
the condition already did.

Closes #748

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the failed re-fetch keeps its bytes on every platform

Test 93 filtered reset.bin out of its bucket lists because a connection killed
before the status line surfaced differently per platform. It no longer does, so
assert the resource like any other: unchanged, with the bytes pass 1 mirrored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: make the re-fetch assertions catch a crawl that never re-fetched

The new test passed with no second pass at all, so a regression that stopped
re-fetching would have looked green. Assert the failure the fixture provokes,
give stay.bin a fresh pass-2 body so a fix that stopped overwriting anything
fails, and compare reset.bin by checksum rather than by length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* --purge-old deleted a file whose re-fetch never got a response

A transfer that dies on the wire leaves the previously mirrored copy in place,
but back_finalize() returned without noting it, so the URL fell out of new.lst
and the end-of-update purge treated the file like a page that had vanished from
the site. Note the surviving copy, the way the incomplete-transfer branch above
already does, for the connection-level failures htsparse.c retries on. A
deliberate skip (too big, MIME-excluded, cancelled) keeps its current fate.

Closes #746

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the copy for every failure, not just the five retryable codes

A malformed status line, an oversized declared length or a mid-flight abort all
land on STATUSCODE_INVALID, outside the retryable set, and the purge still ate
the file. Invert the test: any failure keeps the copy except the codes that
mean the engine passed the resource over on purpose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the retry exhaustion, not the per-platform failure message

The message a cut connection produces depends on whether any bytes arrived, so
matching it would fail on a runner that sees none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 10:03:31 +02:00
Xavier Roche
99bd9bb674 A failed re-fetch overwrote the mirrored file with the aborted read's debris (#763)
* A failed re-fetch overwrote the mirrored file with the aborted read's debris

A transfer that dies before a complete response has no body, yet the save path
still consulted r.adr, which at that point holds whatever the aborted header
read left behind: raw status-line bytes, or an empty buffer that truncated the
file to zero on macOS. Require a successful transfer, as the empty-body half of
the condition already did.

Closes #748

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the failed re-fetch keeps its bytes on every platform

Test 93 filtered reset.bin out of its bucket lists because a connection killed
before the status line surfaced differently per platform. It no longer does, so
assert the resource like any other: unchanged, with the bytes pass 1 mirrored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: make the re-fetch assertions catch a crawl that never re-fetched

The new test passed with no second pass at all, so a regression that stopped
re-fetching would have looked green. Assert the failure the fixture provokes,
give stay.bin a fresh pass-2 body so a fix that stopped overwriting anything
fails, and compare reset.bin by checksum rather than by length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the retry exhaustion, not the per-platform failure message

The message a cut connection produces depends on whether any bytes arrived, so
matching it would fail on a runner that sees none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:44:28 +02:00
Xavier Roche
29dfd2df59 htsserver never returns from main(), it blocks on its own exit wait (#757)
htsthread_wait_n(background_threads - 1) subtracts one more than the count of
threads that must not be joined. Without --ppid that count is zero, so the wait
asks for a negative number of outstanding threads and the counter never gets
there; with --ppid it waits on the pinger, which by design never returns.

Wait for background_threads instead, which is what the sibling call inside
back_launch_cmd() already passes.

Closes #753

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:43:51 +02:00
Xavier Roche
55765815f5 htsAddLink walks back from an empty codebase, one byte before the buffer (#767)
* htsAddLink walks back from an empty codebase, one byte before the buffer

Same idiom as the lienrelatif() underflow fixed in #729: the trim that walks
back to the last '/' starts at codebase + strlen(codebase) - 1, which is
codebase - 1 when the string is empty, and the loop dereferences it before
a > codebase stops it.

Unlike #729 there is no reachable empty value. codebase is copied from a
recorded link's fil, and no hts_record_link() call site can supply an empty
one: every fil is either a literal seed or an ident_url_absolute() /
ident_url_relatif() success return, and fil_simplifie() restores "/" or "./"
rather than leaving a path empty. The guard goes in anyway, and -#test=addlink
drives the walk directly: under the sanitize job's ASan+UBSan build the test
fails on the unfixed walk and passes with the guard. Its second case pins the
ordinary trim, which the guard leaves alone.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* review: add the case that actually notices the codebase trim

For an ordinary relative link ident_url_relatif() re-derives the directory
from the path it is handed, so deleting the trim outright left the two
existing cases green. A query-only link ("?x=1") copies that path whole, and
does catch it.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:40:11 +02:00
Xavier Roche
869b8479e9 structcheck() builds its rename target with an unbounded sprintf (#762)
* structcheck() builds its rename target with an unbounded sprintf

Both structcheck() and structcheck_utf8() move a regular file sitting where a
directory belongs, and build the "<name>.txt" target with a raw sprintf into a
2048-byte buffer. It stays in bounds only because of a strlen(path) >
HTS_URLMAXSIZE guard dozens of lines above, which nothing at the write site
mentions. Route both through sprintfbuff() and fail with ENAMETOOLONG, so the
bound is local.

Armed the probe: with that distant guard patched out, a 2045-byte path makes
the old sprintf write 2049 bytes into tmpbuf[2048] under ASan; the same build
with sprintfbuff() returns -1 and reports nothing.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* review: tighten the structcheck self-test and its comments

The path builder could end a path with a bare separator when the base
directory's length hit the wrong residue, so the test aborted on a long
$TMPDIR. The refusal case also asserted on a component structcheck never
creates; assert on the outermost one instead.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the max-length rename case, macOS PATH_MAX is 1024

The path the guard admits (HTS_URLMAXSIZE) plus ".txt" is longer than macOS
accepts, so fopen() failed there. The case could not tell a fixed build from an
unfixed one anyway; what is left covers the guard and the rename on both entry
points.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:34:02 +02:00
Xavier Roche
9fe47c3986 Remove dead HTS_USEZLIB guards now that zlib is mandatory (#761)
configure has rejected --without-zlib since #750, and htsglobal.h
#errors on a forced HTS_USEZLIB=0, so the macro can only ever be 1.
Collapse every #if/#ifdef HTS_USEZLIB guard (htsweb.c, htsname.c,
htslib.c, htszlib.c, htsselftest.c, htscodec.c) to its always-taken
branch; htsweb.c's guard used #ifdef where every other site used #if,
an inconsistency that no longer matters once the guard is gone.

htscodec.c's #else arms were a genuine zlib-free content-coding path
(Accept-Encoding: identity, hts_codec_unpack returning -1), not stubs.
Removed for consistency with the other nine guards: the cache
(htscache.c, proxy/store.c) already reaches minizip unconditionally
from ~60 call sites with no null backend, so a zlib-free build is not
actually reachable today regardless of this file.

No new test: this is dead-code removal with no behavior change.
Verified by differential build against master: htscodec.o and
htszlib.o are byte-identical; the other touched objects differ only
in __LINE__ immediates shifted by the removed guard lines (plus one
cosmetic objdump label-annotation artifact each in htsselftest.o and
htsweb.o, from string-literal pool reordering). Exported symbols in
libhttrack.so are unchanged. make check: 166/166 (158 pass, 8 skip).

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:05:52 +02:00
Xavier Roche
37fa549ac5 Add the regression test #747 could not carry when it landed (#760)
htsselftest.c and tests/Makefile.am were held by #718 while #747 was fixed, so
the thread-counting fix went in without a test. -#test=threadwait covers it
from both sides: a wait placed right after a spawn joins that thread, and
wait_n(n) leaves n running rather than draining them.

One spawn per round is what makes it bite. A batch gives the earlier threads
time to raise the counter themselves, which is why an eight-thread version
passed on the unfixed engine; one thread per round failed 10 runs out of 10.

The changes-race self-test can now drop the counter it kept because
htsthread_wait() could not be trusted to join.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:05:44 +02:00
418 changed files with 13952 additions and 8163 deletions

3
.gitattributes vendored
View File

@@ -6,3 +6,6 @@
# bash die on $'\r' on every line of them.
*.test text eol=lf
*.sh text eol=lf
# Scoped to this file so a union can never touch build logic.
tests/tests-list.mk merge=union

View File

@@ -12,3 +12,9 @@ updates:
directory: /
schedule:
interval: weekly
# src/coucal is a hard build dependency, and nothing else notices when it moves.
- package-ecosystem: gitsubmodule
directory: /
schedule:
interval: weekly

View File

@@ -79,7 +79,10 @@ jobs:
- name: Build
run: make -j"$(nproc)"
# A backstop only: tests/test-timeout.sh bounds each test, and a healthy run
# is a minute here, two on macOS. Without it a stall ran to the job's 6h default.
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -119,6 +122,7 @@ jobs:
run: make -j"$(nproc)"
- name: Test without python3
timeout-minutes: 20
run: |
set -euo pipefail
# Hide every python3* so `command -v python3` fails like it does in the
@@ -173,6 +177,7 @@ jobs:
sudo ifconfig lo0 alias 127.0.0.3 up
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(sysctl -n hw.ncpu) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -245,6 +250,7 @@ jobs:
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -302,6 +308,7 @@ jobs:
env:
ASAN_OPTIONS: detect_leaks=0:abort_on_error=1:halt_on_error=1:strict_string_checks=1:malloc_fill_byte=202:max_malloc_fill_size=2147483647:free_fill_byte=203:max_free_fill_size=2147483647
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -345,6 +352,7 @@ jobs:
- name: Test (offline self-tests under MSan)
env:
MSAN_OPTIONS: abort_on_error=1:halt_on_error=1
timeout-minutes: 20
run: |
set -euo pipefail
# 01_engine-* only; zlib-dependent self-tests are named 01_zlib-* and
@@ -426,6 +434,7 @@ jobs:
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -434,6 +443,36 @@ jobs:
if: failure()
run: cat tests/test-suite.log 2>/dev/null || true
# OpenSSL 4.0 removes the API 3.x deprecated. Building with that set hidden
# keeps the engine ready for it: a deprecated call fails here rather than the
# day a distro moves. Compile-only, since the headers are what is under test.
openssl-no-deprecated:
name: build (OpenSSL, no deprecated API)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
- name: Configure (deprecated OpenSSL API hidden)
run: |
set -euo pipefail
autoreconf -fi
./configure CPPFLAGS=-DOPENSSL_NO_DEPRECATED
# A build with https off would pass this job vacuously.
grep -q "define HTS_USEOPENSSL 1" config.h
- name: Build
run: make -j"$(nproc)"
# Validate the Debian packaging via the same script maintainers release with.
# One amd64/gcc run is enough: packaging (control/rules/manifest/lintian/quilt
# source build) is arch- and compiler-independent, and the build matrix above
@@ -618,7 +657,12 @@ jobs:
set -euo pipefail
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
# Merge base, not the branch tip: master moves during a run, and a
# tip-relative diff blames its new C on this PR.
if ! base="$(git merge-base "origin/${{ github.base_ref }}" HEAD)"; then
echo "::error::no merge base with origin/${{ github.base_ref }}; cannot scope the check."
exit 1
fi
set +e
diff="$(git clang-format --binary clang-format-19 --style=file \
--diff --extensions c,h "$base")"
@@ -641,6 +685,21 @@ jobs:
exit 1 ;;
esac
docs:
name: documentation (chrome in sync, links resolve)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
# The pages carry their navigation as real markup, because they are read from
# disk as often as over http and file:// has no includes. Regenerating has to
# be a no-op, or a page has drifted from tools/doc-chrome.py.
- name: Shared chrome is in sync
run: python3 tools/doc-chrome.py --check
- name: Relative links and anchors resolve
run: python3 tools/doc-links.py html
man-page-sync:
name: man page / html in sync
if: github.event_name == 'pull_request'
@@ -660,6 +719,8 @@ jobs:
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
# Three dots: merge-base scoped, so commits landing on master mid-run
# are not counted as this PR's.
changed="$(git diff --name-only "$base"...HEAD)"
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
if has man/httrack.1 && ! has html/httrack.man.html; then

72
.github/workflows/screenshots.yml vendored Normal file
View File

@@ -0,0 +1,72 @@
# Documentation screenshots: drive the WebHTTrack UI and capture every screen.
# Manual, and deliberately not part of the build gate -- run it after adding or
# changing a UI page, and the artifact is the new set of shots.
name: screenshots
on:
# A push trigger as well as the button: workflow_dispatch is only offered for
# workflows that already exist on the default branch.
push:
paths:
- '.github/workflows/screenshots.yml'
- 'tools/screenshot-walk.py'
workflow_dispatch:
inputs:
lang:
description: 'lang.indexes number to render the UI in (1 = English)'
default: '1'
scale:
description: 'device pixel ratio'
default: '2'
permissions:
contents: read
jobs:
capture:
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Install build dependencies
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
- name: Build
run: |
set -euo pipefail
autoreconf -fi
./configure
make -j"$(nproc)" -C src htsserver
- name: Install the browser
run: |
set -euo pipefail
pip install --quiet playwright
playwright install --with-deps chromium
- name: Capture
env:
LANGN: ${{ inputs.lang || '1' }}
SCALE: ${{ inputs.scale || '2' }}
run: |
set -euo pipefail
# --root is the dist root; the source tree already holds every piece
# htsserver reads, so this needs no install.
python3 tools/screenshot-walk.py --htsserver src/htsserver --root . \
--lang "$LANGN" --scale "$SCALE" --out shots
- name: Upload the screens
if: always()
uses: actions/upload-artifact@v7
with:
name: screenshots
path: shots
if-no-files-found: error

View File

@@ -189,25 +189,55 @@ jobs:
# A wedged crawl must not eat the job's timeout budget. timeout(1)'s
# signals can't reap a native httrack.exe (MSYS signals don't reach it),
# so a hang orphaned processes that starved the runner; run_with_timeout
# TerminateProcess-es the whole tree. 600s clears the slowest multi-pass
# crawl (a few passes at --max-time=120 each).
# TerminateProcess-es the whole tree. 600s is unchanged: it clears the
# 540s a three-pass crawl may legitimately take under local-crawl.sh's
# own watchdogs, against a slowest healthy test here of 39s.
. ./testlib.sh
per_test=600
pass=0 fail=0 skip=0 failed="" skipped=""
for t in 00_runnable.test 01_engine-*.test 01_zlib-*.test \
*_local-*.test 13_crawl_proxy_https.test 58_watchdog.test \
60_crawl-log-salvage.test; do
# The whole suite must give up before the step timeout above. A cancelled
# step keeps neither its log nor the artifacts the later if:always()
# steps would upload, so an overrun that ends in a cancel tells us
# nothing; failing on our own terms keeps both. Healthy runs take 8-9
# min. The check sits between tests, so the step can still reach 25 min
# plus one per-test budget, and that worst case stays inside the 45.
suite_deadline=1500
started=$SECONDS
# Survives into the artifact even if the tail of the step log does not.
progress=suite-progress.log
: >"$progress"
pass=0 fail=0 skip=0 failed="" skipped="" deadline=0
# Globbed, not enumerated: a new NNN_engine-*.test or NNN_local-*.test
# is picked up automatically instead of silently getting zero coverage.
for t in 00_runnable.test *_engine-*.test *_zlib-*.test \
*_local-*.test *_watchdog*.test *_crawl_proxy_https.test \
*_crawl-log-salvage.test; do
elapsed=$((SECONDS - started))
if [ "$elapsed" -ge "$suite_deadline" ]; then
echo "::error::suite deadline: ${elapsed}s elapsed, stopping before $t"
echo "DEADLINE before $t after ${elapsed}s" >>"$progress"
# Per-test start times, so the slow ones are named rather than guessed.
sed 's/^/ /' "$progress"
deadline=1
break
fi
echo "RUN $t at ${elapsed}s" >>"$progress"
rc=0
run_with_timeout 600 bash "$t" >"$t.log" 2>&1 || rc=$?
# Same guard "make check" uses on POSIX, so a wedge is diagnosed the
# same way on every platform. It dumps before it kills, which a bare
# run_with_timeout cannot: by the time that returns, the tree whose
# stack we wanted is already gone.
HTTRACK_TEST_TIMEOUT=$per_test bash ./test-timeout.sh "$t" >"$t.log" 2>&1 || rc=$?
case "$rc" in
0) pass=$((pass + 1)); echo "PASS $t" ;;
77) skip=$((skip + 1)) skipped="$skipped $t"; echo "SKIP $t" ;;
124)
fail=$((fail + 1)) failed="$failed $t"
# test-timeout.sh has already written the process list, the stacks
# and the killed crawl's own logs into $t.log.
echo "FAIL $t (timed out, tree killed)"
# Re-running a wedge traced would just hang again: salvage the
# killed crawl's own logs into the artifact instead.
dump_crawl_logs >>"$t.log"
tail -n 25 "$t.log" | sed 's/^/ /'
;;
*)
@@ -215,23 +245,53 @@ jobs:
echo "FAIL $t (exit $rc)"
# These assert with `test "$(...)" == "..." || exit 1`, which
# says nothing at all on failure. Re-run traced, still bounded.
run_with_timeout 600 bash -x "$t" >>"$t.log" 2>&1 || true
run_with_timeout "$per_test" bash -x "$t" >>"$t.log" 2>&1 || true
tail -n 25 "$t.log" | sed 's/^/ /'
;;
esac
echo "$rc $t" >>"$progress"
# An orphaned native httrack.exe spins and starves the runner, which
# is how this job dies with "lost communication" rather than a plain
# timeout. Clear them between tests and name whoever leaked them.
reap_leftover_processes "$t" | tee -a "$progress"
done
echo "ran=$((pass + fail + skip)) pass=$pass fail=$fail skip=$skip" |
tee -a "$GITHUB_STEP_SUMMARY"
# Every gate here exits 77, so an all-skipped suite would report green having
# tested nothing: pin the skips, and floor the passes in case the glob empties.
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
# webdav-mime needs a reapable background listener, which MSYS cannot give it;
# One name per line, so two branches each appending one don't collide on the
# same line; compared as a sorted set below, so glob discovery order can't
# cause a false mismatch either.
# footer-overflow and purge-longpath skip on Windows (need a path past MAX_PATH);
# crange pending #581;
# webdav-default and webdav-mime need a reapable background listener, which MSYS cannot give them;
# badmtime needs a filesystem that stores an mtime past gmtime's range;
# single-file ends on a GUI half needing htsserver, which this job does not build.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test 94_local-single-file.test"
# single-file ends on a GUI half needing htsserver, which this job does not build;
# update-304-leak needs a LeakSanitizer build, which MSVC has no equivalent of;
# crash-symbolize needs backtrace(), which Windows has no equivalent of.
expected_skips="01_engine-footer-overflow.test
100_local-purge-longpath.test
114_local-update-304-leak.test
120_local-proxytrack-webdav-default.test
48_local-crange-memresume.test
71_local-crange-repaircache.test
79_local-proxytrack-webdav-mime.test
80_engine-crash-symbolize.test
88_local-proxytrack-badmtime.test
94_local-single-file.test"
# First, or the deadline reads as an unexplained shortfall in the gates below.
[ "$deadline" -eq 0 ] || { echo "::error::suite did not finish within ${suite_deadline}s"; exit 1; }
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
[ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
# Word-split on whitespace (space-joined $skipped, newline-joined
# expected_skips both work) and sort, so the compare is a set, not a string.
got=$(printf '%s\n' $skipped | sort)
want=$(printf '%s\n' $expected_skips | sort)
if [ "$got" != "$want" ]; then
echo "::error::skip set changed from expected; - missing, + newly skipped"
diff -u <(echo "$want") <(echo "$got") | tail -n +3 | sed 's/^/ /'
exit 1
fi
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }
- name: Upload the test logs

View File

@@ -19,19 +19,25 @@ the operational checklist: toolchain, invariants, and how to ship a change.
(`request_queue_size`) so macOS/BSD don't drop connections under a parallel
`-c16` bigcrawl the way Python's default backlog of 5 did.
Or run `sh build.sh` to do bootstrap + configure + make in one shot.
- A `tests/NN_*.test` runs only if listed in `tests/Makefile.am`'s `TESTS`; an
- A `tests/NN_*.test` runs only if listed in `tests/tests-list.mk`; an
unregistered file is silently skipped.
- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
as "marker absent". Capture the reply, assert the status line it must carry
(an empty, truncated or redirected one is marker-free too), then match with a
here-string.
- Run teardown with errexit off: `trap 'set +e; cleanup' EXIT`. Under `set -e` a
failing cleanup command becomes the test's exit status (#773). Keep the other
signals on their own `trap` line, or errexit stays off for the rest of the run.
The guard also resets `$?`, so save it first if teardown reads it.
- Never pipe into `grep -q`: it exits on the first match, so whatever the
producer had left to write takes SIGPIPE, and under `pipefail` that becomes
the pipeline's status. `cmd | grep -q M && fail` then never fires and a probe
that proved nothing reads as "marker absent"; `cmd | grep -q M || fail` fails
a test whose marker was present. bash issues one `write()` per line, so any
match that is not on the last line is exposed. Capture the reply, assert the
status line it must carry (an empty, truncated or redirected one is
marker-free too), then match with a here-string: `grep -q M <<<"$reply"`.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every

View File

@@ -1,6 +1,6 @@
AC_PREREQ([2.71])
AC_INIT([httrack], [3.49.14], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_INIT([httrack], [3.49.15], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_COPYRIGHT([
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998-2015 Xavier Roche and other contributors
@@ -29,12 +29,12 @@ AC_CONFIG_SRCDIR(src/httrack.c)
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_HEADERS(config.h)
AM_INIT_AUTOMAKE([subdir-objects])
# 3:6:0: 3.49.14 grows two installed structs from the inside (htsoptstate sits mid-
# httrackp, htsblk mid-lien_back), so the fields after them shift: httrackp.cookies_file
# +8, lien_back.http11 +48. Soname deliberately stays .so.3: the shifted fields are
# 3.49-era additions no external consumer reaches, and a libhttrack3 rename isn't worth it.
# 3:7:0: htsblk gained a tail field and lien_back embeds it by value, so
# lien_back.is_update and everything after it shift +8 (httrackp's own tail growth
# moves nothing). Soname stays .so.3: HTTrackQt is the only consumer of the installed
# headers, so a libhttrack4 rename isn't worth it.
# (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
VERSION_INFO="3:6:0"
VERSION_INFO="3:7:0"
AM_MAINTAINER_MODE
AC_USE_SYSTEM_EXTENSIONS
@@ -173,10 +173,7 @@ AC_CHECK_TYPE(sa_family_t, [], [AC_DEFINE([sa_family_t], [uint16_t], [sa_family_
AX_CHECK_ALIGNED_ACCESS_REQUIRED
# check for various headers
AC_CHECK_HEADERS([execinfo.h sys/ioctl.h sys/random.h])
## CSPRNG for the --single-file mark; /dev/urandom is the fallback
AC_CHECK_FUNCS([getrandom])
AC_CHECK_HEADERS([execinfo.h sys/ioctl.h])
### zlib (mandatory)
CHECK_ZLIB()

10
debian/changelog vendored
View File

@@ -1,3 +1,13 @@
httrack (3.49.15-1) unstable; urgency=medium
* New upstream release: single-file mirrors with inlined assets, sitemap
ingestion, a change report between crawls, and numerous --update, FTP and
parser fixes; full list in history.txt.
* httrack-doc grows by roughly 1 MB: the offline guide now ships screenshots
of the WinHTTrack, WebHTTrack and Android interfaces.
-- Xavier Roche <xavier@debian.org> Thu, 30 Jul 2026 10:40:08 +0200
httrack (3.49.14-1) unstable; urgency=medium
* New upstream release: WARC/WACZ archive output, Windows long-path support,

View File

@@ -1,4 +1,3 @@
<img src="a.png#!htsinline">
<img src=big.png#!htsinline alt=over-cap>
<img src="../escape.png#!htsinline"><img src="/abs.png#!htsinline">
<img src="data:,x#!htsinline">
<img src="a.png">
<img src=big.png alt=over-cap>
<img src="../escape.png"><img src="/abs.png"><img src="data:,x">

View File

@@ -1,3 +1,4 @@
<link rel="stylesheet" href="s.css#!htsinline">
<link rel="icon" href="a.png#!htsinline">
<link rel="canonical" href="other.html#!htsinline">
<link rel="stylesheet" href="s.css">
<link rel=icon href=a.png>
<link rel="next" href="p2.html">
<link rel="preload" href="j.js">

View File

@@ -1,3 +1,5 @@
<p a=1 b=2 c=3 d=4 e=5 f=6 g=7 h=8 i=9 j=10 title="> <img src=a.png#!htsinline>">x</p>
<!--><img src="a.png#!htsinline">
<img src="a.png#!htsinline" >
<img src="unterminated.png
<div style="background:url(a.png">
<style>@import url(
<!-- unterminated comment
<a href=

View File

@@ -0,0 +1 @@
<img src="a.png" a0="v" a1="v" a2="v" a3="v" a4="v" a5="v" a6="v" a7="v" a8="v" a9="v" a10="v" a11="v" a12="v" a13="v" a14="v" a15="v" a16="v" a17="v" a18="v" a19="v" a20="v" a21="v" a22="v" a23="v" a24="v" a25="v" a26="v" a27="v" a28="v" a29="v" a30="v" a31="v" a32="v" a33="v" a34="v" a35="v" a36="v" a37="v" a38="v" a39="v" a40="v" a41="v" a42="v" a43="v" a44="v" a45="v" a46="v" a47="v" a48="v" a49="v" a50="v" a51="v" a52="v" a53="v" a54="v" a55="v" a56="v" a57="v" a58="v" a59="v" a60="v" a61="v" a62="v" a63="v" a64="v" a65="v" a66="v" a67="v" a68="v" a69="v">

View File

@@ -1 +0,0 @@
a.png#!htsinline

View File

@@ -1,5 +0,0 @@
#!htsinline#!htsinline#!htsinline
=#!htsinline=
"#!htsinline"
a.png#!htsinlin
#!htsinlin

View File

@@ -1 +0,0 @@
#!htsinline

View File

@@ -1,3 +1,4 @@
<script src="j.js#!htsinline"></script>
<script>var u="a.png#!htsinline";</script>
<textarea>a.png#!htsinline</textarea>
<script>var s="</scripting>"; if(a</b) x=1;</script>
<script src="j.js"></script>
<textarea></textareas></textarea>
<title></titles></title>

View File

@@ -1 +1,2 @@
<img srcset="a.png#!htsinline 1x, big.png#!htsinline 2x, a.png#!htsinline 480w">
<img srcset="a.png 1x, big.png 2x, a.png 100w">
<source srcset="a.png,, a.png 2x," src="a.png">

View File

@@ -1,2 +1,2 @@
<div style="background:url(a.png#!htsinline)"></div>
<div style='background:url("a.png#!htsinline")'></div>
<div style="background:url(a.png);list-style:url('a.png')"></div>
<p style='background:url("a.png")'>x</p>

View File

@@ -1,4 +1,5 @@
<style>@import "s.css#!htsinline";
@import url(sub/b.css#!htsinline);
div{background:url(a.png#!htsinline)}
<style>@import "s.css";
@import url(sub/b.css);
div{background:url(a.png)}
/* url(a.png) */ p:after{content:"url(a.png)"}
</style>

View File

@@ -1,4 +0,0 @@
<img src="../../../../etc/passwd#!htsinline">
<img src="a.png#!htsinline#frag?q=1">
<img src="sub/../a.png#!htsinline">
<img src="./a.png#!htsinline">

View File

@@ -95,13 +95,10 @@ static void sf_init(void) {
sf_write("a.png", png, sizeof(png) - 1);
sf_write("big.png", big, sizeof(big));
sf_text("j.js", "var x=1;\n");
/* Marked, so an inlined stylesheet recurses into its own marks and its
un-inlinable reference is rebased; unmarked assets leave the target as a
bare scan that reaches nothing. */
sf_text("s.css", "@import url(sub/b.css" SINGLEFILE_MARK ");\n"
"div{background:url(a.png" SINGLEFILE_MARK ")}\n"
"p{background:url(big.png" SINGLEFILE_MARK ")}\n");
sf_text("sub/b.css", "p{background:url(../a.png" SINGLEFILE_MARK ")}\n");
/* @import plus a url(), so an inlined stylesheet recurses and its own
relative reference is rebased. */
sf_text("s.css", "@import url(sub/b.css);\ndiv{background:url(a.png)}\n");
sf_text("sub/b.css", "p{background:url(../a.png)}\n");
}
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {

View File

@@ -4,6 +4,36 @@ HTTrack Website Copier release history:
This file lists all changes and fixes that have been made for HTTrack
3.49-15
+ New: --single-file rewrites each saved page with its assets inlined as data: URIs (#713)
+ New: --changes reports what a crawl added, updated or removed against the previous mirror (#714)
+ New: --sitemap and --sitemap-url ingest sitemaps, so pages nothing links to are still found (#712)
+ New: webhttrack exposes --warc-cdx, --wacz and --warc-max-size (#862)
+ Fixed: --update and --purge-old destroyed a good local copy when the re-fetch got no response, was aborted mid-read, or when the backup meant to protect it failed (#746, #748, #758, #775)
+ Fixed: an FTP re-fetch truncated the mirrored file, resumed a complete mirror with REST and spliced the old body into the new one, and a successful transfer was blanked when its backlog slot was swapped out (#771, #797, #798, #823)
+ Fixed: a chunked response cut at a chunk boundary was stored and cached as complete (#840)
+ Fixed: cache repair deleted the old cache before a rename it never checked, and the unlink-then-rename fallback could lose the destination (#779, #786, #790, #824)
+ Fixed: several backward scans from strlen(s) - 1 read before their buffer on an empty string (#730, #768, #770, #814, #821)
+ Fixed: a URL could be saved onto the engine's own temporary files, and the final path segment was never clamped (#774, #842, #852)
+ Fixed: a query-string character reference the page charset cannot represent was left unescaped, changing how the query parses (#854)
+ Fixed: a second --update pass overwrote the previous WARC and regenerated a page-less WACZ (#759)
+ Fixed: WARC output dropped URLs of 1005 bytes or more, archived nothing for a 304 revisit, and marked an engine-forced not-modified as one the server sent (#778, #785, #826, #838, #839)
+ Fixed: ProxyTrack overflowed its .arc header block, walked past the end of an .ndx buffer, trusted an unparsed offset, and crashed on a PROPFIND or on an entry with no usable Last-Modified (#793, #820, #825, #828)
+ Fixed: webhttrack leaked the session id into crawled pages, overflowed its command line so a quoted value could inject flags, let a posted project path repoint the served root, and built its redirect Location in a 256-byte stack buffer (#700, #706, #707, #710)
+ Fixed: in the web GUI, options ticked on by default could not be un-ticked, "max site size" set a per-file cap instead of the HTML one, and the mirror link on the finished page could not be followed (#708, #709, #725)
+ Fixed: htsserver labelled every PNG as image/gif and offered JPEG as a download, and an unauthenticated GET of a directory spun the server forever (#724, #875)
+ Fixed: webhttrack hung at exit when no mirror had been launched (#753)
+ Fixed: oversized cache and header fields aborted the engine or overflowed a neighbouring field instead of being clipped (#701, #715, #717, #722, #732)
+ Fixed: a -%S rules file of 4 GB or more overran the heap (#702)
+ Fixed: the CLI display did not repaint when the terminal was resized (#97)
+ Fixed: a document with no declared charset double-encoded the title lifted for the local index (#848)
+ Fixed: a fragment on an inlined reference was dropped, losing an SVG sprite selector (#766)
+ Fixed: several time helpers handed out libc's shared gmtime/localtime static rather than a reentrant breakdown (#794, #806)
+ Changed: fatal-signal backtraces name engine frames instead of a bare module and offset (#705)
+ Changed: --without-zlib is rejected at configure time rather than failing at link (#735)
+ Changed: the offline documentation gains one GUI guide with screenshots, an Android option reference, and a restructured index
+ Changed: multiple internal hardening, test and CI improvements
3.49-14
+ New: WARC/1.1 archive output (--warc), with a sorted CDXJ index (--warc-cdx) and WACZ packaging (--wacz), also available from webhttrack (#668)
+ New: -%F takes named footer fields such as {url}, {lastmodified}, {mime}, {charset} and {status} instead of a fixed layout (#667)

View File

@@ -19,7 +19,7 @@ MetaInfodir = $(datadir)/metainfo
# Explicit filenames (e.g. ../history.txt, div/search.sh) resolve via VPATH and
# need no prefix.
HelpHtmlroot_DATA = ../httrack-doc.html ../history.txt
HelpHtml_DATA = $(srcdir)/*.html
HelpHtml_DATA = $(srcdir)/*.html $(srcdir)/*.js $(srcdir)/*.css
HelpHtmldiv_DATA = div/search.sh
HelpHtmlimg_DATA = $(srcdir)/img/*
HelpHtmlimages_DATA = $(srcdir)/images/*
@@ -39,8 +39,7 @@ MetaInfo_DATA = $(srcdir)/server/div/*.metainfo.xml
EXTRA_DIST = $(HelpHtml_DATA) $(HelpHtmlimg_DATA) $(HelpHtmlimages_DATA) \
$(HelpHtmldiv_DATA) $(WebHtml_DATA) $(WebHtmlimages_DATA) \
$(WebPixmap_DATA) $(WebIcon16x16_DATA) $(WebIcon32x32_DATA) $(WebIcon48x48_DATA) \
$(VFolderEntry_DATA) $(MetaInfo_DATA) \
httrack.css
$(VFolderEntry_DATA) $(MetaInfo_DATA)
install-data-hook:
if test ! -L $(DESTDIR)$(prefix)/share/httrack/html ; then \

View File

@@ -1,111 +1,74 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="How to mirror a site without hammering the server, and what a webmaster can do about crawlers that do.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>Using HTTrack responsibly</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h3>For HTTrack users:</h3>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>On this page</h2>
<ul>
<li><a href="#for-httrack-users">For HTTrack users</a></li>
<li><a href="#for-webmasters-having-problems-with-band">For webmasters having problems with bandwidth abuse / other abuses related to HTTrack</a></li>
<li><a href="#please-follow-these-common-sense-rules-t">Please follow these common sense rules to avoid any network abuse</a></li>
<li><a href="#how-to-limit-network-abuse-httrack-websi">How to limit network abuse HTTrack Website Copier FAQ (updated - DRAFT)</a></li>
</ul>
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html" aria-current="page" class="here">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h3 id="for-httrack-users">For HTTrack users:</h3>
<ul>
<li><a href="#USERS">Advice & what <u>not</u> to do when you are using HTTrack</a></li>
</ul>
<br>
<h3>For webmasters having problems with bandwidth abuse / other abuses related to HTTrack:</h3>
<h3 id="for-webmasters-having-problems-with-band">For webmasters having problems with bandwidth abuse / other abuses related to HTTrack:</h3>
<ul>
<li><a href="#WEBMASTERS">Abuse FAQ for <u>webmasters</a></u></li>
</ul>
@@ -121,9 +84,9 @@ h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
<h2 align="center"><em>Advice & what <u>not</u> to do</em></h2>
<h1>Advice & what <u>not</u> to do</h1>
<center><h3>Please follow these <i>common sense</i> rules to avoid any network abuse</h3></center>
<center><h3 id="please-follow-these-common-sense-rules-t">Please follow these <i>common sense</i> rules to avoid any network abuse</h3></center>
<br>
@@ -178,9 +141,9 @@ Downloading a site can overload it, if you have a fast pipe, or if you capture t
<h2 align="center"><em>Abuse FAQ for <u>webmasters</u></em></h2>
<h2 id="abuse-faq-for-webmasters" align="center"><em>Abuse FAQ for <u>webmasters</u></em></h2>
<center><h3>How to limit network abuse
<center><h3 id="how-to-limit-network-abuse-httrack-websi">How to limit network abuse
<br>
HTTrack Website Copier FAQ (updated - DRAFT)
</h3></center>
@@ -338,19 +301,19 @@ Bad: Radical (your site will only be available online for all users), not easy t
How to do: Create fake links with &quot;killing&quot; targets<br>
Example: Use things like &lt;a href=&quot;killme.cgi&quot;&gt;&lt;nothing&gt;&lt;/a&gt;
(again an example in php:)<br>
<tt>
<tt>
<pre>
&lt;?php
// Add IP.
add_temp_firewall_rule($REMOTE_ADDR,"30s");
?&gt;
function add_temp_firewall_rule($addr) {
// The chain chhttp is flushed in a cron job to avoid ipchains overflow
system("/usr/bin/sudo -u root /sbin/ipchains -I 1 chhttp -p tcp -s ".$addr." --dport 80 -j REJECT");
syslog("user rejected due to too many copy attemps : ".$addr);
}
<pre>
&lt;?php
// Add IP.
add_temp_firewall_rule($REMOTE_ADDR,"30s");
?&gt;
function add_temp_firewall_rule($addr) {
// The chain chhttp is flushed in a cron job to avoid ipchains overflow
system("/usr/bin/sudo -u root /sbin/ipchains -I 1 chhttp -p tcp -s ".$addr." --dport 80 -j REJECT");
syslog("user rejected due to too many copy attemps : ".$addr);
}
<pre>
</tt>
<br>
@@ -409,9 +372,9 @@ Example:<br>
&nbsp;&nbsp;&nbsp; FOS('mycompany.com','smith?subject=Hi, John','Click here to email me!')<br>
&nbsp;&nbsp;&nbsp; // --&gt;<br>
&nbsp;&nbsp;&nbsp; &lt;/script&gt;<br>
&nbsp;&nbsp;&nbsp; &lt;noscript&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;smith at mycompany dot com<br>
&nbsp;&nbsp;&nbsp; &lt;/noscript&gt;<br>
&nbsp;&nbsp;&nbsp; &lt;noscript&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;smith at mycompany dot com<br>
&nbsp;&nbsp;&nbsp; &lt;/noscript&gt;<br>
</tt>
<br>
</li><li>Another one is to create images of emails<br>
@@ -566,26 +529,14 @@ And then, put the email address in your pages through:
<br>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

View File

@@ -1,155 +1,10 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<meta charset="utf-8">
<title>Moved to the HTTrack interface guide</title>
<link rel="canonical" href="guide.html#step-address">
<meta http-equiv="refresh" content="0; url=guide.html#step-address">
</head>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<h2 align="center"><em>Add a URL</em></h2>
<br>
<ol>
<li>Enter a typical Web address</li>
<br><small>Just type in your address in the field</small><br><br>
<center><img src="img/addurl1.gif" border="0"></center>
<br><br>OR<br><br>
<li>Enter a Web address with authentication</li>
<br><small>Useful when you need basic authentication to watch the Web page</small><br><br>
<center><img src="img/addurl2.gif" border="0"></center>
<br><br>OR<br><br>
<li>Capture a link from your Web browser to HTTrack</li>
<br><small>Use this tool only for form-based pages (pages delivered after submiting a form) that need some analysis</small><br><br>
<center><img src="img/addurl3.gif" border="0"></center>
<br>
<br>
<small>Set, as explained, your Web browser proxy preferences to the values indicated : set the proxy's address, and the proxy's port,
then click on the button or link as you usually do in your Web browser.
<br>The temporary proxy, installed by HTTrack, will then
capture the link and display a confirmation page.
<br><br>
</small>
<center><img src="img/addurl4.gif" border="0"></center>
<br>
<center><img src="img/addurl5.gif" border="0"></center>
</ol>
<br><br><br><br>
<p align="right">Back to <a href="index.html">Home</a></p>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
</body>
<body><p>Adding a URL is now part of <a href="guide.html#step-address">the HTTrack interface guide</a>.</p></body>
</html>

View File

@@ -1,223 +1,11 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="How to mirror a website with HTTrack on Android: install the app, create a project, enter the address, run the mirror, and browse the result on your device." />
<meta name="keywords" content="httrack, HTTrack, android, offline browser, web mirror utility, website mirroring, mobile, Google Play" />
<title>HTTrack on Android</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<meta charset="utf-8">
<title>Moved to the HTTrack interface guide</title>
<link rel="canonical" href="guide.html">
<meta http-equiv="refresh" content="0; url=guide.html#droid/">
</head>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<h2 align="center"><em>HTTrack on Android</em></h2>
<br>
<p>HTTrack downloads a website to your device so you can read it offline. The
Android app runs the same mirroring engine as the desktop version behind a touch
interface. It needs Android 7.0 or later, and is available on
<a href="https://play.google.com/store/apps/details?id=com.httrack.android">Google Play</a>.</p>
<p>The steps below follow one mirror from start to finish.</p>
<br>
<ol>
<li><b>Grant storage access</b></li>
<br><small>On first launch the app asks for permission to store mirrors on your
device. The prompt reads <em>"Allow HTTrack Website Copier to access photos,
media, and files on your device?"</em>. Tap <b>ALLOW</b>: without it the app
cannot save the downloaded files.</small>
<br><br><center><img src="img/android_permission.png" width="320" alt="First-run storage permission dialog" border="0"></center>
<br><small>If you have used an older release, a second prompt offers to bring its
mirrors into the app. Tap <b>Import</b> to move them, or <b>Not now</b> to
skip.</small>
<br><br><center><img src="img/android_import.png" width="320" alt="Import mirrors from an older version" border="0"></center>
<br><br>
<li><b>Start</b></li>
<br><small>The welcome screen shows the engine version at the bottom. Tap
<b>Next</b> to create a project, or <b>Browse sites</b> to open a mirror you
already made.</small>
<br><br><center><img src="img/android_startup.png" width="320" alt="Welcome screen" border="0"></center>
<br><br>
<li><b>Name the project</b></li>
<br><small>Give the project a name, and optionally a category to group related
mirrors. <b>Base path</b> shows where the files will be written. Tap
<b>Next</b>.</small>
<br><br><center><img src="img/android_project.png" width="320" alt="Project name, category, and base path" border="0"></center>
<br><br>
<li><b>Enter the address</b></li>
<br><small>Type the site address. If a project of the same name already exists,
pick <b>Continue interrupted download</b> or <b>Update existing download</b>.
Tap <b>Options</b> to adjust the crawl, or <b>Start</b> to begin.</small>
<br><br><center><img src="img/android_url.png" width="320" alt="Web address and download action" border="0"></center>
<br><br>
<li><b>Options (optional)</b></li>
<br><small>The <b>Options</b> screen holds the crawl settings, grouped into
eleven tabs. It uses the same profile format as the desktop version, so a tab you
know from WinHTTrack behaves the same way here.</small>
<br><br><center><img src="img/android_options.png" width="320" alt="Options tab list" border="0"></center>
<br><small>The tabs are:</small>
<ul>
<li><small><b>Scan Rules</b>: wildcard filters for the addresses to keep or skip.</small></li>
<li><small><b>Limits</b>: caps on depth, size, time, speed, and number of connections.</small></li>
<li><small><b>Flow Control</b>: simultaneous connections, timeouts, and retries.</small></li>
<li><small><b>Links</b>: how far to follow links, and which extra files to fetch.</small></li>
<li><small><b>Build</b>: how the saved files and folders are named.</small></li>
<li><small><b>Browser ID</b>: the user-agent, language, and headers sent to servers.</small></li>
<li><small><b>Spider</b>: cookies, robots.txt handling, and request behavior.</small></li>
<li><small><b>Proxy address</b>: proxy host and port.</small></li>
<li><small><b>Log, Index, Cache</b>: log files, the search index, and the update cache.</small></li>
<li><small><b>Type/MIME associations</b>: map file extensions to MIME types.</small></li>
<li><small><b>Experts Only</b>: scan mode, how far the crawl may travel, and link rewriting.</small></li>
</ul>
<br><center><img src="img/android_scanrules.png" width="320" alt="Scan Rules tab" border="0"></center>
<br><center><img src="img/android_experts.png" width="320" alt="Experts Only tab" border="0"></center>
<br><small>The <a href="step9.html">desktop option reference</a> describes every
setting in full. A few desktop options are missing or fixed on Android: the proxy
tab has a host and port but no login and password, the download folder is fixed
inside the app's storage so there is no output-path setting, and the MIME table
holds up to eight entries.</small>
<br><br>
<li><b>Run the mirror</b></li>
<br><small>HTTrack downloads the site and reports live figures: bytes saved,
links scanned, transfer rate, and errors. Tap <b>Abort</b> to stop early; a
partial mirror can be resumed later.</small>
<br><br><center><img src="img/android_progress.png" width="320" alt="Crawl in progress with live statistics" border="0"></center>
<br><br>
<li><b>Browse the result</b></li>
<br><small>When the crawl finishes the app reports <b>Success</b> and the mirror
location. Tap <b>Browse Mirrored Website</b> to read the copy in your browser,
<b>View log</b> to see what happened, or <b>New project</b> to start
again.</small>
<br><br><center><img src="img/android_finished.png" width="320" alt="Mirror finished, success" border="0"></center>
<br><br>
<li><b>Where the files are</b></li>
<br><small>Mirrors are written under <tt>/storage/emulated/0/HTTrack/Websites</tt>,
in a folder named after the project. That folder lives in your device's shared
storage, so a file manager or a USB connection can reach it. Opening
<tt>index.html</tt> there browses a mirror without the app.</small>
</ol>
<br><br><br><br>
<p align="right">Back to <a href="index.html">Home</a></p>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
</body>
<body><p>This page is now part of
<a href="guide.html#droid/">the HTTrack interface guide</a>.</p></body>
</html>

View File

@@ -1,293 +1,238 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Cache format specification</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="The format of the HTTrack cache in hts-cache, and how to read it.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>HTTrack cache format</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>Cache format specification</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html" aria-current="page" class="here">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>Cache format specification</h1>
<br />
For updating purpose, HTTrack stores original (untouched) HTML data,
references to downloaded files, and other meta-data (especially parts of the HTTP headers) in a cache,
located in the hts-cache directory. Because local html pages are always modified to "fit" the local
filesystem structure, and because meta-data such as the last-Modified date and Etag can not be stored
with the associated files, the cache is absolutely mandatory for reprocessing (update/continue) phases.
<br /><br />
<h3>The (new) cache.zip format</h3>
The 3.31 release of HTTrack introduces a new cache format, more extensible and efficient than the previous one (ndx/dat format).
The main advantages of this cache are:
<ul>
<li>One single file for a complete website cache archive</li>
<li>Standard <a href="http://www.pkware.com/products/enterprise/white_papers/appnote.txt" target="_new">ZIP</a> format, that can be easily reused on most platforms and languages</li>
<li>Compressed data with the efficient and opened <a href="http://www.gzip.org/zlib/" target="_new">zlib</a> format</li>
</ul>
The cache is made of ZIP files entries ; with one ZIP file entry per fetched URL (successfully or not - errors are also stored).<br />
For each entry:
<ul>
<li>The ZIP file name is the original URL [<small><a href="#orig">see notes below</a></small>]</li>
<li>The ZIP file contents, <b>if available</b>, is the original (compressed, using the deflate algorythm) data</li>
<li>The ZIP file extra field (in the local file header) contains a list of meta-fields, very similar to the <a href="http://www.ietf.org/rfc/rfc2616.txt?number=2616" target="new_">HTTP</a> headers fields. See also <a href="http://www.ietf.org/rfc/rfc2396.txt?number=2396" target="new_">RFC</a>.</li><br />
<li>The ZIP file timestamp follows the "Last-Modified-Since" field given for this URL, if any</li>
</ul>
Example of cache file:
<br />
<pre>
$ unzip -l hts-cache/new.zip
Archive: hts-cache/new.zip
HTTrack Website Copier/3.31-ALPHA-4 mirror complete in 3 seconds : 5 links scanned,
3 files written (16109 bytes overall) [17690 bytes received at 5896 bytes/sec]
(1 errors, 0 warnings, 0 messages)
Length Date Time Name
-------- ---- ---- ----
94 07-18-03 08:59 http://www.httrack.com/robots.txt
9866 01-17-04 01:09 http://www.httrack.com/html/cache.html
0 05-11-03 13:31 http://www.httrack.com/html/images/bg_rings.gif
207 01-19-04 05:49 http://www.httrack.com/html/fade.gif
0 05-11-03 13:31 http://www.httrack.com/html/images/header_title_4.gif
-------- -------
10167 5 files
</pre>
Example of cache file meta-data:
<br />
<pre>
HTTP/1.1 200 OK
X-In-Cache: 1
X-StatusCode: 200
X-StatusMessage: OK
X-Size: 94
Content-Type: text/plain
Last-Modified: Fri, 18 Jul 2003 08:59:11 GMT
Etag: "40ebb5-5e-3f17b6df"
X-Addr: www.httrack.com
X-Fil: /robots.txt
</pre>
There are also specific issues regarding this format:
<ul>
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
</ul>
<br />
<b>Meta-data stored in the "extra field" of the local file headers</b><br />
The extra field is composed of text data, and this text data is composed of distinct lines of headers.
The end of text, <b>or</b> a double CR/LF, mark the end of this zone.
This method allows you to optionally store original HTTP headers just after the "meta-data" headers for informational use.<br />
<br />
<b>The status line (the first headers line)</b><br />
Status-Line = HTTP-Version SP Status-Code SP X-Reason-Phrase CRLF<br />
<br />
<b>Other lines:</b><br />
<br />
<b>Specific fields:</b><br />
<ul>
<li>X-In-Cache</li><br>
Indicates if the data are present (value=1) in the cache (that is, as ZIP data), or in an external file (value=0).
This field MUST be the first field.
<li>X-StatusCode</li><br>
The modified (by httrack) status code after processing. 304 error codes ("Not modified"), for example, are transformed into "200" codes after processing.
<li>X-StatusMessage</li><br>
The modified (by httrack) status message.
<li>X-Size</li><br>
The stored (either in cache, or in an external file) data size.
<li>X-Charset</li><br>
The original charset.
<li>X-Addr</li><br>
The original URL address part.
<li>X-Fil</li><br>
The original URL path part.
<li>X-Save</li><br>
The local filename, depending on user's "build structure" preferences.
</ul>
<br />
<b>Standard (RFC 2616) "useful" fields:</b><br />
<ul>
<li>Content-Type</li>
<li>Last-Modified</li>
<li>Etag</li>
<li>Location</li>
<li>Content-Disposition</li>
</ul>
<br />
<b>Specific fields in "BNF-like" grammar:</b><br />
<pre>
X-In-Cache = "X-In-Cache" ":" 1*DIGIT
X-StatusCode = "X-StatusCode" ":" 1*DIGIT
X-StatusMessage = "X-StatusMessage" ":" *&lt;TEXT, excluding CR, LF&gt;
X-Size = "X-Size" ":" 1*DIGIT
X-Charset = "X-Charset" ":" value
X-Addr = "X-Addr" ":" scheme ":" "//" authority
X-Fil = "X-Fil" ":" rel_path
X-Save = "X-Save" ":" rel_path
</pre>
<b>RFC standard fields:</b><br />
<pre>
Content-Type = "Content-Type" ":" media-type
Last-Modified = "Last-Modified" ":" HTTP-date
Etag = "ETag" ":" entity-tag
Location = "Location" ":" absoluteURI
Content-Disposition = "Content-Disposition" ":" disposition-type *( ";" disposition-parm )
</pre>
<br />
And, for your information,
<pre>
X-Reason-Phrase = *&lt;TEXT, with a maximum of 32 characters, and excluding CR, LF&gt;
</pre>
<a name="orig">Note: Because the URLs may have an unexpected format, especially with double "/" inside, and other reserved characters ("?", "&" ..),
various ZIP uncompressors can potentially have troubles accessing or decompressing the data.
Libraries should generally handle this peculiar format, however.
</a>
For updating purpose, HTTrack stores original (untouched) HTML data,
references to downloaded files, and other meta-data (especially parts of the HTTP headers) in a cache,
located in the hts-cache directory. Because local html pages are always modified to "fit" the local
filesystem structure, and because meta-data such as the last-Modified date and Etag can not be stored
with the associated files, the cache is absolutely mandatory for reprocessing (update/continue) phases.
<br /><br />
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<h3 id="the-new-cache-zip-format">The (new) cache.zip format</h3>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
The 3.31 release of HTTrack introduces a new cache format, more extensible and efficient than the previous one (ndx/dat format).
The main advantages of this cache are:
<ul>
<li>One single file for a complete website cache archive</li>
<li>Standard <a href="http://www.pkware.com/products/enterprise/white_papers/appnote.txt" target="_new">ZIP</a> format, that can be easily reused on most platforms and languages</li>
<li>Compressed data with the efficient and opened <a href="http://www.gzip.org/zlib/" target="_new">zlib</a> format</li>
</ul>
The cache is made of ZIP files entries ; with one ZIP file entry per fetched URL (successfully or not - errors are also stored).<br />
For each entry:
<ul>
<li>The ZIP file name is the original URL [<small><a href="#orig">see notes below</a></small>]</li>
<li>The ZIP file contents, <b>if available</b>, is the original (compressed, using the deflate algorythm) data</li>
<li>The ZIP file extra field (in the local file header) contains a list of meta-fields, very similar to the <a href="http://www.ietf.org/rfc/rfc2616.txt?number=2616" target="new_">HTTP</a> headers fields. See also <a href="http://www.ietf.org/rfc/rfc2396.txt?number=2396" target="new_">RFC</a>.</li><br />
<li>The ZIP file timestamp follows the "Last-Modified-Since" field given for this URL, if any</li>
</ul>
Example of cache file:
<br />
<pre>
$ unzip -l hts-cache/new.zip
Archive: hts-cache/new.zip
HTTrack Website Copier/3.31-ALPHA-4 mirror complete in 3 seconds : 5 links scanned,
3 files written (16109 bytes overall) [17690 bytes received at 5896 bytes/sec]
(1 errors, 0 warnings, 0 messages)
Length Date Time Name
-------- ---- ---- ----
94 07-18-03 08:59 http://www.httrack.com/robots.txt
9866 01-17-04 01:09 http://www.httrack.com/html/cache.html
0 05-11-03 13:31 http://www.httrack.com/html/images/bg_rings.gif
207 01-19-04 05:49 http://www.httrack.com/html/fade.gif
0 05-11-03 13:31 http://www.httrack.com/html/images/header_title_4.gif
-------- -------
10167 5 files
</pre>
Example of cache file meta-data:
<br />
<pre>
HTTP/1.1 200 OK
X-In-Cache: 1
X-StatusCode: 200
X-StatusMessage: OK
X-Size: 94
Content-Type: text/plain
Last-Modified: Fri, 18 Jul 2003 08:59:11 GMT
Etag: "40ebb5-5e-3f17b6df"
X-Addr: www.httrack.com
X-Fil: /robots.txt
</pre>
There are also specific issues regarding this format:
<ul>
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
</ul>
<br />
<b>Meta-data stored in the "extra field" of the local file headers</b><br />
The extra field is composed of text data, and this text data is composed of distinct lines of headers.
The end of text, <b>or</b> a double CR/LF, mark the end of this zone.
This method allows you to optionally store original HTTP headers just after the "meta-data" headers for informational use.<br />
<br />
<b>The status line (the first headers line)</b><br />
Status-Line = HTTP-Version SP Status-Code SP X-Reason-Phrase CRLF<br />
<br />
<b>Other lines:</b><br />
<br />
<b>Specific fields:</b><br />
<ul>
<li>X-In-Cache</li><br>
Indicates if the data are present (value=1) in the cache (that is, as ZIP data), or in an external file (value=0).
This field MUST be the first field.
<li>X-StatusCode</li><br>
The modified (by httrack) status code after processing. 304 error codes ("Not modified"), for example, are transformed into "200" codes after processing.
<li>X-StatusMessage</li><br>
The modified (by httrack) status message.
<li>X-Size</li><br>
The stored (either in cache, or in an external file) data size.
<li>X-Charset</li><br>
The original charset.
<li>X-Addr</li><br>
The original URL address part.
<li>X-Fil</li><br>
The original URL path part.
<li>X-Save</li><br>
The local filename, depending on user's "build structure" preferences.
</ul>
<br />
<b>Standard (RFC 2616) "useful" fields:</b><br />
<ul>
<li>Content-Type</li>
<li>Last-Modified</li>
<li>Etag</li>
<li>Location</li>
<li>Content-Disposition</li>
</ul>
<br />
<b>Specific fields in "BNF-like" grammar:</b><br />
<pre>
X-In-Cache = "X-In-Cache" ":" 1*DIGIT
X-StatusCode = "X-StatusCode" ":" 1*DIGIT
X-StatusMessage = "X-StatusMessage" ":" *&lt;TEXT, excluding CR, LF&gt;
X-Size = "X-Size" ":" 1*DIGIT
X-Charset = "X-Charset" ":" value
X-Addr = "X-Addr" ":" scheme ":" "//" authority
X-Fil = "X-Fil" ":" rel_path
X-Save = "X-Save" ":" rel_path
</pre>
<b>RFC standard fields:</b><br />
<pre>
Content-Type = "Content-Type" ":" media-type
Last-Modified = "Last-Modified" ":" HTTP-date
Etag = "ETag" ":" entity-tag
Location = "Location" ":" absoluteURI
Content-Disposition = "Content-Disposition" ":" disposition-type *( ";" disposition-parm )
</pre>
<br />
And, for your information,
<pre>
X-Reason-Phrase = *&lt;TEXT, with a maximum of 32 characters, and excluding CR, LF&gt;
</pre>
<a name="orig">Note: Because the URLs may have an unexpected format, especially with double "/" inside, and other reserved characters ("?", "&" ..),
various ZIP uncompressors can potentially have troubles accessing or decompressing the data.
Libraries should generally handle this peculiar format, however.
</a>
<br /><br />
<!-- doc-chrome:bottom -->
</main>
</div>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

View File

@@ -1,105 +1,70 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Change report format specification</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="The change report HTTrack writes after an update, field by field.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>HTTrack change report format</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>Change report format specification</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>On this page</h2>
<ul>
<li><a href="#what-changed-means">What "changed" means</a></li>
<li><a href="#with-the-cache-off">With the cache off</a></li>
<li><a href="#fields">Fields</a></li>
<li><a href="#encoding">Encoding</a></li>
<li><a href="#example">Example</a></li>
<li><a href="#notes">Notes</a></li>
</ul>
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html" aria-current="page" class="here">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>Change report format specification</h1>
<br />
@@ -111,7 +76,7 @@ summary of the same counts.
<br /><br />
<h3>What "changed" means</h3>
<h3 id="what-changed-means">What "changed" means</h3>
A resource is changed when its bytes differ, not when the server merely re-sent
it. HTTrack compares the payload it just received against the copy the previous
@@ -129,7 +94,7 @@ server that answers 200 rather than 304 reads as changed. Keeping the cache on
<br /><br />
<h3>With the cache off</h3>
<h3 id="with-the-cache-off">With the cache off</h3>
<tt>--cache=0</tt> costs the report more than the digest of a parsed page. The
mirror's file index (<tt>hts-cache/new.lst</tt>) is what records which files a
@@ -142,7 +107,7 @@ compare against and fall back to the transfer signal.
<br /><br />
<h3>Fields</h3>
<h3 id="fields">Fields</h3>
<ul>
<li><tt>schema</tt>: format version, currently <tt>1</tt>. It is bumped only
@@ -179,7 +144,7 @@ copy the previous run left.</li>
<br />
<h3>Encoding</h3>
<h3 id="encoding">Encoding</h3>
The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are
not guaranteed to be valid UTF-8; any byte sequence that is not becomes
@@ -188,7 +153,7 @@ rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
<br /><br />
<h3>Example</h3>
<h3 id="example">Example</h3>
<pre>
{
@@ -216,7 +181,7 @@ rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
<br /><br />
<h3>Notes</h3>
<h3 id="notes">Notes</h3>
<ul>
<li>A file listed under <tt>gone</tt> is only deleted when <tt>--purge-old</tt> is
@@ -239,25 +204,14 @@ scope: both change what a mirror directory contains.</li>
<br /><br />
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

10
html/cmddoc.html Normal file
View File

@@ -0,0 +1,10 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Moved to the command-line guide</title>
<link rel="canonical" href="cmdguide.html">
<meta http-equiv="refresh" content="0; url=cmdguide.html">
</head>
<body><p>This page is now part of <a href="cmdguide.html">the HTTrack command-line guide</a>.</p></body>
</html>

View File

@@ -1,102 +1,75 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="A task-oriented guide to the httrack command line: scope, filters, limits, logins, proxies, updates and recipes.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>HTTrack command-line guide</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>Command-Line Guide</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>On this page</h2>
<ul>
<li><a href="#quickstart">1. Quick start</a></li>
<li><a href="#scope">2. Scope: how far the crawl reaches</a></li>
<li><a href="#filters">3. Filters and scan rules</a></li>
<li><a href="#limits">4. Limits and politeness</a></li>
<li><a href="#names">5. File names and types</a></li>
<li><a href="#links">6. Links and page building</a></li>
<li><a href="#identity">7. Identity, cookies and login</a></li>
<li><a href="#proxy">8. Proxy and network</a></li>
<li><a href="#update">9. Update and cache</a></li>
<li><a href="#experts">10. Experts and scripting</a></li>
<li><a href="#recipes">11. Recipes</a></li>
</ul>
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html" aria-current="page" class="here">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>Command-Line Guide</h1>
<p>This is a task-oriented guide to the <tt>httrack</tt> command line: how to do the
things people actually ask for, and the handful of defaults that surprise
@@ -112,21 +85,6 @@ the log. Every project writes <tt>hts-log.txt</tt> (and <tt>hts-err.txt</tt>) in
its output directory, and those files name every URL that was refused, redirected,
or filtered out. Read them first.</p>
<h4>On this page</h4>
<ul class="tblNoBorder">
<li><a href="#quickstart">1. Quick start</a></li>
<li><a href="#scope">2. Scope: how far the crawl reaches</a></li>
<li><a href="#filters">3. Filters and scan rules</a></li>
<li><a href="#limits">4. Limits and politeness</a></li>
<li><a href="#names">5. File names and types</a></li>
<li><a href="#links">6. Links and page building</a></li>
<li><a href="#identity">7. Identity, cookies and login</a></li>
<li><a href="#proxy">8. Proxy and network</a></li>
<li><a href="#update">9. Update and cache</a></li>
<li><a href="#experts">10. Experts and scripting</a></li>
<li><a href="#recipes">11. Recipes</a></li>
</ul>
<h3 id="quickstart">1. Quick start</h3>
<p>A mirror is one command: a start URL and an output directory.</p>
@@ -156,7 +114,7 @@ you write. Get this right and most "it downloaded too much" or "it only grabbed
the index" problems disappear.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--depth (-r)</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
<tr><td><tt>--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
<tr><td><tt>--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
@@ -242,7 +200,7 @@ options will not help there. That is an
bracket forms match narrower sets. The full table, with size and mime rules, is on
<a href="filters.html">the filters page</a>.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
<tr class="head"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> &mdash; any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> &mdash; a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
@@ -261,7 +219,7 @@ accident, and the <a href="abuse.html">abuse page</a> is worth a read. The limit
below let you go faster when you own the target, and slower when you do not.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--max-rate (-A)</tt></td><td>Maximum transfer rate in bytes/sec. <b>The default is about 100 KB/s even without this flag.</b> Raise it to go faster.</td></tr>
<tr><td><tt>--sockets (-c)</tt></td><td>Number of parallel connections (default 4). <tt>--tiny</tt>, <tt>--wide</tt> and <tt>--ultrawide</tt> are presets.</td></tr>
<tr><td><tt>--connection-per-second (-%c)</tt></td><td>New connections opened per second (default 5).</td></tr>
@@ -288,7 +246,7 @@ usually about a URL like <tt>/article?id=42</tt> or a <tt>.php</tt> page that is
really HTML.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
<tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
<tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
@@ -312,7 +270,7 @@ naming.</p>
ones it kept so the local copy browses offline. These options tune both halves.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--keep-links (-K)</tt></td><td>How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare <tt>-K</tt> keeps <b>absolute</b> URLs, and <tt>-K0</tt> is the <b>relative</b> default. <tt>-K3</tt> keeps absolute URIs, <tt>-K4</tt> keeps the original links.</td></tr>
<tr><td><tt>--replace-external (-x), --generate-errors (-o)</tt></td><td>Replace external links with an error page, and generate an error page for links that failed.</td></tr>
<tr><td><tt>--preserve (-%p), --disable-passwords (-%x)</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
@@ -338,7 +296,7 @@ server), and obeys robots. Plenty of sites filter exactly that profile. These
options control what HTTrack says about itself.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--user-agent (-F)</tt></td><td>The <tt>User-Agent</tt>. Set a browser string to get past crawler blocks; <tt>--user-agent ""</tt> sends none.</td></tr>
<tr><td><tt>--referer (-%R), --from (-%E), --language (-%l), --accept (-%a)</tt></td><td>Referer, From, Accept-Language and Accept headers.</td></tr>
<tr><td><tt>--headers (-%X)</tt></td><td>Add raw header lines to every request.</td></tr>
@@ -369,7 +327,7 @@ across links at the same time.</p>
<h3 id="proxy">8. Proxy and network</h3>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
<tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
<tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
@@ -392,7 +350,7 @@ updating possible. It is not a size-limited scratch area you can delete: throw i
away and you lose the ability to continue or update the mirror.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--continue</tt></td><td>Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.</td></tr>
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
<tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
@@ -415,7 +373,7 @@ dynamic pages to look "changed" on every run. See the
mirror behavior and expose the engine.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr class="head"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--get URL</tt></td><td>Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.</td></tr>
<tr><td><tt>--spider --testlinks --skeleton</tt></td><td>Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.</td></tr>
<tr><td><tt>--userdef-cmd (-V)</tt></td><td>Run a shell command on each downloaded file; <tt>$0</tt> is the file path. Good for on-the-fly processing.</td></tr>
@@ -540,24 +498,14 @@ all disabled.</small></p>
<a href="cache.html">cache</a>; for the change report, see
<a href="changes.html">changes</a>.</p>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

View File

@@ -1,120 +1,75 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<script language="javascript">
<!--
function FOS(host,nom,info) {
var s;
if (info == "") info=nom+"@"+host;
s="mail";
document.write("<a href='"+s+"to:"+nom+"@"+host+"'>"+info+"</a>");
}
// -->
</script>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="How to reach the HTTrack project, and who contributed to it.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>Contact and credits</title>
<script language="javascript">
<!--
function FOS(host,nom,info) {
var s;
if (info == "") info=nom+"@"+host;
s="mail";
document.write("<a href='"+s+"to:"+nom+"@"+host+"'>"+info+"</a>");
}
// -->
</script>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>Contact Us!</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html" aria-current="page" class="here">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>Contact Us!</h1>
<br>
<h3>
<h3 id="please-visit-our-website">
<center>
<a href="http://www.httrack.com/" target="_blank">Please visit our website!</a>
<br>
@@ -127,94 +82,94 @@ h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
<pre>
Informations on this project:
This project has been developed by:
Xavier Roche (xroche at httrack dot com)
for the main engine and Windows interface
and maintainer for v2.0 and v3.0
Yann Philippot (yphilippot at lemel dot fr)
past contributor (java binary .class parser)
David Lawrie (dalawrie at lineone dot net)
Robert Lagadec (rlagadec at yahoo dot fr)
for checking both English & French translations
Juan Pablo Barrio Lera (University of León)
for Spanish translations
Rainer Klueting (rainer at klueting dot de)
Bastian Gorke (bastiang at yahoo dot com)
Rudi Ferrari (Wyando at netcologne dot de)
Marcus Gaza (MarcusGaza at t-online dot de)
for German translations
Rudi Ferrari (Wyando at netcologne dot de)
for Dutch translations
Lukasz Jokiel (Opole University of Technology, Lukasz dot Jokiel at po dot opole dot pl)
for Polish translations
Rui Fernandes (CANTIC, ruiefe at mail dot malhatlantica dot pt)
Pedro T dot Pinheiro (Universidade Nova de Lisboa-FCT, ptiago at mail dot iupi dot pt)
for Portuguese translations
Andrei Iliev (iliev at vitaplus dot ru)
for Russian translations
Witold Krakowski (wtkrak at netscape dot net )
for Italian translations
Jozsef Tamas Herczeg (hdodi at freemail dot hu)
for Hungarian translation
Paulo Neto (company at layout dot com dot br)
for Brazilian translation
Brook Qin (brookqwr at sina dot com)
for simplified Chinese translation
David Hing Cheong Hung (DAVEHUNG at mtr dot com dot hk)
Addy Lin (addy1975 at pchome dot com dot tw)
for traditional Chinese translation
Jesper Bramm (bramm at get2net dot dk)
for Danish translation
Tõnu Virma
for Estonian translation
Staffan Ström (staffan at fam-strom dot org)
for Swedish translation
Mehmet Akif Köeoðlu (mak at ttnet dot net dot tr)
for Turkish translation
Aleksandar Savic (aleks at macedonia dot eu dot org)
for Macedonian translation
Takayoshi Nakasikiryo
for Japanese translation
Martin Sereday (sereday at slovanet dot sk)
for Slovak translation
Antonín Matìjèík (matejcik at volny dot cz)
for Czech translation
Andrij Shevchuk (http://programy dot com dot ua)
for Ukrainian translation
Tobias "Spug" Langhoff (spug_enigma at hotmail dot com)
for Norwegian translation
Jadran Rudeciur (jrudec at email dot si)
for Slovenian translation
Alin Gheorghe Miron (miron dot alin at personal dot ro)
for Romanian translation
Michael Papadakis (mikepap at freemail dot gr)
for Greek translation
Thanks to:
Leto Kauler (molotov at tasmail dot com)
for the site/logos design
Special Thanks to:
Patrick Ducrot & Daniel Carré (ENSI of Caen)
for their initial support
Fred Cohen (fc at all dot net)
for HTTrack user's guide
Greetings to:
Christian Marillat (marillat dot christian at wanadoo dot fr)
for autoconf compliance and .deb package
Jean-loup Gailly from gzip dot org
for the help provided with Zlib
Eric A dot Young (eay at cryptsoft dot com)
for SSL cryptographic material (OpenSSL)
Russ Freeman from gipsymedia
for hints on DLL dynamic load
Paul DiLascia
for helping to fix problems with CHtmlView
ISMRA/Ensi of Caen
for their initial support
.. and all users that are using and supporting HTTrack!
This project has been developed by:
Xavier Roche (xroche at httrack dot com)
for the main engine and Windows interface
and maintainer for v2.0 and v3.0
Yann Philippot (yphilippot at lemel dot fr)
past contributor (java binary .class parser)
David Lawrie (dalawrie at lineone dot net)
Robert Lagadec (rlagadec at yahoo dot fr)
for checking both English & French translations
Juan Pablo Barrio Lera (University of León)
for Spanish translations
Rainer Klueting (rainer at klueting dot de)
Bastian Gorke (bastiang at yahoo dot com)
Rudi Ferrari (Wyando at netcologne dot de)
Marcus Gaza (MarcusGaza at t-online dot de)
for German translations
Rudi Ferrari (Wyando at netcologne dot de)
for Dutch translations
Lukasz Jokiel (Opole University of Technology, Lukasz dot Jokiel at po dot opole dot pl)
for Polish translations
Rui Fernandes (CANTIC, ruiefe at mail dot malhatlantica dot pt)
Pedro T dot Pinheiro (Universidade Nova de Lisboa-FCT, ptiago at mail dot iupi dot pt)
for Portuguese translations
Andrei Iliev (iliev at vitaplus dot ru)
for Russian translations
Witold Krakowski (wtkrak at netscape dot net )
for Italian translations
Jozsef Tamas Herczeg (hdodi at freemail dot hu)
for Hungarian translation
Paulo Neto (company at layout dot com dot br)
for Brazilian translation
Brook Qin (brookqwr at sina dot com)
for simplified Chinese translation
David Hing Cheong Hung (DAVEHUNG at mtr dot com dot hk)
Addy Lin (addy1975 at pchome dot com dot tw)
for traditional Chinese translation
Jesper Bramm (bramm at get2net dot dk)
for Danish translation
Tõnu Virma
for Estonian translation
Staffan Ström (staffan at fam-strom dot org)
for Swedish translation
Mehmet Akif Köeoðlu (mak at ttnet dot net dot tr)
for Turkish translation
Aleksandar Savic (aleks at macedonia dot eu dot org)
for Macedonian translation
Takayoshi Nakasikiryo
for Japanese translation
Martin Sereday (sereday at slovanet dot sk)
for Slovak translation
Antonín Matìjèík (matejcik at volny dot cz)
for Czech translation
Andrij Shevchuk (http://programy dot com dot ua)
for Ukrainian translation
Tobias "Spug" Langhoff (spug_enigma at hotmail dot com)
for Norwegian translation
Jadran Rudeciur (jrudec at email dot si)
for Slovenian translation
Alin Gheorghe Miron (miron dot alin at personal dot ro)
for Romanian translation
Michael Papadakis (mikepap at freemail dot gr)
for Greek translation
Thanks to:
Leto Kauler (molotov at tasmail dot com)
for the site/logos design
Special Thanks to:
Patrick Ducrot & Daniel Carré (ENSI of Caen)
for their initial support
Fred Cohen (fc at all dot net)
for HTTrack user's guide
Greetings to:
Christian Marillat (marillat dot christian at wanadoo dot fr)
for autoconf compliance and .deb package
Jean-loup Gailly from gzip dot org
for the help provided with Zlib
Eric A dot Young (eay at cryptsoft dot com)
for SSL cryptographic material (OpenSSL)
Russ Freeman from gipsymedia
for hints on DLL dynamic load
Paul DiLascia
for helping to fix problems with CHtmlView
ISMRA/Ensi of Caen
for their initial support
.. and all users that are using and supporting HTTrack!
</pre>
<hr>
@@ -225,46 +180,35 @@ ISMRA/Ensi of Caen
You can also contact by email, but due to the large volume of messages, it is impossible
to always respond (especially for configuration help or other configuration-related
questions).
<br>
<script language="javascript">
<!--
FOS('httrack.com','httrack','httrack at httrack dot com (HTTrack project)');
document.write('<br>');
FOS('httrack.com','roche','roche at httrack dot com (Xavier ROCHE)');
// -->
</script>
<noscript>
httrack at httrack dot com (HTTrack project)<br>
roche at httrack dot com (Xavier ROCHE)<br>
</noscript>
<br>
<script language="javascript">
<!--
FOS('httrack.com','httrack','httrack at httrack dot com (HTTrack project)');
document.write('<br>');
FOS('httrack.com','roche','roche at httrack dot com (Xavier ROCHE)');
// -->
</script>
<noscript>
httrack at httrack dot com (HTTrack project)<br>
roche at httrack dot com (Xavier ROCHE)<br>
</noscript>
<br><hr><br>
<br>
This program is covered by the GNU General Public License.<br>
HTTrack/HTTrack Website Copier is Copyright (C) 1998-2026 Xavier Roche and other contributors
<br>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

View File

@@ -1,161 +1,111 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Using HTTrack from a script, from the callbacks, or through the libhttrack library.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>Programming with HTTrack</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>HTTrack Programming page</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>On this page</h2>
<ul>
<li><a href="#programming">Programming</a></li>
<li><a href="#technical-references">Technical references</a></li>
</ul>
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html" aria-current="page" class="here">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>HTTrack Programming page</h1>
<br>
HTTrack can be used as a third-party program in batch files, or as library.
HTTrack can be used as a third-party program in batch files, or as library.
Depending on your needs, you may look:
<br><br>
<h3>Programming</h3>
<ul>
<h3 id="programming">Programming</h3>
<ul>
<li><a href="scripting.html">Simple use: scripting examples using the httrack commandline program</a></li><br>
You can use tens of options (see <tt>httrack --help</tt>) to control precisely the mirror, and used embedded commands in httrack.
<br><br>
<li><a href="plug.html">More complex use: plugging external C functions to the httrack library</a></li><br>
For advanced functions, you may have to use external C wrappers ; for example when adding advanced crawl features, such as "tuned" filestructure type
<br><i>Important note: please read the license information of httrack before developing add-ons</i><br />
See also the page <a href="plug_330.html">for versions prior to 3.41</a>
<br><br>
<li><a href="library.html">Advanced use: using the library</a></li><br>
The library can be used to write graphical GUIs for httrack, or to run mirrors from a program.
<br><i>Important note: please read the license information of httrack before developing add-ons</i>
</ul>
<h3>Technical references</h3>
<ul>
<li><a href="cache.html">Cache format</a></li><br>
HTTrack stores original HTML data and references to downloaded files in a cache, located in the hts-cache directory.
This page describes the HTTrack cache format.
<li><a href="changes.html">Change report format</a></li><br>
With --changes, HTTrack writes hts-changes.json describing what the crawl left new, changed, unchanged and gone
compared to the previous mirror. This page describes that file.
</ul>
You can use tens of options (see <tt>httrack --help</tt>) to control precisely the mirror, and used embedded commands in httrack.
<br><br>
<li><a href="plug.html">More complex use: plugging external C functions to the httrack library</a></li><br>
For advanced functions, you may have to use external C wrappers ; for example when adding advanced crawl features, such as "tuned" filestructure type
<br><i>Important note: please read the license information of httrack before developing add-ons</i><br />
See also the page <a href="plug_330.html">for versions prior to 3.41</a>
<br><br>
<li><a href="library.html">Advanced use: using the library</a></li><br>
The library can be used to write graphical GUIs for httrack, or to run mirrors from a program.
<br><i>Important note: please read the license information of httrack before developing add-ons</i>
</ul>
<h3 id="technical-references">Technical references</h3>
<ul>
<li><a href="cache.html">Cache format</a></li><br>
HTTrack stores original HTML data and references to downloaded files in a cache, located in the hts-cache directory.
This page describes the HTTrack cache format.
<li><a href="changes.html">Change report format</a></li><br>
With --changes, HTTrack writes hts-changes.json describing what the crawl left new, changed, unchanged and gone
compared to the previous mirror. This page describes that file.
</ul>
<br><br>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

341
html/doc.css Normal file
View File

@@ -0,0 +1,341 @@
/* Shared chrome for the HTTrack documentation pages. Page-specific rules live
beside the page (guide.css). Read from disk as often as over http, so
everything here has to work from a file:// URL. */
:root {
--field: #77b;
--panel: #ccd;
--accent: #99c;
--ink: #111;
--ink-soft: #444;
--rule: #aab;
--link: #00c;
--link-hover: #c00;
--code-bg: #e8e8f2;
--chip: #dde;
--shadow: rgba(0, 0, 0, .25);
}
@media (prefers-color-scheme: dark) {
:root {
--field: #171a2b;
--panel: #22263a;
--accent: #3b4066;
--ink: #e6e6ee;
--ink-soft: #a9adc4;
--rule: #3b4066;
--link: #9db4ff;
--link-hover: #ff9d9d;
--code-bg: #14172a;
--chip: #2e3350;
--shadow: rgba(0, 0, 0, .6);
}
}
* { box-sizing: border-box; }
/* The plain declaration before each var() is what IE11 gets: it drops the
custom-property line it cannot parse and keeps the literal above it. */
body {
margin: 0;
padding: 0 1rem 3rem;
background: #77b;
background: var(--field);
color: #111;
color: var(--ink);
font: 1em/1.6 "Trebuchet MS", Verdana, system-ui, -apple-system, "Segoe UI", sans-serif;
}
/* ---- page frame ---- */
.masthead, .wrap {
max-width: 68rem;
margin: 0 auto;
}
.masthead {
padding: .6rem 0 0;
}
.masthead img { display: block; max-width: 100%; height: auto; }
.masthead .tagline {
background: #000;
color: #fff;
font-weight: bold;
padding: .25rem .4rem;
margin-top: .4rem;
}
.wrap {
background: #ccd;
background: var(--panel);
border-bottom: 6px solid #000;
padding: 1.5rem;
/* The rings the 2007 pages carried. The image has the light panel colour baked
into it, so it is dropped rather than inverted in dark mode. */
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
@media (prefers-color-scheme: dark) {
.wrap { background-image: none; }
/* The wordmark is black on transparent, and all but vanishes on the dark field. */
.masthead img { filter: invert(1); }
}
@media (min-width: 62rem) {
.wrap {
display: grid;
grid-template-columns: 14rem minmax(0, 1fr);
gap: 2.5rem;
align-items: start;
}
/* The hub is its own navigation, so it spans the full width. */
.wrap.solo { display: block; }
}
footer {
max-width: 68rem;
margin: .5rem auto 0;
color: #eef;
font-size: .8em;
}
@media (prefers-color-scheme: dark) {
footer { color: var(--ink-soft); }
}
/* ---- typography ---- */
h2, h3 { scroll-margin-top: 1rem; }
h1 { font-size: 1.9em; margin: 0 0 .3em; line-height: 1.2; }
h2 { font-size: 1.45em; margin: 2.5em 0 .5em; padding-bottom: .2em; border-bottom: 2px solid var(--rule); }
h3 { font-size: 1.15em; margin: 2em 0 .4em; }
h2:first-child, h1 + h2 { margin-top: 1em; }
p { margin: 0 0 1em; }
a { color: #00c; color: var(--link); }
a:hover { color: #c00; color: var(--link-hover); }
code, kbd, pre {
font-family: ui-monospace, "Cascadia Mono", Consolas, "DejaVu Sans Mono", monospace;
font-size: .9em;
}
code, kbd { background: var(--code-bg); padding: .1em .35em; border-radius: 3px; }
/* Legacy pages carry wide code samples: scroll them, never the page. */
pre {
overflow-x: auto;
max-width: 100%;
padding: .7rem;
background: var(--code-bg);
border-left: 3px solid var(--accent);
}
.lede { font-size: 1.08em; color: var(--ink-soft); }
hr { height: 1px; border: 0; background: var(--rule); margin: 2em 0; }
img { max-width: 100%; height: auto; }
/* Legacy prose carries bare URLs long enough to push a phone sideways. */
body { overflow-wrap: break-word; }
/* The reference pages carry tables wider than any column: scroll the table, not
the page. The guide's own MIME table is narrow and reads better left alone. */
main table:not(.mime) { display: block; overflow-x: auto; }
/* ---- table of contents ---- */
.toc { font-size: .92em; }
@media (min-width: 62rem) {
.toc { position: sticky; top: 1rem; max-height: calc(100vh - 2rem); overflow-y: auto; }
}
.toc h2 {
font-size: .8em;
text-transform: uppercase;
letter-spacing: .08em;
margin: 0 0 .5em;
border: 0;
color: var(--ink-soft);
}
.toc ol, .toc ul { list-style: none; margin: 0 0 1.5em; padding: 0; }
.toc li { margin: 0; }
.toc a {
display: block;
padding: .2rem .5rem;
text-decoration: none;
border-left: 3px solid transparent;
color: var(--ink);
}
.toc a:hover { background: var(--accent); }
.toc a[aria-current="true"] { border-left-color: var(--link); font-weight: bold; }
.toc .here { font-weight: bold; }
/* ---- callouts ---- */
.note {
margin: 0 0 1em;
padding: .7rem 1rem;
border-left: 4px solid var(--accent);
background: var(--chip);
font-size: .95em;
}
.note > :last-child { margin-bottom: 0; }
/* ---- the hub ---- */
.hub {
display: grid;
gap: 1.5rem;
margin: 2rem 0 0;
padding: 0;
list-style: none;
}
@media (min-width: 46rem) {
.hub { grid-template-columns: repeat(2, minmax(0, 1fr)); }
}
.hub > li {
padding: 1.2rem;
background: var(--chip);
border-radius: .5rem;
}
.hub h2 {
margin: 0 0 .8em;
font-size: 1.1em;
text-transform: uppercase;
letter-spacing: .06em;
border: 0;
color: var(--ink-soft);
}
.hub dl { margin: 0; }
.hub dt { margin: 0 0 .1em; font-weight: bold; }
.hub dd { margin: 0 0 1em; font-size: .93em; color: var(--ink-soft); }
.hub dd:last-child { margin-bottom: 0; }
/* Option names in the glosses are short; without this a browser breaks
"--changes" after its hyphens. */
.hub code { white-space: nowrap; }
.dated { font-weight: normal; font-size: .85em; color: var(--ink-soft); }
/* ---- content tables ---- */
/* The reference pages use bare <table> for their content tables. The guide's MIME
table is styled beside the guide and opts out. */
main table:not(.mime) {
border-collapse: collapse;
margin: 0 0 1em;
font-size: .92em;
}
main table:not(.mime) td, main table:not(.mime) th {
border: 1px solid var(--rule);
padding: .3em .6em;
text-align: left;
vertical-align: top;
}
main table:not(.mime) th, main table:not(.mime) tr.head td {
background: var(--accent);
font-weight: bold;
}
/* ---- screenshots ---- */
figure {
margin: 0 0 1.5em;
text-align: center;
}
figure img {
border: 1px solid var(--rule);
background: #fff;
box-shadow: 0 2px 8px var(--shadow);
cursor: zoom-in;
}
figcaption {
margin-top: .5em;
font-size: .88em;
color: var(--ink-soft);
}
/* Zoom overlay. Closed <dialog> is hidden by the UA, so nothing shows without JS. */
dialog#zoom {
max-width: 96vw;
max-height: 96vh;
padding: 0;
border: 0;
background: transparent;
}
dialog#zoom img {
display: block;
max-width: 96vw;
max-height: 96vh;
width: auto;
height: auto;
cursor: zoom-out;
}
dialog#zoom::backdrop { background: rgba(0, 0, 0, .8); }
/* ---- filter box (guide options, and the FAQ) ---- */
.filter {
display: flex;
gap: .5rem;
align-items: center;
margin: 0 0 1.5rem;
}
.filter input {
flex: 1 1 auto;
padding: .5rem .7rem;
border: 1px solid var(--rule);
border-radius: .3rem;
background: var(--panel);
color: var(--ink);
font: inherit;
}
.filter output { font-size: .88em; color: var(--ink-soft); white-space: nowrap; }
/* ---- misc ---- */
.skip {
position: absolute;
left: -9999px;
}
.skip:focus {
left: 1rem;
top: 1rem;
background: var(--panel);
padding: .5rem;
z-index: 10;
}
@media print {
body { background: #fff; }
.toc, .filter { display: none; }
.wrap { display: block; border: 0; }
}

104
html/doc.js Normal file
View File

@@ -0,0 +1,104 @@
/* Shared behaviour for the documentation pages: click an image to enlarge it, and
mark the section being read in the sidebar. Both are optional, and both are
written to fail quietly on an old browser rather than break the page: a Windows 7
machine may still open these files in Internet Explorer. */
(function () {
"use strict";
function each(list, fn) {
Array.prototype.forEach.call(list, fn);
}
/* ---- click to enlarge ---- */
function zoomable(node) {
while (node && node !== document.body) {
if (node.tagName === "IMG" && node.parentNode &&
node.parentNode.tagName === "FIGURE") {
return node;
}
node = node.parentNode;
}
return null;
}
function enlarge() {
var zoom = document.getElementById("zoom");
if (!zoom || typeof zoom.showModal !== "function") {
return;
}
var target = zoom.getElementsByTagName("img")[0];
document.addEventListener("click", function (event) {
var img = zoomable(event.target);
if (!img) {
return;
}
event.preventDefault();
target.src = img.src;
target.alt = img.alt;
zoom.showModal();
});
zoom.addEventListener("click", function () {
zoom.close();
});
}
/* ---- highlight the section being read ---- */
function highlight() {
var marks = [];
each(document.querySelectorAll(".toc a, .tabstrip a"), function (a) {
if (a.hash && a.hash.length > 1) {
marks.push(a);
}
});
if (!marks.length || !window.IntersectionObserver) {
return;
}
var byId = {};
each(marks, function (a) {
var id = a.hash.slice(1);
(byId[id] = byId[id] || []).push(a);
});
var visible = {};
var observer = new IntersectionObserver(function (entries) {
each(entries, function (entry) {
visible[entry.target.id] = entry.isIntersecting;
});
var top = Object.keys(byId).filter(function (id) {
return visible[id];
})[0];
each(marks, function (a) {
if (a.getAttribute("aria-current") === "true") {
a.removeAttribute("aria-current");
}
});
if (top) {
each(byId[top], function (a) {
a.setAttribute("aria-current", "true");
});
}
}, { rootMargin: "-10% 0px -70% 0px" });
Object.keys(byId).forEach(function (id) {
var el = document.getElementById(id);
if (el) {
observer.observe(el);
}
});
}
function start() {
enlarge();
highlight();
}
if (document.readyState === "loading") {
document.addEventListener("DOMContentLoaded", start);
} else {
start();
}
})();

View File

@@ -1,112 +1,76 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, linux, windows, macos, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Frequently asked questions about HTTrack, and what to do when a site does not mirror the way you expected.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>HTTrack FAQ and troubleshooting</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>F A Q</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>On this page</h2>
<ul>
<li><a href="#very-frequently-asked-questions">Very Frequently Asked Questions</a></li>
<li><a href="#general-questions">General questions</a></li>
<li><a href="#troubleshooting">Troubleshooting</a></li>
<li><a href="#questions-concerning-a-mirror">Questions concerning a mirror</a></li>
<li><a href="#other-problems">Other problems</a></li>
</ul>
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html" aria-current="page" class="here">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>F A Q</h1>
<br>
<p><em><br>
<ul>
<strong>Tips:</strong>
<li>In case of troubles/problems during transfer, <b><u><font color="red">first check the hts-log.txt (and hts-err.txt) files to figure out what happened</b></u></font>. These log files report all
<li>In case of troubles/problems during transfer, <b><u>first check the hts-log.txt (and hts-err.txt) files to figure out what happened</b></u>. These log files report all
events that may be useful to detect a problem. You can also ajust the debug level of the log files in the option
</li><li>
The tutorial written by Fred Cohen is a very good document to read, to understand how to use the engine,
@@ -118,12 +82,12 @@ clear language!
<ul><br>
<h3><b>Very</b> Frequently Asked Questions:<br></h3>
<h3 id="very-frequently-asked-questions"><b>Very</b> Frequently Asked Questions:<br></h3>
<li><a href="#VF1">HTTrack does not capture all files I want to capture!</a><br></li>
<br>
<h3>General questions:<br></h3>
<h3 id="general-questions">General questions:<br></h3>
<li><a href="#QG0">Is there any 'spyware' or 'adware' in this program? Can you prove that there isn't any?</a><br></li>
@@ -152,7 +116,7 @@ clear language!
<li><a href="#QG9">Is there a GUI version available for Linux and Un*x?</a><br></li>
<br><h3>Troubleshooting:<br></h3>
<br><h3 id="troubleshooting">Troubleshooting:<br></h3>
<li><a href="#Q0">Some sites are captured very well, other aren't. Why?</a><br></li>
@@ -217,7 +181,7 @@ clear language!
<li><a href="#QT2">With WinHTTrack, sometimes the minimize in system tray causes a crash!</a><br></li>
<h3><br>Questions concerning a mirror:<br></h3></li>
<h3 id="questions-concerning-a-mirror"><br>Questions concerning a mirror:<br></h3></li>
<li><a href="#Q5">I want to mirror a Web site, but there are some files outside the domain, too. How to retrieve them?</a><br></li>
@@ -277,7 +241,7 @@ clear language!
<li><a href="#QM13">How can I grab email addresses in web pages?</a><br></li>
<br><h3>Other problems:<br></h3></a>
<br><h3 id="other-problems">Other problems:<br></h3></a>
<li><a href="#Q300">My problerm is not listed!</a><br></li>
@@ -912,25 +876,14 @@ A: <em>Feel free to <a href="contact.html">contact us</a>!
</em></p><h3><br>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

View File

@@ -1,110 +1,81 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Fred Cohen's HTTrack Users Guide, written for release 3.10.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>HTTrack Users Guide, by Fred Cohen (3.10)</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>On this page</h2>
<ul>
<li><a href="#background-and-introduction">Background and Introduction</a></li>
<li><a href="#basics">Basics</a></li>
<li><a href="#syntax">Syntax</a></li>
<li><a href="#a-thorough-going-over">A Thorough Going Over</a></li>
<li><a href="#filters">Filters</a></li>
<li><a href="#user-authentication-protocols">User Authentication Protocols</a></li>
<li><a href="#httrackrc">.httrackrc</a></li>
<li><a href="#release-notes">Release Notes</a></li>
<li><a href="#some-more-examples">Some More Examples</a></li>
<li><a href="#general-questions">General questions</a></li>
<li><a href="#troubleshooting">Troubleshooting</a></li>
<li><a href="#questions-concerning-a-mirror">Questions concerning a mirror</a></li>
</ul>
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html" aria-current="page" class="here">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<center><h1> Httrack Users Guide (3.10)</h1>
By Fred Cohen </center>
<hr>
<h2> Background and Introduction </h2>
<h2 id="background-and-introduction"> Background and Introduction </h2>
<p align=justify> I started using httrack in mid-2000 and found it to be
an excellent tool for imaging web sites. Various words are used to
@@ -125,7 +96,7 @@ assistance from the authors I volunteered to write a users manual for
httrack - and here it is. I hope it gets the job done.
<hr>
<h2> Basics </h2>
<h2 id="basics"> Basics </h2>
<p align=justify> Httrack is a program that gets information from the
Internet, looks for pointers to other information, gets that
@@ -237,7 +208,7 @@ Build options:
x replace external html links by error pages (--replace-external)
%x do not include any password for external password protected websites (%x0 include) (--no-passwords)
%q *include query string for local files (useless, for information purpose only) (%q0 don't include) (--include-query-string)
o *generate output html file in case of error (404..) (o0 don't generate) (--generate-errors)
o *save the server's error pages (404..) (o0 discard them) (--generate-errors)
X *purge old files after update (X0 keep delete) (--purge-old[=N])
Spider options:
@@ -385,7 +356,7 @@ the rest of us, I will now go through this listing one item at a time
with examples... I will be here a while...
<hr>
<h2> Syntax </h2>
<h2 id="syntax"> Syntax </h2>
<pre><b><i>httrack &lt;URLs&gt; [-option] [+&lt;FILTERs&gt;] [-&lt;FILTERs&gt;] </i></b></pre>
@@ -413,7 +384,7 @@ site. Specifically, the defauls are:
NN name conversion type (0 *original structure, 1+: see below)
LN long names (L1 *long names / L0 8-3 conversion)
K keep original links (e.g. http://www.adr/link) (K0 *relative link)
o *generate output html file in case of error (404..) (o0 don't generate)
o *save the server's error pages (404..) (o0 discard them)
X *purge old files after update (X0 keep delete)
bN accept cookies in cookies.txt (0=do not accept,* 1=accept)
u check document type if unknown (cgi,asp..) (u0 don't check, * u1 check but /, u2 check always)
@@ -473,11 +444,11 @@ store them with the same names used on the web site.
URLs within this web site are adjusted to point to the files in the
mirror.
<pre><b><i> o *generate output html file in case of error (404..) (o0 don't generate) </i></b></pre>
<pre><b><i> o *save the server's error pages (404..) (o0 discard them) </i></b></pre>
<p align=justify> IF there are errors in downloading, create a file that
indicates that the URL was not found. This makes browsing go a lot
smoother.
<p align=justify> IF a page cannot be downloaded, the error page the
server sent is saved in its place, so a broken link still lands on the
site's own 'not found' page. This makes browsing go a lot smoother.
<pre><b><i> X *purge old files after update (X0 keep delete) </i></b></pre>
@@ -571,7 +542,7 @@ www.all.net/bob/deeper/).
<hr>
<h2> A Thorough Going Over </h2>
<h2 id="a-thorough-going-over"> A Thorough Going Over </h2>
<p align=justify> Now that you have an introduction, it's time for a
more though coverage. This is where I go through each of the options
@@ -1011,7 +982,7 @@ Build options:
LN long names (L1 *long names / L0 8-3 conversion)
K keep original links (e.g. http://www.adr/link) (K0 *relative link)
x replace external html links by error pages
o *generate output html file in case of error (404..) (o0 don't generate)
o *save the server's error pages (404..) (o0 discard them)
X *purge old files after update (X0 keep delete)
%x do not include any password for external password protected websites (%x0 include) (--no-passwords)
%q *include query string for local files (information only) (%q0 don't include) (--include-query-string)
@@ -1118,9 +1089,9 @@ deactivated byt his process.
httrack http://www.shoesizes.com -O /tmp/shoesizes -x
</i></b></pre>
<p align=justify> This option prevents the generation of '404' error
files to replace files that were not found even though there were URLs
pointing to them. It is useful for saving space as well as eliminating
<p align=justify> This option keeps the server's '404' error pages out
of the mirror, even though there were URLs pointing to the missing
files. It is useful for saving space as well as eliminating
unnecessary files in operations where a working web site is not the
desired result.
@@ -1930,7 +1901,7 @@ as -%h).
<hr>
<h2> Filters </h2>
<h2 id="filters"> Filters </h2>
<p align=justify> Filters are normally placed at the end of the command
line, but can be intermixed with other command line options if desired,
@@ -2079,7 +2050,7 @@ generated automatically using the interface)
<hr>
<h2> User Authentication Protocols </h2>
<h2 id="user-authentication-protocols"> User Authentication Protocols </h2>
<p align=justify> Smoe servers require user ID and password information
in order to gain access. In this example, the user ID smith with
@@ -2096,7 +2067,7 @@ capability is being provided through th URL capture features
<hr>
<h2> .httrackrc </h2>
<h2 id="httrackrc"> .httrackrc </h2>
<p align=justify> A file called '.httrackrc' can be placed in the
current directory, or if not found there, in the home directory, to
@@ -2185,7 +2156,7 @@ added by the .httrackrc
<hr>
<h2> Release Notes </h2>
<h2 id="release-notes"> Release Notes </h2>
<p align=justify> Some things change between releases. Here are some
recent changes in httrack that may affect some of these options:
@@ -2223,7 +2194,7 @@ collision for large numbers of files.
<hr>
<h2> Some More Examples </h2>
<h2 id="some-more-examples"> Some More Examples </h2>
<p align=justify> Here are some examples of special purpose httrack
command lines that might be useful for your situation.
@@ -2275,7 +2246,7 @@ httrack "http://www.somesite.com/~library/thing/thingmain.htm" -O /tmp/thing -R5
<hr>
<h2>General questions:</h2>
<h2 id="general-questions">General questions:</h2>
<p align=justify><b>Q: The install is not working on NT without administrator rights! </b>
@@ -2327,7 +2298,7 @@ Un*x? </b>
<p align=justify> A: No. Unfortunately, we do not have enough time for
that - if you want to help us, please write one!
<hr><h2> Troubleshooting:</h2>
<hr><h2 id="troubleshooting"> Troubleshooting:</h2>
<p align=justify><b>Q: Only the first page is caught. What's wrong?</b>
</br> A: First, check the <b>hts-err.txt</b> error log file - this can
@@ -2509,7 +2480,7 @@ considered as "newer" than the local ones.. even if there are
identical! Unfortunately, there is no possibility to avoid this problem,
which is strongly linked with the server abilities.
<p align=justify> <hr><h2> Questions concerning a mirror: </h2>
<p align=justify> <hr><h2 id="questions-concerning-a-mirror"> Questions concerning a mirror: </h2>
<p align=justify> <p align=justify><b>Q: I want to mirror a Web site,
but there are some files outside the domain, too. How to retrieve them?
@@ -2693,25 +2664,14 @@ too many sessions, or your system may be unable to provide the necessary
resources. Try redusing this number - for example using the -c2 options
for only 2 simultaneous sesions.
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

View File

@@ -1,105 +1,61 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
<!-- doc-chrome:head -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="HTTrack filter syntax: wildcards, and the size and MIME-type forms of a scan rule.">
<link rel="stylesheet" href="doc.css">
<script src="doc.js" defer></script>
<!-- /doc-chrome:head -->
<title>HTTrack filter syntax</title>
</head>
<body>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<!-- doc-chrome:top -->
<a class="skip" href="#main">Skip to content</a>
<h2 align="center"><em>Filters: Advanced</em></h2>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Documentation">
<h2>Start here</h2>
<ul>
<li><a href="guide.html">Interface guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="abuse.html">Using HTTrack responsibly</a></li>
</ul>
<h2>Command line</h2>
<ul>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="httrack.man.html">Option reference</a></li>
<li><a href="filters.html" aria-current="page" class="here">Filter syntax</a></li>
</ul>
<h2>Developers</h2>
<ul>
<li><a href="dev.html">Programming</a></li>
<li><a href="library.html">libhttrack API</a></li>
<li><a href="plug.html">Callbacks</a></li>
<li><a href="scripting.html">Scripting</a></li>
<li><a href="cache.html">Cache format</a></li>
<li><a href="changes.html">Change report format</a></li>
</ul>
<h2>More</h2>
<ul>
<li><a href="fcguide.html">Users Guide (3.10)</a></li>
<li><a href="contact.html">Contact and credits</a></li>
<li><a href="index.html">All documentation</a></li>
</ul>
</nav>
<main id="main">
<!-- /doc-chrome:top -->
<h1>Filters: Advanced</h1>
<br>
@@ -152,7 +108,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Important notice: size scan rules are checked <font color=red><b>after</b></font> the link was scheduled for download,
Important notice: size scan rules are checked <b>after</b> the link was scheduled for download,
allowing to abort the connection.
@@ -174,7 +130,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
-mime:image/gif
<br>
Important notice: MIME types scan rules are <font color=red><b>only</b></font> checked against links that were
Important notice: MIME types scan rules are <b>only</b> checked against links that were
scheduled for download, i.e. links <b>already authorized</b> by url scan rules.
Hence, using '+mime:image/gif' will only be a hint to accept images that were already authorized,
if previous MIME scan rules excluded them - such as in '-mime:*/* +mime:text/html +mime:image/gif'
@@ -182,7 +138,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
<br>
<h3>Scan rules patterns:</h3>
<h3 id="scan-rules-patterns">Scan rules patterns:</h3>
<p>
<h4>1.a. Scan rules based on URL or extension</h4>
@@ -455,25 +411,14 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!-- doc-chrome:bottom -->
</main>
</div>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
<dialog id="zoom" aria-label="Enlarged image"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
<!-- /doc-chrome:bottom -->
</body>
</html>

149
html/guide.css Normal file
View File

@@ -0,0 +1,149 @@
/* Styles specific to guide.html: the platform switcher, the option tab strip and
the option entries. The shared page chrome is in doc.css, loaded first. */
/* Anchor targets must clear the sticky tab strip. */
h2, h3, section.tab { scroll-margin-top: 5.5rem; }
.cli {
font-family: ui-monospace, "Cascadia Mono", Consolas, "DejaVu Sans Mono", monospace;
font-size: .9em;
}
/* ---- platform switcher ---- */
.platforms {
display: flex;
flex-wrap: wrap;
gap: .3rem;
margin: 0 0 1.5rem;
padding: .3rem;
background: var(--chip);
border-radius: .5rem;
}
.platforms button {
flex: 1 1 8rem;
padding: .5rem .7rem;
border: 0;
border-radius: .35rem;
background: transparent;
color: var(--ink);
font: inherit;
font-size: .95em;
cursor: pointer;
}
.platforms button:hover { background: var(--accent); }
.platforms button[aria-pressed="true"] {
background: var(--panel);
box-shadow: 0 1px 3px var(--shadow);
font-weight: bold;
}
.platforms button small { display: block; font-weight: normal; color: var(--ink-soft); font-size: .82em; }
/* Hide only what the chosen platform excludes: an "unhide" rule would need to
outweigh the hiding one, and loses on specificity. Without scripting no
platform is set, so every variant stays visible. */
html[data-platform="win"] [data-for]:not([data-for~="win"]),
html[data-platform="web"] [data-for]:not([data-for~="web"]),
html[data-platform="droid"] [data-for]:not([data-for~="droid"]) { display: none; }
/* ---- platform badges ---- */
.badge {
display: inline-block;
padding: .05em .5em;
border-radius: 1em;
background: var(--accent);
color: var(--ink);
font-size: .78em;
font-weight: bold;
vertical-align: .1em;
white-space: nowrap;
}
/* ---- screenshots ---- */
figure img[data-for~="droid"] { max-width: 18rem; }
/* ---- option tab strip ---- */
.tabstrip {
position: sticky;
top: 0;
z-index: 2;
display: flex;
flex-wrap: wrap;
gap: .2rem;
margin: 0 0 1.5rem;
padding: .4rem;
background: var(--chip);
border-radius: .4rem;
box-shadow: 0 2px 6px var(--shadow);
}
.tabstrip a {
padding: .3rem .6rem;
border-radius: .25rem;
text-decoration: none;
color: var(--ink);
font-size: .9em;
white-space: nowrap;
}
.tabstrip a:hover { background: var(--accent); }
.tabstrip a[aria-current="true"] { background: var(--panel); font-weight: bold; box-shadow: 0 1px 3px var(--shadow); }
/* ---- option entries ---- */
.opt {
margin: 0 0 1.2em;
padding-bottom: 1.2em;
border-bottom: 1px solid var(--rule);
}
.opt:last-child { border-bottom: 0; }
.opt-head {
display: flex;
flex-wrap: wrap;
gap: .5rem;
align-items: baseline;
justify-content: space-between;
margin-bottom: .3em;
}
.opt-name { font-weight: bold; }
.opt .cli {
background: var(--code-bg);
padding: .1em .45em;
border-radius: 3px;
color: var(--ink-soft);
white-space: nowrap;
}
.opt p { margin: 0 0 .5em; font-size: .95em; }
.opt p:last-child { margin-bottom: 0; }
.opt[hidden] { display: none; }
/* ---- misc ---- */
table.mime {
border-collapse: collapse;
margin: 0 0 1em;
font-size: .92em;
}
table.mime td {
border: 1px solid var(--rule);
padding: .3em .6em;
}
@media print {
.tabstrip, .platforms { display: none; }
html[data-platform] [data-for] { display: revert !important; }
}

911
html/guide.html Normal file
View File

@@ -0,0 +1,911 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="How to mirror a website with the HTTrack graphical interface: a step-by-step walkthrough and a full option reference for WinHTTrack on Windows, WebHTTrack on Linux and Unix, and HTTrack for Android.">
<meta name="keywords" content="httrack, WinHTTrack, WebHTTrack, HTTrack Android, offline browser, website mirroring, web mirror utility, options, scan rules, filters">
<title>The HTTrack interface, step by step</title>
<link rel="stylesheet" href="doc.css">
<link rel="stylesheet" href="guide.css">
<script src="doc.js" defer></script>
<script src="guide.js"></script>
</head>
<body>
<a class="skip" href="#main">Skip to content</a>
<header class="masthead">
<img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier">
<div class="tagline">Open Source offline browser</div>
</header>
<div class="wrap">
<nav class="toc" aria-label="Contents">
<h2>Walkthrough</h2>
<ol>
<li><a href="#before">Before you start</a></li>
<li><a href="#step-start">1. Open HTTrack</a></li>
<li><a href="#step-project">2. Name the project</a></li>
<li><a href="#step-address">3. Enter the addresses</a></li>
<li><a href="#step-ready">4. Last checks</a></li>
<li><a href="#step-run">5. Watch it run</a></li>
<li><a href="#step-done">6. Read the result</a></li>
</ol>
<h2>Options</h2>
<ol>
<li><a href="#opt-scan-rules">Scan Rules</a></li>
<li><a href="#opt-limits">Limits</a></li>
<li><a href="#opt-flow-control">Flow Control</a></li>
<li><a href="#opt-links">Links</a></li>
<li><a href="#opt-build">Build</a></li>
<li><a href="#opt-browser-id">Browser ID</a></li>
<li><a href="#opt-spider">Spider</a></li>
<li><a href="#opt-proxy">Proxy</a></li>
<li><a href="#opt-log-index-cache">Log, Index, Cache</a></li>
<li><a href="#opt-mime-types">MIME Types</a></li>
<li><a href="#opt-experts-only">Experts Only</a></li>
</ol>
<h2>Elsewhere</h2>
<ol>
<li><a href="filters.html">Filter syntax</a></li>
<li><a href="cmdguide.html">Command-line guide</a></li>
<li><a href="faq.html">FAQ and troubleshooting</a></li>
<li><a href="index.html">All documentation</a></li>
</ol>
</nav>
<main id="main">
<h1>The HTTrack interface, step by step</h1>
<p class="lede">HTTrack copies a website to your disk so you can read it offline. The same
engine ships behind three interfaces, and this page covers all three: pick yours below and
the screenshots and notes follow along.</p>
<div class="platforms" role="group" aria-label="Choose your version">
<button type="button" data-platform="win" aria-pressed="false">WinHTTrack <small>Windows</small></button>
<button type="button" data-platform="web" aria-pressed="false">WebHTTrack <small>Linux and Unix</small></button>
<button type="button" data-platform="droid" aria-pressed="false">HTTrack <small>Android</small></button>
</div>
<h2 id="before">Before you start</h2>
<p>The three versions differ in their chrome, not in what they do. Every option below means
the same thing in each of them, and all three write the same profile format. Where a version
genuinely lacks something, it is marked <span class="badge">not on Android</span> or
similar.</p>
<p data-for="web">WebHTTrack runs as a small local web server and opens in your usual browser.
Nothing is uploaded anywhere; the pages you see come from your own machine. Do leave the window
open while a mirror runs: the interface pings the server every 30 seconds, and the server shuts
itself down once those pings stop.</p>
<p data-for="droid">The Android app needs Android 7.0 or later and is on
<a href="https://play.google.com/store/apps/details?id=com.httrack.android">Google Play</a>.</p>
<p>Each option also lists its command-line equivalent, so anything you set here can later be
scripted. The <a href="cmdguide.html">command-line guide</a> covers that side.</p>
<div class="note" data-for="droid">
<p><b>First launch.</b> Android asks for permission to store mirrors on your device. Without
it the app cannot save anything, so tap <b>Allow</b>.</p>
<figure>
<img data-for="droid" src="img/guide-droid-permission.png" alt="Android storage permission dialog" width="600" height="1300">
</figure>
<p>If an older release left mirrors behind, a second prompt offers to bring them in. Tap
<b>Import</b> to move them, or <b>Not now</b>.</p>
<figure>
<img data-for="droid" src="img/guide-droid-legacy-import.png" alt="Prompt offering to import mirrors from an older version" width="600" height="1300">
</figure>
</div>
<div class="note" data-for="win">
<p><b>First launch.</b> WinHTTrack opens on its About box, which carries a <b>Language
preference</b> dropdown at the foot. The choice is remembered, and
<b>Preferences &gt; Language preference...</b> changes it later.</p>
<figure>
<img data-for="win" src="img/guide-win-language.png" alt="WinHTTrack About box with the language preference dropdown" width="369" height="371">
</figure>
</div>
<h2 id="step-start">1. Open HTTrack</h2>
<p>The welcome screen has nothing to fill in. <span data-for="win web">Click <b>Next</b></span><span
data-for="droid">Tap <b>Next</b></span> to create a project, or open one you already made.</p>
<figure>
<img data-for="win" src="img/guide-win-start.png" alt="WinHTTrack welcome pane" width="1040" height="744">
<img data-for="web" src="img/guide-web-start.png" alt="WebHTTrack welcome page" width="1024" height="388">
<img data-for="droid" src="img/guide-droid-start.png" alt="HTTrack for Android welcome screen" width="600" height="1300">
<figcaption>The welcome screen.</figcaption>
</figure>
<p class="note" data-for="web">The language dropdown starts blank on purpose: its first entry
means "leave the interface as it is". Pick a language only if you want to change it.</p>
<h2 id="step-project">2. Name the project</h2>
<p>A project is one mirror: its name becomes the folder your files land in, so give it something
you will recognise in a year. The <b>category</b> is optional and only groups related projects
together in the list.</p>
<figure>
<img data-for="win" src="img/guide-win-project-name.png" alt="WinHTTrack project name, category and base path" width="1040" height="744">
<img data-for="web" src="img/guide-web-project-name.png" alt="WebHTTrack project name, category and base path" width="1024" height="346">
<img data-for="droid" src="img/guide-droid-project-name.png" alt="Android project name, category and base path" width="600" height="1300">
<figcaption>Project name, category, and where the files go.</figcaption>
</figure>
<p data-for="win web"><b>Base path</b> is the folder holding all your mirrors. Keeping every project
under one folder is worth doing, if only so an update finds the previous copy. To reopen or update
an existing project, pick its name from the list instead of typing a new one.</p>
<p data-for="droid">The base path is fixed inside the app's storage and cannot be changed. To update
an existing project, pick its name from the list instead of typing a new one.</p>
<h2 id="step-address">3. Enter the addresses</h2>
<p>Type the address you want to copy. Several addresses, one per line, are mirrored together and
keep their links to each other.</p>
<figure>
<img data-for="win" src="img/guide-win-project-setup.png" alt="WinHTTrack action and web addresses" width="1040" height="744">
<img data-for="web" src="img/guide-web-project-setup.png" alt="WebHTTrack action and web addresses" width="1024" height="496">
<img data-for="droid" src="img/guide-droid-project-setup.png" alt="Android web address and action" width="600" height="1300">
<figcaption>The addresses to copy, and what to do with them.</figcaption>
</figure>
<p><b>Action</b> decides what the engine does with them:</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Download web site(s)</span><span class="cli">-w, --mirror</span></div>
<p>The normal choice. Copies the sites with the current options.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Download web site(s) + questions</span><span class="cli">-W, --mirror-wizard</span></div>
<p>The same, but asks before following anything it is unsure about.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Get individual files</span><span class="cli">-g, --get-files</span></div>
<p>Downloads exactly the addresses you listed and follows nothing. Use it to grab a handful of
files rather than a site.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Download all sites in pages</span><span class="cli">-Y, --mirrorlinks</span></div>
<p>Mirrors every site linked from the pages you listed. Pointed at a bookmarks file, it copies
everything you bookmarked.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Test links in pages</span><span class="cli">--testlinks</span></div>
<p>Checks that the links resolve without saving anything. Useful against a bookmarks file.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Continue interrupted download</span><span class="cli">--continue</span></div>
<p>Picks a crashed or cancelled mirror back up where it stopped. Offered only when the project
already exists.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Update existing download</span><span class="cli">--update</span></div>
<p>Re-checks the whole site and fetches only what changed. Offered only when the project already
exists, and it needs the cache from the previous run, which is why
<a href="#opt-experts-only">Use a cache for updates</a> should stay on.</p>
</div>
<p data-for="win web">The <b>Add a URL</b> button takes one address at a time, and does two
things the address box cannot. It attaches a login and password to that one address, for a
page behind basic authentication. And it captures an address out of your browser, for a page
you can only reach by submitting a form or following a script: HTTrack listens on a local
port, you point your browser at it as a proxy and click the link or the submit button as
usual, and the address the browser asks for is added to the project.</p>
<figure data-for="web">
<img data-for="web" src="img/guide-web-add-url.png" alt="WebHTTrack add-a-URL dialog" width="1024" height="634">
<figcaption>Adding a single address, with an optional login.</figcaption>
</figure>
<p>Everything else lives behind <b>Set options</b>, described in
<a href="#options">the option panel</a> below. The defaults are sane; a first mirror needs nothing
from there except, quite often, a <a href="#opt-scan-rules">scan rule</a>.</p>
<figure data-for="droid">
<img data-for="droid" src="img/guide-droid-options-menu.png" alt="Android options tab list" width="600" height="1300">
<figcaption>On Android the option tabs are a list.</figcaption>
</figure>
<h2 id="step-ready">4. Last checks</h2>
<p data-for="win">One pane stands between you and the mirror. It can dial a connection first,
hang up when the mirror ends, shut the machine down afterwards, or hold the whole thing for up to
24 hours. Leave it alone if you are already online and want to start now, and click
<b>Finish</b>.</p>
<p data-for="web">One page stands between you and the mirror. <b>Start</b> begins it;
the save-only choice writes the project settings and stops, which is handy when you are preparing
a mirror to run later.</p>
<p data-for="droid">Android has no such screen: <b>Start</b> on the address screen begins the
mirror straight away.</p>
<figure data-for="win web">
<img data-for="win" src="img/guide-win-ready.png" alt="WinHTTrack connection settings before starting" width="1040" height="744">
<img data-for="web" src="img/guide-web-ready.png" alt="WebHTTrack ready to start" width="1024" height="508">
<figcaption>The last screen before the mirror runs.</figcaption>
</figure>
<h2 id="step-run">5. Watch it run</h2>
<p>The engine reports what it is doing: bytes written, links scanned, transfer rate, errors so
far, and one line per transfer in flight. A mirror of any size takes a while. That is the
server's pace as much as yours, and the <a href="#opt-limits">Limits</a> options exist to keep it
polite.</p>
<figure>
<img data-for="win" src="img/guide-win-progress.png" alt="WinHTTrack mirror in progress" width="1040" height="744">
<img data-for="web" src="img/guide-web-progress.png" alt="WebHTTrack mirror in progress" width="1024" height="742">
<img data-for="droid" src="img/guide-droid-progress.png" alt="Android mirror in progress" width="600" height="1300">
<figcaption>A crawl in flight.</figcaption>
</figure>
<p data-for="win web">You can stop at any point without losing what has been written, and a
stopped mirror resumes later with <b>Continue interrupted download</b>. Individual transfers can
be cancelled too, which is the quick way past one enormous file. Several options (the number of
connections, the limits) can be changed while the mirror runs.</p>
<p data-for="droid">Tap <b>Abort</b> to stop. Nothing already written is lost, and the mirror
resumes later with <b>Continue interrupted download</b>.</p>
<h2 id="step-done">6. Read the result</h2>
<p>When the crawl ends, open the copy and read it as you would the real site. Links between
mirrored pages are rewritten to point at each other, so it browses offline in any browser.</p>
<figure>
<img data-for="win" src="img/guide-win-finished.png" alt="WinHTTrack mirror complete" width="1040" height="744">
<img data-for="web" src="img/guide-web-finished.png" alt="WebHTTrack mirror complete" width="1024" height="487">
<img data-for="droid" src="img/guide-droid-finished.png" alt="Android mirror complete" width="600" height="1300">
<figcaption>Done.</figcaption>
</figure>
<p>Read the log before you trust a mirror. It records every error, and a site that looks complete
can still be missing images that a <a href="#opt-scan-rules">scan rule</a> would have caught. If
something is missing, the <a href="faq.html">FAQ and troubleshooting</a> page starts with the
usual causes.</p>
<p data-for="win web">The files sit under the base path you chose, in a folder named after the
project. Opening <code>index.html</code> there browses the mirror without HTTrack.</p>
<p data-for="droid">Mirrors are written to
<code>/storage/emulated/0/HTTrack/Websites</code>, in a folder named after the project. That is
shared storage, so a file manager or a USB cable can reach it, and opening
<code>index.html</code> there browses the mirror without the app.</p>
<h2 id="options">The option panel</h2>
<p>Eleven tabs, and you can ignore almost all of them. The defaults mirror a site correctly; the
options are there for the sites that need persuading. Each entry below names the setting as the
interface shows it, and the command-line option it corresponds to.</p>
<nav class="tabstrip" aria-label="Option tabs">
<a href="#opt-scan-rules">Scan Rules</a>
<a href="#opt-limits">Limits</a>
<a href="#opt-flow-control">Flow Control</a>
<a href="#opt-links">Links</a>
<a href="#opt-build">Build</a>
<a href="#opt-browser-id">Browser ID</a>
<a href="#opt-spider">Spider</a>
<a href="#opt-proxy">Proxy</a>
<a href="#opt-log-index-cache">Log, Index, Cache</a>
<a href="#opt-mime-types">MIME Types</a>
<a href="#opt-experts-only">Experts Only</a>
</nav>
<div class="filter">
<label for="optfilter">Find an option</label>
<input type="search" id="optfilter" placeholder="cookies, depth, footer…" autocomplete="off">
<output id="optcount" for="optfilter"></output>
</div>
<section class="tab" id="opt-scan-rules">
<h3>Scan Rules</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-scan-rules.png" alt="WinHTTrack Scan Rules tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-scan-rules.png" alt="WebHTTrack Scan Rules tab" width="1024" height="995">
<img data-for="droid" src="img/guide-droid-opt-scan-rules.png" alt="Android Scan Rules tab" width="600" height="1300">
</figure>
<p>The most useful tab in the panel, and the answer to most "why is this missing?" questions. A
scan rule accepts or refuses addresses by pattern: a whole directory, a domain, a file type. When
a mirror comes back without its images because they live on another host, this is where you let
them in.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Include link(s)</span><span class="cli">+pattern</span></div>
<p>Accept addresses matching the pattern, even ones the engine would otherwise leave alone.
<code>+*.example.com/*.jpg</code> takes the JPEGs from a neighbouring host;
<code>+*/images/landscapes/*</code> takes one directory wherever it appears.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Exclude link(s)</span><span class="cli">-pattern</span></div>
<p>Refuse addresses matching the pattern. <code>-*.zip</code> skips the archives,
<code>-*/forum/*</code> skips a section you do not want.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Rule list from a file</span><span class="cli">-%S, --urllist &lt;file&gt;</span></div>
<p>Reads rules from a text file, one per line, instead of the box.</p>
</div>
<p>Rules are read in order and the last match wins, so a broad exclusion followed by a narrow
inclusion does what it looks like. The full pattern syntax, including size and MIME conditions,
is on the <a href="filters.html">filter page</a>.</p>
</section>
<section class="tab" id="opt-limits">
<h3>Limits</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-limits.png" alt="WinHTTrack Limits tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-limits.png" alt="WebHTTrack Limits tab" width="1024" height="1016">
<img data-for="droid" src="img/guide-droid-opt-limits.png" alt="Android Limits tab" width="600" height="1300">
</figure>
<p>Ceilings on how much the mirror may cost you, and cost the server. The two rate limits at the
bottom are the ones that keep you welcome.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Maximum mirroring depth</span><span class="cli">-rN, --depth N</span></div>
<p>How many clicks from your starting addresses the engine may travel. A depth of 3 means the
pages you listed, plus everything within two more clicks. Left empty it is effectively
unlimited, which is usually right: the engine already refuses to leave the site.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Maximum external depth</span><span class="cli">-%eN, --ext-depth N</span></div>
<p>How far to follow links off the site, past what the scan rules allow. It overrides the other
limits, so raise it with care. The default is zero.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Max size of any non-HTML file</span><span class="cli">-mN, --max-files N</span></div>
<p>Per-file ceiling in bytes: a larger image or archive is skipped.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Max size of any HTML file</span><span class="cli">-mN,N2, --max-files</span></div>
<p>The same for pages. One command-line option carries both ceilings, non-HTML first.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Site size limit</span><span class="cli">-MN, --max-size N</span></div>
<p>Total bytes the mirror may download before it stops.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Pause after downloading</span><span class="cli">-GN, --max-pause N</span></div>
<p>Pauses once that many bytes have arrived and waits for you to delete a lock file. The way to
mirror a site larger than your free space: back up and clear the files during the pause.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Pause between files</span><span class="cli">-%G, --pause MIN[:MAX]</span></div>
<p>Waits a random number of seconds between downloads. <code>2:8</code> picks a fresh delay in
that range each time. The gentlest way to crawl a small server, and less blunt than capping the
rate.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Max time overall</span><span class="cli">-EN, --max-time N</span></div>
<p>Seconds the whole mirror may take. 3600 is an hour.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Max transfer rate</span><span class="cli">-AN, --max-rate N</span></div>
<p>Bytes per second, across the whole mirror. Keeps HTTrack from taking the whole line.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Max connections / seconds</span><span class="cli">-%cN, --connection-per-second N</span></div>
<p>New connections per second, and the politest thing on this page. Fractions are allowed:
<code>0.1</code> is one connection every ten seconds. Setting it to zero removes the limit and
can flatten a small server; do not, unless the server is yours.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Maximum number of links</span><span class="cli">-#LN, --advanced-maxlinks N</span></div>
<p>How many addresses may be held in memory, downloaded or not. The engine stops dead on
reaching it, so do not set it low. The default of 100,000 suits most sites and costs memory to
raise.</p>
</div>
</section>
<section class="tab" id="opt-flow-control">
<h3>Flow Control</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-flow-control.png" alt="WinHTTrack Flow Control tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-flow-control.png" alt="WebHTTrack Flow Control tab" width="1024" height="910">
<img data-for="droid" src="img/guide-droid-opt-flow-control.png" alt="Android Flow Control tab" width="600" height="1300">
</figure>
<p>How hard to push, and when to give up on a server that is not answering.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Number of connections</span><span class="cli">-cN, --sockets N</span></div>
<p>Transfers running at once. Four is the default and eight is comfortable on an ordinary site;
drop to one or two when the files are large, since parallel transfers of big files help nobody.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Timeout</span><span class="cli">-TN, --timeout N</span></div>
<p>Seconds to wait on a silent server before abandoning the link. It also bounds host name
resolution. Around 120 suits most connections.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Retries</span><span class="cli">-RN, --retries N</span></div>
<p>How many times to retry after a timeout or another non-fatal error. It will not rescue a
404: nothing retries a definite answer.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Min transfer rate</span><span class="cli">-JN, --min-rate N</span></div>
<p>Bytes per second below which a transfer is treated as stalled and dropped.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Abandon host on timeout or slow transfer</span><span class="cli">-HN, --host-control N</span></div>
<p>Whether one bad transfer condemns the whole host: never (0), on timeout (1), on a slow
transfer (2), or either (3). The checkboxes beside the timeout and rate fields set this. Anything
but <em>never</em> can drop a lot of links over one slow moment, so leave it off unless a host is
actively wasting your time.</p>
</div>
</section>
<section class="tab" id="opt-links">
<h3>Links</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-links.png" alt="WinHTTrack Links tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-links.png" alt="WebHTTrack Links tab" width="1024" height="781">
<img data-for="droid" src="img/guide-droid-opt-links.png" alt="Android Links tab" width="600" height="1300">
</figure>
<p>What counts as a link worth following.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Attempt to detect all links</span><span class="cli">-%P, --extended-parsing</span></div>
<p>Looks for addresses everywhere, including unknown tags and JavaScript. On by default. It
earns its keep on script-heavy pages, at the price of the occasional request for something that
was never a link.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Get non-HTML files related to a link</span><span class="cli">-n, --near</span></div>
<p>Fetches the images and other files a page refers to even when they sit outside the mirror.
The usual fix for a page whose pictures live on another host.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Test validity of all links</span><span class="cli">-t, --test</span></div>
<p>Requests every link found, including ones the rules forbid, and logs the failures. A link
checker rather than a mirroring option.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Get HTML files first</span><span class="cli">-p7, --priority 7</span></div>
<p>Takes the pages before the images. The structure of the site is known sooner, which makes the
rest of the crawl better informed.</p>
</div>
</section>
<section class="tab" id="opt-build">
<h3>Build</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-build.png" alt="WinHTTrack Build tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-build.png" alt="WebHTTrack Build tab" width="1024" height="986">
<img data-for="droid" src="img/guide-droid-opt-build.png" alt="Android Build tab" width="600" height="1300">
</figure>
<p>How the copy is laid out on disk.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Local structure type</span><span class="cli">-NN, --structure N</span></div>
<p>The default reproduces the site's own folders and names. The alternatives sort files by kind
instead: all the pages here, all the images there. A custom pattern such as
<code>-N "%h%p/%n%q.%t"</code> builds names from the host, path, name, query and type.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">DOS names (8+3)</span><span class="cli">-L0, --long-names 0</span></div>
<p>Cuts every name to eight characters and a three-letter extension.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">ISO9660 names (CDROM)</span><span class="cli">-L2, --long-names 2</span></div>
<p>Names that survive being burned to a CD or DVD.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">No error pages</span><span class="cli">-o0, --do-not-generate-errors</span></div>
<p>By default a page that returned 404 is saved as a note saying so. This drops those, leaving
nothing at all where the page was.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">No external pages</span><span class="cli">-x, --replace-external</span></div>
<p>Replaces links that leave the mirror with a page saying you need to be online. Keeps the
offline copy from silently reaching for the network.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Hide passwords</span><span class="cli">-%x, --disable-passwords</span></div>
<p>Strips credentials out of the links written into the saved pages, so the copy cannot leak
them.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Hide query strings</span><span class="cli">-%q0, --include-query-string 0</span></div>
<p>Local files rarely need the <code>?a=1&amp;b=2</code> part, though it can carry a hint about
what a page was. Some limited browsers choke on it.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Do not purge old files</span><span class="cli">-X0, --purge-old 0</span></div>
<p>After an update HTTrack deletes local files that are gone from the site or now excluded. This
keeps them.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Inline assets as data: URIs</span><span class="cli">-%Z, --single-file</span></div>
<p>Rewrites each saved page with its stylesheets, scripts, images and fonts embedded, so a single
file opens anywhere on its own. Links between mirrored pages stay relative; audio and video stay
links. <code>--single-file-max-size</code> caps each embedded asset, 10 MB by default.</p>
</div>
</section>
<section class="tab" id="opt-browser-id">
<h3>Browser ID</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-browser-id.png" alt="WinHTTrack Browser ID tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-browser-id.png" alt="WebHTTrack Browser ID tab" width="1024" height="706">
<img data-for="droid" src="img/guide-droid-opt-browser-id.png" alt="Android Browser ID tab" width="600" height="1300">
</figure>
<p>What HTTrack says about itself in its requests.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Browser identity</span><span class="cli">-F, --user-agent</span></div>
<p>The <code>User-Agent</code> sent with every request. Some sites serve different pages, or
refuse to serve at all, depending on what they read here.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">HTML footer</span><span class="cli">-%F, --footer</span></div>
<p>A comment added to each saved page, recording where it came from. Named fields are
substituted: <code>{url}</code>, <code>{date}</code>, <code>{addr}</code>, <code>{path}</code>,
<code>{lastmodified}</code>, <code>{version}</code>, <code>{mime}</code>, <code>{charset}</code>,
<code>{status}</code> and <code>{size}</code>. For example
<code>&lt;!-- Mirrored from {url} on {date} --&gt;</code>. You can turn it off, but a page with no
record of its origin is worth less later.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Preferred language</span><span class="cli">-%l, --language</span></div>
<p>The <code>Accept-Language</code> header, as in <code>"fr, en, *"</code>. It decides which
translation a multilingual site hands over.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Default referer</span><span class="cli">-%R, --referer</span></div>
<p>The <code>Referer</code> sent with the first request. Occasionally needed by sites that refuse
traffic arriving from nowhere.</p>
</div>
</section>
<section class="tab" id="opt-spider">
<h3>Spider</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-spider.png" alt="WinHTTrack Spider tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-spider.png" alt="WebHTTrack Spider tab" width="1024" height="1557">
<img data-for="droid" src="img/guide-droid-opt-spider.png" alt="Android Spider tab" width="600" height="1300">
</figure>
<p>How the engine behaves towards the server it is talking to.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Accept cookies</span><span class="cli">-bN, --cookies N</span></div>
<p>On by default. Sites that hand out a session before showing anything need it.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Check document type</span><span class="cli">-uN, --check-type N</span></div>
<p>When the address does not reveal the file type, say a <code>.cgi</code> that returns an image,
the engine asks the server so the file gets a sensible name locally. Turning this off entirely
produces a mirror full of files no browser will open.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Parse scripts</span><span class="cli">-jN, --parse-java N</span></div>
<p>Reads scripts looking for addresses. On by default.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Follow robots.txt</span><span class="cli">-sN, --robots N</span></div>
<p>Whether to respect the site's crawling rules: never (0), sometimes (1), always (2, the
default), or always including the strict rules (3). Ignoring them on a site you do not own is
how mirroring gets people blocked.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Update hacks</span><span class="cli">-%s, --updatehack</span></div>
<p>Works around servers that misreport what changed, treating a file of identical size as
unchanged even when the timestamp moved. Saves a great deal of traffic on dynamic sites, at a
small risk of keeping a stale page.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Tolerant requests</span><span class="cli">-%B, --tolerant</span></div>
<p>Accepts responses that break the rules. Off by default, because a server that lies about a
file's length usually leaves you with a truncated one.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Force HTTP/1.0 requests</span><span class="cli">-%h, --http-10</span></div>
<p>For servers and proxies too old to cope with anything newer. It costs you most of the update
machinery, so use it only when a site genuinely fails without it.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">URL hacks</span><span class="cli">-%u, --urlhack</span></div>
<p>Treats addresses that differ only cosmetically as the same page (<code>www.example.com</code>
and <code>example.com</code>, doubled slashes, reordered query keys), so the same page is not
downloaded twice.</p>
<p>Three checkboxes opt out of one part each when a site really does treat the difference as
meaningful: <b>Keep the www. prefix</b> (<code>--keep-www-prefix</code>), <b>Keep double
slashes</b> (<code>--keep-double-slashes</code>) and <b>Keep the original query-string order</b>
(<code>--keep-query-order</code>).</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Seed the crawl from the site's sitemap</span><span class="cli">-%m, --sitemap</span></div>
<p>Reads the site's sitemap, the <code>Sitemap:</code> lines in <code>robots.txt</code> and then
<code>/sitemap.xml</code>, and adds every address it lists as a starting point. It reaches pages
no link on the site points to. <b>Sitemap address</b> (<code>--sitemap-url</code>) names one
directly instead of probing.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Load cookies from file</span><span class="cli">-%K, --cookies-file</span></div>
<p>Preloads cookies from a Netscape <code>cookies.txt</code> before the crawl starts. How you
mirror something that needs you logged in, by exporting the session from your browser.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Strip query keys</span><span class="cli">-%g, --strip-query</span></div>
<p>Comma-separated query keys to drop when naming saved files, such as
<code>sid,utm_source</code>. Keeps one page from being saved several times under tracking
parameters that change nothing.</p>
</div>
</section>
<section class="tab" id="opt-proxy">
<h3>Proxy</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-proxy.png" alt="WinHTTrack Proxy tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-proxy.png" alt="WebHTTrack Proxy tab" width="1024" height="762">
<img data-for="droid" src="img/guide-droid-opt-proxy.png" alt="Android Proxy tab" width="600" height="1300">
</figure>
<div class="opt">
<div class="opt-head"><span class="opt-name">Proxy type</span><span class="cli">-P scheme://…</span></div>
<p>The proxy protocol. <b>HTTP</b> is an ordinary proxy. <b>HTTP (CONNECT tunnel)</b> sends every
request through a CONNECT tunnel, which CONNECT-only proxies such as Tor's
<code>HTTPTunnelPort</code> require. <b>SOCKS5</b> defaults to port 1080.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Proxy address and port</span><span class="cli">-P, --proxy</span></div>
<p>The proxy to send requests through. On the command line one string carries the lot, including
a scheme and credentials where needed:
<code>--proxy socks5://user:pass@proxy.example.com:1080</code>.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Proxy login and password</span><span class="cli">-P user:pass@host:port</span></div>
<p>Filled in behind the <b>Configure</b> button, for a proxy that demands authentication.</p>
<p><span class="badge">not on Android</span> The Android app takes a host and a port only.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Use proxy for FTP transfers</span><span class="cli">-%f, --httpproxy-ftp</span></div>
<p>Sends <code>ftp://</code> transfers through the HTTP proxy as well. On by default, and worth
keeping: it gets FTP links through a firewall, and proxied FTP tends to be more reliable than the
engine's own client.</p>
</div>
</section>
<section class="tab" id="opt-log-index-cache">
<h3>Log, Index, Cache</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-log-index-cache.png" alt="WinHTTrack Log, Index, Cache tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-log-index-cache.png" alt="WebHTTrack Log, Index, Cache tab" width="1024" height="1278">
<img data-for="droid" src="img/guide-droid-opt-log-index-cache.png" alt="Android Log, Index, Cache tab" width="600" height="1300">
</figure>
<div class="opt">
<div class="opt-head"><span class="opt-name">Store all files in cache</span><span class="cli">-k, --store-all-in-cache</span></div>
<p>Normally only pages are cached, since that is all an update needs. This caches everything,
which lets you rebuild the mirror in a different layout later without downloading it again, and
makes the cache as large as the mirror.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Do not re-download locally erased files</span><span class="cli">-%n, --do-not-recatch</span></div>
<p>A file you deleted stays deleted through the next update, instead of coming straight back.
Useful while pruning something large by hand.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Create log files</span><span class="cli">-f, --file-log</span></div>
<p>On, and worth leaving on: without a log you have no way to find out what failed. The
debug level beside it controls how much detail lands there.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Make an index</span><span class="cli">-I, --index</span></div>
<p>Writes an <code>index.html</code> at the top of the project, listing the mirrored sites. The
front door to the copy.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Make a word database</span><span class="cli">-%I, --search-index</span></div>
<p>Builds a searchable word index of every mirrored page.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Write a WARC archive of the crawl</span><span class="cli">-%r, --warc</span></div>
<p>Saves every fetched response into an ISO-28500 WARC/1.1 archive alongside the mirror, headers
and all. The mirror is a browsable copy; the WARC is the archival record of what the server
actually sent, which is what replay tools and web archives want.</p>
<p><b>WARC archive name</b> (<code>--warc-file</code>) sets the base name, blank to auto-name it
under the output directory. <b>WARC segment size limit</b> (<code>--warc-max-size</code>) starts a
new segment past that many bytes; blank or 0 keeps one file.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Write a CDXJ index of the WARC archive</span><span class="cli">-%rc, --warc-cdx</span></div>
<p>Adds a sorted CDXJ index listing every record in the archive, which replay tools use to find a
capture without reading the whole file.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Package the crawl as a WACZ file</span><span class="cli">--wacz</span></div>
<p>Bundles the WARC, its CDXJ index and the mirrored pages into one WACZ file. It implies both of
the two above.</p>
</div>
<div class="opt" data-for="win web">
<div class="opt-head"><span class="opt-name">Report what changed since the previous mirror</span><span class="cli">-%d, --changes</span></div>
<p>Writes <code>hts-changes.json</code> listing what this crawl left new, changed, unchanged and
gone against the previous mirror. The quick way to see what an update actually did.</p>
</div>
</section>
<section class="tab" id="opt-mime-types">
<h3>MIME Types</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-mime-types.png" alt="WinHTTrack MIME types tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-mime-types.png" alt="WebHTTrack MIME types tab" width="1024" height="1046">
<img data-for="droid" src="img/guide-droid-opt-mime-types.png" alt="Android MIME types tab" width="600" height="1300">
</figure>
<div class="opt">
<div class="opt-head"><span class="opt-name">Type / MIME associations</span><span class="cli">-%A, --assume</span></div>
<p>Tells the engine what an extension always means, so it stops asking the server. On a site full
of <code>.asp</code> links this is a large speed-up: without it every unknown extension costs a
request before the engine knows what it has.</p>
<p>Enter a file type and the MIME type it maps to, <code>asp</code> to <code>text/html</code>,
and list several at once with commas: <code>asp,php,php3</code>. On the command line the same
thing reads <code>--assume asp,php,php3=text/html</code>, and
<code>--assume standard</code> covers the usual page extensions in one go.</p>
<p>It also renames on the way in. If you know the site's <code>.dat</code> files are really ZIPs,
map <code>dat</code> to <code>application/x-zip</code> and they land correctly named. Mapping an
unknown type to <code>application/octet-stream</code> simply stops the engine checking it.</p>
<table class="mime">
<tr><td><code>text/html</code></td><td>pages, which HTTrack parses for links</td></tr>
<tr><td><code>image/gif</code>, <code>image/jpeg</code>, <code>image/png</code></td><td>images</td></tr>
<tr><td><code>application/x-zip</code></td><td>ZIP archives</td></tr>
<tr><td><code>application/octet-stream</code></td><td>anything else, left alone</td></tr>
</table>
<p><span class="badge">Android</span> The table holds up to eight entries.</p>
</div>
</section>
<section class="tab" id="opt-experts-only">
<h3>Experts Only</h3>
<figure>
<img data-for="win" src="img/guide-win-opt-experts-only.png" alt="WinHTTrack Experts Only tab" width="524" height="448">
<img data-for="web" src="img/guide-web-opt-experts-only.png" alt="WebHTTrack Experts Only tab" width="1024" height="1088">
<img data-for="droid" src="img/guide-droid-opt-experts-only.png" alt="Android Experts Only tab" width="600" height="1300">
</figure>
<p>The defaults here are right for almost every mirror. The travel modes are the exception: they
are how you widen a crawl deliberately.</p>
<div class="opt">
<div class="opt-head"><span class="opt-name">Use a cache for updates</span><span class="cli">-CN, --cache N</span></div>
<p>Keep this on. The cache is what makes <b>Update existing download</b> and <b>Continue
interrupted download</b> possible at all; turning it off saves a little disk and costs you both.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Primary scan rule</span><span class="cli">-pN, --priority N</span></div>
<p>What gets saved: nothing (0, for checking links), pages only (1), everything but pages (2),
everything (3, the default), or pages first then the rest (7).</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Travel mode</span><span class="cli">-S -D -U -B</span></div>
<p>Which directories the crawl may enter relative to your starting address: this one only
(<code>-S</code>), it and below (<code>-D</code>, the default), it and above (<code>-U</code>), or
both directions (<code>-B</code>).</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Global travel mode</span><span class="cli">-a -d -l -e</span></div>
<p>How far the crawl may stray from the address you gave it: same address only (<code>-a</code>,
the default), same domain (<code>-d</code>), same top-level domain (<code>-l</code>), or anywhere
at all (<code>-e</code>). The last one will follow the web until a limit stops it.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Rewrite links</span><span class="cli">-KN, --keep-links N</span></div>
<p>What the links in the saved pages look like: relative, so the copy browses offline
(<code>K0</code>, the default), absolute, or the original online addresses (<code>K4</code>), which
gives you the pages as archived documents rather than a browsable mirror.</p>
</div>
<div class="opt">
<div class="opt-head"><span class="opt-name">Activate debug mode</span><span class="cli">-%H, --debug-headers</span></div>
<p>Writes the HTTP headers and other internals to the log. For diagnosing a site that behaves
strangely, not for normal use.</p>
</div>
</section>
<h2 id="next">Where to go next</h2>
<ul>
<li><a href="filters.html">Filter syntax</a>: the full pattern language behind Scan Rules.</li>
<li><a href="cmdguide.html">Command-line guide</a>: the same engine, scripted.</li>
<li><a href="httrack.man.html">Manual page</a>: every option, authoritative.</li>
<li><a href="faq.html">FAQ and troubleshooting</a>: when a mirror comes back wrong.</li>
<li><a href="abuse.html">How not to use HTTrack</a>: worth two minutes before pointing it at
someone else's server.</li>
</ul>
</main>
</div>
<dialog id="zoom" aria-label="Enlarged screenshot"><img src="" alt=""></dialog>
<footer>&copy; 1998-2026 Xavier Roche &amp; other contributors - Web Design: Leto Kauler.</footer>
</body>
</html>

127
html/guide.js Normal file
View File

@@ -0,0 +1,127 @@
/* Progressive enhancement for guide.html: the platform switcher and the option
filter. Image zoom and sidebar highlighting are shared, in doc.js. Everything
here is optional: with scripting off the page still carries every platform's
text and screenshots. */
(function () {
"use strict";
var PLATFORMS = ["win", "web", "droid"];
var STORE = "httrack-guide-platform";
/* Chrome denies storage to file:// origins, and the doc is read from disk. */
function remembered() {
try {
return localStorage.getItem(STORE);
} catch (e) {
return null;
}
}
function remember(platform) {
try {
localStorage.setItem(STORE, platform);
} catch (e) {
/* not fatal: the hash still carries the choice */
}
}
/* "#win/opt-limits" -> {platform: "win", target: "opt-limits"}, and a bare
"#win" is a platform, not a target of that name. */
function parseHash() {
var hash = location.hash.replace(/^#/, "");
var slash = hash.indexOf("/");
if (slash < 0) {
return PLATFORMS.indexOf(hash) !== -1
? { platform: hash, target: "" }
: { platform: null, target: hash };
}
if (PLATFORMS.indexOf(hash.slice(0, slash)) !== -1) {
return { platform: hash.slice(0, slash), target: hash.slice(slash + 1) };
}
return { platform: null, target: hash };
}
function setPlatform(platform) {
document.documentElement.setAttribute("data-platform", platform);
}
/* Runs before first paint, so the page never flashes the other platforms. */
var initial = parseHash().platform || remembered() ||
(/Android/i.test(navigator.userAgent) ? "droid" :
/Windows/i.test(navigator.userAgent) ? "win" : "web");
setPlatform(initial);
document.addEventListener("DOMContentLoaded", function () {
var current = document.documentElement.getAttribute("data-platform");
/* ---- platform switcher ---- */
var buttons = [].slice.call(document.querySelectorAll(".platforms button"));
function select(platform, push) {
current = platform;
setPlatform(platform);
remember(platform);
buttons.forEach(function (b) {
b.setAttribute("aria-pressed", String(b.dataset.platform === platform));
});
if (push) {
var target = parseHash().target;
history.replaceState(null, "", "#" + platform + (target ? "/" + target : ""));
}
}
buttons.forEach(function (b) {
b.addEventListener("click", function () {
select(b.dataset.platform, true);
});
});
select(current, false);
/* A "#win/opt-limits" link has no element of that id: scroll it ourselves. */
function jumpToHash() {
var parsed = parseHash();
if (parsed.platform) {
select(parsed.platform, false);
}
if (parsed.target) {
var el = document.getElementById(parsed.target);
if (el) {
el.scrollIntoView();
}
}
}
window.addEventListener("hashchange", jumpToHash);
if (parseHash().platform) {
jumpToHash();
}
/* ---- option filter ---- */
var filter = document.getElementById("optfilter");
var count = document.getElementById("optcount");
if (filter) {
var options = [].slice.call(document.querySelectorAll(".opt"));
options.forEach(function (opt) {
opt.dataset.haystack = opt.textContent.toLowerCase();
});
filter.addEventListener("input", function () {
var needle = filter.value.trim().toLowerCase();
var shown = 0;
options.forEach(function (opt) {
var hit = !needle || opt.dataset.haystack.indexOf(needle) !== -1;
opt.hidden = !hit;
shown += hit ? 1 : 0;
});
/* Fold away a tab whose options all filtered out. */
[].slice.call(document.querySelectorAll("section.tab")).forEach(function (section) {
section.hidden = needle !== "" &&
!section.querySelector(".opt:not([hidden])");
});
count.textContent = needle ? shown + " of " + options.length : "";
});
}
});
})();

View File

@@ -777,8 +777,8 @@ information purpose only) (%q0 don&rsquo;t include)
<td width="82%">
<p>*generate output html file in case of error (404..) (o0
don&rsquo;t generate) (--generate-errors)</p></td></tr>
<p>*save the server&rsquo;s error pages (404..) (o0 discard
them) (--generate-errors)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 94 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.8 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.8 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.7 KiB

Some files were not shown because too many files have changed in this diff Show More