ZIP_READFIELD_STRING in proxy/store.c took no destination size and used a
raw strcpy, while the engine's namesake in htscache.c has taken a
refvalue_size for years. The source is a cache-entry header line bounded
only by line[HTS_URLMAXSIZE + 2].
contenttype[64] sits just before the location pointer in the calloc'd
_PT_Element, so an over-long Content-Type walks over charset and then
over location, which the next Location: line copies through.
Clip rather than reject: the engine stores Content-Type in 128 bytes, so
a valid cache legitimately carries fields wider than ours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver builds the redirect Location header in a 256-byte stack buffer
The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.
Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.
The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: satisfy shellcheck and shfmt in the new server test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the pre-fix narration from the oversized-value comment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the bound socket, not the announced URL
The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.
Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: bound the response read and the empty --bind run
The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver: the session id must gate the request body, not the reply
Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.
Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.
A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.
Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.
Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsserver: serve the crawled mirror verbatim, never through the expander
WebHTTrack serves the mirror under /website/ from the same small server as its
own GUI, and the decision to run a response through the ${...} template
expander was a substring test for ".htm" on the request path. A mirrored page
therefore had its directives evaluated: ${_sid} rendered the live session id,
handing the crawled site the token that authenticates commands on the local
GUI, and ${do:...} gave it the rest of the template verbs.
The /website/ prefix was already detected, but only to keep the crawl-state
override from hijacking a mirror request. Reuse it as the expansion gate, so
expansion is limited to files under the GUI's html root, and re-evaluate it
after that override, which can substitute a GUI page for a mirror path.
Mirrored pages keep their text/html type: verbatim must not turn browsing the
mirror into a download.
Note that mirrored content still shares the control origin, so a script in it
can read the session id from a GUI page itself; that is a separate fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver: stop letting a posted projpath name the /website/ root
The static-file path in smallserver() was composed with a length guard that
measured only the server root and the request path, then wrote the posted
"projpath" field into a 1024-byte stack buffer: a 2000-byte projpath followed
by any /website/ request smashed the stack (buffer overflow detected, server
gone). The guard also summed two untrusted lengths before comparing, the shape
that can wrap and pass.
Composition now goes through a bounded, non-aborting append that keeps the
untrusted length alone on one side, and /website/ is served from the project
directory the server itself set up, rejecting a ".." in it, rather than from
whatever root the request body claimed. Without that, projpath=/etc/ plus
GET /website/passwd read an arbitrary file, since the ".." check looked at the
request path only. fsfile is also cleared before the error-redirect branch,
which could otherwise reach fopen() uninitialized.
tests/80_webhttrack-projpath.test drives the three cases against a live server
and keeps a legitimate project browsable as the control.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: establish test 79's mirror root through the server
/website/ no longer serves the posted projpath, so injecting one as a
fixture stopped working and test 79 got a 404 instead of the mirrored
page. Save a profile first (no command_do=start, so nothing crawls) to
make the server record the root, then plant the file under it.
Re-checked against a reverted expander gate: still fails there, so the
fixture change did not cost the assertion its teeth.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: cover the running-crawl half of the /website/ override
Test 84 only exercised the idle server, where the override never fires and
the recomputed virtualpath is indistinguishable from the stale one. Drive a
crawl through the server so /website/*.html is rewritten to the GUI refresh
page, which 404s without the recompute.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: cover the overflow, the '..' rejection and the fsfile hoist
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: fix the merged test 84 duplicate post() helper
The rename-side and the branch-side each defined post(), with different
argument shapes; the last one won and silently mangled the save body.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: trim the new comments
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the stale start() arg comments in test 84
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* WebHTTrack's mirror links are dead file:// URLs
The GUI is served from http://127.0.0.1:PORT/, and browsers refuse to navigate
from an http: page to a file: URL, so the "browse the mirror" links in
finished.html and file.html did nothing when clicked. They now point at the
server's own /website/ route, which already serves the project directory. The
desktop entry's browse mode still works, because there the shell hands the
file: URL to the browser instead of navigating from a page.
The per-project picker in file.html goes too: /website/ is bound to the running
project, and reaching the other projects over HTTP would mean serving the whole
mirror tree from the control origin.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* finished.html: link the served mirror, not a dead file:// URL
The page is served over http:, so its file:// link is a cross-scheme
navigation that Chrome and Firefox refuse. Nothing happened on click. The
mirror is already reachable at /website/, which the two list entries just
below were using all along.
file.html keeps its file:// links. It is the fresh-session entry point for
sites mirrored earlier, and /website/ is bound to one project, so pointing
it there would trade a dead link for a 404. Serving an arbitrary past
project needs a route that does not exist yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: tighten the browse-link assertion and trim its comments
The runtime check for href="/website/index.html" also matched the list
entry below the anchor, so it passed on the unfixed page; match the
anchor by its mirror-path label instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsserver builds the redirect Location header in a 256-byte stack buffer
The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.
Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.
The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: satisfy shellcheck and shfmt in the new server test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the pre-fix narration from the oversized-value comment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the bound socket, not the announced URL
The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.
Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: bound the response read and the empty --bind run
The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver: the session id must gate the request body, not the reply
Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.
Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.
A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.
Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.
Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsserver: serve the crawled mirror verbatim, never through the expander
WebHTTrack serves the mirror under /website/ from the same small server as its
own GUI, and the decision to run a response through the ${...} template
expander was a substring test for ".htm" on the request path. A mirrored page
therefore had its directives evaluated: ${_sid} rendered the live session id,
handing the crawled site the token that authenticates commands on the local
GUI, and ${do:...} gave it the rest of the template verbs.
The /website/ prefix was already detected, but only to keep the crawl-state
override from hijacking a mirror request. Reuse it as the expansion gate, so
expansion is limited to files under the GUI's html root, and re-evaluate it
after that override, which can substitute a GUI page for a mirror path.
Mirrored pages keep their text/html type: verbatim must not turn browsing the
mirror into a download.
Note that mirrored content still shares the control origin, so a script in it
can read the session id from a GUI page itself; that is a separate fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: cover the running-crawl half of the /website/ override
Test 84 only exercised the idle server, where the override never fires and
the recomputed virtualpath is indistinguishable from the stale one. Drive a
crawl through the server so /website/*.html is rewritten to the GUI refresh
page, which 404s without the recompute.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* webhttrack: "max site size" set a per-file cap instead of the overall one
step4.html mapped all three size fields of the wizard onto --max-files (-m),
so "Max site size" emitted a per-file limit rather than --max-size (-M), and
landed a second -m on the command line. That second -m also clobbered the HTML
per-file limit: a bare -m<n> resets maxfile_html, so whichever of the two came
last won. Point sizemax at --max-size and emit the bare -m before the -m,<n>
form so both per-file caps survive.
Two template typos in the same family, where a malformed ${...} renders as
nothing or as its own key instead of erroring: the winprofile.ini writer's
Dos=${dos was missing its closing brace, and option2b.html's OK button read
${LANG_OK] with a bracket.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* tests: carry the session id in test 81's POST
The gate that landed with #700 refuses a body without one, so the wizard
POST came back refused and the option audit had nothing to read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: close the confirmation-biased gaps in test 81
Post the size fields empty too, so a step4.html that lost its ${test:} guard
and rendered a valueless --max-files= is caught; assert the winprofile.ini
MaxHtml/MaxOther/MaxAll keys the header claimed to audit; and pin the OK button
label, since an unknown ${LANG_} key renders empty and passed the absence check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `sprintfbuff()`/`slprintfbuff()` to `htssafe.h`: a formatted print that truncates to fit and returns whether it had to, marked `warn_unused_result` so the answer cannot be dropped. It fills the gap between the `strcpybuff` family, which aborts on overflow, and `String`, which grows without bound. Abort is the wrong contract wherever the text is built from a remote peer's reply.
Four `-Wformat-truncation=` sites used the result as if `snprintf` had never truncated. Two only needed a bigger destination, so they get one: `hts_finish_makeindex`'s `tempo` was a flat 1024 against a 2048-byte escaped URL and is now sized off it, and the wizard's `cmd[4096]` could not hold the answers it concatenates. The other two cannot grow. `create_back_tmpfile` formats `<url_sav>.bak` into a buffer the same size as `url_sav`, and that struct is installed, so a dropped extension would alias the backup onto the live file that `back_finalize_backup()` unlinks. ProxyTrack's `startUrl[1024]` is fed by cache content. Both take the error path they already had, and ProxyTrack moves to the next cache entry rather than publishing a clipped one.
On what the wrapper buys, since it is not what I first assumed: checking a raw `snprintf` return inline silences the warning just as well. The wrapper's value is that the capacity comes from `sizeof`, the check is the default rather than the exception, and `warn_unused_result` makes skipping it visible.
`-#test=strsafe` covers the primitive (exact fit, one over, 4 KB source, `size == 1`, trailing canary, destination repoisoned between cases), and `-#test=makeindex` gains a first link whose escaped form overruns the old buffer. Both were mutation-checked. The ProxyTrack change ships without a direct test: its only observable is the catalog page, which renders solely as a PROPFIND fallback I could not drive from curl. 25 gcc warnings down to 21; the rest of the cluster is diagnostic-only and follows separately.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsserver builds the redirect Location header in a 256-byte stack buffer
The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.
Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.
The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: satisfy shellcheck and shfmt in the new server test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the pre-fix narration from the oversized-value comment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the bound socket, not the announced URL
The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.
Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: bound the response read and the empty --bind run
The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver: the session id must gate the request body, not the reply
Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.
Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.
A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.
Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.
Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* webhttrack: bound the argv vector and escape quotes in the wizard command line
The wizard hands its httrack command line to the engine as one string, which
back_launch_cmd() split back into argv. Two things were wrong with that split.
It wrote into a fixed 1024-pointer vector with no bound, and every unquoted
space in the posted string yields an entry, so an ordinary mirror with a few
hundred URLs walked off the allocation. The split now lives in htscmdline.c as
hts_split_cmdline(), which sizes the vector from the separator count before
filling it, and the engine self-tests can reach it.
Quotes were also purely advisory: they toggled the "inside an argument" state
but nothing escaped them, so a double quote typed into a wizard field (user
agent, footer, path, project name) closed the argument early and the rest of
the value was parsed as fresh options -- among them -V, which reaches system().
Escaping has to happen where the argument boundary is known, so the template
gets its own ${arg:} filter for that context; ${html:} keeps its meaning for
the HTML attributes it is used in everywhere else, and HTML escaping would not
help anyway since the browser undoes it when it posts the command line back.
${arg:} backslash-escapes a quote and a backslash, and the splitter reads those
inside a quoted run, the same convention next_token() already implements for
doit.log. A value containing a quote now survives it intact instead of turning
into options.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* webhttrack: keep a stray quote in an unquoted field out of the split
The url and wildcard-filter fields go into the command line outside quotes,
where no backslash can escape anything: a single quote there flips the parity
of every quote after it, so a later escaped value ends up split as flags and
the escaping buys nothing. Emit %22 for those fields instead.
NULL-terminate the argv vector while here, matching the convention the tree
documents in htscharset.c, and fold the third copy of the entity table into
one helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: do not feed snprintf's return value back as its size argument
snprintf returns the length it wanted to write, so accumulating it blind
lets the next size argument wrap. The buffer is sized well past what the
loop needs, but the pattern is the one the project forbids.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Symbolize fatal-signal backtraces through addr2line
backtrace_symbols_fd() resolves names from .dynsym only, and
-fvisibility=hidden keeps every engine frame out of it, so a crash report
arrived as a column of bare module+offset. The handler now emits the raw trace
first and unconditionally, then groups the frames per module and runs addr2line
(or llvm-symbolizer) over the offsets, which reads DWARF and names the static
frames plus their inline chain.
-rdynamic is dropped: it only populated .dynsym and bought exactly one named
frame. -Wl,--build-id replaces it, so a trace from a stripped build can be
matched to its debug symbols.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Move the crash backtrace printer into src/htsbacktrace.c
httrack.c keeps only the two call sites. The symbolizer needs _GNU_SOURCE for
dladdr(), which is now confined to its own translation unit instead of being
forced on the whole CLI front-end.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test: match glibc's backtrace format with or without the space
backtrace_symbols_fd() prints the trailing "[0xADDR]" with a leading space on
some glibc versions and without on others (Ubuntu 24.04), so the raw-frame
assertion failed everywhere but the dev box. Match only up to the offset.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Skip pseudo-modules with no file on disk
A frame in linux-vdso.so.1 made addr2line complain instead of resolving, so
the arm64 leg lost its symbolized output entirely. Renumber the test too:
77 landed on master with #700.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Stop discarding local symbols, which made the names wrong
--discard-all drops the local symbol of every static function, so addr2line
attributes the frame to the nearest surviving global: the abort frame read
dns_timeout_selftests instead of abortf_, with the file and line still right.
A wrong name is worse than none, and because the symbols go at link time no
-dbgsym package can recover them. Costs 21784 bytes on libhttrack.so.3, 0.6%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Symbolize once when the handler itself faults
A fault inside the handler re-enters the printer, which interleaved a second
symbolized trace on the same fd and spent a second budget: 3.04s and two
overlapping reports, measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop NULL tests on inline array members
`lien_back::url_sav`, `htsblk::msg` and POSIX `dirent::d_name` are arrays,
so testing their address folds to a constant and gcc/clang report it
(-Waddress, -Wpointer-bool-conversion). Every site keeps whatever real
condition sat beside the dead one, so behavior is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* help_wizard: check the allocation, not the arrays it contains
The out-of-memory guard has been constant-false since d593418 folded the
nine separate wizard buffers into one struct: the names it tests are now
inline arrays, so `malloct()`'s result is never checked and an exhausted
heap gets a NULL-page write instead of the intended message. Also switch
the raw free() to freet() and release the struct on the two early returns
that leaked it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* proxytrack: test the WebDAV header fields, not their addresses
`PT_Element::lastmodified` and `::contenttype` are inline arrays, so both
`if`s were constant-true (-Waddress); use the `[0]` form the same file
already uses when it emits the GET headers. Neither is observable:
get_time_rfc822("") returns 0 and falls through to the index timestamp,
and proxytrack_add_DAV_Item already substitutes application/octet-stream
for an empty mime, which a PROPFIND probe against a cache entry carrying
no Content-Type confirms both before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the discarded const qualifiers rather than casting them away
`binput` and `cache_binput` only ever read through their source pointer,
so they take `const char *` now; that alone clears the cast in
htsrobots.c, and neither is exported nor declared in an installed header,
so no ABI question arises. `treathead` keeps `char *rcvd` because it does
NUL-cut the header in place, and the two selftest calls that fed it a
string literal get a mutable buffer instead, matching their three
siblings and removing a latent write to .rodata. The remaining two are
one-liners: zlib's `next_in` is already `const` under -DZLIB_CONST, and
htsback can call the non-const `jump_protocol` twin on its mutable
`url_adr`. libhttrack.vcxproj gains ZLIB_CONST so the MSVC build agrees
with autotools, as webhttrack and proxytrack already do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: prove the WebDAV mime/timestamp fallback survives an empty field
proxytrack's DAV PROPFIND response computes a fallback content-type and
timestamp when a cache entry has no Content-Type/Last-Modified; that
fallback already existed before commit eae1dd0 changed the surrounding
always-true array-address checks, so this test guards the equivalence
rather than a bug. Verified it fails when the fallback default is
disabled, and passes unmodified against the pre-eae1dd0 code too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: bound the PROPFIND request and cut the header comment
curl had no --max-time; an unbounded read wedges the runner instead of
failing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: skip the WebDAV mime test on Windows
It is the first test to run proxytrack as a live listener, and MSYS cannot
reap a native one: the orphan wedged the whole Windows suite past its
45-minute budget, twice, destroying the log upload with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* htsserver builds the redirect Location header in a 256-byte stack buffer
The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.
Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.
The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: satisfy shellcheck and shfmt in the new server test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: drop the pre-fix narration from the oversized-value comment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the bound socket, not the announced URL
The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.
Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: bound the response read and the empty --bind run
The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* htsserver: the session id must gate the request body, not the reply
Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.
Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.
A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.
Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.
Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* -%S list file over 4GB overflows the heap
hts_main_internal() sized the -%S buffer as `cl + fz + 8192` and stored
the sum in an int url_sz, then fread() the untruncated 64-bit fz into it.
A 4GB+100KB rules file wraps the capacity to 110602 bytes on x64, the
realloct() succeeds, and the read walks off the heap. Intermediate sizes
land on a negative int and fail the allocation, which is luck, not design.
Route the file-size arithmetic through llint_grow_size_t(), a saturating
sibling of llint_to_size_t() that refuses a total it cannot represent, and
widen url_sz and the filelist offsets to size_t. The "config" sizing in the
same function and htscore.c's primary_len had the same shape: a file size
accumulated into an int before reaching an allocator. htscache.c's two
mirrored-file comparisons held a 64-bit fsize_utf8() in a size_t, which
truncates on Win32 and re-downloads a >4GB file already on disk.
Found by MSVC C4244 on x64; invisible to gcc/clang because int64_t to
size_t is width-preserving on LP64.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Print T_SOC with the format matching its width, not a bare %d
T_SOC is unsigned __int64 on Win64 (htsglobal.h): passing it to fprintf's
%d is undefined behavior, flagged by MSVC C4477. Add T_SOCP beside the
typedef, following the existing LLintP/INTsysP precedent, and use it at
both deletesoc() call sites (htslib.c:2601, :2607).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix MSVC C2099 in the new growsize self-test
A static const object used inside another object's static initializer is a
GNU/clang extension, not standard C: MSVC's /TC C mode rejects it ("initializer
is not a constant"). Replace the over32 local with a macro.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: cover the slack-only overrun and the largest capacity
A helper dropping the slack bound passed the table; -1 as extra also refused
either way, since llint_to_size_t() maps it to SIZE_MAX regardless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ARC cache replays a truncated HTTP reason phrase
proxytrack bounded the reason-phrase copy out of an ARC index by
sizeof(pos) - 1 where pos is a const char *, so a stored "404 Not Found"
replays as "404 Not Fou" (and "404 Not" on 32-bit). Use strncatbuff with
the destination's own size.
Fold the nine copies of the buff() family's source-capacity expression
into HTS_SIZEOF_SRC_, applying sizeof to the type so a decayed operand no
longer trips -Wsizeof-array-decay at five call sites; MSVC keeps the old
expression behind the guard HTS_IS_CHAR_BUFFER already uses. The two
other raw strncat calls become strncatbuff, htsbuff_catn stops handing
strnlen the (size_t)-1 sentinel, and htsweb.c no longer compares ep
against a NULL eps.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Size the new strsafe buffers away from sizeof(char*)
char[8] equals a pointer on LP64, so MSVC's array-vs-pointer heuristic
read the unterminated source as a pointer, skipped the bound and never
aborted; the x64 build failed while Win32 passed. Same trap the existing
comment in that function already warns about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* configure: fix flag probes, move -rdynamic to the link line
AX_CHECK_COMPILE_FLAG only checks the exit status, and clang merely warns on
an unknown -W name, so -Wmissing-parameter-type reached every clang build and
warned on all 66 TUs. Probe with -Werror; -Wformat-nonliteral also needs
-Wformat there or gcc rejects it and the flag would be lost.
-rdynamic lived in DEFAULT_CFLAGS, i.e. AM_CPPFLAGS, so no link line ever saw
it; make it a link check. Drop -pie from CFLAGS_PIE (LDFLAGS_PIE has it), and
drop -Wdeclaration-after-statement, a C90 rule the gnu17 build does not follow
anywhere else.
Distinct build warnings: gcc 82 -> 54, clang 54 -> 41.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* configure: tighten the two new flag-block comments
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
WinHTTrack is an MBCS build, so it still lands on the ANSI-only Win32 entry points (TTN_NEEDTEXTA and the like) while the engine hands it UTF-8. The other direction is already exported as `hts_convertStringSystemToUTF8`, so this adds the mirror, `hts_convertStringUTF8ToSystem`, a one-liner over the existing `hts_convertStringCPFromUTF8`. The GUI can then delegate instead of keeping its own MultiByteToWideChar/WideCharToMultiByte copy (`CopyTextUTF8ToCP` in newlang.cpp, added while fixing #114). `hts_convertStringFromUTF8` would also have done the job, but it is declared plain `extern` and never reaches the DLL export table; left alone here. Windows-only addition, so no POSIX ABI change and no soname move.
The new `-#test=syscharset` self-test round-trips against the raw Win32 two-step it replaces, and skips off Windows.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* CLI: redraw the whole screen when the terminal is resized
The animated -%v display assumed a fixed 80x24 layout: it cleared the screen
once at start, and afterwards cleared only to end of line on the rows it
wrote. A resize left stale wrapped text around the frame, and on a terminal
shorter than the 20-row frame the stats block scrolled off the top at every
refresh.
Poll the terminal geometry at each refresh (TIOCGWINSZ, or the console screen
buffer info on Windows) and repaint in full when it moved. The in-progress
list is capped to the rows that fit, and the URL column follows the width; it
stays at the historical 40 characters on an 80-column terminal.
Closes#97
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: assert the resize repaint, the row clamp and the URL width
The first pass only checked that a clear-screen followed one resize, which a
display clearing on every frame passes just as well, and neither the row clamp
nor the URL budget was exercised at all: the single 24x80 to 40x100 resize
leaves both at their 80x24 values.
The harness now waits for the display instead of sleeping, resizes width and
height separately, and asserts a repaint after each one, none in between, a
frame that fits a 10-row terminal, and a long URL that stops being truncated at
200 columns. The long path is a new /trickle/deep route, so the shared
/trickle/ index other tests assert on stays untouched. Mutating any of the four
behaviors away fails the test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dependabot only watched vcpkg, so the action pins drifted by hand:
windows-build.yml sat on actions/checkout@v4 while everything else moved to
v6. Add the github-actions ecosystem and bump that stray checkout to v6.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vcpkg builds openssl/brotli/zlib/zstd from source on every Windows run.
Redirect its files binary cache into the workspace and persist it with
actions/cache, keyed on the manifest (which carries the builtin-baseline).
x-gha, which used to do this, was dropped from vcpkg-tool (#1662) after
GitHub changed the Actions cache API; actions/cache over the archive dir is
the endorsed replacement and stays within the GitHub-owned-actions policy.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop the removed Java .class parser from the credits
The Java binary .class parser was removed in #552, but its credit lingered:
"JavaParserClasses: Yann Philippot" in the WinHTTrack About box (lang.def plus
every lang/*.txt), and "for the java binary .class parser" in greetings.txt and
the contact page. Remove the About-box line and the now-defunct descriptor,
keeping Yann Philippot's name in the developed-by acknowledgments.
lang.def and lang/*.txt are read as strict line pairs, so the edit deletes the
substring inside the single credits line rather than removing any line, keeping
the pairing intact and each translated file's English msgid in lockstep with
lang.def. Verified by tests/62_lang-integrity.test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Also drop the localized java-parser credit from the translations
The About box shows the translation (msgstr), not the English msgid. The first
pass only removed the literal English "JavaParserClasses: Yann Philippot", so the
12 files whose translators localized the credit (Chinese, French, Russian,
Turkish, ...) still displayed it. Remove the localized credit line from each of
those msgstrs too; the pairing and every English msgid are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Reword Yann Philippot's credit as a past contributor
Dropping the descriptor left a bare name in the developed-by list. Mark the
java binary .class parser as a past contribution instead, keeping the
acknowledgment meaningful now that the code is gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* contact.html: use a literal dot in .class
The " dot " spelling is the page's email anti-harvest obfuscation; applied to a
filename extension it just left a stray double space. The page already writes
literal dots elsewhere (v2.0, v3.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the version in configure.ac, htsglobal.h and version.rc, move VERSION_INFO to 3:6:0, and add the history.txt and debian/changelog entries.
The WARC fields landed at the tail of httrackp and htsblk, but the embedded htsoptstate and htsblk members sit mid-struct, so the fields after them shifted. Installed layouts moved while the soname stays .so.3; the VERSION_INFO comment records that rather than claiming the layouts are unchanged.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Windows: import IE cookies via FindFirstFileW so long/non-ASCII jars load
cookie_load's IE-cookie scan and its cookies.txt read used the narrow
ANSI file API (FindFirstFileA, fopen, remove), so a cookie folder with a
non-ASCII or >MAX_PATH path was silently skipped, and any matched IE
cookie name was fed back through the mirror path as CP_ACP bytes. Route
the glob through hts_pathToUCS2 + FindFirstFileW, convert cFileName to
UTF-8 before rebuilding the path, and use the FOPEN/UNLINK wrappers.
The A->W find and the sink swaps land together on purpose: swapping only
the sink would feed a CP_ACP name into hts_fopen_utf8's UTF-8 decode and
mojibake the accidentally-consistent ANSI path. hts_pathToUCS2 is
un-static'd (internal, still -fvisibility=hidden / not HTSEXT_API) so the
glob gets the same \\?\ long-path treatment as the other wrappers.
Last piece of the #133 Windows long-path series. Self-test
-#test=cookieimport drives cookie_load against a long, non-ASCII folder;
POSIX is a positive control (IE block compiled out), the wide glob and
IE import run on the Windows legs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim review-flagged comments to one line
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Windows: enumerate long or non-ASCII directories via FindFirstFileW
The opendir/readdir emulation was fully ANSI (FindFirstFileA, CP_ACP
cFileName), so it capped listing at MAX_PATH and mis-decoded a non-ASCII
path — the odd wrapper out in an engine that feeds UTF-8 paths and reads
d_name as UTF-8. The one live Windows caller, the end-of-crawl
hts-cache/ref cleanup, silently no-op'd at a long or non-ASCII project
root, leaking the temp ref/ directory into the finished mirror.
Route both through hts_pathToUCS2 (\\?\ prefixing, #684) and the wide
APIs, converting cFileName back to UTF-8. d_name now holds UTF-8, so
HTS_DIRENT_SIZE grows to 1024 to fit MAX_PATH's worst-case expansion; the
struct is _WIN32-only, no POSIX ABI impact. Self-test direnum drives it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim comments to one line each (review follow-up)
Condense the opendir/readdir and direnum-selftest comments per the
review's comment-conciseness pass; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* topindex: convert the winprofile.ini category to UTF-8 on Windows (#216)
The project category read from hts-cache/winprofile.ini comes back in the
Windows ANSI codepage and is written into the charset=utf-8 topindex
template, so a non-ASCII category renders as mojibake. Convert it the same
way #681 did the project name. The topindex self-test now writes a
winprofile.ini with a non-ASCII category and asserts the generated index
carries the UTF-8 form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* topindex: drop the redundant category-assert comment
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 01_engine-fsize self-test builds a 5GB sparse file to exercise the 32-bit size wrap. On GNU/Hurd i386 the ext2fs translator rejects the extend with EFBIG, failing the Debian build on that port. Return 77 (automake skip) instead of 1 when the extend fails with EFBIG, and map that to a skip in the .test wrapper. Any other errno, or a wrong size report, still fails, so real regressions are not masked.
* Windows: route mirror-tree file ops through the UTF-8/long-path wrappers
The engine's raw fopen/remove/rename/rmdir/mkdir/fexist/fsize calls on
mirror-tree paths bypassed the UTF-8 (_w*) wrappers, so on Windows a mirror
under a non-ASCII or >MAX_PATH directory silently broke: a long/non-ASCII
file read as absent (its unlink/reget skipped), or its bytes went to a
mojibaked, truncated path.
Swap those call sites in htscoremain/htscore/htsparse/htsindex/htshelp to
the FOPEN/UNLINK/RENAME/MKDIR/fexist_utf8/fsize_utf8 twins, and add
hts_rmdir_utf8 (RMDIR) for the three rmdir sites that lacked a wrapper. The
arguments are already UTF-8 (Windows argv is transcoded at startup, fconcat
is byte-transparent), so there is no double-conversion. Raw calls on
genuinely system-charset or ASCII paths (getenv/$HOME, structcheck's ASCII
twin, "config") are left as-is.
-#test=mirrorio drives a long AND non-ASCII path through the wrapped guards
and the new rmdir wrapper; on POSIX it stands as a byte-transparent control.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* test: cover RENAME on the long+non-ASCII mirror path
The mirrorio self-test drove FOPEN/UNLINK/RMDIR but not RENAME, which the
sweep newly routes to. Rename the leaf to a non-ASCII sibling and verify
the move before teardown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Windows: reach past MAX_PATH via \\?\ at the file-op choke-point (#133)
The _w* file wrappers on Windows all funnel their path through one
converter. Route it through a new hts_pathToUCS2 that, for a path near
MAX_PATH, absolutizes and normalizes it with GetFullPathNameW and
prepends the "\\?\" verbatim prefix ("\\?\UNC\" for UNC shares) so the
wide file APIs accept paths past 260 chars. Shorter paths keep today's
exact behavior, so nothing changes until the naming ceiling in htsname.c
is raised (a follow-up); a -#test=longpath self-test drives a >260-char
path through the wrappers to exercise the new branch on Windows CI now.
Partially addresses #133.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Run the long-path self-test on the Windows CI leg; tighten comments
The Windows job iterates an explicit test glob (01_engine-*, 01_zlib-*,
*_local-*, ...), so 76_engine-longpath-io.test matched nothing and the
\\?\ path never ran on the one platform that needs it. Rename it to
01_engine-longpath-io.test so the glob picks it up and the prefixing is
actually exercised on Windows. Also trim the review comments to the why.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use the platform path limit off Windows, not the Windows MAX_PATH (#133)
url_savename capped every saved path at 236 chars (MAX_PATH minus 8.3
headroom minus the ".delayed" marker) on all platforms, so Linux, macOS
and Android hashed long names to fit a limit only Windows has. Off
Windows, derive the ceiling from the platform's own PATH_MAX/NAME_MAX,
clamped to the fixed save buffer so an oversized output dir can never
push the final path past it and abort. Windows keeps the MAX_PATH
ceiling until the engine can prefix its paths with \\?\.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cut the save name in bytes, not codepoints, so multibyte paths can't overflow
The path-length guard measured length in UTF-8 codepoints (hts_stringLengthUTF8)
while the buffer that receives parent+name is a fixed byte array whose overflow
aborts() rather than truncates. With the ceiling raised to ~1984 on POSIX, a
multibyte name (e.g. a long CJK path) can stay under the codepoint cap yet run
its byte length past the 2048-byte buffer, crashing the crawl on the final
prepend. Add an unconditional byte-boundary cut before the parent is prepended,
and drop the earlier codepoint-unit clamp it supersedes. Covers the oversized
-O parent case in the same measure. Regression test uses a 339-char CJK name
under a ~1020-byte output dir, which aborted before the cut.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Leave an oversized parent to the existing abort, not an empty colliding name
When the output dir alone fills the save buffer, forcing the name to empty made
every URL under it collapse to the same path, feeding the unbounded collision
sprintf and turning a clean abort into a 1-2 byte overflow. Only shrink the
name; a buffer-filling parent aborts on the prepend exactly as before (that
oversized-parent path is pre-existing and unrelated to the #133 ceiling).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* topindex: non-ASCII project names render as mojibake on Windows (#216)
hts_buildtopindex() lists each sub-project by the name FindFirstFileA
returns in the ANSI codepage, then writes it into a document declaring
charset=utf-8, so a non-ASCII name shows up as mojibake. Convert the name
to UTF-8 on Windows, matching the gif-path fix already in this function
for #217. The topindex self-test now builds a non-ASCII sub-project and
asserts the generated index carries the UTF-8 name.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* topindex: drop const so freet() can null the converted name (MSVC)
The #ifdef _WIN32 conversion block only compiles on Windows, where MSVC
rejected freet() nulling a char *const (C2166). Make the pointer mutable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* topindex: tighten the added comments
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The command-line guide's recipe section had no entry for WARC output, so
the five --warc* flags were discoverable only through --help or the man
page. Add a §11 recipe leading with the gotcha that trips people: the
archive is written alongside the browsable mirror, not instead of it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WARC is a transaction-level archive of what was fetched (a sibling of the
hts-cache), not a browsable-mirror build format like MHTML, so it belongs in
the Log/index/cache group next to the cache options. Help text only; the
webhttrack GUI already places it on the Log/Index/Cache tab (option9).
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verbatim compressed bodies (Content-Encoding kept, only hop-by-hop
Transfer-Encoding dropped, Content-Length rewritten to the stored length)
are now the sole behavior, matching wget --warc and Heritrix. The former
default that decoded the body and stripped Content-Encoding is gone, along
with the --warc-verbatim switch (added the same day, never released), the
warc_verbatim option field, and the dual-mode plumbing in
normalize_http_headers.
This also fixes a cap-truncated compressed response: it previously stored
the raw compressed partial while stripping Content-Encoding, mislabeling
gzip bytes as identity. Keeping Content-Encoding makes the record's label
match its body; the truncation still carries WARC-Truncated: length/time.
header_is now tolerates whitespace before the ':' so a non-compliant
"Content-Encoding : gzip" is still recognized.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add --warc-cdx: sorted CDXJ index alongside the WARC archive
WARC v2 PR B. --warc-cdx (alias --warc-cdxj, sub-flag -%rc) writes a sorted
CDXJ index next to the .warc.gz, one line per response/revisit/resource record
(warcinfo/request are not indexed): the SURT sort key, a 14-digit timestamp,
and a JSON object with url, mime, status, payload digest, and the record's byte
offset and length in the gzip stream so a replay tool can seek and inflate a
single member. Rotation-aware (the filename field tracks the current segment).
SURT canonicalization is new (surt_canon in htswarc.c): scheme/userinfo
dropped, host lowercased with a leading www[digits] label and the scheme
default port stripped, labels reversed and comma-joined, a non-default port
kept, IPv4 and [IPv6] literals left verbatim. Lines are accumulated in the
writer and qsort'd in LC_ALL=C byte order at close.
Tested by -#test=warc-surt (SURT vectors, 01_engine, MSan-instrumented) and
-#test=warc-cdx (end-to-end: sorted, one line per record, each offset/length
independently inflates to the matching WARC-Target-URI; 01_zlib).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Add --wacz: package the WARC archive, CDXJ index and pages as a WACZ file
--wacz bundles the crawl's WARC segment(s), the sorted CDXJ index and a
generated pages.jsonl into a single WACZ 1.1.1 package at crawl end, using
the in-tree minizip. It implies --warc and --warc-cdx. Every ZIP entry is
stored (no re-deflate of the already-gzipped WARC), as the WACZ spec
requires. datapackage.json lists each file with its SHA-256 and size, and
datapackage-digest.json chains the digest of datapackage.json.
SHA-256 comes from OpenSSL; a build without OpenSSL cannot emit conformant
digests, so --wacz is refused there with a clear log line while the WARC and
CDXJ are still written.
pages.jsonl captures the top-level 200 text/html responses (URL + WARC-Date),
bounded like the CDX accumulator; the first seeds datapackage mainPageUrl.
Tests: an in-process self-test (-#test=warc-wacz) unzips the package and
asserts the layout, STORE mode on every entry, each recomputed SHA-256, the
digest chain, and the pages header; a crawl test validates a real .wacz with
a stdlib validator (py-wacz/pywb when importable). Both skip cleanly without
OpenSSL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Package WACZ atomically so a failed re-run can't destroy a good archive
warc_wacz_package opened <base>.wacz directly with a create/truncate zip
handle, wiping the previous archive before it checked its inputs existed. A
re-run that produced zero indexable records (empty crawl, or every fetch
erroring so warc_cdx_flush writes no .cdx) truncated the good .wacz, failed to
find indexes/index.cdx, then unlinked the now-empty file with no log. Same
data-loss class as the hard-abort file truncation (#522).
Build the package into <base>.wacz.tmp and only rename it over <base>.wacz on
full success (RENAME, with an unlink+rename fallback so Windows can clobber).
On any error unlink the temp, leave the existing archive untouched, and warn
(the old error path was silent); the writer keeps opt for close-time logging.
The warc-wacz self-test now proves it: after a good package it drops the .cdx,
re-runs empty, and asserts the .wacz is byte-unchanged. Also tightens both the
C self-test and the Python validator against the WACZ spec (profile,
wacz_version, digest path, and >= 1 pages.jsonl body row with url + ts) so a
writer emitting an empty package can't pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* WARC: add --warc-verbatim to store compressed bodies as received
--warc-verbatim (internal -%rv, implies --warc) captures a compressed
response in its as-received content-coded form instead of the decoded
strategy-B body. back_finalize already materializes the whole de-chunked
compressed body as a temp file just before decoding it, so the tee is a
save-before-unlink: adopt that spool onto the new htsblk.warc_rawpath /
warc_rawsize instead of unlinking it, and emit the response record with
Content-Encoding kept and Content-Length set to the compressed length
(normalize_http_headers gains a keep_ce mode). WARC-Payload-Digest is
then over the coded payload, which is what the record carries.
Non-compressed responses, the never-spooled is_write archive-ext case,
and the cap-truncated second emit site all leave warc_rawpath NULL and
fall back to strategy B unchanged. The spool is owned by the entry:
freed and unlinked in warc_free_request, NULLed in back_copy_static and
back_unserialize so a shallow copy never double-unlinks.
Output is now byte-replay-identical to what the origin sent, matching
wget --warc and Heritrix.
Tests: a warc-verbatim engine self-test asserts the response keeps
Content-Encoding: gzip, Content-Length == the compressed length, the
stored bytes equal the gzip input and inflate to the known plaintext
(bite-checked); tests/74_local-warc-verbatim crawls a gzip-served
fixture and runs the strategy-A/B differential through
warc-validate.py --verbatim (inflate(stored) == the decoded body).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* WARC verbatim: cover the direct-to-disk adoption path, harden the validator
Review follow-up to --warc-verbatim (strategy A).
The spool adoption in back_finalize runs on both the in-memory and the
direct-to-disk (is_write) branch, but only the in-memory path was crawl-tested.
Add a non-hypertext gzip fixture (application/octet-stream) so the crawler
streams it to disk and the is_write adoption is exercised; the warc-validate
--verbatim gate now asserts Content-Encoding kept, Content-Length == compressed
length, and inflate(stored) == the served plaintext for both assets. Confirmed
at runtime that page.html takes is_write=0 and data.bin is_write=1, and
bite-checked that dropping the is_write adoption fails the new assertion.
warc-validate --verbatim now requires WARC-Payload-Digest on a verbatim record
when the file emits digests at all (an OpenSSL build), so a regression that
drops the digest is caught without breaking the no-OpenSSL leg.
Extract warc_adopt_rawspool() so the WARC spool bookkeeping lives behind the
htswarc boundary, mirroring warc_stash_response. Comment trims throughout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WARC v2 PR B. --warc-cdx (alias --warc-cdxj, sub-flag -%rc) writes a sorted
CDXJ index next to the .warc.gz, one line per response/revisit/resource record
(warcinfo/request are not indexed): the SURT sort key, a 14-digit timestamp,
and a JSON object with url, mime, status, payload digest, and the record's byte
offset and length in the gzip stream so a replay tool can seek and inflate a
single member. Rotation-aware (the filename field tracks the current segment).
SURT canonicalization is new (surt_canon in htswarc.c): scheme/userinfo
dropped, host lowercased with a leading www[digits] label and the scheme
default port stripped, labels reversed and comma-joined, a non-default port
kept, IPv4 and [IPv6] literals left verbatim. Lines are accumulated in the
writer and qsort'd in LC_ALL=C byte order at close.
Tested by -#test=warc-surt (SURT vectors, 01_engine, MSan-instrumented) and
-#test=warc-cdx (end-to-end: sorted, one line per record, each offset/length
independently inflates to the matching WARC-Target-URI; 01_zlib).
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Expose --warc / --warc-file in the webhttrack GUI
Add a WARC output checkbox and an optional archive-name field to the
"Log, Index, Cache" option tab, beside store-all-in-cache. The checkbox
emits --warc (auto-named archive); the text field emits --warc-file NAME.
Wiring mirrors how #589 added cookies-file and strip-query: the option9.html
form fields, the generated httrack command and winprofile.ini in step4.html,
and the reload remap in step2.html.
New LANG_WARC / LANG_WARCFILE strings and tooltips land in lang.def,
English.txt and Francais.txt; the remaining 28 language files fall back to
French for now and are a follow-up.
The webhttrack smoke test now also fetches option9.html and requires the
WARC control to render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Translate the webhttrack WARC strings into the remaining languages
PR #672 added a WARC toggle to the webhttrack GUI with 4 new LANG keys,
translated only in English and French. Append the same 4 msgid/translation
pairs to the other 28 lang/*.txt so those locales stop falling back to
English. Each translation is encoded in the file's declared LANGUAGE_CHARSET
(the charset the server serves the page and parses the form as) with a strict
lossless round-trip, matching each file's existing CRLF/LF line ending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add WARC/1.1 v1.1: --warc-max-size rotation, WARC-Truncated, FTP resource records
Extends the merged WARC writer with the three v1.1 items. All logic stays in
htswarc.c; the engine hooks stay thin.
--warc-max-size N (-%rs) rotates the archive into NAME-00000.warc.gz, -00001,
... once a segment passes N bytes (wget naming), each segment led by its own
warcinfo and never splitting a record. N<=0 keeps the single-file behavior.
WARC-Truncated tags a body cut short by a cap. The mirror size (-M) and time
(-E) caps abort in-flight transfers and overwrite the slot's status to a
negative TIMEOUT, after which back_finalize and the WARC hook both bail; so the
partial is archived at the abort site, before the clobber, with WARC-Truncated:
length/time. HTTrack's own incomplete/retry bookkeeping is untouched, and a
genuine broken transfer (not a cap) is still not archived. The per-file cap (-m)
rejects on Content-Length rather than truncating, so it has no truncated body.
Standard ISO 28500 tokens only (length/time/disconnect); the task's non-standard
"disk" is left out.
FTP transfers have no HTTP envelope, so an ftp:// capture becomes one resource
record: WARC-Type: resource, the payload's own Content-Type, block = payload,
no request/response. FTP back_finalize runs on the main loop (the worker thread
only flips status), so the single-writer no-lock design holds.
Self-tests warc-trunc / warc-ftp / warc-rotate added to the -#test=warc family
and 01_zlib-warc.test; each was confirmed to fail without the corresponding
code. man/httrack.1 + html regenerated for the %r help line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Range-check --warc-max-size and regenerate the man page
Replace the unchecked sscanf on the --warc-max-size argument with a
strtoll parse that rejects non-numeric, negative, and overflowing values,
leaving the default 0 (single archive) intact. The man page and its HTML
render were already regenerated for the option's help line in the feature
commit, so this only tightens the parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `--warc` / `--warc-file NAME`, writing an ISO-28500 WARC/1.1 archive of a crawl (warcinfo + request + response records, gzip per record, SHA-1 digests under OpenSSL) so HTTrack output replays in Wayback-style tools (ReplayWeb.page, pywb). Under `--update`, unchanged resources become `revisit` records instead of duplicate copies. No new dependency.
Fidelity note: HTTrack decodes gzip/br/zstd inline, so v1 stores the decoded body and normalizes the encoding headers (drops `Content-Encoding`/`Transfer-Encoding`, recomputes `Content-Length`). Valid and replayable; verbatim-compressed capture and WACZ are v2. Logic is isolated in a new `src/warc.c`.
First phase (v1) of #668.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Skip an oversized -%F footer instead of aborting the crawl
The per-page footer is expanded into a fixed ~3KB stack buffer. On overflow
hts_footer_format returns <0 and leaves the buffer unterminated, but the call
site ignored the return and ran strcatbuff on it regardless; strcatbuff's
bounded strlen finds no terminator within capacity and abort()s, killing the
crawl with SIGABRT. Reachable whenever the footer expansion exceeds the buffer
(a field referenced several times with a long URL/path, or a URL that triples
under the HTML-comment escaping); the pre-existing default footer has the same
exposure.
Guard the emit on the formatter's return: on overflow, drop the footer rather
than crash. Regression test drives a file:// crawl of a deep path with a footer
repeating {path} and asserts the crawl completes; it aborts on the pre-fix
binary.
Closes#669
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Skip the footer-overflow test on Windows (MAX_PATH)
The test triggers the overflow with a path longer than Windows MAX_PATH (260),
which the source tree and mirror output both exceed there, so httrack can't
create it and the crawl fails for an unrelated reason. Restrict it to POSIX; the
fix is platform-independent and the formatter's overflow-return contract is
still covered cross-platform by the footerfmt self-test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Allowlist the Windows skip of the footer-overflow test
The Windows CI pins expected skips so an all-skipped suite can't pass green;
01_engine-footer-overflow.test now skips there (MAX_PATH), so add it to the list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add named {addr}{path}{date}{version} footer fields alongside the legacy %s form
The -%F footer was a bare positional printf: each %s consumed the next of a
fixed addr/path/date/version arg list. It is order-coupled and inexpressive
(you cannot reach {date} without emitting addr and path first, cannot reorder,
and a wrong %s count silently yields "???" or shifted values), and the help
text advertised a cryptic bracket syntax.
hts_footer_format now dispatches on content: a footer containing %s keeps the
legacy positional model byte-for-byte (the default footer and every existing
-%F string are unaffected), while a footer without %s uses named fields
{addr} {path} {date} {version}, with "{{"/"}}" for a literal brace and any
unrecognized {...} left verbatim so typos stay visible. Sanitization is
unchanged: addr/path still pass through html_inline_safe() at the call site,
so the comment-injection guard (#165) still holds.
Driven by a new -#test=footerfmt self-test (tests/01_engine-footerfmt.test)
covering both models, brace escaping, the mixed-mode dispatch boundary, and
the overflow/zero-size return paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Regenerate html/httrack.man.html for the -%F help change
The man/html sync CI guard requires html/httrack.man.html to change whenever
man/httrack.1 does; the -%F help-text update regenerated the man page but not
its html rendering.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Add {url}{lastmodified}{mime}{charset}{status}{size} footer fields
Extends the named footer set beyond addr/path/date/version. hts_footer_format
now takes a {name,value} table instead of four fixed parameters, so the field
set is data rather than signature and the legacy %s path looks addr/path/date/
version up by name (order-independent, still byte-for-byte). The call site
supplies the new fields from the current response: {url} (scheme + host + path,
credentials stripped like {addr}), {lastmodified}, {mime}, {charset}, {status}
and {size}.
Every network-derived string is html_inline_safe()'d, since the footer sits
inside an HTML comment and a value holding "-->" would otherwise close it and
inject markup (#165); {status}/{size} are formatted integers and need none.
Documented in --help, the man pages and the Command-Line Guide, and covered by
the -#test=footerfmt self-test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The offline crawl harness reads the server's ephemeral port from the "PORT <n>"
line it prints once bound, but only waited 5s (50 x 0.1s) and matched head -n1.
Under `make check -jN` up to 16 Python servers cold-start at once; on a loaded
Windows runner a cold MSYS Python start can lag past 5s, so discovery timed out
with an empty log ("could not discover server port:") and failed 13_local-cookies
spuriously. Give it a 30s deadline, and match the PORT line anywhere so a stray
startup warning merged via 2>&1 can't wedge discovery until timeout.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#R was listed twice: "cache repair" and "old FTP routines". The FTP handler
is commented out in the parser, so the only live meaning is cache repair;
drop the bogus line. #X is a no-op (the parser prints "option has no effect"),
so remove its help line and misleading default marker.
Add help lines for three real but undocumented options: -%z
(--disable-compression), -%t (keep the original file extension), and -y
(--background-on-suspend).
Regenerate man/httrack.1 and html/httrack.man.html from the updated --help.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The offline suite waited on each httrack crawl with a bare wait bounded only by the engine's --max-time; a fetch that wedges past that (a Windows socket stall the engine misses) blocked wait forever and ran the test step to its 45-minute cap. wait_bounded attaches the #595 kill_tree reaper to the crawl pid so an overrun is reaped in seconds, stop_server now reaps the server's native tree, and 72_watchdog-crawl proves the fail-fast against an always-stall endpoint.
The "Select URLs" and "Start" section headings on the webhttrack server
wizard were literal English, bypassing the ${LANG_...} substitution, so
they stayed English in all 30 locales. Reuse existing, already-translated
keys: step3 -> ${LANG_G44} ("Web Addresses: (URL)"), step4 -> ${LANG_J9}
("Start"). No lang/*.txt changes needed.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct spelling errors (beginning, dishonest, redistributing, forbid,
personal, chosen, address, occurrences) and refresh stale content: collapse
the obsolete per-Windows-version questions into a single current statement,
drop dead OS keywords, describe the cookies.txt / --cookies-file workflow
instead of the Netscape/IE folder steps, and reword the rtsp note to say
streaming protocols are out of scope rather than "not supported yet".
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct several spelling typos in html/abuse.html and html/filters.html,
and point filters.html readers at the --why (-%Y) option, which reports
which filter rule accepted or blocked a given URL.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2007 cmddoc.html beginner walkthrough is fully subsumed by cmdguide.html.
Repoint index.html's "Command-line version" bullet to httrack.man.html (the
generated reference, previously unlinked) and mark the now-orphaned legacy
options.html as possibly-stale, pointing at the man page and the guide. Also
fixes two long-standing index.html typos (relese, Developper).
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix the download-PDFs example in the command-line guide
The old example ran httrack with no filters, so it mirrored the whole
site rather than downloading its PDFs. The corrected command restricts
to the site, admits the HTML pages as scaffolding for link discovery
(including directory-index pages via *[file]/ so sub-directory PDFs are
found), keeps the PDFs, and drops everything else.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Use *[path]/ so the PDF recipe follows nested directory indexes
*[file] cannot cross a slash, so *[file]/ only admits a single directory level and
silently drops PDFs linked under nested paths (dated archives, doc trees). *[path]/
spans slashes and follows directory-index pages at any depth, with no over-admission
(only trailing-slash URLs match, never files).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
html/httrack.man.html is groff-rendered from man/httrack.1 and committed, but
had drifted far behind the roff page (missing the option table-of-contents and
many options such as --pause and --delayed-type-check). Regenerate it, and make
regen-man-html strip groff's version-stamp and creation-date comments so the
committed file no longer churns across groff versions or rebuilds.
Add a lightweight CI guard: a PR that changes man/httrack.1 must also change
html/httrack.man.html, catching the "regenerated the roff page, forgot the HTML"
case that caused this drift. Rendering needs the full groff html device, so CI
verifies the two move together rather than regenerating in-CI.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Filter wildcards" subsection listing the bracket wildcard forms
the matcher (src/htsfilters.c) implements: *, *[file]/*[name], *[path],
*[param], character sets and ranges, literal escapes, size rules, and
the *[] end anchor. It links to the full filters page for the rest.
Added as an h4 under the filters section so section numbering and the
page ToC are untouched.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
src/minizip/ vendors patched minizip as X.orig + X.diff pairs; X.diff is
replayed onto fresh upstream on the next re-sync, so a live edit that skips
X.diff silently reverts then. PR #640 fixed a signed-shift UB in the live
mztools.c (READ_32 now casts each recovered 16-bit half to uLong before the
shift) but never regenerated mztools.c.diff, so a re-sync would drop it.
Regenerate mztools.c.diff from the live file; it reproduces mztools.c
byte-for-byte again. Header date labels keep the existing convention so the
change is content-only. The other four snapshots already match and are untouched.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Non-ASCII single -O drops the hts-cache into a mangled twin directory on Windows
PR #636 fixed the log half of #630 and left the cache for a follow-up. With a
single -O café, path_log holds UTF-8 bytes, but cache_init created hts-cache and
opened new.zip through ANSI calls, so on Windows the cache landed in a second
mangled café-twin directory beside the mirror, and a later --update read the
cache from the wrong place.
This routes every path_log filesystem op in cache_init and the reconcile helpers
through the UTF-8 wrappers (structcheck_utf8, MKDIR, FOPEN, UNLINK, RENAME,
fexist_utf8, fsize_utf8), and opens the cache ZIP through minizip's filefunc
entry points (zipOpen2/unzOpen2) backed by hts_fopen_utf8. On POSIX the wrappers
resolve to the same libc calls, so only Windows changes. The conversion is
all-or-nothing: a directory created UTF-8 but a ZIP opened ANSI would split the
cache, so the whole cache-open path flips together.
The corrupt-cache repair path stays ANSI. mztools' unzRepair takes no filefunc,
so a corrupt cache under a non-ASCII path fails to repair (the site is
re-crawled) rather than forking a twin.
Test 69 now also asserts the cache lands under the café directory through a new
--cache-under-logroot audit. Like the log assertion it only bites on the Windows
CI leg; on POSIX it exercises the code path without reproducing the mojibake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cache UTF-8 filefunc must keep 64-bit offsets and the cache dir 0700
The #630 cache-UTF-8 hook built a 32-bit zlib_filefunc_def via
fill_fopen_filefunc and called unzOpen2/zipOpen2. Minizip runs a 32-bit
filefunc through fill_zlib_filefunc64_32_def_from_filefunc32, which NULLs the
64-bit seek so every seek truncates the offset to uLong: on Windows LLP64 a
cache >=4GB hard-fails and >=2GB corrupts. The ANSI code this replaced went
through fill_fopen64_filefunc and never truncated. Keep 64-bit: fill the
zlib_filefunc64_def and override only zopen64_file with the UTF-8 opener, then
call unzOpen2_64/zipOpen2_64.
Also restore the POSIX cache-dir mode: the MKDIR macro creates with
HTS_ACCESS_FOLDER (0755), loosening the 0700 the explicit mkdir used. There is
no UTF-8 mkdir-with-mode, so keep the platform split - UTF-8 MKDIR on Windows
(which ignores the mode), mkdir(HTS_PROTECT_FOLDER) on POSIX.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten the #630 comments
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Strip only the scheme's own default port, not :80 on every scheme
hts_strip_default_port treated 80 as the default for every scheme, so an
explicit :80 on a non-http URL was dropped and the fetch silently moved to that
scheme's real default: https://h:80/x became https://h/x (fetched on 443), ftp
likewise. The reverse gap: a scheme's own default (443 https, 21 ftp) was never
stripped, so https://h:443/x kept a redundant port and would not dedup against
the portless form.
Derive the default from lien's scheme (80 http, 443 https, 21 ftp; 80 when
absent or unknown) and strip only when the port equals that. Same family as
#627/#614.
Closes#638
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Exclude the auth-bypass CodeQL query and pin case-insensitive scheme matching
cpp/user-controlled-bypass models auth-bypass-by-spoofing, but httrack has no
auth or access-control surface; its nearest downstream decision, the +/- crawl
filter, is a mirror boundary, not a security one. Exclude it like the
world-writable query.
The stripport self-test's scheme detection is case-insensitive (strfield/streql)
but nothing pinned it: a case-sensitive rewrite would strip :80 from HTTPS://
unnoticed. Replace the non-discriminating :8443 case (no scheme defaults to 8443,
so it passes for any impl) with HTTPS://h:80, which must keep its port. Note at
scheme_default_port that schemeless/protocol-relative links default to 80.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten the #638 comments
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Force a whole-file refetch when a rejected 206 resume can loop (#581)
Resuming an interrupted download, the engine sends a Range and, if the
server answers an unusable 206 (Content-Range not matching the partial on
disk), drops the partial and retries the whole file. On Windows a hard-killed
first pass leaves a truncated cache that the next pass must repair; the retry
can then re-derive a Range from a partial or temp-ref that outlived the
restart, meet the same unusable 206, and loop until retries run out. The
partial is removed but never refetched, so the file is lost from the mirror.
Carry a refetch-whole signal from the restart-whole decision onto the
requeued link, so the retry's back_add drops any stale temp-ref and skips the
partial/temp-ref resume branches: it sends no Range and GETs the whole file,
regardless of whether a partial survived the restart. On POSIX the restart
already removes both sources, so the retry was already Range-less and behavior
is unchanged; test 71 still recovers the file whole.
back_add gains an internal (hidden, non-exported) parameter; htsblk and
lien_url gain one trailing field each. No exported symbol changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Use hts_boolean and HTS_TRUE/HTS_FALSE for the refetch-whole flag
The #581 fix carried the whole-file refetch flag under three inconsistent
types (short int, char, int) set with bare 1/0. Normalize all three to the
house hts_boolean type and use the HTS_TRUE/HTS_FALSE macros for the literal
sets. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Advertise -%N's long option and document the real long forms
The help display appends each option's long alias by looking it up in the
alias table, but it first stripped a trailing N to turn placeholders like
cN into c. That also turned -%N into -%, so --delayed-type-check was never
advertised in --help (nor in the generated man page). Try the flag as-is
first and only strip a trailing N on a miss; -%N now shows its long form,
and cN still resolves to --sockets.
The command-line guide had several rows marked short-only that in fact
have long aliases (I had read them off a stale installed binary): fill in
--pause, --strip-query, --disable-compression, the three --keep-* dedup
opts, and --delayed-type-check. Regenerate man/httrack.1 to match, and
drop the long-dead commented-out -%O chroot help line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bound the option-token sscanf to the buffer size
CodeQL flagged the %s read into cmd[32] as an unbounded copy on the line
the previous commit touched. The tokens come from HTTrack's own help
strings, not hostile input, so it was not reachable in practice, but the
copy should be bounded regardless. Limit it to %30s (the buffer holds the
leading '-' plus 30 chars and a NUL).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add a task-oriented command-line guide to the offline docs
The command-line docs so far are the option list and the generated manual
page: exhaustive, but organized by flag, not by task. Newcomers arrive
expecting wget/curl syntax and hit the same walls (only the index came
down, filters that do the opposite of what they read, an update that
deletes files), because the reference answers "what does -X do", not "how
do I do the thing I want".
cmdguide.html is a task-oriented layer on top of the manual page: quick
start, scope, filters, limits, naming, links, identity/login, proxy,
update/cache, scripting, then eleven copy-ready recipes with the one
gotcha each. It foregrounds the defaults that actually surprise people
(the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge
semantics, the depth off-by-one, mime filters running after headers) and
links the manual page for per-option detail. Linked from cmddoc.html and
the index. Every documented default and behavior was checked against the
engine source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Correct four scope/option descriptions in the guide
Adversarial review against the engine source caught four mislabels:
-d is "same principal domain" and -l is "same TLD" (not "same directory"
and "same domain"); -U goes up only and -B goes both ways (the guide had
-U as up-and-down and -B as "anywhere"); -%M archives the whole mirror
into one index.mht, not each page; and -t HEAD-tests links outside scope
rather than reporting "what a scope would reach". The rest of the guide's
documented behavior verified against code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the PDF recipe, prefer long options, drop the section rules
Review feedback from Xavier:
- The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages
that carry the PDF links, so the crawl only keeps PDFs linked from the
entry page (verified on a local fixture). There is no PDF-only crawl:
HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse,
with the off-host case handled by a `+host/*.pdf` rule.
- Recipes now use long options throughout. `-%!` in particular is replaced
by `--disable-security-limits`: the bare `!` triggers shell history
expansion and is easy to fumble, and the long name says what it does.
- Dropped the per-section `<hr>` rules; the sibling doc pages don't use
them and the section headings already separate the content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Key the reference tables by long option name, short in parens
Follows the recipe conversion: every table row now leads with the long
option (--depth, --stay-on-same-domain, ...) and carries the short flag in
parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y,
-%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command
snippets in descriptions to long form (--assume, --structure,
--user-agent "").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--help (and the manual page generated from it) claimed the default socket
count is 8 and the default new-connections-per-second is 10. The engine
actually defaults to 4 sockets (htslib.c) and 5 connections/second, which
is also the -%c ceiling. The 8 in the old text is the -c clamp, not the
default, so the line conflated the two.
Correct the two markers to (*c4) and (*%c5) in htshelp.c and regenerate
man/httrack.1 via man/makeman.sh. html/httrack.man.html carries the same
two-token fix applied directly, since regenerating it needs groff's html
device (Debian: full groff, not groff-base); a later full regen on such a
host reproduces the identical output.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The option parser's `case 'K':` had no `break`, so a bare -K (no trailing
digit) fell through into `case 'c':`, hit its no-digit branch, and forced
the socket count back to the default of 4. `-c8 -K` ended up with 4
connections, not 8; the -K rewrite mode quietly overrode an earlier -c.
Only a trailing bare -K triggered it (`-K -c8` and `-Kc8` were fine),
which is why it went unnoticed.
Add the missing break. The regression test drives the observable that
maxsoc has: a socket count above 8 trips the "limited to 8" security
warning in hts-log.txt, so `-c16` warns and `-c16 -K` warns only if the
16 survives. It fails on the pre-fix build and passes after.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The command-line documentation (cmddoc.html, options.html, and the generated httrack.man.html) was unreachable from the documentation index; they only linked each other. Link cmddoc.html into the "How to Use" list so the whole set is reachable, and replace options.html's stale ~2007 option list with a pointer to the generated man page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Adds html/android.html, a step-by-step guide for the HTTrack Android app (install, project, address, options, run, browse, storage), with screenshots of the shipped app and an options section covering all eleven tabs and the settings that are missing or fixed on Android. Linked from the documentation index between the WinHTTrack/WebHTTrack and Fred Cohen entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Bumps the copyright year in the html/ documentation footers (and the inline notice in contact.html) from 2007 to 1998-2026. Footer text only, no content changes; fcguide.html (Fred Cohen, upstream) is left untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Four factual bugs in the shipped html/ help pages: the FAQ's stale "SOCKS? Not yet!" answer (SOCKS5 and HTTP CONNECT proxies have shipped), cache.html's false ">4GiB ZIP64 not supported" claim, two -mime:video/* example rows in filters.html mislabeled "application/", and a compile-breaking fprintf(stder, ...) in plug.html's sample module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
src/vcpkg.json had no builtin-baseline, so vcpkg resolved OpenSSL/zlib/brotli/zstd in Classic mode from whatever ports tree the build runner happened to sit at: the crypto shipped next to libhttrack.dll was a property of the runner image, not of any commit. Pin the baseline (resolves OpenSSL 3.6.3, zlib 1.3.2#1, what the Windows build already ships) so it becomes deterministic, and add a Dependabot vcpkg entry so the baseline advances on a schedule instead of freezing into a stale, CVE-bearing OpenSSL. The Windows workflow now fetches the pinned baseline before building, read back from the manifest so Dependabot bumps need no workflow edit.
Closes#642
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold repo-specific conventions from private notes into the tracked
guide: a C conventions section (htssafe.h *t allocators, HTSEXT_API as
the exported ABI surface, Windows-breakable/POSIX-stable ABI split),
the concrete Latin-1 file list behind the byte-safe-edits rule, and
three test gotchas (register NN_*.test in TESTS, installed-binary PATH
shadowing, set -e in new .test scripts).
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#581 reports a Windows-only data loss: an interrupted mirror that resumes into an unusable Content-Range 206 drops its partial and never refetches, where the same sequence recovers the file whole on Linux. The trigger is the damaged cache a hard TerminateProcess leaves behind (MSYS can't signal a native exe), which pass 2 has to repair before resuming.
This does not fix the engine. I couldn't reproduce the failure on Linux, so an engine change would be guesswork. What it adds is a deterministic test that drives the damaged-cache regime, plus an analysis of where the two platforms part.
Test 71 leaves a partial and a temp-ref in pass 1, truncates `new.zip` past its last local entry so the central directory a hard kill never wrote is gone and the repair path runs, then resumes into the hostile 206. On Linux that fires the repair, takes the "unusable range -> restart whole" branch, and recovers the file whole every time, across every damage severity I tried, including a repair that recovers zero entries or fails outright. So the cache repair and the restart-whole logic are not themselves where Linux and Windows differ.
Root cause, as far as I can pin it from Linux: restart-whole doesn't refetch. It removes the partial and the temp-ref, flags `STATUSCODE_NON_FATAL`, and leans on the ordinary retry to requeue the URL. The requeued attempt rebuilds the request from the cache, then the temp-ref, then the on-disk partial. On Linux the removals leave none of those, so the retry is a clean whole-file GET and it succeeds. For the file to be lost, the retried attempt has to send a Range again: a second unusable 206, a second -5, and once the retry budget is spent the partial is already gone. That second Range can only come from a temp-ref or partial that outlived the restart-whole removal, which points at a Windows-specific removal or path effect I can't confirm from here.
For the maintainer: the fragile hinge is that restart-whole depends on a budget-consuming retry that re-derives its Range state from disk. A sturdier fix would make the retried attempt refuse to resume, via a per-link "refetch whole, no Range" flag the request builder honors, so a leftover temp-ref or partial can't re-enter the 206 loop whatever the removal quirk turns out to be. Checking that `UNLINK` and `url_savename_refname_remove` actually succeed on Windows would confirm the mechanism first.
Test 71 carries the same Windows skip as test 48. Lifting that skip should reproduce the failure, and turn the test into the fix's verification.
Refs #581.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
webhttrack splits the raw HTTP POST body into argv and calls hts_main2 directly, but that body is in the web form's declared charset (LANGUAGE_CHARSET: ISO-8859-1, windows-125x, BIG5, gb2312, shift-jis, depending on the language), not UTF-8. The httrack CLI and WinHTTrack both hand the engine UTF-8, which htsname's path budget and htscache's format detection now assume, so a non-ASCII output path or URL from the web UI reached the engine as raw form-charset bytes and put the mirror in a mojibake directory instead of the one the user named.
The command line is now converted from the current LANGUAGE_CHARSET to UTF-8 before the crawl starts, in htsserver.c right before webhttrack_main() where that charset is known via LANGSEL. ASCII and already-UTF-8 input pass through untouched. webhttrack's own argv also gets the Windows hts_argv_utf8 treatment the CLI already has.
This exports hts_convertStringToUTF8 so htsserver (which links the shared library) can reach it: an additive ABI change, new symbol, soname unchanged, alongside the already-exported hts_convertStringSystemToUTF8. Flagging it since it touches the public export set.
Test 68 drives the real htsserver over HTTP, posts a start command whose -O dir is café in ISO-8859-1, and checks the mirror lands under the UTF-8 café directory, not the ISO-8859-1 twin. It fails on master and passes with the fix.
Closes#629
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
unzRepair parses the local file header of a damaged cache zip during
repair. READ_32 combined two int-typed READ_16 halves as
READ_16(adr) | (READ_16((adr)+2) << 16); when the high half has bit 15
set (>= 0x8000), shifting it left by 16 exceeds INT_MAX, which is signed
overflow. UBSan aborts on the CRC/size fields of a header whose high
16-bit word has that bit set. Repair runs on hostile input (a corrupt or
foreign new.zip). Cast the halves to uLong before the shift, matching the
sibling minizip readers in unzip.c and zip.c.
Closes#639
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single `-O` sets both `path_html` and `path_log`, so `httrack -O café url` sends the logs through `path_log` too. argv is UTF-8, so `path_log` holds UTF-8 bytes, but the two-file log branch in `hts_main_internal()` still created its directory and opened `hts-log.txt`/`hts-err.txt` through the ANSI `structcheck()`/`fopen()`. On Windows those read the bytes as the codepage and dropped the logs into a second, mangled `caf<mojibake>/` directory beside the mirror. This routes that branch through the same UTF-8 wrappers #628 used for the mirror root (`structcheck_utf8`/`FOPEN`/`UNLINK`/`fexist_utf8`), so the logs land under `café/` with the mirror. On a UTF-8 filesystem the wrappers resolve to the same calls, so only Windows changes.
This is the log half of #630. I left the cache half out on purpose: relocating `hts-cache` is a much larger, coupled change, because the cache is written through minizip's own ANSI `fopen` and cleaned up through an ANSI `opendir`, so half-converting it would split or break the cache rather than move it. That wants its own PR, and #630 should stay open for it.
Test 69 mirrors into a single non-ASCII `-O` and asserts the audits read `hts-log.txt` from that directory. Like test 64 it only bites on the Windows CI leg, where the two encodings differ; on Linux the change is a no-op and the whole suite stays green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HTTrack gives a file a temporary `.delayed` placeholder name while it can't resolve the file type yet, then renames the placeholder once the type is known. The problem is that url_savename appends that marker before it enforces the 236-char path ceiling, and the ceiling cuts the tail of the last path segment, which is exactly where the marker lives. Once the trailing `.delayed` is gone the name no longer matches `IS_DELAYED_EXT`, so back_delayed_rename bails out ("nothing bound to the placeholder name") and the downloaded file is never moved to its final name. It is reachable through #133-style deep paths whose final segment is a long hashed filename.
The fix reserves the trailing `.<id>.delayed` in the last-segment copy loop and trims the head of the segment instead, so the result still fits under the ceiling. The hex-id scan stops at its dot separator, so a wholly-hex hashed base is never mistaken for the collision tag and pulled into the marker.
Test 67 drives the naming path over a deep, over-long delayed URL and checks that the marker survives the cut. It fails on master and passes with the fix.
Closes#623
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test matched engine output with `echo "$out" | grep -q PATTERN`. Under
`set -o pipefail`, grep -q exits the moment it matches, so echo takes a SIGPIPE
writing the rest and the pipeline reports failure even though the pattern was
found. The `|| { echo FAIL; exit 1; }` guard then fired on a passing case,
turning it into an intermittent, output-size-dependent failure (seen on the
Debian buildd leg of #632).
Match with here-strings instead, dropping the pipe and the race entirely.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The link-rewrite step that drops a redundant :80 read the port with a
hand-rolled digit accumulator, then skipped a hardcoded 3 chars (":80")
when the value equaled 80. Any longer spelling that still evaluates to
80 lost only its first 3 chars and glued the rest onto the host:
http://127.0.0.1:080/x became 127.0.0.10/x, :0080 became 127.0.0.180.
The accumulator was also unchecked, so :4294967376 wrapped to 80 and
took the same path (#614 shape).
Extract the strip into hts_strip_default_port() and parse the matched
digits with the range-checked hts_parse_url_port(): a wrapped or
out-of-range value no longer aliases 80, and a genuine default is
dropped by its full matched length instead of 3. Covered by the
"stripport" engine self-test (tests/66_engine-port80-strip.test).
Closes#627
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the version in configure.ac, htsglobal.h and the Windows version
resource, and move VERSION_INFO to 3:5:0. The installed struct layouts and
the exported symbol set are unchanged since 3.49.12, so the soname stays
libhttrack.so.3 and no Debian package rename is needed.
The release is a large one: SOCKS5 and CONNECT proxy support, brotli and
zstd content decoding, a family of --update data-loss fixes, 2 GB+ file
handling on Windows and 32-bit builds, and the Windows toolchain port to
VS2022 with its own CI test suite.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Range-check the listen and ftp:// ports, not just the crawled-URL one
The --port listen port, proxytrack's host:port arguments and an ftp:// URL
port all took their value from sscanf("%d"), which range-checks nothing and
is undefined past INT_MAX (glibc wraps). "--port 4295009395" listened on
42099, and proxytrack, which had no range check at all, listened on port 80
for ":65616".
Reuse hts_parse_url_port rather than range-check after the fact. It moves to
its own file: htsserver cannot see a hidden libhttrack symbol and proxytrack
does not link the library at all, so neither could reach it in htslib.c.
htsweb's --port and proxytrack's arguments are operator input for a listen
socket, so both refuse a bad value at startup through the error path each
already had, rather than falling back to a default: silently listening
somewhere the operator did not name is the bug, and a default would keep the
same wrong behavior. The ftp:// port is a URL port and follows the
crawled-URL policy, refusing the link.
--port 0 (which asked for the auto-pick) and 65535 (refused by an off-by-one
">= 65535") both now mean what they say.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Wire htsurlport into the MSVC projects and make test 64 portable
The new htsurlport.c was added to the autotools targets but not the MSVC
projects, so libhttrack.dll, proxytrack.exe and webhttrack.exe failed to
link on Windows with hts_parse_url_port unresolved. Add it to the three
.vcxproj that compile a caller, mirroring the autotools split.
Test 64 drove the listen servers with timeout(1), which is absent on macOS,
so every assertion tripped over a missing command. Source testlib.sh and use
its run_with_timeout instead, the same helper the other blocking-server
tests use. The htsserver and proxytrack listeners block once a port is
accepted, so both need the bound regardless.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Since argv became UTF-8 through hts_argv_utf8(), opt->path_html is already
UTF-8, but hts_main_internal() still re-encoded it with
hts_convertStringSystemToUTF8(), reinterpreting the bytes as the ANSI codepage.
path_html_utf8 prefixes every saved file, so a non-ASCII -O path sent the whole
mirror into a double-encoded directory. Collapse the Windows branch to the
plain UTF-8 copy the POSIX side already used, and create the mirror root with
structcheck_utf8() so the pre-created directory matches where files are written.
On a UTF-8 filesystem both resolve to the same mkdir, so only Windows changes.
This fixes the CLI and the WinHTTrack GUI, which both hand the engine UTF-8 argv.
A local-crawl test mirrors into a non-ASCII -O directory and asserts the files
land there; it runs on the Windows CI leg where the bug is reachable.
Two adjacent gaps stay open with their own issues: a single-directory -O still
leaves a mangled path_log twin holding logs and cache (#630), and webhttrack
never converts its argv to UTF-8 (#629).
Closes#621
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #619 fixed hts_gethome() for an exported-but-empty $HOME, where getenv()
returns "" rather than NULL and the old NULL-only check let "~/foo" collapse
into the absolute "/foo". The same shape lives in two static clones that the
fix missed; htsserver's is live.
With HOME= exported, htsserver renders /websites as the default base path,
reads /.httrack.ini on the first request of every session, and writes
/.httrack.ini when a mirror starts. Running as root that last one lands a file
at the filesystem root; otherwise it silently fails on EACCES.
The clones cannot just call hts_gethome(): it is hidden by -fvisibility=hidden
(nm reports it as a local "t" in libhttrack.so), so htsserver cannot reach it
even though it links the library, and proxytrack does not link libhttrack at
all. Exporting it would grow the public ABI for no other caller, so each copy
keeps its own body. proxytrack's is dead today and fixed anyway: this bug got a
second round precisely because one copy kept the old shape.
Test 63 drives htsserver and reads the base path back out of the rendered UI,
since a static in htsserver.h is out of reach of the -#test registry. It pins a
real HOME too, or the empty case would also pass against a gethomedir() that
ignored the environment and always answered ".".
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hts_buildtopindex() works in the system charset, but verif_backblue(), which
writes backblue.gif and fade.gif, expects UTF-8, and every other caller already
hands it the UTF-8 path. Only this one passed a system-charset path, so on
Windows the gifs were written into a mangled twin directory instead of the
project. Convert at the call site, guarded by _WIN32 since the converter is
Windows-only and the system charset is UTF-8 everywhere else.
Adds a topindex self-test (driven from the offline suite, so it runs on the
Windows leg where the bug is reachable) that asserts the gifs land in the
project directory. It failed on both Windows legs before the fix.
Sibling #216, the mojibake project name in the generated index, is a separate
leak in the same function and is not addressed here.
Closes#217
Refs #216
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Refuse a URL whose port is outside 1..65535
newhttp_addr() parsed a crawled URL's port with sscanf("%d") and validated it
with "i != -1", then cast it to unsigned short. An out-of-range port therefore
folded into 0..65535 and httrack connected there: http://host:105579/ mirrored
off port 40043, storing the result under host_105579 while the bytes came from
somewhere else. A value past INT_MAX is also UB, and wraps on glibc.
What carries the weight is that a user's port exclusion filter stops describing
the port httrack opens, since the filter sees the literal spelling and the
connect uses the folded one.
Parse it with hts_parse_url_port() instead, factored out of #610's
parse_proxy_port(), and refuse the link through newhttp_addr()'s existing
"msg + INVALID_SOCKET" path. A proxy port can fall back to the scheme default;
a URL port cannot, because the URL named one and named a nonsense one.
socks5_handshake_stream() had the same shape, range-checking only after the
sscanf had already wrapped a huge value into a plausible port; it now shares
the parser and its existing socks5_fail() rejection.
Covered by the dns self-test, which drives newhttp_addr() through the mocked
resolver: *addr_count separates a refused port (0, no resolve) from one merely
truncated or defaulted (2), so a "fall back to 80" fix would not pass.
Closes#614
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep accepting an empty "host:" port, which means the default
WHATWG and curl both treat an empty port as valid and meaning the scheme
default, and httrack's URL parser hands "http://host:/x" to newhttp_addr() as
adr="host:", so refusing it as an unparseable port would drop a legal link that
master crawls today.
Guard the parse on a non-empty port text, and move the case to the accepted
side of the self-test. Removing the guard fails that test, so the case is
covered rather than merely asserted.
The SOCKS5 origin path already refused an empty port before this series and
still does; that predates #614 and is left alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the SOCKS5 origin port in the self-test, and trim comments
Reverting htsproxy.c alone to the wrap-then-check version passed the whole
suite, so nothing pinned that half of the fix and a later refactor could have
dropped it silently. The scripted SOCKS5 harness now asserts a bad origin port
is refused with no byte sent.
65616 would not have tested anything: it fits an int, so the old range check
already caught it. The case the old check could not see is 4294967376, which
overflowed the sscanf("%d") into a plausible 80 and passed; the trailing-junk
and signed spellings sscanf also accepted are covered alongside it. Against
master's htsproxy.c the new assertion fails.
Also note at the SOCKS5 call site that an empty "host:" stays refused there
while the direct path accepts it, so the asymmetry reads as intended rather
than as an oversight, and halve two comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SOCKS5 test server read the sub-negotiation version byte and dropped it. That byte must be 0x01, RFC 1929's own version, not the 0x05 of the SOCKS5 greeting. `htsproxy.c` gets it right and even comments the trap, but since the server never looked, a client that regressed to 0x05 would have kept `52_local-socks5.test` green forever while every real proxy rejected the handshake.
The server now rejects a mismatch the way a real proxy does, and the test names it, so a bad version byte reads as itself rather than as the vaguer "origin not downloaded".
This turned up while checking the proxy client against implementations we didn't write (microsocks, squid, tinyproxy, tor) over the live site. The client is correct on every leg, but this test could not have told us that. Verified by mutation: with `htsproxy.c` patched to send 0x05, master's test passes and the fixed one fails.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restore ~/ expansion in the -O base path
expand_home() has never fired since 660b569 (3.41.2) refactored it from char*
to String: the correct str[0] == '~' became StringSub(*str, 1) == '~', but the
body still skips a single leading character. "~/foo"[1] is '/', so the guard
never matches, while any path whose *second* char is '~' silently lost its
first character and gained $HOME ("a~/x" -> "/home/smith~/x").
Index from 0 and gate on a following '/' so ~user/ is left alone rather than
mangled into $HOME + "user/" -- resolving it needs getpwnam, which this does
not do. An empty $HOME now falls back to '.' like an unset one, so ~/foo can
no longer expand to the absolute /foo, and an oversized $HOME leaves the path
untouched instead of aborting inside strcatbuff.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the expandhome test to a $HOME that MSYS leaves alone
The Win32 leg spawns a native httrack.exe, and MSYS rewrites a POSIX absolute
$HOME into a Windows path on the way in, so the engine read
C:\Program Files\Git\home\smith and the pinned /home/smith never matched. A
$HOME with no leading '/' is passed through verbatim; expand_home only
concatenates, so the sentinel exercises the same code.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the clang-format run over the gate condition
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Ask the engine for $HOME rather than pinning it in the expandhome test
MSYS rewrites $HOME into a cwd-relative Windows path before the native
httrack.exe reads it (HTSHOME became D:\a\httrack\httrack\tests\HTSHOME), so
no pinned value survives the Win32 leg. Take the expansion of a bare '~' as
the reference instead, and assert it moved off '~' so the derived cases cannot
go vacuous. The ~user/ and a~/ negatives are literals and stay exact.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin what the expandhome test could not distinguish
A hts_gethome() that ignored $HOME and always returned the fallback passed
the whole suite: the empty-HOME cases expect ./foo, which a constant "."
satisfies. That is the same silent no-op the fix exists to remove. Also cover
the oversized-$HOME bail, which had no coverage at all.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the oversized-$HOME case inside Windows' environment limit
70000 chars cannot be set as an environment variable on Windows, so the case
failed the MSVC legs, and $(seq 1 70000) flooded the -x log. The buffer is
2 * HTS_URLMAXSIZE, so 4096 clears it and still kills the no-guard mutant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Run the oversized-$HOME case only where HOME survives the shell
MSYS rewrites HOME into a path of its own, so the engine never saw the long
value: it expanded ~/foo against the build directory instead of bailing. Gate
the case on the engine reporting the HOME we set; it still runs, and still
fails without the guard, on every platform that passes HOME through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix three broken msgids in lang/ and assert the pairing in a test
lang/*.txt are consecutive line pairs, an English msgid then its translation,
and nothing validates that shape. Finnish.txt had ended with a blank line since
2012, which was harmless while it was the last line; #588 appended new strings
after it, so the blank became an interior line and knocked every pair past it
out of phase. All 14 strings #588 added to Finnish, plus the CONNECT proxy one
from 938a873, were keyed off a translation instead of an msgid and rendered as
English. Drop the blank.
Italiano and Portugues-Brasil each carry one msgid that matches nothing in
English.txt ("ISO 9660" for "ISO9660", "Relative URL" for "Relative URI"), so
those two entries never resolved either.
The test is the point: it rebuilds the msgid set from English.txt and checks
every file pairs against it, byte-wise, since each file is in its own legacy
charset. It also checks lang.def's English values, which are the join key into
the msgids. It fails on all three defects before this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Renumber the lang integrity test to 62
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Note the test's physical-line assumption
The engine continues a msgid ending in \ onto the next line; the test does
not. Latent today: no lang line ends in one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix webhttrack picking the wrong language from the locale
The launcher resolved LC_MESSAGES/LC_ALL/LANG into HTSLANG, then immediately
overwrote it with a value recomputed from $LANG alone, so LC_ALL and
LC_MESSAGES were ignored: a user with only LC_ALL=fr_FR.UTF-8 set got English.
That same line cut the region off with "cut -f1 -d'_'", leaving the zh_tw and
pt_br entries of lang.indexes unreachable, so Traditional Chinese fell back to
Simplified and Brazilian Portuguese to European Portuguese.
Resolve the locale in POSIX precedence order and look the tag up region-first,
falling back to the bare language. Also fix Slovenian's ISO 639-1 code, which
was "si" (Sinhala) in both lang.indexes and Slovenian.txt: sl_SI users got
English, and the served pages claimed lang="si".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Trim the locale test's comments
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Anchor the locale test's lift so it cannot silently stop testing
The lift ended at the first blank line, so a blank line grown inside the
locale block would truncate it: reinstating the old LANG-only bug below one
left the test passing 14/14 while the launcher returned English. Anchor on
the next section instead, and pin exported-empty as unset, which lang_of
cannot express.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Salvage a killed crawl's logs into the Windows test artifact
A test the suite watchdog kills at 600s left nothing to diagnose it with.
local-crawl.sh sends the crawl's output to its tmpdir and hts-log.txt lives
there too, but the artifact only uploads tests/*.log, so the killed test's
entry was a single truncated line. The hard kill also skips local-crawl.sh's
cleanup trap, so the tmpdir survives on the runner and is then discarded.
Dump those logs into the test's own .log on a timeout, and clear the tmpdir so
a later timeout cannot re-report it. hts-log.txt is where "More than N seconds
passed.. giving up" lands, which is the question #605 asks of a wedge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Never let the log salvage's cleanup fail the test
dump_crawl_logs runs on the timeout path, where the caller is already
handling a failure: rm -rf can fail on Windows while the killed httrack
still holds a file, and under set -e that would abort the branch whose
whole job is to salvage the evidence.
Renumber the test to 60: #611 adds a 59 too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Do not mistake -%i and -@i for the -i continue flag
The pass that counts URLs also detects -i (continue), and for anything
cmdl_opt() calls an option it looked for a bare 'i' with strchr(). That runs
after optalias_check() has expanded the long aliases, so --build-top-index
(-%i) and --protocol N (-@iN) matched too: they wiped the URL list and left
the run on the usage screen, exit 255. The neighbouring 'q' test had the same
flaw, with -%q silently forcing quiet mode.
Look for the flag in plain short-option clusters only, so -i, -iC1, -iC2 and
--continue keep forcing continue mode while prefixed options stay distinct.
Closes#615
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Walk the option cluster the way the parser does
The helper tested the second character only, which disagreed with the parser
in both directions: it dropped continue mode for -%qi and -%Mi, where the
trailing i does reach the main set, and it still mistook the i of -q%i for
-i, leaving that spelling on the usage screen.
Walk the cluster instead, letting %, &, @ and # take the letter after them
into another set wherever they sit. & matters because the -& to -% rewrite
only fires at position 0, so -q&i reaches the parser as-is.
None of this is reachable through a documented option or an alias, which all
place the prefix first; it keeps the helper honest against hand-written
clusters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bound DNS resolution so a slow resolver cannot outlast --max-time
host_wait() is a stub returning 1, so a STATUS_WAIT_DNS slot resolves
synchronously through hts_dns_resolve_all() -> getaddrinfo() on the engine
thread. Nothing bounded that call, and it ran holding opt->state.lock, so a
black-hole resolver stalled the crawl past --max-time and --timeout and blocked
hts_request_stop()/hts_has_stopped() with it. resolv.conf caps getaddrinfo on
POSIX and hides this most of the time; Windows has no equivalent.
Resolve a cache miss on a worker thread and wait at most opt->timeout for it,
with the lock held only for the cache lookup and the store. A timed-out resolve
is abandoned rather than cancelled (getaddrinfo has no portable cancellation),
so the job is refcounted and the worker touches nothing but it. A timeout is
reported as "does not resolve" but never cached, so the host is retried instead
of written off for the rest of the crawl.
This bounds the wedge; it does not make resolution asynchronous. The engine
thread still stalls other slots for up to --timeout per resolve, which only
driving STATUS_WAIT_DNS from the poll loop would fix.
New -#test=dnstimeout self-test drives a mock resolver that blocks for 3s past
a 1s --timeout; all five of its checks fail on master.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Derive the DNS timeout assertion from the spec, and join the mock workers
The self-test's ceiling came from the mock's sleep (MOCK_SLOW_MS - 500), not
from opt->timeout: a resolve using a hardcoded 2400ms deadline that ignored
opt->timeout entirely still passed it. Derive the bound from opt->timeout so
the assertion tests the contract instead of the fixture.
The scaffolding also raced its own abandoned workers. mock_host.calls was read
while a worker was still writing it, and tearing the backend down at the end
raced the worker's freeaddrinfo read, ordered only by a sleep. Serialize the
counters under a mutex, wait for the backend calls to actually return instead
of guessing, and leave the backend installed, since an abandoned worker still
reads it. Drops the drain sleep: 6.0s -> 4.0s.
Also: --timeout now bounds resolution, so say so in the help text (the man page
is generated from it), and factor the three address-copy loops into a helper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reap a stalled TLS handshake with the per-slot --timeout
back_wait only runs its per-slot timeout check when the local gestion_timeout
flag is armed. The CONNECTING, WAIT_DNS and receiving handlers arm it, but the
STATUS_SSL_WAIT_HANDSHAKE handler did not, so a peer that completes the TCP
connect and never speaks TLS left SSL_connect returning WANT_READ until
--max-time fired, ignoring --timeout entirely.
Arm the flag in the handshake handler and start a fresh timeout window when the
slot enters the handshake, so it is measured from there rather than from the
connect. The generic check already reaps any status > 0 slot once armed; give it
a distinct message instead of the generic "Receive Time Out".
Test 59 crawls a server that accepts the connect and stays silent: it ends in
--timeout seconds with the fix, and hangs until the kill guard without it.
Closes#607
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Cover the handshake timeout window, and share the ephemeral-bind helper
Test 59 only bounded the crawl from above, so an engine that reaped every
handshake instantly passed it, and nothing exercised the timeout_refresh at the
handshake entry: on loopback the connect is instant, so the handshake's own
window is indistinguishable from the connect's.
Add a floor to the first case, and a second one behind a proxy that takes 4s to
answer CONNECT. The handshake must still get its full --timeout=5 from there
(~9s); sharing the connect's clock reaps at ~5s. Dropping the refresh now fails
the test, as does collapsing the window to zero.
The stall server grows a proxy mode for that, and takes its listening socket
from a new proxytestlib bind_ephemeral(), which replaces the same boilerplate
in the socks5 and proxy servers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reject an out-of-range -P proxy port instead of wrapping it
hts_parse_proxy read the port with sscanf(a, "%d", &p), which is signed
overflow UB past INT_MAX. glibc wraps rather than fails, so a garbage
port got through: -P 'http://host:99999999999999' parsed to port
276447231, while 'http://host:4294967296' happened to fail and fall back
to the default. 65536 was accepted as-is.
Parse with strtol and range-check to *DIGIT in 1..65535 (RFC 3986); an
out-of-range or malformed port now falls back to proxy_default_port,
exactly as an unparsable one already did.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Attribute the proxy port range to TCP, not RFC 3986
RFC 3986 defines port as *DIGIT with no range; the 1..65535 bound comes from
TCP's 16-bit port field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both files still describe the pre-July policy and tell contributors to polish
each commit message because the branch lands on master as-is. Under squash only
the PR title and description survive, so the advice pointed at the wrong thing.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It points at a path under my home directory and was committed by accident
in #556, so a fresh clone gets a dangling symlink. The file's own preamble
calls it untracked guidance for one checkout; ignore it like coucal and
httrack-windows already do.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CONNECT request is always ATYP=domain, so socks5_handshake_stream rejects a
bracketed origin rather than send the brackets as a domain name the proxy could
never resolve. Nothing pinned that branch: stub the reject out and the suite
still passed.
The message check keeps the assertion tied to the IPv6 case. Asserting only the
failure and an empty sent buffer is a weak signal that many bugs satisfy: a
stricter host validator would reject these too, for the wrong reason, and pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three test servers import proxytestlib from their own directory, which works because Python puts the running script's directory on sys.path. PYTHONSAFEPATH (python3 -P) drops it and the import fails. Nothing in the harness or CI sets it, and the failure is loud rather than a silent skip, so this is belt-and-braces for anyone running a server by hand under -P.
Each server now inserts its own directory before the import. Verified by reproducing the ModuleNotFoundError first, then confirming all three announce ready under -P afterwards. No file in tests/ shadows a stdlib module, so the extra sys.path entry is inert in the normal case.
LFS_FLAG is an AC_SUBST make variable holding the -D large-file flags; it is
never itself a preprocessor macro, so "#ifdef LFS_FLAG" was always false and
INTsys stayed int on POSIX. HTS_LFS is the AC_DEFINE that config.h carries, and
htsglobal.h already includes config.h well above this point.
Mostly a wart, but not a pure no-op: htsindex.c casts fread's return to INTsys
and compares it against an LLint file size, so a temp index of 2 GB or more
truncated to a mismatch and the index was silently not rebuilt; and a tampered
cache length wrapping mod 2^32 used to be accepted where it now fails the
32768 guard. Neither comes from a legitimate writer, but the new behavior is
the safer one.
INTsys is in no struct field and no exported signature: the exported symbol set
and every installed struct layout are unchanged, so no soname bump. INTsysP
tracks the typedef, so the cache text stays identical decimal digits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hts_parse_proxy scanned backward for the port ':' and stopped only at the
authority start or a userinfo '@'. A bracketed IPv6 literal carries colons of
its own, so a portless -P 'http://[2001:db8::1]' split on one of them, giving
name='http://[2001:db8:' and port 1. An explicit port hid it, which is why the
usual form worked.
Predates the socks5 work: the old inline parser in htscoremain.c had the same
scan, and #566 kept its shape when factoring out hts_parse_proxy.
Closes#598
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Windows x64 CI leg intermittently died with "the hosted runner lost communication with the server": a crawl wedged and the per-test timeout(1) watchdog could not kill it. MSYS signals do not reach a native httrack.exe, so timeout reaped only the outer bash and left the engine spinning until the runner starved.
run_with_timeout/kill_tree in testlib.sh replace timeout(1): poll for the deadline, then on Windows end the whole tree by Windows PID with taskkill /T (single-slash, since the workflow passes args verbatim so //T would be rejected); on POSIX run the job in its own process group and signal the group. 58_watchdog.test verifies it end-to-end with positive controls so the reap checks cannot pass vacuously.
proxy-https-server.py (#85) and proxy-connect-server.py (#564) were ~90% identical: the CONNECT proxy, the relay and the origin were copies, differing only in the origin body, the TLS wrap and the argv shape. The common half moves to tests/proxytestlib.py, leaving each server as a docstring plus a serve() call (153 and 152 lines become 32 and 33). socks5-server.py reuses the relay too, but keeps its own origin, which is specialized for keep-alive reuse and subpages.
Behavior-preserving except that the shared origin always logs the request line, which previously only proxy-connect-server.py did. That is harmless for #85, whose origin-log assertion only checks that no Proxy-Authorization arrives. Tests 13, 52 and 57 pass unchanged, the full suite is identical at 101 pass / 7 skip, and the new module ships in the release tarball.
HTTPS already reaches the origin through a CONNECT tunnel (#85), but plain HTTP still used the absolute-URI request form, which a CONNECT-only proxy like Tor's HTTPTunnelPort rejects. Adds a `connect://` proxy scheme (`-P connect://host:port`) that drives the proxy with CONNECT for every request: httrack opens a tunnel to the origin (port 80 for http, 443 for https), then sends the request origin-form with no Proxy-Authorization, the same way https does.
It can't be the default (CONNECT-to-:80 breaks proxies that only allow CONNECT on 443) and auto-detection is ruled out, so the mode rides the proxy scheme like socks5:// does. That keeps it ABI- and cache-safe (the scheme lives in proxy.name, no new struct field) and drops into the webhttrack GUI as one more entry beside HTTP and SOCKS5. The tooltip was reworded and retranslated across the lang/*.txt files.
Test 57 fronts a plain-http origin with a CONNECT-only proxy (it 501s anything but CONNECT) and checks the request arrives origin-form, that dropping connect:// gets the classic form refused, that credentials ride the CONNECT and never reach the origin, that a portless origin tunnels to :80, and that a flooding proxy does not hang.
Closes#564.
The web GUI carried per-control help via onMouseOver="info(...)", which
only wrote window.status. Browsers stopped honoring scripted window.status
writes years ago, so those hints render nowhere.
Emit each hint as a native title='...' attribute alongside the existing
handler, reusing the same ${html:LANG_*} string. Single-quoted so the
filter's ' -> ' (and <>&) escaping keeps the attribute well-formed; a
literal " is harmless inside single quotes. The window.status calls stay
(inert) to keep the diff to added attributes.
The webhttrack smoke test now also fetches an option page and requires a
rendered title=' tooltip.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The connect-fallback probe resolved the origin locally and, on a proxy-connect
failure, fell back to dialing the origin direct — bypassing the proxy, leaking
the origin's DNS query, and sending it the request (with the full absolute URI)
over a direct connection. Under a proxy the socket dials the proxy, not the
origin, so an origin-address list is meaningless here; gate the probe on
!proxy.active. A dead proxy now fails cleanly instead of falling open to the
origin.
Test 55 drives a crawl through a dead proxy at a two-address origin decoy and
asserts the decoy is never contacted; it fails on the pre-fix engine.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cap the chunked-transfer buffer at 2GB before realloc
In the chunked-transfer automaton, back[i].r.totalsize is a 64-bit signed value
that accumulates the attacker-declared size of each chunk, and the in-memory
branch reallocs to (size_t) totalsize + 1 before the chunk body arrives. With no
upper bound the sum can exceed 2GB across chunks; cast to a 32-bit size_t that
truncates and under-allocates the buffer, so the following chunk data overflows
the heap. The sibling in-memory Content-Length path already caps at INT32_MAX;
apply the same bound here and abort the transfer (refetchable) when it trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: mirror a chunked response (regression for the 2GB cap)
Add a Transfer-Encoding: chunked route to the test server and a crawl test that
mirrors it, asserting the chunk automaton joins the bodies (no leftover chunk
framing) and the new 2GB in-RAM cap stays quiet on ordinary traffic. The cap's
overflow threshold isn't reachable at test scale (it needs gigabytes on the
wire), so this pins the non-regression side; the abort path was verified by
hand with a lowered cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Route the too-large chunk through the invalid-chunk teardown
Review found the first cut hand-rolled the abort (statuscode/status/finished)
and then fell through into the shared chunk dispatch, which re-ran finalize and
stomped the message; it was only safe by accident. Set chunk_size = -1 instead
and let the existing chunk_size < 0 path tear the transfer down (deletehttp +
deleteaddr + STATUSCODE_INVALID), matching the sibling error handling. Verified
by hand with a lowered cap: the resource is rejected as "Invalid chunk", the
socket is closed, and no partial file is written.
Also fix the test's decoded-vs-raw discriminator: '^40$' never matched the
CRLF-framed size line, and '</body></html>' is split by HTTrack's mirror
comment. Assert no CR (framing decoded) and '</body>' (no truncation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
libtest/ holds demo callback samples that aren't part of libhttrack, and
src/proxy/ is ProxyTrack, a separate legacy binary with no auth surface whose
recv/cache-parse code trips the taint queries by design. Together they account
for ~26 alerts that are false-positive or by-design. Exclude both so the
dashboard tracks the engine, where the queries earn their keep.
Genuine bugs found in that code are fixed on their own, independent of scanning.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
strjoker_rec charged its work budget only after the depth-cap early-return, so depth-capped recursive calls cost nothing. A class pattern like *[a]*[a]... drives recursion to STRJOKER_MAXDEPTH, sets the sticky cut flag (disabling the failure memo), and the band below the cap then re-explores exponentially without ever tripping the budget — a ~2KB filter ran ~80s under ASan. Moving the increment ahead of the depth-cap return makes every call count, bounding total work; the match fails safe. Behavior is unchanged for real filters. Adds a fuzzer corpus regression seed and a class-branch lower-bound pin to the filterbounds self-test.
The web GUI was frozen before the engine gained SOCKS5 proxying, --cookies-file,
--pause, --strip-query, and the three --keep-* URL-hack opt-outs. Wire each into
the option tabs, the generated httrack command, the winprofile.ini persistence,
and the reload remap so they reach parity with WinHTTrack (xroche/httrack-windows#31).
The SOCKS5 selector reuses the single proxy field and prepends socks5:// to -P;
it is gated on a non-empty host so a SOCKS5-selected-but-blank config can't emit a
stray scheme token. winprofile.ini key names match WinHTTrack so projects round-trip
across both GUIs. Labels/tooltips reuse the LANG_ strings already added in #588.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add 14 new LANG_ keys (proxy type, cookies-file, pause-between-files, keep-www / keep-double-slashes / keep-query-order, strip-query, each with a tooltip) to lang.def and English.txt, and translate them across all 30 language files. Purely additive; each file keeps its own legacy on-disk charset and line endings (Svenska stores Latin-1 despite its ISO-8859-2 header; Chinese-BIG5 is cp950; Uzbek is Latin-script and LF-terminated). Windows libhttrack checks were red on an unrelated vcpkg download.qt.io network flake.
Adds fuzz-htsparse, a libFuzzer harness driving the real htsparse() over a
mocked engine: the tag/attribute/JS-inscript scanner, the link rewriter, and
the accept -> url_savename -> hts_record_link path that runs on every crawled
page, previously reached only by slow full-crawl tests. The parser's coupling
is state, not network, so the harness builds the minimal crawl state
httpmirror() sets up (opt, cache, hash, filters, robots, backing, a seeded
link heap with ptr=1) and walks a NUL-terminated page through the parser, then
discards it. Four seeds cover links, srcset, base/meta/usemap, a
document.write body, and malformed tags. Clean under ASan+UBSan+LeakSanitizer.
Bundles the src/coucal bump to ab59c6a (coucal#13): this harness is the first
fuzzer to hash keys through coucal under UBSan, which tripped a pre-existing
getblock32 pointer-overflow.
Merged past a transient arm64 apt-mirror outage; every code-exercising check
(fuzz, sanitize, msan, distcheck, all non-arm64 builds) passed.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hts_free_opt frees about twenty of the option struct's String fields but omits
opt->accept and opt->headers, both allocated in hts_create_opt. Every opt
create/free cycle leaks the two. Harmless for the CLI but real, and it makes
LeakSanitizer noisy: the fuzz job runs detect_leaks=1, so any harness that
builds an opt trips on it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Keep the mirrored file when an --update re-fetch comes in short (#562)
An --update re-fetch that arrives incomplete (server declares a Content-Length
it does not deliver, or the connection drops mid-body) is correctly refused:
httrack logs "incomplete transfer ... will be retried on the next update" and
does not store the partial body. But the incomplete-transfer branch in
back_finalize() returned without noting the surviving copy, so the URL never
reached new.lst and the end-of-update purge deleted the good file it had just
declined to overwrite.
filenote() the previous copy when it still exists on disk, mirroring the
decode-failure path added in #560. Not compression-specific and needs no abort:
a plain HTML page reproduces it, so it is broader than #521 (abort) and #557
(decode failure).
Closes#562
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Reflow the #562 comment to satisfy clang-format
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: share one fetch-pass counter across the update-refetch routes
The #562 /uptrunc/ routes copied upcodec's per-path body-fetch counter verbatim. Fold both onto one refetch_pass() helper and one dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#580 fixed the Winsock connect fallback (closed#579), so 19_local-connect-fallback
can exercise the real path on Windows. Drop its Windows skip and remove it from
the workflow's expected-skips.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mark the CP_ACP->UTF-8 converter HTSEXT_API, matching hts_argv_utf8 next
to it, so WinHTTrack can call the engine's converter instead of keeping a
private copy. Windows-only (inside the _WIN32 block), so no POSIX ABI
change and no soname bump.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* tests: run the loopback crawl suite on Windows
The ~40 *_local-* tests crawl the bundled Python server: the real TLS
handshake, cache, and file writer, none of which any Windows check covered.
Three things stopped them running there. python3 is python.exe on Windows;
MSYS hands out /d/a/... paths a native python.exe cannot resolve (and arg
rewriting is off, so nothing fixes them up); and Python's text layer turns the
"PORT <n>" discovery line into CRLF, so the \r landed in the parsed port.
Factor the python lookup and the path conversion into tests/testlib.sh, fix the
newline at the source in the three servers, and skip the --file-mode assertion
on Windows, where the engine does not chmod. Behaviour on POSIX is unchanged:
the full suite still passes 97/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: share the server shutdown, and make a skip fail the Windows run
stop_server moves to testlib.sh: the MSYS "a signal cannot reach a native
python.exe, only -9 lands" knowledge was in one of the nine places that kill a
python server. Every step is "|| true" because the callers run under set -e and
reaping a server we just signalled makes wait return 143.
Nothing is expected to skip on Windows, so treat a skip as a failure: the pass
floor alone left slack for exactly the tests that can silently gate themselves
off (TLS, the content codings, socks5, connect-fallback). Add the local proxy
crawl, which the glob was missing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: resolve the server path before backgrounding the server
The port-discovery poll raced the server it had just started: the native path
conversion sat inside the backgrounded command, so the child ran two forks
(command -v cygpath, cygpath) before applying its ">server.log" redirect, and
on Windows the parent's first "head" reached the file first. head then failed,
and under set -e that killed the test silently.
Resolve the paths in the parent, and create the log before the launch so the
first poll cannot lose the race. nativepath keys off the platform, not off
cygpath happening to be on PATH.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: skip the connect-fallback crawl on Windows
19_local-connect-fallback fails there because the engine never falls back to
the next address: Winsock reports a failed connect in select()'s exception set
rather than as writable, and the exception loop fails the slot before
back_connect_next() is reached. Skip it pending the engine fix (#579).
Pin the expected skips instead of counting them, so a gate that silently turns
some other test off still fails the run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: trim the comments added by the Windows port
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: say why the crange resume lost its file
48_local-crange-memresume fails on Windows with a bare "blob.bin missing", and
the crawl log it would have to explain that lives in the tmpdir the test wipes.
Dump the engine's errors and the mirror on that branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: dump the crawl log when the crange resume loses its file
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: check the resume pass survived, don't assume it
48_local-crange-memresume ran pass 2 and printed "terminated" whatever came
back, so a crashed engine read as a clean run. It fails on Windows with an
empty log and an empty mirror, which is what that blind spot looks like.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: dump both passes' state for the Windows crange failure
Temporary: pass-1 mirror + log and pass-2 log/mirror, to see why the resume
mirrors and logs nothing on Windows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: skip the crange memory-resume crawl on Windows
48_local-crange-memresume needs a graceful pass-1 interrupt so the cache is
clean when pass 2 resumes; MSYS can only hard-kill a native exe, and the
engine's restart-whole path after an unusable 206 then fails on the repaired
cache (#581). Skip on Windows pending that fix.
Keep the pass-2 exit-code check the investigation added: the test printed
"terminated" whatever came back, so a crashed engine read as a clean run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Winsock reports a failed non-blocking connect in select()'s exception set,
not as writable. The exception loop in back_wait runs before the connect
handler and finished the slot with STATUSCODE_CONNERROR, so back_connect_next
never got a chance and a multi-homed host with one dead address failed the
whole fetch.
Skip connecting slots in that loop and treat their presence in the exception
set as "the connect finished": SO_ERROR is probed as on POSIX, and the next
resolved address is dialed.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cap the wildcard matcher's recursion depth (#574)
strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.
Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.
The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Take argv as UTF-8 on Windows, not the ANSI codepage (#573)
httrack.exe is an MBCS build, so the CRT transcodes the UTF-16 command
line down to the machine's ANSI codepage before handing us char **argv.
Non-ASCII arguments are lossy, and anything outside that codepage is
destroyed: a Cyrillic or CJK URL on a Western-codepage box becomes '?'.
Everything else in the engine already treats char* as UTF-8 on Windows -
FOPEN, STAT, UNLINK are hts_*_utf8 wrappers converting to UTF-16 at the
syscall boundary. argv is the one thing that never got the memo.
Decode the real UTF-16 command line at the entry point instead. The
manifest's activeCodePage would fix it in one line, but it needs Windows
10 1903+ and is silently ignored below that, which is precisely the
population running the codepages that mangle non-Latin URLs today.
Windows-only, so no POSIX symbol and no soname change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the engine tests pass on Windows
Three of them could never pass there, and none of the three was an engine
bug:
- rcfile wrote .httrackrc, but HTS_HTTRACKRC is "httrackrc" on Windows,
so the engine never found it. Write both names.
- filelist made a file unreadable with chmod 000, which Windows ignores.
Probe whether the mode is enforced instead of assuming, which also
subsumes the root special-case.
- ftp-line was #ifndef _WIN32 for its socketpair(), though get_ftp_line()
itself is portable. Pair over loopback TCP and register it everywhere.
The charset self-tests fed raw non-UTF-8 bytes through argv to probe the
decoders. Those bytes cannot survive any Windows command line, whatever
argv does, so take them as hex - the convention the sniff self-test
already uses - and keep the coverage on every platform.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Run the engine test suite on Windows CI
The ~90 tests only ever ran on Linux and macOS, so a Windows-only
regression was invisible until a user hit it - and the first run of the
offline ones found a third of them failing.
Drive them from Git Bash against the native MSVC httrack.exe. Two things
that silently ruin the run: MSYS rewrites any argument shaped like a POSIX
path, and a URL path is shaped exactly like one, so "-#test=mime /a/b.html"
reached the engine as "C:/Program Files/Git/a/b.html"; and a suite that
degrades to all-skipped would report green having tested nothing, hence
the floor on tests actually passed.
This subsumes the codec and cache self-tests the workflow ran inline. The
.gitattributes pins the test scripts to LF: a converting checkout rewrites
them to CRLF and bash then dies on $'\r' on every line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Log the launch banner argv without re-encoding it
The banner ran hts_convertStringSystemToUTF8() over each argv element on
Windows, which was right when argv came in the ANSI codepage. argv is
UTF-8 on every platform now, so that pass double-encoded a non-ASCII
argument into mojibake in hts-log.txt. Log the bytes as they are.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the Win32 CLI build: include htscharset.h after winsock2.h
htscharset.h pulls in <windows.h>, which drags in the legacy <winsock.h>
unless <winsock2.h> was included first. In httrack.c it sat ahead of the
net headers, so the x86 build redefined every sockaddr/winsock symbol
(C2011). Move it below htslib.h, where winsock2.h is already in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix two more tests the Win32 CI leg exposed
Both are test bugs surfaced by running on 32-bit Windows for the first
time, not engine bugs:
- strsafe drove its overflow through a char[4]. On a 32-bit build that
equals sizeof(char*), so htssafe's array-vs-pointer heuristic
(sizeof(A) != sizeof(char*), the MSVC path) reads the array as a
pointer and skips the bound: the copy then genuinely overflows and
crashes instead of aborting cleanly. Size the buffer off the pointer
width so the checked path runs everywhere.
- idna's malformed-UTF-8 rejection case passed the bad bytes through
argv. They cannot survive a Windows command line: the UTF-16 round-trip
turns them into valid U+FFFD, which the encoder then accepts. Feed them
as hex so the exact bytes reach the bundled punycode encoder, which
rejects them identically on every platform.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cap the wildcard matcher's recursion depth (#574)
strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.
Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.
The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the matcher depth cap from below in the self-test
The self-test asserted the cap is reached, which a cap set absurdly low
would also satisfy: real filters use fewer than ten segments, so a cap of
5 would break them and still ship green. Assert a 32-segment pattern still
matches, so the cap cannot be lowered past real use unnoticed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
proxy/store.c carried its own copy of htscache.c's old-format (.dat/.ndx) cache readers without the bounds hardening the engine copy got in #568. cache_brstr/cache_rstr trusted a length prefix from the cache file and wrote it into fixed buffers unbounded, and cache_rint/cache_rLLint used an unchecked sscanf result as a size or offset, so a crafted cache overflowed firstline[256] and the PT_Element fields and could wrap a malloc(size+1).
Ports the engine's discipline into the proxy copy: thread the destination capacity through the readers and clamp while consuming the declared length, check every sscanf, allocate the body through a bounded helper, fix an index[index_id] off-by-one, and bound the startUrl/previous_save concatenations. Also casts the coucal void* key at the %s snprintf. The .dat/.ndx format is dead (zip only now), so this is defense-in-depth on the legacy reader; a corrupt-cache test guards it under the sanitizer CI.
* Extract the proxy tunneling code into htsproxy.c
htslib.c is the engine's catch-all "subroutines" file; the two proxy
tunnels living in it (HTTP CONNECT and the SOCKS5 handshake) are one
coherent thing: blocking raw-socket negotiation with the proxy, run from
the same back_wait callsite before TLS. Move both, plus proxy_getline and
the socks5 statics, into htsproxy.c/.h, which also gives the socks5_test_io
self-test seam a home away from the public-ish htslib.h.
Pure code motion: the moved bytes are identical, and hts_proxy_is_socks
stays in htslib.c next to hts_parse_proxy and jump_protocol_const, being a
URL-scheme predicate rather than handshake logic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the vcxproj source list alphabetical
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Report file sizes as 64-bit, so Windows stops mishandling files past 2GB
MSVC's off_t is a 32-bit long in x64 builds as well as Win32, and struct
_stat expands to _stat64i32, whose st_size is that same 32-bit long. So
fsize(), fsize_utf8() and fpsize() truncated twice over: once in the stat,
once in their own return type. A mirrored file of 2GB or more got a wrapped,
often negative size, which fed resume/206 handling, the cache body-size
reads, and readfile2()'s allocation.
Use the 64-bit _stat64 family on Windows and return LLint from the three
size helpers, widening the callers' locals to match. INTsys and HTS_FSEEKO
now cover MSVC too, so readfile2() and fpsize() reach the _fseeki64/_ftelli64
aliases htslib.h already declared. htsback.c's two 'not (off_t)' comments
described this trap but the value feeding that path was still 32-bit.
STRUCT_STAT is in an installed header, so hts_stat_utf8()'s signature changes:
a deliberate Windows-only ABI break, agreed with Xavier. The DLL ships next to
the exe with no soname contract. POSIX is untouched, where off_t is already
64-bit under LFS.
The new -#test=fsize self-test builds a 3GB sparse file and asserts both the
reported size and the 8-byte return width. Forcing the helpers back to a
32-bit return reproduces the Windows failure exactly (-1073741824) and the
test catches it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Report errno when the fsize self-test cannot create or reopen its file
The linux i386 leg fails with 'fpsize is -1', which only says the reopen
returned NULL. Print strerror(errno) on both the create and the reopen path,
and check fclose() so a failed flush stops being silent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Size files with fseeko/ftello, never ftell: ftell is 32-bit on i386
fpsize() gated fseeko/ftello behind HTS_FSEEKO, which is defined from
'#ifdef LFS_FLAG'. LFS_FLAG is a configure make variable carrying the
-D_FILE_OFFSET_BITS=64 flags, never a C macro, so that test is dead and the
fallback ftell/fseek branch is the one every autotools build compiled. ftell
returns long: 64-bit on x86-64, which hid the bug, but 32-bit on i386, where
it fails with EOVERFLOW past 2GB and fpsize() returns -1.
Call fseeko/ftello unconditionally, as htsback.c already does; MSVC gets them
via the _fseeki64/_ftelli64 aliases in htslib.h. HTS_FSEEKO had no other user.
The dead LFS_FLAG test also leaves INTsys as int on POSIX, capping readfile2()
at 2GB. Widening it changes a typedef in an installed header, so that is left
for a separate change; the FIXME records it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the INTsys widening's fallout, and make the fsize test bite
Widening INTsys to 64-bit on MSVC broke two call sites that assumed it was
an int, both found by an adversarial review of this branch:
- proxy/store.c's cache_rstr_addr() parsed it with sscanf("%d"), writing 4
bytes into an 8-byte object and leaving the high word indeterminate; that
value then sized a malloc() and an fread(). Its sibling cache_rstr() already
used INTsysP. proxytrack.vcxproj builds this file, so it was live on MSVC.
- readfile2()/readfile_utf8() fed a now-64-bit length to malloct() and fread(),
which take size_t: 32-bit on Win32 and i386. The two truncate independently,
so a file of 4GiB-1 gave malloc(0) with a 4GiB read into it. Route the size
through llint_to_size_t() and fail closed when it does not fit.
The self-test only exercised fsize_utf8(), so reverting fsize()'s _stat64 hunk
left it green, and fsize() is what sizes file:// bodies and --list. It now
checks both variants, the absent-file -1 contract, and uses 5GB rather than
3GB: a 32-bit truncation of 3GB is negative and obvious, of 5GB it is
1073741824, a plausible size. NTFS needs FSCTL_SET_SPARSE to leave the hole
unallocated; POSIX gives it for free.
Windows CI now runs the cache, cache-corrupt and fsize self-tests. The cache
fixes in #567 cannot be verified on POSIX at all, where fconv() is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Check the sscanf return in proxytrack's cache reader
cache_rstr() and cache_rstr_addr() declared an uninitialized INTsys and never
checked whether sscanf matched. On a truncated or malformed cache entry the
length stays indeterminate, and it then sizes an fread() and, in the addr
case, a malloc(). Initialize it and require a match, so a bad entry reads as
length 0.
Flagged by CodeQL (cpp/missing-check-scanf) once the widening touched that
line; the alert predates it on master, and its sibling had the same hole.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Tunnel crawls through a SOCKS5 proxy (RFC 1928/1929)
-P socks5://[user:pass@]host[:port] now negotiates a SOCKS5 tunnel to the
origin instead of speaking HTTP to the proxy. The handshake runs on the raw
socket right after connect, so it covers plain http as well as https (TLS then
runs end-to-end over the tunnel, no CONNECT); the origin request stays
origin-form, and the proxy credentials ride the RFC 1929 sub-negotiation rather
than a Proxy-Authorization header.
The origin hostname is sent as-is (ATYP=domain): the proxy resolves it, like
curl's socks5h, which is what makes .onion and split-horizon names work and
keeps the crawl from leaking DNS. ftp:// links fail under a socks proxy rather
than silently dial the origin direct.
Reply frames are drained exactly per ATYP, since the socket is shared with the
origin stream: one byte too few and the leftovers corrupt the first TLS record.
-#test=socks5 asserts that byte count against scripted replies, plus the
truncated/hostile cases; 52_local-socks5.test crawls http and https through a
local SOCKS5 server that only answers for a .invalid name, so a locally-resolved
origin could not reach it.
Closes#563
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Don't re-run the SOCKS5 handshake on a reused keep-alive socket
back_wait re-enters the connect-completed block for a keep-alive socket
revived by back_trylive (plain http; https is covered by the ssl_con guard).
The socket is already tunneled, so re-sending the SOCKS greeting injected
\x05\x01\x00... into the established origin stream and corrupted every request
after the first. Gate the handshake on !keep_alive, which back_connxfr carries
onto the revived slot and which is zero on a fresh connect.
52_local-socks5.test now crawls a keep-alive origin (advertising max>1) with
-c1: the whole plain-http crawl rides one tunnel, exactly one handshake, every
subpage intact. Without the gate the reused stream desyncs and the crawl hangs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Use scheme-less origins in the SOCKS5 self-test (CodeQL cpp/non-https-url)
The scripted handshake strips the scheme and takes the TLS flag as a parameter,
so the "http://origin.test" fixtures were inert cleartext-URL literals that
tripped CodeQL. Pass the bare authority instead; the parsing is identical and
the real scheme handling is covered end-to-end by 52_local-socks5.test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cache self-tests built the X-Save path with fconcat(), which rewrites
every '/' to '\' on Windows -- including the path_html_utf8 prefix the
reader matches on. cache_readex then took the pre-3.40 compat branch and
re-rooted the name into a path that does not exist, so -#test=cache's
disk-fallback read and -#test=cache-corrupt's headers-only and in-memory
cases all failed on MSVC builds. The engine keeps save names in '/' form and
DOS-ifies only at the syscall boundary; the self-tests now do the same.
The sniff test passed a UTF-8 BOM as raw bytes through argv, where Windows
routes it via the ANSI codepage and U+FEFF has no mapping, so it arrived as
'?'. Use the hex: body form the self-test already accepts.
Both are self-test bugs: neither path is reachable from the crawler, which
never sniffs argv and never DOS-ifies a stored save name.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Parse -P proxy URLs scheme-aware, and recognize socks5 schemes
The -P argument parser scanned backward for the port ':' from the end of the
string with no notion of a scheme, so a scheme-prefixed proxy with no explicit
port ("-P http://proxy") mistook the scheme's own colon for a port separator:
the name came out as "http" with a garbage port. A port or a "user:pass@" prefix
happened to shield the scan, so this only bit the no-port case.
Extract the parse into hts_parse_proxy(), which skips any "scheme://" before
hunting the port, so the scheme colon is never read as a port. Teach
jump_protocol_const the socks5/socks5h schemes (so the proxy host resolves with
the scheme stripped) and default the port by scheme (1080 for socks, 8080
otherwise). This is the parsing groundwork for SOCKS5 support (#563); the
handshake lands separately.
Add a -#test=proxyurl self-test and tests/01_engine-proxyurl.test covering bare
hosts, user:pass, the scheme-colon regression, and the socks defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* hts_parse_proxy: scan from one-past-end, avoid forming arg-1
The backward port scan started at arg+strlen(arg)-1 and decremented while
a >= authority, so on any argument with no ':' or '@' in the authority (every
bare-host proxy, and -P "") it stepped a to arg-1: a pointer formed before the
string, which is undefined behavior. It is not a memory-safety bug (never
dereferenced, no wrap, so UBSan stays green), but it is latent UB reachable from
-P.
Scan from one-past-end and test a[-1] with a > authority instead, so no pointer
below the string is ever formed. Guard name_size == 0 against a future
small-buffer caller (the name_size - 1 truncation would otherwise underflow).
Add a userinfo-free socks explicit-port case and a lone ':' boundary case to the
self-test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* proxyurl test: percent-encoded specials and UTF-8
Pin that the host/port split is byte-transparent: a percent-encoded '@' (%40) or
':' (%3A) in the userinfo stays encoded and never mis-splits host or port, a
second raw '@' is handled by the last-'@' host rule, and UTF-8 bytes in the
username and host label pass through intact. Decoding happens later at auth time,
not in the parser. Raw UTF-8 CLI args match the existing 01_engine-idna.test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Keep the mirrored file when a coded re-fetch fails to decode
An --update re-fetch of a content-coded URL decoded straight into url_sav: the
live mirror was truncated (filecreateempty) before the decode was even attempted,
so a truncated gzip, a corrupt br/zstd stream, a decode-size overrun or a coding
we have no decoder for destroyed the good copy we were merely refreshing. The
uncompressed path has been rollback-safe since #522, but its backup is explicitly
skipped for coded bodies, so nothing covered them.
Decode into a temporary file instead and only commit it over the mirror once the
decode succeeded; on failure url_sav is never touched. The failure path also has
to filenote() the surviving copy, or the end-of-update purge (in old.lst, absent
from new.lst) would delete the very file the decode took care not to overwrite.
Closes#557
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Never drop the decoded body when the mirror replace fails, and probe the good path
replace_file() unlinks the destination and retries when the first rename is
refused (Windows never clobbers). If that retry also fails, the previous copy is
gone, so deleting the decoded temp as well destroyed the last copy of the body:
keep it and log where it went, the way the backup restore does. That restore is
the same remove-then-rename, so it now shares the helper.
The test grew the missing positive control: freshdisk.bin decodes on both passes
on the direct-to-disk path, so the update pass renames a decoded temp over an
existing mirror file, which no case exercised. The leftover-temporaries sweep in
local-crawl.sh now covers the .z/.u decode temps for every crawl test, not just
the three names this one asserted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the 0644 mode on a decoded direct-to-disk file
Committing the decode by renaming the temp bypassed filecreate(), which is what
chmods the mirrored file to HTS_ACCESS_FILE. Under a restrictive umask a coded
binary landed 0600 while every other mirrored file stayed 0644, so a mirror
served from a web root or shared with a group silently lost read access. chmod
it at the rename, as the plain direct-to-disk path already does.
The test now crawls under umask 077 and asserts the mode, via a --file-mode
assertion in local-crawl.sh; without the chmod it fails with 0600.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the codec-update assertions to the URLs and to fresh content
The exact error count did not say which links failed: an engine that logged one
URL twice while another failed silently still counted three and passed. Match the
three decode errors per URL instead. The content checks only looked for the new
marker, so a fix that appended the decoded body to the old file rather than
replacing it would pass too; assert the pass-1 marker is gone. Also assert the
mode on a kept file, not just on the two rewritten ones.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.gitattributes marks *.rc as "text eol=crlf": the blob is LF, the checkout is
CRLF. The two resource scripts added in #558 went in with CRLF blobs, so git
renormalizes them on every add and every checkout reports them as modified.
Renormalized with git add --renormalize; version.rc was already correct. The
working-tree files stay CRLF, which is what the Windows toolchain reads.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Decode the br and zstd content codings, and bound every decode
httrack advertised gzip and deflate only, so it took the oldest coding on
offer while every browser negotiates brotli (46% of CDN traffic) or zstd.
Worse, the response side had no codec identity at all: Content-Encoding was
collapsed into a boolean and hts_zunpack guessed the framing from the body
bytes, so a br or zstd body sent unsolicited (mis-keyed Vary caches do this)
fell through the identity path and was saved as the page, coded bytes and all.
Content-Encoding now maps to a codec (htscodec.c), and the decode dispatches on
it: brotli and zstd get streaming decoders, a known coding we cannot undo fails
the fetch instead of saving garbage, and an unrecognized token still means
identity, because servers do put charsets in that header. br and zstd are
advertised over TLS only, as browsers do.
On any decompression failure the undecoded body is now dropped rather than left
in memory for the writer to commit as the page. That closes the coded-bytes-as-
page hole for the new unsupported-coding path, and with it a latent leak on the
gzip path, where a body that failed to inflate (a truncated stream) was written
to disk verbatim.
Every decode is now bounded: 4096x the coded body, floor 1 MiB, ceiling INT_MAX.
Nothing capped the decoded size before, which was survivable while deflate was
the only codec (it cannot pass 1032x) but not with brotli and zstd, which reach
a million to one. The zlib accumulator also moves to LLint, so a body over 2 GiB
fails instead of overflowing an int.
libbrotlidec and libzstd are optional at configure time and required on Windows,
where libhttrack.dll links them from vcpkg and CI runs the self-tests to prove
the decoders are in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix copyright year on the new codec files (2026, not 1998)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: probe the direct-to-disk decode-failure discard, and assert absence
The wire test only exercised the in-memory discard path (bad.html is HTML).
Add a non-HTML body under an unsupported coding (bin.dat) so the is_write
branch is covered too, and assert both are absent from the mirror rather than
grepping a file that the fix now removes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Factor the budget-enforced write shared by the brotli and zstd decoders
The two streaming decoders duplicated the decoded-size budget check, the
security-critical part of each loop. Pull it into one codec_sink helper so the
bomb ceiling lives in a single place. Behavior-preserving; the codec self-test
(exact decoded bytes, truncation and bomb rejection) proves equivalence.
Also compile the streaming-decode body only when a streaming codec is built,
which drops an unused-variable warning in the --without-brotli --without-zstd
build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The enum said HTS_ROBOTS_SOMETIMES was the default. It is not: hts_init_opt
sets opt->robots = HTS_ROBOTS_ALWAYS (htslib.c:5990), and SOMETIMES is what
the -s option selects when the user asks for it (htscoremain.c:1234).
So HTTrack obeys robots.txt fully by default, and the header said the opposite.
Anyone reading it to answer "does this crawler respect robots.txt" got the
wrong answer from the one place they would look.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
libhttrack.dll and httrack.exe have never carried a VERSIONINFO: no product
name, no version, nothing. Right-click either one on Windows and Properties
has no Details worth reading.
That becomes a hard problem now that we are signing them. SignPath enforce
file metadata restrictions: every binary in a release must name the same
product and the same product version, and a binary that names nothing cannot
satisfy the rule. It fails the signing request, on release day, rather than
the build.
So add src/version.rc, and two wrappers that set the parts which differ per
binary. The version is spelled out there as numbers, because a VERSIONINFO
needs 3,49,12,0 and cannot take a string apart, and because htsglobal.h drags
in C declarations that a resource compiler has no business parsing.
That leaves the version written twice, which is how a version resource quietly
starts lying about which release it is. tests/01_engine-version-macros.test
fails the build if version.rc and htsglobal.h ever disagree, on any of the four
values, and on the product name too. I checked that it fires rather than
assuming: drift the numbers and it fails, restore them and it passes.
The product name is HTTrack Website Copier, on all of them. WinHTTrack is the
name of the Windows GUI, not of the command line program or of the engine.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 20:20:02 +02:00
281 changed files with 18611 additions and 3589 deletions
This file lists all changes and fixes that have been made for HTTrack
3.49-14
+ New: WARC/1.1 archive output (--warc), with a sorted CDXJ index (--warc-cdx) and WACZ packaging (--wacz), also available from webhttrack (#668)
+ New: -%F takes named footer fields such as {url}, {lastmodified}, {mime}, {charset} and {status} instead of a fixed layout (#667)
+ New: a command-line guide organized by task ships with the offline documentation (#649)
+ Fixed: on Windows, paths beyond MAX_PATH truncated files, and mirroring into a long or non-ASCII directory silently failed (#133)
+ Fixed: the top index showed mojibake for non-ASCII project names and categories on Windows (#216)
+ Fixed: webhttrack handed the engine the web form's charset rather than UTF-8, so a non-ASCII path mirrored into a mojibake directory (#629)
+ Fixed: a non-ASCII single -O left the logs and the cache in a mangled twin directory on Windows (#630)
+ Fixed: a path-ceiling truncation dropped the .delayed marker, losing the file (#623)
+ Fixed: an oversized -%F footer aborted the crawl instead of being skipped (#669)
+ Fixed: a rejected 206 resume could loop and lose the file rather than refetch it whole (#581)
+ Fixed: default-port stripping was scheme-blind and dropped explicit ports from https and ftp URLs, and a :80 written with leading zeros mangled the host (#627, #638)
+ Fixed: -K silently reset the -c socket count (#650)
+ Fixed: signed-shift undefined behaviour in the zip-repair local-header read (#639)
+ Changed: the offline documentation drops stale facts, gains an Android help page, and documents the filter wildcards and the real long option forms
+ Changed: multiple internal hardening, test and CI improvements
3.49-13
+ New: SOCKS5 proxy support, with scheme-aware -P URLs (socks5://, socks5h://, connect://) and plain HTTP tunneled through a CONNECT-only proxy (#563, #564)
+ New: decode brotli and zstd content codings, advertised over TLS only as browsers do (#556)
+ New: webhttrack exposes the engine options added since 3.49-2, among them --cookies-file, --pause and --strip-query (#587)
+ Fixed: files of 2 GB or more were mishandled on Windows and on every 32-bit build (#569)
+ Fixed: --update destroyed a good local copy when the re-fetch returned an HTTP error, was aborted by -M/-E, failed to decode, or came in short (#176, #521, #557, #562)
+ Fixed: a self-redirect cookie wall was dropped instead of being re-fetched with the cookie (#15)
+ Fixed: a stalled TLS handshake ignored --timeout, and synchronous DNS resolution could wedge a crawl past --max-time (#607, #613)
+ Fixed: -M metered saved bytes rather than received volume, and overshot its cap under a slow server (#77, #520)
+ Fixed: several network-facing overflows and denial-of-service paths in the Content-Range, chunked-transfer, cookie, filter and ProxyTrack cache parsers
+ Fixed: a failed connect did not fall back to the next address on Windows (#579)
+ Fixed: -P took an out-of-range port as a garbage port, and scanned past an IPv6 literal's closing bracket (#598, #602)
+ Fixed: reject a port outside 1..65535 wherever one is parsed (a crawled URL, the htsserver and proxytrack listen arguments, an ftp:// URL), instead of letting a bare sscanf wrap a huge value into a plausible port and silently use it (#614, #626)
+ Fixed: a configured proxy still resolved and dialed the origin itself (#592)
+ Fixed: ~/ in the -O base path was never expanded (#270)
+ Fixed: a non-ASCII -O output path was double-encoded on Windows once argv became UTF-8 (#621)
+ Fixed: files under a non-ASCII project path were saved to a mangled directory on Windows (#217)
+ Fixed: --build-top-index (-%i) and --protocol (-@i) were taken for the -i continue flag, wiping the URL list and exiting on the usage screen (#615)
+ Fixed: webhttrack ignored LC_ALL/LC_MESSAGES and picked the wrong Chinese and Portuguese (#95)
+ Fixed: webhttrack wrote its base path and httrack.ini to the filesystem root when $HOME was empty (#625)
+ Fixed: crawls on a non-default port were slowed by a per-request pre-resolve (#181)
+ Changed: Windows builds moved to Visual Studio 2022 and OpenSSL 3.x, the VS2008 project files are retired, and the binaries carry a version resource
+ Changed: removed the obsolete Java-applet (.class) parser and the dead SWF module remnants
+ Changed: multiple internal hardening, test and CI improvements (Windows and macOS crawl suites, HTML-parser fuzzing, parallel make check)
3.49-12
+ New: --why explains which filter rule accepts or rejects a given URL, then exits (#505)
+ Fixed: links carrying raw UTF-8 bytes were fetched double-encoded and 404'd (#516)
shows. They might do that because they are connected through expensive modem connection,
or because they would like to consult pages while travelling, or archive sites that may be
removed
one day, make some data mining, comiling information ("if only I could find this
one day, make some data mining, compiling information ("if only I could find this
website I saw one day.."). <br>
There are many good reasons to mirror websites, and this helps many good people.<br>
As a webmaster, you might be interested to use such tools, too: test broken links, move a
@@ -229,7 +229,7 @@ test the webserver response and performances, index it..<br>
Anyway, bandwidth abuse can be a problem. If your site is regularly "clobbered"
by evil downloaders, you have <br>
various solutions. You have radical solutions, and intermediate solutions. I strongly
recomment not to use<br>
recommend not to use<br>
radical solutions, because of the previous remarks (good people often mirror websites).<br>
<br>
In general, for all solutions,<br>
@@ -244,7 +244,7 @@ or, to be extreme: if you unplug the wire, there will be no bandwidth abuse<br>
Good: Will work with good people. Many good people just don't KNOW that they can slow down
a network.<br>
Bad: Will **only** work with good people<br>
How to do: Obvious - place a note, a warning, an article, a draw, a poeme or whatever you
How to do: Obvious - place a note, a warning, an article, a draw, a poem or whatever you
want<br>
<br>
</li><li>Use "robots.txt" file<br>
@@ -266,7 +266,7 @@ Good: Efficient<br>
Bad: Multiple users behind proxies will be slow down, not really easy to setup<br>
How to do: Depends on webserver. Might be done with low-level IP rules (QoS)<br>
<br>
</li><li>Priorize small files, against large files<br>
</li><li>Prioritize small files, against large files<br>
Good: Efficient if large files are the cause of abuse<br>
Bad: Not always efficient<br>
How to do: Depends on the webserver<br>
@@ -283,7 +283,7 @@ How to do: Use routine QoS (fair queuing), or webserver options<br>
<br>
</li><li>Use technical tricks (like javascript) to hide URLs<br>
Good: Efficient<br>
Bad: The most efficient tricks will also cause your website to he heavy, and not
Bad: The most efficient tricks will also cause your website to be heavy, and not
user-friendly (and therefore less attractive, even for surfing users). Remember: clients
or visitors might want to consult offline your website. Advanced users will also be still
able to note the URLs and catch them. Will not work on non-javascript browsers. It will
@@ -335,7 +335,7 @@ Example: Use things like
</li><li>Use technical tricks to temporarily ban IPs<br>
Good: Efficient<br>
Bad: Radical (your site will only be available online for all users), not easy to setup<br>
How to to: Create fake links with "killing" targets<br>
How to do: Create fake links with "killing" targets<br>
Example: Use things like <a href="killme.cgi"><nothing></a>
(again an example in php:)<br>
<tt>
@@ -417,7 +417,7 @@ Example:<br>
</li><li>Another one is to create images of emails<br>
Good: Efficient, does not require javascript<br>
Bad: There is still the problem of the link (mailto:), images are bigger than text, and it can cause problems for blind people (a good solution is use an ALT attribute with the email written like "smith at mycompany dot com")<br>
How to do: Not so obvious of you do not want to create images by yourself<br>
How to do: Not so obvious if you do not want to create images by yourself<br>
Example: (php, Unix)<br>
<tt>
@@ -491,7 +491,7 @@ echo <br>
</li><li>You can also create temporary email aliases, each week, for all users<br>
Good: Efficient, and you can give your real email in your reply-to address<br>
Bad: Temporary emails<br>
How to do: Not so hard todo<br>
How to do: Not so hard todo<br>
Example: (script & php, Unix)<br>
<tt>
@@ -579,7 +579,7 @@ And then, put the email address in your pages through:
<metaname="description"content="How to mirror a website with HTTrack on Android: install the app, create a project, enter the address, run the mirror, and browse the result on your device."/>
<metaname="keywords"content="httrack, HTTrack, android, offline browser, web mirror utility, website mirroring, mobile, Google Play"/>
@@ -173,7 +173,7 @@ There are also specific issues regarding this format:
<ul>
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory must be present <b>(not currently supported)</b></li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
</ul>
<br/>
@@ -282,7 +282,7 @@ Libraries should generally handle this peculiar format, however.
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<p>With no other options HTTrack mirrors that site, stays on the same host, follows
links to any depth, rebuilds them to browse offline, and stores everything under
<tt>mydir</tt>. The same directory also holds the log files and the
<tt>hts-cache/</tt> folder that makes a later update or resume possible.</p>
<p>Two defaults are worth knowing up front, because both catch people out:</p>
<ul>
<li>HTTrack throttles itself to about <b>100 KB/s</b> even when you pass no rate
option. If a mirror feels slow, that is why. See
<ahref="#limits">Limits</a> for how to lift it.</li>
<li>The download proceeds as a well-behaved robot: it identifies itself as
<tt>HTTrack</tt>, obeys <tt>robots.txt</tt>, and sends a Referer with each
request. A site that blocks that behavior needs the levers in
<ahref="#identity">Identity</a>, not brute force.</li>
</ul>
<h3id="scope">2. Scope: how far the crawl reaches</h3>
<p>Scope decides which links HTTrack is even willing to follow, before any filter
you write. Get this right and most "it downloaded too much" or "it only grabbed
the index" problems disappear.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--depth (-r)</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
<tr><td><tt>--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
<tr><td><tt>--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
<tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
<tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
<tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
</table>
<p>The single most common surprise is "only the home page came down." That is
usually not a scope option at all: it is an off-host redirect. A start URL of
<tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
lands you on a different host, and same-host scope stops the crawl there. Start
from the final URL, or add a filter that re-admits the real host (see
<ahref="#filters">Filters</a>). The log will show the redirect.</p>
<p><tt>-n</tt> is the fix for pages that render locally without their images or
stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note
that its embedded-asset handling (following <tt>img</tt>, <tt>link</tt>,
<tt>script</tt>, <tt>style</tt> and HTML5 <tt>source</tt>/<tt>track</tt> targets
past the normal depth and filter limits) applies only when <tt>-n</tt> is on; it
is not automatic. It can also over-fetch by dragging in a whole external host from
a single link, in which case name the assets you want with a filter instead.</p>
<h3id="filters">3. Filters and scan rules</h3>
<p>Filters are the number-one source of confusion, and also the tool that solves
most scope problems once you understand them. A filter is a rule that accepts
(<tt>+</tt>) or rejects (<tt>-</tt>) URLs by pattern. The sign is mandatory:
<tt>+pattern</tt> adds, <tt>-pattern</tt> removes, and a bare pattern is an error.</p>
<p>The rules that matter:</p>
<ul>
<li><b>Last match wins.</b> Rules are applied in order and the last one that
matches a URL decides its fate. Order your rules from general to specific.</li>
<li><b>Wildcards.</b><tt>*</tt> matches any run of characters;
<tt>*[a-z]</tt>, <tt>*[0-9]</tt> and similar classes match sets. So
<tt>+*.pdf</tt> means "any URL ending in .pdf".</li>
<li><b>Whitelisting.</b> To keep one site and nothing else, deny everything then
re-admit the host: <tt>"-*" "+example.com/*"</tt>. A lone <tt>+</tt> rule only
adds to the default scope; it never restricts.</li>
<li><b>Size rules.</b><tt>*[>100000]</tt> and <tt>*[<1000]</tt> filter by
byte size. Because size is only known once the transfer starts, an oversize file
is fetched partway and then aborted, not skipped for free.</li>
<li><b>mime: rules.</b> A rule like <tt>-mime:video/*</tt> matches the
<tt>Content-Type</tt>. That type is only known after the response headers arrive,
so a mime rule <b>cannot stop a request</b>; it can only abort the body. Use a
URL pattern when you want to avoid the fetch entirely.</li>
</ul>
<p>Quote your filters. Shells treat <tt>*</tt>, <tt>[</tt> and sometimes <tt>+</tt>
specially, so wrap each rule in quotes as shown above. The full pattern language,
with tables for wildcards, size and mime, is in
<ahref="filters.html">the filters page</a>, and the
<ahref="faq.html">FAQ</a> has a worked tutorial.</p>
<p><b>robots.txt.</b> By default HTTrack obeys <tt>robots.txt</tt> (<tt>-s2</tt>).
<tt>-s0</tt> ignores it entirely, <tt>-s1</tt> obeys it but lets one of your
<tt>+</tt> filters override a disallow for a URL you explicitly asked for. Note
that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
options will not help there. That is an
<ahref="#identity">identity</a> problem.</p>
<h4>Filter wildcards</h4>
<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
bracket forms match narrower sets. The full table, with size and mime rules, is on
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt>— any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt>— a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
<tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
<tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
<tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
<tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
<tr><td><tt>*[<NN]</tt>, <tt>*[>NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[<5]</tt> skips GIFs under 5 KB</td></tr>
<tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
</table>
<h3id="limits">4. Limits and politeness</h3>
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
accident, and the <ahref="abuse.html">abuse page</a> is worth a read. The limits
below let you go faster when you own the target, and slower when you do not.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--max-rate (-A)</tt></td><td>Maximum transfer rate in bytes/sec. <b>The default is about 100 KB/s even without this flag.</b> Raise it to go faster.</td></tr>
<tr><td><tt>--sockets (-c)</tt></td><td>Number of parallel connections (default 4). <tt>--tiny</tt>, <tt>--wide</tt> and <tt>--ultrawide</tt> are presets.</td></tr>
<tr><td><tt>--connection-per-second (-%c)</tt></td><td>New connections opened per second (default 5).</td></tr>
<tr><td><tt>--max-size (-M)</tt></td><td>Stop after N bytes <b>received from the network</b> across the whole mirror (this counts what was transferred, not what was saved).</td></tr>
<tr><td><tt>--max-time (-E)</tt></td><td>Stop after N seconds of wall-clock time.</td></tr>
<tr><td><tt>--timeout (-T), --retries (-R), --min-rate (-J), --host-control (-H)</tt></td><td>Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.</td></tr>
<tr><td><tt>--max-pause (-G), --pause (-%G)</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
</table>
<p><b>The security clamps.</b> To keep an accidental typo from turning into a flood,
HTTrack silently caps a few values: at most 8 connections (<tt>-c</tt>), at most
10 MB/s (<tt>-A</tt>), and at most 5 new connections per second (<tt>-%c</tt>).
Ask for more and you get the ceiling, quietly. The single flag
<tt>--disable-security-limits</tt> lifts all three (the short form <tt>-%!</tt>
also works, but the bare <tt>!</tt> is awkward to type safely in a shell). Use it
only against infrastructure you are allowed to load that hard.</p>
<h3id="names">5. File names and types</h3>
<p>Where local files land, and what they are called, is controlled by the naming
options. This is the second-biggest source of "why did it do that" questions,
usually about a URL like <tt>/article?id=42</tt> or a <tt>.php</tt> page that is
really HTML.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
<tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
<tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
<tr><td><tt>--delayed-type-check (-%N), --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
<tr><td><tt>--include-query-string (-%q), --strip-query (-%g)</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
</table>
<p>The <tt>-N</tt> presets are built from modular arithmetic on the name fields, so
undocumented number combinations often "work" by accident. If you care about the
exact layout, use an explicit template (the <tt>%h %p %n %q %t</tt> placeholders)
rather than a magic number, and check the result on a small crawl first.</p>
<p>A dynamic page served as <tt>.php</tt> or <tt>.asp</tt> that is actually HTML is
the classic case: without help it can be saved with an extension a browser will
not open locally. <tt>--assume php=text/html</tt> fixes both the extension and the
naming.</p>
<h3id="links">6. Links and page building</h3>
<p>After a page is downloaded, HTTrack parses it for more links and rewrites the
ones it kept so the local copy browses offline. These options tune both halves.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--keep-links (-K)</tt></td><td>How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare <tt>-K</tt> keeps <b>absolute</b> URLs, and <tt>-K0</tt> is the <b>relative</b> default. <tt>-K3</tt> keeps absolute URIs, <tt>-K4</tt> keeps the original links.</td></tr>
<tr><td><tt>--replace-external (-x), --generate-errors (-o)</tt></td><td>Replace external links with an error page, and generate an error page for links that failed.</td></tr>
<tr><td><tt>--preserve (-%p), --disable-passwords (-%x)</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
<tr><td><tt>--extended-parsing (-%P), --parse-java (-j)</tt></td><td>Aggressive link discovery, and how much script content is parsed for links.</td></tr>
<tr><td><tt>--mime-html (-%M)</tt></td><td>Save the whole mirror as a single MIME-encapsulated <tt>.mht</tt> archive (<tt>index.mht</tt>).</td></tr>
<tr><td><tt>--index (-I), --build-top-index (-%i), --search-index (-%I)</tt></td><td>Build a per-mirror index, a top index across projects, and a searchable keyword index.</td></tr>
</table>
<p>HTTrack finds links by parsing HTML and CSS. It does not run JavaScript, so any
URL a page builds at runtime in script (a lazy-loaded image, a
JavaScript-assembled path) is invisible to the crawler and will be missing from
the mirror. There is no flag that fixes this; the asset has to appear in the
static HTML or CSS to be found. <tt>-%P</tt> widens discovery for links that are
present but awkwardly formatted, not for links that do not exist until script
runs.</p>
<h3id="identity">7. Identity, cookies and login</h3>
<p>By default HTTrack is an honest robot: it sends a <tt>User-Agent</tt> of
<tt>HTTrack</tt>, a Referer with each link (which reveals the crawl path to the
server), and obeys robots. Plenty of sites filter exactly that profile. These
options control what HTTrack says about itself.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--user-agent (-F)</tt></td><td>The <tt>User-Agent</tt>. Set a browser string to get past crawler blocks; <tt>--user-agent ""</tt> sends none.</td></tr>
<tt>{lastmodified}</tt> (the page's Last-Modified), <tt>{version}</tt>,
<tt>{mime}</tt>, <tt>{charset}</tt>, <tt>{status}</tt> and <tt>{size}</tt>; write
<tt>{{</tt> or <tt>}}</tt> for a literal brace. A footer that contains <tt>%s</tt>
keeps the older positional form (host, path, date in that order). Example:
<tt>-%F "<!-- Mirrored from {url} on {date} -->"</tt>.</p>
<p><b>Login.</b> For HTTP Basic auth, put the credentials in the URL:
<tt>http://user:pass@host/</tt>. An <tt>@</tt> inside the username must be written
<tt>%40</tt>. Only Basic is supported, not Digest.</p>
<p>For cookie or form logins, the simplest path is to log in with a browser, export
its <tt>cookies.txt</tt>, and drop that file in the project directory so HTTrack
sends the session cookie. For a form that needs a POST, <tt>--catchurl</tt> can
capture the exact request your browser sends and replay it. A few cookie caveats
to know: expiry is ignored, there is a silent cap of about 8 cookies sent per
request, and <tt>-b0</tt> disables cookies and the reuse of Basic credentials
across links at the same time.</p>
<h3id="proxy">8. Proxy and network</h3>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
<tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
<tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
<tr><td><tt>--http-10 (-%h), --keep-alive (-%k), --disable-compression (-%z)</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
<tr><td><tt>--bind (-%b), --tolerant (-%B)</tt></td><td>Bind to a local address, and accept technically-bogus responses some servers send.</td></tr>
host names at the proxy (remote DNS) for both <tt>socks5://</tt> and
<tt>socks5h://</tt>, so your local resolver is never consulted. And HTTrack does
not verify TLS certificates: HTTPS gives you an encrypted transport, but not an
authenticated one. That is a deliberate choice for a mirroring tool, not a bug,
but it is worth knowing if you are relying on it for trust.</p>
<h3id="update">9. Update and cache</h3>
<p>Every project keeps a cache under <tt>hts-cache/</tt>. It records every URL that
was fetched, together with the options you used, and it is what makes resuming and
updating possible. It is not a size-limited scratch area you can delete: throw it
away and you lose the ability to continue or update the mirror.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--continue</tt></td><td>Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.</td></tr>
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
<tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
<tr><td><tt>--cache (-C)</tt></td><td>Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.</td></tr>
<tr><td><tt>--debug-cache (-#C), --repair-cache (-#R), --clean</tt></td><td>Inspect the cache, repair its ZIP, and erase cache plus logs.</td></tr>
</table>
<p><b>The purge trap.</b> An <tt>--update</tt> run rebuilds the list of files the
mirror should contain, then deletes any previously-mirrored file that is not on the
new list. This is what keeps a mirror in sync with a shrinking site, but it means a
partial or interrupted update can delete files you meant to keep. If an update might
not complete cleanly, add <tt>-X0</tt> to protect the existing tree, and expect
dynamic pages to look "changed" on every run. See the
<ahref="cache.html">cache page</a> for the details.</p>
<h3id="experts">10. Experts and scripting</h3>
<p>HTTrack is also a scriptable fetch-and-scan tool. These options turn off the
mirror behavior and expose the engine.</p>
<tableclass="tblRegular tableWidth"border="0">
<trclass="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--get URL</tt></td><td>Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.</td></tr>
<tr><td><tt>--spider --testlinks --skeleton</tt></td><td>Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.</td></tr>
<tr><td><tt>--userdef-cmd (-V)</tt></td><td>Run a shell command on each downloaded file; <tt>$0</tt> is the file path. Good for on-the-fly processing.</td></tr>
<tr><td><tt>--callback (-%W)</tt></td><td>Load an external callback module to hook the engine.</td></tr>
<tr><td><tt>--do-not-log (-Q), --quiet (-q), --verbose (-v), --file-log (-f), --extra-log (-z), --debug-log (-Z)</tt></td><td>Logging: quiet, no questions, verbose on screen, and the various log-to-file levels.</td></tr>
</table>
<p>The <ahref="dev.html">developer page</a> covers the callback API and batch use
in more depth.</p>
<h3id="recipes">11. Recipes</h3>
<p>Copy-ready command lines for the tasks people ask about most. Each has the one
<metaname="description"content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows vista, windows seven, windows 8, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
<metaname="keywords"content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, linux, windows, macos, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software"/>
You have noticed the <tt>-</tt> in the begining of the third rule: this means "refuse links matching the rule"
; and the rule is "any files begining with <tt>www.example.com/gallery/trees/hugetrees/</tt><br>
You have noticed the <tt>-</tt> in the beginning of the third rule: this means "refuse links matching the rule"
; and the rule is "any files beginning with <tt>www.example.com/gallery/trees/hugetrees/</tt><br>
Voila! With these three rules, you have precisely defined what you wanted to capture.<br>
<br>
@@ -361,12 +359,12 @@ You can freely download it, without paying any fees, copy it to your friends, an
There are NO official/authorized resellers, because HTTrack is <b>NOT</b> a commercial product.
But you can be charged for duplication fees, or any other services (example: software CDroms or shareware collections, or fees for maintenance),
but you should have been informed that the software was free software/GPL, and you <b><u>MUST</u></b> have received a copy of the GNU General Public License.
Otherwise this is dishonnest and unfair (ie. selling httrack on ebay without telling that it was a free software is a scam).
Otherwise this is dishonest and unfair (ie. selling httrack on ebay without telling that it was a free software is a scam).
</em>
<br><br><aNAME="QG0b">Q: <strong>Are there any risks of viruses with this software?</strong></a><br>
A: <em>For the software itself:
All official releases (at httrack.com) are checked against all known viruses, and the packaging process is also checked. Archives are stored on Un*x servers, not really concerned by viruses. It has been reported, however, that some rogue freeware sites are embedding free softwares and freewares inside badware installers. Always download httrack from the main site (www.httrack.com), and never from an untrusted source!<br>
All official releases (at httrack.com) are checked against all known viruses, and the packaging process is also checked. Archives are stored on Un*x servers, not really concerned by viruses. It has been reported, however, that some rogue freeware sites are embedding free software and freeware inside badware installers. Always download httrack from the main site (www.httrack.com), and never from an untrusted source!<br>
For files you are downloading on the WWW using HTTrack: You may encounter websites which were corrupted by viruses, and downloading data on these websites might be dangerous if you execute downloaded executables, or if embedded pages contain infected material (as dangerous as if using a regular Browser). Always ensure that websites you are crawling are safe.
(Note: remember that using an antivirus software is a good idea once you are connected to the Internet)</em>
@@ -376,11 +374,8 @@ A: <em>That's right. You can, however, install WinHTTrack on your own machine, a
<br><br><aNAME="QG2">Q: <strong>Where can I find French/other languages documentation?</strong></a><br>
A: <em>Windows interface is available on several languages, but not yet the documentation!</em>
<br><br><aNAME="QG3b">Q: <strong>Is HTTrack working on Windows Vista/Windows Seven/Windows 8 ?</strong></a><br>
A: <em>Yes, it does</em>
<br><br><aNAME="QG3">Q: <strong>Is HTTrack working on Windows 95/98 ?</strong></a><br>
A: <em>No, not anymore. You may try to pick an older release (such as 3.33)</em>
<br><br><aNAME="QG3">Q: <strong>Which systems does HTTrack run on?</strong></a><br>
A: <em>HTTrack runs on current Windows, Linux and other Unix-like systems, and macOS. Very old platforms such as Windows 95/98 are no longer supported; you may try an older release (such as 3.33) on those.</em>
<br><br><aNAME="QG4">Q: <strong>What's the difference between HTTrack, WinHTTrack and WebHTTrack?</strong></a><br>
A: <em>WinHTTrack is the Windows GUI release of HTTrack (with a native graphic shell) and WebHTTrack is the Linux/Posix release of HTTrack (with an html graphic shell)</em>
@@ -394,7 +389,7 @@ A: <em>It should. The <tt>configure.ac</tt> may be modified in some cases, howev
<br><br><aNAME="QG7">Q: <strong>I use HTTrack for professional purpose. What about restrictions/license fee?</strong></a><br>
A: <em>HTTrack is covered by the GNU General Public License (GPL). There is no restrictions using HTTrack for professional purpose,
except if you develop a software which uses HTTrack components (parts of the source, or any other component).
See the <tt>license.txt</tt> file for more information</em>. See also the next question regarding copyright issues when reditributing downloaded material.
See the <tt>license.txt</tt> file for more information</em>. See also the next question regarding copyright issues when redistributing downloaded material.
<br><br><aNAME="QG7b">Q: <strong>Is there any license royalties for distributing a mirror made with HTTrack?</strong></a><br>
A: <em>On the HTTrack side, no. However, sharing, publishing or reusing copyrighted material downloaded from a site requires the authorization of the copyright holders, and possibly paying royalty fees. Always ask the authorization before creating a mirror of a site, even if the site appears to be royalty-free and/or without copyright notice.</em>
@@ -415,7 +410,7 @@ There are several reasons (and solutions) for a mirror to fail. Reading the log
<ul>
<li>Links within the site refers to external links, or links located in another (or upper) directories, not captured by default - the use of filters is generally THE solution, as this is one of the powerful option in HTTrack. <u>See the above questions/answers</u>.</li>
<li>Website <ahref="#Q1b1">'robots.txt' rules</a> forbide access to several website parts - you can disable them, but only with great care!</li>
<li>Website <ahref="#Q1b1">'robots.txt' rules</a> forbid access to several website parts - you can disable them, but only with great care!</li>
<li>HTTrack is filtered (by its default User-agent IDentity) - you can change the Browser User-Agent identity to an anonymous one (MSIE, Netscape..) - here again, use this option with care, as this measure might have been put to avoid some bandwidth abuse (see also the <ahref="abuse.html">abuse faq</a>!)</li>
</ul>
@@ -457,14 +452,14 @@ A: <em>Yes, HTTrack does support (since 3.20 release) ipv6 sites, using A/AAAA e
A: <em>Check the build options (you may have selected user-defined structure with wrong parameters!)</em>
<br><br><aNAME="QT5">Q: <strong>When capturing real audio/video links (.ram), I only get a shortcut!</a></strong></a></br>
A: <em>Yes, but .ra/.rm associated file should be captured together - except if rtsp:// protocol is used (not supported by HTTrack yet), or if proper filters are needed</em>
A: <em>The .ra/.rm associated file can be captured together with the shortcut, if proper filters are set. Streaming protocols such as rtsp:// are out of scope: HTTrack is an HTTP/FTP mirror and does not capture rtsp streams.</em>
<br><br><aNAME="QT6">Q: <strong>Using user:password@address is not working!</a></strong></a></br>
A: <em>Again, first check the <tt>hts-log.txt</tt> and <tt>hts-err.txt</tt> error log files - this can give you precious information<br>
The site may have a different authentication scheme - form based authentication, for example.
In this case, use the URL capture features of HTTrack, it might work.
<br>Note: If your username and/or password contains a '<tt>@</tt>' character, you may have to replace all '<tt>@</tt>'
occurences by '<tt>%40</tt>' so that it can work, such as in <tt>user%40domain.com:foobar@www.foo.com/auth/.
occurrences by '<tt>%40</tt>' so that it can work, such as in <tt>user%40domain.com:foobar@www.foo.com/auth/.
You may have to do the same for all "special" characters like spaces (%20), quotes (%22)..</tt>
</em>
<br><br>
@@ -520,7 +515,7 @@ These rules, stored in a file called robots.txt, are given by the website, to sp
- for example, /cgi-bin or large images files.
They are followed by default by HTTrack, as it is advised. Therefore, you may miss some files that would have been downloaded without
these rules - check in your logs if it is the case:<br>
<tt>Info: Note: due to www.foobar.com remote robots.txt rules, links begining with these path will be forbidden: /cgi-bin/,/images/ (see in the options to disable this)
<tt>Info: Note: due to www.foobar.com remote robots.txt rules, links beginning with these path will be forbidden: /cgi-bin/,/images/ (see in the options to disable this)
</tt>
<br>
If you want to disable them, just change the corresponding option in the option list! (but only disable this option with great care,
@@ -540,7 +535,7 @@ HTTrack must find one. Therefore, two index.html will be produced, one with the
<br>
It might be a good idea to consider that http://www.foobar.com/ and http://www.foobar.com/index.html are the same links, to avoid
duplicate files, isn't it?
NO, because the top index (/) can refer to ANY filename, and if index.html is generally the default name, index.htm can be choosen,
NO, because the top index (/) can refer to ANY filename, and if index.html is generally the default name, index.htm can be chosen,
or index.php3, mydog.jpg, or anything you may imagine. (some webmasters are really crazy)
<br>
<br>
@@ -595,8 +590,7 @@ A: <em>Simply use the <tt>--assume dat=application/x-zip</tt> option
A: <em>You may need cookies! Cookies are specific data (for example, your username or password) that are sent to your browser once
you have logged in certain sites so that you only have to log-in once. For example, after having entered your username in a website, you can
view pages and articles, and the next time you will go to this site, you will not have to re-enter your username/password.<br>
To "merge" your personnal cookies to an HTTrack project, just copy the cookies.txt file from your Netscape folder (or the cookies located into the Temporary Internet Files folder for IE)
into your project folder (or even the HTTrack folder)
To supply your own cookies to an HTTrack project, put a Netscape-format <tt>cookies.txt</tt> file in your project folder, or point HTTrack at one with the <tt>--cookies-file</tt> option. You can export your browser's session cookies to that format with a browser extension.
</a><aNAME="Q3b">Q: <strong>I want to update a site, but it's taking too much time! What's happening?</strong><br>
A: <em>First, HTTrack always tries to minimize the download flow by interrogating the server about the
file changes. But, because HTTrack has to rescan all files from the begining to rebuild the local site structure,
file changes. But, because HTTrack has to rescan all files from the beginning to rebuild the local site structure,
it can take some time.
Besides, some servers are not very smart and always consider that they get newer files, forcing HTTrack to reload them,
even if no changes have been made!
@@ -740,7 +734,7 @@ retransferred.</em><br>
</a><aNAME="Q7">Q: <strong>I just want to retrieve all ZIP files or other files in a web
site/in a page. How do I do it?</strong><br>
A: <em>You can use different methods. You can use the 'get files near a link' option if
files are in a foreign domain. You can use, too, a filter adress: adding <tt>+*.zip</tt>
files are in a foreign domain. You can use, too, a filter address: adding <tt>+*.zip</tt>
in the URL list (or in the filter list) will accept all ZIP files, even if these files are
outside the address. <br>
Example : <tt>httrack www.example.com/someaddress.html +*.zip</tt> will allow
@@ -835,8 +829,8 @@ A: <em>Yes. Use user:password@your_proxy_name as your proxy name (example: <tt>s
<br><br><aNAME="QM8">Q: <strong>Can HTTrack generates HP-UX or ISO9660 compatible files?</strong></a><br>
A: <em>Yes. See the build options (-N, or see the WinHTTrack options)</em>
<br><br><aNAME="QM9">Q: <strong>If there any SOCKS support?</strong></a><br>
A: <em>Not yet!</em>
<br><br><aNAME="QM9">Q: <strong>Is there any SOCKS support?</strong></a><br>
A: <em>Yes. HTTrack supports SOCKS5 and HTTP CONNECT proxies: give the proxy with a scheme prefix, e.g. <tt>-P socks5://host:port</tt> or <tt>-P connect://host:port</tt> (prefix <tt>user:pass@</tt> before the host for authenticated proxies).</em>
<br><br><aNAME="QM10">Q: <strong>What's this hts-cache directory? Can I remove it?</strong></a><br>
A: <em>NO if you want to update the site, because this directory is used by HTTrack for this purpose.
@@ -111,7 +111,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
starts links, the default mode is to mirror these links - i.e. if one of your start page is
www.example.com/test/index.html, all links starting with www.example.com/test/ will be
accepted. But links directly in www.example.com/.. will not be accepted, however, because
they are in a higher strcuture. This prevent HTTrack from mirroring the whole site. (All
they are in a higher structure. This prevent HTTrack from mirroring the whole site. (All
files in structure levels equal or lower than the primary links will be retrieved.)<br>
</i>
<br>
@@ -127,13 +127,15 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
an authorization filter, like <b><tt>+*.gif</tt></b>. The pattern is a plus (this one: <b><tt>+</tt></b>),
followed by a pattern composed of letters and wildcards (this one: <b><tt>*</tt></b>).
<br><br>
To forbide a family of links, define
To forbid a family of links, define
an authorization filter, like <b><tt>-*.gif</tt></b>. The pattern is a dash (this one: <b><tt>-</tt></b>),
followed by a the same kind of pattern as for the authorization filter.
<br><br>
Example: +*.gif will accept all files finished by .gif<br>
Example: -*.gif will refuse all files finished by .gif<br>
<br>
To see which rule accepted or blocked a given URL, run HTTrack with the <b><tt>--why</tt></b> (<b><tt>-%Y</tt></b>) option, described in <ahref="httrack.man.html#OPTIONS">the manual page</a>.<br>
<br>
<p>
<h4>Scan rules based on size (e.g. accept or refuse files bigger/smaller than a certain size)</h4>
@@ -143,7 +145,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
size to ensure that you won't reach a defined limit.
Example: You may want to accept all files on the domain www.example.com, using '+www.example.com/*',
including gif files inside this domain and outside (eternal images), but not take to large images,
including gif files inside this domain and outside (external images), but not take to large images,
or too small ones (thumbnails)<br>
Excluding gif images smaller than 5KB and images larger than 100KB is therefore a good option;
+www.example.com +*.gif -*.gif*[<5]-*.gif*[>100]
@@ -189,7 +191,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Filters are analyzed by HTTrack from the first filter to the last one. The complete URL
name is compared to filters defined by the user or added automatically by HTTrack. <br><br>
A scan rule has an higher priority is it is declared later - hierarchy is important: <br>
A scan rule has a higher priority if it is declared later - hierarchy is important: <br>
<br>
<tableBORDER="1"CELLPADDING="2">
@@ -307,7 +309,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Filters are analyzed by HTTrack from the first filter to the last one. The sizes
are compared against scan rules defined by the user.<br><br>
A scan rule has an higher priority is it is declared later - hierarchy is important.<br>
A scan rule has a higher priority if it is declared later - hierarchy is important.<br>
Note: scan rules based on size can be mixed with regular URL patterns<br>
@@ -361,7 +363,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Filters are analyzed by HTTrack from the first filter to the last one. The complete MIME
type is compared against scan rules defined by the user.<br><br>
A scan rule has an higher priority is it is declared later - hierarchy is important<br>
A scan rule has a higher priority if it is declared later - hierarchy is important<br>
Note: scan rules based on MIME types can <b>NOT</b> be mixed with regular URL patterns or size patterns within the same rule, but you can use both of them in distinct ones<br>
@@ -387,12 +389,12 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<tr>
<tdnowrap><tt>-mime:video/*</tt></td>
<td>This will refuse all video links that were already scheduled for download
(i.e. all other 'application/' link download will be aborted)</td>
<p><strong>Note:</strong> this is a legacy option list and may be out of date.
For the authoritative, always-current reference see <ahref="httrack.man.html">the httrack manual page</a>,
and for a task-oriented walkthrough see the <ahref="cmdguide.html">Command-line Guide</a>.</p>
<ul>
<li>Filters: <ahref="filters.html">how to use them</a></li>
<br><small>Here you can find informations on filters: how to accept all gif files in a mirror, for example</small>
<br><br>
<li>List of options</li>
</ul>
</ul>
<tt>
<pre>
<p>The complete, current list of command-line options is generated from the
program itself, in <ahref="httrack.man.html">the httrack manual page</a>.</p>
w mirror with automatic wizard
This is the default scanning option, the engine automatically scans links according to the default options, and filters defined. It does not prompt a message when a "foreign" link is reached.
<p>WinHTTrack and WebHTTrack offer the same options through their graphical option
panels, described in the <ahref="step9.html">step-by-step guide</a>.</p>
W semi-automatic mirror with help-wizard (asks questions)
This option lets the engine ask the user if a link must be mirrored or not, when a new web has been found.
g just get files (saved in the current directory)
This option forces the engine not to scan the files indicated - i.e. the engine only gets the files indicated.
i continue an interrupted mirror using the cache
This option indicates to the engine that a mirror must be updated or continued.
rN recurse get with limited link depth of N
This option sets the maximum recurse level. Default is infinite (the engine "knows" that it should not go out of current domain)
a stay on the same address
This is the default primary scanning option, the engine does not go out of domains without permissions (filters, for example)
d stay on the same principal domain
This option lets the engine go on all sites that exist on the same principal domain.
Example: a link located at www.example.com that goes to members.example.com will be followed.
l stay on the same location (.com, etc.)
This option lets the engine go on all sites that exist on the same location.
Example: a link located at www.example.com that goes to www.anyotherweb.com will be followed.
Warning: this is a potentially dangerous option, limit the recurse depth with r option.
e go everywhere on the web
This option lets the engine go on any sites.
Example: a link located at www.example.com that goes to www.anyotherweb.org will be followed.
Warning: this is a potentially dangerous option, limit the recurse depth with r option.
n get non-html files 'near' an html file (ex: an image located outside)
This option lets the engine catch all files that have references on a page, but that exist outside the web site.
Example: List of ZIP files links on a page.
t test all URLs (even forbidden ones)
This option lets the engine test all links that are not caught.
Example: to test broken links in a site
x replace external html links by error pages
This option tells the engine to rewrite all links not taken into warning pages.
Example: to browse offline a site, and to warn people that they must be online if they click to external links.
sN follow robots.txt and meta robots tags
This option sets the way the engine treats "robots.txt" files. This file is often set by webmasters to avoir cgi-bin directories, or other irrevelant pages.
Values:
s0 Do not take robots.txt rules
s1 Follow rules, if compatible with internal filters
s2 Always follow site's rules
bN accept cookies in cookies.txt
This option activates or unactivates the cookie
b0 do not accept cookies
b1 accept cookies
S stay on the same directory
This option asks the engine to stay on the same folder level.
Example: A link in /index.html that points to /sub/other.html will not be followed
D can only go down into subdirs
This is the default option, the engine can go everywhere on the same directoy, or in lower structures
U can only go to upper directories
This option asks the engine to stay on the same folder level or in upper structures
B can both go up&down into the directory structure
This option lets the engine to go in any directory level
Y mirror ALL links located in the first level pages (mirror links)
This option is activated for the links typed in the command line
Example: if you have a list of web sites in www.asitelist.com/index.html, then all these sites will be mirrored
NN name conversion type (0 *original structure 1,2,3 html/data in one directory)
N0 Site-structure (default)
N1 Html in web/, images/other files in web/images/
N2 Html in web/html, images/other in web/images
N3 Html in web/, images/other in web/
N4 Html in web/, images/other in web/xxx, where xxx is the file extension (all gif will be placed onto web/gif, for example)
N5 Images/other in web/xxx and Html in web/html
N99 All files in web/, with random names (gadget !)
N100 Site-structure, without www.domain.xxx/
N101 Identical to N1 except that "web" is replaced by the site's name
N102 Identical to N2 except that "web" is replaced by the site's name
N103 Identical to N3 except that "web" is replaced by the site's name
N104 Identical to N4 except that "web" is replaced by the site's name
N105 Identical to N5 except that "web" is replaced by the site's name
N199 Identical to N99 except that "web" is replaced by the site's name
N1001 Identical to N1 except that there is no "web" directory
N1002 Identical to N2 except that there is no "web" directory
N1003 Identical to N3 except that there is no "web" directory (option set for g option)
N1004 Identical to N4 except that there is no "web" directory
N1005 Identical to N5 except that there is no "web" directory
N1099 Identical to N99 except that there is no "web" directory
LN long names
L0 Filenames and directory names are limited to 8 characters + 3 for extension
L1 No restrictions (default)
K keep original links (e.g. http://www.adr/link) (K0 *relative link)
This option has only been kept for compatibility reasons
pN priority mode:
p0 just scan, don't save anything (for checking links)
p1 save only html files
p2 save only non html files
p3 save all files
p7 get html files before, then treat other files
cN number of multiple connections (*c8)
Set the numer of multiple simultaneous connections
O path for mirror/logfiles+cache (-O path_mirror[,path_cache_and_logfiles])
This option define the path for mirror and log files
Example: -P "/user/webs","/user/logs"
P proxy use (-P proxy:port or -P user:pass@proxy:port)
This option define the proxy used in this mirror
Example: -P proxy.myhost.com:8080
F user-agent field (-F \"user-agent name\
This option define the user-agent field
Example: -F "Mozilla/4.5 (compatible; HTTrack 1.2x; Windows 98)"
mN maximum file length for a non-html file
This option define the maximum size for non-html files
Example: -m100000
mN,N' for non html (N) and html (N')
This option define the maximum size for non-html files and html-files
Example: -m100000,250000
MN maximum overall size that can be uploaded/scanned
This option define the maximum amount of bytes that can be downloaded
Example: -M1000000
EN maximum mirror time in seconds (60=1 minute, 3600=1 hour)
This option define the maximum time that the mirror can last
Example: -E3600
AN maximum transfer rate in bytes/seconds (1000=1kb/s max)
This option define the maximum transfer rate
Example: -A2000
GN pause transfer if N bytes reached, and wait until lock file is deleted
This option asks the engine to pause every time N bytes have been transferred, and restarts when the lock file "hts-pause.lock" is being deleted
Example: -G20000000
u check document type if unknown (cgi,asp..)
This option define the way the engine checks the file type
u0 do not check
u1 check but /
u2 check always
RN number of retries, in case of timeout or non-fatal errors (*R0)
This option sets the maximum number of tries that can be processed for a file
o *generate output html file in case of error (404..) (o0 don't generate)
This option define whether the engine has to generate html output file or not if an error occurred
TN timeout, number of seconds after a non-responding link is shutdown
This option define the timeout
Example: -T120
JN traffic jam control, minimum transfert rate (bytes/seconds) tolerated for a link
This option define the minimum transfer rate
Example: -J200
HN host is abandoned if: 0=never, 1=timeout, 2=slow, 3=timeout or slow
This option define whether the engine has to abandon a host if a timeout/"too slow" error occurred
&P extended parsing, attempt to parse all links (even in unknown tags or Javascript)
This option activates the extended parsing, that attempt to find links in unknown Html code/javascript
j *parse scripts (j0 don't parse)
This option defines whether the engine has to parse scripts or not to catch included files
I *make an index (I0 don't make)
This option define whether the engine has to generate an index.html on the top directory
X *delete old files after update (X0 keep delete)
This option define whether the engine has to delete locally, after an update, files that have been deleted in the remote mirror, or that have been excluded
C *create/use a cache for updates and retries (C0 no cache)
This option define whether the engine has to generate a cache for retries and updates or not
k store all files in cache (not useful if files on disk)
This option define whether the engine has to store all files in cache or not
V execute system command after each files ($0 is the filename: -V \"rm \\$0\
This option lets the engine execute a command for each file saved on disk
q quiet mode (no questions)
Do not ask questions (for example, for confirm an option)
Q log quiet mode (no log)
Do not generate log files
v verbose screen mode
Log files are printed in the screen
f *log file mode
Log files are generated into two log files
z extra infos log
Add more informations on log files
Z debug log
Add debug informations on log files
--mirror <URLs> *make a mirror of site(s)
--get <URLs> get the files indicated, do not seek other URLs
--mirrorlinks <URLs> test links in pages (identical to -Y)
--testlinks <URLs> test links in pages
--spider <URLs> spider site(s), to test links (reports Errors & Warnings)
--update <URLs> update a mirror, without confirmation
--skeleton<URLs> make a mirror, but gets only html files
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(notifíquenos fallos o problemas)\r\n\r\nDesarrollo:\r\nInterface (Windows): Xavier Roche\r\nMotor: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for Spanish translations to:\r\nJuan Pablo Barrio Lera (Universidad de León)
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(notifíquenos fallos o problemas)\r\n\r\nDesarrollo:\r\nInterface (Windows): Xavier Roche\r\nMotor: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for Spanish translations to:\r\nJuan Pablo Barrio Lera (Universidad de León)
About WinHTTrack Website Copier
Acerca de...
Please visit our Web page
@@ -928,3 +928,39 @@ Server terminated
Servidor desconectado
A fatal error has occurred during this mirror
Ha ocurrido un error fatal durante esta copia
Proxy type:
Tipo de proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
Protocolo del proxy. HTTP: proxy estándar. HTTP (túnel CONNECT): envía cada petición a través de un túnel CONNECT, para proxys que solo admiten CONNECT como el HTTPTunnelPort de Tor. SOCKS5: puerto predeterminado 1080.
Load cookies from file:
Cargar cookies desde un archivo:
Preload cookies from a Netscape cookies.txt file before crawling.
Precargar cookies desde un archivo Netscape cookies.txt antes de comenzar la descarga.
Pause between files:
Pausa entre archivos:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
Retardo aleatorio entre descargas de archivos, en segundos. Use MIN:MAX para un rango aleatorio (p. ej. 2:8).
Keep the www. prefix (do not merge www.host with host)
Mantener el prefijo www. (no combinar www.host con host)
Do not treat www.host and host as the same site.
No tratar www.host y host como el mismo sitio.
Keep double slashes in URLs
Mantener las barras dobles en las URL
Do not collapse duplicate slashes in URLs.
No colapsar las barras duplicadas en las URL.
Keep the original query-string order
Mantener el orden original de la query string
Do not reorder query-string parameters when deduplicating URLs.
No reordenar los parámetros de la query string al deduplicar las URL.
Strip query keys:
Eliminar claves de query string:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Claves de query string, separadas por comas, que se eliminarán del nombre de los archivos guardados (p. ej. sid,utm_source).
Write a WARC archive of the crawl
Escribir un archivo WARC del rastreo
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Guardar también cada respuesta descargada en un archivo WARC/1.1 ISO-28500, junto a la réplica.
WARC archive name:
Nombre del archivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Prosíme o podání hlášení o jakýchkoliv chybách)\r\n\r\nVývoj:\r\nRozhraní (Windows): Xavier Roche\r\nPavouk: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche a ostatní, kdo pøispìli\r\nÈeský pøeklad:\r\nAntonín Matìjèík (matejcik@volny.cz)
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Prosíme o podání hlášení o jakýchkoliv chybách)\r\n\r\nVývoj:\r\nRozhraní (Windows): Xavier Roche\r\nPavouk: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche a ostatní, kdo pøispìli\r\nÈeský pøeklad:\r\nAntonín Matìjèík (matejcik@volny.cz)
About WinHTTrack Website Copier
O programu WinHTTrack Website Copier
Please visit our Web page
@@ -928,3 +928,39 @@ Server terminated
A fatal error has occurred during this mirror
Proxy type:
Typ proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
Protokol proxy. HTTP: standardní proxy. HTTP (tunel CONNECT): odešle každý požadavek pøes tunel CONNECT, pro proxy podporující jen CONNECT jako HTTPTunnelPort v Toru. SOCKS5: výchozí port 1080.
Load cookies from file:
Naèíst cookies ze souboru:
Preload cookies from a Netscape cookies.txt file before crawling.
Naèíst cookies ze souboru Netscape cookies.txt pøed zahájením stahování.
Pause between files:
Prodleva mezi soubory:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
Náhodná prodleva mezi stahováním souborù, v sekundách. Pro náhodný rozsah použijte MIN:MAX (napø. 2:8).
Keep the www. prefix (do not merge www.host with host)
Zachovat pøedponu www. (nesluèovat www.host s host)
Do not treat www.host and host as the same site.
Nepovažovat www.host a host za stejný web.
Keep double slashes in URLs
Zachovat dvojitá lomítka v URL
Do not collapse duplicate slashes in URLs.
Nesluèovat opakovaná lomítka v URL.
Keep the original query-string order
Zachovat pùvodní poøadí øetìzce dotazu
Do not reorder query-string parameters when deduplicating URLs.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(有任何程序錯誤或問題請聯絡我們)\r\n\r\n開發:\r\n界面設計 (Windows): Xavier Roche\r\n分析引擎: Xavier Roche\r\nJava分析引擎: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\n感謝:\r\nDavid Hing Cheong Hung (DAVEHUNG@mtr.com.hk) 提供翻譯
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(有任何程序錯誤或問題請聯絡我們)\r\n\r\n開發:\r\n界面設計 (Windows): Xavier Roche\r\n分析引擎: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\n感謝:\r\nDavid Hing Cheong Hung (DAVEHUNG@mtr.com.hk) 提供翻譯
About WinHTTrack Website Copier
關於WinHTTrack Website Copier
Please visit our Web page
@@ -928,3 +928,39 @@ Server terminated
伺服器已終止
A fatal error has occurred during this mirror
這鏡像發生了不可回復的錯誤
Proxy type:
proxy 類型:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
代理協定。HTTP:標準代理。HTTP(CONNECT 隧道):透過 CONNECT 隧道傳送每個請求,用於僅支援 CONNECT 的代理,例如 Tor 的 HTTPTunnelPort。SOCKS5:預設連接埠 1080。
Load cookies from file:
從檔案載入 cookies:
Preload cookies from a Netscape cookies.txt file before crawling.
在抓取前從 Netscape 格式的 cookies.txt 檔案預先載入 cookies。
Pause between files:
檔案之間暫停:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
檔案下載之間的隨機延遲(秒)。使用 MIN:MAX 指定隨機範圍 (例如 2:8)。
Keep the www. prefix (do not merge www.host with host)
保留 www. 前綴 (不將 www.host 與 host 合併)
Do not treat www.host and host as the same site.
不將 www.host 與 host 視為同一站點。
Keep double slashes in URLs
保留 URL 中的雙斜線
Do not collapse duplicate slashes in URLs.
不合併 URL 中重複的斜線。
Keep the original query-string order
保留查詢字串的原始順序
Do not reorder query-string parameters when deduplicating URLs.
在對 URL 去重時不重新排列查詢字串參數。
Strip query keys:
移除查詢鍵:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
以逗號分隔的查詢鍵,將其從儲存檔案的命名中移除 (例如 sid,utm_source)。
Write a WARC archive of the crawl
寫入此次抓取的 WARC 封存檔
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
同時將每個已擷取的回應儲存為 ISO-28500 WARC/1.1 封存檔,置於鏡像網站旁。
WARC archive name:
WARC 封存檔名稱:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(有任何程序错误或问题请联系我们)\r\n\r\n开发:\r\n界面设计 (Windows): Xavier Roche\r\n解析引擎: Xavier Roche\r\nJava解析引擎: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\n感谢:\r\nRobert Lagadec (rlagadec@yahoo.fr) 提供翻译技巧
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(有任何程序错误或问题请联系我们)\r\n\r\n开发:\r\n界面设计 (Windows): Xavier Roche\r\n解析引擎: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\n感谢:\r\nRobert Lagadec (rlagadec@yahoo.fr) 提供翻译技巧
About WinHTTrack Website Copier
关于WinHTTrack Website Copier
Please visit our Web page
@@ -928,3 +928,39 @@ Server terminated
A fatal error has occurred during this mirror
Proxy type:
代理类型:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
代理协议。HTTP:标准代理。HTTP(CONNECT 隧道):通过 CONNECT 隧道发送每个请求,用于仅支持 CONNECT 的代理,例如 Tor 的 HTTPTunnelPort。SOCKS5:默认端口 1080。
Load cookies from file:
从文件加载 cookies:
Preload cookies from a Netscape cookies.txt file before crawling.
在抓取前从 Netscape 格式的 cookies.txt 文件预加载 cookies。
Pause between files:
文件间暂停:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
文件下载之间的随机延迟(秒)。使用 MIN:MAX 指定随机范围 (例如 2:8)。
Keep the www. prefix (do not merge www.host with host)
保留 www. 前缀 (不将 www.host 与 host 合并)
Do not treat www.host and host as the same site.
不把 www.host 和 host 视为同一站点。
Keep double slashes in URLs
保留 URL 中的双斜杠
Do not collapse duplicate slashes in URLs.
不合并 URL 中重复的斜杠。
Keep the original query-string order
保留查询字符串的原始顺序
Do not reorder query-string parameters when deduplicating URLs.
在对 URL 去重时不重新排列查询字符串参数。
Strip query keys:
剥离查询键:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
用逗号分隔的查询键,将其从保存文件的命名中删除 (例如 sid,utm_source)。
Write a WARC archive of the crawl
写入本次抓取的 WARC 归档
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
同时将每个已获取的响应保存为 ISO-28500 WARC/1.1 归档,置于镜像站点旁边。
WARC archive name:
WARC 归档名称:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Molimo da nas obavijestite o svim pogreškama i poteškoæama)\r\n\r\nRazvoj:\r\nSuèelje (Windows): Xavier Roche\r\nPauk: Xavier Roche\r\nRazrediRašèlanjivaèaJave: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche i drugi suradnici\r\nPUNO HVALA za prijevodne preporuke upuæujemo:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Molimo da nas obavijestite o svim pogreškama i poteškoæama)\r\n\r\nRazvoj:\r\nSuèelje (Windows): Xavier Roche\r\nPauk: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche i drugi suradnici\r\nPUNO HVALA za prijevodne preporuke upuæujemo:\r\nRobert Lagadec (rlagadec@yahoo.fr)
About WinHTTrack Website Copier
O programu WinHTTrack Website Copier
Please visit our Web page
@@ -930,3 +930,39 @@ Server terminated
Poslužitelj je razriješen
A fatal error has occurred during this mirror
Tijekom ovog zrcaljenja je nastala fatalna pogreška
Proxy type:
Vrsta posrednika:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
Protokol posrednika. HTTP: standardni posrednik. HTTP (CONNECT tunel): ¹alje svaki zahtjev kroz CONNECT tunel, za posrednike koji podr¾avaju samo CONNECT poput Torovog HTTPTunnelPorta. SOCKS5: zadani port 1080.
Load cookies from file:
Uèitati kolaèiæe iz datoteke:
Preload cookies from a Netscape cookies.txt file before crawling.
Prije zrcaljenja uèitati kolaèiæe iz datoteke Netscape cookies.txt.
Pause between files:
Stanka izmeðu datoteka:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
Nasumièna odgoda izmeðu preuzimanja datoteka, u sekundama. Za nasumièni raspon koristiti MIN:MAX (npr. 2:8).
Keep the www. prefix (do not merge www.host with host)
Zadr¾ati www. predmetak (ne spajati www.host s host)
Do not treat www.host and host as the same site.
www.host i host ne smatrati istim web-mjestom.
Keep double slashes in URLs
Zadr¾ati dvostruke kose crte u URL-ovima
Do not collapse duplicate slashes in URLs.
Ne sa¾imati ponovljene kose crte u URL-ovima.
Keep the original query-string order
Zadr¾ati izvorni redoslijed teksta za upite
Do not reorder query-string parameters when deduplicating URLs.
Ne mijenjati redoslijed parametara teksta za upite pri uklanjanju dvostrukih URL-ova.
Strip query keys:
Ukloniti kljuèeve upita:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Zarezom odvojeni kljuèevi upita koji se izostavljaju iz naziva spremljenih datoteka (npr. sid,utm_source).
Write a WARC archive of the crawl
Zapi¹i WARC arhivu obilaska
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Spremi i svaki preuzeti odgovor u ISO-28500 WARC/1.1 arhivu, uz zrcalo.
WARC archive name:
Naziv WARC arhive:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(finder du fejl eller opstår der problemer så kontakt os venligst)\r\n\r\nUdvikling:\r\nBrugerflade (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche og andre bidragydere\r\nMange tak for dansk oversættelse til:\r\nJesper Bramm (bramm@get2net.dk)\r\nscootergrisen
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(finder du fejl eller opstår der problemer så kontakt os venligst)\r\n\r\nUdvikling:\r\nBrugerflade (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche og andre bidragydere\r\nMange tak for dansk oversættelse til:\r\nJesper Bramm (bramm@get2net.dk)\r\nscootergrisen
About WinHTTrack Website Copier
Om WinHTTrack Website Copier
Please visit our Web page
@@ -976,3 +976,39 @@ Click on this notification to restart the interrupted mirror
Klik på denne notifikation for at genstarte den afbrudte spejlkopiering
HTTrack: could not save profile for '%s'!
HTTrack: kunne ikke gemme profil for '%s'!
Proxy type:
Proxytype:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
Proxyprotokol. HTTP: standardproxy. HTTP (CONNECT-tunnel): sender hver forespørgsel gennem en CONNECT-tunnel, til proxyer der kun understøtter CONNECT som Tors HTTPTunnelPort. SOCKS5: standardport 1080.
Load cookies from file:
Indlæs cookies fra fil:
Preload cookies from a Netscape cookies.txt file before crawling.
Indlæs cookies fra en Netscape cookies.txt-fil, før crawlingen starter.
Pause between files:
Pause mellem filer:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
Tilfældig forsinkelse mellem filoverførsler, i sekunder. Brug MIN:MAX for et tilfældigt interval (f.eks. 2:8).
Keep the www. prefix (do not merge www.host with host)
Behold www.-præfikset (sammenlæg ikke www.host med host)
Do not treat www.host and host as the same site.
Behandl ikke www.host og host som det samme websted.
Keep double slashes in URLs
Behold dobbelte skråstreger i URL'er
Do not collapse duplicate slashes in URLs.
Sammenfold ikke gentagne skråstreger i URL'er.
Keep the original query-string order
Behold den oprindelige rækkefølge i forespørgselsstrengen
Do not reorder query-string parameters when deduplicating URLs.
Omorden ikke forespørgselsstrengens parametre, når dublerede URL'er fjernes.
Strip query keys:
Fjern forespørgselsnøgler:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kommaseparerede forespørgselsnøgler, der udelades i navngivningen af gemte filer (f.eks. sid,utm_source).
Write a WARC archive of the crawl
Skriv et WARC-arkiv af gennemsøgningen
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Gem også hvert hentet svar i et ISO-28500 WARC/1.1-arkiv ved siden af spejlet.
WARC archive name:
Navn på WARC-arkiv:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen.
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Bitte benachrichtigen Sie uns über Fehler und Probleme)\r\n\r\nEntwicklung:\r\nOberfläche (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\nJavaParserClasses: Yann Philippot\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for German translations to:\r\nRainer Klueting (rainer@klueting.de)
\r\n(Please notify us of any bug or problem)\r\n\r\nDevelopment:\r\nInterface (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for translation tips to:\r\nRobert Lagadec (rlagadec@yahoo.fr)
\r\n(Bitte benachrichtigen Sie uns über Fehler und Probleme)\r\n\r\nEntwicklung:\r\nOberfläche (Windows): Xavier Roche\r\nSpider: Xavier Roche\r\n\r\n(C)1998-2003 Xavier Roche and other contributors\r\nMANY THANKS for German translations to:\r\nRainer Klueting (rainer@klueting.de)
About WinHTTrack Website Copier
Über WinHTTrack Website Copier
Please visit our Web page
@@ -928,3 +928,39 @@ Server terminated
Der Server wurde beendet
A fatal error has occurred during this mirror
Fataler Fehler während der Webseiten-Kopie
Proxy type:
Proxy-Typ:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
Proxy-Protokoll. HTTP: normaler Proxy. HTTP (CONNECT-Tunnel): sendet jede Anfrage durch einen CONNECT-Tunnel, für Proxys, die nur CONNECT unterstützen, wie Tors HTTPTunnelPort. SOCKS5: Standardport 1080.
Load cookies from file:
Cookies aus Datei laden:
Preload cookies from a Netscape cookies.txt file before crawling.
Cookies aus einer Netscape-cookies.txt-Datei laden, bevor das Crawlen beginnt.
Pause between files:
Pause zwischen Dateien:
Random delay between file downloads, in seconds. Use MIN:MAX for a random range (e.g. 2:8).
Zufällige Verzögerung zwischen Datei-Downloads, in Sekunden. MIN:MAX für einen Zufallsbereich verwenden (z. B. 2:8).
Keep the www. prefix (do not merge www.host with host)
www.-Präfix beibehalten (www.host nicht mit host zusammenführen)
Do not treat www.host and host as the same site.
www.host und host nicht als dieselbe Website behandeln.
Keep double slashes in URLs
Doppelte Schrägstriche in URLs beibehalten
Do not collapse duplicate slashes in URLs.
Doppelte Schrägstriche in URLs nicht zusammenführen.
Keep the original query-string order
Ursprüngliche Reihenfolge des Query-Strings beibehalten
Do not reorder query-string parameters when deduplicating URLs.
Query-String-Parameter beim Entfernen doppelter URLs nicht neu anordnen.
Strip query keys:
Query-Schlüssel entfernen:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kommagetrennte Query-Schlüssel, die bei der Benennung gespeicherter Dateien entfallen (z. B. sid,utm_source).
Write a WARC archive of the crawl
WARC-Archiv des Crawls schreiben
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Jede heruntergeladene Antwort zusätzlich in einem ISO-28500-WARC/1.1-Archiv neben dem Spiegel speichern.
WARC archive name:
Name des WARC-Archivs:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.