Compare commits

...

24 Commits

Author SHA1 Message Date
Xavier Roche
dc5caa8920 changes: translate the new GUI strings into the remaining 28 locales
The feature PR added the two LANG_CHANGES* entries to lang.def with English and
Francais only; every other locale fell back to English in the WebHTTrack form.
Each file is written in its own declared charset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 21:33:18 +02:00
Xavier Roche
6f4b390aa5 Merge origin/master into feat/change-report
Records the second parent: the previous commit carried master's content but
was committed after a stash cleared MERGE_HEAD, so git never saw it as a merge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

# Conflicts:
#	tests/Makefile.am
2026-07-26 19:04:08 +02:00
Xavier Roche
b099e303e5 Merge origin/master into feat/change-report
Both sides appended to tests/Makefile.am's TESTS; kept master's
86_local-proxytrack-cache-longfields.test alongside 88_local-changes.test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 19:00:24 +02:00
Xavier Roche
ca533abefd Unbounded writes into the 80-byte htsblk failure message (#722)
* Bound the remaining writes into htsblk.msg[80]

Nine writers still filled the 80-byte msg with no bound. Six sprintf the
result of strerror(), whose longest glibc string is 49 bytes in the C
locale and 72 in fr_FR against 46 bytes of room after the longest prefix.
Realistic connect() errnos still fit, so this was latent rather than live.

The FTP helper's .ok result file had no excuse: it was copied byte by byte
until EOF into the same 80 bytes. Split that parse out as
back_read_ftp_result() so a self-test can drive it, and stop at capacity.

These sites never appeared in the -Wformat-truncation cluster that added
htsblk_failf: the diagnostic only fires on a bounded snprintf whose return
is discarded, so a raw sprintf is invisible to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the self-test's blind spots around msg[]

Six of eight mutants survived the first version. The neighbour canary
compared against zero, so it saw a stray 'X' but not the stray NUL an
off-by-one terminator actually writes; poison it instead. Only the
over-capacity case existed, so padding every message to 79 bytes or eating
its last character both passed, as did a sign-extended 0xff reading as EOF
and the new unparseable-status branch, which had no coverage at all.

Same zero-comparison weakness applied to the htsblk_failf canary inherited
from #715, so that one is poisoned too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:51:41 +02:00
Xavier Roche
dc6819b7b7 Cached headers longer than ProxyTrack's fields overflow into the location pointer (#717)
* Bound ProxyTrack's cache header copies

ZIP_READFIELD_STRING in proxy/store.c took no destination size and used a
raw strcpy, while the engine's namesake in htscache.c has taken a
refvalue_size for years. The source is a cache-entry header line bounded
only by line[HTS_URLMAXSIZE + 2].

contenttype[64] sits just before the location pointer in the calloc'd
_PT_Element, so an over-long Content-Type walks over charset and then
over location, which the next Location: line copies through.

Clip rather than reject: the engine stores Content-Type in 128 bytes, so
a valid cache legitimately carries fields wider than ours.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the ARC reader's header copies too

Review found store.c carries a second copy of the same macro:
HTTP_READFIELD_STRING feeds the ARC reader from index->line[2048], twice
the ZIP path's reach, into the same contenttype[64] and its neighbouring
location pointer. proxytrack --convert on a plain-text ARC segfaults.

Test 86 covered one of eight fields, so a per-field clip and a
one-size-fits-all clip were indistinguishable. It now overshoots every
destination and asserts each surviving length against its own capacity,
and exercises the ARC reader without needing python.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:51:13 +02:00
Xavier Roche
6a20dfd998 changes: refresh two stale test comments
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 18:37:17 +02:00
Xavier Roche
08b32f4878 changes: document the renamed-file case in the format page
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 18:36:21 +02:00
Xavier Roche
a7ed9c3ca0 changes: fix the review's three findings
The size shortcut compared rendered on-disk sizes even for parsed pages, whose
payload digests describe something else entirely, so it decided the outcome
before the payload comparison could run: a page whose payload never changed but
whose rewritten links moved read as changed. It now only applies when both
digests describe the file on disk.

file_notify() reaches the accumulator from the FTP download thread as well as
the main one, and the lazy allocation, the coucal write and the entries realloc
were all unguarded. Every entry point now takes a mutex, and the HTML hook does
its cache read before taking it, since that read can itself re-enter
file_notify() and move the array.

Two fixtures cover what nothing did: a gzipped direct-to-disk body that changes
at constant length (without the pre-sample before the decoded temp is renamed,
it reads as unchanged), and a page with a fixed payload behind a redirect whose
target is renamed, so its file on disk changes length while its bytes do not.
Both were checked against builds with the respective fix reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 18:35:43 +02:00
Xavier Roche
11c0aa59df changes: drop em dashes from the format page
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 18:10:20 +02:00
Xavier Roche
f7f428bc93 changes: never leave a stale report behind
A crawl that mirrored nothing created no accumulator, so hts_changes_close_opt
returned without writing and the previous run's report stayed on disk as if it
described this one. Write it whenever --changes is on. The no-data rollback is
the deliberate exception, and is now documented: it restores the previous cache
generation, so leaving the matching report alone is the consistent behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 18:08:04 +02:00
Xavier Roche
9061f372e4 Merge origin/master into feat/change-report 2026-07-26 18:03:01 +02:00
Xavier Roche
3247e5b183 Diagnostic messages built from remote replies discard snprintf's truncation signal (#715)
* Fold htsblk/PT_Element failure messages into a clipping helper

The 18 remaining -Wformat-truncation warnings all came from building a
diagnostic string out of a remote server's reply and dropping snprintf's
return. Add htsblk_failf() for htsblk.msg[80] and PT_Element_failf() for
ProxyTrack's msg[1024]: both clip to fit and absorb the discard once, so
the obligation is not silently laundered at each call site.

msg[80] lives in the installed htsopt.h, so growing it would break the
ABI; a clipped FTP banner is the intended outcome anyway. The display
StatsBuffer.state is not installed, so it grows to fit back->info instead.

Also bounds two unbounded sprintf(r->msg, ...) in ProxyTrack's cache
reader and moves the raw strcpy neighbours to strcpybuff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch a one-past-the-end write into msg[]'s neighbour

Review found the new self-test blind to a store at msg[sizeof(msg)]: it
lands in the adjacent contenttype field, which no assertion read, and an
intra-struct overflow is invisible to ASan and _FORTIFY_SOURCE. Check the
neighbour after every call, and add the exact-fit case the block lacked.

Correct the contract comment too: msg is not purely diagnostic, it
round-trips through the cache as X-StatusMessage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:00:21 +02:00
Xavier Roche
c38bf69c7d Report what a crawl changed against the previous mirror (--changes)
--update already knows which resources were new, which changed and which the
server called unchanged, and throws it away: the flags reach file_notify() and
go no further than a log line, while deletions exist only as a side effect of
purging. --changes (-%d) keeps all of it and writes hts-changes.json plus a
one-line summary in the log.

"Changed" means the bytes differ, not that the server re-sent the resource.
Comparing the mirrored files directly would not work: HTTrack stamps every
parsed page with the crawl date via the footer, so those bytes differ on every
run. Payloads are compared instead, the previous one coming from the cache for
parsed pages and from the local copy sampled just before it is overwritten for
everything else.

The mirror-relative path, not the URL, is the accumulator's key, so a redirect
and its target that share a save name are one entry; and only the first notify
for a file samples its pre-run state, so a retried transfer is not counted
twice. What counts as already mirrored comes from the previous run's file
index rather than from the file's presence on disk: a partial left by this
crawl's own failed attempt is on disk but was never part of the previous
mirror.

The deleted set is now computed whether or not purging is enabled; unlinking
still happens only under --purge-old.

Closes #714

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-26 17:57:07 +02:00
Xavier Roche
bef7120423 tests: test 82 fails on master after #707 and #709 landed together (#716)
* tests: arm test 82's mirror root through a profile save

#707 made /website/ serve only the root htsserver recorded at structcheck
success, so the posted projpath test 82 relied on no longer names anything.
Each PR was green alone; the pair was not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the now-unused project-path argument

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 17:44:55 +02:00
Xavier Roche
7cf54af485 htsserver: a posted projpath repoints the served root and overflows the composed path (#707)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* htsserver: serve the crawled mirror verbatim, never through the expander

WebHTTrack serves the mirror under /website/ from the same small server as its
own GUI, and the decision to run a response through the ${...} template
expander was a substring test for ".htm" on the request path. A mirrored page
therefore had its directives evaluated: ${_sid} rendered the live session id,
handing the crawled site the token that authenticates commands on the local
GUI, and ${do:...} gave it the rest of the template verbs.

The /website/ prefix was already detected, but only to keep the crawl-state
override from hijacking a mirror request. Reuse it as the expansion gate, so
expansion is limited to files under the GUI's html root, and re-evaluate it
after that override, which can substitute a GUI page for a mirror path.
Mirrored pages keep their text/html type: verbatim must not turn browsing the
mirror into a download.

Note that mirrored content still shares the control origin, so a script in it
can read the session id from a GUI page itself; that is a separate fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: stop letting a posted projpath name the /website/ root

The static-file path in smallserver() was composed with a length guard that
measured only the server root and the request path, then wrote the posted
"projpath" field into a 1024-byte stack buffer: a 2000-byte projpath followed
by any /website/ request smashed the stack (buffer overflow detected, server
gone). The guard also summed two untrusted lengths before comparing, the shape
that can wrap and pass.

Composition now goes through a bounded, non-aborting append that keeps the
untrusted length alone on one side, and /website/ is served from the project
directory the server itself set up, rejecting a ".." in it, rather than from
whatever root the request body claimed. Without that, projpath=/etc/ plus
GET /website/passwd read an arbitrary file, since the ".." check looked at the
request path only. fsfile is also cleared before the error-redirect branch,
which could otherwise reach fopen() uninitialized.

tests/80_webhttrack-projpath.test drives the three cases against a live server
and keeps a legitimate project browsable as the control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: establish test 79's mirror root through the server

/website/ no longer serves the posted projpath, so injecting one as a
fixture stopped working and test 79 got a 404 instead of the mirrored
page. Save a profile first (no command_do=start, so nothing crawls) to
make the server record the root, then plant the file under it.

Re-checked against a reverted expander gate: still fails there, so the
fixture change did not cost the assertion its teeth.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the running-crawl half of the /website/ override

Test 84 only exercised the idle server, where the override never fires and
the recomputed virtualpath is indistinguishable from the stale one. Drive a
crawl through the server so /website/*.html is rewritten to the GUI refresh
page, which 404s without the recompute.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the overflow, the '..' rejection and the fsfile hoist

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: fix the merged test 84 duplicate post() helper

The rename-side and the branch-side each defined post(), with different
argument shapes; the last one won and silently mangled the save body.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: trim the new comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the stale start() arg comments in test 84

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 17:18:43 +02:00
Xavier Roche
c32a47110e webhttrack: the mirror link on the finished page cannot be followed (#709)
* WebHTTrack's mirror links are dead file:// URLs

The GUI is served from http://127.0.0.1:PORT/, and browsers refuse to navigate
from an http: page to a file: URL, so the "browse the mirror" links in
finished.html and file.html did nothing when clicked. They now point at the
server's own /website/ route, which already serves the project directory. The
desktop entry's browse mode still works, because there the shell hands the
file: URL to the browser instead of navigating from a page.

The per-project picker in file.html goes too: /website/ is bound to the running
project, and reaching the other projects over HTTP would mean serving the whole
mirror tree from the control origin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* finished.html: link the served mirror, not a dead file:// URL

The page is served over http:, so its file:// link is a cross-scheme
navigation that Chrome and Firefox refuse. Nothing happened on click. The
mirror is already reachable at /website/, which the two list entries just
below were using all along.

file.html keeps its file:// links. It is the fresh-session entry point for
sites mirrored earlier, and /website/ is bound to one project, so pointing
it there would trade a dead link for a 404. Serving an arbitrary past
project needs a route that does not exist yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: tighten the browse-link assertion and trim its comments

The runtime check for href="/website/index.html" also matched the list
entry below the anchor, so it passed on the unfixed page; match the
anchor by its mirror-path label instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 17:18:17 +02:00
Xavier Roche
1027a9f392 htsserver: crawled pages run through the template expander and leak the session id (#706)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* htsserver: serve the crawled mirror verbatim, never through the expander

WebHTTrack serves the mirror under /website/ from the same small server as its
own GUI, and the decision to run a response through the ${...} template
expander was a substring test for ".htm" on the request path. A mirrored page
therefore had its directives evaluated: ${_sid} rendered the live session id,
handing the crawled site the token that authenticates commands on the local
GUI, and ${do:...} gave it the rest of the template verbs.

The /website/ prefix was already detected, but only to keep the crawl-state
override from hijacking a mirror request. Reuse it as the expansion gate, so
expansion is limited to files under the GUI's html root, and re-evaluate it
after that override, which can substitute a GUI page for a mirror path.
Mirrored pages keep their text/html type: verbatim must not turn browsing the
mirror into a download.

Note that mirrored content still shares the control origin, so a script in it
can read the session id from a GUI page itself; that is a separate fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the running-crawl half of the /website/ override

Test 84 only exercised the idle server, where the override never fires and
the recomputed virtualpath is indistinguishable from the stale one. Drive a
crawl through the server so /website/*.html is rewritten to the GUI refresh
page, which 404s without the recompute.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:57:59 +02:00
Xavier Roche
d0a1573a04 webhttrack: "max site size" sets a per-file cap and discards the HTML one (#708)
* webhttrack: "max site size" set a per-file cap instead of the overall one

step4.html mapped all three size fields of the wizard onto --max-files (-m),
so "Max site size" emitted a per-file limit rather than --max-size (-M), and
landed a second -m on the command line. That second -m also clobbered the HTML
per-file limit: a bare -m<n> resets maxfile_html, so whichever of the two came
last won. Point sizemax at --max-size and emit the bare -m before the -m,<n>
form so both per-file caps survive.

Two template typos in the same family, where a malformed ${...} renders as
nothing or as its own key instead of erroring: the winprofile.ini writer's
Dos=${dos was missing its closing brace, and option2b.html's OK button read
${LANG_OK] with a bracket.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* tests: carry the session id in test 81's POST

The gate that landed with #700 refuses a body without one, so the wizard
POST came back refused and the option audit had nothing to read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: close the confirmation-biased gaps in test 81

Post the size fields empty too, so a step4.html that lost its ${test:} guard
and rendered a valueless --max-files= is caught; assert the winprofile.ini
MaxHtml/MaxOther/MaxAll keys the header claimed to audit; and pin the OK button
label, since an unknown ${LANG_} key renders empty and passed the absence check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:40:40 +02:00
Xavier Roche
4362ac34cb Check snprintf truncation where the result must be complete (#711)
Adds `sprintfbuff()`/`slprintfbuff()` to `htssafe.h`: a formatted print that truncates to fit and returns whether it had to, marked `warn_unused_result` so the answer cannot be dropped. It fills the gap between the `strcpybuff` family, which aborts on overflow, and `String`, which grows without bound. Abort is the wrong contract wherever the text is built from a remote peer's reply.

Four `-Wformat-truncation=` sites used the result as if `snprintf` had never truncated. Two only needed a bigger destination, so they get one: `hts_finish_makeindex`'s `tempo` was a flat 1024 against a 2048-byte escaped URL and is now sized off it, and the wizard's `cmd[4096]` could not hold the answers it concatenates. The other two cannot grow. `create_back_tmpfile` formats `<url_sav>.bak` into a buffer the same size as `url_sav`, and that struct is installed, so a dropped extension would alias the backup onto the live file that `back_finalize_backup()` unlinks. ProxyTrack's `startUrl[1024]` is fed by cache content. Both take the error path they already had, and ProxyTrack moves to the next cache entry rather than publishing a clipped one.

On what the wrapper buys, since it is not what I first assumed: checking a raw `snprintf` return inline silences the warning just as well. The wrapper's value is that the capacity comes from `sizeof`, the check is the default rather than the exception, and `warn_unused_result` makes skipping it visible.

`-#test=strsafe` covers the primitive (exact fit, one over, 4 KB source, `size == 1`, trailing canary, destination repoisoned between cases), and `-#test=makeindex` gains a first link whose escaped form overruns the old buffer. Both were mutation-checked. The ProxyTrack change ships without a direct test: its only observable is the catalog page, which renders solely as a PROPFIND fallback I could not drive from curl. 25 gcc warnings down to 21; the rest of the cluster is diagnostic-only and follows separately.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:37:48 +02:00
Xavier Roche
6579436607 webhttrack: the command line overflows its argv vector and a quoted value can inject flags (#710)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* webhttrack: bound the argv vector and escape quotes in the wizard command line

The wizard hands its httrack command line to the engine as one string, which
back_launch_cmd() split back into argv. Two things were wrong with that split.

It wrote into a fixed 1024-pointer vector with no bound, and every unquoted
space in the posted string yields an entry, so an ordinary mirror with a few
hundred URLs walked off the allocation. The split now lives in htscmdline.c as
hts_split_cmdline(), which sizes the vector from the separator count before
filling it, and the engine self-tests can reach it.

Quotes were also purely advisory: they toggled the "inside an argument" state
but nothing escaped them, so a double quote typed into a wizard field (user
agent, footer, path, project name) closed the argument early and the rest of
the value was parsed as fresh options -- among them -V, which reaches system().
Escaping has to happen where the argument boundary is known, so the template
gets its own ${arg:} filter for that context; ${html:} keeps its meaning for
the HTML attributes it is used in everywhere else, and HTML escaping would not
help anyway since the browser undoes it when it posts the command line back.
${arg:} backslash-escapes a quote and a backslash, and the splitter reads those
inside a quoted run, the same convention next_token() already implements for
doit.log. A value containing a quote now survives it intact instead of turning
into options.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* webhttrack: keep a stray quote in an unquoted field out of the split

The url and wildcard-filter fields go into the command line outside quotes,
where no backslash can escape anything: a single quote there flips the parity
of every quote after it, so a later escaped value ends up split as flags and
the escaping buys nothing. Emit %22 for those fields instead.

NULL-terminate the argv vector while here, matching the convention the tree
documents in htscharset.c, and fold the third copy of the entity table into
one helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: do not feed snprintf's return value back as its size argument

snprintf returns the length it wanted to write, so accumulating it blind
lets the next size argument wrap. The buffer is sized well past what the
loop needs, but the pattern is the one the project forbids.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:26:55 +02:00
Xavier Roche
cb98b1b197 Fatal-signal backtraces are unreadable: every engine frame is a bare module+offset (#705)
* Symbolize fatal-signal backtraces through addr2line

backtrace_symbols_fd() resolves names from .dynsym only, and
-fvisibility=hidden keeps every engine frame out of it, so a crash report
arrived as a column of bare module+offset. The handler now emits the raw trace
first and unconditionally, then groups the frames per module and runs addr2line
(or llvm-symbolizer) over the offsets, which reads DWARF and names the static
frames plus their inline chain.

-rdynamic is dropped: it only populated .dynsym and bought exactly one named
frame. -Wl,--build-id replaces it, so a trace from a stripped build can be
matched to its debug symbols.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Move the crash backtrace printer into src/htsbacktrace.c

httrack.c keeps only the two call sites. The symbolizer needs _GNU_SOURCE for
dladdr(), which is now confined to its own translation unit instead of being
forced on the whole CLI front-end.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test: match glibc's backtrace format with or without the space

backtrace_symbols_fd() prints the trailing "[0xADDR]" with a leading space on
some glibc versions and without on others (Ubuntu 24.04), so the raw-frame
assertion failed everywhere but the dev box. Match only up to the offset.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Skip pseudo-modules with no file on disk

A frame in linux-vdso.so.1 made addr2line complain instead of resolving, so
the arm64 leg lost its symbolized output entirely. Renumber the test too:
77 landed on master with #700.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Stop discarding local symbols, which made the names wrong

--discard-all drops the local symbol of every static function, so addr2line
attributes the frame to the nearest surviving global: the abort frame read
dns_timeout_selftests instead of abortf_, with the file and line still right.
A wrong name is worse than none, and because the symbols go at link time no
-dbgsym package can recover them. Costs 21784 bytes on libhttrack.so.3, 0.6%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Symbolize once when the handler itself faults

A fault inside the handler re-enters the printer, which interleaved a second
symbolized trace on the same fd and spent a second budget: 3.04s and two
overlapping reports, measured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 15:43:17 +02:00
Xavier Roche
da8fbfff49 Fix dead array-address checks and discarded-qualifier casts (-Waddress, -Wcast-qual) (#703)
* Drop NULL tests on inline array members

`lien_back::url_sav`, `htsblk::msg` and POSIX `dirent::d_name` are arrays,
so testing their address folds to a constant and gcc/clang report it
(-Waddress, -Wpointer-bool-conversion). Every site keeps whatever real
condition sat beside the dead one, so behavior is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* help_wizard: check the allocation, not the arrays it contains

The out-of-memory guard has been constant-false since d593418 folded the
nine separate wizard buffers into one struct: the names it tests are now
inline arrays, so `malloct()`'s result is never checked and an exhausted
heap gets a NULL-page write instead of the intended message. Also switch
the raw free() to freet() and release the struct on the two early returns
that leaked it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* proxytrack: test the WebDAV header fields, not their addresses

`PT_Element::lastmodified` and `::contenttype` are inline arrays, so both
`if`s were constant-true (-Waddress); use the `[0]` form the same file
already uses when it emits the GET headers. Neither is observable:
get_time_rfc822("") returns 0 and falls through to the index timestamp,
and proxytrack_add_DAV_Item already substitutes application/octet-stream
for an empty mime, which a PROPFIND probe against a cache entry carrying
no Content-Type confirms both before and after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix the discarded const qualifiers rather than casting them away

`binput` and `cache_binput` only ever read through their source pointer,
so they take `const char *` now; that alone clears the cast in
htsrobots.c, and neither is exported nor declared in an installed header,
so no ABI question arises. `treathead` keeps `char *rcvd` because it does
NUL-cut the header in place, and the two selftest calls that fed it a
string literal get a mutable buffer instead, matching their three
siblings and removing a latent write to .rodata. The remaining two are
one-liners: zlib's `next_in` is already `const` under -DZLIB_CONST, and
htsback can call the non-const `jump_protocol` twin on its mutable
`url_adr`. libhttrack.vcxproj gains ZLIB_CONST so the MSVC build agrees
with autotools, as webhttrack and proxytrack already do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: prove the WebDAV mime/timestamp fallback survives an empty field

proxytrack's DAV PROPFIND response computes a fallback content-type and
timestamp when a cache entry has no Content-Type/Last-Modified; that
fallback already existed before commit eae1dd0 changed the surrounding
always-true array-address checks, so this test guards the equivalence
rather than a bug. Verified it fails when the fallback default is
disabled, and passes unmodified against the pre-eae1dd0 code too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the PROPFIND request and cut the header comment

curl had no --max-time; an unbounded read wedges the runner instead of
failing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: skip the WebDAV mime test on Windows

It is the first test to run proxytrack as a live listener, and MSYS cannot
reap a native one: the orphan wedged the whole Windows suite past its
45-minute budget, twice, destroying the log upload with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 14:49:29 +02:00
Xavier Roche
a7fbd3f739 htsserver builds the redirect Location header in a 256-byte stack buffer (#700)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 13:59:15 +02:00
Xavier Roche
c4b803eb33 -%S list file over 4GB overflows the heap (#702)
* -%S list file over 4GB overflows the heap

hts_main_internal() sized the -%S buffer as `cl + fz + 8192` and stored
the sum in an int url_sz, then fread() the untruncated 64-bit fz into it.
A 4GB+100KB rules file wraps the capacity to 110602 bytes on x64, the
realloct() succeeds, and the read walks off the heap. Intermediate sizes
land on a negative int and fail the allocation, which is luck, not design.

Route the file-size arithmetic through llint_grow_size_t(), a saturating
sibling of llint_to_size_t() that refuses a total it cannot represent, and
widen url_sz and the filelist offsets to size_t. The "config" sizing in the
same function and htscore.c's primary_len had the same shape: a file size
accumulated into an int before reaching an allocator. htscache.c's two
mirrored-file comparisons held a 64-bit fsize_utf8() in a size_t, which
truncates on Win32 and re-downloads a >4GB file already on disk.

Found by MSVC C4244 on x64; invisible to gcc/clang because int64_t to
size_t is width-preserving on LP64.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Print T_SOC with the format matching its width, not a bare %d

T_SOC is unsigned __int64 on Win64 (htsglobal.h): passing it to fprintf's
%d is undefined behavior, flagged by MSVC C4477. Add T_SOCP beside the
typedef, following the existing LLintP/INTsysP precedent, and use it at
both deletesoc() call sites (htslib.c:2601, :2607).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix MSVC C2099 in the new growsize self-test

A static const object used inside another object's static initializer is a
GNU/clang extension, not standard C: MSVC's /TC C mode rejects it ("initializer
is not a constant"). Replace the over32 local with a macro.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the slack-only overrun and the largest capacity

A helper dropping the slack bound passed the table; -1 as extra also refused
either way, since llint_to_size_t() maps it to SIZE_MAX regardless.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 13:23:03 +02:00
98 changed files with 4585 additions and 470 deletions

View File

@@ -225,8 +225,9 @@ jobs:
# Every gate here exits 77, so an all-skipped suite would report green having
# tested nothing: pin the skips, and floor the passes in case the glob empties.
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test"
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
# webdav-mime needs a reapable background listener, which MSYS cannot give it.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
[ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

View File

@@ -100,7 +100,8 @@ AX_CHECK_COMPILE_FLAG([-fstack-protector-strong], [DEFAULT_CFLAGS="$DEFAULT_CFLA
[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"], [], [-Werror])], [-Werror])
AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"], [], [-Werror])
AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"], [], [-Werror])
AX_CHECK_LINK_FLAG([-Wl,--discard-all], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--discard-all"])
# No --discard-all: it drops the local symbols naming every static function, so
# a trace misattributes them to the nearest surviving global. Costs 0.6% size.
AX_CHECK_LINK_FLAG([-Wl,--no-undefined], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--no-undefined"])
AX_CHECK_LINK_FLAG([-Wl,-z,relro,-z,now], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,relro,-z,now"])
AX_CHECK_LINK_FLAG([-Wl,-z,noexecstack], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,noexecstack"])
@@ -127,8 +128,8 @@ AX_CHECK_LINK_FLAG([-pie], [LDFLAGS_PIE="-pie"])
AC_SUBST([CFLAGS_PIE])
AC_SUBST([LDFLAGS_PIE])
## -rdynamic must be a link flag; DEFAULT_CFLAGS (AM_CPPFLAGS) never reaches the linker.
AX_CHECK_LINK_FLAG([-rdynamic], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -rdynamic"])
# Ties a crash trace from a stripped build back to its separate debug symbols.
AX_CHECK_LINK_FLAG([-Wl,--build-id], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--build-id"])
### Check for -fvisibility=hidden support
gl_VISIBILITY

245
html/changes.html Normal file
View File

@@ -0,0 +1,245 @@
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Change report format specification</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
</head>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<h2 align="center"><em>Change report format specification</em></h2>
<br />
Run with <tt>--changes</tt> (<tt>-%d</tt>), HTTrack writes <tt>hts-changes.json</tt>
in the project directory, next to <tt>hts-log.txt</tt>, describing what the crawl
left new, changed, unchanged and gone compared to the previous mirror. The file is
rewritten from scratch at the end of every run, and the log carries a one-line
summary of the same counts.
<br /><br />
<h3>What "changed" means</h3>
A resource is changed when its bytes differ, not when the server merely re-sent
it. HTTrack compares the payload it just received against the copy the previous
run left behind: for pages it parses, the previous payload comes from the cache
(the file on disk carries the mirror footer and its crawl date, so its bytes
differ on every run); for everything else, the mirrored file is the payload
verbatim and is compared directly.
<br /><br />
Where no digest can be taken on either side, because the cache is disabled or
the previous copy is gone, the report falls back to the transfer signal, and a
server that answers 200 rather than 304 reads as changed. Keeping the cache on
(the default) is what makes the report precise.
<br /><br />
<h3>Fields</h3>
<ul>
<li><tt>schema</tt>: format version, currently <tt>1</tt>. It is bumped only
on an incompatible change; new fields may appear without one.</li>
<li><tt>generator</tt>: the HTTrack build that wrote the file.</li>
<li><tt>date</tt>: when the report was written, UTC, <tt>YYYY-MM-DDThh:mm:ssZ</tt>.</li>
<li><tt>first_crawl</tt>: true when no index of a previous mirror
(<tt>hts-cache/old.lst</tt>) was found, so there was nothing to compare against and
everything is listed as new.</li>
<li><tt>partial</tt>: true when the report ran out of memory and lists only
part of the mirror.</li>
<li><tt>purged</tt>: true when <tt>--purge-old</tt> was in effect, so the
files under <tt>gone</tt> were also deleted from disk.</li>
<li><tt>counts</tt>: the size of each of the four lists.</li>
<li><tt>new</tt>, <tt>changed</tt>, <tt>unchanged</tt>, <tt>gone</tt>: the
lists themselves. Every mirrored file appears in exactly one of them.</li>
</ul>
Each entry is an object:
<ul>
<li><tt>url</tt>: the absolute URL the file came from. Empty under
<tt>gone</tt>: deletions are computed from the mirror's file index, which records
paths, not URLs.</li>
<li><tt>file</tt>: the path relative to the mirror root, with forward
slashes. This is the entry's identity: a URL and a redirect that resolve to the
same local file are one entry, not two.</li>
<li><tt>size</tt>: the mirrored file's size in bytes, absent when the file
is not on disk.</li>
<li><tt>previous_size</tt>: under <tt>changed</tt> only, the size of the
copy the previous run left.</li>
</ul>
<br />
<h3>Encoding</h3>
The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are
not guaranteed to be valid UTF-8; any byte sequence that is not becomes
U+FFFD (<tt>\ufffd</tt>), so the file always parses. Compare on <tt>file</tt>
rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
<br /><br />
<h3>Example</h3>
<pre>
{
"schema": 1,
"generator": "HTTrack Website Copier/3.49-14",
"date": "2026-07-26T15:29:03Z",
"first_crawl": false,
"partial": false,
"purged": true,
"counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
"new": [
{ "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
],
"changed": [
{ "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
],
"unchanged": [
{ "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
],
"gone": [
{ "url": "", "file": "example.com/c.html" }
]
}
</pre>
<br /><br />
<h3>Notes</h3>
<ul>
<li>A file listed under <tt>gone</tt> is only deleted when <tt>--purge-old</tt> is
on. Left in place it drops out of the mirror's index, so it is reported once and
not again.</li>
<li>A resource whose local file name changed since the previous mirror (a new
MIME type, say) is reported as <tt>new</tt> under its new name; the old name is
reported as <tt>gone</tt> only if the file is still on disk. The two entries are
not paired.</li>
<li>A run that transfers no data at all is rolled back: HTTrack restores the
previous cache generation and leaves the previous report in place, so a lost
connection does not overwrite a good report with an empty one.</li>
<li>Content diffs, and keeping the previous copy of a changed page, are out of
scope: both change what a mirror directory contains.</li>
</ul>
<br /><br />
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
</body>
</html>

View File

@@ -482,6 +482,15 @@ add <tt>--warc-cdx</tt> for a sorted CDXJ index, or <tt>--wacz</tt> to bundle th
archive, index and pages into one WACZ for replay tools such as
replayweb.page.</small></p>
<h4>See what a re-crawl changed</h4>
<p><tt>httrack https://example.com/ --update --changes --path mydir</tt><br>
<small>Writes <tt>hts-changes.json</tt> in the project folder, listing every
mirrored file as new, changed, unchanged or gone, plus a one-line summary in the
log. &quot;Changed&quot; means the bytes really differ: a server that answers 200
with the same content it served last time lands in <tt>unchanged</tt>. Deletions
are reported whether or not <tt>--purge-old</tt> is deleting them. The format is
documented in <a href="changes.html">the change report specification</a>.</small></p>
<h4>HTTrack as a fetch tool</h4>
<p><tt>httrack --get https://host/file.bin --path tmp</tt><br>
<small><tt>--get</tt> fetches one file with cache, index, depth, cookies and robots
@@ -490,7 +499,8 @@ all disabled.</small></p>
<p><br>For the complete, always-current option list, see
<a href="httrack.man.html">the manual page</a>. For the filter language, see
<a href="filters.html">filters</a>; for the cache and updates, see
<a href="cache.html">cache</a>.</p>
<a href="cache.html">cache</a>; for the change report, see
<a href="changes.html">changes</a>.</p>
<!-- ==================== Start epilogue ==================== -->
</td>

View File

@@ -129,6 +129,9 @@ The library can be used to write graphical GUIs for httrack, or to run mirrors f
<li><a href="cache.html">Cache format</a></li><br>
HTTrack stores original HTML data and references to downloaded files in a cache, located in the hts-cache directory.
This page describes the HTTrack cache format.
<li><a href="changes.html">Change report format</a></li><br>
With --changes, HTTrack writes hts-changes.json describing what the crawl left new, changed, unchanged and gone
compared to the previous mirror. This page describes that file.
</ul>

View File

@@ -108,16 +108,17 @@ offline browser : copy websites to a local directory</p>
--footer</b> ] [ <b>-%l, --language</b> ] [ <b>-%a,
--accept</b> ] [ <b>-%X, --headers</b> ] [ <b>-C,
--cache[=N]</b> ] [ <b>-k, --store-all-in-cache</b> ] [
<b>-%r, --warc</b> ] [ <b>-%n, --do-not-recatch</b> ] [
<b>-%v, --display</b> ] [ <b>-Q, --do-not-log</b> ] [ <b>-q,
--quiet</b> ] [ <b>-z, --extra-log</b> ] [ <b>-Z,
--debug-log</b> ] [ <b>-v, --verbose</b> ] [ <b>-f,
--file-log</b> ] [ <b>-f2, --single-log</b> ] [ <b>-I,
--index</b> ] [ <b>-%i, --build-top-index</b> ] [ <b>-%I,
--search-index</b> ] [ <b>-pN, --priority[=N]</b> ] [ <b>-S,
--stay-on-same-dir</b> ] [ <b>-D, --can-go-down</b> ] [
<b>-U, --can-go-up</b> ] [ <b>-B, --can-go-up-and-down</b> ]
[ <b>-a, --stay-on-same-address</b> ] [ <b>-d,
<b>-%r, --warc</b> ] [ <b>-%d, --changes</b> ] [ <b>-%n,
--do-not-recatch</b> ] [ <b>-%v, --display</b> ] [ <b>-Q,
--do-not-log</b> ] [ <b>-q, --quiet</b> ] [ <b>-z,
--extra-log</b> ] [ <b>-Z, --debug-log</b> ] [ <b>-v,
--verbose</b> ] [ <b>-f, --file-log</b> ] [ <b>-f2,
--single-log</b> ] [ <b>-I, --index</b> ] [ <b>-%i,
--build-top-index</b> ] [ <b>-%I, --search-index</b> ] [
<b>-pN, --priority[=N]</b> ] [ <b>-S, --stay-on-same-dir</b>
] [ <b>-D, --can-go-down</b> ] [ <b>-U, --can-go-up</b> ] [
<b>-B, --can-go-up-and-down</b> ] [ <b>-a,
--stay-on-same-address</b> ] [ <b>-d,
--stay-on-same-domain</b> ] [ <b>-l, --stay-on-same-tld</b>
] [ <b>-e, --go-everywhere</b> ] [ <b>-%H,
--debug-headers</b> ] [ <b>-%!,
@@ -1143,6 +1144,19 @@ past N bytes, --warc-cdx also writes a sorted CDXJ index,
<td width="4%">
<p>-%d</p></td>
<td width="5%"></td>
<td width="82%">
<p>write hts-changes.json listing what this crawl left new,
changed, unchanged and gone compared to the previous mirror
(--changes)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%n</p></td>
<td width="5%"></td>
<td width="82%">

View File

@@ -97,9 +97,8 @@ ${do:end-if}
</pre>
${LANG_G8} :
${do:output-mode:html-urlescaped}
<a href="file://${path}/${projname}/" target="_new">
${do:output-mode:}
${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
<a href="/website/index.html" target="_new">
${path}/${projname}
</a></li>
<ul>

View File

@@ -103,7 +103,7 @@ ${LANG_Q3}
<tr><td>
<table width="100%">
<tr><td align="left">
<input type="submit" value="${LANG_OK]"
<input type="submit" value="${LANG_OK}"
${do:output-mode:html-urlescaped}
onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
${do:output-mode:}

View File

@@ -114,6 +114,11 @@ ${LANG_WARCFILE}
>
<br><br>
<input type="checkbox" name="changes" ${checked:changes}
title='${html:LANG_CHANGESTIP}' onMouseOver="info('${html:LANG_CHANGESTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_CHANGES}
<br><br>
<input type="checkbox" name="norecatch" ${checked:norecatch}
title='${html:LANG_I5b}' onMouseOver="info('${html:LANG_I5b}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I34b}

View File

@@ -143,6 +143,7 @@ ${do:copy:StripQuery:stripquery}
${do:copy:StoreAllInCache:cache2}
${do:copy:Warc:warc}
${do:copy:WarcFile:warcfile}
${do:copy:Changes:changes}
${do:copy:LogType:logtype}
${do:copy:UseHTTPProxyForFTP:ftpprox}
${do:copy:ProxyType:proxytype}

View File

@@ -121,9 +121,9 @@ httrack \
--quiet \
--build-top-index \
${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
${urls}
${test:filelist:-%L "}${filelist}${test:filelist:"}
--path "${html:path}/${html:projname}"
${unquoted:urls}
${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
--path "${arg:path}/${arg:projname}"
\
${test:parseall:--near}
${test:link:--test}
@@ -131,7 +131,7 @@ httrack \
${test:htmlfirst::--priority=7}
\
${do:if-not-empty:BuildString}
--structure "${BuildString}"
--structure "${arg:BuildString}"
${do:end-if}
${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
\
@@ -150,29 +150,30 @@ ${do:end-if}
${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
${test:windebug:--debug-headers}
\
${test:connexion:--sockets=}${connexion}
${test:connexion:--sockets=}${unquoted:connexion}
${test:ka:--keep-alive}
${test:timeout:--timeout=}${timeout}
${test:timeout:--timeout=}${unquoted:timeout}
${test:remt:--host-control=1}
${test:retry:--retries=}${retry}
${test:rate:--min-rate=}${rate}
${test:retry:--retries=}${unquoted:retry}
${test:rate:--min-rate=}${unquoted:rate}
${test:rems:--host-control=2}
\
${test:depth:--depth=}${depth}
${test:depth2:--ext-depth=}${depth2}
${test:maxhtml:--max-files=,}${maxhtml}
${test:othermax:--max-files=}${othermax}
${test:sizemax:--max-files=}${sizemax}
${test:pausebytes:--max-pause=}${pausebytes}
${test:maxtime:--max-time=}${maxtime}
${test:maxrate:--max-rate=}${maxrate}
${test:maxconn:--connection-per-second=}${maxconn}
${test:maxlinks:--advanced-maxlinks=}${maxlinks}
${test:depth:--depth=}${unquoted:depth}
${test:depth2:--ext-depth=}${unquoted:depth2}
${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
${test:othermax:--max-files=}${unquoted:othermax}
${test:maxhtml:--max-files=,}${unquoted:maxhtml}
${test:sizemax:--max-size=}${unquoted:sizemax}
${test:pausebytes:--max-pause=}${unquoted:pausebytes}
${test:maxtime:--max-time=}${unquoted:maxtime}
${test:maxrate:--max-rate=}${unquoted:maxrate}
${test:maxconn:--connection-per-second=}${unquoted:maxconn}
${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
\
--user-agent "${html:user}"
--footer "${html:footer}"
--user-agent "${arg:user}"
--footer "${arg:footer}"
\
${url2}
${unquoted:url2}
\
${test:cookies:--cookies=0:}
${test:parsejava:--parse-java=0:}
@@ -181,20 +182,21 @@ ${do:end-if}
${test:keepwww:--keep-www-prefix}
${test:keepslashes:--keep-double-slashes}
${test:keepqueryorder:--keep-query-order}
${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
${test:toler:--tolerant}
${test:http10:--http-10}
${test:cache2:--store-all-in-cache}
${test:warc:--warc}
${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
${test:changes:--changes}
${test:norecatch:--do-not-recatch}
${test:logf:--single-log}
${test:logtype:::--extra-log:--debug-log}
${test:index:--index=0:}
${test:index2:--search-index=0:--search-index}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
</textarea>
@@ -213,7 +215,7 @@ ParseAll=${ztest:parseall:0:1}
HTMLFirst=${ztest:htmlfirst:0:1}
Cache=${ztest:cache:0:1}
NoRecatch=${ztest:norecatch:0:1}
Dos=${dos
Dos=${dos}
Index=${ztest:index:0:1}
WordIndex=${ztest:index2:0:1}
Log=${ztest:logf:0:1:2}
@@ -241,6 +243,7 @@ StripQuery=${stripquery}
StoreAllInCache=${ztest:cache2:0:1}
Warc=${ztest:warc:0:1}
WarcFile=${warcfile}
Changes=${ztest:changes:0:1}
LogType=${logtype}
UseHTTPProxyForFTP=${ztest:ftpprox:0:1}
ProxyType=${proxytype}

View File

@@ -1042,3 +1042,7 @@ LANG_WARCFILE
WARC archive name:
LANG_WARCFILETIP
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
LANG_CHANGES
Report what changed since the previous mirror
LANG_CHANGESTIP
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Èìå íà WARC àðõèâà:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Íåçàäúëæèòåëíî áàçîâî èìå çà WARC àðõèâà; îñòàâåòå ïðàçíî çà àâòîìàòè÷íî èìåíóâàíå â èçõîäíàòà äèðåêòîðèÿ.
Report what changed since the previous mirror
Îò÷åò çà ïðîìåíèòå ñïðÿìî ïðåäèøíîòî îãëåäàëî
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Çàïèñâàíå è íà hts-changes.json ñúñ ñïèñúê íà íîâèòå, ïðîìåíåíèòå, íåïðîìåíåíèòå è èç÷åçíàëèòå ôàéëîâå ñïðÿìî ïðåäèøíîòî îãëåäàëî.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Nombre del archivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida.
Report what changed since the previous mirror
Informar de los cambios desde la copia anterior
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Escribir también hts-changes.json con la lista de lo que esta captura deja como nuevo, modificado, sin cambios o desaparecido respecto a la copia anterior.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Název archivu WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Volitelný základní název archivu WARC; ponechte prázdné pro automatické pojmenování ve výstupním adresáøi.
Report what changed since the previous mirror
Hlásit, co se od pøedchozího zrcadlení zmìnilo
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Zapsat také hts-changes.json se seznamem toho, co je oproti pøedchozímu zrcadlení nové, zmìnìné, nezmìnìné nebo chybìjící.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC 封存檔名稱:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC 封存檔的選用基本名稱;留空則於輸出目錄中自動命名。
Report what changed since the previous mirror
回報自上次鏡射以來的變更
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
同時寫入 hts-changes.json列出這次擷取相對於上次鏡射的新增、變更、未變更與消失的項目。

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC 归档名称:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC 归档的可选基本名称;留空则在输出目录中自动命名。
Report what changed since the previous mirror
报告自上次镜像以来的变更
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
同时写入 hts-changes.json列出本次抓取相对于上次镜像的新增、更改、未更改和消失的项目。

View File

@@ -966,3 +966,7 @@ WARC archive name:
Naziv WARC arhive:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju.
Report what changed since the previous mirror
Prijavi ¹to se promijenilo od prethodnog zrcaljenja
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Zapi¹i i hts-changes.json s popisom onoga ¹to je u odnosu na prethodno zrcaljenje novo, promijenjeno, nepromijenjeno ili nestalo.

View File

@@ -1012,3 +1012,7 @@ WARC archive name:
Navn på WARC-arkiv:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen.
Report what changed since the previous mirror
Rapportér hvad der er ændret siden den forrige spejling
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Skriv også hts-changes.json med en liste over, hvad denne gennemgang efterlader som nyt, ændret, uændret eller forsvundet i forhold til den forrige spejling.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Name des WARC-Archivs:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen.
Report what changed since the previous mirror
Melden, was sich seit der vorherigen Spiegelung geändert hat
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Zusätzlich hts-changes.json schreiben, das auflistet, was dieser Durchlauf gegenüber der vorherigen Spiegelung als neu, geändert, unverändert oder verschwunden hinterlässt.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC-arhiivi nimi:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC-arhiivi valikuline põhinimi; jäta tühjaks, et see väljundkataloogis automaatselt nimetada.
Report what changed since the previous mirror
Teata, mis on eelmisest peegeldusest saadik muutunud
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Kirjuta ka hts-changes.json, mis loetleb, mis on võrreldes eelmise peegeldusega uus, muutunud, muutumatu või kadunud.

View File

@@ -1012,3 +1012,7 @@ WARC archive name:
WARC archive name:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Report what changed since the previous mirror
Report what changed since the previous mirror
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.

View File

@@ -966,3 +966,7 @@ WARC archive name:
WARC-arkiston nimi:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valinnainen WARC-arkiston perusnimi; jätä tyhjäksi, jotta se nimetään automaattisesti tulostehakemistoon.
Report what changed since the previous mirror
Raportoi, mikä on muuttunut edellisen peilauksen jälkeen
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Kirjoita myös hts-changes.json, joka luettelee, mikä on edelliseen peilaukseen verrattuna uutta, muuttunutta, muuttumatonta tai kadonnutta.

View File

@@ -1012,3 +1012,7 @@ WARC archive name:
Nom de l'archive WARC :
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.
Report what changed since the previous mirror
Signaler ce qui a changé depuis le miroir précédent
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Écrire aussi hts-changes.json, qui liste ce que ce crawl laisse nouveau, modifié, inchangé ou disparu par rapport au miroir précédent.

View File

@@ -966,3 +966,7 @@ WARC archive name:
Όνομα αρχείου WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Προαιρετικό βασικό όνομα για το αρχείο WARC. Αφήστε το κενό για αυτόματη ονομασία στον κατάλογο εξόδου.
Report what changed since the previous mirror
Αναφορά των αλλαγών από το προηγούμενο αντίγραφο
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Εγγραφή και του hts-changes.json, που παραθέτει τι είναι νέο, τι άλλαξε, τι έμεινε ίδιο και τι χάθηκε σε σχέση με το προηγούμενο αντίγραφο.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Nome dell'archivio WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nome di base facoltativo per l'archivio WARC; lascia vuoto per assegnarlo automaticamente nella directory di output.
Report what changed since the previous mirror
Segnala che cosa è cambiato dalla copia precedente
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Scrive anche hts-changes.json, che elenca ciò che questa scansione lascia come nuovo, modificato, invariato o scomparso rispetto alla copia precedente.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC アーカイブ名:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC アーカイブの任意のベース名。空欄にすると出力ディレクトリ内で自動的に名前が付けられます。
Report what changed since the previous mirror
前回のミラーからの変更点を報告する
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
hts-changes.json も書き出し、前回のミラーと比べて新規、変更、変更なし、消滅となったものを一覧にします。

View File

@@ -964,3 +964,7 @@ WARC archive name:
¸ÜÕ ÝÐ WARC ÐàåØÒÐâÐ:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
¸×ÑÞàÝÞ ÞáÝÞÒÝÞ ØÜÕ ×Ð WARC ÐàåØÒÐâÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÐÒâÞÜÐâáÚÞ ØÜÕÝãÒÐúÕ ÒÞ Ø×ÛÕ×ÝØÞâ ÔØàÕÚâÞàØãÜ.
Report what changed since the previous mirror
¸×ÒÕáâØ èâÞ Õ ßàÞÜÕÝÕâÞ ÞÔ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
·ÐßØèØ Ø hts-changes.json áÞ áߨáÞÚ ÝÐ âÞÐ èâÞ Õ ÝÞÒÞ, ßàÞÜÕÝÕâÞ, ÝÕßàÞÜÕÝÕâÞ ØÛØ ØáçÕ×ÝÐâÞ ÒÞ ÞÔÝÞá ÝÐ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC archívum neve:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
A WARC archívum opcionális alapneve; hagyja üresen az automatikus elnevezéshez a kimeneti könyvtárban.
Report what changed since the previous mirror
Jelentés arról, mi változott az elõzõ tükrözés óta
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
A hts-changes.json fájl írása is, amely felsorolja, mi új, mi változott, mi maradt változatlan és mi tûnt el az elõzõ tükrözéshez képest.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Naam van WARC-archief:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optionele basisnaam voor het WARC-archief; laat leeg om het automatisch een naam te geven in de uitvoermap.
Report what changed since the previous mirror
Rapporteren wat er sinds de vorige spiegeling is gewijzigd
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Ook hts-changes.json schrijven met een overzicht van wat deze doorloop nieuw, gewijzigd, ongewijzigd of verdwenen laat ten opzichte van de vorige spiegeling.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Navn på WARC-arkiv:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valgfritt basisnavn for WARC-arkivet; la feltet stå tomt for automatisk navngivning i utdatamappen.
Report what changed since the previous mirror
Rapporter hva som er endret siden forrige speiling
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Skriv også hts-changes.json som lister opp hva denne gjennomgangen etterlater som nytt, endret, uendret eller forsvunnet sammenlignet med forrige speiling.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Nazwa archiwum WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Opcjonalna nazwa bazowa archiwum WARC; pozostaw puste, aby nazwaæ je automatycznie w katalogu wyj¶ciowym.
Report what changed since the previous mirror
Zg³o¶, co zmieni³o siê od poprzedniej kopii
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Zapisz równie¿ plik hts-changes.json z list± tego, co w porównaniu z poprzedni± kopi± jest nowe, zmienione, niezmienione lub usuniête.

View File

@@ -1012,3 +1012,7 @@ WARC archive name:
Nome do arquivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nome base opcional para o arquivo WARC; deixe em branco para nomeá-lo automaticamente no diretório de saída.
Report what changed since the previous mirror
Relatar o que mudou desde o espelhamento anterior
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Gravar também hts-changes.json listando o que esta captura deixa como novo, alterado, inalterado ou removido em relação ao espelhamento anterior.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Nome do arquivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nome base opcional para o arquivo WARC; deixe em branco para o nomear automaticamente no diretório de saída.
Report what changed since the previous mirror
Comunicar o que mudou desde o espelho anterior
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Escrever também hts-changes.json, que lista o que esta recolha deixa como novo, alterado, inalterado ou desaparecido em relação ao espelho anterior.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Numele arhivei WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nume de baza optional pentru arhiva WARC; lasati gol pentru a-l denumi automat in directorul de iesire.
Report what changed since the previous mirror
Raporteaza ce s-a schimbat fata de copia anterioara
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Scrie si hts-changes.json, care listeaza ce este nou, modificat, nemodificat sau disparut fata de copia anterioara.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Èìÿ WARC-àðõèâà:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Íåîáÿçàòåëüíîå áàçîâîå èìÿ WARC-àðõèâà; îñòàâüòå ïóñòûì äëÿ àâòîìàòè÷åñêîãî èìåíîâàíèÿ â âûõîäíîì êàòàëîãå.
Report what changed since the previous mirror
Ñîîáùàòü, ÷òî èçìåíèëîñü ñ ïðåäûäóùåãî çåðêàëà
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Òàêæå çàïèñûâàòü hts-changes.json ñî ñïèñêîì òîãî, ÷òî ïî ñðàâíåíèþ ñ ïðåäûäóùèì çåðêàëîì ñòàëî íîâûì, èçìåí¸ííûì, íåèçìåí¸ííûì èëè èñ÷åçëî.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Názov archívu WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Voliteµný základný názov archívu WARC; ponechajte prázdne pre automatické pomenovanie vo výstupnom adresári.
Report what changed since the previous mirror
Oznámi», èo sa od predchádzajúceho zrkadlenia zmenilo
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Zapísa» aj hts-changes.json so zoznamom toho, èo je oproti predchádzajúcemu zrkadleniu nové, zmenené, nezmenené alebo chýbajúce.

View File

@@ -964,3 +964,7 @@ WARC archive name:
Ime arhiva WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Neobvezno osnovno ime arhiva WARC; pustite prazno za samodejno poimenovanje v izhodni mapi.
Report what changed since the previous mirror
Porocaj, kaj se je spremenilo od prejsnjega zrcaljenja
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Zapisi tudi hts-changes.json s seznamom tega, kar je v primerjavi s prejsnjim zrcaljenjem novo, spremenjeno, nespremenjeno ali izginilo.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC-arkivets namn:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valfritt basnamn för WARC-arkivet; lämna tomt för att namnge det automatiskt i utdatakatalogen.
Report what changed since the previous mirror
Rapportera vad som ändrats sedan föregående spegling
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Skriv även hts-changes.json som listar vad den här genomgången lämnar som nytt, ändrat, oförändrat eller försvunnet jämfört med föregående spegling.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC arþivi adý:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC arþivi için isteðe baðlý temel ad; çýktý dizininde otomatik adlandýrma için boþ býrakýn.
Report what changed since the previous mirror
Önceki yansýmadan bu yana deðiþenleri bildir
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Ayrýca hts-changes.json yazarak bu taramanýn önceki yansýmaya göre neyi yeni, deðiþmiþ, deðiþmemiþ veya kaybolmuþ býraktýðýný listele.

View File

@@ -964,3 +964,7 @@ WARC archive name:
²ì'ÿ WARC-àðõ³âó:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Íåîáîâ'ÿçêîâà áàçîâà íàçâà WARC-àðõ³âó; çàëèøòå ïîðîæí³ì äëÿ àâòîìàòè÷íîãî íàéìåíóâàííÿ ó âèõ³äíîìó êàòàëîç³.
Report what changed since the previous mirror
Ïîâ³äîìëÿòè, ùî çì³íèëîñÿ ç ïîïåðåäíüîãî äçåðêàëà
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
Òàêîæ çàïèñóâàòè hts-changes.json ç³ ñïèñêîì òîãî, ùî ïîð³âíÿíî ç ïîïåðåäí³ì äçåðêàëîì º íîâèì, çì³íåíèì, íåçì³íåíèì àáî çíèêëèì.

View File

@@ -964,3 +964,7 @@ WARC archive name:
WARC arxivi nomi:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC arxivi uchun ixtiyoriy asosiy nom; chiqish katalogida avtomatik nomlash uchun bo'sh qoldiring.
Report what changed since the previous mirror
Oldingi nusxadan beri nima ozgarganini xabar qilish
Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
hts-changes.json ham yoziladi: unda ushbu yigish oldingi nusxaga nisbatan nimani yangi, ozgargan, ozgarmagan yoki yoqolgan holda qoldirgani royxati boladi.

View File

@@ -3,7 +3,7 @@
.\"
.\" This file is generated by man/makeman.sh; do not edit by hand.
.\" SPDX-License-Identifier: GPL-3.0-or-later
.TH httrack 1 "23 July 2026" "httrack website copier"
.TH httrack 1 "26 July 2026" "httrack website copier"
.SH NAME
httrack \- offline browser : copy websites to a local directory
.SH SYNOPSIS
@@ -75,6 +75,7 @@ httrack \- offline browser : copy websites to a local directory
[ \fB\-C, \-\-cache[=N]\fR ]
[ \fB\-k, \-\-store\-all\-in\-cache\fR ]
[ \fB\-%r, \-\-warc\fR ]
[ \fB\-%d, \-\-changes\fR ]
[ \fB\-%n, \-\-do\-not\-recatch\fR ]
[ \fB\-%v, \-\-display\fR ]
[ \fB\-Q, \-\-do\-not\-log\fR ]
@@ -279,6 +280,8 @@ create/use a cache for updates and retries (C0 no cache,C1 cache is prioritary,*
store all files in cache (not useful if files on disk) (\-\-store\-all\-in\-cache)
.IP \-%r
write an ISO\-28500 WARC/1.1 archive; \-\-warc\-file NAME sets the output name, \-\-warc\-max\-size N rotates segments past N bytes, \-\-warc\-cdx also writes a sorted CDXJ index, \-\-wacz packages it all as a WACZ file (\-\-warc)
.IP \-%d
write hts\-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror (\-\-changes)
.IP \-%n
do not re\-download locally erased files (\-\-do\-not\-recatch)
.IP \-%v

View File

@@ -32,7 +32,9 @@ AM_LDFLAGS = \
bin_PROGRAMS = proxytrack httrack htsserver
httrack_LDADD = $(THREADS_LIBS) libhttrack.la
httrack_SOURCES = httrack.c htsbacktrace.c htsbacktrace.h
# $(DL_LIBS): dladdr() in the crash handler, still in libdl on pre-2.34 glibc.
httrack_LDADD = $(THREADS_LIBS) $(DL_LIBS) libhttrack.la
htsserver_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS) libhttrack.la
proxytrack_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS)
@@ -47,6 +49,7 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
lib_LTLIBRARIES = libhttrack.la
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
htscmdline.c htscmdline.h \
htsurlport.c htsurlport.h
proxytrack_SOURCES = proxy/main.c \
proxy/proxytrack.c proxy/store.c \
@@ -60,21 +63,21 @@ whttrackrun_SCRIPTS = webhttrack
libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htscache_selftest.c htsdns_selftest.c htsselftest.c \
htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
htshelp.c htslib.c htsurlport.c htscoremain.c \
htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
htsname.c htsrobots.c htstools.c htswizard.c \
htsalias.c htsthread.c htsindex.c htsbauth.c \
htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
htsmd5.c htscodec.c htswarc.c htschanges.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
md5.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h \
htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
htsmodules.h htsname.h htsnet.h htssniff.h \
htsopt.h htsrobots.h htsthread.h \
htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h \
htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htschanges.h htsproxy.h htszlib.h \
htsstrings.h htsarrays.h httrack-library.h \
htscharset.h punycode.h htsencoding.h \
htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \

View File

@@ -114,6 +114,8 @@ const char *hts_optalias[][4] = {
"strip [host/pattern=]key1,key2,... from URLs"},
{"cookies-file", "-%K", "param1",
"load extra cookies from a Netscape cookies.txt"},
{"changes", "-%d", "single",
"write hts-changes.json: what this crawl changed vs. the previous mirror"},
{"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
{"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
{"warc-max-size", "-%rs", "param1",

View File

@@ -38,6 +38,7 @@ Please visit our Website: http://www.httrack.com
#include "htsnet.h"
#include "htscore.h"
#include "htswarc.h"
#include "htschanges.h"
#include "htsthread.h"
#include <time.h>
/* END specific definitions */
@@ -125,6 +126,22 @@ void back_free(struct_back ** sback) {
above a normal handshake. The last candidate still gets the full timeout. */
#define HTS_CONNECT_FALLBACK_TIMEOUT 10
void back_read_ftp_result(FILE *fp, htsblk *r) {
size_t j = 0;
if (fscanf(fp, "%d ", &r->statuscode) != 1)
r->statuscode = STATUSCODE_INVALID;
// an external helper writes this file: stop at capacity, not at EOF
while (j + 1 < sizeof(r->msg)) {
const int c = fgetc(fp);
if (c == EOF)
break;
r->msg[j++] = (char) c;
}
r->msg[j] = '\0';
}
int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
int timeout) {
int deadline;
@@ -541,9 +558,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
const char *ext) {
// do not use tempnam() but a regular filename
back->tmpfile_buffer[0] = '\0';
if (back->url_sav != NULL && back->url_sav[0] != '\0') {
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
back->url_sav, ext);
if (back->url_sav[0] != '\0') {
/* same capacity as url_sav, so truncation drops the extension and aliases
the temp name onto the live file that back_finalize_backup() UNLINKs */
if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
back->url_sav);
back->tmpfile_buffer[0] = '\0';
return -1;
}
back->tmpfile = back->tmpfile_buffer;
if (structcheck(back->tmpfile) != 0) {
hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
@@ -551,8 +574,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
return -1;
}
} else {
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
/* truncation here would collide distinct tmpnameid's onto one name */
if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
ext)) {
hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
StringBuff(opt->path_html_utf8));
back->tmpfile_buffer[0] = '\0';
return -1;
}
back->tmpfile = back->tmpfile_buffer;
}
/* OK */
@@ -699,6 +729,13 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
if ((size = hts_codec_unpack(codec, back[p].tmpfile,
unpacked)) >= 0) {
back[p].r.size = back[p].r.totalsize = size;
if (back[p].r.is_write) {
/* Sample the previous copy now: the rename below replaces
it, and file_notify() only fires once it is gone. */
hts_changes_notify(
opt, back[p].url_adr, back[p].url_fil, back[p].url_sav,
HTS_TRUE, back[p].r.notmodified ? HTS_TRUE : HTS_FALSE);
}
if (!back[p].r.is_write) {
// fichier -> mémoire ; le fichier est écrit plus tard
deleteaddr(&back[p].r);
@@ -887,8 +924,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
HTS_STAT.stat_bytes += back[p].r.size;
HTS_STAT.stat_files++;
hts_log_print(opt, LOG_TRACE, "added file %s%s => %s",
back[p].url_adr, back[p].url_fil,
back[p].url_sav != NULL ? back[p].url_sav : "");
back[p].url_adr, back[p].url_fil, back[p].url_sav);
}
if ((!back[p].r.notmodified) && (opt->is_update)) {
HTS_STAT.stat_updated_files++; // page modifiée
@@ -2302,26 +2338,24 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
&& slot_can_be_finalized(opt, &back[i]);
int may_serialize = slot_can_be_cached_on_disk(&back[i]);
hts_log_print(opt, LOG_DEBUG,
"back[%03d]: may_clean=%d, may_finalize_disk=%d, may_serialize=%d:"
LF "\t"
"finalized(%d), status(%d), locked(%d), delayed(%d), test(%d), "
LF "\t"
"statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), "
LF "\t" "contenttype(%s), url(%s%s), save(%s)", i,
may_clean, may_finalize, may_serialize,
back[i].finalized, back[i].status, back[i].locked,
IS_DELAYED_EXT(back[i].url_sav), back[i].testmode,
back[i].r.statuscode, (int) back[i].r.size,
back[i].r.is_write, may_be_hypertext_mime(opt,
back[i].r.
contenttype,
back[i].
url_fil),
/* */
back[i].r.contenttype, back[i].url_adr,
back[i].url_fil,
back[i].url_sav ? back[i].url_sav : "<null>");
hts_log_print(
opt, LOG_DEBUG,
"back[%03d]: may_clean=%d, may_finalize_disk=%d, "
"may_serialize=%d:" LF "\t"
"finalized(%d), status(%d), locked(%d), delayed(%d), "
"test(%d), " LF "\t"
"statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), " LF
"\t"
"contenttype(%s), url(%s%s), save(%s)",
i, may_clean, may_finalize, may_serialize, back[i].finalized,
back[i].status, back[i].locked, IS_DELAYED_EXT(back[i].url_sav),
back[i].testmode, back[i].r.statuscode, (int) back[i].r.size,
back[i].r.is_write,
may_be_hypertext_mime(opt, back[i].r.contenttype,
back[i].url_fil),
/* */
back[i].r.contenttype, back[i].url_adr, back[i].url_fil,
back[i].url_sav);
}
}
}
@@ -2857,7 +2891,8 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
// new session
back[i].r.ssl_con = SSL_new(openssl_ctx);
if (back[i].r.ssl_con) {
const char* hostname = jump_protocol_const(back[i].url_adr);
/* non-const twin: the OpenSSL macro casts the qualifier away */
char *hostname = jump_protocol(back[i].url_adr);
// some servers expect the hostname on the clienthello (SNI TLS extension)
SSL_set_tlsext_host_name(back[i].r.ssl_con, hostname);
SSL_clear(back[i].r.ssl_con);
@@ -2949,7 +2984,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
back[i].r.msg[0] = '\0';
strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
if (!strnotempty(back[i].r.msg)) {
sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
}
deletehttp(&back[i].r);
back[i].r.soc = INVALID_SOCKET;
@@ -3022,16 +3057,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
"rb");
if (fp) {
int j = 0;
fscanf(fp, "%d ", &(back[i].r.statuscode));
while(!feof(fp)) {
int c = fgetc(fp);
if (c != EOF)
back[i].r.msg[j++] = c;
}
back[i].r.msg[j++] = '\0';
back_read_ftp_result(fp, &back[i].r);
fclose(fp);
UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
strcpybuff(fconcat
@@ -3332,10 +3358,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
deleteaddr(&back[i].r);
if (back[i].r.size < back[i].r.totalsize)
back[i].r.statuscode = STATUSCODE_CONNERROR; // recatch
sprintf(back[i].r.msg,
"Incorrect length (" LLintP " Bytes, " LLintP
" expected)", (LLint) back[i].r.size,
(LLint) back[i].r.totalsize);
htsblk_failf(&back[i].r,
"Incorrect length (" LLintP " Bytes, " LLintP
" expected)",
(LLint) back[i].r.size,
(LLint) back[i].r.totalsize);
} else {
// Un warning suffira..
hts_log_print(opt, LOG_WARNING,

View File

@@ -74,6 +74,10 @@ void back_free(struct_back ** sback);
// backing
#define BACK_ADD_TEST "(dummy)"
#define BACK_ADD_TEST2 "(dummy2)"
/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
clipping the message to r->msg. */
void back_read_ftp_result(FILE *fp, htsblk *r);
int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
const char *sav);
int back_available(const struct_back * sback);

238
src/htsbacktrace.c Normal file
View File

@@ -0,0 +1,238 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: crash backtrace printer */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
/* Before every header: glibc gates dladdr() on it. */
#if defined(__linux) && !defined(_GNU_SOURCE)
#define _GNU_SOURCE
#endif
#include "htsbacktrace.h"
#include "htsglobal.h"
#include <stdlib.h>
#include <string.h>
#ifdef _WIN32
#include <io.h> /* write */
#endif
#ifdef HAVE_UNISTD_H
#include <unistd.h>
#endif
#if (defined(__linux) && defined(HAVE_EXECINFO_H))
#include <dlfcn.h>
#include <errno.h>
#include <execinfo.h>
#include <signal.h>
#include <time.h>
#include <sys/wait.h>
#define USES_BACKTRACE
#endif
#ifdef USES_BACKTRACE
#define BT_MAX_FRAMES 64 /* frames we try to name */
#define BT_MAX_MODULES 8 /* distinct modules, one child each */
#define BT_HEX_SIZE 19 /* "0x" + 16 nibbles + NUL */
#define BT_PATH_SIZE 1024 /* module path; longer is skipped */
#define BT_WAIT_TICKS 300 /* 10ms ticks, shared: cap a slow child */
#define BT_NO_SYMBOLIZER 127 /* child exit: execvp() found none */
static hts_boolean symbolize_crash = HTS_TRUE;
/* "0x"-prefixed hex: the handler must stay stdio-free. */
static void print_hex(char *buffer, uintptr_t value) {
static const char digits[] = "0123456789abcdef";
size_t i = 2, a, b;
buffer[0] = '0';
buffer[1] = 'x';
do {
buffer[i++] = digits[value & 0xf];
value >>= 4;
} while (value != 0);
buffer[i] = '\0';
for (a = 2, b = i - 1; a < b; a++, b--) {
const char c = buffer[a];
buffer[a] = buffer[b];
buffer[b] = c;
}
}
/* HTS_FALSE if src does not fit: a truncated module path would point the
symbolizer at the wrong file. */
static hts_boolean copy_bounded(char *dest, size_t size, const char *src) {
size_t i;
for (i = 0; i < size - 1 && src[i] != '\0'; i++) {
dest[i] = src[i];
}
dest[i] = '\0';
return src[i] == '\0' ? HTS_TRUE : HTS_FALSE;
}
/* Run the symbolizer on argv, output on fd, within *budget ticks. HTS_FALSE
only if none could be run at all; otherwise silent, the raw trace stands. */
static hts_boolean spawn_symbolizer(char **argv, int fd, int *budget) {
const pid_t pid = fork();
int status = 0;
if (pid == -1)
return HTS_FALSE;
if (pid == 0) {
static char llvm_prog[] = "llvm-symbolizer";
static char llvm_opts[] = "-p";
dup2(fd, 1); /* both symbolizers write on stdout */
execvp(argv[0], argv);
argv[0] = llvm_prog; /* an LLVM-only install ships no addr2line */
argv[1] = llvm_opts;
execvp(argv[0], argv);
_exit(BT_NO_SYMBOLIZER);
}
for (; *budget > 0; (*budget)--) {
const struct timespec tick = {0, 10 * 1000 * 1000};
const pid_t reaped = waitpid(pid, &status, WNOHANG);
if (reaped == pid)
return WIFEXITED(status) && WEXITSTATUS(status) == BT_NO_SYMBOLIZER
? HTS_FALSE
: HTS_TRUE;
if (reaped == -1 && errno != EINTR)
return HTS_TRUE;
nanosleep(&tick, NULL);
}
kill(pid, SIGKILL);
waitpid(pid, NULL, 0);
return HTS_TRUE;
}
/* Name the frames backtrace_symbols_fd() leaves as module+offset:
-fvisibility=hidden keeps them out of .dynsym, but DWARF has them. dladdr()
is not formally async-signal-safe; accepted, this path is already fatal. */
static void symbolize_backtrace(void *const *stack, int size, int fd) {
static char prog[] = "addr2line";
static char opts[] = "-Cfipa";
static char dashe[] = "-e";
char hex[BT_MAX_FRAMES][BT_HEX_SIZE];
const void *base[BT_MAX_FRAMES];
const char *name[BT_MAX_FRAMES];
hts_boolean grouped[BT_MAX_FRAMES];
char module[BT_PATH_SIZE];
char *argv[4 + BT_MAX_FRAMES + 1];
int budget = BT_WAIT_TICKS;
int i, spawned;
if (size > BT_MAX_FRAMES)
size = BT_MAX_FRAMES;
for (i = 0; i < size; i++) {
Dl_info info;
grouped[i] = HTS_TRUE; /* skipped unless dladdr() places the frame */
if (dladdr(stack[i], &info) == 0 || info.dli_fname == NULL ||
info.dli_fname[0] == '\0')
continue;
base[i] = info.dli_fbase;
name[i] = info.dli_fname;
print_hex(hex[i], (uintptr_t) ((const char *) stack[i] -
(const char *) info.dli_fbase));
grouped[i] = HTS_FALSE;
}
/* One child per module: addr2line takes a single -e. Each frame is claimed
once, so argc cannot exceed argv[]. */
for (spawned = 0; spawned < BT_MAX_MODULES; spawned++) {
int first, j, argc = 0;
for (first = 0; first < size && grouped[first]; first++)
;
if (first >= size)
break;
argv[argc++] = prog;
argv[argc++] = opts;
argv[argc++] = dashe;
argv[argc++] = module;
for (j = first; j < size; j++) {
if (grouped[j] || base[j] != base[first])
continue;
grouped[j] = HTS_TRUE;
argv[argc++] = hex[j];
}
argv[argc] = NULL;
/* access(): skip pseudo-modules like linux-vdso, which have no file and
would draw nothing but an addr2line complaint. */
if (copy_bounded(module, sizeof(module), name[first]) &&
access(module, R_OK) == 0) {
const size_t len = strlen(module);
/* addr2line -a prints offsets only: say which module they are in. */
(void) (write(fd, module, len) == (ssize_t) len);
(void) (write(fd, ":\n", 2) == 2);
if (!spawn_symbolizer(argv, fd, &budget))
break; /* no symbolizer: stop at one header */
}
}
}
#endif
void hts_backtrace_init(void) {
#ifdef USES_BACKTRACE
symbolize_crash =
getenv("HTTRACK_NO_SYMBOLIZE") == NULL ? HTS_TRUE : HTS_FALSE;
#endif
}
void hts_print_backtrace(int fd) {
#ifdef USES_BACKTRACE
void *stack[256];
const int size = backtrace(stack, sizeof(stack) / sizeof(stack[0]));
/* A fault inside the handler lands back here: symbolizing twice interleaves
two traces on fd and spends a second budget. */
static volatile sig_atomic_t entered = 0;
if (size != 0) {
backtrace_symbols_fd(stack, size, fd);
if (symbolize_crash && entered == 0) {
entered = 1;
symbolize_backtrace(stack, size, fd);
entered = 0;
}
}
#else
const char msg[] = "No stack trace available on this OS :(\n";
if (write(fd, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
/* sorry GCC */
}
#endif
}

45
src/htsbacktrace.h Normal file
View File

@@ -0,0 +1,45 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: crash backtrace printer */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTSBACKTRACE_DEFH
#define HTSBACKTRACE_DEFH
/* Sample HTTRACK_NO_SYMBOLIZE before any crash: getenv() is not signal-safe.
Call once, from the process that installs the fatal-signal handlers. */
void hts_backtrace_init(void);
/* Write the calling thread's stack to fd, callable from a fatal signal handler:
raw frames first, then whatever an external symbolizer can name. Allocates
nothing; prints a one-line notice where the OS has no backtrace(). */
void hts_print_backtrace(int fd);
#endif

View File

@@ -629,8 +629,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
// File exists on disk with declared cache name (this is expected!)
if (fexist_utf8(fconv(catbuff, sizeof(catbuff), previous_save))) { // un fichier existe déja
// Expected size ?
const size_t fsize =
fsize_utf8(fconv(catbuff, sizeof(catbuff), previous_save));
const LLint fsize = fsize_utf8(
fconv(catbuff, sizeof(catbuff), previous_save));
if (fsize == r.size) {
// Target name is the previous name, and the file looks good: nothing to do!
if (strcmp(previous_save, target_save) == 0) {
@@ -666,7 +666,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
// Suppose a broken mirror, with a file being renamed: OK
else if (fexist_utf8(fconv(catbuff, sizeof(catbuff), target_save))) {
// Expected size ?
const size_t fsize = fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
const LLint fsize =
fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
if (fsize == r.size) {
// So far so good
@@ -1440,7 +1441,7 @@ int cache_brstr(char *adr, char *s, size_t s_size) {
/* binput bounded to a NUL-terminated buffer: refuse to start a read at or
past `end`, so a prior over-advance can't walk a cache-index parse OOB. */
int cache_binput(char *adr, const char *end, char *s, int max) {
int cache_binput(const char *adr, const char *end, char *s, int max) {
if (adr >= end) {
s[0] = '\0';
return 0;

View File

@@ -93,7 +93,7 @@ void cache_rstr(FILE *fp, char *s, size_t s_size);
char *cache_rstr_addr(FILE * fp);
int cache_brstr(char *adr, char *s, size_t s_size);
/* binput over a NUL-terminated buffer, bounded: no read starts at/past end. */
int cache_binput(char *adr, const char *end, char *s, int max);
int cache_binput(const char *adr, const char *end, char *s, int max);
int cache_brint(char *adr, int *i);
void cache_rint(FILE * fp, int *i);
void cache_rLLint(FILE * fp, LLint * i);

618
src/htschanges.c Normal file
View File

@@ -0,0 +1,618 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: htschanges.c subroutines: */
/* --changes: what this crawl changed vs. the previous */
/* mirror (hts-changes.json) */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#define HTS_INTERNAL_BYTECODE
#include "htschanges.h"
#include "htscache.h"
#include "htscharset.h"
#include "htscore.h"
#include "htslib.h"
#include "htsmd5.h"
#include "htssafe.h"
#include "htsthread.h"
#include "htstools.h"
#include "coucal/coucal.h"
#include "md5.h"
#include <stdlib.h>
#include <string.h>
#include <time.h>
#define DIGEST_SIZE 16
typedef struct hts_changes hts_changes;
/* One mirrored file. The mirror-relative path is the key, not the URL: a
redirect and its target can share a save name, and counting that file twice
would put it in two buckets. */
typedef struct {
char *url; /* absolute URL, "" for an engine-generated file */
char *file; /* mirror-relative path, as listed in new.lst */
hts_boolean rewritten; /* the crawl wrote over the local copy */
hts_boolean not_updated; /* transfer signal: the server reported no change */
hts_boolean existed; /* the file was part of the previous mirror */
hts_boolean on_disk; /* a copy was on disk when the crawl first saw it */
hts_boolean listed_prev; /* the previous mirror's index listed this file */
hts_boolean has_prev; /* prev_digest holds the previous payload's digest */
hts_boolean has_new; /* new_digest was taken from the payload, not disk */
unsigned char prev_digest[DIGEST_SIZE];
unsigned char new_digest[DIGEST_SIZE];
LLint prev_size; /* -1 when unknown */
LLint size; /* size after the crawl, -1 when unknown */
hts_change_bucket bucket;
} changes_entry;
struct hts_changes {
coucal index; /* mirror-relative path -> entry slot + 1 */
changes_entry *entries;
size_t count;
size_t capacity;
hts_boolean old_index; /* the previous mirror's index was read */
hts_boolean overflow; /* an allocation failed; the report is partial */
};
/* ------------------------------------------------------------ */
/* JSON */
/* ------------------------------------------------------------ */
/* Length of the UTF-8 sequence led by c, or 0 if c cannot lead one. */
static size_t utf8_lead_length(unsigned char c) {
if (c >= 0xc2 && c <= 0xdf)
return 2;
if (c >= 0xe0 && c <= 0xef)
return 3;
if (c >= 0xf0 && c <= 0xf4)
return 4;
return 0;
}
void hts_changes_json_string(String *out, const char *s) {
const unsigned char *p = (const unsigned char *) s;
StringAddchar(*out, '"');
while (*p != '\0') {
const unsigned char c = *p;
if (c == '"' || c == '\\') {
StringAddchar(*out, '\\');
StringAddchar(*out, (char) c);
p++;
} else if (c < 0x20 || c == 0x7f) {
char esc[8];
snprintf(esc, sizeof(esc), "\\u%04x", (unsigned) c);
StringCat(*out, esc);
p++;
} else if (c < 0x80) {
StringAddchar(*out, (char) c);
p++;
} else {
/* Non-ASCII: emit it verbatim only if it is a well-formed sequence, so
a legacy-charset URL cannot produce unparseable JSON. */
const size_t len = utf8_lead_length(c);
if (len != 0 && strnlen((const char *) p, len) == len &&
hts_isStringUTF8((const char *) p, len)) {
StringMemcat(*out, (const char *) p, len);
p += len;
} else {
StringCat(*out, "\\ufffd");
p++;
}
}
}
StringAddchar(*out, '"');
}
/* ------------------------------------------------------------ */
/* Accumulator */
/* ------------------------------------------------------------ */
static hts_changes *changes_new(void) {
hts_changes *changes = calloct(1, sizeof(*changes));
if (changes == NULL)
return NULL;
changes->index = coucal_new(0);
if (changes->index == NULL) {
freet(changes);
return NULL;
}
return changes;
}
static void changes_free(hts_changes **pchanges) {
hts_changes *changes = *pchanges;
if (changes == NULL)
return;
if (changes->entries != NULL) {
size_t i;
for (i = 0; i < changes->count; i++) {
freet(changes->entries[i].url);
freet(changes->entries[i].file);
}
freet(changes->entries);
}
if (changes->index != NULL)
coucal_delete(&changes->index);
freet(changes);
*pchanges = NULL;
}
/* Slot of `file`, or -1 when it has not been recorded. */
static intptr_t changes_find(const hts_changes *changes, const char *file) {
intptr_t slot = 0;
return coucal_read(changes->index, file, &slot) ? slot - 1 : -1;
}
/* Slot of `file`, appending a fresh entry when it is new; -1 if allocation
failed, which flags the report partial. */
static intptr_t changes_slot(hts_changes *changes, const char *file,
hts_boolean *is_new) {
intptr_t slot = 0;
*is_new = HTS_FALSE;
if (coucal_read(changes->index, file, &slot))
return slot - 1;
if (changes->count == changes->capacity) {
const size_t capacity = changes->capacity != 0 ? changes->capacity * 2 : 64;
changes_entry *const entries =
realloct(changes->entries, capacity * sizeof(*entries));
if (entries == NULL) {
changes->overflow = HTS_TRUE;
return -1;
}
changes->entries = entries;
changes->capacity = capacity;
}
slot = (intptr_t) changes->count;
memset(&changes->entries[slot], 0, sizeof(changes->entries[slot]));
changes->entries[slot].file = strdupt(file);
changes->entries[slot].prev_size = -1;
changes->entries[slot].size = -1;
if (changes->entries[slot].file == NULL) {
changes->overflow = HTS_TRUE;
return -1;
}
changes->count++;
coucal_write(changes->index, file, slot + 1);
*is_new = HTS_TRUE;
return slot;
}
/* MD5 of the file at `path`; change detection over our own mirror, not a
security boundary. HTS_FALSE if it cannot be read. */
static hts_boolean digest_file(const char *path,
unsigned char digest[DIGEST_SIZE]) {
const int endian = 1;
struct MD5Context ctx;
char BIGSTK buffer[32768];
FILE *fp = FOPEN(path, "rb");
size_t nread;
if (fp == NULL)
return HTS_FALSE;
MD5Init(&ctx, *((const char *) &endian));
while ((nread = fread(buffer, 1, sizeof(buffer), fp)) > 0)
MD5Update(&ctx, (const unsigned char *) buffer, (unsigned int) nread);
if (ferror(fp) != 0) {
fclose(fp);
return HTS_FALSE;
}
fclose(fp);
MD5Final(digest, &ctx);
return HTS_TRUE;
}
static void digest_mem(const char *buffer, size_t len,
unsigned char digest[DIGEST_SIZE]) {
domd5mem(buffer, len, (char *) digest, 0);
}
hts_change_bucket hts_changes_classify(hts_boolean rewritten,
hts_boolean existed,
hts_boolean not_updated,
hts_boolean have_digests,
hts_boolean digests_equal) {
if (!rewritten)
return HTS_CHANGE_UNCHANGED; /* the crawl left the copy alone */
if (!existed)
return HTS_CHANGE_NEW;
if (have_digests)
return digests_equal ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED;
/* No digest to compare: a server with no validators answers 200 with the
same bytes, so this signal alone over-reports. */
return not_updated ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED;
}
/* ------------------------------------------------------------ */
/* Engine hooks */
/* ------------------------------------------------------------ */
/* FTP transfers reach file_notify() from their own detached thread
(htsftp.c:631/635), so every entry point below runs under this lock. */
static htsmutex changes_mutex = HTSMUTEX_INIT;
/* The live accumulator, created on first use; NULL when --changes is off. */
static hts_changes *changes_get(httrackp *opt) {
if (!opt->changes)
return NULL;
if (opt->changes_state == NULL)
opt->changes_state = changes_new();
return (hts_changes *) opt->changes_state;
}
void hts_changes_notify(httrackp *opt, const char *adr, const char *fil,
const char *save, hts_boolean rewritten,
hts_boolean not_updated) {
hts_changes *changes;
char BIGSTK file[HTS_URLMAXSIZE * 2];
hts_boolean is_new;
intptr_t slot;
changes_entry *entry;
/* Engine-generated scaffolding (the top index) carries no URL and is not a
mirrored resource. */
if (save == NULL || !strnotempty(save) || adr == NULL || fil == NULL ||
(!strnotempty(adr) && !strnotempty(fil)))
return;
hts_mutexlock(&changes_mutex);
changes = changes_get(opt);
if (changes == NULL) {
hts_mutexrelease(&changes_mutex);
return;
}
hts_savename_listed(&opt->state.strc, save, file, sizeof(file));
slot = changes_slot(changes, file, &is_new);
if (slot < 0) {
hts_mutexrelease(&changes_mutex);
return;
}
entry = &changes->entries[slot];
if (!is_new) {
/* Retry, or a second call site for the same file: the pre-run state was
already sampled and the copy on disk may no longer be it. */
entry->rewritten = entry->rewritten || rewritten;
hts_mutexrelease(&changes_mutex);
return;
}
{
char BIGSTK url[HTS_URLMAXSIZE * 4 + 8]; /* holds adr + fil + a scheme */
url[0] = '\0';
if (!link_has_authority(adr))
strlcatbuff(url, "http://", sizeof(url));
strlcatbuff(url, adr, sizeof(url));
strlcatbuff(url, fil, sizeof(url));
entry->url = strdupt(url);
if (entry->url == NULL)
changes->overflow = HTS_TRUE;
}
entry->rewritten = rewritten;
entry->not_updated = not_updated;
entry->prev_size = fsize_utf8(save);
entry->on_disk = entry->prev_size >= 0;
/* Hash the previous copy only when it is about to be overwritten: this is
the last moment those bytes exist. */
if (entry->on_disk && rewritten)
entry->has_prev = digest_file(save, entry->prev_digest);
hts_mutexrelease(&changes_mutex);
}
void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r,
const char *adr, const char *fil, const char *save) {
hts_changes *changes;
char BIGSTK file[HTS_URLMAXSIZE * 2];
char BIGSTK location[HTS_URLMAXSIZE * 2];
unsigned char new_digest[DIGEST_SIZE];
unsigned char prev_digest[DIGEST_SIZE];
hts_boolean has_prev = HTS_FALSE;
intptr_t slot;
htsblk prev;
if (r->adr == NULL || r->size < 0 || save == NULL || !strnotempty(save))
return;
/* On disk this is the payload plus rewritten links and a footer dated by the
crawl, so it differs every run; compare payloads, the previous one being
the body the cache kept. */
digest_mem(r->adr, (size_t) r->size, new_digest);
prev = cache_read_ro(opt, cache, adr, fil, "", location);
if (HTTP_IS_OK(prev.statuscode) && prev.adr != NULL && prev.size >= 0) {
digest_mem(prev.adr, (size_t) prev.size, prev_digest);
has_prev = HTS_TRUE;
}
freet(prev.adr);
/* Only now take the lock and look the entry up: cache_read_ro() can itself
reach file_notify(), which would move the entries array. */
hts_mutexlock(&changes_mutex);
changes = changes_get(opt);
/* file_notify() records the entry; this call only refines it. */
if (changes != NULL) {
hts_savename_listed(&opt->state.strc, save, file, sizeof(file));
slot = changes_find(changes, file);
if (slot >= 0) {
changes_entry *const entry = &changes->entries[slot];
entry->has_new = HTS_TRUE;
memcpy(entry->new_digest, new_digest, DIGEST_SIZE);
entry->has_prev = has_prev;
if (has_prev)
memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE);
}
}
hts_mutexrelease(&changes_mutex);
}
void hts_changes_gone(httrackp *opt, const char *file) {
hts_changes *changes;
hts_boolean is_new;
intptr_t slot;
if (file == NULL || !strnotempty(file))
return;
hts_mutexlock(&changes_mutex);
changes = changes_get(opt);
if (changes != NULL) {
slot = changes_slot(changes, file, &is_new);
if (slot >= 0 && is_new) {
changes->entries[slot].url = strdupt("");
changes->entries[slot].bucket = HTS_CHANGE_GONE;
changes->entries[slot].listed_prev = HTS_TRUE;
}
}
hts_mutexrelease(&changes_mutex);
}
void hts_changes_previous(httrackp *opt, const char *file) {
hts_changes *changes;
intptr_t slot;
if (file == NULL)
return;
hts_mutexlock(&changes_mutex);
changes = changes_get(opt);
if (changes != NULL) {
changes->old_index = HTS_TRUE;
slot = changes_find(changes, file);
if (slot >= 0)
changes->entries[slot].listed_prev = HTS_TRUE;
}
hts_mutexrelease(&changes_mutex);
}
/* ------------------------------------------------------------ */
/* Report */
/* ------------------------------------------------------------ */
/* Assign each entry its final bucket from the bytes now on disk. */
static void changes_resolve(hts_changes *changes, httrackp *opt) {
char catbuff[CATBUFF_SIZE];
size_t i;
for (i = 0; i < changes->count; i++) {
changes_entry *const entry = &changes->entries[i];
const char *path;
unsigned char digest[DIGEST_SIZE];
hts_boolean have_digests = HTS_FALSE;
hts_boolean digests_equal = HTS_FALSE;
if (entry->bucket == HTS_CHANGE_GONE)
continue;
/* The previous mirror's index is the authority on what was there before:
a partial left by this crawl's own failed attempt is on disk but was
never part of the previous mirror. Without an index, fall back to what
the first notify saw on disk. */
entry->existed = changes->old_index ? entry->listed_prev : entry->on_disk;
path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_html),
entry->file);
entry->size = fsize_utf8(path);
if (entry->rewritten && entry->existed) {
/* The size shortcut only holds when both digests describe the file on
disk; has_new means they are payload digests, and a parsed page's
rendered size moves with its links and footer. */
if (!entry->has_new && entry->size >= 0 && entry->prev_size >= 0 &&
entry->size != entry->prev_size) {
have_digests = HTS_TRUE; /* different lengths: no need to hash */
digests_equal = HTS_FALSE;
} else if (entry->has_prev &&
(entry->has_new
? (memcpy(digest, entry->new_digest, DIGEST_SIZE), 1)
: digest_file(path, digest))) {
have_digests = HTS_TRUE;
digests_equal = memcmp(digest, entry->prev_digest, DIGEST_SIZE) == 0
? HTS_TRUE
: HTS_FALSE;
}
}
entry->bucket =
hts_changes_classify(entry->rewritten, entry->existed,
entry->not_updated, have_digests, digests_equal);
}
}
static const char *const bucket_names[HTS_CHANGE_BUCKETS] = {
"new", "changed", "unchanged", "gone"};
/* Callers below run after the crawl's threads are done, so they take no lock.
*/
void hts_changes_report(httrackp *opt, String *out) {
hts_changes *const changes = (hts_changes *) opt->changes_state;
size_t counts[HTS_CHANGE_BUCKETS];
char date[32];
char scratch[64];
int bucket;
size_t i;
StringClear(*out);
if (changes == NULL)
return;
changes_resolve(changes, opt);
memset(counts, 0, sizeof(counts));
for (i = 0; i < changes->count; i++)
counts[changes->entries[i].bucket]++;
hts_now_iso8601(date);
StringCat(*out, "{\n \"schema\": ");
snprintf(scratch, sizeof(scratch), "%d", HTS_CHANGES_SCHEMA);
StringCat(*out, scratch);
StringCat(*out, ",\n \"generator\": ");
hts_changes_json_string(out, "HTTrack Website Copier/" HTTRACK_VERSION);
StringCat(*out, ",\n \"date\": ");
hts_changes_json_string(out, date);
StringCat(*out, ",\n \"first_crawl\": ");
StringCat(*out, changes->old_index ? "false" : "true");
StringCat(*out, ",\n \"partial\": ");
StringCat(*out, changes->overflow ? "true" : "false");
StringCat(*out, ",\n \"purged\": ");
StringCat(*out, opt->delete_old ? "true" : "false");
StringCat(*out, ",\n \"counts\": {");
for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) {
StringCat(*out, bucket != 0 ? ", " : " ");
hts_changes_json_string(out, bucket_names[bucket]);
snprintf(scratch, sizeof(scratch), ": %d", (int) counts[bucket]);
StringCat(*out, scratch);
}
StringCat(*out, " }");
for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) {
hts_boolean first = HTS_TRUE;
StringCat(*out, ",\n ");
hts_changes_json_string(out, bucket_names[bucket]);
StringCat(*out, ": [");
for (i = 0; i < changes->count; i++) {
const changes_entry *const entry = &changes->entries[i];
if (entry->bucket != bucket)
continue;
StringCat(*out, first ? "\n { \"url\": " : ",\n { \"url\": ");
first = HTS_FALSE;
hts_changes_json_string(out, entry->url != NULL ? entry->url : "");
StringCat(*out, ", \"file\": ");
hts_changes_json_string(out, entry->file);
if (entry->size >= 0) {
snprintf(scratch, sizeof(scratch), ", \"size\": " LLintP,
(LLint) entry->size);
StringCat(*out, scratch);
}
if (bucket == HTS_CHANGE_CHANGED && entry->prev_size >= 0) {
snprintf(scratch, sizeof(scratch), ", \"previous_size\": " LLintP,
(LLint) entry->prev_size);
StringCat(*out, scratch);
}
StringCat(*out, " }");
}
StringCat(*out, first ? "]" : "\n ]");
}
StringCat(*out, "\n}\n");
}
void hts_changes_close_opt(httrackp *opt) {
/* changes_get(), not the raw field: a crawl that mirrored nothing still owes
the user a report, and a stale one from the previous run must not survive
on disk as if it described this one. */
hts_changes *changes = changes_get(opt);
char catbuff[CATBUFF_SIZE];
const char *path;
String report = STRING_EMPTY;
size_t counts[HTS_CHANGE_BUCKETS];
size_t i;
FILE *fp;
if (changes == NULL)
return;
hts_changes_report(opt, &report);
memset(counts, 0, sizeof(counts));
for (i = 0; i < changes->count; i++)
counts[changes->entries[i].bucket]++;
path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_log),
HTS_CHANGES_FILE);
fp = FOPEN(path, "wb");
if (fp != NULL) {
const size_t len = StringLength(report);
if (len != 0 && fwrite(StringBuff(report), 1, len, fp) != len)
hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
"Unable to write the change report %s", path);
fclose(fp);
} else {
hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
"Unable to create the change report %s", path);
}
if (!changes->old_index) {
hts_log_print(opt, LOG_NOTICE,
"Change report: first crawl, %d files mirrored, nothing to "
"compare against (%s)",
(int) changes->count, HTS_CHANGES_FILE);
} else {
hts_log_print(opt, LOG_NOTICE,
"Change report: %d new, %d changed, %d unchanged, %d gone "
"(%s)",
(int) counts[HTS_CHANGE_NEW],
(int) counts[HTS_CHANGE_CHANGED],
(int) counts[HTS_CHANGE_UNCHANGED],
(int) counts[HTS_CHANGE_GONE], HTS_CHANGES_FILE);
}
StringFree(report);
changes_free(&changes);
opt->changes_state = NULL;
}
void hts_changes_free_opt(httrackp *opt) {
hts_changes *changes = (hts_changes *) opt->changes_state;
changes_free(&changes);
opt->changes_state = NULL;
}

126
src/htschanges.h Normal file
View File

@@ -0,0 +1,126 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* HTTrack change report (--changes). Internal, not installed.
Accumulates what the crawl did to each mirrored file, compares the bytes
against the copy the previous run left behind, and writes
hts-changes.json next to the log. */
/* ------------------------------------------------------------ */
#ifndef HTS_CHANGES_DEFH
#define HTS_CHANGES_DEFH
#include "htsopt.h"
#include "htsstrings.h"
#ifndef HTS_DEF_FWSTRUCT_cache_back
#define HTS_DEF_FWSTRUCT_cache_back
typedef struct cache_back cache_back;
#endif
#ifdef __cplusplus
extern "C" {
#endif
/* Report file name, written under the project's log directory. */
#define HTS_CHANGES_FILE "hts-changes.json"
/* Schema version carried by the report; bump on an incompatible change. */
#define HTS_CHANGES_SCHEMA 1
/* Which side of the comparison a mirrored file ended up on. */
typedef enum {
HTS_CHANGE_NEW = 0, /* no local copy before this run */
HTS_CHANGE_CHANGED, /* rewritten, and the bytes differ */
HTS_CHANGE_UNCHANGED, /* the bytes are those of the previous mirror */
HTS_CHANGE_GONE, /* in the previous mirror, absent from this one */
HTS_CHANGE_BUCKETS
} hts_change_bucket;
/* Record what this crawl is doing to the local file `save` (an absolute path;
adr/fil form the URL it comes from, either may be empty for an engine
generated file). `rewritten` means the copy on disk is being written over,
`not_updated` is the 200-versus-304 transfer signal, used only when no
digest can be taken. Only the first call for a given file samples the
previous copy, so a retried or twice-notified resource is counted once.
No-op unless --changes is on. */
void hts_changes_notify(httrackp *opt, const char *adr, const char *fil,
const char *save, hts_boolean rewritten,
hts_boolean not_updated);
/* Refine the entry for a parsed HTML file, whose local copy is the payload
after link rewriting plus a footer carrying the crawl date: its bytes differ
on every run, so the comparison uses the payload `r` holds against the body
the previous run left in the cache. Called once the page is ready to be
written, from hts_finish_html_file(). */
void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r,
const char *adr, const char *fil, const char *save);
/* Record `file` (mirror-relative, as listed in new.lst) as present in the
previous mirror and absent from this one. Whether it is also deleted from
disk is the purge option's business, not this one's. */
void hts_changes_gone(httrackp *opt, const char *file);
/* Record that the previous mirror's index listed `file`. That index, not the
file's presence on disk, decides what counts as already mirrored: a partial
left by this crawl's own failed attempt is on disk but was never part of the
previous mirror. Calling it at all marks the run as having one. */
void hts_changes_previous(httrackp *opt, const char *file);
/* Resolve every entry against the bytes now on disk and serialize the report
into `out` (replaced). Exposed for the `changes` self-test; the engine goes
through hts_changes_close_opt(). */
void hts_changes_report(httrackp *opt, String *out);
/* Write the report and log a one-line summary, then free the accumulator.
Null-safe and idempotent. */
void hts_changes_close_opt(httrackp *opt);
/* Drop the accumulator without writing anything, for a run that never reached
its end. Null-safe and idempotent. */
void hts_changes_free_opt(httrackp *opt);
/* Bucket for one entry, from what was observed. No filesystem access:
`have_digests` says both sides could be hashed, `digests_equal` compares
them, and `not_updated` is the fallback signal when they could not. */
hts_change_bucket hts_changes_classify(hts_boolean rewritten,
hts_boolean existed,
hts_boolean not_updated,
hts_boolean have_digests,
hts_boolean digests_equal);
/* Append `s` to `out` as a quoted JSON string. Byte sequences that are not
valid UTF-8 become U+FFFD, so a mirror carrying legacy-charset URLs still
produces parseable JSON. */
void hts_changes_json_string(String *out, const char *s);
#ifdef __cplusplus
}
#endif
#endif

95
src/htscmdline.c Normal file
View File

@@ -0,0 +1,95 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: command line splitter, shared by the engine and */
/* htsserver */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#include "htscmdline.h"
#include "htssafe.h"
#include <limits.h>
#include <stdint.h>
char **hts_split_cmdline(char *cmd, int *nargs) {
size_t nsep = 0;
size_t capacity;
size_t r;
size_t w;
int argc = 0;
hts_boolean quoted = HTS_FALSE;
char **argv;
*nargs = 0;
/* fold the other separators, so counting them sizes the vector exactly */
for (r = 0; cmd[r] != '\0'; r++) {
if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
cmd[r] = ' ';
}
if (cmd[r] == ' ') {
nsep++;
}
}
/* at most one argument per separator, plus the leading one and the NULL */
if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
return NULL;
}
capacity = nsep + 2;
argv = (char **) malloct(capacity * sizeof(char *));
if (argv == NULL) {
return NULL;
}
argv[argc++] = cmd;
for (r = 0, w = 0; cmd[r] != '\0';) {
if (quoted && cmd[r] == '\\' &&
(cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
r++;
cmd[w++] = cmd[r++];
} else if (cmd[r] == '\"') {
quoted = !quoted;
cmd[w++] = cmd[r++];
} else if (cmd[r] == ' ' && !quoted) {
cmd[w++] = '\0';
assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
argv[argc++] = cmd + w;
r++;
} else {
cmd[w++] = cmd[r++];
}
}
cmd[w] = '\0';
argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
*nargs = argc;
return argv;
}

45
src/htscmdline.h Normal file
View File

@@ -0,0 +1,45 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: command line splitter, shared by the engine and */
/* htsserver */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTSCMDLINE_DEFH
#define HTSCMDLINE_DEFH
#include "htsglobal.h"
/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
argv[0] being the program name and quotes left for the engine to strip.
Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
entries), or NULL when it cannot be sized or allocated. */
char **hts_split_cmdline(char *cmd, int *nargs);
#endif

View File

@@ -40,6 +40,7 @@ Please visit our Website: http://www.httrack.com
/* File defs */
#include "htscore.h"
#include "htswarc.h"
#include "htschanges.h"
/* specific definitions */
#include "htsbase.h"
@@ -447,13 +448,22 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
const char *fil) {
if (!*makeindex_done) {
if (*makeindex_fp) {
char BIGSTK tempo[1024];
/* sized off link_escaped below: at the old flat 1024 a long first link
produced a redirect to a clipped URL */
char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
if (makeindex_links == 1) {
char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
snprintf(tempo, sizeof(tempo),
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped);
/* no redirect beats one pointing at a clipped URL */
if (!sprintfbuff(
tempo,
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped)) {
hts_log_print(opt, LOG_WARNING,
"index redirect omitted: first link too long (%s)",
makeindex_firstlink);
tempo[0] = '\0';
}
} else
tempo[0] = '\0';
hts_template_format(*makeindex_fp, template_footer,
@@ -479,6 +489,7 @@ void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r,
const char *adr, const char *fil, const char *save) {
{
file_notify(opt, adr, fil, save, 1, 1, r->notmodified);
hts_changes_html(opt, cache, r, adr, fil, save);
*fp = filecreate(&opt->state.strc, save);
if (*fp) {
if (ht_len > 0 && fwrite(ht_buff, 1, ht_len, *fp) != ht_len) {
@@ -696,17 +707,19 @@ int httpmirror(char *url1, httrackp * opt) {
// copier adresse(s) dans liste des adresses
{
char *a = url1;
int primary_len = 8192;
if (StringNotEmpty(opt->filelist)) {
primary_len += max(0, fsize_utf8(StringBuff(opt->filelist)) * 2);
}
primary_len += (int) strlen(url1) * 2;
const LLint list_sz = StringNotEmpty(opt->filelist)
? fsize_utf8(StringBuff(opt->filelist))
: 0;
/* two bytes reserved per list byte; -1 makes an undoublable size refused */
const LLint list_room =
list_sz > 0 ? (list_sz <= INT64_MAX / 2 ? list_sz * 2 : -1) : 0;
const size_t primary_len =
llint_grow_size_t(8192 + strlen(url1) * 2, list_room, 0);
// création de la première page, qui contient les liens de base à scanner
// c'est plus propre et plus logique que d'entrer à la main les liens dans la pile
// on bénéficie ainsi des vérifications et des tests du robot pour les liens "primaires"
primary = (char *) malloct(primary_len);
primary = primary_len != (size_t) -1 ? (char *) malloct(primary_len) : NULL;
if (!primary) {
printf("PANIC! : Not enough memory [%d]\n", __LINE__);
XH_extuninit;
@@ -887,7 +900,7 @@ int httpmirror(char *url1, httrackp * opt) {
}
if (filelist_buff != NULL) {
int filelist_ptr = 0;
size_t filelist_ptr = 0;
int n = 0;
char BIGSTK line[HTS_URLMAXSIZE * 2];
@@ -2078,7 +2091,9 @@ int httpmirror(char *url1, httrackp * opt) {
if (cache.lst) {
fclose(cache.lst);
cache.lst = opt->state.strc.lst = NULL;
if (opt->delete_old) {
/* old.lst minus new.lst is the set of files the previous mirror had and
this one does not. --changes reports it; only --purge-old acts on it. */
if (opt->delete_old || opt->changes) {
FILE *old_lst, *new_lst;
//
@@ -2106,21 +2121,27 @@ int httpmirror(char *url1, httrackp * opt) {
int purge = 0;
while(!feof(old_lst)) {
linput(old_lst, line, 1000);
if (!strstr(adr, line)) { // not found in the new list?
char BIGSTK file[HTS_URLMAXSIZE * 2];
char BIGSTK file[HTS_URLMAXSIZE * 2];
strcpybuff(file, StringBuff(opt->path_html));
strcatbuff(file, line + 1);
file[strlen(file) - 1] = '\0';
if (fexist_utf8(file)) { // toujours sur disque: virer
hts_log_print(opt, LOG_INFO, "Purging %s", file);
UNLINK(file);
purge = 1;
linput(old_lst, line, 1000);
if (!strnotempty(line))
continue;
strcpybuff(file, StringBuff(opt->path_html));
strcatbuff(file, line + 1);
file[strlen(file) - 1] = '\0';
hts_changes_previous(opt, file + StringLength(opt->path_html));
if (!strstr(adr, line)) { // not found in the new list?
if (fexist_utf8(file)) { // still on disk
hts_changes_gone(opt, file + StringLength(opt->path_html));
if (opt->delete_old) {
hts_log_print(opt, LOG_INFO, "Purging %s", file);
UNLINK(file);
purge = 1;
}
}
}
}
{
if (opt->delete_old) { // emptied directories go with the files
fseek(old_lst, 0, SEEK_SET);
while(!feof(old_lst)) {
linput(old_lst, line, 1000);
@@ -2155,7 +2176,7 @@ int httpmirror(char *url1, httrackp * opt) {
}
}
//
if (!purge) {
if (opt->delete_old && !purge) {
hts_log_print(opt, LOG_INFO, "No files purged");
}
}
@@ -2245,6 +2266,7 @@ int httpmirror(char *url1, httrackp * opt) {
// ending
usercommand(opt, 0, NULL, NULL, NULL, NULL);
warc_close_opt(opt);
hts_changes_close_opt(opt);
// désallocation mémoire & buffers
XH_uninit;
@@ -2870,6 +2892,18 @@ int filecreateempty(filenote_strc * strc, const char *filename) {
return 0;
}
void hts_savename_listed(const filenote_strc *strc, const char *s, char *dest,
size_t destsize) {
char catbuff[CATBUFF_SIZE];
strlcpybuff(dest, fslash(catbuff, sizeof(catbuff), s), destsize);
if (strnotempty(strc->path) &&
strncmp(fslash(catbuff, sizeof(catbuff), strc->path), dest,
strlen(strc->path)) == 0) {
strlcpybuff(dest, s + strlen(strc->path), destsize);
}
}
// noter fichier
int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
// gestion du fichier liste liste
@@ -2879,15 +2913,8 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
return 0;
} else if (strc->lst) {
char BIGSTK savelst[HTS_URLMAXSIZE * 2];
char catbuff[CATBUFF_SIZE];
strcpybuff(savelst, fslash(catbuff, sizeof(catbuff), s));
// couper chemin?
if (strnotempty(strc->path)) {
if (strncmp(fslash(catbuff, sizeof(catbuff), strc->path), savelst, strlen(strc->path)) == 0) { // couper
strcpybuff(savelst, s + strlen(strc->path));
}
}
hts_savename_listed(strc, s, savelst, sizeof(savelst));
fprintf(strc->lst, "[%s]" LF, savelst);
fflush(strc->lst);
}
@@ -2897,6 +2924,9 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
/* Note: utf-8 */
void file_notify(httrackp * opt, const char *adr, const char *fil,
const char *save, int create, int modify, int not_updated) {
hts_changes_notify(opt, adr, fil, save,
(create || modify) ? HTS_TRUE : HTS_FALSE,
not_updated ? HTS_TRUE : HTS_FALSE);
RUN_CALLBACK6(opt, filesave2, adr, fil, save, create, modify, not_updated);
}
@@ -3631,6 +3661,7 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
to->warc_max_size = from->warc_max_size;
to->warc_cdx = from->warc_cdx;
to->warc_wacz = from->warc_wacz;
to->changes = from->changes;
if (from->pause_max_ms > 0) {
to->pause_min_ms = from->pause_min_ms;

View File

@@ -354,6 +354,13 @@ int filecreateempty(filenote_strc * strct, const char *filename);
int filenote(filenote_strc * strct, const char *s, filecreate_params * params);
/* Copy into dest (destsize bytes) the form under which the local path s is
listed in new.lst: forward slashes, and the mirror root stripped when s sits
under it. That listed form is also the change report's key, so the two must
not drift. */
void hts_savename_listed(const filenote_strc *strct, const char *s, char *dest,
size_t destsize);
void file_notify(httrackp * opt, const char *adr, const char *fil,
const char *save, int create, int modify, int wasupdated);

View File

@@ -41,6 +41,7 @@ Please visit our Website: http://www.httrack.com
#include "htsdefines.h"
#include "htsalias.h"
#include "htswarc.h"
#include "htschanges.h"
#include "htsbauth.h"
#include "htswrap.h"
#include "htsmodules.h"
@@ -145,7 +146,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
int argv_url = -1; // ==0 : utiliser cache et doit.log
char *argv_firsturl = NULL; // utilisé pour nommage par défaut
char *url = NULL; // URLS séparées par un espace
int url_sz = 65535;
size_t url_sz = 65535;
// the parametres
int httrack_logmode = 3; // ONE log file
@@ -224,21 +225,22 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
/* create x_argvblk buffer for transformed command line */
{
int current_size = 0;
int size;
size_t current_size = 0;
const LLint size = fsize("config");
size_t blk_size;
int na;
for(na = 0; na < argc; na++)
current_size += (int) (strlen(argv[na]) + 1);
if ((size = fsize("config")) > 0)
current_size += size;
x_argvblk = (char *) malloct(current_size + 32768);
current_size += strlen(argv[na]) + 1;
/* a huge file named "config" must saturate, not wrap, the capacity */
blk_size = llint_grow_size_t(current_size, size > 0 ? size : 0, 32768);
x_argvblk = blk_size != (size_t) -1 ? (char *) malloct(blk_size) : NULL;
if (x_argvblk == NULL) {
HTS_PANIC_PRINTF("Error, not enough memory");
htsmain_free();
return -1;
}
x_argvblk_size = (size_t) (current_size + 32768);
x_argvblk_size = blk_size;
x_argvblk[0] = '\0';
x_ptr = 0;
@@ -1456,20 +1458,29 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
FILE *fp = FOPEN(argv[na], "rb");
if (fp != NULL) {
int cl = (int) strlen(url);
size_t cl = strlen(url);
const size_t fzs = llint_to_size_t(fz);
const size_t capa = llint_grow_size_t(cl, fz, 8192);
ensureUrlCapacity(url, url_sz, cl + fz + 8192);
if (capa == (size_t) -1) {
fclose(fp);
HTS_PANIC_PRINTF("File url list too large");
htsmain_free();
return -1;
}
ensureUrlCapacity(url, url_sz, capa);
if (cl > 0) { /* don't stick! (3.43) */
url[cl] = ' ';
cl++;
}
if (fread(url + cl, 1, fz, fp) != fz) {
if (fread(url + cl, 1, fzs, fp) != fzs) {
fclose(fp);
HTS_PANIC_PRINTF("File url list could not be read");
htsmain_free();
return -1;
}
fclose(fp);
*(url + cl + fz) = '\0';
*(url + cl + fzs) = '\0';
}
}
}
@@ -1737,6 +1748,13 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
StringCopy(opt->cookies_file, argv[na]);
}
break;
case 'd': // --changes: report what this crawl changed
opt->changes = HTS_TRUE;
if (*(com + 1) == '0') {
opt->changes = HTS_FALSE;
com++;
}
break;
case 'r': // warc / warc-file: write an ISO-28500 WARC archive
if (*(com + 1) == 'f') { // --warc-file NAME: explicit basename
com++;
@@ -2296,8 +2314,8 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
} else { // URL/filters
char catbuff[CATBUFF_SIZE];
const int urlSize = (int) strlen(argv[na]);
const int capa = (int) (strlen(url) + urlSize + 32);
const size_t urlSize = strlen(argv[na]);
const size_t capa = strlen(url) + urlSize + 32;
assertf(urlSize < HTS_URLMAXSIZE);
if (urlSize < HTS_URLMAXSIZE) {

View File

@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// folding a nonsense port into 1..65535 fetches one the link never named;
// an empty "host:" just means the default (#614)
if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
htsblk_failf(&back->r, "Invalid port: %s", a + 1);
back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
_HALT_FTP return 0;
}
@@ -262,8 +262,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// récupérer adresse résolue
strcpybuff(back->info, "host name");
if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
snprintf(back->r.msg, sizeof(back->r.msg),
"Unable to get server's address: %s", error);
htsblk_failf(&back->r, "Unable to get server's address: %s", error);
back->r.statuscode = STATUSCODE_NON_FATAL;
_HALT_FTP return 0;
}
@@ -332,18 +331,15 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
linejmp(line));
htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
linejmp(line));
htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
linejmp(line));
htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
@@ -410,8 +406,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
// -- fin analyse de l'adresse IP et du port --
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
@@ -442,13 +437,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
linejmp(line));
htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
}
@@ -554,8 +547,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
deletesoc(soc_dat);
soc_dat = INVALID_SOCKET;
//
snprintf(back->r.msg, sizeof(back->r.msg),
"RETR command error: %s", linejmp(line));
htsblk_failf(&back->r, "RETR command error: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
@@ -573,13 +566,12 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
snprintf(back->r.msg, sizeof(back->r.msg),
"Unable to resolve IP %s: %s", adr_ip, error);
htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
error);
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
#else
@@ -603,13 +595,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg),
"RETR command error: %s", linejmp(line));
htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
linejmp(line));
htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
#ifdef _WIN32
@@ -652,8 +642,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
len = 0; // fin
break;
case 0:
snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
timeout);
htsblk_failf(&back->r, "Time out (%d)", timeout);
back->r.statuscode = STATUSCODE_INVALID;
len = 0; // fin
break;
@@ -716,8 +705,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
strcpybuff(back->r.msg, "OK");
back->r.statuscode = HTTP_OK;
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {

View File

@@ -72,7 +72,8 @@ Please visit our Website: http://www.httrack.com
HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
compiler type-checks the format string at argument index fmt against the
varargs starting at arg. */
varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
error signal, so dropping it is a bug; a (void) cast does not silence it. */
#ifndef HTS_UNUSED
#ifdef __GNUC__
#define HTS_UNUSED __attribute__((unused))
@@ -80,10 +81,13 @@ Please visit our Website: http://www.httrack.com
#define HTS_STATIC static __attribute__((unused))
#define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
#else
#define HTS_UNUSED
#define HTS_STATIC static
#define HTS_PRINTF_FUN(fmt, arg)
#define HTS_CHECK_RESULT
#endif
#endif
@@ -336,16 +340,20 @@ typedef int INTsys;
#endif
/* Socket-handle type. An unsigned integer wide enough for a Windows SOCKET;
a plain int file descriptor on POSIX. */
a plain int file descriptor on POSIX. T_SOCP is its printf conversion,
'%' included: unsigned __int64 on Win64 must not be printed with "%d". */
#ifdef _WIN32
#if defined(_WIN64)
typedef unsigned __int64 T_SOC;
#define T_SOCP "%" PRIu64
#else
typedef unsigned __int32 T_SOC;
#define T_SOCP "%" PRIu32
#endif
#else
typedef int T_SOC;
#define T_SOCP "%d"
#endif
/* Buffer size for a printed network address (IPv4 or IPv6, NUL included). */

View File

@@ -124,7 +124,9 @@ typedef struct help_wizard_buffers {
char stropt[2048]; // options
char stropt2[2048]; // options longues
char strwild[2048]; // wildcards
char cmd[4096];
/* holds all four of the above plus separators: at 4096 a long answer set
clipped the filters off the command line */
char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
char str[256];
char *argv[256];
} help_wizard_buffers;
@@ -156,9 +158,7 @@ void help_wizard(httrackp * opt) {
char *a;
//
if (urls == NULL || mainpath == NULL || projname == NULL || stropt == NULL
|| stropt2 == NULL || strwild == NULL || cmd == NULL || str == NULL
|| argv == NULL) {
if (buffers == NULL) {
fprintf(stderr, "* memory exhausted in %s, line %d\n", __FILE__, __LINE__);
return;
}
@@ -251,6 +251,7 @@ void help_wizard(httrackp * opt) {
strcatbuff(stropt2, "--update ");
break;
case 0:
freet(buffers);
return;
break;
}
@@ -309,14 +310,23 @@ void help_wizard(httrackp * opt) {
printf("\n");
if (strlen(stropt) == 1)
stropt[0] = '\0'; // aucune
snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
/* the tail is the filter list, and cmd is split into the argv handed to
hts_main() below: a clipped line would silently widen the crawl */
if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
printf("* command line too long (%d bytes max)\n",
(int) sizeof(cmd) - 1);
freet(buffers);
return;
}
printf("---> Wizard command line: httrack %s\n\n", cmd);
printf("Ready to launch the mirror? (Y/n) :");
fflush(stdout);
linput(stdin, str, 250);
if (strnotempty(str)) {
if (!((str[0] == 'y') || (str[0] == 'Y')))
if (!((str[0] == 'y') || (str[0] == 'Y'))) {
freet(buffers);
return;
}
}
printf("\n");
@@ -340,7 +350,7 @@ void help_wizard(httrackp * opt) {
}
/* Free buffers */
free(buffers);
freet(buffers);
#undef urls
#undef mainpath
#undef projname
@@ -597,6 +607,8 @@ void help(const char *app, int more) {
"output name, --warc-max-size N rotates segments past N bytes, "
"--warc-cdx also writes a sorted CDXJ index, --wacz packages it all "
"as a WACZ file");
infomsg(" %d write hts-changes.json listing what this crawl left new, "
"changed, unchanged and gone compared to the previous mirror");
infomsg(" %n do not re-download locally erased files");
infomsg
(" %v display on screen filenames downloaded (in realtime) - * %v1 short version - %v2 full animation");

View File

@@ -37,6 +37,7 @@ Please visit our Website: http://www.httrack.com
#include "htscore.h"
#include "htswarc.h"
#include "htschanges.h"
/* specific definitions */
#include "htsbase.h"
@@ -704,18 +705,16 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
/* Check for errors */
if (soc == INVALID_SOCKET) {
if (retour) {
if (retour->msg) {
if (!strnotempty(retour->msg)) {
if (!strnotempty(retour->msg)) {
#ifdef _WIN32
int last_errno = WSAGetLastError();
int last_errno = WSAGetLastError();
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
#else
int last_errno = errno;
int last_errno = errno;
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
#endif
}
}
}
}
@@ -2207,7 +2206,7 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
#if DEBUG
printf("erreur gethostbyname\n");
#endif
if (retour && retour->msg) {
if (retour != NULL) {
#ifdef _WIN32
snprintf(retour->msg, sizeof(retour->msg),
"Unable to get server's address: %s", error);
@@ -2238,17 +2237,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
DEBUG_W("socket()=%d\n" _(int) soc);
#endif
if (soc == INVALID_SOCKET) {
if (retour && retour->msg) {
if (retour != NULL) {
#ifdef _WIN32
int last_errno = WSAGetLastError();
sprintf(retour->msg, "Unable to create a socket: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to create a socket: %s",
strerror(last_errno));
#else
int last_errno = errno;
sprintf(retour->msg, "Unable to create a socket: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to create a socket: %s",
strerror(last_errno));
#endif
}
return INVALID_SOCKET; // erreur création socket impossible
@@ -2262,17 +2261,8 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
&bind_addr, &error) == NULL
|| bind(soc, &SOCaddr_sockaddr(bind_addr),
SOCaddr_size(bind_addr)) != 0) {
if (retour && retour->msg) {
#ifdef _WIN32
snprintf(retour->msg, sizeof(retour->msg),
"Unable to bind the specificied server address: %s",
error);
#else
snprintf(retour->msg, sizeof(retour->msg),
"Unable to bind the specificied server address: %s",
error);
#endif
}
snprintf(retour->msg, sizeof(retour->msg),
"Unable to bind the specificied server address: %s", error);
deletesoc(soc);
return INVALID_SOCKET;
}
@@ -2319,17 +2309,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
#if HDEBUG
printf("unable to connect!\n");
#endif
if (retour != NULL && retour->msg) {
if (retour != NULL) {
#ifdef _WIN32
const int last_errno = WSAGetLastError();
sprintf(retour->msg, "Unable to connect to the server: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to connect to the server: %s",
strerror(last_errno));
#else
const int last_errno = errno;
sprintf(retour->msg, "Unable to connect to the server: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to connect to the server: %s",
strerror(last_errno));
#endif
}
/* Close the socket and notify the error!!! */
@@ -2548,6 +2538,15 @@ void fil_simplifie(char *f) {
}
}
void htsblk_failf(htsblk *r, const char *fmt, ...) {
va_list args;
va_start(args, fmt);
// deliberate clip: the reason is quoted from a remote reply
(void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
va_end(args);
}
// fermer liaison fichier ou socket
void deletehttp(htsblk * r) {
#if HTS_DEBUG_CLOSESOCK
@@ -2598,13 +2597,15 @@ void deletesoc(T_SOC soc) {
if (closesocket(soc) != 0) {
int err = WSAGetLastError();
fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
strerror(err));
}
#else
if (close(soc) != 0) {
const int err = errno;
fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
strerror(err));
}
#endif
#if HTS_WIDE_DEBUG
@@ -2707,6 +2708,24 @@ void time_gmt_rfc822(char *s) {
time_rfc822(s, A);
}
void hts_now_iso8601(char out[32]) {
time_t t = time(NULL);
struct tm tmv;
#if defined(_WIN32)
struct tm *g = gmtime(&t);
if (g != NULL)
tmv = *g;
else
memset(&tmv, 0, sizeof(tmv));
#else
if (gmtime_r(&t, &tmv) == NULL)
memset(&tmv, 0, sizeof(tmv));
#endif
strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
}
// heure actuelle, format rfc (taille buffer 256o)
void time_local_rfc822(char *s) {
time_t tt;
@@ -3017,7 +3036,7 @@ int finput(T_SOC fd, char *s, int max) {
}
// Like linput, but in memory (optimized)
int binput(char *buff, char *s, int max) {
int binput(const char *buff, char *s, int max) {
int count = 0;
int destCount = 0;
@@ -6020,6 +6039,8 @@ HTSEXT_API httrackp *hts_create_opt(void) {
StringCopy(opt->cookies_file, "");
StringCopy(opt->warc_file, "");
opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
opt->changes = HTS_FALSE;
opt->changes_state = NULL;
StringCopy(opt->why_url, "");
opt->pause_min_ms = 0;
opt->pause_max_ms = 0;
@@ -6173,6 +6194,8 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
StringFree(opt->why_url);
StringFree(opt->warc_file);
hts_changes_free_opt(opt);
StringFree(opt->path_html);
StringFree(opt->path_html_utf8);
StringFree(opt->path_log);

View File

@@ -199,6 +199,10 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
etc.). */
T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
int waitconnect, int addr_index, int *addr_count);
/* Clips the formatted failure reason into r->msg, which also round-trips
through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
void htsblk_failf(htsblk *r, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
HTS_INLINE void deletehttp(htsblk * r);
HTS_INLINE int deleteaddr(htsblk * r);
HTS_INLINE void deletesoc(T_SOC soc);
@@ -253,6 +257,10 @@ void sec2str(char *s, TStamp t);
void time_gmt_rfc822(char *s);
void time_local_rfc822(char *s);
/* Current UTC time as "YYYY-MM-DDThh:mm:ssZ". */
void hts_now_iso8601(char out[32]);
struct tm *convert_time_rfc822(struct tm *buffer, const char *s);
int set_filetime(const char *file, struct tm *tm_time);
int set_filetime_rfc822(const char *file, const char *date);
@@ -262,7 +270,7 @@ HTS_INLINE void time_rfc822_local(char *s, struct tm *A);
HTS_INLINE int sendc(htsblk * r, const char *s);
int finput(T_SOC fd, char *s, int max);
int binput(char *buff, char *s, int max);
int binput(const char *buff, char *s, int max);
int linput(FILE * fp, char *s, int max);
int linputsoc(T_SOC soc, char *s, int max);
int linputsoc_t(T_SOC soc, char *s, int max, int timeout);
@@ -607,6 +615,21 @@ static HTS_UNUSED size_t llint_to_size_t(LLint o) {
}
}
/* Capacity for @p used bytes plus @p extra more plus @p slack spare;
(size_t) -1 if the total exceeds (size_t) -2 or @p extra is negative
(llint_to_size_t() would map that to a huge valid-looking size). */
static HTS_UNUSED size_t llint_grow_size_t(size_t used, LLint extra,
size_t slack) {
const size_t max = (size_t) -2; /* (size_t) -1 is the error value */
const size_t e = extra >= 0 ? llint_to_size_t(extra) : (size_t) -1;
if (e == (size_t) -1 || used > max || slack > max - used ||
e > max - used - slack) {
return (size_t) -1;
}
return used + e + slack;
}
/* dirent() compatibility */
#ifdef _WIN32
/* Holds a UTF-8 d_name: MAX_PATH (260) UTF-16 units expand to <=3 bytes each.

View File

@@ -547,6 +547,10 @@ struct httrackp {
archive. Tail: ABI */
hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip
(implies --warc + --warc-cdx). Tail: ABI */
hts_boolean changes; /**< --changes: report what this crawl changed against
the previous mirror. Tail: ABI */
void *changes_state; /**< live change-report accumulator (hts_changes*),
engine-owned. Tail: ABI */
};
/* Running statistics for a mirror. */

View File

@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
code = 0;
if (code < 200 || code >= 300) {
snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
htsblk_failf(retour, "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
return 0;
}

View File

@@ -33,6 +33,7 @@ Please visit our Website: http://www.httrack.com
#ifndef HTSSAFE_DEFH
#define HTSSAFE_DEFH
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -459,6 +460,49 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
return b->buf;
}
/**
* Callers that deliberately ignore truncation use this instead of
* slprintfbuff(), so it is not HTS_CHECK_RESULT.
*/
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
int ret;
assertf(dest != NULL && size != 0);
ret = vsnprintf(dest, size, fmt, args);
/* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
dest[size - 1] = '\0';
return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
}
/**
* Formatted print into dest (capacity size, NUL included), truncating to fit
* and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
* result is the only truncation signal, so it must be acted on. Unlike
* strcpybuff() it never aborts, so it suits text built from remote input.
*/
static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
va_list args;
hts_boolean ret;
va_start(args, fmt);
ret = vslprintfbuff(dest, size, fmt, args);
va_end(args);
return ret;
}
/**
* slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
* On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
*/
#if (defined(__GNUC__) && !defined(__cplusplus))
#define sprintfbuff(ARR, ...) \
slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
#else
#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
#endif
/* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
added bounds checking. freet() also NULLs the freed pointer and tolerates
NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

View File

@@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htsback.h"
#include "htsdefines.h"
#include "htslib.h"
#include "htsalias.h"
@@ -51,6 +52,7 @@ Please visit our Website: http://www.httrack.com
#include "htscache_selftest.h"
#include "htsdns_selftest.h"
#include "htscharset.h"
#include "htscmdline.h"
#include "htsencoding.h"
#include "htsftp.h"
#include "htsmd5.h"
@@ -58,6 +60,7 @@ Please visit our Website: http://www.httrack.com
#include "htscodec.h"
#include "htsproxy.h"
#include "htswarc.h"
#include "htschanges.h"
#if HTS_USEZLIB
#include "htszlib.h"
#endif
@@ -568,6 +571,154 @@ static int string_safety_selftests(void) {
return 1;
}
/* sprintfbuff: truncate-and-report. Must never abort (its callers format
remote banners) nor write past the array, which the canary catches. */
{
struct {
char dst[8];
char canary[8];
} s;
const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
/* repoison before every call, or an implementation that measures first and
writes nothing "passes" the truncating cases on the previous content */
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
memset(&s, '#', sizeof(s));
if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
return 1;
/* exact fit: 7 characters plus the NUL */
POISON_DST();
if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
return 1;
/* one over, then far over: truncated to the prefix, terminated, reported */
POISON_DST();
if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
return 1;
POISON_DST();
if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
return 1;
/* explicit-capacity form, down to the degenerate size 1 */
{
char *const p = s.dst;
POISON_DST();
if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
return 1;
POISON_DST();
if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
return 1;
}
#undef POISON_DST
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
return 1;
}
/* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
touches nothing else in the block */
{
htsblk r;
char expect[sizeof(r.msg)];
char big[4 * sizeof(r.msg)];
/* contenttype abuts msg, so a one-past-the-end store lands in it rather
than in padding. Poison it: a stray NUL is invisible against zeroes,
and a stray NUL is exactly what an off-by-one terminator writes. */
#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
memset(&r, 0, sizeof(r));
memset(r.contenttype, '#', sizeof(r.contenttype));
r.statuscode = 1234;
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
return 1;
/* exact fit: capacity - 1 characters plus the NUL */
memset(expect, 'y', sizeof(expect) - 1);
expect[sizeof(expect) - 1] = '\0';
memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "%s", expect);
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
return 1;
/* far over: the expected bytes differ from the cases above, so writing
nothing cannot pass on the leftovers */
memset(big, 'z', sizeof(big) - 1);
big[sizeof(big) - 1] = '\0';
memset(expect, 'z', sizeof(expect) - 1);
expect[sizeof(expect) - 1] = '\0';
memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "Bad user name: %s", big);
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
return 1;
#undef NEIGHBOURS_INTACT
}
/* back_read_ftp_result: the helper's result file is external input, so an
over-long message must stop at msg[]'s capacity */
{
htsblk r;
size_t k;
/* poisoned so a short message cannot pass on leftovers, and so a stray
NUL past msg[] is visible in the neighbour */
#define FTP_RESULT_CASE(BODY) \
do { \
FILE *fp_ = tmpfile(); \
\
if (fp_ == NULL) \
return 1; \
BODY; \
rewind(fp_); \
memset(&r, 0, sizeof(r)); \
memset(r.msg, '#', sizeof(r.msg)); \
memset(r.contenttype, '#', sizeof(r.contenttype)); \
back_read_ftp_result(fp_, &r); \
fclose(fp_); \
if (r.contenttype[0] != '#') \
return 1; \
} while (0)
/* over capacity: clipped to 79 payload bytes plus the NUL */
FTP_RESULT_CASE({
fprintf(fp_, "226 ");
for (k = 0; k < 4 * sizeof(r.msg); k++)
fputc('q', fp_);
});
if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
return 1;
for (k = 0; k < sizeof(r.msg) - 1; k++) {
if (r.msg[k] != 'q')
return 1;
}
/* well under capacity: nothing padded, nothing eaten off the end */
FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
return 1;
/* a byte over 0x7f must not read as EOF and cut the message short */
FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
return 1;
/* unparseable status: the message still loads, the code reports failure */
FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
if (r.statuscode != STATUSCODE_INVALID)
return 1;
#undef FTP_RESULT_CASE
}
/* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
n_eval == 2 on a double-eval macro. */
{
@@ -1126,6 +1277,128 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
return 0;
}
// hts_split_cmdline(): the vector must grow with the argument count, and a
// quote inside a value must not end the argument and hand -V to the parser.
static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
char line[512];
char **args;
int nargs = 0;
(void) opt;
(void) argc;
(void) argv;
// control: every separator splits, and argv[0] is the program name
strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
args = hts_split_cmdline(line, &nargs);
assertf(args != NULL && nargs == 6);
assertf(args[nargs] == NULL); // callers may walk to the terminator
assertf(strcmp(args[0], "httrack") == 0);
assertf(strcmp(args[1], "http://x/") == 0);
assertf(strcmp(args[2], "--quiet") == 0);
assertf(strcmp(args[3], "-c8") == 0);
assertf(strcmp(args[4], "-O") == 0);
assertf(strcmp(args[5], "out") == 0);
freet(args);
// the template pads with whitespace: empty arguments are kept (the engine
// skips them), so the count is one per separator
strcpybuff(line, "httrack --quiet");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3 && args[1][0] == '\0');
assertf(strcmp(args[2], "--quiet") == 0);
freet(args);
// a quoted run keeps both its spaces and its quotes: the engine unquotes
strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 4);
assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
assertf(strcmp(args[3], "-c8") == 0);
freet(args);
// an escaped quote is a literal quote, not the end of the argument: the
// engine strips only the outer pair
strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 4);
assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
assertf(strcmp(args[3], "-c8") == 0);
freet(args);
// \\ is a literal backslash, so a Windows path survives
strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
freet(args);
// outside a quoted run a backslash is literal: the url and wildcard-filter
// fields, which the wizard cannot quote, read as before
strcpybuff(line, "httrack -*\\** +*.png");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[1], "-*\\**") == 0);
assertf(strcmp(args[2], "+*.png") == 0);
freet(args);
// a quoted run leaves slots unused, so the terminator has to be written and
// not inherited: size the vector from a full line first, so freeing it hands
// the same chunk back with stale pointers in those slots
strcpybuff(line, "httrack a b c d e");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 6);
freet(args);
strcpybuff(line, "httrack \"a b c d e\"");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 2);
assertf(args[nargs] == NULL);
freet(args);
// an unterminated quote protects the rest of the line, as one argument
strcpybuff(line, "httrack --footer \"unbalanced -V x");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
freet(args);
// past the 1024 entries the vector used to hold: distinct arguments, so a
// write beyond the allocation cannot read back as the expected parse
{
const int n = 2000;
const size_t size = 16 * (size_t) n + 16;
char *big = malloct(size);
size_t pos = 0;
int i;
assertf(big != NULL);
pos = (size_t) snprintf(big, size, "httrack");
assertf(pos < size);
for (i = 0; i < n; i++) {
// snprintf returns what it wanted to write, so accumulating it blind
// would let the next size argument wrap
const int len = snprintf(big + pos, size - pos, " a%d", i);
assertf(len > 0 && (size_t) len < size - pos);
pos += (size_t) len;
}
args = hts_split_cmdline(big, &nargs);
assertf(args != NULL && nargs == n + 1);
assertf(args[nargs] == NULL);
for (i = 0; i < n; i++) {
char expect[16];
snprintf(expect, sizeof(expect), "a%d", i);
assertf(strcmp(args[i + 1], expect) == 0);
}
freet(args);
freet(big);
}
printf("cmdline-split self-test OK\n");
return 0;
}
static int st_hashtable(httrackp *opt, int argc, char **argv) {
char *snum;
unsigned long count = 0;
@@ -1402,6 +1675,12 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0)
err = 1;
from->changes = HTS_TRUE;
to->changes = HTS_FALSE;
copy_htsopt(from, to);
if (to->changes != HTS_TRUE)
err = 1;
/* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
from->pause_min_ms = 5000;
from->pause_max_ms = 10000;
@@ -2023,6 +2302,77 @@ static int st_fsize(httrackp *opt, int argc, char **argv) {
return rc;
}
/* 4GB+100KB wraps to ~108KB through an int, and needs 33 unsigned bits. A
macro, not a static const: MSVC's C mode (/TC) rejects a const object
used inside another object's static initializer below (C2099). */
#define HTS_ST_GROWSIZE_OVER32 (4LL * 1024 * 1024 * 1024 + 100 * 1024)
/* llint_grow_size_t() sizes the buffer holding a whole -%S list file: the
result must be the exact 64-bit sum or a clean refusal, never a short one. */
static int st_growsize(httrackp *opt, int argc, char **argv) {
enum { REFUSE, ACCEPT, WIDTH };
static const struct {
size_t used;
LLint extra;
size_t slack;
int want;
} cases[] = {
{0, 0, 0, ACCEPT},
{10, 100, 8192, ACCEPT},
{(size_t) -2 - 8, 4, 4, ACCEPT}, /* exact fit, no room to spare */
{(size_t) -2, 0, 0, ACCEPT}, /* largest representable capacity */
{0, -1, 0, REFUSE}, /* fsize() failure */
/* -1 already maps to SIZE_MAX; only this exercises the negative guard */
{0, -4096, 0, REFUSE},
{(size_t) -1, 1, 0, REFUSE},
{(size_t) -2, 0, 1, REFUSE}, /* slack alone overruns */
{(size_t) -1 - 8, 4, 4, REFUSE}, /* total would be the error value */
{(size_t) -1 - 8, 4, 8, REFUSE},
{0, HTS_ST_GROWSIZE_OVER32, 8192,
WIDTH}, /* 32-bit size_t can't hold these */
{10, HTS_ST_GROWSIZE_OVER32, 8192, WIDTH},
};
size_t k;
int rc = 0;
(void) opt;
(void) argc;
(void) argv;
for (k = 0; k < sizeof(cases) / sizeof(cases[0]); k++) {
const size_t used = cases[k].used, slack = cases[k].slack;
const LLint extra = cases[k].extra;
const size_t got = llint_grow_size_t(used, extra, slack);
const hts_boolean refused = got == (size_t) -1 ? HTS_TRUE : HTS_FALSE;
const hts_boolean exact =
!refused && extra >= 0 && got - used - slack == (size_t) extra;
hts_boolean ok;
switch (cases[k].want) {
case ACCEPT:
ok = exact;
break;
case REFUSE:
ok = refused;
break;
default:
ok = sizeof(size_t) >= sizeof(LLint) ? exact : refused;
break;
}
if (!ok) {
fprintf(stderr,
"growsize: grow(" LLintP ", " LLintP ", " LLintP ") = " LLintP
" (want %s)\n",
(LLint) used, extra, (LLint) slack, (LLint) got,
cases[k].want == REFUSE ? "refusal" : "exact sum");
rc = 1;
}
}
printf("growsize self-test %s\n", rc == 0 ? "OK" : "FAILED");
return rc;
}
static int st_savename(httrackp *opt, int argc, char **argv) {
lien_adrfilsave afs;
cache_back cache;
@@ -2417,17 +2767,20 @@ static int st_cookies(httrackp *opt, int argc, char **argv) {
static t_cookie ck2;
htsblk r;
char host[600];
char line[64]; /* treathead NUL-cuts the header in place: never a literal */
memset(&r, 0, sizeof(r));
memset(host, 'a', sizeof(host) - 1);
host[sizeof(host) - 1] = '\0';
ck2.max_len = (int) sizeof(ck2.data);
ck2.data[0] = '\0';
treathead(&ck2, host, "/", &r, "Set-Cookie: SID=1; path=/");
strcpybuff(line, "Set-Cookie: SID=1; path=/");
treathead(&ck2, host, "/", &r, line);
if (strnotempty(ck2.data)) // oversize-host cookie was not dropped
err = 1;
/* control: a normal host still yields a cookie through treathead */
treathead(&ck2, dom, "/", &r, "Set-Cookie: SID=1; path=/");
strcpybuff(line, "Set-Cookie: SID=1; path=/");
treathead(&ck2, dom, "/", &r, line);
if (strstr(ck2.data, "SID") == NULL) // guard wrongly dropped a valid cookie
err = 1;
}
@@ -2688,6 +3041,32 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
assertf(strstr(buf, "Refresh") != NULL);
assertf(strstr(buf, "example.com") != NULL);
/* a first link whose escaped form overruns the old flat 1024-byte tempo: the
redirect must carry the whole URL, not a clipped prefix */
{
char BIGSTK link[HTS_URLMAXSIZE * 2];
char *p = link;
strcpybuff(link, "http://example.com/");
p += strlen(link);
memset(p, 'a', 1200);
p += 1200;
strcpy(p, "/end.html");
done = 0;
fp = fopen(path, "wb");
assertf(fp != NULL);
hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
assertf(fp == NULL);
fp = fopen(path, "rb");
assertf(fp != NULL);
n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
buf[n] = '\0';
/* the closing quote proves the URL was not clipped mid-way */
assertf(strstr(buf, "/end.html\">") != NULL);
}
/* no single link: footer only, no refresh meta */
done = 0;
fp = fopen(path, "wb");
@@ -2917,7 +3296,7 @@ static int ae_write_packed(const char *path, int windowBits,
deflateEnd(&strm);
return 1;
}
strm.next_in = (Bytef *) src;
strm.next_in = (const Bytef *) src;
strm.avail_in = (uInt) len;
do {
size_t n;
@@ -4727,10 +5106,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
char fpath[HTS_URLMAXSIZE * 2];
char file[HTS_URLMAXSIZE * 2];
snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
/* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
snprintf(file, sizeof(file), "%scookies.txt", fpath);
assertf(sprintfbuff(file, "%scookies.txt", fpath));
{
FILE *fp = FOPEN(file, "wb");
@@ -4740,7 +5119,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
}
/* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
snprintf(file, sizeof(file), "%su@v.txt", fpath);
assertf(sprintfbuff(file, "%su@v.txt", fpath));
{
FILE *fp = FOPEN(file, "wb");
@@ -4762,7 +5141,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
#endif
(void) UNLINK(file); /* u@v.txt (already gone on Windows) */
snprintf(file, sizeof(file), "%scookies.txt", fpath);
assertf(sprintfbuff(file, "%scookies.txt", fpath));
(void) UNLINK(file);
dir[dirlen] = '\0';
while (strlen(dir) > base) {
@@ -4788,6 +5167,63 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
return 0;
}
/* --changes bucket accounting and JSON escaping (#714). */
static int st_changes(httrackp *opt, int argc, char **argv) {
String out = STRING_EMPTY;
int err = 0;
(void) opt;
(void) argc;
(void) argv;
/* A file the crawl did not rewrite is unchanged whatever the wire said. */
assertf(hts_changes_classify(HTS_FALSE, HTS_TRUE, HTS_FALSE, HTS_FALSE,
HTS_FALSE) == HTS_CHANGE_UNCHANGED);
/* Rewritten with no previous copy: new, digests or not. */
assertf(hts_changes_classify(HTS_TRUE, HTS_FALSE, HTS_FALSE, HTS_TRUE,
HTS_FALSE) == HTS_CHANGE_NEW);
/* Digests decide, and outrank the transfer signal both ways: a server with
no validators answers 200 with the same bytes, and a 304 can still sit in
front of a locally damaged copy. */
assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_FALSE, HTS_TRUE,
HTS_TRUE) == HTS_CHANGE_UNCHANGED);
assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_TRUE, HTS_TRUE,
HTS_FALSE) == HTS_CHANGE_CHANGED);
/* Only with no digest at all does the transfer signal get a say. */
assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_TRUE, HTS_FALSE,
HTS_FALSE) == HTS_CHANGE_UNCHANGED);
assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_FALSE, HTS_FALSE,
HTS_FALSE) == HTS_CHANGE_CHANGED);
#define JSON_IS(SRC, WANT) \
do { \
StringClear(out); \
hts_changes_json_string(&out, SRC); \
if (strcmp(StringBuff(out), WANT) != 0) { \
fprintf(stderr, "changes: %s -> %s, expected %s\n", #SRC, \
StringBuff(out), WANT); \
err = 1; \
} \
} while (0)
JSON_IS("/a/b.html", "\"/a/b.html\"");
JSON_IS("a\"b\\c", "\"a\\\"b\\\\c\"");
JSON_IS("tab\there", "\"tab\\u0009here\"");
/* Valid UTF-8 rides through; a lone Latin-1 byte, a truncated sequence and
an overlong encoding of '/' each become U+FFFD rather than invalid JSON. */
JSON_IS("caf\xc3\xa9", "\"caf\xc3\xa9\"");
JSON_IS("caf\xe9", "\"caf\\ufffd\"");
JSON_IS("\xc3", "\"\\ufffd\"");
JSON_IS("\xc0\xaf", "\"\\ufffd\\ufffd\"");
/* A UTF-16 surrogate half is well-formed UTF-8 by shape only. */
JSON_IS("\xed\xa0\x80", "\"\\ufffd\\ufffd\\ufffd\"");
#undef JSON_IS
StringFree(out);
printf("changes self-test: %s\n", err ? "FAIL" : "OK");
return err;
}
/* ------------------------------------------------------------ */
/* Registry: name -> handler, with a usage hint and a one-line description. */
/* ------------------------------------------------------------ */
@@ -4836,10 +5272,15 @@ static const struct selftest_entry {
st_footerfmt},
{"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
st_unescape_bounds},
{"cmdline-split", "",
"webhttrack command-line to argv split (bounds, quoting)",
st_cmdlinesplit},
{"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
{"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
"bounded string-op self-test", st_strsafe},
{"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt},
{"changes", "", "--changes bucket accounting and JSON escaping (#714)",
st_changes},
{"pause", "", "randomized inter-file pause target self-test", st_pause},
{"relative", "<link> <curr-file>", "relative link between two paths",
st_relative},
@@ -4868,6 +5309,8 @@ static const struct selftest_entry {
{"sniff", "<content-type> <hex:..|text>", "MIME magic consistency",
st_sniff},
{"fsize", "<dir>", "file size past the 2GB signed-32-bit wrap", st_fsize},
{"growsize", "", "buffer capacity for a 64-bit file size (no int wrap)",
st_growsize},
{"cache", "<dir>", "cache read/write round-trip self-test", st_cache},
{"cacheindex", "", "cache-index (.ndx) parse must stay in bounds",
st_cacheindex},

View File

@@ -147,7 +147,8 @@ HTS_UNUSED static int LANG_LIST(const char *path, char *buffer, size_t size);
// 0- Init the URL catcher with standard port
// smallserver_init(&port,&return_host);
T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort) {
T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort,
const char *bindAddr) {
T_SOC soc;
if (defaultPort <= 0) {
@@ -160,12 +161,12 @@ T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort) {
int i = 0;
do {
soc = smallserver_init(&try_to_listen_to[i], adr_prox);
soc = smallserver_init(&try_to_listen_to[i], adr_prox, bindAddr);
*port_prox = try_to_listen_to[i];
i++;
} while((soc == INVALID_SOCKET) && (try_to_listen_to[i] >= 0));
} else {
soc = smallserver_init(&defaultPort, adr_prox);
soc = smallserver_init(&defaultPort, adr_prox, bindAddr);
*port_prox = defaultPort;
}
return soc;
@@ -243,9 +244,10 @@ static int my_gethostname(char *h_loc, size_t size) {
}
// smallserver_init(&port,&return_host);
T_SOC smallserver_init(int *port, char *adr) {
T_SOC smallserver_init(int *port, char *adr, const char *bindAddr) {
T_SOC soc = INVALID_SOCKET;
char h_loc[256 + 2];
SOCaddr server;
commandRunning = commandEnd = commandReturn = commandReturnSet =
commandEndRequested = 0;
@@ -256,25 +258,23 @@ T_SOC smallserver_init(int *port, char *adr) {
free(commandReturnCmdl);
commandReturnCmdl = NULL;
if (my_gethostname(h_loc, 256) == 0) { // host name
SOCaddr server;
SOCaddr_initany(server);
if (bindAddr != NULL && *bindAddr != '\0') {
/* advertise the bound address, else the URL we print is unreachable */
if (strlen(bindAddr) >= sizeof(h_loc) || !gethost(bindAddr, &server)) {
return INVALID_SOCKET;
}
strcpybuff(h_loc, bindAddr);
} else if (my_gethostname(h_loc, 256) != 0) { // host name
return INVALID_SOCKET;
}
SOCaddr_initany(server);
if ((soc =
(T_SOC) socket(SOCaddr_sinfamily(server), SOCK_STREAM,
0)) != INVALID_SOCKET) {
SOCaddr_initport(server, *port);
if (bind(soc, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
if (listen(soc, 10) >= 0) {
strcpy(adr, h_loc);
} else {
#ifdef _WIN32
closesocket(soc);
#else
close(soc);
#endif
soc = INVALID_SOCKET;
}
if ((soc = (T_SOC) socket(SOCaddr_sinfamily(server), SOCK_STREAM, 0)) !=
INVALID_SOCKET) {
SOCaddr_initport(server, *port);
if (bind(soc, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
if (listen(soc, 10) >= 0) {
strcpy(adr, h_loc);
} else {
#ifdef _WIN32
closesocket(soc);
@@ -283,6 +283,13 @@ T_SOC smallserver_init(int *port, char *adr) {
#endif
soc = INVALID_SOCKET;
}
} else {
#ifdef _WIN32
closesocket(soc);
#else
close(soc);
#endif
soc = INVALID_SOCKET;
}
}
return soc;
@@ -311,6 +318,104 @@ typedef struct {
error_redirect = "/server/error.html"; \
} while(0)
/* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
so anything near this is already invalid and is rejected unread. */
#define SID_VALUE_MAX 64
/** Does the urlencoded request body present the expected session id?
True only if at least one "sid" field is present and every occurrence
matches, so it holds whichever one a later last-write-wins parse keeps.
Non-destructive: it runs before the body is tokenized in place. */
static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
const char *s = body;
hts_boolean seen = HTS_FALSE;
while (s != NULL && *s != '\0') {
const char *const amp = strchr(s, '&');
const char *const eq = strchr(s, '=');
if (eq != NULL && (amp == NULL || eq < amp) && (size_t) (eq - s) == 3 &&
strncmp(s, "sid", 3) == 0) {
const size_t len = amp != NULL ? (size_t) (amp - eq - 1) : strlen(eq + 1);
hts_boolean match = HTS_FALSE;
if (len < SID_VALUE_MAX) {
char raw[SID_VALUE_MAX];
String value = STRING_EMPTY;
memcpy(raw, eq + 1, len);
raw[len] = '\0';
unescapehttp(raw, &value);
/* StringBuff is NULL until written, so an empty value lands here. */
if (StringBuff(value) != NULL &&
strcmp(StringBuff(value), expected) == 0) {
match = HTS_TRUE;
}
StringFree(value);
}
if (!match) {
return HTS_FALSE;
}
seen = HTS_TRUE;
}
s = amp != NULL ? amp + 1 : NULL;
}
return seen;
}
/** Append src to the NUL-terminated dst of capacity size (NUL included).
False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
never aborts, because every piece appended here is client-supplied. */
static hts_boolean path_append(char *dst, size_t size, const char *src) {
const size_t used = strlen(dst);
const size_t len = strlen(src);
/* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
len stays alone: "used + len < size" could wrap and pass. */
if (len >= size - used) {
return HTS_FALSE;
}
memcpy(dst + used, src, len + 1);
return HTS_TRUE;
}
/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
static hts_boolean cat_html_escaped(String *dst, char c) {
switch (c) {
case '<':
StringCat(*dst, "&lt;");
break;
case '>':
StringCat(*dst, "&gt;");
break;
case '&':
StringCat(*dst, "&amp;");
break;
case '\'':
StringCat(*dst, "&#39;");
break;
default:
return HTS_FALSE;
}
return HTS_TRUE;
}
/* Append the value of a double-quoted command-line argument: escaped for HTML,
which the browser undoes when it posts the command line back, and for the
argv splitter, which does not. */
static void cat_cmdline_arg(String *output, const char *value) {
const char *a;
for (a = value; *a != '\0'; a++) {
if (*a == '\\' || *a == '\"') {
StringCat(*output, "\\");
}
if (!cat_html_escaped(output, *a)) {
StringMemcat(*output, a, 1);
}
}
}
int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
int timeout = 30;
int retour = 0;
@@ -322,6 +427,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
String tmpbuff = STRING_EMPTY;
String tmpbuff2 = STRING_EMPTY;
String fspath = STRING_EMPTY;
/* Project directory this server set up; the only root /website/ serves from,
and deliberately not cleared between requests. */
String website = STRING_EMPTY;
char catbuff[CATBUFF_SIZE];
/* Load strings */
@@ -395,6 +503,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
T_SOC soc_c;
LLint length = 0;
const char *error_redirect = NULL;
hts_boolean denied = HTS_FALSE;
line[0] = '\0';
buffer[0] = '\0';
@@ -506,6 +615,22 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
/* Authenticate the body before parsing it: every field it carries is
written straight into the global key store below, "command" included,
and that one reaches the engine. Checking afterwards cannot work — the
damage is already done, and the pre-seeded "sid" above would compare
equal to itself for a request that simply omits the field. */
if (meth && buffer[0]) {
intptr_t expected = 0;
if (!coucal_readptr(NewLangList, "_sid", &expected) ||
!body_sid_is_valid(buffer, (const char *) expected)) {
buffer[0] = '\0';
meth = 0;
denied = HTS_TRUE;
}
}
/* check variables */
if (meth && buffer[0]) {
char *s = buffer;
@@ -526,20 +651,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
/* Error check */
{
intptr_t adr = 0;
intptr_t adr2 = 0;
if (coucal_readptr(NewLangList, "sid", &adr)) {
if (coucal_readptr(NewLangList, "_sid", &adr2)) {
if (strcmp((char *) adr, (char *) adr2) != 0) {
meth = 0;
}
}
}
}
/* Check variables (internal) */
if (meth) {
int doLoad = 0;
@@ -730,6 +841,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
if (!structcheck(StringBuff(tmpbuff))) {
FILE *fp;
/* Both halves of fspath come from posted fields, so a ".."
in them would escape the mirror once served. */
if (strstr(StringBuff(fspath), "..") == NULL) {
StringCopy(website, StringBuff(fspath));
}
StringCat(tmpbuff, "winprofile.ini");
fp = fopen(StringBuff(tmpbuff), "wb");
if (fp != NULL) {
@@ -802,7 +918,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
/* Response */
if (meth) {
int virtualpath = 0;
hts_boolean virtualpath = HTS_FALSE;
char *pos;
char *url = strchr(line1, ' ');
@@ -813,11 +929,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
char *qpos;
/* get the URL */
fsfile[0] = '\0';
if (error_redirect == NULL) {
if ((qpos = strchr(url, '?'))) {
*qpos = '\0';
}
fsfile[0] = '\0';
if (strcmp(url, "/") == 0) {
file = "/server/index.html";
meth = 2;
@@ -830,7 +946,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
if (strncmp(file, "/website/", 9) == 0) {
virtualpath = 1;
virtualpath = HTS_TRUE;
}
/* override */
@@ -844,18 +960,26 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
if (strncmp(file, "/website/", 9) != 0) {
sprintf(fsfile, "%shtml%s", path, file);
} else {
intptr_t adr = 0;
/* the override above may have swapped a mirror path for a GUI page */
virtualpath = strncmp(file, "/website/", 9) == 0;
if (coucal_readptr(NewLangList, "projpath", &adr)) {
sprintf(fsfile, "%s%s", (char *) adr, file + 9);
}
if (!virtualpath) {
if (!path_append(fsfile, sizeof(fsfile), path) ||
!path_append(fsfile, sizeof(fsfile), "html") ||
!path_append(fsfile, sizeof(fsfile), file)) {
fsfile[0] = '\0';
}
} else if (StringNotEmpty(website)) {
/* Never the posted "projpath": a client root reads any file. */
if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
!path_append(fsfile, sizeof(fsfile), "/") ||
!path_append(fsfile, sizeof(fsfile), file + 9)) {
fsfile[0] = '\0';
}
}
/* path itself may hold ".." (webhttrack passes "<bin>/../share"), so
only the untrusted halves are checked: file here, website above. */
if (fsfile[0] && strstr(file, "..") == NULL
&& (fp = fopen(fsfile, "rb"))) {
char ok[] =
@@ -903,16 +1027,16 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
StringMemcat(headers, redir, strlen(redir));
{
char tmp[256];
if (strlen(file) < sizeof(tmp) - 32) {
sprintf(tmp, "Location: %s\r\n", newfile);
StringMemcat(headers, tmp, strlen(tmp));
}
/* client-supplied: a CR/LF here would split the response */
if (newfile[strcspn(newfile, "\r\n")] == '\0') {
StringCat(headers, "Location: ");
StringCat(headers, newfile);
StringCat(headers, "\r\n");
}
coucal_write(NewLangList, "redirect", (intptr_t) NULL);
} else if (is_html(file)) {
} else if (!virtualpath && is_html(file)) {
/* GUI templates only: ${_sid} in a mirrored page would hand the
crawled site the session id that authenticates commands */
int outputmode = 0;
StringMemcat(headers, ok, sizeof(ok) - 1);
@@ -940,6 +1064,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
int p;
int format = 0;
int listDefault = 0;
hts_boolean unquoted = HTS_FALSE;
name[0] = '\0';
strlncatbuff(name, str, sizeof(name_), n);
@@ -949,6 +1074,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
} else if ((p = strfield(name, "html:"))) {
name += p;
format = 1;
} else if ((p = strfield(name, "unquoted:"))) {
name += p;
unquoted = HTS_TRUE;
} else if ((p = strfield(name, "arg:"))) {
name += p;
format = 5;
} else if ((p = strfield(name, "list:"))) {
name += p;
format = 2;
@@ -1085,8 +1216,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
test:<if ==0>:<if ==1>:<if == 2>..
ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
*/
else if ((p = strfield(name, "test:"))
|| (p = strfield(name, "ztest:"))) {
else if ((p = strfield(name, "test:")) ||
(p = strfield(name, "ztest:"))) {
intptr_t adr = 0;
char *pos2;
int ztest = (name[0] == 'z');
@@ -1189,6 +1320,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
}
/* consumed here: it shares nothing with the list and
option formats below */
if (format == 5 && langstr != NULL && outputmode != -1) {
cat_cmdline_arg(&output, langstr);
langstr = NULL;
}
if (langstr && outputmode != -1) {
switch (format) {
case 0:
@@ -1206,18 +1343,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringMemcat(output, &c, 1);
}
a += 2;
} else if (outputmode && a[0] == '<') {
StringCat(output, "&lt;");
} else if (outputmode && a[0] == '>') {
StringCat(output, "&gt;");
} else if (outputmode && a[0] == '&') {
StringCat(output, "&amp;");
} else if (outputmode && a[0] == '\'') {
StringCat(output, "&#39;");
} else if (unquoted && a[0] == '\"') {
/* the browser posts an entity back as a raw
quote, which would open a quoted run in the
argv splitter; a URI cannot hold one anyway */
StringCat(output, "%22");
} else if (outputmode &&
cat_html_escaped(&output, a[0])) {
/* appended as an entity */
} else if (outputmode == 3 && a[0] == ' ') {
StringCat(output, "%20");
} else if (outputmode >= 2
&& ((unsigned char) a[0]) < 32) {
} else if (outputmode >= 2 &&
((unsigned char) a[0]) < 32) {
char tmp[32];
sprintf(tmp, "%%%02x", (unsigned char) a[0]);
@@ -1278,20 +1415,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
StringClear(tmpbuff);
break;
case '<':
StringCat(tmpbuff, "&lt;");
break;
case '>':
StringCat(tmpbuff, "&gt;");
break;
case '&':
StringCat(tmpbuff, "&amp;");
break;
case '\'':
StringCat(tmpbuff, "&#39;");
break;
default:
StringMemcat(tmpbuff, fstr, 1);
if (!cat_html_escaped(&tmpbuff, *fstr)) {
StringMemcat(tmpbuff, fstr, 1);
}
break;
}
fstr++;
@@ -1331,7 +1458,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
#endif
} else {
if (is_text(file)) {
if (is_html(file)) {
StringMemcat(headers, ok, sizeof(ok) - 1);
} else if (is_text(file)) {
StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
} else if (is_js(file)) {
StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
@@ -1368,6 +1497,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringCat(output, error);
}
}
} else if (denied) {
StringCat(headers, "HTTP/1.0 403 Forbidden\r\n"
"Server: httrack small server\r\n"
"Content-type: text/html\r\n");
StringCat(output, "Missing or invalid session id.\r\n");
} else {
#ifdef _DEBUG
char error_hdr[] =
@@ -1433,6 +1567,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringFree(tmpbuff);
StringFree(tmpbuff2);
StringFree(fspath);
StringFree(website);
if (buffer)
free(buffer);

View File

@@ -43,8 +43,11 @@ Please visit our Website: http://www.httrack.com
// Fonctions
void socinput(T_SOC soc, char *s, int max);
T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort);
T_SOC smallserver_init(int *port, char *adr);
/* Listen on bindAddr, or every interface if NULL/empty; adr (>= 258 bytes) gets
the address to advertise. INVALID_SOCKET on error. */
T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort,
const char *bindAddr);
T_SOC smallserver_init(int *port, char *adr, const char *bindAddr);
int smallserver(T_SOC soc, char *url, char *method, char *data, char *path);
#define CATCH_RESPONSE \

View File

@@ -1318,19 +1318,19 @@ HTSEXT_API hts_boolean hts_findnext(find_handle find) {
if (find) {
#ifdef _WIN32
if ((FindNextFileA(find->handle, &find->hdata)))
return 1;
return HTS_TRUE;
#else
char catbuff[CATBUFF_SIZE];
memset(&(find->filestat), 0, sizeof(find->filestat));
if ((find->dirp = readdir(find->hdir)))
if (find->dirp->d_name)
if (!STAT
(concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name), &find->filestat))
return 1;
if (!STAT(
concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name),
&find->filestat))
return HTS_TRUE;
#endif
}
return 0;
return HTS_FALSE;
}
HTSEXT_API int hts_findclose(find_handle find) {

View File

@@ -462,22 +462,6 @@ static void warc_make_id(warc_writer *w, char out[64]) {
b[11], b[12], b[13], b[14], b[15]);
}
static void warc_now_iso8601(char out[32]) {
time_t t = time(NULL);
struct tm tmv;
#if defined(_WIN32)
struct tm *g = gmtime(&t);
if (g != NULL)
tmv = *g;
else
memset(&tmv, 0, sizeof(tmv));
#else
if (gmtime_r(&t, &tmv) == NULL)
memset(&tmv, 0, sizeof(tmv));
#endif
strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
}
/* Case-insensitive "is this the header named name?" test, tolerating optional
whitespace before the ':' (non-compliant "Name : value" is still matched). */
static int header_is(const char *line, size_t line_len, const char *name) {
@@ -1063,7 +1047,7 @@ static void warc_wacz_package(warc_writer *w) {
wbuf_free(&pages);
/* datapackage.json listing every stored file with its sha256 + size. */
warc_now_iso8601(created);
hts_now_iso8601(created);
memset(&dp, 0, sizeof(dp));
if (!err &&
(wbuf_printf(&dp,
@@ -1166,7 +1150,7 @@ static int warc_emit(warc_writer *w, const char *type, const char *content_type,
memset(&hdr, 0, sizeof(hdr));
warc_make_id(w, id);
warc_now_iso8601(date);
hts_now_iso8601(date);
#if HTS_USEOPENSSL
/* Block digest over the whole block, in one streaming pass. */
@@ -1670,7 +1654,7 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
char date[32];
char *slot;
size_t need;
warc_now_iso8601(date);
hts_now_iso8601(date);
need = strlen(target_uri) + 1 + strlen(date) + 1;
slot = malloct(need);
if (slot != NULL) {

View File

@@ -62,6 +62,7 @@ Please visit our Website: http://www.httrack.com
#include "htsmd5.c"
#include "md5.c"
#include "htscmdline.h"
#include "htsserver.h"
#include "htsurlport.h"
#include "htsweb.h"
@@ -88,7 +89,7 @@ Please visit our Website: http://www.httrack.com
static htsmutex refreshMutex = HTSMUTEX_INIT;
static int help_server(char *dest_path, int defaultPort);
static int help_server(char *dest_path, int defaultPort, const char *bindAddr);
extern int commandRunning;
extern int commandEnd;
extern int commandReturn;
@@ -153,6 +154,8 @@ int main(int argc, char *argv[]) {
int ret = 0;
int defaultPort = 0;
int parentPid = 0;
/* loopback by default: the handler trusts its input; --bind widens it */
const char *bindAddr = "127.0.0.1";
printf("Initializing the server..\n");
@@ -179,7 +182,8 @@ int main(int argc, char *argv[]) {
if (argc < 2 || (argc % 2) != 0) {
fprintf(stderr, "** Warning: use the webhttrack frontend if available\n");
fprintf(stderr,
"usage: %s [--port <port>] [--ppid parent-pid] <path-to-html-root-dir> [key value [key value]..]\n",
"usage: %s [--port <port>] [--bind <address>] [--ppid parent-pid] "
"<path-to-html-root-dir> [key value [key value]..]\n",
argv[0]);
fprintf(stderr, "example: %s /usr/share/httrack/\n", argv[0]);
return 1;
@@ -267,6 +271,14 @@ int main(int argc, char *argv[]) {
fprintf(stderr, "couldn't set the port number to %s\n", argv[i + 1]);
return -1;
}
} else if (strcmp(argv[i], "--bind") == 0 && i + 1 < argc) {
/* empty would fall back to every interface, silently undoing the default
*/
if (!strnotempty(argv[i + 1])) {
fprintf(stderr, "--bind needs an address\n");
return -1;
}
bindAddr = argv[i + 1];
} else if (strcmp(argv[i], "--ppid") == 0 && i + 1 < argc) {
if (sscanf(argv[i + 1], "%u", &parentPid) != 1) {
fprintf(stderr, "couldn't set the parent PID to %s\n", argv[i + 1]);
@@ -293,7 +305,7 @@ int main(int argc, char *argv[]) {
}
/* launch */
ret = help_server(argv[1], defaultPort);
ret = help_server(argv[1], defaultPort, bindAddr);
htsthread_wait_n(background_threads - 1);
hts_uninit();
@@ -308,10 +320,8 @@ int main(int argc, char *argv[]) {
static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
static void back_launch_cmd(void *pP) {
char *cmd = (char *) pP;
char **argv = (char **) malloct(1024 * sizeof(char *));
char **argv;
int argc = 0;
int i = 0;
int g = 0;
//
httrackp *opt;
@@ -322,28 +332,19 @@ static void back_launch_cmd(void *pP) {
commandReturnCmdl = strdup(cmd);
/* split */
argv[0] = strdup("webhttrack");
argv[1] = cmd;
argc++;
i = 0;
while(cmd[i]) {
if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
cmd[i] = ' ';
}
i++;
}
i = 0;
while(cmd[i]) {
if (cmd[i] == '\"')
g = !g;
if (cmd[i] == ' ') {
if (!g) {
cmd[i] = '\0';
argv[argc++] = cmd + i + 1;
}
}
i++;
argv = hts_split_cmdline(cmd, &argc);
if (argv == NULL) {
if (commandReturnMsg)
free(commandReturnMsg);
commandReturnMsg = strdup("could not parse the command line");
commandReturn = -1;
commandRunning = 0;
commandEnd = 1;
free(cmd);
return;
}
/* drop the program name the posted command line carries */
argv[0] = strdupt("webhttrack");
/* init */
hts_init();
@@ -372,6 +373,7 @@ static void back_launch_cmd(void *pP) {
/* free */
free(cmd);
freet(argv[0]);
freet(argv);
return;
}
@@ -427,11 +429,11 @@ static int webhttrack_runmain(httrackp * opt, int argc, char **argv) {
return ret;
}
static int help_server(char *dest_path, int defaultPort) {
static int help_server(char *dest_path, int defaultPort, const char *bindAddr) {
int returncode = 0;
char adr_prox[HTS_URLMAXSIZE * 2];
int port_prox;
T_SOC soc = smallserver_init_std(&port_prox, adr_prox, defaultPort);
T_SOC soc = smallserver_init_std(&port_prox, adr_prox, defaultPort, bindAddr);
if (soc != INVALID_SOCKET) {
char url[HTS_URLMAXSIZE * 2];

View File

@@ -42,7 +42,7 @@ Please visit our Website: http://www.httrack.com
typedef struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[256];
char state[288]; // a short label plus back->info[256]
char url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char url_adr[HTS_URLMAXSIZE * 2];
char url_fil[HTS_URLMAXSIZE * 2];

View File

@@ -46,6 +46,7 @@ Please visit our Website: http://www.httrack.com
#include "httrack.h"
#include "htslib.h"
#include "htscharset.h" // after htslib.h: winsock2.h must precede windows.h
#include "htsbacktrace.h"
/* Static definitions */
static int fexist(const char *s);
@@ -72,10 +73,6 @@ static int linput(FILE * fp, char *s, int max);
#include <sys/ioctl.h>
#endif
#include <ctype.h>
#if (defined(__linux) && defined(HAVE_EXECINFO_H))
#include <execinfo.h>
#define USES_BACKTRACE
#endif
/* END specific definitions */
static void __cdecl htsshow_init(t_hts_callbackarg * carg);
@@ -880,21 +877,6 @@ static void sig_doback(int blind) { // mettre en backing
#undef FD_ERR
#define FD_ERR 2
static void print_backtrace(void) {
#ifdef USES_BACKTRACE
void *stack[256];
const int size = backtrace(stack, sizeof(stack)/sizeof(stack[0]));
if (size != 0) {
backtrace_symbols_fd(stack, size, FD_ERR);
}
#else
const char msg[] = "No stack trace available on this OS :(\n";
if (write(FD_ERR, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
/* sorry GCC */
}
#endif
}
static size_t print_num(char *buffer, int num) {
size_t i, j;
if (num < 0) {
@@ -928,7 +910,7 @@ static void sig_fatal(int code) {
size += print_num(&buffer[size], code);
buffer[size++] = '\n';
(void) (write(FD_ERR, buffer, size) == size);
print_backtrace();
hts_print_backtrace(FD_ERR);
(void) (write(FD_ERR, msgreport, sizeof(msgreport) - 1)
== sizeof(msgreport) - 1);
abort();
@@ -951,6 +933,7 @@ static void sig_leave(int code) {
}
static void signal_handlers(void) {
hts_backtrace_init();
#ifdef _WIN32
signal(SIGINT, sig_leave); // ^C
signal(SIGTERM, sig_finish); // kill <process>

View File

@@ -44,7 +44,7 @@ typedef struct t_StatsBuffer t_StatsBuffer;
struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[256];
char state[288]; // a short label plus back->info[256]
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];

View File

@@ -100,6 +100,7 @@
<ItemGroup>
<ClCompile Include="httrack.c" />
<ClCompile Include="htsbacktrace.c" />
</ItemGroup>
<!-- Pulls in libhttrack.lib and builds it first. -->

View File

@@ -59,9 +59,10 @@
<ItemDefinitionGroup>
<ClCompile>
<!-- LIBHTTRACK_EXPORTS turns HTSEXT_API into __declspec(dllexport); ZLIB_DLL
imports zlib from its DLL. HTS_USEBROTLI/HTS_USEZSTD enable the br and
zstd content codings. Windows 7 floor, matching WinHTTrack. -->
<PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
imports zlib from its DLL, ZLIB_CONST makes its next_in const as the
autotools build does. HTS_USEBROTLI/HTS_USEZSTD enable the br and zstd
content codings. Windows 7 floor, matching WinHTTrack. -->
<PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;ZLIB_CONST;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<AdditionalIncludeDirectories>$(MSBuildThisFileDirectory);$(MSBuildThisFileDirectory)coucal;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
<WarningLevel>Level3</WarningLevel>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
@@ -121,6 +122,7 @@
<ClCompile Include="htshelp.c" />
<ClCompile Include="htsindex.c" />
<ClCompile Include="htslib.c" />
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="htsmodules.c" />
@@ -136,6 +138,7 @@
<ClCompile Include="htswrap.c" />
<ClCompile Include="htszlib.c" />
<ClCompile Include="htswarc.c" />
<ClCompile Include="htschanges.c" />
<ClCompile Include="md5.c" />
<ClCompile Include="minizip\ioapi.c" />
<ClCompile Include="minizip\iowin32.c" />

View File

@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
PT_Element elt = PT_ElementNew();
elt->statuscode = 405;
strcpy(elt->msg, "Method Not Allowed");
strcpybuff(elt->msg, "Method Not Allowed");
return elt;
}
@@ -771,7 +771,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
PT_ReadIndex(indexes, StringBuff(itemUrl) + 1, FETCH_HEADERS);
if (file != NULL && file->statuscode == HTTP_OK) {
size = file->size;
if (file->lastmodified) {
if (file->lastmodified[0] != '\0') {
timestamp = get_time_rfc822(file->lastmodified);
}
if (timestamp == (time_t) 0) {
@@ -785,7 +785,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
}
timestamp = timestampRep;
}
if (file->contenttype) {
if (file->contenttype[0] != '\0') {
mimeType = file->contenttype;
}
}
@@ -822,7 +822,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
elt->statuscode = 207; /* Multi-Status */
strcpy(elt->charset, "utf-8");
strcpy(elt->contenttype, "text/xml");
strcpy(elt->msg, "Multi-Status");
strcpybuff(elt->msg, "Multi-Status");
StringFree(response);
fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
@@ -879,7 +879,7 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
elt->statuscode = HTTP_OK;
strcpy(elt->charset, "iso-8859-1");
strcpy(elt->contenttype, "text/html");
strcpy(elt->msg, "OK");
strcpybuff(elt->msg, "OK");
StringFree(html);
return elt;
}

View File

@@ -407,7 +407,7 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
elt->statuscode = HTTP_OK;
strcpy(elt->charset, "iso-8859-1");
strcpy(elt->contenttype, "text/html");
strcpy(elt->msg, "OK");
strcpybuff(elt->msg, "OK");
StringFree(html);
return elt;
}
@@ -579,9 +579,10 @@ PT_Index PT_LoadCache(const char *filename) {
if (chain != NULL) {
const char *scheme = link_has_authority(chain->name) ? "" : "http://";
snprintf(index->slots.common.startUrl,
sizeof(index->slots.common.startUrl), "%s%s", scheme,
(const char *) chain->name);
/* dropped rather than truncated: empty already reads as "unset" */
if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
(const char *) chain->name))
index->slots.common.startUrl[0] = '\0';
}
}
}
@@ -827,6 +828,18 @@ PT_Element PT_ElementNew(void) {
return r;
}
/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
static void PT_Element_failf(PT_Element r, const char *fmt, ...)
HTS_PRINTF_FUN(2, 3);
static void PT_Element_failf(PT_Element r, const char *fmt, ...) {
va_list args;
va_start(args, fmt);
(void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
va_end(args);
}
PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
if (index != NULL && SAFE_INDEX(index)) {
return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
@@ -865,12 +878,17 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
(headersSize) += (int) strlen(headers + headersSize); \
} while(0)
#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strcpy(refvalue, value); \
line[0] = '\0'; \
} \
} while(0)
/* refvalue_size is mandatory: the cache line is bounded only by the line
buffer, not by the destination. Clip rather than reject, since an
engine-written field can be wider than ours. */
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(refvalue)[0] = '\0'; \
strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1); \
line[0] = '\0'; \
} \
} while (0)
#define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
int intval = 0; \
@@ -972,9 +990,12 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
const char *scheme =
link_has_authority(filenameIndex) ? "" : "http://";
firstSeen = 1;
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
scheme, filenameIndex);
/* dropped rather than truncated; try the next entry */
if (sprintfbuff(index->startUrl, "%s%s", scheme,
filenameIndex))
firstSeen = 1;
else
index->startUrl[0] = '\0';
}
}
} else {
@@ -1070,16 +1091,23 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
value++;
ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg); // msg
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
sizeof(r->msg));
ZIP_READFIELD_INT(line, value, "X-Size", r->size); // size
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset); // contenttype
ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
ZIP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
ZIP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
sizeof(r->contenttype));
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
sizeof(r->charset));
ZIP_READFIELD_STRING(line, value, "Last-Modified",
r->lastmodified, sizeof(r->lastmodified));
ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
sizeof(r->etag));
ZIP_READFIELD_STRING(line, value, "Location", r->location,
sizeof(location_default));
ZIP_READFIELD_STRING(line, value, "Content-Disposition",
r->cdispo); // Content-disposition
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_); // Original save filename
r->cdispo, sizeof(r->cdispo));
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
sizeof(previous_save_));
if (line[0] != '\0') {
int len = r->headers ? ((int) strlen(r->headers)) : 0;
int nlen =
@@ -1135,8 +1163,9 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
snprintf(previous_save, sizeof(previous_save), "%s%s",
index->path, previous_save_ + index->fixedPath);
} else {
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
PT_Element_failf(
r, "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
r->statuscode = STATUSCODE_INVALID;
}
} else {
@@ -1155,7 +1184,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
// Peut-on stocker le fichier directement sur disque?
if (ok) {
if (r->msg[0] == '\0') {
strcpy(r->msg, "Cache Read Error : Unexpected error");
strcpybuff(r->msg, "Cache Read Error : Unexpected error");
}
} else { // lire en mémoire
@@ -1173,24 +1202,27 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
int last_errno = errno;
r->statuscode = STATUSCODE_INVALID;
sprintf(r->msg, "Read error in cache disk data: %s",
strerror(last_errno));
PT_Element_failf(r,
"Read error in cache disk data: %s",
strerror(last_errno));
}
r->adr[r->size] = '\0';
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg,
"Read error (memory exhausted) from cache");
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
}
fclose(fp);
} else {
r->statuscode = STATUSCODE_INVALID;
snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
file_convert(catbuff, sizeof(catbuff), previous_save));
PT_Element_failf(
r, "Read error (can't open '%s') from cache",
file_convert(catbuff, sizeof(catbuff),
previous_save));
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cached file name is invalid");
strcpybuff(r->msg, "Cached file name is invalid");
}
}
} else {
@@ -1202,12 +1234,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
free(r->adr);
r->adr = NULL;
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Data");
strcpybuff(r->msg, "Cache Read Error : Read Data");
} else
*(r->adr + r->size) = '\0';
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Memory Error");
strcpybuff(r->msg, "Cache Memory Error");
}
}
}
@@ -1215,21 +1247,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
} // si save==null, ne rien charger (juste en tête)
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header Data");
strcpybuff(r->msg, "Cache Read Error : Read Header Data");
}
unzCloseCurrentFile(index->zFile);
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Open File");
strcpybuff(r->msg, "Cache Read Error : Open File");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Bad Offset");
strcpybuff(r->msg, "Cache Read Error : Bad Offset");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "File Cache Entry Not Found");
strcpybuff(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -1570,9 +1602,11 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
const char *scheme =
link_has_authority(line) ? "" : "http://";
firstSeen = 1;
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
scheme, line);
/* dropped rather than truncated; try the next entry */
if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
firstSeen = 1;
else
index->startUrl[0] = '\0';
}
}
@@ -1793,8 +1827,9 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
snprintf(previous_save, sizeof(previous_save), "%s%s",
index->path, previous_save_ + index->fixedPath);
} else {
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
PT_Element_failf(r,
"Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
r->statuscode = STATUSCODE_INVALID;
}
} else {
@@ -1820,17 +1855,18 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
if (r->adr != NULL) {
if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Read error in cache disk data");
strcpybuff(r->msg, "Read error in cache disk data");
}
r->adr[r->size] = '\0';
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Read error (memory exhausted) from cache");
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
}
fclose(fp);
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Previous cache file not found (2)");
strcpybuff(r->msg, "Previous cache file not found (2)");
}
}
} else {
@@ -1842,30 +1878,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
free(r->adr);
r->adr = NULL;
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Data");
strcpybuff(r->msg, "Cache Read Error : Read Data");
} else
r->adr[r->size] = '\0';
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Memory Error");
strcpybuff(r->msg, "Cache Memory Error");
}
}
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Bad Data");
strcpybuff(r->msg, "Cache Read Error : Bad Data");
}
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header");
strcpybuff(r->msg, "Cache Read Error : Read Header");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Seek Failed");
strcpybuff(r->msg, "Cache Read Error : Seek Failed");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "File Cache Entry Not Found");
strcpybuff(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -2128,12 +2164,15 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
return 0;
}
#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strcpy(refvalue, value); \
line[0] = '\0'; \
} \
} while(0)
/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(refvalue)[0] = '\0'; \
strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1); \
line[0] = '\0'; \
} \
} while (0)
#define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
int intval = 0; \
@@ -2202,11 +2241,16 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
*value = '\0';
for(value++; *value == ' ' || *value == '\t'; value++) ;
HTTP_READFIELD_INT(line, value, "Content-Length", r->size); // size
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
HTTP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
HTTP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo); // Content-disposition
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
sizeof(r->contenttype));
HTTP_READFIELD_STRING(line, value, "Last-Modified",
r->lastmodified, sizeof(r->lastmodified));
HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
sizeof(r->etag));
HTTP_READFIELD_STRING(line, value, "Location", r->location,
sizeof(location_default));
HTTP_READFIELD_STRING(line, value, "Content-Disposition",
r->cdispo, sizeof(r->cdispo));
if (line[0] != '\0') {
int len = r->headers ? ((int) strlen(r->headers)) : 0;
int nlen =
@@ -2246,7 +2290,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
fetchSize = dataLength - metaSize;
} else if (fetchSize > dataLength - metaSize) {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Truncated Data");
strcpybuff(r->msg, "Cache Read Error : Truncated Data");
}
r->size = 0;
if (r->statuscode != STATUSCODE_INVALID) {
@@ -2259,12 +2303,13 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
int last_errno = errno;
r->statuscode = STATUSCODE_INVALID;
sprintf(r->msg, "Read error in cache disk data: %s",
strerror(last_errno));
PT_Element_failf(r, "Read error in cache disk data: %s",
strerror(last_errno));
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Read error (memory exhausted) from cache");
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
}
}
}
@@ -2272,21 +2317,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header Error");
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header Error");
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Seek Error");
strcpybuff(r->msg, "Cache Read Error : Seek Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "File Cache Entry Not Found");
strcpybuff(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);

View File

@@ -97,6 +97,7 @@
<ItemGroup>
<ClCompile Include="htsserver.c" />
<ClCompile Include="htsweb.c" />
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
</ItemGroup>

View File

@@ -0,0 +1,7 @@
#!/bin/bash
#
set -euo pipefail
# --changes bucket accounting and JSON escaping (#714).
httrack -O /dev/null -#test=changes run | grep -q "changes self-test: OK"

View File

@@ -0,0 +1,8 @@
#!/bin/bash
#
set -euo pipefail
# webhttrack posts its command line as one string: the argv split must grow past
# 1024 arguments and keep a quote inside a value out of the option parser.
httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

View File

@@ -0,0 +1,25 @@
#!/bin/bash
#
# Buffer capacity for a 64-bit file size ('httrack -#test=growsize'): a -%S list
# file past 4GB must size its buffer exactly or be refused, never wrap.
set -euo pipefail
out=$(httrack -#test=growsize)
echo "$out"
test "$out" == "growsize self-test OK"
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_growsize.XXXXXX")
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
echo '<html><body>hi</body></html>' >"$tmp/index.html"
printf -- '-*/zzmarker*\n' >"$tmp/rules.txt"
# the rules file lands in the URL/filter string, echoed back by the banner
run=$(httrack -O "$tmp/out" --quiet -n "-%S" "$tmp/rules.txt" \
"file://$tmp/index.html" 2>&1) || true
printf '%s\n' "$run" | grep -q 'zzmarker' || {
echo "FAIL: -%S rules file was not loaded"
printf '%s\n' "$run"
exit 1
}

View File

@@ -60,13 +60,20 @@ test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
# Post a "start" whose -O dir is 'café' in the form's ISO-8859-1 charset.
"${python}" - "${url}" "${work}" <<'PY'
import sys, urllib.parse, urllib.request
import re, sys, urllib.parse, urllib.request
url, work = sys.argv[1], sys.argv[2]
outdir = work + "/caf\xe9" # 'café' as the single byte the browser would send
# Port 1 refuses at once: only the decoded -O dir matters, not the fetch.
cmd = "httrack --quiet --robots=0 http://127.0.0.1:1/x.html -O " + outdir
fields = [("path", work), ("projname", "proj"), ("command_do", "start"),
# The body is refused without the session id the server renders into each form.
# Note the wizard page is under /server/; a bare /step4.html is the doc page.
page = urllib.request.urlopen(url + "server/step4.html", timeout=20).read()
m = re.search(rb'name="sid" value="([0-9a-f]+)"', page)
if not m:
raise SystemExit("no session id in server/step4.html")
fields = [("sid", m.group(1).decode()),
("path", work), ("projname", "proj"), ("command_do", "start"),
("winprofile", "x"), ("command", cmd)]
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="", encoding="latin-1"))
for k, v in fields)

View File

@@ -0,0 +1,185 @@
#!/bin/bash
#
# htsserver's POST redirect: the Location header is built from a client-supplied
# value, and the listen address is loopback unless --bind widens it.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# run_with_timeout/kill_tree: timeout(1) is absent on macOS
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
log=$(mktemp)
# start() runs inside a command substitution, so its $! never reaches this
# shell. Take the pid from the server's own announcement instead: a missed kill
# leaves an orphan holding the CI job open long after the suite has passed.
srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
cleanup() {
stop
rm -f "${log}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Start htsserver, echo the announced URL. Extra args are passed through.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" "$@" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
stop() {
local pid
pid=$(srvpid)
test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
}
# The session id gates the request body, so a POST has to carry one. The server
# renders it into every form, which is where a browser picks it up too.
scrape_sid() {
python3 -c 'import socket, sys
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(20)
s.sendall(b"GET /server/index.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n")
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
}
# POST redirect=$1 to 127.0.0.1:$2 with session id $3, print the raw headers.
post_redirect() {
python3 -c 'import socket, sys, urllib.parse
body = ("sid=" + sys.argv[3] + "&redirect="
+ urllib.parse.quote(sys.argv[1], safe=""))
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", int(sys.argv[2])), 10)
s.settimeout(20)
s.sendall(req.encode())
out = b""
# stop at the end of the header block: the server need not close the connection,
# and an unbounded recv() hung the macOS runner for over an hour.
while b"\r\n\r\n" not in out:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.split(b"\r\n\r\n")[0].decode("latin-1"))' "$1" "$2" "$3"
}
portof() { echo "${1##*:}" | tr -d /; }
# "free"/"inuse"/"unusable": can $1 still be bound on 127.0.0.2? A wildcard
# listener takes the port on every address, a loopback-only one does not, so
# this discriminates where the announced URL cannot.
probe_alias() {
python3 -c 'import socket, sys
s = socket.socket()
try:
s.bind(("127.0.0.2", int(sys.argv[1])))
except OSError as e:
print("unusable" if e.errno in (99, 49, 10049) else "inuse")
else:
print("free")
finally:
s.close()' "$1"
}
# Non-repeating and not a round number: a truncation, a reorder or a dropped
# interior byte all stay visible.
long=$(python3 -c 'print("".join(chr(33 + i % 90) for i in range(4097)))')
url=$(start)
port=$(portof "${url}")
sid=$(scrape_sid "${port}")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
resp=$(post_redirect "${long}" "${port}" "${sid}") ||
fail "no response to an oversized redirect value"
stop
loc=$(printf '%s' "${resp}" | sed -n 's/^Location: //p' | tr -d '\r')
test "${loc}" = "${long}" ||
fail "oversized redirect not echoed whole (got ${#loc} bytes, want ${#long})"
# CR/LF must suppress the header outright. Sanitising it instead would keep the
# grep for an injected header quiet while still emitting a mangled Location.
url=$(start)
port=$(portof "${url}")
sid=$(scrape_sid "${port}")
resp=$(post_redirect '/foo
X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
stop
printf '%s' "${resp}" | grep -qi 'X-Injected' &&
fail "CRLF in the redirect value reached the response"
test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
fail "CRLF redirect still emitted a Location header"
test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
fail "CRLF redirect did not yield exactly one status line"
# Loopback unless asked otherwise. Assert the socket, not the announcement.
url=$(start)
port=$(portof "${url}")
alias_state=$(probe_alias "${port}")
stop
if test "${alias_state}" = unusable; then
echo "no 127.0.0.2 alias; skipping the listen-address assertions" >&2
else
test "${alias_state}" = free ||
fail "default listen address is not loopback-only (127.0.0.2:${port} taken)"
case "${url}" in
http://127.0.0.1:*) ;;
*) fail "default announcement is not loopback: ${url}" ;;
esac
url=$(start --bind 0.0.0.0)
port=$(portof "${url}")
alias_state=$(probe_alias "${port}")
stop
test "${alias_state}" = inuse ||
fail "--bind 0.0.0.0 did not take the wildcard (127.0.0.2:${port} ${alias_state})"
fi
# An empty --bind must not quietly fall back to every interface. Bounded: if it
# were accepted the server would listen instead of exiting.
run_with_timeout 15 htsserver "${distdir}/" --bind "" >"${log}" 2>&1 || true
grep -q -- "--bind needs an address" "${log}" ||
fail "empty --bind not refused: $(cat "${log}")"
echo "PASS"

View File

@@ -0,0 +1,142 @@
#!/bin/bash
#
# htsserver's session id must gate the request body: every field in it is
# written into the global key store, and "command" from there reaches the engine.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
# The server reports its own pid; the aliveness assertions below hang off it.
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
# default index). Prints the reply.
request() {
python3 -c 'import socket, sys
port, body = int(sys.argv[1]), sys.argv[2]
page = sys.argv[3] if len(sys.argv) > 3 else "index"
if body:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
else:
req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
}
# The server hands the sid to any client in every form; scraping it is the
# legitimate flow, and it is what makes the accept case below meaningful.
scrape_sid() {
request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
alive "${srv}" || fail "htsserver is not running"
sid=$(scrape_sid "${port}")
# Control: without a real sid every assertion below would pass vacuously.
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
# Accept: the legitimate flow still works. "redirect" is the cheapest field
# with a reply that is visible in the headers.
resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
fail "a body carrying the correct sid was refused"
# Refuse: missing and wrong. A missing one is the regression under test — the
# expected value used to be pre-seeded into the compared key, so omitting the
# field compared equal to itself.
for bad in "redirect=/nosid" "sid=&redirect=/empty" \
"sid=00000000000000000000000000000000&redirect=/wrong"; do
resp=$(request "${port}" "${bad}")
printf '%s' "${resp}" | grep -q '^Location:' &&
fail "body accepted without a valid sid: ${bad}"
# A refusal has to be a well-formed reply, not a headerless fragment: the
# release build used to emit only Content-length, which reads as a protocol
# error to any client and hides the reason.
printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
fail "refusal was not a 403: ${bad}"
done
# Suppressing the reply is not the same as refusing the write: the body is
# applied to one global key store that later requests render from, and the
# command dispatcher reads it from there. Probe the store itself — step3
# interpolates ${projname} into its title — rather than the refused reply.
title() { request "$1" "" step3; }
request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
title "${port}" | grep -q 'UNAUTHWRITE' &&
fail "a body without a valid sid was written to the key store"
# Paired accept case: without it the assertion above passes even if the server
# simply ignores every body, which would prove nothing.
request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
title "${port}" | grep -q 'AUTHWRITE' ||
fail "a body carrying the correct sid was not written to the key store"
echo "PASS"

View File

@@ -0,0 +1,90 @@
#!/bin/bash
#
# proxytrack's WebDAV PROPFIND fallback must behave identically whether or not
# the cache entry supplies Content-Type and Last-Modified.
set -euo pipefail
: "${top_srcdir:=..}"
# shellcheck source=tests/testlib.sh
. "$top_srcdir/tests/testlib.sh"
python=$(find_python) || {
echo "python3 missing, skipping"
exit 77
}
command -v curl >/dev/null 2>&1 || {
echo "curl missing, skipping"
exit 77
}
# First test to run proxytrack as a live server, and MSYS cannot reap a native
# listener: the orphan wedged the whole Windows suite for 48 minutes (#595).
if is_windows; then
echo "windows: cannot reap a backgrounded proxytrack, skipping"
exit 77
fi
dir=$(mktemp -d)
ptpid=
cleanup() {
stop_server "$ptpid"
rm -rf "$dir"
}
trap cleanup EXIT
# Neither header is present, so file->contenttype and ->lastmodified stay "".
printf 'HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
printf 'hello' >"$dir/body"
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
{
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
printf '2 0 test\n'
printf '\n\n'
printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
cat "$dir/hdr" "$dir/body"
} >"$dir/in.arc"
freeport() {
"$python" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'
}
proxyport=$(freeport)
icpport=$(freeport)
proxytrack "127.0.0.1:$proxyport" "127.0.0.1:$icpport" "$dir/in.arc" >"$dir/pt.log" 2>&1 &
ptpid=$!
waited=0
until grep -qE "HTTP Proxy installed on|Unable to (initialize a temporary server|create the server)" "$dir/pt.log"; do
kill -0 "$ptpid" 2>/dev/null || {
echo "FAIL: proxytrack exited before listening"
cat "$dir/pt.log"
exit 1
}
test "$waited" -lt 50 || {
echo "FAIL: proxytrack never announced its listen port"
exit 1
}
sleep 0.1
waited=$((waited + 1))
done
grep -q "HTTP Proxy installed on" "$dir/pt.log" || {
echo "FAIL: proxytrack failed to bind"
cat "$dir/pt.log"
exit 1
}
# --max-time: an unbounded read here would wedge the runner rather than fail
curl -s --max-time 30 -X PROPFIND -H "Depth: 1" \
"http://127.0.0.1:$proxyport/webdav/example.com/" >"$dir/resp.xml"
grep -q '<getcontenttype>application/octet-stream</getcontenttype>' "$dir/resp.xml" || {
echo "FAIL: missing Content-Type did not fall back to application/octet-stream"
cat "$dir/resp.xml"
exit 1
}
grep -q '<getlastmodified>Wed, 01 Jan 2025 00:00:00 GMT</getlastmodified>' "$dir/resp.xml" || {
echo "FAIL: missing Last-Modified did not fall back to the index timestamp"
cat "$dir/resp.xml"
exit 1
}
echo "OK: WebDAV listing falls back correctly for an entry with no Content-Type/Last-Modified"

View File

@@ -0,0 +1,97 @@
#!/bin/bash
# A fatal signal prints the raw backtrace, then names the frames
# backtrace_symbols_fd() leaves as module+offset (-fvisibility=hidden hides them).
set -eu
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
if is_windows; then
echo "no backtrace() on Windows; skipping" >&2
exit 77
fi
command -v httrack >/dev/null || {
echo "could not find httrack" >&2
exit 1
}
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_symbolize.XXXXXX") || exit 1
trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM
out="${tmpdir}/trace"
raw="${tmpdir}/trace-optout"
overflow="this string is far too long for the buffer"
# A frame glibc could not name. Stop at the offset: it prints the trailing
# "[0xADDR]" with or without a leading space depending on its version.
rawframe='(+0x[0-9a-f][0-9a-f]*)'
# The strsafe selftest aborts on purpose, which lands in sig_fatal.
rc=0
run_with_timeout 60 httrack -#test=strsafe overflow "$overflow" >"$out" 2>&1 || rc=$?
test "$rc" -ne 124 || {
echo "the crash handler did not finish within the deadline" >&2
exit 1
}
grep -q '^Caught signal ' "$out" || {
echo "no 'Caught signal' line:" >&2
cat "$out" >&2
exit 1
}
if grep -q 'No stack trace available on this OS' "$out"; then
echo "no backtrace() on this platform; skipping" >&2
exit 77
fi
# Unconditional and first: symbolizing must never cost the raw trace.
grep -q "$rawframe" "$out" || {
echo "raw module+offset frames are gone:" >&2
cat "$out" >&2
exit 1
}
command -v addr2line >/dev/null || {
echo "addr2line not installed; skipping" >&2
exit 77
}
# Control: resolve the same frames ourselves, so a stripped build cannot turn
# the assertion below into a vacuous pass.
oracle="${tmpdir}/oracle"
sed -n 's/^\([^(]*\)(+\(0x[0-9a-f]*\)).*$/\1 \2/p' "$out" |
while read -r mod off; do
test -f "$mod" || continue
addr2line -Cfip -e "$mod" "$off" 2>/dev/null || true
done >"$oracle"
grep -qE 'st_strsafe|strcpy_safe_' "$oracle" || {
echo "addr2line names no hidden symbol in this build; skipping" >&2
exit 77
}
# The payload. Dropping the raw frames first is what makes it specific: both
# names are static, so .dynsym could never have carried them. Not anchored on
# the "0xOFF:" line, the inline chain puts the name on either half.
grep -v "$rawframe" "$out" | grep -qE 'st_strsafe|strcpy_safe_' || {
echo "the handler did not name the hidden frames:" >&2
cat "$out" >&2
exit 1
}
# Opt-out: raw trace kept, nothing spawned.
export HTTRACK_NO_SYMBOLIZE=1
rc=0
run_with_timeout 60 httrack -#test=strsafe overflow "$overflow" >"$raw" 2>&1 || rc=$?
unset HTTRACK_NO_SYMBOLIZE
test "$rc" -ne 124 || {
echo "the opt-out run did not finish within the deadline" >&2
exit 1
}
grep -q "$rawframe" "$raw" || {
echo "the opt-out lost the raw trace:" >&2
cat "$raw" >&2
exit 1
}
! grep -q '^0x[0-9a-f]*: ' "$raw" || {
echo "the opt-out still symbolized:" >&2
cat "$raw" >&2
exit 1
}

View File

@@ -0,0 +1,142 @@
#!/bin/bash
#
# The wizard's size fields must render as distinct caps: sizemax is -M, othermax
# then maxhtml share -m.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_maxsize.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
srv=
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
# ~/.httrack.ini out of it.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
# Audit what step4.html renders back; without command_do nothing is launched.
"${python}" - "${url}" <<'PY' || fail "wizard rendered the wrong options (see above)"
import re, sys, urllib.parse, urllib.request
url = sys.argv[1]
html, other, site = "111111", "222222", "333333"
def post(fields):
# The body is refused without the session id the server puts in each form.
form = urllib.request.urlopen(url + "server/index.html", timeout=20).read()
m = re.search(rb'name="sid" value="([0-9a-f]+)"', form)
if not m:
raise SystemExit("no session id in server/index.html")
fields = [("sid", m.group(1).decode())] + fields
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in fields)
req = urllib.request.Request(url + "server/step4.html",
data=body.encode("latin-1"), method="POST")
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
def textarea(page, name):
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
if m is None:
sys.exit("no %s textarea in the rendered step4.html" % name)
return m.group(1)
def want(ok, msg, ctx):
if not ok:
sys.exit("%s\nrendered:%s" % (msg, ctx))
page = post([("maxhtml", html), ("othermax", other), ("sizemax", site),
("dos", "2")])
cmd = textarea(page, "command")
want("--max-size=" + site in cmd, "site cap not emitted as --max-size", cmd)
want("--max-files=" + site not in cmd, "site cap still an --max-files", cmd)
# Positive controls: both per-file caps still ride -m, one option each.
want("--max-files=" + other in cmd, "non-HTML cap not emitted as --max-files",
cmd)
want("--max-files=," + html in cmd, "HTML cap not emitted as --max-files=,",
cmd)
want(cmd.count("--max-files=") == 2, "unexpected --max-files count", cmd)
want(cmd.index("--max-files=" + other) < cmd.index("--max-files=," + html),
"the bare --max-files must precede the --max-files=, form", cmd)
# winprofile.ini feeds the same three caps to the Windows GUI, one key each.
ini = textarea(page, "winprofile").replace("\r\n", "\n") # the ini is CRLF
want("\nDos=2" in ini, "Dos not written from the dos field", ini)
want("\nMaxHtml=" + html + "\n" in ini, "MaxHtml not written from maxhtml", ini)
want("\nMaxOther=" + other + "\n" in ini, "MaxOther not written from othermax",
ini)
want("\nMaxAll=" + site + "\n" in ini, "MaxAll not written from sizemax", ini)
# A bare --max-files= (an unguarded empty field) parses as -m with no digits,
# silently clearing the html cap.
page = post([("maxhtml", ""), ("othermax", ""), ("sizemax", ""), ("dos", "2")])
cmd = textarea(page, "command")
want("--max-rate=" in cmd, "empty size fields rendered no command line", cmd)
want("--max-files" not in cmd and "--max-size" not in cmd,
"empty size fields still rendered a size option", cmd)
# A mistyped ${LANG_OK] key renders the OK button's label empty.
opts = urllib.request.urlopen(url + "server/option2b.html",
timeout=20).read().decode("latin-1")
want("LANG_OK" not in opts, "unexpanded LANG_OK in option2b.html", "")
want('<input type="submit" value="OK"' in opts, "no OK label in option2b.html",
"")
PY
cleanup
# A bare -m<n> resets the html limit, so only the bare-then-comma order keeps
# both caps live. basic.html is 487 bytes, well past the 10-byte html cap.
crawl() {
bash "${distdir}/tests/local-crawl.sh" "$@"
}
crawl --errors 1 --log-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=,10'
crawl --errors 1 --log-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=500000' '--max-files=,10'
crawl --errors 0 --found 'simple/basic.html' --log-not-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=,10' '--max-files=500000'
echo "PASS"

View File

@@ -0,0 +1,135 @@
#!/bin/bash
#
# A browser will not follow an http: page to a file: URL, so the GUI's "browse
# the mirror" link has to go through the server's own /website/ route.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
# Control: on a path that does not resolve, grep "passes" without reading anything.
test -f "${distdir}/html/server/finished.html" || fail "no GUI pages under ${distdir}"
bad=$(grep -l 'file://' "${distdir}"/html/server/finished.html || true)
test -z "${bad}" || fail "file: link left in: ${bad}"
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_browse.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
srvpid=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# Stand in for a finished mirror: the crawl itself is not under test.
proj="${work}/websites/proj"
mkdir -p "${proj}"
printf 'MIRROR-PROBE-OK\n' >"${proj}/probe.txt"
printf '<html><body>MIRROR-INDEX-OK</body></html>\n' >"${proj}/index.html"
# An isolated HOME keeps a stray ~/.httrack.ini out of the server's settings.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
# htsserver resolves the posted paths itself, so hand it native ones.
"${python}" - "${url}" "$(nativepath "${work}/websites")" <<'PY' || fail "browse-link checks failed"
import re, sys, urllib.error, urllib.parse, urllib.request
url, base = sys.argv[1].rstrip("/"), sys.argv[2]
rc = 0
def check(ok, what):
global rc
print(("ok: " if ok else "FAIL: ") + what)
if not ok:
rc = 1
def get(path):
# The UI is served ISO-8859-1, so stay in bytes.
return urllib.request.urlopen(url + path, timeout=20).read()
# No crawl has run, so no commandRunning/commandEnd override swaps the page out.
finished = get("/server/finished.html").decode("latin-1")
# Control: an empty or unrendered page would pass "no file:" on its own.
check("HTTrack Website Copier" in finished, "finished.html rendered")
check("file://" not in finished, "finished.html has no file: link")
# Control: /website/ 404s until a project is set, so the fetches below are real.
try:
get("/website/probe.txt")
check(False, "/website/ served with no project set")
except urllib.error.HTTPError as e:
check(e.code == 404, "/website/ is bound to a project (got %d)" % e.code)
# Point the server at the mirror the way step4.html does; a body without the
# session id is refused outright. Only a profile save records the served root,
# so post one; command_do=save runs no crawl.
sid = re.search(r'name="sid" value="([0-9a-f]+)"', finished)
if sid is None:
print("FAIL: no sid in the rendered form")
sys.exit(1)
fields = [("sid", sid.group(1)), ("path", base), ("projname", "proj"),
("command", "httrack"), ("command_do", "save"), ("winprofile", "x")]
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="")) for k, v in fields)
urllib.request.urlopen(urllib.request.Request(
url + "/server/step4.html", data=body.encode("latin-1"), method="POST"),
timeout=20).read()
check(get("/website/probe.txt") == b"MIRROR-PROBE-OK\n", "mirror file over http")
check("MIRROR-INDEX-OK" in get("/website/index.html").decode("latin-1"),
"mirror index over http")
body = get("/server/finished.html").decode("latin-1")
# Match the anchor by its mirror-path label: the list below it links /website/
# too, so a bare "is the route mentioned" check passes on the unfixed page.
check(re.search(r'href="/website/index\.html"[^>]*>\s*' +
re.escape(base + "/proj"), body) is not None,
"the mirror-path link points at the served mirror")
check("file://" not in body, "finished.html has no file: link")
sys.exit(rc)
PY
# A leaked htsserver wedges the parallel harness behind a green log.
cleanup
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
echo "PASS"

View File

@@ -0,0 +1,148 @@
#!/bin/bash
#
# The wizard renders its httrack command line as one string, which the engine
# splits back into argv: a double quote in a field must reach the split escaped,
# or the rest of the value is parsed as fresh options (-V runs a shell command).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
# default index). Prints the reply.
request() {
python3 -c 'import socket, sys
port, body = int(sys.argv[1]), sys.argv[2]
page = sys.argv[3] if len(sys.argv) > 3 else "index"
if body:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
else:
req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
}
scrape_sid() {
request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
}
# urlencode the key=value pairs given as arguments
formencode() {
python3 -c 'import sys, urllib.parse
print(urllib.parse.urlencode([tuple(a.split("=", 1)) for a in sys.argv[1:]]))' "$@"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
sid=$(scrape_sid "${port}")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
# Fill the wizard fields the command line quotes, then read back the generated
# command line: the user-agent carries a break-out attempt, the footer a
# backslash (which must survive the escape round trip), the project name a plain
# value.
body=$(formencode "sid=${sid}" 'user=Moz" -V "touch /tmp/pwn' 'footer=a\b"c' \
"path=/tmp/p" "projname=plain proj" 'urls=http://x/a"b' 'url2=+*.png"')
request "${port}" "${body}" >/dev/null
cmdline=$(request "${port}" "" step4 |
sed -n '/<textarea name="command"/,/<\/textarea>/p')
# Control: without the fields in the page every assertion below is vacuous.
grep -q -- '--user-agent' <<<"${cmdline}" ||
fail "no --user-agent in the generated command line (probe blind)"
# A plain value is passed through untouched: the escaping must not mangle the
# ordinary case.
grep -qF -- '--path "/tmp/p/plain proj"' <<<"${cmdline}" ||
fail "a plain quoted value was not passed through: ${cmdline}"
# The quote is escaped, so the split keeps it inside the value...
grep -qF -- '--user-agent "Moz\" -V \"touch /tmp/pwn"' <<<"${cmdline}" ||
fail "the quote in the user-agent was not escaped: ${cmdline}"
# ...and the raw form, which would end the argument and hand -V to the option
# parser, is gone.
grep -qF -- '--user-agent "Moz" -V "touch' <<<"${cmdline}" &&
fail "the user-agent still closes its argument early: ${cmdline}"
# A backslash is escaped too, or the split would eat it along with the quote
# that follows.
grep -qF -- '--footer "a\\b\"c"' <<<"${cmdline}" ||
fail "the backslash in the footer was not escaped: ${cmdline}"
# The url and filter fields sit outside quotes, where a backslash cannot escape
# anything: one quote there flips the parity of every quote after it, so the
# escaping above would protect nothing. They must not emit a raw quote at all.
grep -qF -- 'http://x/a%22b' <<<"${cmdline}" ||
fail "the quote in the url field was not neutralised: ${cmdline}"
grep -qF -- '+*.png%22' <<<"${cmdline}" ||
fail "the quote in the filter field was not neutralised: ${cmdline}"
grep -qF -- 'http://x/a"b' <<<"${cmdline}" &&
fail "the url field still emits a raw quote: ${cmdline}"
echo "PASS"

View File

@@ -0,0 +1,167 @@
#!/bin/bash
#
# htsserver serves the crawled mirror under /website/ alongside its own GUI:
# mirrored pages must skip the ${...} expander, or ${_sid} leaks the session id.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
log=$(mktemp)
work=$(mktemp -d)
csrv=
# start() runs in a command substitution, so its $! never reaches this shell. A
# missed kill leaves an orphan holding the CI job open long after a green suite.
srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
cleanup() {
local pid
pid=$(srvpid)
test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
test -z "${csrv}" || kill -9 "${csrv}" 2>/dev/null || true
wait "${csrv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${log}" "${work}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
portof() { echo "${1##*:}" | tr -d /; }
# GET $2 from 127.0.0.1:$1, headers into $3 and the body, byte for byte, into $4.
fetch() {
python3 -c 'import socket, sys
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(20)
s.sendall(("GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % sys.argv[2]).encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
head, _, body = out.partition(b"\r\n\r\n")
open(sys.argv[3], "wb").write(head)
open(sys.argv[4], "wb").write(body)' "$1" "$2" "$3" "$4"
}
# POST the remaining args as urlencoded key=value fields to 127.0.0.1:$1.
post() {
local port=$1
shift
python3 -c 'import socket, sys, urllib.parse
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe=""))
for k, v in (a.split("=", 1) for a in sys.argv[2:])).encode()
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(30)
s.sendall(b"POST /step4.html HTTP/1.0\r\nHost: 127.0.0.1\r\n"
b"Content-type: application/x-www-form-urlencoded\r\n"
b"Content-length: %d\r\n\r\n" % len(body) + body)
while s.recv(65536):
pass
s.close()' "${port}" "$@" >/dev/null
}
# LF-only, and a line ending in a backslash: the expander rewrites both, so a
# mangled reply fails the byte comparison even where no directive is present.
mirror="${work}/proj/hostile.html"
hdr="${work}/hdr"
body="${work}/body"
# The server merges $HOME/.httrack.ini into the same store on the first request;
# point it somewhere empty so a developer's own file cannot shadow the fields.
export HOME="${work}"
url=$(start)
port=$(portof "${url}")
# Positive control: the GUI's own templates must still expand. This is also
# where the real token comes from, so its absence below can be asserted.
fetch "${port}" /server/index.html "${hdr}" "${body}"
sid=$(sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' "${body}" | head -1)
test "${#sid}" -eq 32 || fail "GUI page did not expand \${sid} (got '${sid}')"
# /website/ serves only the root the server itself recorded, so save a profile
# (no command_do=start, so nothing crawls) to create it, then plant the file.
post "${port}" "sid=${sid}" command=httrack command_do=save winprofile=x \
"path=${work}" projname=proj
test -f "${work}/proj/hts-cache/winprofile.ini" ||
fail "profile save did not create the project: $(cat "${log}")"
mkdir -p "$(dirname "${mirror}")"
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
printf 'Hostile mirrored page.\nsid=${_sid} copy=${sid}\ntrailing backslash: \\\n' \
>"${mirror}"
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
grep -q '^HTTP/1\.0 200 ' "${hdr}" || fail "mirrored page not served: $(head -1 "${hdr}")"
grep -qF "${sid}" "${body}" &&
fail "the session id was expanded into mirrored content"
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
grep -qF '${_sid}' "${body}" || fail "\${_sid} did not survive verbatim"
# shellcheck disable=SC2016
grep -qF '${sid}' "${body}" || fail "\${sid} did not survive verbatim"
cmp -s "${mirror}" "${body}" || fail "mirrored file not served byte for byte"
# The mirror stays browsable: verbatim must not mean served as a download.
grep -qi '^Content-type: text/html' "${hdr}" ||
fail "mirrored page lost its text/html type: $(cat "${hdr}")"
# The other direction: while a crawl runs, every .html request is overridden to
# the GUI's own refresh page, so a /website/ URL stops naming mirrored content.
# /trickle/ dribbles for a minute, which holds the crawl open for the probe.
clog="${work}/content.log"
python3 "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
csrv=$!
for _ in $(seq 1 40); do
cport=$(sed -n 's/^PORT //p' "${clog}") && test -n "${cport}" && break
kill -0 "${csrv}" 2>/dev/null || break
sleep 0.25
done
test -n "${cport:-}" || fail "content server did not come up: $(cat "${clog}")"
post "${port}" "sid=${sid}" "path=${work}" projname=crawl winprofile=x \
command_do=start \
"command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/trickle/ -O ${work}/crawl"
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
grep -q '^HTTP/1\.0 200 ' "${hdr}" ||
fail "running crawl: /website/ was not overridden to the GUI page: $(head -1 "${hdr}")"
grep -qF "'crawl' - HTTrack Website Copier" "${body}" ||
fail "the overridden GUI page was not the expanded refresh page"
echo "PASS"

View File

@@ -0,0 +1,168 @@
#!/bin/bash
#
# /website/ is served from the project directory htsserver set up itself, never
# from a root the request body names, and composing that path must stay bounded.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
base=$(mktemp -d)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
# The server reports its own pid; the aliveness assertion below hangs off it.
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
# $2 is empty. Prints the reply.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
get() { request "$1" "$2" ""; }
post() { request "$1" "" "$2"; }
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Every request body is gated by the session id (78_webhttrack-sid.test).
sid=$(get "${port}" /server/index.html |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
# A file the mirror must never expose, next to the project that may.
echo "SECRETMARKER" >"${base}/secret.txt"
mkdir -p "${base}/proj"
echo "LOGMARKER" >"${base}/proj/hts-log.txt"
# error_redirect is the branch that skipped the fsfile clearing, so fail the save
# with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
# any successful save: commandEnd then swaps the error page for the finished one.
get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p" |
grep -q '^Location: /server/error.html' ||
fail "the refused save did not redirect to the error page"
# No project yet, so no root to serve from.
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
fail "a posted projpath served a file outside any project"
# Positive control: step4's "save settings" flow registers the project without
# crawling, and browsing its mirror keeps working. Without it the assertions
# around it would pass on a server that never serves /website/ at all.
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
post "${port}" "${body}" >/dev/null
test -f "${base}/proj/hts-cache/winprofile.ini" ||
fail "the project was not registered: $(cat "${log}")"
get "${port}" /website/hts-log.txt | grep -q LOGMARKER ||
fail "the registered project's mirror is not served"
# Same request with the root repointed: the project is legitimate, projpath is
# not what decides where the bytes come from.
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
fail "a posted projpath repointed the served root"
post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
get "${port}" /website/passwd | grep -q '^root:' &&
fail "a posted projpath read an arbitrary system file"
# A ".." anywhere in the recorded root would escape the mirror on every later
# request, so the save must be refused and the previous root kept.
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
fail "a '..' in the saved project path escaped the mirror root"
get "${port}" /website/hts-log.txt | grep -q LOGMARKER ||
fail "rejecting the '..' root also lost the previous one"
# The root is now the server's own, but it is still built from two posted
# fields: 800-odd bytes of them used to be sprintf'd into a 1024-byte buffer.
seg=$(printf '%0200d' 0)
longpath="${base}/${seg}/${seg}/${seg}/${seg}"
fspath="${longpath}/proj"
# structcheck() refuses a root over HTS_URLMAXSIZE, so the URL carries the rest.
test "$((${#fspath} + 11))" -le 1024 ||
fail "the long project path (${#fspath}) would not pass structcheck"
longurl=$(printf '%0800d' 0)
test "$((${#fspath} + 1 + ${#longurl}))" -gt 1024 ||
fail "the composed path (${#fspath} + ${#longurl}) would not overflow"
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${longpath}&projname=proj" >/dev/null
test -f "${fspath}/hts-cache/winprofile.ini" ||
fail "the long-path project was not registered: $(cat "${log}")"
get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
get "${port}" /server/index.html | grep -q '200 OK' ||
fail "the server stopped answering after the over-long project path"
echo "PASS"

View File

@@ -0,0 +1,111 @@
#!/bin/bash
# A cached header longer than proxytrack's field must be clipped to that field,
# not overflow into the next one (contenttype[64] abuts the location pointer).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
pad() { printf "%${1}s" '' | tr ' ' "$2"; }
# Longest surviving run of char $2 in file $1, or 0.
runlen() {
grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
}
# --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
# Each header overshoots its own destination, so a per-field clip is the only
# way to get every expected length right at once.
{
printf 'HTTP/1.1 200 OK\r\n'
printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)"
printf 'Etag: %s\r\n' "$(pad 400 E)"
printf 'Content-Disposition: %s\r\n' "$(pad 400 D)"
printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
printf 'Content-Length: 5\r\n\r\n'
} >"$dir/hdr"
printf 'hello' >"$dir/body"
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
{
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
printf '2 0 test\n'
printf '\n\n'
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
cat "$dir/hdr" "$dir/body"
} >"$dir/in.arc"
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2
exit 1
}
test -s "$dir/out.arc" || {
echo "FAIL: ARC entry dropped instead of clipped" >&2
exit 1
}
# Only contenttype survives into the ARC output; the over-long Etag and
# Content-Disposition above are still load-bearing, since bounding just one
# field leaves the others smashing the element (the run above would crash).
got=$(runlen "$dir/out.arc" A)
test "${got:-0}" = 54 || {
echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2
exit 1
}
# --- ZIP reader (ZIP_READFIELD_STRING) --------------------------------------
python=$(find_python) || {
echo "python3 missing; ARC half only" >&2
exit 0
}
"$python" - "$dir/in.zip" <<'EOF'
import sys, zipfile
body = b"hello world"
meta = (
"X-In-Cache: 1\r\n"
"X-StatusCode: 200\r\n"
"X-StatusMessage: ok%s\r\n"
"X-Size: %d\r\n"
"Content-Type: text/html%s\r\n"
"X-Charset: %s\r\n"
"Etag: %s\r\n"
"Content-Disposition: %s\r\n"
# a parseable date is required: the ARC writer dereferences it unchecked
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
"X-Save: out/page.html\r\n"
% ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400)
).encode()
zi = zipfile.ZipInfo("example.com/page.html")
zi.compress_type = zipfile.ZIP_STORED
zi.extra = meta
with zipfile.ZipFile(sys.argv[1], "w") as z:
z.writestr(zi, body)
EOF
proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2
exit 1
}
test -s "$dir/out2.arc" || {
echo "FAIL: zip entry dropped instead of clipped" >&2
exit 1
}
# three destinations, two capacities: msg[1024] keeps all 400 M's where a
# one-size-fits-all clip would cut it to 63 like the others
while read -r ch want; do
got=$(runlen "$dir/out2.arc" "$ch")
test "${got:-0}" = "$want" || {
echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2
exit 1
}
done <<'EOF'
A 54
C 63
M 400
EOF

162
tests/88_local-changes.test Normal file
View File

@@ -0,0 +1,162 @@
#!/bin/bash
# --changes (#714): the report must match the mirror's real deltas. Every route
# answers 200 with no validators on both passes, so the transfer signal alone
# would call the whole site changed; the stable resources land in "unchanged"
# only if payloads are really compared.
set -eu
: "${top_srcdir:=..}"
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
server=$(nativepath "${testdir}/local-server.py")
root=$(nativepath "${testdir}/server-root")
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_changes.XXXXXX") || exit 1
serverpid=
cleanup() {
stop_server "$serverpid"
rm -rf "$tmpdir"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
serverlog="${tmpdir}/server.log"
: >"$serverlog"
"$python" "$server" --root "$root" >"$serverlog" 2>&1 &
serverpid=$!
port=
for _ in $(seq 1 50); do
line=$(head -n1 "$serverlog" 2>/dev/null)
if test "${line%% *}" == "PORT"; then
port="${line#PORT }"
break
fi
kill -0 "$serverpid" 2>/dev/null || {
echo "server exited early: $(cat "$serverlog")"
exit 1
}
sleep 0.1
done
test -n "$port" || {
echo "could not discover server port"
exit 1
}
base="http://127.0.0.1:${port}"
which httrack >/dev/null || {
echo "could not find httrack"
exit 1
}
out="${tmpdir}/crawl"
mkdir "$out"
report="${out}/hts-changes.json"
# Purging off throughout: the deleted set must be computed with nothing acting
# on it.
common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --changes --purge-old=0 --retries=2)
# Report field extractor: bucket counts, and the files listed under a bucket.
count() { sed -n 's/.*"'"$1"'": \([0-9]*\).*/\1/p' "$report" | head -1; }
# tr: python prints CRLF on Windows, and the expectations below are LF.
listed() {
"$python" - "$report" "$1" <<'EOF' | tr -d '\r'
import json
import sys
with open(sys.argv[1], encoding="utf-8") as fp:
report = json.load(fp)
for entry in report[sys.argv[2]]:
print(entry["file"])
EOF
}
expect() {
local label="$1" want="$2" got="$3"
printf '[%s] ..\t' "$label"
test "$want" == "$got" || {
echo "FAIL: expected '${want}', got '${got}'"
cat "$report"
exit 1
}
echo "OK"
}
# --- pass 1: nothing to compare against --------------------------------------
httrack "${common[@]}" "${base}/changes/index.html" >"${tmpdir}/log1" 2>&1
test -s "$report" || {
echo "FAIL: pass 1 wrote no ${report}"
exit 1
}
expect "pass 1 declares a first crawl" "true" \
"$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)"
# Every mirrored file: index, stable.html, moved.html, stable.bin, moved.bin,
# doomed.html, redirtarget.html, flaky.bin, coded.bin, codedstable.bin,
# sized.html and the redirect target it points at.
expect "pass 1 counts 12 new" "12" "$(count new)"
expect "pass 1 counts 0 changed" "0" "$(count changed)"
expect "pass 1 counts 0 gone" "0" "$(count gone)"
grep -aq "first crawl" "${out}/hts-log.txt" || {
echo "FAIL: pass 1 log carries no first-crawl summary"
exit 1
}
# --- pass 2: two bodies move, one page appears, one disappears ---------------
httrack "${common[@]}" --update "${base}/changes/index.html" >"${tmpdir}/log2" 2>&1
expect "pass 2 is not a first crawl" "false" \
"$(sed -n 's/.*"first_crawl": \([a-z]*\).*/\1/p' "$report" | head -1)"
host="127.0.0.1_${port}"
# The redirect behind sized.html now points at a much longer target name.
long="${host}/changes/$(printf 's%.0s' $(seq 40)).html"
expect "new is fresh.html and the renamed redirect target" \
"$(printf '%s\n%s' "${host}/changes/fresh.html" "$long" | sort)" \
"$(listed new | sort)"
expect "gone is doomed.html and the old redirect target" \
"$(printf '%s\n%s' "${host}/changes/doomed.html" "${host}/changes/s.html" | sort)" \
"$(listed gone | sort)"
# index.html changes because its link list does; moved.* change their bodies.
# coded.bin arrives gzipped and direct-to-disk, so its previous copy has to be
# sampled before the decoded temp is renamed over it; it is the same length on
# both passes, so only a digest separates it from codedstable.bin.
expect "changed is exactly the four moved resources" \
"$(printf '%s\n%s\n%s\n%s' "${host}/changes/coded.bin" \
"${host}/changes/index.html" "${host}/changes/moved.bin" \
"${host}/changes/moved.html")" \
"$(listed changed | sort)"
# The teeth: re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin
# gets there through a failed transfer and a retry, so its file is notified
# twice; redirtarget.html arrives behind a 302.
# sized.html is the sharpest of these: its payload is byte-identical, but the
# rewritten link to the renamed redirect target makes the file on disk change
# length, so a size comparison alone would call it changed.
expect "unchanged is exactly the six stable resources" \
"$(printf '%s\n%s\n%s\n%s\n%s\n%s' "${host}/changes/codedstable.bin" \
"${host}/changes/flaky.bin" "${host}/changes/redirtarget.html" \
"${host}/changes/sized.html" "${host}/changes/stable.bin" \
"${host}/changes/stable.html" | sort)" \
"$(listed unchanged | sort)"
# Every mirrored file appears once and only once, across all four buckets.
printf '[each file in exactly one bucket] ..\t'
dupes=$( (
listed new
listed changed
listed unchanged
listed gone
) | sort | uniq -d)
test -z "$dupes" || {
echo "FAIL: listed in more than one bucket: ${dupes}"
exit 1
}
echo "OK"
# Before #714 the deleted set only existed as a side effect of purging; it must
# now be computed without one, and reporting it must not delete anything.
expect "the report says nothing was purged" "false" \
"$(sed -n 's/.*"purged": \([a-z]*\).*/\1/p' "$report" | head -1)"
printf '[reporting gone did not delete it] ..\t'
test -f "${out}/${host}/changes/doomed.html" || {
echo "FAIL: doomed.html was deleted with --purge-old=0"
exit 1
}
echo "OK"

View File

@@ -30,8 +30,10 @@ TEST_EXTENSIONS = .test
TEST_LOG_COMPILER = $(BASH)
TESTS = \
00_runnable.test \
01_engine-changes.test \
01_engine-charset.test \
01_engine-cmdline.test \
01_engine-cmdline-split.test \
01_engine-cookies.test \
01_engine-copyopt.test \
01_engine-crange.test \
@@ -64,6 +66,7 @@ TESTS = \
01_engine-reconcile.test \
01_engine-expandhome.test \
01_engine-fsize.test \
01_engine-growsize.test \
01_engine-redirect.test \
01_engine-longpath-io.test \
01_engine-mirror-io.test \
@@ -171,6 +174,17 @@ TESTS = \
74_local-warc-wacz.test \
74_local-warc-verbatim.test \
75_engine-longpath-posix.test \
76_cli-resize.test
76_cli-resize.test \
77_webhttrack-redirect.test \
78_webhttrack-sid.test \
79_local-proxytrack-webdav-mime.test \
80_engine-crash-symbolize.test \
81_webhttrack-maxsize.test \
82_webhttrack-browse-links.test \
83_webhttrack-argescape.test \
84_webhttrack-mirror-verbatim.test \
85_webhttrack-projpath.test \
86_local-proxytrack-cache-longfields.test \
88_local-changes.test
CLEANFILES = check-network_sh.cache

View File

@@ -1571,6 +1571,124 @@ class Handler(SimpleHTTPRequestHandler):
if self.command != "HEAD":
self.wfile.write(body)
# --changes (#714). Every route answers 200 with no validators, so the
# transfer signal alone would call the whole site changed on pass 2; only a
# payload comparison can tell stable.html and stable.bin from moved.*.
def route_changes_index(self):
links = "".join(
'\t<a href="%s">%s</a>\n' % (name, name)
for name in (
"stable.html",
"moved.html",
"stable.bin",
"moved.bin",
"redir.html",
"flaky.bin",
"coded.bin",
"codedstable.bin",
"sized.html",
)
)
if self.refetch_pass() == 1:
links += '\t<a href="doomed.html">doomed</a>\n'
else:
links += '\t<a href="fresh.html">fresh</a>\n'
self.send_html(links)
def route_changes_stable(self):
self.refetch_pass()
self.send_raw(b"<html><body><p>CHANGES-STABLE</p></body></html>", "text/html")
def route_changes_moved(self):
pass1 = self.refetch_pass() == 1
self.send_raw(
b"<html><body><p>CHANGES-MOVED-V%d</p></body></html>" % (1 if pass1 else 2),
"text/html",
)
def route_changes_stable_bin(self):
self.refetch_pass()
self.send_raw(
b"CHANGES-STABLE-BIN\n" + b"\x00\x01\x02\xff" * 512,
"application/octet-stream",
)
def route_changes_moved_bin(self):
pass1 = self.refetch_pass() == 1
self.send_raw(
b"CHANGES-MOVED-BIN-V%d\n" % (1 if pass1 else 2)
+ b"\x03\x02\x01\xfe" * 512,
"application/octet-stream",
)
def route_changes_doomed(self):
self.refetch_pass()
self.send_raw(b"<html><body><p>CHANGES-DOOMED</p></body></html>", "text/html")
def route_changes_fresh(self):
self.refetch_pass()
self.send_raw(b"<html><body><p>CHANGES-FRESH</p></body></html>", "text/html")
def route_changes_redir(self):
self.send_response(302)
self.send_header("Location", "/changes/redirtarget.html")
self.send_header("Content-Length", "0")
self.end_headers()
def route_changes_redirtarget(self):
self.refetch_pass()
self.send_raw(b"<html><body><p>CHANGES-REDIR</p></body></html>", "text/html")
# Direct-to-disk, and the first body fetch of each pass dies mid-transfer:
# the retry notifies the same file a second time, so the accumulator has to
# keep the pre-run sample from the first notify.
FLAKY = b"CHANGES-FLAKY\n" + b"\x07\x06\x05\x04" * 512
# A page whose payload never changes, linking through a redirect whose
# target is renamed between passes: the rewritten link makes the file on
# disk change length while the payload is byte-identical. Only a payload
# comparison can call it unchanged.
SIZED_SHORT = "s.html"
SIZED_LONG = "s" * 40 + ".html"
def route_changes_sized(self):
self.refetch_pass()
self.send_raw(
b'<html><body><a href="sizedredir.html">t</a></body></html>',
"text/html",
)
def route_changes_sizedredir(self):
target = self.SIZED_SHORT if self.refetch_pass() == 1 else self.SIZED_LONG
self.send_response(302)
self.send_header("Location", "/changes/" + target)
self.send_header("Content-Length", "0")
self.end_headers()
def route_changes_sizedtarget(self):
self.refetch_pass()
self.send_raw(b"<html><body><p>SIZED-TARGET</p></body></html>", "text/html")
# Content-Encoding on a direct-to-disk body: the decoded temp is renamed
# over the mirror file, so the previous copy has to be sampled before the
# rename. Same length on both passes, so only a digest can tell them apart.
def route_changes_coded(self):
pass1 = self.refetch_pass() == 1
body = b"CHANGES-CODED-V%d\n" % (1 if pass1 else 2) + b"\x11\x22" * 1024
self.send_coded(self.gzipped(body), "application/octet-stream")
# Control: same coding, same bytes on both passes.
def route_changes_coded_stable(self):
self.refetch_pass()
body = b"CHANGES-CODED-STABLE\n" + b"\x33\x44" * 1024
self.send_coded(self.gzipped(body), "application/octet-stream")
def route_changes_flaky(self):
if self.refetch_pass() % 2 == 1:
self.send_truncated(self.FLAKY, "application/octet-stream")
else:
self.send_raw(self.FLAKY, "application/octet-stream")
ROUTES = {
"/cookies/entrance.php": route_entrance,
"/cookies/second.php": route_second,
@@ -1588,6 +1706,22 @@ class Handler(SimpleHTTPRequestHandler):
"/codec/bad.html": route_codec_bad,
"/codec/bin.dat": route_codec_bin,
"/codec/ae.html": route_codec_ae,
"/changes/index.html": route_changes_index,
"/changes/stable.html": route_changes_stable,
"/changes/moved.html": route_changes_moved,
"/changes/stable.bin": route_changes_stable_bin,
"/changes/moved.bin": route_changes_moved_bin,
"/changes/doomed.html": route_changes_doomed,
"/changes/fresh.html": route_changes_fresh,
"/changes/redir.html": route_changes_redir,
"/changes/redirtarget.html": route_changes_redirtarget,
"/changes/flaky.bin": route_changes_flaky,
"/changes/coded.bin": route_changes_coded,
"/changes/sized.html": route_changes_sized,
"/changes/sizedredir.html": route_changes_sizedredir,
"/changes/" + SIZED_SHORT: route_changes_sizedtarget,
"/changes/" + SIZED_LONG: route_changes_sizedtarget,
"/changes/codedstable.bin": route_changes_coded_stable,
"/upcodec/index.html": route_upcodec_index,
"/upcodec/mem.html": route_upcodec_mem,
"/upcodec/disk.bin": route_upcodec_disk,

View File

@@ -46,12 +46,14 @@ cat >"$stubdir/x-www-browser" <<EOF
echo "stub browser invoked with: \$1" >&2
# Also fetch an option page and require a rendered title='' tooltip: proves the
# option template expands and the \${html:} filter escapes into the attribute.
# option9 additionally proves the WARC control renders with its expanded label.
# option9 additionally proves the WARC and change-report controls render with
# their expanded labels.
opturl="\${1%/}/server/option2.html"
warcurl="\${1%/}/server/option9.html"
if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html &&
opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" &&
warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then
warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC &&
printf '%s' "\$warc" | grep -qaF 'name="changes"' && printf '%s' "\$warc" | grep -qaF hts-changes.json; then
echo PASS >"$marker"
else
echo "FAIL: unexpected response from \$1" >"$marker"