Compare commits

..

25 Commits

Author SHA1 Message Date
Xavier Roche
9484b32ecd An .ndx entry's two URL halves overflow the buffer they share (#744)
Both binput calls pass HTS_URLMAXSIZE, but they write into one
line[HTS_URLMAXSIZE * 2] and binput puts its NUL at s[max]. A first field
that fills its whole bound leaves the second writing line[2048], one past
the array. ASan reports a stack-buffer-overflow on a crafted .ndx. Bound
the second by what the first left.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 23:20:10 +02:00
Xavier Roche
1d647bfecd PT_GetTime's gmtime-failure fallback would print a day of 00 (#742)
* PT_GetTime's failure fallback formats as day 00

An all-zero struct tm has tm_mday == 0, and the ARC filedesc line prints
tm_mday raw, so a gmtime failure would emit "...0100" where the day
belongs. Use the epoch, as PT_SaveCache__Arc_Fun already does for an
unparseable Last-Modified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The fallback is reachable, so test it

I claimed no reachable trigger. Wrong: file_timestamp() passes st_mtime
through untouched, and a cache whose mtime is past gmtime's range takes
the fallback. Only the ARC loader is safe, because it overwrites the
timestamp with the filedesc line's own 4-digit-year date.

Test 88 sets an out-of-range mtime on a zip cache and pins the emitted
date; without the fix it reads 19000100000000.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Register the Windows skip, and make the skip path real

The Windows job pins an exact expected-skip list, so a new conditional
skip fails it; test 88 skips there because NTFS will not hold an mtime
past gmtime's range. Its own skip logic was also dead code: "rc=0 || rc=$?"
never runs the right-hand side, and only set -e was carrying python's 77
through. Capture the status properly and tell a clamp apart from a real
failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 23:19:51 +02:00
Xavier Roche
b2dc012263 Route ProxyTrack's remaining raw strcpy through the wrappers (#743)
Fourteen sites, all safe today: literals into sized fields, and two
same-sized array copies. The three writing "" through r->location, a
char *, become a direct terminator, since strcpybuff would have taken its
pointer path and lost the bound.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:59:43 +02:00
Xavier Roche
aa1131982b Three hand-written copies of the same clipping contract (#741)
* Fold the three copies of the clip idiom into strclipbuff()

htscache.c and the two readers in proxy/store.c each spelled out
clear-then-strlncatbuff, in binaries that share no code. A helper in
htssafe.h states the contract once, and evaluates its arguments once: the
macro form expanded refvalue and refvalue_size twice, and (refvalue_size)
- 1 would have wrapped to SIZE_MAX had any call site ever passed 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the capacity-2 boundary

The cases jumped from the degenerate capacity 1 straight to 8, so a
defect confined to small-but-not-degenerate sizes passed: clipping a
two-byte destination to the empty string instead of one character.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:58:58 +02:00
Xavier Roche
f72e7ebe96 htsserver: an unauthenticated GET of a directory spins the server forever (#724)
* htsserver: a request naming a directory spins the server forever

fopen() succeeds on a directory on POSIX and every read from it fails with
EISDIR without ever raising EOF, so smallserver()'s "while (!feof(fp))" serving
loop never terminates. GET /server/ needs no session id and no project to reach
it, and the accept loop is single-threaded, so one unauthenticated local request
wedges WebHTTrack for good.

Refuse a directory before fopen() so the 404 branch answers, rather than only
ending the loop: a loop-only fix would serve every directory as an empty 200.
The two loops fed a client-influenced path also stop on ferror(), since a read
that fails for any other reason spins the same way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: whitelist regular files instead of blacklisting directories

fexist() (htsserver.h) is the stat + S_ISREG predicate this file already uses
for the "file-exists:" template op, so gate the serving fopen() on it rather
than on a fresh negated is_directory(). Refusing only directories still handed
FIFOs to the same code path, where fopen() blocks with no writer and kills the
single-threaded accept loop for good.

Test 91 gains the FIFO case and a POST that loads a project whose
hts-cache/winprofile.ini is a directory: that fopen() sits ahead of every guard,
so it covers the ferror() check on the project-load read loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: fold the two comments above the serve guard into one

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:49:59 +02:00
Xavier Roche
3b53bf85f2 A posted project path overflows htsserver's "save settings" error messages (#723)
* htsserver: clip the "save settings" failure messages

The three sprintf(tmp[1024], ...) sites reporting a failed profile save
quote a project path composed from the posted "path" and "projname"
fields, neither of them bounded. A sid-authenticated save with a
1200-byte path smashes the stack buffer and takes the server down.

Route them through a SET_ERRORF() that formats into a fixed buffer and
absorbs the truncation once, so the message clips instead of aborting:
the text comes from the client, and a quieter denial of service is no
better than a loud one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: reach the second message without a 1030-byte mkdir

The tree the case needed came within a few bytes of macOS's PATH_MAX
once /var resolves to /private/var. A symlinked hts-cache gets there
instead: structcheck() lets a non-directory through, and opening
winprofile.ini under it fails with ENOTDIR whatever the uid.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htssafe: one clipping printf helper for the three failf wrappers

htsblk_failf(), PT_Element_failf() and htsserver's format_error() all had
the same body; slprintfbuff_clip() now owns it. A (void) cast on
slprintfbuff() is no substitute: GCC warns through warn_unused_result.

vslprintfbuff() also empties dest before formatting, so a vsnprintf that
fails outright cannot leave the caller publishing uninitialized stack.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the clipped length, not just the message text

Two of the three sites only checked that the message appeared, which the
old sprintf did equally well; each now compares the rendered message
against the 1023-byte clip. The failing-write branch no longer needs
/dev/full either: the server runs under a file-size limit, so macOS gets
the same coverage.

Renumbered 86 to 89, which no other pending change claims.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: build the oversized profile without a 16k printf width

The macOS leg reached the third message with no error set at all, so the
write the file-size limit was supposed to break had gone through. Build the
profile by doubling a 1024-wide printf, assert its length, and assert the
init file came out short, so a limit that does not bite names itself
instead of surfacing as a missing message.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: build the long paths from the physical temp directory

That is what broke the macOS leg: TMPDIR there resolves through the /var
symlink, so the 1020-byte init file the third case opens was really 1028
bytes to the kernel and the open failed with ENAMETOOLONG. Both cases sit
within a few bytes of macOS's 1024-byte PATH_MAX, so the paths have to be
measured after resolution.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:25:52 +02:00
Xavier Roche
ce7dcfa9de Options ticked on by default cannot be un-ticked in the web GUI (#725)
* Options ticked on by default cannot be un-ticked in the web GUI

An unchecked HTML checkbox posts nothing, so htsserver never overwrites the
value it already holds. Every box in the wizard is one-way: once the stored
value is "1", whether htsserver seeded it at startup, a loaded profile set it,
or the user ticked it earlier in the session, un-ticking and submitting leaves
the option on and draws the box ticked again. Only four boxes, all in
option1.html, carried the companion hidden field that guards against this.

Add it to every remaining bare checkbox, and switch cookies and parsejava to
${ztest:...} so a cleared box emits --cookies=0 / --parse-java=0; ${test:...}
renders nothing at all when the value is empty, which is not "off" for an
option the engine turns on by default.

index, urlhack and keep-alive are deliberately left alone: their long options
are declared "single" in htsalias.c and optalias_check drops the =value, so
--index=0 resolves to -I and turns the option back on. That parser bug is
pre-existing and needs its own fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: pin last-write-wins and cover every checkbox

The runtime leg posted each name once, so a first-wins body parser would
have passed while the fix did nothing in a real browser: post the
duplicated name in both orders and assert the last value wins. Replace the
four hand-written option cases with a table covering all 28 non-skipped
boxes, asserting the command-line token and the Windows-profile key each
state emits, plus a completeness check so a new box cannot slip through
unexercised.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: rename the loop variable shadowing the scanned page

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:08:46 +02:00
Xavier Roche
069573edc3 Test assertions read a padded or truncated reply as a clean security verdict (#728)
* tests: a failed request must not read as a clean security verdict

Under pipefail, "request | grep -q MARKER && fail" skips the fail when the
request itself errors: the leak checks in tests 78 and 85 then pass without
ever having run. Capture the reply first and fail loudly if it never arrived.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* AGENTS.md: record the fail-open assertion shape

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: a reply that proves nothing must not read as a clean verdict

The previous commit converted some of the fail-open assertions and left three.
78's refusal loop still piped into "grep -q ... && fail": grep -q exits on the
first match and SIGPIPEs the producer, so under pipefail a hostile reply that
pads its Location past the 64 KB pipe buffer suppresses the failure exactly as
a dead probe would. 85's fetch() only required a non-empty reply, so a
truncated body or a 302 to the file passed the leak checks marker-free, and no
assertion looked at the status line at all. 78's store probe had no emptiness
guard, so an empty page read as "the store was not written".

Match from here-strings throughout, give fetch() the status each caller
expects, and route 78's store probe through a helper that requires a served
page. 77's X-Injected check had the same shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:08:10 +02:00
Xavier Roche
a75f437df9 lienrelatif() reads one byte before its stack buffer on an empty path (#729)
The trim that walks back to the last '/' starts at `curr + strlen(curr) - 1`,
which is `curr - 1` when the path is empty. The loop then dereferences it.

An empty path is reachable today: the pre-pass that strips a query does
`strncatbuff(newcurr_fil, curr_fil, a - curr_fil)`, so any `curr_fil` starting
with '?' hands the walk an empty string. `-#test=relative "dir/page.html" "?x"`
under ASan reports the underflow.

The read is one byte and the loop stops immediately either way, so the guard
changes no output: over the 484 ordered pairs of a 22-value path corpus, run
against builds that force the byte before the buffer to 0 and to '/', the
guarded and unguarded results are identical.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 22:06:11 +02:00
Xavier Roche
783f6ee1f5 AGENTS.md: record what the msg[80] hardening batch taught (#737)
Five PRs across the engine and ProxyTrack turned up the same few traps
more than once, and none of them are obvious from the code.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 21:39:57 +02:00
Xavier Roche
913caf68be Clear the last three compiler warnings (#733)
* Clear the last three compiler warnings

finalurl was sized for one of the two URLs it concatenates. The IIS-bug
example callback overwrites a suffix in place with a same-length
replacement and must not terminate the string, which is memcpy, not
strncpy. The coucal bench's if/else chain has no final else, so result was
only initialised on the paths gcc could not prove exhaustive.

A clean build now reports zero warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the IIS suffix copy by what matched, not by the table

Copying strlen(replacement) leaves the "MUST be the same sizes" comment as
the only thing standing between a future table edit and an overflow. j is
the number of bytes just matched in the destination, so copying j is safe
whatever the table holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 20:37:52 +02:00
Xavier Roche
59660102d6 A cache field wider than ours aborts the engine instead of clipping (#732)
* A cache field wider than ours aborts the engine instead of clipping

The read-side ZIP_READFIELD_STRING used strlcpybuff, and the whole *_safe_
family aborts on overflow rather than truncating. Since the header line is
bounded only by HTS_URLMAXSIZE and msg is 80 bytes, a cache written by
another build, or a corrupt one, kills the crawl outright. The corrupt-cache
self-test already promises "rejected per-entry, never crash".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the clip to each field's own capacity

Review found one case exercised only msg[80], so a hardcoded clip length
passed. lastmodified[64] is narrower, and no single constant satisfies
both. The forged replacement was also one byte longer than the line it
overwrote, which only worked because corrupt_patch copies exactly the
pattern length.

Also stop claiming another build's cache can trigger this: the writer
emits each field from the same struct the reader fills, so it takes a
corrupt cache.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 20:37:38 +02:00
Xavier Roche
e96399910b Share the in-progress display struct instead of copying it (#734)
t_StatsBuffer was defined twice, byte for byte, in httrack.h and htsweb.h,
with NStatsBuffer duplicated alongside. httrack and htsserver each keep
their own array, so nothing catches the two drifting apart, and the last
change to the struct had to be applied to both by hand. Both now include
src/htsstats.h; sizes and offsets are unchanged.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 20:19:01 +02:00
Xavier Roche
52d0ab2356 An entry with no usable Last-Modified crashes proxytrack --convert (#731)
* Cached entry with no usable date crashes the ARC writer

PT_SaveCache__Arc_Fun dereferenced convert_time_rfc822() straight into the
record line, so any entry whose Last-Modified is absent or unparseable took
proxytrack --convert down. The sibling caller a thousand lines up already
guards the same call; this one fills in the epoch instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Assert the archive date, not just the entry

Review found the test blind to the two mutants that matter: a guard firing
unconditionally clobbers every valid date to the epoch, and one that skips
the year and day emits a month and day of 00. Grepping only for the URL saw
neither. Assert the date field exactly, and add a valid-date case so the
untouched path is pinned too.

A bare "Last-Modified: 0" crashes the same way, so it joins the cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 20:18:58 +02:00
Xavier Roche
ca533abefd Unbounded writes into the 80-byte htsblk failure message (#722)
* Bound the remaining writes into htsblk.msg[80]

Nine writers still filled the 80-byte msg with no bound. Six sprintf the
result of strerror(), whose longest glibc string is 49 bytes in the C
locale and 72 in fr_FR against 46 bytes of room after the longest prefix.
Realistic connect() errnos still fit, so this was latent rather than live.

The FTP helper's .ok result file had no excuse: it was copied byte by byte
until EOF into the same 80 bytes. Split that parse out as
back_read_ftp_result() so a self-test can drive it, and stop at capacity.

These sites never appeared in the -Wformat-truncation cluster that added
htsblk_failf: the diagnostic only fires on a bounded snprintf whose return
is discarded, so a raw sprintf is invisible to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Close the self-test's blind spots around msg[]

Six of eight mutants survived the first version. The neighbour canary
compared against zero, so it saw a stray 'X' but not the stray NUL an
off-by-one terminator actually writes; poison it instead. Only the
over-capacity case existed, so padding every message to 79 bytes or eating
its last character both passed, as did a sign-extended 0xff reading as EOF
and the new unparseable-status branch, which had no coverage at all.

Same zero-comparison weakness applied to the htsblk_failf canary inherited
from #715, so that one is poisoned too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:51:41 +02:00
Xavier Roche
dc6819b7b7 Cached headers longer than ProxyTrack's fields overflow into the location pointer (#717)
* Bound ProxyTrack's cache header copies

ZIP_READFIELD_STRING in proxy/store.c took no destination size and used a
raw strcpy, while the engine's namesake in htscache.c has taken a
refvalue_size for years. The source is a cache-entry header line bounded
only by line[HTS_URLMAXSIZE + 2].

contenttype[64] sits just before the location pointer in the calloc'd
_PT_Element, so an over-long Content-Type walks over charset and then
over location, which the next Location: line copies through.

Clip rather than reject: the engine stores Content-Type in 128 bytes, so
a valid cache legitimately carries fields wider than ours.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Bound the ARC reader's header copies too

Review found store.c carries a second copy of the same macro:
HTTP_READFIELD_STRING feeds the ARC reader from index->line[2048], twice
the ZIP path's reach, into the same contenttype[64] and its neighbouring
location pointer. proxytrack --convert on a plain-text ARC segfaults.

Test 86 covered one of eight fields, so a per-field clip and a
one-size-fits-all clip were indistinguishable. It now overshoots every
destination and asserts each surviving length against its own capacity,
and exercises the ARC reader without needing python.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:51:13 +02:00
Xavier Roche
3247e5b183 Diagnostic messages built from remote replies discard snprintf's truncation signal (#715)
* Fold htsblk/PT_Element failure messages into a clipping helper

The 18 remaining -Wformat-truncation warnings all came from building a
diagnostic string out of a remote server's reply and dropping snprintf's
return. Add htsblk_failf() for htsblk.msg[80] and PT_Element_failf() for
ProxyTrack's msg[1024]: both clip to fit and absorb the discard once, so
the obligation is not silently laundered at each call site.

msg[80] lives in the installed htsopt.h, so growing it would break the
ABI; a clipped FTP banner is the intended outcome anyway. The display
StatsBuffer.state is not installed, so it grows to fit back->info instead.

Also bounds two unbounded sprintf(r->msg, ...) in ProxyTrack's cache
reader and moves the raw strcpy neighbours to strcpybuff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Catch a one-past-the-end write into msg[]'s neighbour

Review found the new self-test blind to a store at msg[sizeof(msg)]: it
lands in the adjacent contenttype field, which no assertion read, and an
intra-struct overflow is invisible to ASan and _FORTIFY_SOURCE. Check the
neighbour after every call, and add the exact-fit case the block lacked.

Correct the contract comment too: msg is not purely diagnostic, it
round-trips through the cache as X-StatusMessage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:00:21 +02:00
Xavier Roche
bef7120423 tests: test 82 fails on master after #707 and #709 landed together (#716)
* tests: arm test 82's mirror root through a profile save

#707 made /website/ serve only the root htsserver recorded at structcheck
success, so the posted projpath test 82 relied on no longer names anything.
Each PR was green alone; the pair was not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the now-unused project-path argument

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 17:44:55 +02:00
Xavier Roche
7cf54af485 htsserver: a posted projpath repoints the served root and overflows the composed path (#707)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* htsserver: serve the crawled mirror verbatim, never through the expander

WebHTTrack serves the mirror under /website/ from the same small server as its
own GUI, and the decision to run a response through the ${...} template
expander was a substring test for ".htm" on the request path. A mirrored page
therefore had its directives evaluated: ${_sid} rendered the live session id,
handing the crawled site the token that authenticates commands on the local
GUI, and ${do:...} gave it the rest of the template verbs.

The /website/ prefix was already detected, but only to keep the crawl-state
override from hijacking a mirror request. Reuse it as the expansion gate, so
expansion is limited to files under the GUI's html root, and re-evaluate it
after that override, which can substitute a GUI page for a mirror path.
Mirrored pages keep their text/html type: verbatim must not turn browsing the
mirror into a download.

Note that mirrored content still shares the control origin, so a script in it
can read the session id from a GUI page itself; that is a separate fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: stop letting a posted projpath name the /website/ root

The static-file path in smallserver() was composed with a length guard that
measured only the server root and the request path, then wrote the posted
"projpath" field into a 1024-byte stack buffer: a 2000-byte projpath followed
by any /website/ request smashed the stack (buffer overflow detected, server
gone). The guard also summed two untrusted lengths before comparing, the shape
that can wrap and pass.

Composition now goes through a bounded, non-aborting append that keeps the
untrusted length alone on one side, and /website/ is served from the project
directory the server itself set up, rejecting a ".." in it, rather than from
whatever root the request body claimed. Without that, projpath=/etc/ plus
GET /website/passwd read an arbitrary file, since the ".." check looked at the
request path only. fsfile is also cleared before the error-redirect branch,
which could otherwise reach fopen() uninitialized.

tests/80_webhttrack-projpath.test drives the three cases against a live server
and keeps a legitimate project browsable as the control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: establish test 79's mirror root through the server

/website/ no longer serves the posted projpath, so injecting one as a
fixture stopped working and test 79 got a 404 instead of the mirrored
page. Save a profile first (no command_do=start, so nothing crawls) to
make the server record the root, then plant the file under it.

Re-checked against a reverted expander gate: still fails there, so the
fixture change did not cost the assertion its teeth.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the running-crawl half of the /website/ override

Test 84 only exercised the idle server, where the override never fires and
the recomputed virtualpath is indistinguishable from the stale one. Drive a
crawl through the server so /website/*.html is rewritten to the GUI refresh
page, which 404s without the recompute.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the overflow, the '..' rejection and the fsfile hoist

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: fix the merged test 84 duplicate post() helper

The rename-side and the branch-side each defined post(), with different
argument shapes; the last one won and silently mangled the save body.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: trim the new comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the stale start() arg comments in test 84

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 17:18:43 +02:00
Xavier Roche
c32a47110e webhttrack: the mirror link on the finished page cannot be followed (#709)
* WebHTTrack's mirror links are dead file:// URLs

The GUI is served from http://127.0.0.1:PORT/, and browsers refuse to navigate
from an http: page to a file: URL, so the "browse the mirror" links in
finished.html and file.html did nothing when clicked. They now point at the
server's own /website/ route, which already serves the project directory. The
desktop entry's browse mode still works, because there the shell hands the
file: URL to the browser instead of navigating from a page.

The per-project picker in file.html goes too: /website/ is bound to the running
project, and reaching the other projects over HTTP would mean serving the whole
mirror tree from the control origin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* finished.html: link the served mirror, not a dead file:// URL

The page is served over http:, so its file:// link is a cross-scheme
navigation that Chrome and Firefox refuse. Nothing happened on click. The
mirror is already reachable at /website/, which the two list entries just
below were using all along.

file.html keeps its file:// links. It is the fresh-session entry point for
sites mirrored earlier, and /website/ is bound to one project, so pointing
it there would trade a dead link for a 404. Serving an arbitrary past
project needs a route that does not exist yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: tighten the browse-link assertion and trim its comments

The runtime check for href="/website/index.html" also matched the list
entry below the anchor, so it passed on the unfixed page; match the
anchor by its mirror-path label instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 17:18:17 +02:00
Xavier Roche
1027a9f392 htsserver: crawled pages run through the template expander and leak the session id (#706)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* htsserver: serve the crawled mirror verbatim, never through the expander

WebHTTrack serves the mirror under /website/ from the same small server as its
own GUI, and the decision to run a response through the ${...} template
expander was a substring test for ".htm" on the request path. A mirrored page
therefore had its directives evaluated: ${_sid} rendered the live session id,
handing the crawled site the token that authenticates commands on the local
GUI, and ${do:...} gave it the rest of the template verbs.

The /website/ prefix was already detected, but only to keep the crawl-state
override from hijacking a mirror request. Reuse it as the expansion gate, so
expansion is limited to files under the GUI's html root, and re-evaluate it
after that override, which can substitute a GUI page for a mirror path.
Mirrored pages keep their text/html type: verbatim must not turn browsing the
mirror into a download.

Note that mirrored content still shares the control origin, so a script in it
can read the session id from a GUI page itself; that is a separate fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: cover the running-crawl half of the /website/ override

Test 84 only exercised the idle server, where the override never fires and
the recomputed virtualpath is indistinguishable from the stale one. Drive a
crawl through the server so /website/*.html is rewritten to the GUI refresh
page, which 404s without the recompute.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:57:59 +02:00
Xavier Roche
d0a1573a04 webhttrack: "max site size" sets a per-file cap and discards the HTML one (#708)
* webhttrack: "max site size" set a per-file cap instead of the overall one

step4.html mapped all three size fields of the wizard onto --max-files (-m),
so "Max site size" emitted a per-file limit rather than --max-size (-M), and
landed a second -m on the command line. That second -m also clobbered the HTML
per-file limit: a bare -m<n> resets maxfile_html, so whichever of the two came
last won. Point sizemax at --max-size and emit the bare -m before the -m,<n>
form so both per-file caps survive.

Two template typos in the same family, where a malformed ${...} renders as
nothing or as its own key instead of erroring: the winprofile.ini writer's
Dos=${dos was missing its closing brace, and option2b.html's OK button read
${LANG_OK] with a bracket.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* tests: carry the session id in test 81's POST

The gate that landed with #700 refuses a body without one, so the wizard
POST came back refused and the option audit had nothing to read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: close the confirmation-biased gaps in test 81

Post the size fields empty too, so a step4.html that lost its ${test:} guard
and rendered a valueless --max-files= is caught; assert the winprofile.ini
MaxHtml/MaxOther/MaxAll keys the header claimed to audit; and pin the OK button
label, since an unknown ${LANG_} key renders empty and passed the absence check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:40:40 +02:00
Xavier Roche
4362ac34cb Check snprintf truncation where the result must be complete (#711)
Adds `sprintfbuff()`/`slprintfbuff()` to `htssafe.h`: a formatted print that truncates to fit and returns whether it had to, marked `warn_unused_result` so the answer cannot be dropped. It fills the gap between the `strcpybuff` family, which aborts on overflow, and `String`, which grows without bound. Abort is the wrong contract wherever the text is built from a remote peer's reply.

Four `-Wformat-truncation=` sites used the result as if `snprintf` had never truncated. Two only needed a bigger destination, so they get one: `hts_finish_makeindex`'s `tempo` was a flat 1024 against a 2048-byte escaped URL and is now sized off it, and the wizard's `cmd[4096]` could not hold the answers it concatenates. The other two cannot grow. `create_back_tmpfile` formats `<url_sav>.bak` into a buffer the same size as `url_sav`, and that struct is installed, so a dropped extension would alias the backup onto the live file that `back_finalize_backup()` unlinks. ProxyTrack's `startUrl[1024]` is fed by cache content. Both take the error path they already had, and ProxyTrack moves to the next cache entry rather than publishing a clipped one.

On what the wrapper buys, since it is not what I first assumed: checking a raw `snprintf` return inline silences the warning just as well. The wrapper's value is that the capacity comes from `sizeof`, the check is the default rather than the exception, and `warn_unused_result` makes skipping it visible.

`-#test=strsafe` covers the primitive (exact fit, one over, 4 KB source, `size == 1`, trailing canary, destination repoisoned between cases), and `-#test=makeindex` gains a first link whose escaped form overruns the old buffer. Both were mutation-checked. The ProxyTrack change ships without a direct test: its only observable is the catalog page, which renders solely as a PROPFIND fallback I could not drive from curl. 25 gcc warnings down to 21; the rest of the cluster is diagnostic-only and follows separately.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:37:48 +02:00
Xavier Roche
6579436607 webhttrack: the command line overflows its argv vector and a quoted value can inject flags (#710)
* htsserver builds the redirect Location header in a 256-byte stack buffer

The POST redirect path checks strlen(file) but sprintf's newfile, which comes
straight from the client's "redirect" POST field with no length cap. A 300-byte
value overflows tmp[256]. The same value reached the Location header with no
CR/LF check, so it could split the response and inject headers.

Append into the dynamic String the other headers already use, and drop the
header entirely when the value carries a CR or LF.

The listen socket was SOCaddr_initany, so the server answered the LAN and not
just the local browser it exists to serve. Bind 127.0.0.1 by default, with
--bind <addr> to widen it again, resolved through the existing gethost() helper
the way proxytrack already does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: satisfy shellcheck and shfmt in the new server test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: drop the pre-fix narration from the oversized-value comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the bound socket, not the announced URL

The listen-address assertions only compared the URL= banner, which is a
literal echo of argv: a build that announced 127.0.0.1 while binding the
wildcard passed. Probe 127.0.0.2 on the same port instead, which a wildcard
listener takes and a loopback-only one leaves free.

Also refuse an empty --bind, which fell through to every interface and
silently undid the new default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: bound the response read and the empty --bind run

The recv() loop had no timeout and read until EOF; htsserver need not close
the connection after responding, which wedged the macOS runner for over an
hour. Stop at the end of the header block, which is all the test reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* webhttrack: bound the argv vector and escape quotes in the wizard command line

The wizard hands its httrack command line to the engine as one string, which
back_launch_cmd() split back into argv. Two things were wrong with that split.

It wrote into a fixed 1024-pointer vector with no bound, and every unquoted
space in the posted string yields an entry, so an ordinary mirror with a few
hundred URLs walked off the allocation. The split now lives in htscmdline.c as
hts_split_cmdline(), which sizes the vector from the separator count before
filling it, and the engine self-tests can reach it.

Quotes were also purely advisory: they toggled the "inside an argument" state
but nothing escaped them, so a double quote typed into a wizard field (user
agent, footer, path, project name) closed the argument early and the rest of
the value was parsed as fresh options -- among them -V, which reaches system().
Escaping has to happen where the argument boundary is known, so the template
gets its own ${arg:} filter for that context; ${html:} keeps its meaning for
the HTML attributes it is used in everywhere else, and HTML escaping would not
help anyway since the browser undoes it when it posts the command line back.
${arg:} backslash-escapes a quote and a backslash, and the splitter reads those
inside a quoted run, the same convention next_token() already implements for
doit.log. A value containing a quote now survives it intact instead of turning
into options.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* webhttrack: keep a stray quote in an unquoted field out of the split

The url and wildcard-filter fields go into the command line outside quotes,
where no backslash can escape anything: a single quote there flips the parity
of every quote after it, so a later escaped value ends up split as flags and
the escaping buys nothing. Emit %22 for those fields instead.

NULL-terminate the argv vector while here, matching the convention the tree
documents in htscharset.c, and fold the third copy of the entity table into
one helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: do not feed snprintf's return value back as its size argument

snprintf returns the length it wanted to write, so accumulating it blind
lets the next size argument wrap. The buffer is sized well past what the
loop needs, but the pattern is the one the project forbids.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 16:26:55 +02:00
Xavier Roche
cb98b1b197 Fatal-signal backtraces are unreadable: every engine frame is a bare module+offset (#705)
* Symbolize fatal-signal backtraces through addr2line

backtrace_symbols_fd() resolves names from .dynsym only, and
-fvisibility=hidden keeps every engine frame out of it, so a crash report
arrived as a column of bare module+offset. The handler now emits the raw trace
first and unconditionally, then groups the frames per module and runs addr2line
(or llvm-symbolizer) over the offsets, which reads DWARF and names the static
frames plus their inline chain.

-rdynamic is dropped: it only populated .dynsym and bought exactly one named
frame. -Wl,--build-id replaces it, so a trace from a stripped build can be
matched to its debug symbols.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Move the crash backtrace printer into src/htsbacktrace.c

httrack.c keeps only the two call sites. The symbolizer needs _GNU_SOURCE for
dladdr(), which is now confined to its own translation unit instead of being
forced on the whole CLI front-end.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Test: match glibc's backtrace format with or without the space

backtrace_symbols_fd() prints the trailing "[0xADDR]" with a leading space on
some glibc versions and without on others (Ubuntu 24.04), so the raw-frame
assertion failed everywhere but the dev box. Match only up to the offset.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Skip pseudo-modules with no file on disk

A frame in linux-vdso.so.1 made addr2line complain instead of resolving, so
the arm64 leg lost its symbolized output entirely. Renumber the test too:
77 landed on master with #700.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Stop discarding local symbols, which made the names wrong

--discard-all drops the local symbol of every static function, so addr2line
attributes the frame to the nearest surviving global: the abort frame read
dns_timeout_selftests instead of abortf_, with the file and line still right.
A wrong name is worse than none, and because the symbols go at link time no
-dbgsym package can recover them. Costs 21784 bytes on libhttrack.so.3, 0.6%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Symbolize once when the handler itself faults

A fault inside the handler re-enters the printer, which interleaved a second
symbolized trace on the same fd and spent a second budget: 3.04s and two
overlapping reports, measured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 15:43:17 +02:00
56 changed files with 2911 additions and 342 deletions

View File

@@ -226,8 +226,9 @@ jobs:
# Every gate here exits 77, so an all-skipped suite would report green having
# tested nothing: pin the skips, and floor the passes in case the glob empties.
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
# webdav-mime needs a reapable background listener, which MSYS cannot give it.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
# webdav-mime needs a reapable background listener, which MSYS cannot give it;
# badmtime needs a filesystem that stores an mtime past gmtime's range.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test"
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
[ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

View File

@@ -26,6 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
as "marker absent". Capture the reply, assert the status line it must carry
(an empty, truncated or redirected one is marker-free too), then match with a
here-string.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every
@@ -48,6 +54,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
- Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
`untrusted < limit - controlled` — never `controlled + untrusted < limit`,
which can wrap and pass.
- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
(`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
our own data, wrong for anything read back from a cache, a header or the
wire, where it trades a memory smash for a crash on malformed input. Clip
with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
into the same buffer never appears on it. Before scoping a hardening pass off
compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
`\bstrcpy\s*\(`, `\bstrcat\s*\(`).
## C conventions
- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -84,6 +100,17 @@ Before pushing, and when reviewing others, don't skim for bugs:
layout/ABI, cache/wire format, or a security path? A static or unit check
isn't enough; exercise the wrong behavior at runtime. Claude Code:
`/review-recipe`.
- **Poison a canary, never compare it against zero.** Checking that a
neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
terminator writes — the exact bug the canary is there for. Fill it with a
non-zero byte, and prove it by killing both the stray-`'X'` and the
stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
lands inside the same struct.
- **Overshoot every destination, not one.** A bounds test that oversizes a
single field cannot tell a per-field bound from a one-size-fits-all one, nor
from a fix that bounds that field and leaves its neighbours raw. Exercise
each destination the path touches, spanning at least two capacities, and
check what the code actually emits before writing the expected values.
## Commits
- **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

View File

@@ -97,9 +97,8 @@ ${do:end-if}
</pre>
${LANG_G8} :
${do:output-mode:html-urlescaped}
<a href="file://${path}/${projname}/" target="_new">
${do:output-mode:}
${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
<a href="/website/index.html" target="_new">
${path}/${projname}
</a></li>
<ul>

View File

@@ -98,6 +98,9 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="ftpprox" value="">
${LANG_PROXYTYPE}
<select name="proxytype"
title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

View File

@@ -98,6 +98,13 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="errpage" value="">
<input type="hidden" name="external" value="">
<input type="hidden" name="hidepwd" value="">
<input type="hidden" name="hidequery" value="">
<input type="hidden" name="nopurge" value="">
${LANG_I33}
<br>
<select name="build"

View File

@@ -103,7 +103,7 @@ ${LANG_Q3}
<tr><td>
<table width="100%">
<tr><td align="left">
<input type="submit" value="${LANG_OK]"
<input type="submit" value="${LANG_OK}"
${do:output-mode:html-urlescaped}
onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
${do:output-mode:}

View File

@@ -98,6 +98,9 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="windebug" value="">
${LANG_I40c}
<br>

View File

@@ -98,6 +98,11 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="ka" value="">
<input type="hidden" name="remt" value="">
<input type="hidden" name="rems" value="">
<table border="0" width="100%" cellspacing="0">
<tr><td>

View File

@@ -98,6 +98,17 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="cookies" value="">
<input type="hidden" name="parsejava" value="">
<input type="hidden" name="updhack" value="">
<input type="hidden" name="urlhack" value="">
<input type="hidden" name="keepwww" value="">
<input type="hidden" name="keepslashes" value="">
<input type="hidden" name="keepqueryorder" value="">
<input type="hidden" name="toler" value="">
<input type="hidden" name="http10" value="">
<input type="checkbox" name="cookies" ${checked:cookies}
title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I58}

View File

@@ -98,6 +98,13 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="warc" value="">
<input type="hidden" name="norecatch" value="">
<input type="hidden" name="logf" value="">
<input type="hidden" name="index" value="">
<input type="hidden" name="index2" value="">
<input type="checkbox" name="cache2" ${checked:cache2}
title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I61}

View File

@@ -114,16 +114,16 @@ ${do:end-if}
${/* Real commands and ini file generated below */}
<!-- engine commandline -->
<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
${do:output-mode:html}
<textarea name="command" cols="50" rows="4" style="visibility:hidden">
httrack \
--quiet \
--build-top-index \
${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
${urls}
${test:filelist:-%L "}${filelist}${test:filelist:"}
--path "${html:path}/${html:projname}"
${unquoted:urls}
${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
--path "${arg:path}/${arg:projname}"
\
${test:parseall:--near}
${test:link:--test}
@@ -131,7 +131,7 @@ httrack \
${test:htmlfirst::--priority=7}
\
${do:if-not-empty:BuildString}
--structure "${BuildString}"
--structure "${arg:BuildString}"
${do:end-if}
${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
\
@@ -150,51 +150,52 @@ ${do:end-if}
${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
${test:windebug:--debug-headers}
\
${test:connexion:--sockets=}${connexion}
${test:connexion:--sockets=}${unquoted:connexion}
${test:ka:--keep-alive}
${test:timeout:--timeout=}${timeout}
${test:timeout:--timeout=}${unquoted:timeout}
${test:remt:--host-control=1}
${test:retry:--retries=}${retry}
${test:rate:--min-rate=}${rate}
${test:retry:--retries=}${unquoted:retry}
${test:rate:--min-rate=}${unquoted:rate}
${test:rems:--host-control=2}
\
${test:depth:--depth=}${depth}
${test:depth2:--ext-depth=}${depth2}
${test:maxhtml:--max-files=,}${maxhtml}
${test:othermax:--max-files=}${othermax}
${test:sizemax:--max-files=}${sizemax}
${test:pausebytes:--max-pause=}${pausebytes}
${test:maxtime:--max-time=}${maxtime}
${test:maxrate:--max-rate=}${maxrate}
${test:maxconn:--connection-per-second=}${maxconn}
${test:maxlinks:--advanced-maxlinks=}${maxlinks}
${test:depth:--depth=}${unquoted:depth}
${test:depth2:--ext-depth=}${unquoted:depth2}
${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
${test:othermax:--max-files=}${unquoted:othermax}
${test:maxhtml:--max-files=,}${unquoted:maxhtml}
${test:sizemax:--max-size=}${unquoted:sizemax}
${test:pausebytes:--max-pause=}${unquoted:pausebytes}
${test:maxtime:--max-time=}${unquoted:maxtime}
${test:maxrate:--max-rate=}${unquoted:maxrate}
${test:maxconn:--connection-per-second=}${unquoted:maxconn}
${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
\
--user-agent "${html:user}"
--footer "${html:footer}"
--user-agent "${arg:user}"
--footer "${arg:footer}"
\
${url2}
${unquoted:url2}
\
${test:cookies:--cookies=0:}
${test:parsejava:--parse-java=0:}
${ztest:cookies:--cookies=0:}
${ztest:parsejava:--parse-java=0:}
${test:updhack:--updatehack}
${test:urlhack:--urlhack=0:--urlhack}
${test:keepwww:--keep-www-prefix}
${test:keepslashes:--keep-double-slashes}
${test:keepqueryorder:--keep-query-order}
${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
${test:toler:--tolerant}
${test:http10:--http-10}
${test:cache2:--store-all-in-cache}
${test:warc:--warc}
${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
${test:norecatch:--do-not-recatch}
${test:logf:--single-log}
${test:logtype:::--extra-log:--debug-log}
${test:index:--index=0:}
${test:index2:--search-index=0:--search-index}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
</textarea>
@@ -213,7 +214,7 @@ ParseAll=${ztest:parseall:0:1}
HTMLFirst=${ztest:htmlfirst:0:1}
Cache=${ztest:cache:0:1}
NoRecatch=${ztest:norecatch:0:1}
Dos=${dos
Dos=${dos}
Index=${ztest:index:0:1}
WordIndex=${ztest:index2:0:1}
Log=${ztest:logf:0:1:2}

View File

@@ -71,7 +71,9 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
if (iisBogus[i][j] == '\0'
&& (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
/* j bytes matched, so j fit: copying j cannot overrun whatever the
table holds, and the tail must survive untouched */
memcpy(a, iisBogusReplace[i], (size_t) j);
break;
}
}

View File

@@ -48,7 +48,8 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
lib_LTLIBRARIES = libhttrack.la
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
htscmdline.c htscmdline.h \
htsurlport.c htsurlport.h
proxytrack_SOURCES = proxy/main.c \
proxy/proxytrack.c proxy/store.c \
@@ -62,7 +63,7 @@ whttrackrun_SCRIPTS = webhttrack
libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htscache_selftest.c htsdns_selftest.c htsselftest.c \
htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
htshelp.c htslib.c htsurlport.c htscoremain.c \
htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
htsname.c htsrobots.c htstools.c htswizard.c \
htsalias.c htsthread.c htsindex.c htsbauth.c \
htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
@@ -71,7 +72,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h \
htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
htsmodules.h htsname.h htsnet.h htssniff.h \
@@ -86,7 +87,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
EXTRA_DIST = httrack.h webhttrack \
EXTRA_DIST = httrack.h htsstats.h webhttrack \
version.rc \
libhttrack.rc \
httrack.rc \

View File

@@ -125,6 +125,22 @@ void back_free(struct_back ** sback) {
above a normal handshake. The last candidate still gets the full timeout. */
#define HTS_CONNECT_FALLBACK_TIMEOUT 10
void back_read_ftp_result(FILE *fp, htsblk *r) {
size_t j = 0;
if (fscanf(fp, "%d ", &r->statuscode) != 1)
r->statuscode = STATUSCODE_INVALID;
// an external helper writes this file: stop at capacity, not at EOF
while (j + 1 < sizeof(r->msg)) {
const int c = fgetc(fp);
if (c == EOF)
break;
r->msg[j++] = (char) c;
}
r->msg[j] = '\0';
}
int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
int timeout) {
int deadline;
@@ -542,8 +558,14 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
// do not use tempnam() but a regular filename
back->tmpfile_buffer[0] = '\0';
if (back->url_sav[0] != '\0') {
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
back->url_sav, ext);
/* same capacity as url_sav, so truncation drops the extension and aliases
the temp name onto the live file that back_finalize_backup() UNLINKs */
if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
back->url_sav);
back->tmpfile_buffer[0] = '\0';
return -1;
}
back->tmpfile = back->tmpfile_buffer;
if (structcheck(back->tmpfile) != 0) {
hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
@@ -551,8 +573,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
return -1;
}
} else {
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
/* truncation here would collide distinct tmpnameid's onto one name */
if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
ext)) {
hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
StringBuff(opt->path_html_utf8));
back->tmpfile_buffer[0] = '\0';
return -1;
}
back->tmpfile = back->tmpfile_buffer;
}
/* OK */
@@ -2947,7 +2976,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
back[i].r.msg[0] = '\0';
strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
if (!strnotempty(back[i].r.msg)) {
sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
}
deletehttp(&back[i].r);
back[i].r.soc = INVALID_SOCKET;
@@ -3020,16 +3049,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
"rb");
if (fp) {
int j = 0;
fscanf(fp, "%d ", &(back[i].r.statuscode));
while(!feof(fp)) {
int c = fgetc(fp);
if (c != EOF)
back[i].r.msg[j++] = c;
}
back[i].r.msg[j++] = '\0';
back_read_ftp_result(fp, &back[i].r);
fclose(fp);
UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
strcpybuff(fconcat
@@ -3330,10 +3350,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
deleteaddr(&back[i].r);
if (back[i].r.size < back[i].r.totalsize)
back[i].r.statuscode = STATUSCODE_CONNERROR; // recatch
sprintf(back[i].r.msg,
"Incorrect length (" LLintP " Bytes, " LLintP
" expected)", (LLint) back[i].r.size,
(LLint) back[i].r.totalsize);
htsblk_failf(&back[i].r,
"Incorrect length (" LLintP " Bytes, " LLintP
" expected)",
(LLint) back[i].r.size,
(LLint) back[i].r.totalsize);
} else {
// Un warning suffira..
hts_log_print(opt, LOG_WARNING,

View File

@@ -74,6 +74,10 @@ void back_free(struct_back ** sback);
// backing
#define BACK_ADD_TEST "(dummy)"
#define BACK_ADD_TEST2 "(dummy2)"
/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
clipping the message to r->msg. */
void back_read_ftp_result(FILE *fp, htsblk *r);
int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
const char *sav);
int back_available(const struct_back * sback);

View File

@@ -142,10 +142,12 @@ struct cache_back_zip_entry {
int compressionMethod;
};
/* A corrupt cache can carry a field wider than ours; clipping it keeps the
entry, where aborting would take the crawl down. */
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strlcpybuff(refvalue, value, refvalue_size); \
(void) strclipbuff(refvalue, refvalue_size, value); \
line[0] = '\0'; \
} \
} while (0)

View File

@@ -1195,6 +1195,12 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
/* --- read-side corruption injection --------------------------------------- */
/* 100 'A's: a placeholder header line long enough to be overwritten by a
forged, over-long X-StatusMessage of the same byte length. */
#define CORRUPT_LONG_ETAG \
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" \
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
/* canary read back intact after each corruption; victim gets the byte surgery
*/
#define CORRUPT_ADR "corrupt.example.com"
@@ -1219,6 +1225,24 @@ static void corrupt_build(httrackp *opt) {
selftest_close(&cache);
}
/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
static void corrupt_build_longetag(httrackp *opt) {
cache_back cache;
memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
remove(reconcile_st_path(opt, "hts-cache/new.zip"));
remove(reconcile_st_path(opt, "hts-cache/old.zip"));
selftest_open_for_write(&cache, opt);
store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
"OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
strlen(corrupt_body_a));
store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
"OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
corrupt_body_b, strlen(corrupt_body_b));
selftest_close(&cache);
}
/* Like corrupt_build, but the victim carries a 20-char Etag whose header line
is later overwritten with a forged oversized X-Size (same byte length). */
static void corrupt_build_etag(httrackp *opt) {
@@ -1403,6 +1427,48 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
return fail;
}
/* An over-long field from a foreign cache must clip, not abort: the entry
still reads, clipped to capacity, and the canary survives. */
static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
size_t wantlastmod, const char *what) {
cache_back cache;
htsblk v, c;
char BIGSTK lv[HTS_URLMAXSIZE * 2];
char BIGSTK lc[HTS_URLMAXSIZE * 2];
int fail = 0;
selftest_open_for_read(&cache, opt);
lv[0] = lc[0] = '\0';
v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
if (v.statuscode != 200) {
fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
v.statuscode);
fail++;
}
if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
(unsigned) strlen(v.msg), (unsigned) wantmsg);
fail++;
}
if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
fail++;
}
c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
if (c.statuscode != 200) {
fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
c.statuscode);
fail++;
}
if (v.adr != NULL)
freet(v.adr);
if (c.adr != NULL)
freet(c.adr);
selftest_close(&cache);
return fail;
}
/* One zip corruption case: build, patch, then check victim+canary in-session.
*/
static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1439,6 +1505,31 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
"garbled deflate stream");
/* A corrupt cache can hold a field wider than ours. Clipping keeps the
entry; aborting would take the crawl down. Overwrite the placeholder Etag
line in place, same byte length, so the zip offsets stay intact. */
corrupt_build_longetag(opt);
corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
"X-StatusMessage: " /* 17 + 89 = 106 */
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
1, 1);
failures +=
corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
(size_t) -1, "over-long X-StatusMessage");
/* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
cannot satisfy both. */
corrupt_build_longetag(opt);
corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
"Last-Modified: " /* 15 + 91 = 106 */
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
1, 1);
failures += corrupt_expect_victim_clipped(
opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
"over-long Last-Modified");
/* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
truncates negative in the (int) cast the malloc uses: a wraparound alloc.
cache_add asserts size fits an int, so such a value only reaches the reader

95
src/htscmdline.c Normal file
View File

@@ -0,0 +1,95 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: command line splitter, shared by the engine and */
/* htsserver */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#include "htscmdline.h"
#include "htssafe.h"
#include <limits.h>
#include <stdint.h>
char **hts_split_cmdline(char *cmd, int *nargs) {
size_t nsep = 0;
size_t capacity;
size_t r;
size_t w;
int argc = 0;
hts_boolean quoted = HTS_FALSE;
char **argv;
*nargs = 0;
/* fold the other separators, so counting them sizes the vector exactly */
for (r = 0; cmd[r] != '\0'; r++) {
if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
cmd[r] = ' ';
}
if (cmd[r] == ' ') {
nsep++;
}
}
/* at most one argument per separator, plus the leading one and the NULL */
if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
return NULL;
}
capacity = nsep + 2;
argv = (char **) malloct(capacity * sizeof(char *));
if (argv == NULL) {
return NULL;
}
argv[argc++] = cmd;
for (r = 0, w = 0; cmd[r] != '\0';) {
if (quoted && cmd[r] == '\\' &&
(cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
r++;
cmd[w++] = cmd[r++];
} else if (cmd[r] == '\"') {
quoted = !quoted;
cmd[w++] = cmd[r++];
} else if (cmd[r] == ' ' && !quoted) {
cmd[w++] = '\0';
assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
argv[argc++] = cmd + w;
r++;
} else {
cmd[w++] = cmd[r++];
}
}
cmd[w] = '\0';
argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
*nargs = argc;
return argv;
}

45
src/htscmdline.h Normal file
View File

@@ -0,0 +1,45 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: command line splitter, shared by the engine and */
/* htsserver */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTSCMDLINE_DEFH
#define HTSCMDLINE_DEFH
#include "htsglobal.h"
/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
argv[0] being the program name and quotes left for the engine to strip.
Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
entries), or NULL when it cannot be sized or allocated. */
char **hts_split_cmdline(char *cmd, int *nargs);
#endif

View File

@@ -447,13 +447,22 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
const char *fil) {
if (!*makeindex_done) {
if (*makeindex_fp) {
char BIGSTK tempo[1024];
/* sized off link_escaped below: at the old flat 1024 a long first link
produced a redirect to a clipped URL */
char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
if (makeindex_links == 1) {
char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
snprintf(tempo, sizeof(tempo),
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped);
/* no redirect beats one pointing at a clipped URL */
if (!sprintfbuff(
tempo,
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped)) {
hts_log_print(opt, LOG_WARNING,
"index redirect omitted: first link too long (%s)",
makeindex_firstlink);
tempo[0] = '\0';
}
} else
tempo[0] = '\0';
hts_template_format(*makeindex_fp, template_footer,

View File

@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// folding a nonsense port into 1..65535 fetches one the link never named;
// an empty "host:" just means the default (#614)
if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
htsblk_failf(&back->r, "Invalid port: %s", a + 1);
back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
_HALT_FTP return 0;
}
@@ -262,8 +262,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// récupérer adresse résolue
strcpybuff(back->info, "host name");
if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
snprintf(back->r.msg, sizeof(back->r.msg),
"Unable to get server's address: %s", error);
htsblk_failf(&back->r, "Unable to get server's address: %s", error);
back->r.statuscode = STATUSCODE_NON_FATAL;
_HALT_FTP return 0;
}
@@ -332,18 +331,15 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
linejmp(line));
htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
linejmp(line));
htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
linejmp(line));
htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
@@ -410,8 +406,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
// -- fin analyse de l'adresse IP et du port --
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
@@ -442,13 +437,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
linejmp(line));
htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
}
@@ -554,8 +547,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
deletesoc(soc_dat);
soc_dat = INVALID_SOCKET;
//
snprintf(back->r.msg, sizeof(back->r.msg),
"RETR command error: %s", linejmp(line));
htsblk_failf(&back->r, "RETR command error: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
@@ -573,13 +566,12 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
snprintf(back->r.msg, sizeof(back->r.msg),
"Unable to resolve IP %s: %s", adr_ip, error);
htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
error);
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
#else
@@ -603,13 +595,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg),
"RETR command error: %s", linejmp(line));
htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
linejmp(line));
htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
#ifdef _WIN32
@@ -652,8 +642,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
len = 0; // fin
break;
case 0:
snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
timeout);
htsblk_failf(&back->r, "Time out (%d)", timeout);
back->r.statuscode = STATUSCODE_INVALID;
len = 0; // fin
break;
@@ -716,8 +705,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
strcpybuff(back->r.msg, "OK");
back->r.statuscode = HTTP_OK;
} else {
snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
linejmp(line));
htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {

View File

@@ -72,7 +72,8 @@ Please visit our Website: http://www.httrack.com
HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
compiler type-checks the format string at argument index fmt against the
varargs starting at arg. */
varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
error signal, so dropping it is a bug; a (void) cast does not silence it. */
#ifndef HTS_UNUSED
#ifdef __GNUC__
#define HTS_UNUSED __attribute__((unused))
@@ -80,10 +81,13 @@ Please visit our Website: http://www.httrack.com
#define HTS_STATIC static __attribute__((unused))
#define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
#else
#define HTS_UNUSED
#define HTS_STATIC static
#define HTS_PRINTF_FUN(fmt, arg)
#define HTS_CHECK_RESULT
#endif
#endif

View File

@@ -124,7 +124,9 @@ typedef struct help_wizard_buffers {
char stropt[2048]; // options
char stropt2[2048]; // options longues
char strwild[2048]; // wildcards
char cmd[4096];
/* holds all four of the above plus separators: at 4096 a long answer set
clipped the filters off the command line */
char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
char str[256];
char *argv[256];
} help_wizard_buffers;
@@ -308,7 +310,14 @@ void help_wizard(httrackp * opt) {
printf("\n");
if (strlen(stropt) == 1)
stropt[0] = '\0'; // aucune
snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
/* the tail is the filter list, and cmd is split into the argv handed to
hts_main() below: a clipped line would silently widen the crawl */
if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
printf("* command line too long (%d bytes max)\n",
(int) sizeof(cmd) - 1);
freet(buffers);
return;
}
printf("---> Wizard command line: httrack %s\n\n", cmd);
printf("Ready to launch the mirror? (Y/n) :");
fflush(stdout);
@@ -424,7 +433,8 @@ void help_catchurl(const char *dest_path) {
}
// former URL!
{
char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
/* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
inplace_escape_check_url(dest, sizeof(dest));
snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);

View File

@@ -708,11 +708,11 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
#ifdef _WIN32
int last_errno = WSAGetLastError();
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
#else
int last_errno = errno;
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
#endif
}
}
@@ -2240,13 +2240,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
#ifdef _WIN32
int last_errno = WSAGetLastError();
sprintf(retour->msg, "Unable to create a socket: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to create a socket: %s",
strerror(last_errno));
#else
int last_errno = errno;
sprintf(retour->msg, "Unable to create a socket: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to create a socket: %s",
strerror(last_errno));
#endif
}
return INVALID_SOCKET; // erreur création socket impossible
@@ -2312,13 +2312,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
#ifdef _WIN32
const int last_errno = WSAGetLastError();
sprintf(retour->msg, "Unable to connect to the server: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to connect to the server: %s",
strerror(last_errno));
#else
const int last_errno = errno;
sprintf(retour->msg, "Unable to connect to the server: %s",
strerror(last_errno));
htsblk_failf(retour, "Unable to connect to the server: %s",
strerror(last_errno));
#endif
}
/* Close the socket and notify the error!!! */

View File

@@ -199,6 +199,11 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
etc.). */
T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
int waitconnect, int addr_index, int *addr_count);
/* Clips the formatted failure reason into r->msg, which also round-trips
through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
#define htsblk_failf(R, ...) \
slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
HTS_INLINE void deletehttp(htsblk * r);
HTS_INLINE int deleteaddr(htsblk * r);
HTS_INLINE void deletesoc(T_SOC soc);

View File

@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
code = 0;
if (code < 200 || code >= 300) {
snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
htsblk_failf(retour, "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
return 0;
}

View File

@@ -33,6 +33,7 @@ Please visit our Website: http://www.httrack.com
#ifndef HTSSAFE_DEFH
#define HTSSAFE_DEFH
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -459,6 +460,85 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
return b->buf;
}
/**
* Copy src into dest (capacity size, NUL included), truncating to fit and
* always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
* value read back from a cache, a header or the wire, where refusing the whole
* record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
* callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
*/
static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
const char *src) {
size_t len, copy;
assertf(dest != NULL && src != NULL && size != 0);
len = strlen(src);
copy = len < size ? len : size - 1;
memcpy(dest, src, copy);
dest[copy] = '\0';
return copy == len ? HTS_TRUE : HTS_FALSE;
}
/**
* Callers that deliberately ignore truncation use this instead of
* slprintfbuff(), so it is not HTS_CHECK_RESULT.
*/
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
int ret;
assertf(dest != NULL && size != 0);
/* a vsnprintf failing outright may write nothing at all, leaving whatever
the caller had on the stack for it to publish */
dest[0] = '\0';
ret = vsnprintf(dest, size, fmt, args);
/* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
dest[size - 1] = '\0';
return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
}
/**
* Formatted print into dest (capacity size, NUL included), truncating to fit
* and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
* result is the only truncation signal, so it must be acted on. Unlike
* strcpybuff() it never aborts, so it suits text built from remote input.
*/
static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
va_list args;
hts_boolean ret;
va_start(args, fmt);
ret = vslprintfbuff(dest, size, fmt, args);
va_end(args);
return ret;
}
/**
* slprintfbuff() for diagnostics quoting remote or client text, which are
* meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
* cast on slprintfbuff() is no substitute, GCC warns through it.
*/
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
char *dest, size_t size, const char *fmt, ...) {
va_list args;
va_start(args, fmt);
(void) vslprintfbuff(dest, size, fmt, args);
va_end(args);
}
/**
* slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
* On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
*/
#if (defined(__GNUC__) && !defined(__cplusplus))
#define sprintfbuff(ARR, ...) \
slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
#else
#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
#endif
/* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
added bounds checking. freet() also NULLs the freed pointer and tolerates
NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

View File

@@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htsback.h"
#include "htsdefines.h"
#include "htslib.h"
#include "htsalias.h"
@@ -51,6 +52,7 @@ Please visit our Website: http://www.httrack.com
#include "htscache_selftest.h"
#include "htsdns_selftest.h"
#include "htscharset.h"
#include "htscmdline.h"
#include "htsencoding.h"
#include "htsftp.h"
#include "htsmd5.h"
@@ -455,6 +457,12 @@ static void basic_selftests(void) {
// link one level up -> a "../" prefix
assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
assertf(strcmp(s, "../a.html") == 0);
// an empty current path: the trim used to walk off the front of it, which
// "?x" reaches too because the query pre-pass hands on the part before it
assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
assertf(strcmp(s, "dir/page.html") == 0);
assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
assertf(strcmp(s, "dir/page.html") == 0);
}
}
@@ -568,6 +576,214 @@ static int string_safety_selftests(void) {
return 1;
}
/* sprintfbuff: truncate-and-report. Must never abort (its callers format
remote banners) nor write past the array, which the canary catches. */
{
struct {
char dst[8];
char canary[8];
} s;
const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
/* repoison before every call, or an implementation that measures first and
writes nothing "passes" the truncating cases on the previous content */
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
memset(&s, '#', sizeof(s));
if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
return 1;
/* exact fit: 7 characters plus the NUL */
POISON_DST();
if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
return 1;
/* one over, then far over: truncated to the prefix, terminated, reported */
POISON_DST();
if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
return 1;
POISON_DST();
if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
return 1;
/* explicit-capacity form, down to the degenerate size 1 */
{
char *const p = s.dst;
POISON_DST();
if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
return 1;
POISON_DST();
if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
return 1;
}
#undef POISON_DST
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
return 1;
}
/* strclipbuff: truncate-and-report, never abort. Same canary shape; the
destination is poisoned before every call so a case cannot pass on the
previous one's bytes. */
{
struct {
char dst[8];
char canary[8];
} s;
memset(&s, '#', sizeof(s));
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
/* well under capacity: no padding, nothing eaten off the end */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "abc") || strcmp(s.dst, "abc") != 0)
return 1;
/* exact fit: capacity - 1 characters plus the NUL */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "1234567") ||
strcmp(s.dst, "1234567") != 0)
return 1;
/* one over, then far over: clipped, terminated, reported. The expected
bytes differ from the case above, so a write-nothing implementation
cannot pass on the leftovers. */
POISON_DST();
if (strclipbuff(s.dst, sizeof(s.dst), "abcdefgh") ||
strcmp(s.dst, "abcdefg") != 0)
return 1;
POISON_DST();
if (strclipbuff(s.dst, sizeof(s.dst), "0123456789abcdef") ||
strcmp(s.dst, "0123456") != 0)
return 1;
/* degenerate capacity 1: only the NUL fits. Capacity 2 pins the boundary
between that and the sizes above: one character plus the NUL. */
POISON_DST();
if (strclipbuff(s.dst, 1, "x") || s.dst[0] != '\0')
return 1;
POISON_DST();
if (strclipbuff(s.dst, 2, "yz") || strcmp(s.dst, "y") != 0)
return 1;
/* the empty string fits any non-zero capacity */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "") || s.dst[0] != '\0')
return 1;
/* a byte over 0x7f must not end the copy early */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "\xff\xfe") ||
strcmp(s.dst, "\xff\xfe") != 0)
return 1;
#undef POISON_DST
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
return 1;
}
/* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
touches nothing else in the block */
{
htsblk r;
char expect[sizeof(r.msg)];
char big[4 * sizeof(r.msg)];
/* contenttype abuts msg, so a one-past-the-end store lands in it rather
than in padding. Poison it: a stray NUL is invisible against zeroes,
and a stray NUL is exactly what an off-by-one terminator writes. */
#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
memset(&r, 0, sizeof(r));
memset(r.contenttype, '#', sizeof(r.contenttype));
r.statuscode = 1234;
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
return 1;
/* exact fit: capacity - 1 characters plus the NUL */
memset(expect, 'y', sizeof(expect) - 1);
expect[sizeof(expect) - 1] = '\0';
memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "%s", expect);
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
return 1;
/* far over: the expected bytes differ from the cases above, so writing
nothing cannot pass on the leftovers */
memset(big, 'z', sizeof(big) - 1);
big[sizeof(big) - 1] = '\0';
memset(expect, 'z', sizeof(expect) - 1);
expect[sizeof(expect) - 1] = '\0';
memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "Bad user name: %s", big);
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
return 1;
#undef NEIGHBOURS_INTACT
}
/* back_read_ftp_result: the helper's result file is external input, so an
over-long message must stop at msg[]'s capacity */
{
htsblk r;
size_t k;
/* poisoned so a short message cannot pass on leftovers, and so a stray
NUL past msg[] is visible in the neighbour */
#define FTP_RESULT_CASE(BODY) \
do { \
FILE *fp_ = tmpfile(); \
\
if (fp_ == NULL) \
return 1; \
BODY; \
rewind(fp_); \
memset(&r, 0, sizeof(r)); \
memset(r.msg, '#', sizeof(r.msg)); \
memset(r.contenttype, '#', sizeof(r.contenttype)); \
back_read_ftp_result(fp_, &r); \
fclose(fp_); \
if (r.contenttype[0] != '#') \
return 1; \
} while (0)
/* over capacity: clipped to 79 payload bytes plus the NUL */
FTP_RESULT_CASE({
fprintf(fp_, "226 ");
for (k = 0; k < 4 * sizeof(r.msg); k++)
fputc('q', fp_);
});
if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
return 1;
for (k = 0; k < sizeof(r.msg) - 1; k++) {
if (r.msg[k] != 'q')
return 1;
}
/* well under capacity: nothing padded, nothing eaten off the end */
FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
return 1;
/* a byte over 0x7f must not read as EOF and cut the message short */
FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
return 1;
/* unparseable status: the message still loads, the code reports failure */
FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
if (r.statuscode != STATUSCODE_INVALID)
return 1;
#undef FTP_RESULT_CASE
}
/* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
n_eval == 2 on a double-eval macro. */
{
@@ -1126,6 +1342,128 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
return 0;
}
// hts_split_cmdline(): the vector must grow with the argument count, and a
// quote inside a value must not end the argument and hand -V to the parser.
static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
char line[512];
char **args;
int nargs = 0;
(void) opt;
(void) argc;
(void) argv;
// control: every separator splits, and argv[0] is the program name
strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
args = hts_split_cmdline(line, &nargs);
assertf(args != NULL && nargs == 6);
assertf(args[nargs] == NULL); // callers may walk to the terminator
assertf(strcmp(args[0], "httrack") == 0);
assertf(strcmp(args[1], "http://x/") == 0);
assertf(strcmp(args[2], "--quiet") == 0);
assertf(strcmp(args[3], "-c8") == 0);
assertf(strcmp(args[4], "-O") == 0);
assertf(strcmp(args[5], "out") == 0);
freet(args);
// the template pads with whitespace: empty arguments are kept (the engine
// skips them), so the count is one per separator
strcpybuff(line, "httrack --quiet");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3 && args[1][0] == '\0');
assertf(strcmp(args[2], "--quiet") == 0);
freet(args);
// a quoted run keeps both its spaces and its quotes: the engine unquotes
strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 4);
assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
assertf(strcmp(args[3], "-c8") == 0);
freet(args);
// an escaped quote is a literal quote, not the end of the argument: the
// engine strips only the outer pair
strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 4);
assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
assertf(strcmp(args[3], "-c8") == 0);
freet(args);
// \\ is a literal backslash, so a Windows path survives
strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
freet(args);
// outside a quoted run a backslash is literal: the url and wildcard-filter
// fields, which the wizard cannot quote, read as before
strcpybuff(line, "httrack -*\\** +*.png");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[1], "-*\\**") == 0);
assertf(strcmp(args[2], "+*.png") == 0);
freet(args);
// a quoted run leaves slots unused, so the terminator has to be written and
// not inherited: size the vector from a full line first, so freeing it hands
// the same chunk back with stale pointers in those slots
strcpybuff(line, "httrack a b c d e");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 6);
freet(args);
strcpybuff(line, "httrack \"a b c d e\"");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 2);
assertf(args[nargs] == NULL);
freet(args);
// an unterminated quote protects the rest of the line, as one argument
strcpybuff(line, "httrack --footer \"unbalanced -V x");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
freet(args);
// past the 1024 entries the vector used to hold: distinct arguments, so a
// write beyond the allocation cannot read back as the expected parse
{
const int n = 2000;
const size_t size = 16 * (size_t) n + 16;
char *big = malloct(size);
size_t pos = 0;
int i;
assertf(big != NULL);
pos = (size_t) snprintf(big, size, "httrack");
assertf(pos < size);
for (i = 0; i < n; i++) {
// snprintf returns what it wanted to write, so accumulating it blind
// would let the next size argument wrap
const int len = snprintf(big + pos, size - pos, " a%d", i);
assertf(len > 0 && (size_t) len < size - pos);
pos += (size_t) len;
}
args = hts_split_cmdline(big, &nargs);
assertf(args != NULL && nargs == n + 1);
assertf(args[nargs] == NULL);
for (i = 0; i < n; i++) {
char expect[16];
snprintf(expect, sizeof(expect), "a%d", i);
assertf(strcmp(args[i + 1], expect) == 0);
}
freet(args);
freet(big);
}
printf("cmdline-split self-test OK\n");
return 0;
}
static int st_hashtable(httrackp *opt, int argc, char **argv) {
char *snum;
unsigned long count = 0;
@@ -1247,7 +1585,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
size_t i;
for (i = bench[loop].offset; i < (size_t) count;
i += bench[loop].modulus) {
int result;
int result = 0; /* no final else: an unknown type reports failure */
FMT();
if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
size_t k;
@@ -2762,6 +3100,32 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
assertf(strstr(buf, "Refresh") != NULL);
assertf(strstr(buf, "example.com") != NULL);
/* a first link whose escaped form overruns the old flat 1024-byte tempo: the
redirect must carry the whole URL, not a clipped prefix */
{
char BIGSTK link[HTS_URLMAXSIZE * 2];
char *p = link;
strcpybuff(link, "http://example.com/");
p += strlen(link);
memset(p, 'a', 1200);
p += 1200;
strcpy(p, "/end.html");
done = 0;
fp = fopen(path, "wb");
assertf(fp != NULL);
hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
assertf(fp == NULL);
fp = fopen(path, "rb");
assertf(fp != NULL);
n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
buf[n] = '\0';
/* the closing quote proves the URL was not clipped mid-way */
assertf(strstr(buf, "/end.html\">") != NULL);
}
/* no single link: footer only, no refresh meta */
done = 0;
fp = fopen(path, "wb");
@@ -4801,10 +5165,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
char fpath[HTS_URLMAXSIZE * 2];
char file[HTS_URLMAXSIZE * 2];
snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
/* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
snprintf(file, sizeof(file), "%scookies.txt", fpath);
assertf(sprintfbuff(file, "%scookies.txt", fpath));
{
FILE *fp = FOPEN(file, "wb");
@@ -4814,7 +5178,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
}
/* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
snprintf(file, sizeof(file), "%su@v.txt", fpath);
assertf(sprintfbuff(file, "%su@v.txt", fpath));
{
FILE *fp = FOPEN(file, "wb");
@@ -4836,7 +5200,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
#endif
(void) UNLINK(file); /* u@v.txt (already gone on Windows) */
snprintf(file, sizeof(file), "%scookies.txt", fpath);
assertf(sprintfbuff(file, "%scookies.txt", fpath));
(void) UNLINK(file);
dir[dirlen] = '\0';
while (strlen(dir) > base) {
@@ -4910,6 +5274,9 @@ static const struct selftest_entry {
st_footerfmt},
{"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
st_unescape_bounds},
{"cmdline-split", "",
"webhttrack command-line to argv split (bounds, quoting)",
st_cmdlinesplit},
{"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
{"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
"bounded string-op self-test", st_strsafe},

View File

@@ -318,6 +318,18 @@ typedef struct {
error_redirect = "/server/error.html"; \
} while(0)
/* Longest error message shown on the error page; the rest is clipped. */
#define ERROR_MESSAGE_MAX 1024
/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
whose length the client picks. */
#define SET_ERRORF(...) \
do { \
char errbuf[ERROR_MESSAGE_MAX]; \
slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__); \
SET_ERROR(errbuf); \
} while (0)
/* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
so anything near this is already invalid and is rejected unread. */
#define SID_VALUE_MAX 64
@@ -363,6 +375,59 @@ static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
return seen;
}
/** Append src to the NUL-terminated dst of capacity size (NUL included).
False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
never aborts, because every piece appended here is client-supplied. */
static hts_boolean path_append(char *dst, size_t size, const char *src) {
const size_t used = strlen(dst);
const size_t len = strlen(src);
/* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
len stays alone: "used + len < size" could wrap and pass. */
if (len >= size - used) {
return HTS_FALSE;
}
memcpy(dst + used, src, len + 1);
return HTS_TRUE;
}
/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
static hts_boolean cat_html_escaped(String *dst, char c) {
switch (c) {
case '<':
StringCat(*dst, "&lt;");
break;
case '>':
StringCat(*dst, "&gt;");
break;
case '&':
StringCat(*dst, "&amp;");
break;
case '\'':
StringCat(*dst, "&#39;");
break;
default:
return HTS_FALSE;
}
return HTS_TRUE;
}
/* Append the value of a double-quoted command-line argument: escaped for HTML,
which the browser undoes when it posts the command line back, and for the
argv splitter, which does not. */
static void cat_cmdline_arg(String *output, const char *value) {
const char *a;
for (a = value; *a != '\0'; a++) {
if (*a == '\\' || *a == '\"') {
StringCat(*output, "\\");
}
if (!cat_html_escaped(output, *a)) {
StringMemcat(*output, a, 1);
}
}
}
int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
int timeout = 30;
int retour = 0;
@@ -374,6 +439,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
String tmpbuff = STRING_EMPTY;
String tmpbuff2 = STRING_EMPTY;
String fspath = STRING_EMPTY;
/* Project directory this server set up; the only root /website/ serves from,
and deliberately not cleared between requests. */
String website = STRING_EMPTY;
char catbuff[CATBUFF_SIZE];
/* Load strings */
@@ -669,7 +737,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
fp = fopen(StringBuff(fspath), "rb");
if (fp) {
/* Read file */
while(!feof(fp)) {
while (!feof(fp) && !ferror(fp)) {
char *str = line;
char *pos;
@@ -785,6 +853,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
if (!structcheck(StringBuff(tmpbuff))) {
FILE *fp;
/* Both halves of fspath come from posted fields, so a ".."
in them would escape the mirror once served. */
if (strstr(StringBuff(fspath), "..") == NULL) {
StringCopy(website, StringBuff(fspath));
}
StringCat(tmpbuff, "winprofile.ini");
fp = fopen(StringBuff(tmpbuff), "wb");
if (fp != NULL) {
@@ -818,28 +891,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
commandEnd = 1;
}
} else {
char tmp[1024];
sprintf(tmp,
"Unable to write %d bytes in the the init file %s",
count, StringBuff(fspath));
SET_ERROR(tmp);
SET_ERRORF(
"Unable to write %d bytes in the the init file %s",
count, StringBuff(fspath));
}
fclose(fp);
} else {
char tmp[1024];
sprintf(tmp, "Unable to create the init file %s",
StringBuff(fspath));
SET_ERROR(tmp);
SET_ERRORF("Unable to create the init file %s",
StringBuff(fspath));
}
} else {
char tmp[1024];
sprintf(tmp,
"Unable to create the directory structure in %s",
StringBuff(fspath));
SET_ERROR(tmp);
SET_ERRORF("Unable to create the directory structure in %s",
StringBuff(fspath));
}
} else {
@@ -857,7 +920,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
/* Response */
if (meth) {
int virtualpath = 0;
hts_boolean virtualpath = HTS_FALSE;
char *pos;
char *url = strchr(line1, ' ');
@@ -868,11 +931,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
char *qpos;
/* get the URL */
fsfile[0] = '\0';
if (error_redirect == NULL) {
if ((qpos = strchr(url, '?'))) {
*qpos = '\0';
}
fsfile[0] = '\0';
if (strcmp(url, "/") == 0) {
file = "/server/index.html";
meth = 2;
@@ -885,7 +948,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
if (strncmp(file, "/website/", 9) == 0) {
virtualpath = 1;
virtualpath = HTS_TRUE;
}
/* override */
@@ -899,20 +962,28 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
if (strncmp(file, "/website/", 9) != 0) {
sprintf(fsfile, "%shtml%s", path, file);
} else {
intptr_t adr = 0;
/* the override above may have swapped a mirror path for a GUI page */
virtualpath = strncmp(file, "/website/", 9) == 0;
if (coucal_readptr(NewLangList, "projpath", &adr)) {
sprintf(fsfile, "%s%s", (char *) adr, file + 9);
}
if (!virtualpath) {
if (!path_append(fsfile, sizeof(fsfile), path) ||
!path_append(fsfile, sizeof(fsfile), "html") ||
!path_append(fsfile, sizeof(fsfile), file)) {
fsfile[0] = '\0';
}
} else if (StringNotEmpty(website)) {
/* Never the posted "projpath": a client root reads any file. */
if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
!path_append(fsfile, sizeof(fsfile), "/") ||
!path_append(fsfile, sizeof(fsfile), file + 9)) {
fsfile[0] = '\0';
}
}
if (fsfile[0] && strstr(file, "..") == NULL
&& (fp = fopen(fsfile, "rb"))) {
/* Regular files only: reading a directory or FIFO never ends, and
"path" may hold "..", so only the untrusted halves are checked. */
if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
(fp = fopen(fsfile, "rb"))) {
char ok[] =
"HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
"Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
@@ -965,11 +1036,13 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringCat(headers, "\r\n");
}
coucal_write(NewLangList, "redirect", (intptr_t) NULL);
} else if (is_html(file)) {
} else if (!virtualpath && is_html(file)) {
/* GUI templates only: ${_sid} in a mirrored page would hand the
crawled site the session id that authenticates commands */
int outputmode = 0;
StringMemcat(headers, ok, sizeof(ok) - 1);
while(!feof(fp)) {
while (!feof(fp) && !ferror(fp)) {
char *str = line;
int prevlen = (int) StringLength(output);
int nocr = 0;
@@ -993,6 +1066,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
int p;
int format = 0;
int listDefault = 0;
hts_boolean unquoted = HTS_FALSE;
name[0] = '\0';
strlncatbuff(name, str, sizeof(name_), n);
@@ -1002,6 +1076,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
} else if ((p = strfield(name, "html:"))) {
name += p;
format = 1;
} else if ((p = strfield(name, "unquoted:"))) {
name += p;
unquoted = HTS_TRUE;
} else if ((p = strfield(name, "arg:"))) {
name += p;
format = 5;
} else if ((p = strfield(name, "list:"))) {
name += p;
format = 2;
@@ -1138,8 +1218,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
test:<if ==0>:<if ==1>:<if == 2>..
ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
*/
else if ((p = strfield(name, "test:"))
|| (p = strfield(name, "ztest:"))) {
else if ((p = strfield(name, "test:")) ||
(p = strfield(name, "ztest:"))) {
intptr_t adr = 0;
char *pos2;
int ztest = (name[0] == 'z');
@@ -1242,6 +1322,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
}
/* consumed here: it shares nothing with the list and
option formats below */
if (format == 5 && langstr != NULL && outputmode != -1) {
cat_cmdline_arg(&output, langstr);
langstr = NULL;
}
if (langstr && outputmode != -1) {
switch (format) {
case 0:
@@ -1259,18 +1345,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringMemcat(output, &c, 1);
}
a += 2;
} else if (outputmode && a[0] == '<') {
StringCat(output, "&lt;");
} else if (outputmode && a[0] == '>') {
StringCat(output, "&gt;");
} else if (outputmode && a[0] == '&') {
StringCat(output, "&amp;");
} else if (outputmode && a[0] == '\'') {
StringCat(output, "&#39;");
} else if (unquoted && a[0] == '\"') {
/* the browser posts an entity back as a raw
quote, which would open a quoted run in the
argv splitter; a URI cannot hold one anyway */
StringCat(output, "%22");
} else if (outputmode &&
cat_html_escaped(&output, a[0])) {
/* appended as an entity */
} else if (outputmode == 3 && a[0] == ' ') {
StringCat(output, "%20");
} else if (outputmode >= 2
&& ((unsigned char) a[0]) < 32) {
} else if (outputmode >= 2 &&
((unsigned char) a[0]) < 32) {
char tmp[32];
sprintf(tmp, "%%%02x", (unsigned char) a[0]);
@@ -1331,20 +1417,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
StringClear(tmpbuff);
break;
case '<':
StringCat(tmpbuff, "&lt;");
break;
case '>':
StringCat(tmpbuff, "&gt;");
break;
case '&':
StringCat(tmpbuff, "&amp;");
break;
case '\'':
StringCat(tmpbuff, "&#39;");
break;
default:
StringMemcat(tmpbuff, fstr, 1);
if (!cat_html_escaped(&tmpbuff, *fstr)) {
StringMemcat(tmpbuff, fstr, 1);
}
break;
}
fstr++;
@@ -1384,7 +1460,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
#endif
} else {
if (is_text(file)) {
if (is_html(file)) {
StringMemcat(headers, ok, sizeof(ok) - 1);
} else if (is_text(file)) {
StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
} else if (is_js(file)) {
StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
@@ -1398,14 +1476,15 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
while(!feof(fp)) {
int n = (int) fread(line, 1, sizeof(line) - 2, fp);
if (n > 0) {
StringMemcat(output, line, n);
if (n <= 0) {
break; /* short read: EOF or error, never a retry */
}
StringMemcat(output, line, n);
}
}
fclose(fp);
} else if (strcmp(file, "/ping") == 0
|| strncmp(file, "/ping?", 6) == 0) {
} else if (strcmp(file, "/ping") == 0 ||
strncmp(file, "/ping?", 6) == 0) {
char error_hdr[] =
"HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
"Content-type: text/html\r\n";
@@ -1491,6 +1570,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringFree(tmpbuff);
StringFree(tmpbuff2);
StringFree(fspath);
StringFree(website);
if (buffer)
free(buffer);

63
src/htsstats.h Normal file
View File

@@ -0,0 +1,63 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: in-progress display rows, shared by httrack and htsserver .h */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTS_STATS_DEFH
#define HTS_STATS_DEFH
#include "htsglobal.h"
/* One row of the "in progress" display, shared so the CLI (httrack) and the
web GUI (htsserver) cannot drift apart: each fills its own array. */
#define NStatsBuffer 14
#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
#define HTS_DEF_FWSTRUCT_t_StatsBuffer
typedef struct t_StatsBuffer t_StatsBuffer;
#endif
struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[288]; // a short label plus back->info[256]
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
LLint size;
LLint sizetot;
int offset;
//
int back;
//
int actived; // pour disabled
};
#endif

View File

@@ -308,8 +308,10 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
// copy only the current path
curr = _curr;
strlcpybuff(curr, curr_fil, sizeof(_curr));
if ((a = strchr(curr, '?')) == NULL) // couper au ? (params)
a = curr + strlen(curr) - 1; // pas de params: aller à la fin
if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
// an empty path has no last character: curr-1 would read before the buffer
a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
}
while((*a != '/') && (a > curr))
a--; // chercher dernier / du chemin courant
if (*a == '/')

View File

@@ -62,6 +62,7 @@ Please visit our Website: http://www.httrack.com
#include "htsmd5.c"
#include "md5.c"
#include "htscmdline.h"
#include "htsserver.h"
#include "htsurlport.h"
#include "htsweb.h"
@@ -319,10 +320,8 @@ int main(int argc, char *argv[]) {
static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
static void back_launch_cmd(void *pP) {
char *cmd = (char *) pP;
char **argv = (char **) malloct(1024 * sizeof(char *));
char **argv;
int argc = 0;
int i = 0;
int g = 0;
//
httrackp *opt;
@@ -333,28 +332,19 @@ static void back_launch_cmd(void *pP) {
commandReturnCmdl = strdup(cmd);
/* split */
argv[0] = strdup("webhttrack");
argv[1] = cmd;
argc++;
i = 0;
while(cmd[i]) {
if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
cmd[i] = ' ';
}
i++;
}
i = 0;
while(cmd[i]) {
if (cmd[i] == '\"')
g = !g;
if (cmd[i] == ' ') {
if (!g) {
cmd[i] = '\0';
argv[argc++] = cmd + i + 1;
}
}
i++;
argv = hts_split_cmdline(cmd, &argc);
if (argv == NULL) {
if (commandReturnMsg)
free(commandReturnMsg);
commandReturnMsg = strdup("could not parse the command line");
commandReturn = -1;
commandRunning = 0;
commandEnd = 1;
free(cmd);
return;
}
/* drop the program name the posted command line carries */
argv[0] = strdupt("webhttrack");
/* init */
hts_init();
@@ -383,6 +373,7 @@ static void back_launch_cmd(void *pP) {
/* free */
free(cmd);
freet(argv[0]);
freet(argv);
return;
}

View File

@@ -35,26 +35,10 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htsstats.h"
#define NStatsBuffer 14
#define MAX_LEN_INPROGRESS 40
typedef struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[256];
char url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char url_adr[HTS_URLMAXSIZE * 2];
char url_fil[HTS_URLMAXSIZE * 2];
LLint size;
LLint sizetot;
int offset;
//
int back;
//
int actived; // pour disabled
} t_StatsBuffer;
typedef struct t_InpInfo {
int ask_refresh;
int refresh;

View File

@@ -230,8 +230,7 @@ static hts_boolean vt_size_refresh(void) {
*/
#define STYLE_STATVALUES VT_BOLD
#define STYLE_STATTEXT VT_UNBOLD
#define STYLE_STATRESET VT_UNBOLD
#define NStatsBuffer 14
#define STYLE_STATRESET VT_UNBOLD
/* Rows the stats block and "Current job" take above the in-progress list. */
#define NStatsHeaderRows 7

View File

@@ -36,26 +36,7 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htssafe.h"
#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
#define HTS_DEF_FWSTRUCT_t_StatsBuffer
typedef struct t_StatsBuffer t_StatsBuffer;
#endif
struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[256];
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
LLint size;
LLint sizetot;
int offset;
//
int back;
//
int actived; // pour disabled
};
#include "htsstats.h"
#ifndef HTS_DEF_FWSTRUCT_t_InpInfo
#define HTS_DEF_FWSTRUCT_t_InpInfo

View File

@@ -122,6 +122,7 @@
<ClCompile Include="htshelp.c" />
<ClCompile Include="htsindex.c" />
<ClCompile Include="htslib.c" />
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="htsmodules.c" />

View File

@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
PT_Element elt = PT_ElementNew();
elt->statuscode = 405;
strcpy(elt->msg, "Method Not Allowed");
strcpybuff(elt->msg, "Method Not Allowed");
return elt;
}
@@ -820,9 +820,9 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
elt->size = StringLength(response);
elt->adr = StringAcquire(&response);
elt->statuscode = 207; /* Multi-Status */
strcpy(elt->charset, "utf-8");
strcpy(elt->contenttype, "text/xml");
strcpy(elt->msg, "Multi-Status");
strcpybuff(elt->charset, "utf-8");
strcpybuff(elt->contenttype, "text/xml");
strcpybuff(elt->msg, "Multi-Status");
StringFree(response);
fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
@@ -877,9 +877,9 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
elt->size = StringLength(html);
elt->adr = StringAcquire(&html);
elt->statuscode = HTTP_OK;
strcpy(elt->charset, "iso-8859-1");
strcpy(elt->contenttype, "text/html");
strcpy(elt->msg, "OK");
strcpybuff(elt->charset, "iso-8859-1");
strcpybuff(elt->contenttype, "text/html");
strcpybuff(elt->msg, "OK");
StringFree(html);
return elt;
}

View File

@@ -368,7 +368,11 @@ HTS_UNUSED static struct tm PT_GetTime(time_t t) {
if (tm != NULL)
return *tm;
else {
/* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
day of "00"; the epoch is the conventional "date unknown" */
memset(&tmbuf, 0, sizeof(tmbuf));
tmbuf.tm_year = 70;
tmbuf.tm_mday = 1;
return tmbuf;
}
}

View File

@@ -405,9 +405,9 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
elt->size = StringLength(html);
elt->adr = StringAcquire(&html);
elt->statuscode = HTTP_OK;
strcpy(elt->charset, "iso-8859-1");
strcpy(elt->contenttype, "text/html");
strcpy(elt->msg, "OK");
strcpybuff(elt->charset, "iso-8859-1");
strcpybuff(elt->contenttype, "text/html");
strcpybuff(elt->msg, "OK");
StringFree(html);
return elt;
}
@@ -579,9 +579,10 @@ PT_Index PT_LoadCache(const char *filename) {
if (chain != NULL) {
const char *scheme = link_has_authority(chain->name) ? "" : "http://";
snprintf(index->slots.common.startUrl,
sizeof(index->slots.common.startUrl), "%s%s", scheme,
(const char *) chain->name);
/* dropped rather than truncated: empty already reads as "unset" */
if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
(const char *) chain->name))
index->slots.common.startUrl[0] = '\0';
}
}
}
@@ -827,6 +828,10 @@ PT_Element PT_ElementNew(void) {
return r;
}
/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
#define PT_Element_failf(R, ...) \
slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
if (index != NULL && SAFE_INDEX(index)) {
return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
@@ -865,12 +870,15 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
(headersSize) += (int) strlen(headers + headersSize); \
} while(0)
#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strcpy(refvalue, value); \
line[0] = '\0'; \
} \
} while(0)
/* refvalue_size is mandatory: the cache line is bounded only by the line
buffer, not by the destination. */
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(void) strclipbuff(refvalue, refvalue_size, value); \
line[0] = '\0'; \
} \
} while (0)
#define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
int intval = 0; \
@@ -972,9 +980,12 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
const char *scheme =
link_has_authority(filenameIndex) ? "" : "http://";
firstSeen = 1;
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
scheme, filenameIndex);
/* dropped rather than truncated; try the next entry */
if (sprintfbuff(index->startUrl, "%s%s", scheme,
filenameIndex))
firstSeen = 1;
else
index->startUrl[0] = '\0';
}
}
} else {
@@ -1021,7 +1032,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
previous_save[0] = previous_save_[0] = '\0';
memset(r, 0, sizeof(_PT_Element));
r->location = location_default;
strcpy(r->location, "");
r->location[0] = '\0';
if (strncmp(url, "http://", 7) == 0)
url += 7;
hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -1070,16 +1081,23 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
value++;
ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg); // msg
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
sizeof(r->msg));
ZIP_READFIELD_INT(line, value, "X-Size", r->size); // size
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset); // contenttype
ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
ZIP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
ZIP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
sizeof(r->contenttype));
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
sizeof(r->charset));
ZIP_READFIELD_STRING(line, value, "Last-Modified",
r->lastmodified, sizeof(r->lastmodified));
ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
sizeof(r->etag));
ZIP_READFIELD_STRING(line, value, "Location", r->location,
sizeof(location_default));
ZIP_READFIELD_STRING(line, value, "Content-Disposition",
r->cdispo); // Content-disposition
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_); // Original save filename
r->cdispo, sizeof(r->cdispo));
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
sizeof(previous_save_));
if (line[0] != '\0') {
int len = r->headers ? ((int) strlen(r->headers)) : 0;
int nlen =
@@ -1099,7 +1117,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
int pathLen = (int) strlen(index->path);
if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) { // old (<3.40) buggy format
strcpy(previous_save, previous_save_);
strcpybuff(previous_save, previous_save_);
}
// relative ? (hack)
else if (index->safeCache || (previous_save_[0] != '/' // /home/foo/bar.gif
@@ -1135,8 +1153,9 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
snprintf(previous_save, sizeof(previous_save), "%s%s",
index->path, previous_save_ + index->fixedPath);
} else {
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
PT_Element_failf(
r, "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
r->statuscode = STATUSCODE_INVALID;
}
} else {
@@ -1155,7 +1174,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
// Peut-on stocker le fichier directement sur disque?
if (ok) {
if (r->msg[0] == '\0') {
strcpy(r->msg, "Cache Read Error : Unexpected error");
strcpybuff(r->msg, "Cache Read Error : Unexpected error");
}
} else { // lire en mémoire
@@ -1173,24 +1192,27 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
int last_errno = errno;
r->statuscode = STATUSCODE_INVALID;
sprintf(r->msg, "Read error in cache disk data: %s",
strerror(last_errno));
PT_Element_failf(r,
"Read error in cache disk data: %s",
strerror(last_errno));
}
r->adr[r->size] = '\0';
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg,
"Read error (memory exhausted) from cache");
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
}
fclose(fp);
} else {
r->statuscode = STATUSCODE_INVALID;
snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
file_convert(catbuff, sizeof(catbuff), previous_save));
PT_Element_failf(
r, "Read error (can't open '%s') from cache",
file_convert(catbuff, sizeof(catbuff),
previous_save));
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cached file name is invalid");
strcpybuff(r->msg, "Cached file name is invalid");
}
}
} else {
@@ -1202,12 +1224,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
free(r->adr);
r->adr = NULL;
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Data");
strcpybuff(r->msg, "Cache Read Error : Read Data");
} else
*(r->adr + r->size) = '\0';
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Memory Error");
strcpybuff(r->msg, "Cache Memory Error");
}
}
}
@@ -1215,21 +1237,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
} // si save==null, ne rien charger (juste en tête)
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header Data");
strcpybuff(r->msg, "Cache Read Error : Read Header Data");
}
unzCloseCurrentFile(index->zFile);
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Open File");
strcpybuff(r->msg, "Cache Read Error : Open File");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Bad Offset");
strcpybuff(r->msg, "Cache Read Error : Bad Offset");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "File Cache Entry Not Found");
strcpybuff(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -1522,7 +1544,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
cache->version = (int) (firstline[8] - '0'); // cache 1.x
if (cache->version <= 5) {
a += cache_brstr(a, firstline, sizeof(firstline));
strcpy(cache->lastmodified, firstline);
strcpybuff(cache->lastmodified, firstline);
} else {
fclose(cache->dat);
cache->dat = NULL;
@@ -1539,7 +1561,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
} else { // Vieille version du cache
/* */
cache->version = 0; // cache 1.0
strcpy(cache->lastmodified, firstline);
strcpybuff(cache->lastmodified, firstline);
}
/* Create hash table for the cache (MUCH FASTER!) */
@@ -1555,7 +1577,14 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
a++;
/* read "host/file" */
a += binput(a, line, HTS_URLMAXSIZE);
a += binput(a, line + strlen(line), HTS_URLMAXSIZE);
{
/* binput writes its NUL at s[max], so the second field must be
bounded by what is left of line[], not by the same constant
*/
const size_t used = strlen(line);
a += binput(a, line + used, (int) (sizeof(line) - used - 1));
}
/* read position */
a += binput(a, linepos, 200);
sscanf(linepos, "%d", &pos);
@@ -1570,9 +1599,11 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
const char *scheme =
link_has_authority(line) ? "" : "http://";
firstSeen = 1;
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
scheme, line);
/* dropped rather than truncated; try the next entry */
if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
firstSeen = 1;
else
index->startUrl[0] = '\0';
}
}
@@ -1650,7 +1681,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
previous_save[0] = previous_save_[0] = '\0';
memset(r, 0, sizeof(_PT_Element));
r->location = location_default;
strcpy(r->location, "");
r->location[0] = '\0';
if (strncmp(url, "http://", 7) == 0)
url += 7;
hash_pos_return = coucal_read(cache->hash, url, &hash_pos);
@@ -1757,7 +1788,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
int pathLen = (int) strlen(index->path);
if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) { // old (<3.40) buggy format
strcpy(previous_save, previous_save_);
strcpybuff(previous_save, previous_save_);
}
// relative ? (hack)
else if (index->safeCache || (previous_save_[0] != '/' // /home/foo/bar.gif
@@ -1793,8 +1824,9 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
snprintf(previous_save, sizeof(previous_save), "%s%s",
index->path, previous_save_ + index->fixedPath);
} else {
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
PT_Element_failf(r,
"Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
r->statuscode = STATUSCODE_INVALID;
}
} else {
@@ -1820,17 +1852,18 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
if (r->adr != NULL) {
if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Read error in cache disk data");
strcpybuff(r->msg, "Read error in cache disk data");
}
r->adr[r->size] = '\0';
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Read error (memory exhausted) from cache");
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
}
fclose(fp);
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Previous cache file not found (2)");
strcpybuff(r->msg, "Previous cache file not found (2)");
}
}
} else {
@@ -1842,30 +1875,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
free(r->adr);
r->adr = NULL;
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Data");
strcpybuff(r->msg, "Cache Read Error : Read Data");
} else
r->adr[r->size] = '\0';
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Memory Error");
strcpybuff(r->msg, "Cache Memory Error");
}
}
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Bad Data");
strcpybuff(r->msg, "Cache Read Error : Bad Data");
}
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header");
strcpybuff(r->msg, "Cache Read Error : Read Header");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Seek Failed");
strcpybuff(r->msg, "Cache Read Error : Seek Failed");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "File Cache Entry Not Found");
strcpybuff(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -2128,12 +2161,14 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
return 0;
}
#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strcpy(refvalue, value); \
line[0] = '\0'; \
} \
} while(0)
/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(void) strclipbuff(refvalue, refvalue_size, value); \
line[0] = '\0'; \
} \
} while (0)
#define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
int intval = 0; \
@@ -2169,7 +2204,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
location_default[0] = '\0';
memset(r, 0, sizeof(_PT_Element));
r->location = location_default;
strcpy(r->location, "");
r->location[0] = '\0';
if (strncmp(url, "http://", 7) == 0)
url += 7;
hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -2202,11 +2237,16 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
*value = '\0';
for(value++; *value == ' ' || *value == '\t'; value++) ;
HTTP_READFIELD_INT(line, value, "Content-Length", r->size); // size
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
HTTP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
HTTP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo); // Content-disposition
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
sizeof(r->contenttype));
HTTP_READFIELD_STRING(line, value, "Last-Modified",
r->lastmodified, sizeof(r->lastmodified));
HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
sizeof(r->etag));
HTTP_READFIELD_STRING(line, value, "Location", r->location,
sizeof(location_default));
HTTP_READFIELD_STRING(line, value, "Content-Disposition",
r->cdispo, sizeof(r->cdispo));
if (line[0] != '\0') {
int len = r->headers ? ((int) strlen(r->headers)) : 0;
int nlen =
@@ -2246,7 +2286,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
fetchSize = dataLength - metaSize;
} else if (fetchSize > dataLength - metaSize) {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Truncated Data");
strcpybuff(r->msg, "Cache Read Error : Truncated Data");
}
r->size = 0;
if (r->statuscode != STATUSCODE_INVALID) {
@@ -2259,12 +2299,13 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
int last_errno = errno;
r->statuscode = STATUSCODE_INVALID;
sprintf(r->msg, "Read error in cache disk data: %s",
strerror(last_errno));
PT_Element_failf(r, "Read error in cache disk data: %s",
strerror(last_errno));
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Read error (memory exhausted) from cache");
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
}
}
}
@@ -2272,21 +2313,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header Error");
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Read Header Error");
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "Cache Read Error : Seek Error");
strcpybuff(r->msg, "Cache Read Error : Seek Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpy(r->msg, "File Cache Entry Not Found");
strcpybuff(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -2335,8 +2376,18 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
FILE *const fp = st->fp;
struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
struct tm unknown_date;
int size_headers;
/* a cached entry with no parseable Last-Modified must not take the writer
down; the epoch is the conventional "date unknown" */
if (tm == NULL) {
memset(&unknown_date, 0, sizeof(unknown_date));
unknown_date.tm_year = 70;
unknown_date.tm_mday = 1;
tm = &unknown_date;
}
sprintf(st->headers,
"HTTP/1.0 %d %s"
"\r\n"
@@ -2372,7 +2423,7 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
if (element->adr != NULL) {
domd5mem(element->adr, element->size, st->md5, 1);
} else {
strcpy(st->md5, "-");
strcpybuff(st->md5, "-");
}
fprintf(fp,
/* nl */

View File

@@ -97,6 +97,7 @@
<ItemGroup>
<ClCompile Include="htsserver.c" />
<ClCompile Include="htsweb.c" />
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
</ItemGroup>

View File

@@ -0,0 +1,8 @@
#!/bin/bash
#
set -euo pipefail
# webhttrack posts its command line as one string: the argv split must grow past
# 1024 arguments and keep a quote inside a value out of the option parser.
httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

View File

@@ -146,11 +146,13 @@ sid=$(scrape_sid "${port}")
resp=$(post_redirect '/foo
X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
stop
printf '%s' "${resp}" | grep -qi 'X-Injected' &&
# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
# pipefail then turns the hit into a silent pass.
grep -qi 'X-Injected' <<<"${resp}" &&
fail "CRLF in the redirect value reached the response"
test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
fail "CRLF redirect still emitted a Location header"
test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
fail "CRLF redirect did not yield exactly one status line"
# Loopback unless asked otherwise. Assert the socket, not the announcement.

View File

@@ -103,9 +103,10 @@ sid=$(scrape_sid "${port}")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
# Accept: the legitimate flow still works. "redirect" is the cheapest field
# with a reply that is visible in the headers.
# with a reply that is visible in the headers. Matched from a here-string, not a
# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
grep -q '^Location: /accepted' <<<"${resp}" ||
fail "a body carrying the correct sid was refused"
# Refuse: missing and wrong. A missing one is the regression under test — the
@@ -114,12 +115,12 @@ printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
for bad in "redirect=/nosid" "sid=&redirect=/empty" \
"sid=00000000000000000000000000000000&redirect=/wrong"; do
resp=$(request "${port}" "${bad}")
printf '%s' "${resp}" | grep -q '^Location:' &&
grep -q '^Location:' <<<"${resp}" &&
fail "body accepted without a valid sid: ${bad}"
# A refusal has to be a well-formed reply, not a headerless fragment: the
# release build used to emit only Content-length, which reads as a protocol
# error to any client and hides the reason.
printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
fail "refusal was not a 403: ${bad}"
done
@@ -127,16 +128,23 @@ done
# applied to one global key store that later requests render from, and the
# command dispatcher reads it from there. Probe the store itself — step3
# interpolates ${projname} into its title — rather than the refused reply.
title() { request "$1" "" step3; }
store_page() {
# a dead probe or a non-page reply must not read as "the store was not written"
page=$(request "$1" "" step3) || fail "the key store probe failed"
grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
}
request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
title "${port}" | grep -q 'UNAUTHWRITE' &&
store_page "${port}"
grep -q 'UNAUTHWRITE' <<<"${page}" &&
fail "a body without a valid sid was written to the key store"
# Paired accept case: without it the assertion above passes even if the server
# simply ignores every body, which would prove nothing.
request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
title "${port}" | grep -q 'AUTHWRITE' ||
store_page "${port}"
grep -q 'AUTHWRITE' <<<"${page}" ||
fail "a body carrying the correct sid was not written to the key store"
echo "PASS"

View File

@@ -0,0 +1,142 @@
#!/bin/bash
#
# The wizard's size fields must render as distinct caps: sizemax is -M, othermax
# then maxhtml share -m.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_maxsize.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
srv=
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
# ~/.httrack.ini out of it.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
# Audit what step4.html renders back; without command_do nothing is launched.
"${python}" - "${url}" <<'PY' || fail "wizard rendered the wrong options (see above)"
import re, sys, urllib.parse, urllib.request
url = sys.argv[1]
html, other, site = "111111", "222222", "333333"
def post(fields):
# The body is refused without the session id the server puts in each form.
form = urllib.request.urlopen(url + "server/index.html", timeout=20).read()
m = re.search(rb'name="sid" value="([0-9a-f]+)"', form)
if not m:
raise SystemExit("no session id in server/index.html")
fields = [("sid", m.group(1).decode())] + fields
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in fields)
req = urllib.request.Request(url + "server/step4.html",
data=body.encode("latin-1"), method="POST")
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
def textarea(page, name):
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
if m is None:
sys.exit("no %s textarea in the rendered step4.html" % name)
return m.group(1)
def want(ok, msg, ctx):
if not ok:
sys.exit("%s\nrendered:%s" % (msg, ctx))
page = post([("maxhtml", html), ("othermax", other), ("sizemax", site),
("dos", "2")])
cmd = textarea(page, "command")
want("--max-size=" + site in cmd, "site cap not emitted as --max-size", cmd)
want("--max-files=" + site not in cmd, "site cap still an --max-files", cmd)
# Positive controls: both per-file caps still ride -m, one option each.
want("--max-files=" + other in cmd, "non-HTML cap not emitted as --max-files",
cmd)
want("--max-files=," + html in cmd, "HTML cap not emitted as --max-files=,",
cmd)
want(cmd.count("--max-files=") == 2, "unexpected --max-files count", cmd)
want(cmd.index("--max-files=" + other) < cmd.index("--max-files=," + html),
"the bare --max-files must precede the --max-files=, form", cmd)
# winprofile.ini feeds the same three caps to the Windows GUI, one key each.
ini = textarea(page, "winprofile").replace("\r\n", "\n") # the ini is CRLF
want("\nDos=2" in ini, "Dos not written from the dos field", ini)
want("\nMaxHtml=" + html + "\n" in ini, "MaxHtml not written from maxhtml", ini)
want("\nMaxOther=" + other + "\n" in ini, "MaxOther not written from othermax",
ini)
want("\nMaxAll=" + site + "\n" in ini, "MaxAll not written from sizemax", ini)
# A bare --max-files= (an unguarded empty field) parses as -m with no digits,
# silently clearing the html cap.
page = post([("maxhtml", ""), ("othermax", ""), ("sizemax", ""), ("dos", "2")])
cmd = textarea(page, "command")
want("--max-rate=" in cmd, "empty size fields rendered no command line", cmd)
want("--max-files" not in cmd and "--max-size" not in cmd,
"empty size fields still rendered a size option", cmd)
# A mistyped ${LANG_OK] key renders the OK button's label empty.
opts = urllib.request.urlopen(url + "server/option2b.html",
timeout=20).read().decode("latin-1")
want("LANG_OK" not in opts, "unexpanded LANG_OK in option2b.html", "")
want('<input type="submit" value="OK"' in opts, "no OK label in option2b.html",
"")
PY
cleanup
# A bare -m<n> resets the html limit, so only the bare-then-comma order keeps
# both caps live. basic.html is 487 bytes, well past the 10-byte html cap.
crawl() {
bash "${distdir}/tests/local-crawl.sh" "$@"
}
crawl --errors 1 --log-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=,10'
crawl --errors 1 --log-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=500000' '--max-files=,10'
crawl --errors 0 --found 'simple/basic.html' --log-not-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=,10' '--max-files=500000'
echo "PASS"

View File

@@ -0,0 +1,135 @@
#!/bin/bash
#
# A browser will not follow an http: page to a file: URL, so the GUI's "browse
# the mirror" link has to go through the server's own /website/ route.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
# Control: on a path that does not resolve, grep "passes" without reading anything.
test -f "${distdir}/html/server/finished.html" || fail "no GUI pages under ${distdir}"
bad=$(grep -l 'file://' "${distdir}"/html/server/finished.html || true)
test -z "${bad}" || fail "file: link left in: ${bad}"
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_browse.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
srvpid=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# Stand in for a finished mirror: the crawl itself is not under test.
proj="${work}/websites/proj"
mkdir -p "${proj}"
printf 'MIRROR-PROBE-OK\n' >"${proj}/probe.txt"
printf '<html><body>MIRROR-INDEX-OK</body></html>\n' >"${proj}/index.html"
# An isolated HOME keeps a stray ~/.httrack.ini out of the server's settings.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
# htsserver resolves the posted paths itself, so hand it native ones.
"${python}" - "${url}" "$(nativepath "${work}/websites")" <<'PY' || fail "browse-link checks failed"
import re, sys, urllib.error, urllib.parse, urllib.request
url, base = sys.argv[1].rstrip("/"), sys.argv[2]
rc = 0
def check(ok, what):
global rc
print(("ok: " if ok else "FAIL: ") + what)
if not ok:
rc = 1
def get(path):
# The UI is served ISO-8859-1, so stay in bytes.
return urllib.request.urlopen(url + path, timeout=20).read()
# No crawl has run, so no commandRunning/commandEnd override swaps the page out.
finished = get("/server/finished.html").decode("latin-1")
# Control: an empty or unrendered page would pass "no file:" on its own.
check("HTTrack Website Copier" in finished, "finished.html rendered")
check("file://" not in finished, "finished.html has no file: link")
# Control: /website/ 404s until a project is set, so the fetches below are real.
try:
get("/website/probe.txt")
check(False, "/website/ served with no project set")
except urllib.error.HTTPError as e:
check(e.code == 404, "/website/ is bound to a project (got %d)" % e.code)
# Point the server at the mirror the way step4.html does; a body without the
# session id is refused outright. Only a profile save records the served root,
# so post one; command_do=save runs no crawl.
sid = re.search(r'name="sid" value="([0-9a-f]+)"', finished)
if sid is None:
print("FAIL: no sid in the rendered form")
sys.exit(1)
fields = [("sid", sid.group(1)), ("path", base), ("projname", "proj"),
("command", "httrack"), ("command_do", "save"), ("winprofile", "x")]
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="")) for k, v in fields)
urllib.request.urlopen(urllib.request.Request(
url + "/server/step4.html", data=body.encode("latin-1"), method="POST"),
timeout=20).read()
check(get("/website/probe.txt") == b"MIRROR-PROBE-OK\n", "mirror file over http")
check("MIRROR-INDEX-OK" in get("/website/index.html").decode("latin-1"),
"mirror index over http")
body = get("/server/finished.html").decode("latin-1")
# Match the anchor by its mirror-path label: the list below it links /website/
# too, so a bare "is the route mentioned" check passes on the unfixed page.
check(re.search(r'href="/website/index\.html"[^>]*>\s*' +
re.escape(base + "/proj"), body) is not None,
"the mirror-path link points at the served mirror")
check("file://" not in body, "finished.html has no file: link")
sys.exit(rc)
PY
# A leaked htsserver wedges the parallel harness behind a green log.
cleanup
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
echo "PASS"

View File

@@ -0,0 +1,148 @@
#!/bin/bash
#
# The wizard renders its httrack command line as one string, which the engine
# splits back into argv: a double quote in a field must reach the split escaped,
# or the rest of the value is parsed as fresh options (-V runs a shell command).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
# default index). Prints the reply.
request() {
python3 -c 'import socket, sys
port, body = int(sys.argv[1]), sys.argv[2]
page = sys.argv[3] if len(sys.argv) > 3 else "index"
if body:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
else:
req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
}
scrape_sid() {
request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
}
# urlencode the key=value pairs given as arguments
formencode() {
python3 -c 'import sys, urllib.parse
print(urllib.parse.urlencode([tuple(a.split("=", 1)) for a in sys.argv[1:]]))' "$@"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
sid=$(scrape_sid "${port}")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
# Fill the wizard fields the command line quotes, then read back the generated
# command line: the user-agent carries a break-out attempt, the footer a
# backslash (which must survive the escape round trip), the project name a plain
# value.
body=$(formencode "sid=${sid}" 'user=Moz" -V "touch /tmp/pwn' 'footer=a\b"c' \
"path=/tmp/p" "projname=plain proj" 'urls=http://x/a"b' 'url2=+*.png"')
request "${port}" "${body}" >/dev/null
cmdline=$(request "${port}" "" step4 |
sed -n '/<textarea name="command"/,/<\/textarea>/p')
# Control: without the fields in the page every assertion below is vacuous.
grep -q -- '--user-agent' <<<"${cmdline}" ||
fail "no --user-agent in the generated command line (probe blind)"
# A plain value is passed through untouched: the escaping must not mangle the
# ordinary case.
grep -qF -- '--path "/tmp/p/plain proj"' <<<"${cmdline}" ||
fail "a plain quoted value was not passed through: ${cmdline}"
# The quote is escaped, so the split keeps it inside the value...
grep -qF -- '--user-agent "Moz\" -V \"touch /tmp/pwn"' <<<"${cmdline}" ||
fail "the quote in the user-agent was not escaped: ${cmdline}"
# ...and the raw form, which would end the argument and hand -V to the option
# parser, is gone.
grep -qF -- '--user-agent "Moz" -V "touch' <<<"${cmdline}" &&
fail "the user-agent still closes its argument early: ${cmdline}"
# A backslash is escaped too, or the split would eat it along with the quote
# that follows.
grep -qF -- '--footer "a\\b\"c"' <<<"${cmdline}" ||
fail "the backslash in the footer was not escaped: ${cmdline}"
# The url and filter fields sit outside quotes, where a backslash cannot escape
# anything: one quote there flips the parity of every quote after it, so the
# escaping above would protect nothing. They must not emit a raw quote at all.
grep -qF -- 'http://x/a%22b' <<<"${cmdline}" ||
fail "the quote in the url field was not neutralised: ${cmdline}"
grep -qF -- '+*.png%22' <<<"${cmdline}" ||
fail "the quote in the filter field was not neutralised: ${cmdline}"
grep -qF -- 'http://x/a"b' <<<"${cmdline}" &&
fail "the url field still emits a raw quote: ${cmdline}"
echo "PASS"

View File

@@ -0,0 +1,167 @@
#!/bin/bash
#
# htsserver serves the crawled mirror under /website/ alongside its own GUI:
# mirrored pages must skip the ${...} expander, or ${_sid} leaks the session id.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
log=$(mktemp)
work=$(mktemp -d)
csrv=
# start() runs in a command substitution, so its $! never reaches this shell. A
# missed kill leaves an orphan holding the CI job open long after a green suite.
srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
cleanup() {
local pid
pid=$(srvpid)
test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
test -z "${csrv}" || kill -9 "${csrv}" 2>/dev/null || true
wait "${csrv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${log}" "${work}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
portof() { echo "${1##*:}" | tr -d /; }
# GET $2 from 127.0.0.1:$1, headers into $3 and the body, byte for byte, into $4.
fetch() {
python3 -c 'import socket, sys
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(20)
s.sendall(("GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % sys.argv[2]).encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
head, _, body = out.partition(b"\r\n\r\n")
open(sys.argv[3], "wb").write(head)
open(sys.argv[4], "wb").write(body)' "$1" "$2" "$3" "$4"
}
# POST the remaining args as urlencoded key=value fields to 127.0.0.1:$1.
post() {
local port=$1
shift
python3 -c 'import socket, sys, urllib.parse
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe=""))
for k, v in (a.split("=", 1) for a in sys.argv[2:])).encode()
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(30)
s.sendall(b"POST /step4.html HTTP/1.0\r\nHost: 127.0.0.1\r\n"
b"Content-type: application/x-www-form-urlencoded\r\n"
b"Content-length: %d\r\n\r\n" % len(body) + body)
while s.recv(65536):
pass
s.close()' "${port}" "$@" >/dev/null
}
# LF-only, and a line ending in a backslash: the expander rewrites both, so a
# mangled reply fails the byte comparison even where no directive is present.
mirror="${work}/proj/hostile.html"
hdr="${work}/hdr"
body="${work}/body"
# The server merges $HOME/.httrack.ini into the same store on the first request;
# point it somewhere empty so a developer's own file cannot shadow the fields.
export HOME="${work}"
url=$(start)
port=$(portof "${url}")
# Positive control: the GUI's own templates must still expand. This is also
# where the real token comes from, so its absence below can be asserted.
fetch "${port}" /server/index.html "${hdr}" "${body}"
sid=$(sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' "${body}" | head -1)
test "${#sid}" -eq 32 || fail "GUI page did not expand \${sid} (got '${sid}')"
# /website/ serves only the root the server itself recorded, so save a profile
# (no command_do=start, so nothing crawls) to create it, then plant the file.
post "${port}" "sid=${sid}" command=httrack command_do=save winprofile=x \
"path=${work}" projname=proj
test -f "${work}/proj/hts-cache/winprofile.ini" ||
fail "profile save did not create the project: $(cat "${log}")"
mkdir -p "$(dirname "${mirror}")"
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
printf 'Hostile mirrored page.\nsid=${_sid} copy=${sid}\ntrailing backslash: \\\n' \
>"${mirror}"
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
grep -q '^HTTP/1\.0 200 ' "${hdr}" || fail "mirrored page not served: $(head -1 "${hdr}")"
grep -qF "${sid}" "${body}" &&
fail "the session id was expanded into mirrored content"
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
grep -qF '${_sid}' "${body}" || fail "\${_sid} did not survive verbatim"
# shellcheck disable=SC2016
grep -qF '${sid}' "${body}" || fail "\${sid} did not survive verbatim"
cmp -s "${mirror}" "${body}" || fail "mirrored file not served byte for byte"
# The mirror stays browsable: verbatim must not mean served as a download.
grep -qi '^Content-type: text/html' "${hdr}" ||
fail "mirrored page lost its text/html type: $(cat "${hdr}")"
# The other direction: while a crawl runs, every .html request is overridden to
# the GUI's own refresh page, so a /website/ URL stops naming mirrored content.
# /trickle/ dribbles for a minute, which holds the crawl open for the probe.
clog="${work}/content.log"
python3 "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
csrv=$!
for _ in $(seq 1 40); do
cport=$(sed -n 's/^PORT //p' "${clog}") && test -n "${cport}" && break
kill -0 "${csrv}" 2>/dev/null || break
sleep 0.25
done
test -n "${cport:-}" || fail "content server did not come up: $(cat "${clog}")"
post "${port}" "sid=${sid}" "path=${work}" projname=crawl winprofile=x \
command_do=start \
"command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/trickle/ -O ${work}/crawl"
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
grep -q '^HTTP/1\.0 200 ' "${hdr}" ||
fail "running crawl: /website/ was not overridden to the GUI page: $(head -1 "${hdr}")"
grep -qF "'crawl' - HTTrack Website Copier" "${body}" ||
fail "the overridden GUI page was not the expanded refresh page"
echo "PASS"

View File

@@ -0,0 +1,183 @@
#!/bin/bash
#
# /website/ is served from the project directory htsserver set up itself, never
# from a root the request body names, and composing that path must stay bounded.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
base=$(mktemp -d)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
# The server reports its own pid; the aliveness assertion below hangs off it.
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
# $2 is empty. Prints the reply.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
get() { request "$1" "$2" ""; }
post() { request "$1" "" "$2"; }
# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
# reads as marker-absent, and so do a truncated body and a 302 to the file.
reply=
fetch() {
reply=$(get "$1" "$2") || fail "the request for $2 failed"
grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Every request body is gated by the session id (78_webhttrack-sid.test).
sid=$(get "${port}" /server/index.html |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
# A file the mirror must never expose, next to the project that may.
echo "SECRETMARKER" >"${base}/secret.txt"
mkdir -p "${base}/proj"
echo "LOGMARKER" >"${base}/proj/hts-log.txt"
# error_redirect is the branch that skipped the fsfile clearing, so fail the save
# with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
# any successful save: commandEnd then swaps the error page for the finished one.
get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
grep -q '^Location: /server/error.html' <<<"${saved}" ||
fail "the refused save did not redirect to the error page"
# No project yet, so no root to serve from.
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
fetch "${port}" /website/secret.txt 404
grep -q SECRETMARKER <<<"${reply}" &&
fail "a posted projpath served a file outside any project"
# Positive control: step4's "save settings" flow registers the project without
# crawling, and browsing its mirror keeps working. Without it the assertions
# around it would pass on a server that never serves /website/ at all.
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
post "${port}" "${body}" >/dev/null
test -f "${base}/proj/hts-cache/winprofile.ini" ||
fail "the project was not registered: $(cat "${log}")"
fetch "${port}" /website/hts-log.txt 200
grep -q LOGMARKER <<<"${reply}" ||
fail "the registered project's mirror is not served"
# Same request with the root repointed: the project is legitimate, projpath is
# not what decides where the bytes come from.
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
fetch "${port}" /website/secret.txt 404
grep -q SECRETMARKER <<<"${reply}" &&
fail "a posted projpath repointed the served root"
post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
fetch "${port}" /website/passwd 404
grep -q '^root:' <<<"${reply}" &&
fail "a posted projpath read an arbitrary system file"
# A ".." anywhere in the recorded root would escape the mirror on every later
# request, so the save must be refused and the previous root kept.
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
fetch "${port}" /website/secret.txt 404
grep -q SECRETMARKER <<<"${reply}" &&
fail "a '..' in the saved project path escaped the mirror root"
fetch "${port}" /website/hts-log.txt 200
grep -q LOGMARKER <<<"${reply}" ||
fail "rejecting the '..' root also lost the previous one"
# The root is now the server's own, but it is still built from two posted
# fields: 800-odd bytes of them used to be sprintf'd into a 1024-byte buffer.
seg=$(printf '%0200d' 0)
longpath="${base}/${seg}/${seg}/${seg}/${seg}"
fspath="${longpath}/proj"
# structcheck() refuses a root over HTS_URLMAXSIZE, so the URL carries the rest.
test "$((${#fspath} + 11))" -le 1024 ||
fail "the long project path (${#fspath}) would not pass structcheck"
longurl=$(printf '%0800d' 0)
test "$((${#fspath} + 1 + ${#longurl}))" -gt 1024 ||
fail "the composed path (${#fspath} + ${#longurl}) would not overflow"
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${longpath}&projname=proj" >/dev/null
test -f "${fspath}/hts-cache/winprofile.ini" ||
fail "the long-path project was not registered: $(cat "${log}")"
get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
# Not just alive: still answering.
fetch "${port}" /server/index.html 200
echo "PASS"

View File

@@ -0,0 +1,111 @@
#!/bin/bash
# A cached header longer than proxytrack's field must be clipped to that field,
# not overflow into the next one (contenttype[64] abuts the location pointer).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
pad() { printf "%${1}s" '' | tr ' ' "$2"; }
# Longest surviving run of char $2 in file $1, or 0.
runlen() {
grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
}
# --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
# Each header overshoots its own destination, so a per-field clip is the only
# way to get every expected length right at once.
{
printf 'HTTP/1.1 200 OK\r\n'
printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)"
printf 'Etag: %s\r\n' "$(pad 400 E)"
printf 'Content-Disposition: %s\r\n' "$(pad 400 D)"
printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
printf 'Content-Length: 5\r\n\r\n'
} >"$dir/hdr"
printf 'hello' >"$dir/body"
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
{
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
printf '2 0 test\n'
printf '\n\n'
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
cat "$dir/hdr" "$dir/body"
} >"$dir/in.arc"
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2
exit 1
}
test -s "$dir/out.arc" || {
echo "FAIL: ARC entry dropped instead of clipped" >&2
exit 1
}
# Only contenttype survives into the ARC output; the over-long Etag and
# Content-Disposition above are still load-bearing, since bounding just one
# field leaves the others smashing the element (the run above would crash).
got=$(runlen "$dir/out.arc" A)
test "${got:-0}" = 54 || {
echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2
exit 1
}
# --- ZIP reader (ZIP_READFIELD_STRING) --------------------------------------
python=$(find_python) || {
echo "python3 missing; ARC half only" >&2
exit 0
}
"$python" - "$dir/in.zip" <<'EOF'
import sys, zipfile
body = b"hello world"
meta = (
"X-In-Cache: 1\r\n"
"X-StatusCode: 200\r\n"
"X-StatusMessage: ok%s\r\n"
"X-Size: %d\r\n"
"Content-Type: text/html%s\r\n"
"X-Charset: %s\r\n"
"Etag: %s\r\n"
"Content-Disposition: %s\r\n"
# a parseable date is required: the ARC writer dereferences it unchecked
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
"X-Save: out/page.html\r\n"
% ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400)
).encode()
zi = zipfile.ZipInfo("example.com/page.html")
zi.compress_type = zipfile.ZIP_STORED
zi.extra = meta
with zipfile.ZipFile(sys.argv[1], "w") as z:
z.writestr(zi, body)
EOF
proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2
exit 1
}
test -s "$dir/out2.arc" || {
echo "FAIL: zip entry dropped instead of clipped" >&2
exit 1
}
# three destinations, two capacities: msg[1024] keeps all 400 M's where a
# one-size-fits-all clip would cut it to 63 like the others
while read -r ch want; do
got=$(runlen "$dir/out2.arc" "$ch")
test "${got:-0}" = "$want" || {
echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2
exit 1
}
done <<'EOF'
A 54
C 63
M 400
EOF

View File

@@ -0,0 +1,51 @@
#!/bin/bash
# A cached entry whose Last-Modified is missing or unparseable must still
# convert: the ARC writer used to dereference the parsed date unchecked.
set -euo pipefail
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
# $1 Last-Modified value (empty = omit the header), $2 expected archive date,
# $3 label. The date is asserted exactly: a guard that fires unconditionally
# would clobber the valid case, and one that skips the year/day would emit a
# month and day of 00.
run_case() {
lastmod=$1
wantdate=$2
what=$3
{
printf 'HTTP/1.1 200 OK\r\n'
printf 'Content-Type: text/html\r\n'
test -z "$lastmod" || printf 'Last-Modified: %s\r\n' "$lastmod"
printf 'Content-Length: 5\r\n\r\n'
} >"$dir/hdr"
printf 'hello' >"$dir/body"
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
{
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
printf '2 0 test\n'
printf '\n\n'
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
cat "$dir/hdr" "$dir/body"
} >"$dir/in.arc"
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed on $what" >&2
exit 1
}
grep -aq "^http://example.com/p.html 0.0.0.0 ${wantdate} " "$dir/out.arc" || {
echo "FAIL: $what: expected archive date ${wantdate}, got:" >&2
grep -a '^http://example.com/' "$dir/out.arc" >&2 || echo "(entry dropped)" >&2
exit 1
}
}
# the epoch stands in for "date unknown"
run_case '' 19700101000000 'a missing Last-Modified'
run_case 'not a date at all' 19700101000000 'an unparseable Last-Modified'
run_case '0' 19700101000000 'a bare 0 Last-Modified'
# a parseable date must still come through untouched
run_case 'Sun, 06 Nov 1994 08:49:37 GMT' 19941106084937 'a valid Last-Modified'

View File

@@ -0,0 +1,69 @@
#!/bin/bash
# A cache file whose mtime is beyond what gmtime can represent must not put a
# day of "00" in the ARC filedesc date: PT_GetTime fell back to an all-zero tm.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
rc=0
"$python" - "$dir/in.zip" <<'EOF' || rc=$?
import os, sys, zipfile
body = b"hello world"
meta = (
"X-In-Cache: 1\r\n"
"X-StatusCode: 200\r\n"
"X-StatusMessage: OK\r\n"
"X-Size: %d\r\n"
"Content-Type: text/html\r\n"
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
"X-Save: out/page.html\r\n" % len(body)
).encode()
zi = zipfile.ZipInfo("example.com/page.html")
zi.compress_type = zipfile.ZIP_STORED
zi.extra = meta
with zipfile.ZipFile(sys.argv[1], "w") as z:
z.writestr(zi, body)
# past gmtime's range; the index timestamp is this file's mtime
huge = 4611686018427387903
try:
os.utime(sys.argv[1], (huge, huge))
except OSError:
sys.exit(77)
if int(os.stat(sys.argv[1]).st_mtime) < huge:
sys.exit(77) # filesystem clamped it, nothing to test
EOF
test "$rc" -eq 0 || {
test "$rc" -eq 77 || {
echo "FAIL: could not build the cache (python exit $rc)" >&2
exit 1
}
echo "filesystem will not hold an out-of-range mtime; skipping" >&2
exit 77
}
proxytrack --convert "$dir/out.arc" "$dir/in.zip" >/dev/null 2>&1 || {
echo "FAIL: proxytrack failed on a cache with an out-of-range mtime" >&2
exit 1
}
# field 3 of the filedesc line is YYYYMMDDhhmmss; the epoch stands in for
# "unknown", and 14 digits with a real day is the point (00 is what broke)
date=$(head -n1 "$dir/out.arc" | awk '{ print $3 }')
test "$date" = 19700101000000 || {
echo "FAIL: filedesc date is '$date', expected 19700101000000" >&2
exit 1
}

View File

@@ -0,0 +1,174 @@
#!/bin/bash
#
# The "save settings" failure messages quote a project path the request body
# supplies, so composing them must clip instead of running off a fixed buffer.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
# ERROR_MESSAGE_MAX - 1 in htsserver.c.
msgmax=1023
srv=
log=$(mktemp)
# Physical path: macOS resolves TMPDIR through /var -> private/var, and the
# paths built below sit within a few bytes of its 1024-byte PATH_MAX.
base=$(cd "$(mktemp -d)" && pwd -P)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU XFSZ
# Caps regular-file writes, which is how the failing-write branch below
# is reached without /dev/full; the announce rides a pipe, exempt.
ulimit -f 8
exec htsserver "${distdir}/" --port "${port}" 2>&1
) | cat >"${log}" &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
# $2 is empty. Prints the reply.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
get() { request "$1" "$2" ""; }
post() { request "$1" "" "$2"; }
# Echo a path under the root $1 exactly $2 bytes long, in NAME_MAX-sized parts.
padpath() {
local root=$1 want=$2 p=$1 last
while test $((${#p} + 203)) -lt "${want}"; do
p="${p}/$(printf '%0200d' 0)"
done
last=$((want - ${#p} - 1))
test "${last}" -ge 1 || fail "cannot build a ${want}-byte path under ${root}"
printf '%s/%0*d' "${p}" "${last}" 0
}
# Fail unless the error page renders the message opening with $1 about project
# path $2, clipped to msgmax. error.html puts ${error} alone on its own line.
check_clipped() {
local prefix=$1 full=$((${#1} + ${#2})) page line shown
test "${full}" -gt "${msgmax}" ||
fail "a ${full}-byte message fits in ${msgmax}: it cannot show a clip"
page=$(get "${port}" /server/error.html | tr -d '\r')
line=$(printf '%s\n' "${page}" | grep "^${prefix}") || {
shown=$(printf '%s\n' "${page}" | grep '^Unable to ' | cut -c1-80 || true)
fail "the error page reports '${shown:-nothing}', want: ${prefix}"
}
test "${#line}" -eq "${msgmax}" ||
fail "the message rendered ${#line} bytes, want ${full} clipped to ${msgmax}"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Every request body is gated by the session id (78_webhttrack-sid.test).
sid=$(get "${port}" /server/index.html |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
save() {
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=$2&path=$1&projname=$3"
}
# A project path too long to be created at all.
path=$(padpath "${base}/nodir" 1098)
save "${path}" x p >/dev/null
alive "${srv}" ||
fail "an over-long project path crashed the server: $(cat "${log}")"
check_clipped "Unable to create the directory structure in " "${path}/p"
# A creatable project path whose hts-cache is not a directory, so the init file
# under it cannot be opened.
path=$(padpath "${base}/isdir" 993)
mkdir -p "${path}/p"
ln -s /dev/null "${path}/p/hts-cache"
save "${path}" x p >/dev/null
alive "${srv}" || fail "an unwritable init file crashed the server: $(cat "${log}")"
check_clipped "Unable to create the init file " "${path}/p"
# The init file opens, then the write fails: the profile is past both stdio's
# buffer, so fwrite() reaches the descriptor, and the server's file-size limit.
path=$(padpath "${base}/full" 993)
mkdir -p "${path}/p/hts-cache"
profile=$(printf '%01024d' 0)
profile=${profile}${profile}${profile}${profile}
profile=${profile}${profile}${profile}${profile}
test "${#profile}" -eq 16384 || fail "built a ${#profile}-byte profile, want 16384"
save "${path}" "${profile}" p >/dev/null
alive "${srv}" || fail "a short init file write crashed the server: $(cat "${log}")"
written=$(wc -c <"${path}/p/hts-cache/winprofile.ini" | tr -d ' ')
test "${written}" -lt "${#profile}" ||
fail "the file-size limit did not stop the write (${written} bytes landed)"
check_clipped "Unable to write ${#profile} bytes in the the init file " "${path}/p"
get "${port}" /server/index.html | grep -q '200 OK' ||
fail "the server stopped answering after the refused saves"
echo "PASS"

View File

@@ -0,0 +1,195 @@
#!/bin/bash
#
# An unchecked box posts nothing and the stored "1" survives: hence a hidden
# companion per box, plus a disabling flag for the default-on ones.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
srvpid=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
import glob, os, re, sys, urllib.parse, urllib.request
url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
rc = 0
def check(ok, what):
global rc
print(("ok: " if ok else "FAIL: ") + what)
if not ok:
rc = 1
# cache/cache2 are left to the separate rework of the dead "Cache" seed.
skip = {"cache", "cache2"}
page_of = {}
boxes = 0
for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
page = open(path, "rb").read().decode("latin-1")
cleared = dict((m.group(1), m.start()) for m in
re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
boxes += 1
if m.group(1) in skip:
continue
page_of[m.group(1)] = os.path.basename(path)
at = cleared.get(m.group(1))
check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
% (os.path.basename(path), m.group(1)))
# Control: a regex that stopped matching would leave nothing to assert on.
check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
sid = None
def get(path):
# The UI is served ISO-8859-1, so decode, do not assume UTF-8.
return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
def post(fields):
global sid
if sid is None:
m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
if m is None:
sys.exit("no session id in server/index.html")
sid = m.group(1)
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
[("sid", sid)] + fields)
req = urllib.request.Request(url + "/server/step4.html",
data=body.encode("latin-1"), method="POST")
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
def textarea(page, name):
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
if m is None:
sys.exit("no %s textarea in the rendered step4.html" % name)
return m.group(1)
def checked(page, name):
return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
# What each state puts on the command line (None: nothing) and its profile key.
# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
BOXES = [
# field profile key set cleared
("parseall", "ParseAll", "--near", None),
("link", "Near", "--test", None),
("testall", "Test", "--extended-parsing", None),
# htmlfirst emits --priority=7, which the scan-priority list also emits.
("htmlfirst", "HTMLFirst", None, None),
("errpage", "NoErrorPages", "--generate-errors=0", None),
("external", "NoExternalPages", "--replace-external", None),
("hidepwd", "NoPwdInPages", "--disable-passwords", None),
("hidequery", "NoQueryStrings", "--include-query-string=0", None),
("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
("windebug", None, "--debug-headers", None),
("ka", "KeepAlive", "--keep-alive", None),
("remt", "RemoveTimeout", "--host-control=1", None),
("rems", "RemoveRateout", "--host-control=2", None),
("cookies", "Cookies", None, "--cookies=0"),
("parsejava", "ParseJava", None, "--parse-java=0"),
("updhack", "UpdateHack", "--updatehack", None),
("urlhack", "URLHack", "--urlhack", None),
("keepwww", "KeepWww", "--keep-www-prefix", None),
("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
("toler", "TolerantRequests", "--tolerant", None),
("http10", "HTTP10", "--http-10", None),
("warc", "Warc", "--warc", None),
("norecatch", "NoRecatch", "--do-not-recatch", None),
("logf", "Log", "--single-log", None),
("index", "Index", None, None),
("index2", "WordIndex", "--search-index", None),
("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
]
for field, key, when_set, when_clear in BOXES:
for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
("", when_clear, when_set, "0")):
rendered = post([(field, value)])
cmd = textarea(rendered, "command").split()
label = "%s %s" % (field, "set" if value else "cleared")
if want:
check(want in cmd, "%s emits %s" % (label, want))
if unwanted:
check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
if key:
# The Windows GUI reads the same option out of the profile.
check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
"%s writes %s=%s" % (label, key, stored))
check(checked(page_of[field], field) == bool(value),
"%s draws %s box" % (label, "a ticked" if value else "an empty"))
missing = sorted(set(page_of) - set(b[0] for b in BOXES))
check(not missing, "every checkbox is exercised (missing %s)" % missing)
# A browser posts the companion and the ticked box under one name, so the fix
# rests on the body loop keeping the last value; both orders pin that down.
cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
sys.exit(rc)
PY
# A leaked htsserver wedges the parallel harness behind a green log.
cleanup
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
echo "PASS"

View File

@@ -0,0 +1,181 @@
#!/bin/bash
#
# fopen() succeeds on a directory on POSIX and reading one never reaches EOF; a
# FIFO blocks in fopen() instead. Either used to wedge the single-threaded server.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
base=$(mktemp -d)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# First line only. A "| head -1" would close the pipe early and, under pipefail,
# SIGPIPE the producer into a spurious failure.
firstline() { echo "${1%%$'\n'*}"; }
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
# The server reports its own pid; the aliveness assertion below hangs off it.
srvpid() { firstline "$(sed -n 's/^PID=//p' "${log}")"; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# GET the path $2 from 127.0.0.1:$1, or POST the body $3 to / when $2 is empty.
# The socket timeout is what makes an unfixed tree fail rather than wedge CI.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(15)
s.sendall(req.encode())
out = b""
try:
while True:
b = s.recv(65536)
if not b:
break
out += b
except socket.timeout:
sys.stderr.write("timed out after %d bytes\n" % len(out))
sys.exit(9)
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
# GET $2, or fail with $3 when the reply never comes.
get() {
local out
out=$(request "$1" "$2" "") || fail "$2 did not answer: $3"
echo "${out}"
}
post() { request "$1" "" "$2"; }
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Needs no session id and no project: html/server is a directory of the install.
reply=$(get "${port}" /server/ "an unauthenticated request wedged the server")
case "${reply}" in
"HTTP/1.0 404 "*) ;;
*) fail "a GUI directory did not answer 404: $(firstline "${reply}")" ;;
esac
# Every request body is gated by the session id.
reply=$(get "${port}" /server/index.html "the GUI is not served")
sid=$(firstline "$(echo "${reply}" |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p')")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
mkdir -p "${base}/proj/sub"
echo "PROBEMARKER" >"${base}/proj/probe.txt"
# mkfifo and ln -s may not work on a Windows checkout, so both stay optional.
refuse=(/website/ /website/sub/)
if mkfifo "${base}/proj/fifo.txt" 2>/dev/null; then
refuse+=(/website/fifo.txt)
fi
ln -s probe.txt "${base}/proj/link.txt" 2>/dev/null || true
# step4's "save settings" flow registers the project without crawling, and that
# is what arms the /website/ root.
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
body="${body}&path=${base}&projname=proj"
post "${port}" "${body}" >/dev/null || fail "the save request did not answer"
test -f "${base}/proj/hts-cache/winprofile.ini" ||
fail "the project was not registered: $(cat "${log}")"
# Positive control: every assertion below would pass on a server serving nothing.
served() {
case "$(get "$1" "$2" "$3")" in
*PROBEMARKER*) ;;
*) fail "$3" ;;
esac
}
served "${port}" /website/probe.txt "the registered project's mirror is not served"
if test -L "${base}/proj/link.txt"; then
# The guard stats rather than lstats, so a symlinked mirror file still serves.
served "${port}" /website/link.txt "a symlink to a mirror file is not served"
fi
# /website/ is where the GUI's own "browse mirrored site" link points.
for path in "${refuse[@]}"; do
reply=$(get "${port}" "${path}" "the server wedged on ${path}")
case "${reply}" in
"HTTP/1.0 404 "*) ;;
*) fail "${path} did not answer 404: $(firstline "${reply}")" ;;
esac
done
# The accept loop is single-threaded: one spin denies every later request.
alive "${srv}" || fail "the server died: $(cat "${log}")"
served "${port}" /website/probe.txt "the server stopped answering after a directory request"
# This fopen() is reached before any guard, so its read loop must stop on ferror().
mkdir -p "${base}/proj2/hts-cache/winprofile.ini"
reply=$(post "${port}" "sid=${sid}&path=${base}&loadprojname=proj2") ||
fail "a directory winprofile.ini wedged the server"
case "${reply}" in
"HTTP/1.0 3"*) ;;
*) fail "loading a project did not redirect: $(firstline "${reply}")" ;;
esac
alive "${srv}" || fail "the server died: $(cat "${log}")"
served "${port}" /website/probe.txt "the server stopped answering after a project load"
echo "PASS"

View File

@@ -0,0 +1,37 @@
#!/bin/bash
# The two halves of an .ndx entry's URL share one buffer, so the second must be
# bounded by what the first left. The sanitizer CI build turns a regression
# here into a hard stack-buffer-overflow.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
# each field fills binput's own bound, so together they run one past line[]
"$python" - "$dir/foo.ndx" <<'EOF'
import sys
with open(sys.argv[1], "w") as f:
f.write("CACHE-1.4\n")
f.write("Wed, 01 Jan 2025 00:00:00 GMT\n")
f.write("h" * 1024 + "\n" + "f" * 1024 + "\n" + "0\n")
# a trailing entry keeps the walk inside the buffer, so this exercises the
# field bound alone
f.write("tail/entry\nx\n1\n")
EOF
: >"$dir/foo.dat" # the reader only proceeds when the sibling .dat opens
proxytrack --convert "$dir/out.arc" "$dir/foo.ndx" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed on an .ndx with two full-length URL fields" >&2
exit 1
}

View File

@@ -32,6 +32,7 @@ TESTS = \
00_runnable.test \
01_engine-charset.test \
01_engine-cmdline.test \
01_engine-cmdline-split.test \
01_engine-cookies.test \
01_engine-copyopt.test \
01_engine-crange.test \
@@ -176,6 +177,18 @@ TESTS = \
77_webhttrack-redirect.test \
78_webhttrack-sid.test \
79_local-proxytrack-webdav-mime.test \
80_engine-crash-symbolize.test
80_engine-crash-symbolize.test \
81_webhttrack-maxsize.test \
82_webhttrack-browse-links.test \
83_webhttrack-argescape.test \
84_webhttrack-mirror-verbatim.test \
85_webhttrack-projpath.test \
86_local-proxytrack-cache-longfields.test \
87_local-proxytrack-nodate.test \
88_local-proxytrack-badmtime.test \
89_webhttrack-error-overflow.test \
90_webhttrack-checkbox-clear.test \
91_webhttrack-directory.test \
92_local-proxytrack-ndx-fields.test
CLEANFILES = check-network_sh.cache