mirror of
https://github.com/xroche/httrack.git
synced 2026-07-11 03:16:17 +03:00
1abf867333
* Fix double-encoded requests for raw UTF-8 links (#180) Links holding raw non-ASCII bytes were always converted from the page charset to UTF-8, even when already UTF-8: with the iso-8859-1 fallback (reached whenever the charset was declared via HTML5 <meta charset=..>, which the meta scanner predates), each UTF-8 byte was re-encoded and the mirror requested %c3%a7%c2%bb... instead of %e7%bb..., saving a 404. Gate the conversion on a strict (RFC 3629) hts_isStringUTF8, and rewrite hts_getCharsetFromMeta as a size-bounded attribute scanner that handles both the HTML5 and http-equiv forms in any attribute order, first tag wins. Drop the dormant, broken hts_getCharsetFromContentType. Covered by -#test=metacharset/-#test=isutf8 engine self-tests, an 11-variant local-server crawl matrix with an update pass (41_local-utf8-link.test), and new fuzz corpus seeds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: international crawl tracked the double-encoded behavior The default.html sub-crawl asserted the pre-fix output: two mojibake café*.html filenames from double-encoded fetches and 4 errors where the fix leaves 2 (the caf%e9.html Latin-1 escapes the server really lacks). Update to the corrected names and counts; the other three sub-crawls are unchanged. Only runs where ut.httrack.com is reachable (in-tree builds), which is why just the deb CI job caught it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fuzzing httrack
libFuzzer harnesses for the pure hostile-input parsers (charset/UTF-8/IDNA codecs, entity and percent decoders, wildcard filters, URL splitter). Off by default; needs clang.
./bootstrap
mkdir /var/tmp/bld-fuzz && cd /var/tmp/bld-fuzz
CC=clang CFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all -g -O1" \
LDFLAGS="-fsanitize=address,undefined" \
bash /path/to/httrack/configure --enable-fuzzers --disable-shared
make
bash /path/to/httrack/fuzz/run-fuzzers.sh fuzz 60 # 60s per target
Run one target by hand: fuzz/fuzz-url -max_total_time=300 corpusdir fuzz/corpus/url. Seed corpora live in corpus/<target>/; a crash reproducer is replayed with fuzz/fuzz-url crash-file.