mirror of
https://github.com/xroche/httrack.git
synced 2026-07-24 01:29:49 +03:00
b026c80bb6
* Cap the wildcard matcher's recursion depth (#574) strjoker() recursed once per pattern segment, bounded only by the length cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack. That fits Linux's 8MB but not the 1MB a Windows thread gets, so -#test=filterbounds died silently on MSVC x64 and Win32 instead of rejecting the pattern it is meant to reject. Cap the depth at 256 (real filters use fewer than ten). A cut branch does not memoize its failure: the same pair may still match when reached at a shallower depth. The self-test now asserts the depth reached equals the cap, and the .test re-runs it under a 512K stack, which segfaults without the cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Take argv as UTF-8 on Windows, not the ANSI codepage (#573) httrack.exe is an MBCS build, so the CRT transcodes the UTF-16 command line down to the machine's ANSI codepage before handing us char **argv. Non-ASCII arguments are lossy, and anything outside that codepage is destroyed: a Cyrillic or CJK URL on a Western-codepage box becomes '?'. Everything else in the engine already treats char* as UTF-8 on Windows - FOPEN, STAT, UNLINK are hts_*_utf8 wrappers converting to UTF-16 at the syscall boundary. argv is the one thing that never got the memo. Decode the real UTF-16 command line at the entry point instead. The manifest's activeCodePage would fix it in one line, but it needs Windows 10 1903+ and is silently ignored below that, which is precisely the population running the codepages that mangle non-Latin URLs today. Windows-only, so no POSIX symbol and no soname change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Make the engine tests pass on Windows Three of them could never pass there, and none of the three was an engine bug: - rcfile wrote .httrackrc, but HTS_HTTRACKRC is "httrackrc" on Windows, so the engine never found it. Write both names. - filelist made a file unreadable with chmod 000, which Windows ignores. Probe whether the mode is enforced instead of assuming, which also subsumes the root special-case. - ftp-line was #ifndef _WIN32 for its socketpair(), though get_ftp_line() itself is portable. Pair over loopback TCP and register it everywhere. The charset self-tests fed raw non-UTF-8 bytes through argv to probe the decoders. Those bytes cannot survive any Windows command line, whatever argv does, so take them as hex - the convention the sniff self-test already uses - and keep the coverage on every platform. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Run the engine test suite on Windows CI The ~90 tests only ever ran on Linux and macOS, so a Windows-only regression was invisible until a user hit it - and the first run of the offline ones found a third of them failing. Drive them from Git Bash against the native MSVC httrack.exe. Two things that silently ruin the run: MSYS rewrites any argument shaped like a POSIX path, and a URL path is shaped exactly like one, so "-#test=mime /a/b.html" reached the engine as "C:/Program Files/Git/a/b.html"; and a suite that degrades to all-skipped would report green having tested nothing, hence the floor on tests actually passed. This subsumes the codec and cache self-tests the workflow ran inline. The .gitattributes pins the test scripts to LF: a converting checkout rewrites them to CRLF and bash then dies on $'\r' on every line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Log the launch banner argv without re-encoding it The banner ran hts_convertStringSystemToUTF8() over each argv element on Windows, which was right when argv came in the ANSI codepage. argv is UTF-8 on every platform now, so that pass double-encoded a non-ASCII argument into mojibake in hts-log.txt. Log the bytes as they are. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the Win32 CLI build: include htscharset.h after winsock2.h htscharset.h pulls in <windows.h>, which drags in the legacy <winsock.h> unless <winsock2.h> was included first. In httrack.c it sat ahead of the net headers, so the x86 build redefined every sockaddr/winsock symbol (C2011). Move it below htslib.h, where winsock2.h is already in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix two more tests the Win32 CI leg exposed Both are test bugs surfaced by running on 32-bit Windows for the first time, not engine bugs: - strsafe drove its overflow through a char[4]. On a 32-bit build that equals sizeof(char*), so htssafe's array-vs-pointer heuristic (sizeof(A) != sizeof(char*), the MSVC path) reads the array as a pointer and skips the bound: the copy then genuinely overflows and crashes instead of aborting cleanly. Size the buffer off the pointer width so the checked path runs everywhere. - idna's malformed-UTF-8 rejection case passed the bad bytes through argv. They cannot survive a Windows command line: the UTF-16 round-trip turns them into valid U+FFFD, which the encoder then accepts. Feed them as hex so the exact bytes reach the bundled punycode encoder, which rejects them identically on every platform. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
99 lines
3.8 KiB
Bash
Executable File
99 lines
3.8 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
|
|
# Config-file alias loading (no network). A .httrackrc in the working directory
|
|
# is read by optinclude_file(), whose cmdl_ins macro inserts each alias-expanded
|
|
# token into the x_argvblk block. That macro used to copy with an unbounded
|
|
# strcpy on a bare char*; it is now bounded (strlcpybuff + cmdl_room over the
|
|
# block capacity). Two properties are checked:
|
|
# 1. The bound does not truncate: a long user-agent alias reaches doit.log
|
|
# intact. user-agent expands to two tokens (-F <value>), so it exercises
|
|
# both cmdl_ins insertions.
|
|
# 2. The bound holds under exhaustion: a pathological .httrackrc whose alias
|
|
# expansions overflow the block aborts cleanly through the htssafe bounds
|
|
# check (a message naming htsalias.c) instead of overrunning the heap. The
|
|
# unbounded version segfaulted here.
|
|
|
|
# set -e with the intentional-nonzero httrack runs guarded explicitly (the
|
|
# crawls below are expected to fail/abort and their status is inspected by hand).
|
|
set -euo pipefail
|
|
|
|
# Resolve httrack to an absolute path before we cd: PATH may hold a build-relative
|
|
# entry that would not resolve from the temp directory.
|
|
bin=$(command -v httrack) || {
|
|
echo "FAIL: httrack not found on PATH"
|
|
exit 1
|
|
}
|
|
case "$bin" in
|
|
/*) ;;
|
|
*) bin="$(cd "$(dirname "$bin")" && pwd)/$(basename "$bin")" ;;
|
|
esac
|
|
|
|
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_rcfile.XXXXXX") || exit 1
|
|
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
|
|
|
|
# HTS_HTTRACKRC is ".httrackrc" on POSIX but "httrackrc" on Windows: write both,
|
|
# each platform reads the one it knows.
|
|
write_rc() {
|
|
cat >"$1/.httrackrc"
|
|
cp "$1/.httrackrc" "$1/httrackrc"
|
|
}
|
|
|
|
# --- 1. alias token survives the bound intact -------------------------------
|
|
d1="$tmp/intact"
|
|
mkdir -p "$d1"
|
|
echo '<html><body>hello</body></html>' >"$d1/index.html"
|
|
|
|
# optinclude_file() lowercases each config line, so the marker is lowercase to
|
|
# survive the comparison verbatim.
|
|
marker='zzz_rcfile_marker_0123456789_abcdefghijklmnopqrstuvwxyz_intact'
|
|
printf 'user-agent=%s\n' "$marker" | write_rc "$d1"
|
|
|
|
# Run with no -O so the working-directory .httrackrc is loaded (an -O path makes
|
|
# the engine skip the rc files). Output lands in the temp dir. Guard the run so a
|
|
# nonzero exit is captured for the assertion instead of tripping set -e.
|
|
rc=0
|
|
(cd "$d1" && "$bin" "file://$d1/index.html" --quiet -n >.log 2>&1) || rc=$?
|
|
|
|
test "$rc" -eq 0 || {
|
|
echo "FAIL: rc-file crawl exited $rc"
|
|
exit 1
|
|
}
|
|
test -f "$d1/hts-cache/doit.log" || {
|
|
echo "FAIL: doit.log not written (rc file not processed)"
|
|
exit 1
|
|
}
|
|
# A truncated copy would cut the token; require the full -F value.
|
|
grep -q -- "-F $marker" "$d1/hts-cache/doit.log" || {
|
|
echo "FAIL: user-agent alias missing or truncated in doit.log"
|
|
head -1 "$d1/hts-cache/doit.log"
|
|
exit 1
|
|
}
|
|
|
|
# --- 2. block exhaustion aborts through the bound, not the heap -------------
|
|
d2="$tmp/exhaust"
|
|
mkdir -p "$d2"
|
|
echo '<html><body>hi</body></html>' >"$d2/index.html"
|
|
|
|
# Each line inserts ~two tokens of ~200 bytes; 400 lines overflow the block's
|
|
# fixed slack (current_size + 32768) many times over, deterministically.
|
|
val=$(printf 'a%.0s' $(seq 1 200))
|
|
for _ in $(seq 1 400); do
|
|
printf 'user-agent=%s\n' "$val"
|
|
done | write_rc "$d2"
|
|
|
|
# The process aborts (httrack turns the fatal signal into exit 134 either way),
|
|
# so the exit code does not distinguish the bounded abort from a heap overflow;
|
|
# the stderr diagnostic does. The htssafe bounds check names the offending file.
|
|
# Expected to fail, so the nonzero exit is swallowed; only the log is inspected.
|
|
(cd "$d2" && "$bin" "file://$d2/index.html" --quiet -n >.log 2>&1) || true
|
|
|
|
grep -Eq "overflow while copying.*htsalias\.c" "$d2/.log" || {
|
|
echo "FAIL: exhausted rc file did not abort through the htsalias.c bound"
|
|
echo "(an unbounded copy would overrun the heap here)"
|
|
tail -3 "$d2/.log"
|
|
exit 1
|
|
}
|
|
|
|
exit 0
|