Compare commits

..

5 Commits

Author SHA1 Message Date
Xavier Roche
ee550c3972 Bound test 19's dead-address connects so it is fast on macOS
19_local-connect-fallback pins dead loopback IPs (127.0.0.2/.3) before the live
one to exercise address fallback. On Linux those refuse instantly (all of
127.0.0.0/8 is loopback); macOS has only 127.0.0.1, so the connects block to
--timeout, and default retries replay the whole crawl, so the exhaustion case
took ~2 minutes. It passed but dominated the macOS suite once the rest runs in
parallel. Drop --timeout to 5s (bounds the last candidate's stall; the earlier
ones already fall back at min(timeout,10)) and set --retries=0 (no ×3 replay).
Linux is unchanged: the refused connects never reach the timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-11 16:54:25 +02:00
Xavier Roche
145ea4769f Make test 31 fail loudly when the java plugin doesn't load (#529)
31_local-javaclass verifies the java plugin parses a .class constant pool
by asserting a resource named only inside Foo.class gets crawled. When the
plugin fails to load, that assertion just reports hello.gif "not found",
with no hint the plugin was the cause -- which is why an intermittent
local failure sat uninvestigated.

The failure mode is a shadowing system libhtsjava: on a box with httrack
installed, if the soname httrack dlopens is absent from the fresh build
but present in /usr/lib, the loader picks up the system plugin (a
mismatched, non-instrumented build) and it silently parses nothing. It
looked ASan-specific only because that was the build in use; clean CI
runners have no system plugin, so they never hit it. #475 aligned the
dlopen name with the build's own soname, closing the gap.

Assert the plugin's launched-banner suffix ("+libhtsjava") first, so a
load failure trips there with a clear message instead of the opaque
hello.gif miss. The "+libhtsjava" prefix survives future soname bumps.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 16:33:45 +02:00
Xavier Roche
c5da5a2f2e Stop test 36 flaking on macOS when loopback drops connections (#527)
* Stop test 36 flaking on macOS when loopback drops connections

The 36_local-bigcrawl crawl runs -c8 over 361 files and asserted its
error count exactly (--errors 4, the planted 404/410/500/gztrunc). On the
macOS CI runner the busy loopback intermittently resets a few in-flight
connections, so the count came back 7 and the job failed while Linux
passed.

Add a --errors-content harness assertion that counts every "Error:" line
minus the transient-network family (statuscodes -2..-7:
timeout/connect/reset), and switch test 36 to it. The four planted
content errors (404/410/500 and the -1 decompression failure) still count
exactly, so a real regression that adds a content/HTTP error still fails;
only loopback flakiness is tolerated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten --errors-content: drop -7 from transient, match retried errors

Review follow-up. Two fixes to the transient-error filter:

- Narrow the excluded band from -2..-7 to -2..-6. STATUSCODE_TOO_BIG (-7)
  is a deterministic size-cap rejection, not a network transient, so it
  must count as a real error; excluding it could mask a -M regression in
  a future test that reuses this option.

- Also match the "after N retries" error format. htsparse.c logs
  "<msg>" (<code>) after N retries at link ... when retries are enabled,
  which the old "(-code) at link" regex missed. Test 36 forces
  --retries=0 so only the immediate format appears there, but the option
  is a general harness primitive and should hold under default retries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:25:59 +02:00
Xavier Roche
6eaefe6233 Type LLint/TStamp as int64_t and drop the per-platform boilerplate (#526)
* Type LLint/TStamp as int64_t, drop the per-platform boilerplate

The wide-integer typedefs picked long/long long/__int64 per platform and
carried a matching printf-format ladder plus the HTS_LONGLONG capability
macro. That is what let x32 pick a 32-bit long for a "64-bit" LLint (#524).

<stdint.h>/<inttypes.h> answer both questions directly: int64_t is exactly
64-bit signed with defined wrap, and "%" PRId64 is its conversion. LLint
stays signed because -1 is a size/range sentinel across the engine (a
uint64_t would silently break every "< 0" check). The names LLint, TStamp,
and the LLintP macro are kept, so the ~70 call sites are untouched.

HTS_LONGLONG is now dead (a 64-bit type is a hard C99 dependency here:
md5.h includes <stdint.h> and the installed httrack-library.h includes
<inttypes.h>), so its detection blocks and htslib.c's #ifdef around the
GiB/TiB/PiB formatting go too; those branches are always valid now.

No ABI change: int64_t is `long` on LP64, so httrackp/htsblk stay
byte-identical and the exported symbol set is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the LLint typedef comments

Two one-line blocks instead of a four-line header; drop the change-narration
("no per-platform ladder"). Keep the load-bearing note that LLintP carries its
own '%'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:09:54 +02:00
Xavier Roche
33249be5a0 Assert kept content in the update-error test, not just size (#176 follow-up) (#525)
44_local-update-errormask checked only keep.dat's size (>= 1024). The good
body carries a distinctive KEEP marker that nothing asserted, so a same-size
wrong-content overwrite would still pass. Also assert the content matches
^KEEP and lacks the 403 error body, so the test discriminates the fix from
an equal-size clobber the size check alone misses.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:49:01 +02:00
5 changed files with 25 additions and 7 deletions

View File

@@ -66,9 +66,12 @@ test -n "$port" || {
}
out="$tmpdir/crawl"
# macOS has only 127.0.0.1, so the dead 127.0.0.x connects block to the timeout
# instead of refusing instantly (as on Linux); a small --timeout + --retries=0
# bound the stall without changing what the fallback exercises.
HTTRACK_DEBUG_RESOLVE="deadhost:127.0.0.2,127.0.0.1" \
httrack "http://deadhost:$port/simple/basic.html" -O "$out" \
-c1 --robots=0 --timeout=30 --quiet -Z >"$tmpdir/log" 2>&1
-c1 --robots=0 --timeout=5 --retries=0 --quiet -Z >"$tmpdir/log" 2>&1
log="$out/hts-log.txt"
@@ -92,12 +95,12 @@ test -f "$out/deadhost_$port/simple/basic.html" || {
exit 1
}
# every address refused: the slot exhausts the list, then errors out (the
# harness timeout would catch a hang/loop; refused connects are instant)
# every address dead: the slot exhausts the list, then errors out (the harness
# timeout would catch a hang/loop)
out2="$tmpdir/crawl2"
HTTRACK_DEBUG_RESOLVE="alldead:127.0.0.2,127.0.0.3" \
httrack "http://alldead:$port/simple/basic.html" -O "$out2" \
-c1 --robots=0 --timeout=30 --quiet -Z >"$tmpdir/log2" 2>&1
-c1 --robots=0 --timeout=5 --retries=0 --quiet -Z >"$tmpdir/log2" 2>&1
log2="$out2/hts-log.txt"
grep -q "trying next address" "$log2" || {

View File

@@ -17,7 +17,10 @@ printf 'GIF89a' >"$tmproot/javaclass/hello.gif"
printf '\xCA\xFE\xBA\xBE\x00\x00\x00\x32\x00\x02\x01\x00\x09hello.gif\x00\x00\x00\x00' \
>"$tmproot/javaclass/Foo.class"
# --log-found first: a plugin-load failure (wrong soname, shadowing system
# libhtsjava) trips here on the banner suffix, not as an opaque "not found".
bash "$top_srcdir/tests/local-crawl.sh" --root "$tmproot" --errors 0 \
--log-found '\+libhtsjava' \
--found 'javaclass/Foo.class' \
--found 'javaclass/hello.gif' \
httrack 'BASEURL/javaclass/index.html'

View File

@@ -9,7 +9,7 @@ set -euo pipefail
: "${top_srcdir:=..}"
bash "$top_srcdir/tests/local-crawl.sh" --rerun \
--errors 4 --files 361 \
--errors-content 4 --files 361 \
--found 'big/p/95.html' \
--found 'big/a/d1/d2/d3/d4/d5/d6/d7/d8/deep.png' \
--found 'big/a/f2-2x.png' \

View File

@@ -8,4 +8,6 @@ set -euo pipefail
bash "$top_srcdir/tests/local-crawl.sh" --rerun \
--found 'errmask/keep.dat' \
--file-min-bytes 'errmask/keep.dat' 1024 \
--file-matches 'errmask/keep.dat' '^KEEP' \
--file-not-matches 'errmask/keep.dat' 'error 403' \
httrack 'BASEURL/errmask/index.html'

View File

@@ -14,11 +14,13 @@
# Usage:
# bash local-crawl.sh [--tls] [--root DIR] [--cookie NAME=VALUE ...] \
# [--rerun-args 'ARGS'] \
# --errors N --files N --found PATH ... --directory PATH ... \
# --errors N --errors-content N --files N --found PATH ... --directory PATH ... \
# --log-found REGEX ... --log-not-found REGEX ... \
# --file-matches PATH REGEX ... --file-not-matches PATH REGEX ... \
# --file-min-bytes PATH N --max-mirror-bytes N \
# httrack BASEURL/some/path [httrack-args...]
# --errors counts every "Error:" log line; --errors-content drops transient
# network failures (codes -2..-6) that flake on busy loopback under -c8.
# --log-found/--log-not-found grep (ERE) the crawl's hts-log.txt.
# --max/--min-mirror-bytes bound the mirrored content bytes (host root).
# --file-matches/--file-not-matches grep (ERE) a mirrored file (PATH under the
@@ -131,7 +133,7 @@ while test "$pos" -lt "$nargs"; do
pos=$((pos + 1))
rerun_args="${args[$pos]}"
;;
--errors | --files)
--errors | --errors-content | --files)
audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
pos=$((pos + 1))
;;
@@ -334,6 +336,14 @@ while test "$i" -lt "${#audit[@]}"; do
assert_equals "checking errors" "${audit[$i]}" \
"$(grep -iEc "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt")"
;;
--errors-content)
i=$((i + 1))
total=$(grep -icE "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt")
# transient network failures (statuscode -2..-6) flake on busy loopback;
# the code parens are followed by " at link" or " after N retries at link"
transient=$(grep -cE '\(-[2-6]\) (at link|after )' "${out}/hts-log.txt" || true)
assert_equals "checking content errors" "${audit[$i]}" "$((total - transient))"
;;
--files)
i=$((i + 1))
nFiles=$(grep -E "^HTTrack Website Copier/[^ ]* mirror complete in " "${out}/hts-log.txt" |