mirror of
https://github.com/xroche/httrack.git
synced 2026-07-28 11:32:48 +03:00
1fa3cb4e74
A test that wedges runs until CI cancels the step, and a cancelled step keeps neither its log nor the artifacts its `if: always()` uploads would have produced. That is why nobody has ever been able to say which Windows test hangs, across 19 dead jobs in the last day alone (#795). Each test now runs under a wall-clock budget at the automake harness level, so an overrun names the test, dumps the surviving process tree and an engine stack, and exits 124. The step then fails rather than being cancelled, which is what keeps the log. Every POSIX `make check` leg gets this; the Windows leg runs its own serial loop and now calls the same wrapper. The budget is 600s, the value the Windows leg already used: it has to clear the 540s a three-pass crawl may legitimately take under `local-crawl.sh`'s own 180s-per-pass watchdogs, against a slowest healthy test that actually measures 39s. Two things sit on top of the per-test bound. The Windows suite gives up at 25 minutes so it fails on its own terms well before the 45-minute step timeout, and it sweeps leaked engine processes between tests, naming whichever test left them. An orphaned `httrack.exe` starving the runner is the leading theory for the hang, and that sweep is what would confirm it. This also fixes an unbounded `wait` after `kill_tree` in testlib.sh. When the kill failed to reap, which is exactly the native-Windows case those watchdogs exist for, the watchdog blocked forever and never printed the timeout it was about to report. Stacks differ by platform, and each branch says which one it took, because a dump that silently produces nothing reads as coverage. Linux sends SIGABRT and lets httrack's own crash handler symbolize itself, verified against a real wedged crawl where it named `back_wait` at htsback.c:2710. macOS cannot do that, since htsbacktrace.c gates the handler on `__linux`, so it uses `sample(1)`. Windows uses `cdb` from the SDK, and that is the one path I could not exercise from here; it probes for the binary, reports when it is absent, and is bounded so a debugger that wedges cannot become the new hang. Does not fix #795, only makes it diagnosable. The between-test sweep costs the Windows step about two minutes (506s before, 619s on this run). That buys naming whichever test leaks, which is the whole lead on #795; it can be narrowed or dropped once the leak is found. Signed-off-by: Xavier Roche <roche@httrack.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
77 lines
2.7 KiB
Bash
77 lines
2.7 KiB
Bash
#!/bin/bash
|
|
#
|
|
# automake TEST_LOG_COMPILER: run one .test under a wall-clock budget. A wedged
|
|
# test otherwise hangs the whole job until CI cancels it, and a cancelled step
|
|
# keeps neither its log nor its artifacts -- so nobody ever learns which test
|
|
# hung. On expiry this names the test, dumps what is still running, and exits
|
|
# 124, which makes the step FAIL instead: logs survive a failure.
|
|
#
|
|
# Output goes to the test's own tests/NN_*.log (automake redirects us there) and
|
|
# from there into test-suite.log, which CI prints.
|
|
|
|
set -u
|
|
|
|
testdir=$(cd "$(dirname "$0")" && pwd)
|
|
# shellcheck source=tests/testlib.sh
|
|
. "${testdir}/testlib.sh"
|
|
|
|
# 600s is what the Windows leg already bounded each test at, and it clears the
|
|
# 540s that local-crawl.sh's own watchdogs tolerate for a three-pass crawl
|
|
# (CRAWL_DEADLINE, 180s a pass) -- budget below that and a slow-but-legitimate
|
|
# run would be killed. The slowest healthy test measures 39s. A non-numeric or
|
|
# absurd value falls back; 0 disables the guard, for use under a debugger.
|
|
budget=${HTTRACK_TEST_TIMEOUT:-600}
|
|
case "$budget" in
|
|
'' | *[!0-9]*) budget=600 ;;
|
|
esac
|
|
test "$budget" -gt 0 || exec "$BASH" "$@"
|
|
|
|
# The test script is the last argument; automake passes no others today.
|
|
for name in "$@"; do :; done
|
|
name=$(basename "$name")
|
|
|
|
# Give the test its own TMPDIR, so the hang dump can salvage exactly this test's
|
|
# crawl logs instead of racing (and deleting) a sibling's under "make check -j".
|
|
tmproot=${TMPDIR:-/tmp}
|
|
if mkdir -p "$tmproot/ht.$$" 2>/dev/null; then
|
|
TMPDIR="$tmproot/ht.$$"
|
|
export TMPDIR
|
|
trap 'rm -rf "$TMPDIR" 2>/dev/null || true' EXIT
|
|
fi
|
|
|
|
windows=
|
|
is_windows && windows=1
|
|
|
|
had_m=
|
|
case "$-" in *m*) had_m=1 ;; esac
|
|
test -n "$windows" || set -m # own process group, so kill_tree can signal the group
|
|
"$BASH" "$@" &
|
|
pid=$!
|
|
test -n "$had_m" || test -n "$windows" || set +m
|
|
|
|
# Poll, because bash cannot wait with a deadline and a watchdog subshell would
|
|
# have to signal across process groups, which MSYS cannot do. The interval is the
|
|
# latency this adds to every healthy test, so keep it small where fork is cheap;
|
|
# under MSYS fork costs tens of milliseconds, and a second is what the Windows
|
|
# suite already paid before this wrapper existed.
|
|
if test -n "$windows"; then
|
|
tick=1 per_sec=1
|
|
else
|
|
tick=0.1 per_sec=10
|
|
fi
|
|
|
|
ticks=0
|
|
while kill -0 "$pid" 2>/dev/null; do
|
|
if test "$((ticks / per_sec))" -ge "$budget"; then
|
|
dump_hang_diagnostics "$pid" "$name" "$budget"
|
|
kill_tree "$pid"
|
|
reap_bounded "$pid" || echo "hang: the tree outlived the kill; see the process list above"
|
|
# After the kill, so a crawl log holds the backtrace its engine just wrote.
|
|
dump_crawl_logs
|
|
exit 124
|
|
fi
|
|
sleep "$tick"
|
|
ticks=$((ticks + 1))
|
|
done
|
|
wait "$pid"
|