Three suite tests fail on an emulated buildd for being slow, not broken (#1231)

* Three suite tests fail on an emulated buildd for being slow, not broken

hppa reported 3.49.21-1 as Build-Attempted with 105, 151 and 269 red, all three
on wall clock under qemu-user: the guard's own diagnostics dump took 31s of a
30s bound, one configure run passed a hard 300s cap, and the pairwise header
sweep outran the 600s harness budget mid-batch.

Each now measures the property instead of the host. 105 times the guard to its
DUMP announcement, leaving the dump (minutes, emulated) out of the bound. 151
watches configure's output for silence rather than capping its total, and skips
when the budget runs out while it is still making progress. The sweep runs in
slices so it can be given up on, paces itself against what 269 hands it, and 269
declares the larger budget its n^2 compiles need, so the emulated leg still runs
it to completion instead of pacing out.

The emulated leg is the CI counterpart of that buildd and was green on the same
commit: it prints test-suite.log now, so a test that skipped rather than ran
stops reading as coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix the review's two skip-masking holes and the unbounded budget raise

A hung configure could reach the budget skip before the silence window closed, so
a wedge reported SKIP; the skip now needs a run that is still writing and enough
budget left for the detector to speak. 151 also paced inside run(), before
accept/reject read the verdict, so a configure that answered wrongly could exit 77
instead of 1: pacing moved to the callers, which judge first.

The per-test raise is bounded and normalized through one budget parser, since bash
test errors rather than compares past intmax and would have left the guard unarmed,
and a leading zero read as octal in arithmetic and decimal in test. Renamed
TEST_TIMEOUT_AT_LEAST so each use site carries the upwards-only rule, documented in
AGENTS.md, and the sweep takes an explicit --budget rather than sniffing the
environment, so the MSVC job cannot report a paced skip as a header break.

Tests for each: 151 drives run() through a configure_cmd seam with a child that
hangs and one that only crawls, 105 requires the DUMP announcement to be seen while
the guard runs (its fallback made the latency bound vacuous) and pins the hostile
budget values, and the sweep counts the units that reached the compiler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Two of the new assertions passed a mutant; make them bite

Mutation-testing the added tests found two that could not see the bug they were
written for. The verdict check stubbed run() out, so a pacer left inside the real
one stayed invisible: it now drives the real run() through a child that answers
wrongly with the budget spent. The announcement check only asked that the marker
appear while the guard ran, which a driver announcing after the dump still
satisfies: a slow ps widens the dump, and the marker must now precede it.

Writing the first exposed a third: reject takes one argument fewer than accept, so
the extra one reached run() as an env assignment and the child failed to exec,
which made the probe pass on the wrong answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The verdict probe left the pacer disarmed

It numbered its run past the case count, so the pacer it exists to catch declined
to fire on a negative "steps left", and the mutant that puts pacing back inside
run() survived. Keep cases ahead of the run number.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A killed configure left its child running and hung the macOS job

The macOS leg reported no test failure and then sat until its 20-minute step
timeout: bash 3.2 does not replace the subshell around the child with the child
itself, so killing that subshell left the run alive, and it outlived make check.
Give each run its own process group and kill the group.

The probe that exposed it now covers it: a child that spawns a child of its own,
and nothing of it left running afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -30,6 +30,11 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
+- Each test runs under a 600s wall-clock guard that reports a wedge as 124. A test
+  whose own work outlasts it raises the budget with a `# TEST_TIMEOUT_AT_LEAST: N`
+  line, at column 0 within its first 40 lines, and paces itself with
+  `skip_if_out_of_budget` so a host too slow to finish skips instead. The value only
+  ever raises the budget: nothing can disarm the guard.
 - Run teardown with errexit off: `trap 'set +e; cleanup' EXIT`. Under `set -e` a
   failing cleanup command becomes the test's exit status (#773). Keep the other
   signals on their own `trap` line, or errexit stays off for the rest of the run.

tests/105_suite-timeout.test

@@ -36,11 +36,43 @@ echo "marker: the wedged test started"
 sleep 300
 EOF
 
-start=$SECONDS
+# Time the guard to its DUMP announcement, not to the driver's exit: the dump that
+# follows runs for minutes on an emulated host, and that is not what is under test.
 rc=0
-HTTRACK_PROGRESS_LOG="$tmp/progress" HTTRACK_TEST_TIMEOUT=5 \
-    bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 || rc=$?
-elapsed=$((SECONDS - start))
+fired=
+marked=
+total=
+run_wedged() { # run_wedged <budget> [VAR=VAL...]
+    local budget=$1 start=$SECONDS pid
+    shift
+    : >"$tmp/progress"
+    rc=0
+    fired=
+    marked=
+    env "$@" HTTRACK_PROGRESS_LOG="$tmp/progress" HTTRACK_TEST_TIMEOUT="$budget" \
+        bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 &
+    pid=$!
+    while kill -0 "$pid" 2>/dev/null; do
+        # Read with the shell: a fork per poll would blur the latency being measured.
+        if read -r line <"$tmp/progress" 2>/dev/null && test "$line" = "DUMP 90_wedged.test"; then
+            fired=$((SECONDS - start))
+            marked=1
+            break
+        fi
+        poll_wait 0.1 || sleep 1
+    done
+    wait "$pid" || rc=$?
+    total=$((SECONDS - start))
+    # Missed between two polls: the whole run then bounds the latency from above.
+    test -n "$fired" || fired=$total
+}
+
+# Retried once, because a dump short enough to fall between two polls is a race, not a
+# regression; announcing after the dump loses both attempts.
+run_wedged 5
+test -n "$marked" || run_wedged 5
+test -n "$marked" ||
+    fail "the announcement was never seen while the guard ran, so its latency is unknown"
 
 # Announced when the dump starts, which runs for minutes: a suite watchdog
 # reading that log would take the silence for a wedge and kill the step.
@@ -49,8 +81,8 @@ grep -qx 'DUMP 90_wedged.test' "$tmp/progress" ||
 
 test "$rc" -eq 124 || fail "wedged test reported $rc, want 124"
 # Never before the budget, or a slow-but-healthy test would be killed too.
-test "$elapsed" -ge 5 || fail "the guard fired early (${elapsed}s of a 5s budget)"
-test "$elapsed" -lt 30 || fail "the guard fired late (${elapsed}s)"
+test "$fired" -ge 5 || fail "the guard fired early (${fired}s of a 5s budget)"
+test "$fired" -lt 30 || fail "the guard fired late (${fired}s)"
 grep -q 'marker: the wedged test started' "$out" || fail "the test's own output was lost"
 # The header, not a bare name: the process list quotes the test's path too, so a
 # wrapper that named nothing would still match that.
@@ -58,6 +90,20 @@ grep -q '^===== TIMEOUT: 90_wedged.test exceeded' "$out" ||
     fail "the diagnostics do not name the test"
 grep -q "own process tree" "$out" || fail "no process list in the diagnostics"
 
+# Announced BEFORE the dump, not merely at some point during it: the watchdog reading
+# that log takes the silence of a dump for a wedge. Only a slow dump tells the two
+# orderings apart, and the dump's own ps is what this makes slow.
+if ! is_windows; then
+    printf '#!/bin/sh\nsleep 3\nexec %s "$@"\n' "$(command -v ps)" >"$shim/ps"
+    chmod +x "$shim/ps"
+    run_wedged 5 "PATH=$shim:$PATH"
+    rm -f "$shim/ps" # before the starve shim shares this directory
+    test "$rc" -eq 124 || fail "the guard reported $rc under a slow dump, want 124"
+    test -n "$marked" || fail "no announcement under a slow dump"
+    test "$((total - fired))" -ge 2 ||
+        fail "announced with the dump (${fired}s of ${total}s), not before it"
+fi
+
 # The budget is read, not hard-coded: well under it, the same shape survives.
 printf 'sleep 3\necho "slow but healthy"\n' >"$tmp/92_slow.test"
 rc=0
@@ -78,15 +124,11 @@ sleep 1
 # the loop it stretches is the same one poll_wait's fd tick drives.
 (
     starve_sleep "$shim" 4 || fail "could not install the slow sleep"
-    start=$SECONDS
-    rc=0
-    HTTRACK_POLL_SLEEP=1 HTTRACK_TEST_TIMEOUT=1 \
-        bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 || rc=$?
-    elapsed=$((SECONDS - start))
+    run_wedged 1 HTTRACK_POLL_SLEEP=1
     test "$rc" -eq 124 || fail "starved guard reported $rc, want 124"
-    # Generous: the diagnostics dump runs inside this window too.
-    test "$elapsed" -lt 25 ||
-        fail "budget counted polls, not seconds: ${elapsed}s for a 1s budget"
+    # 10 stretched polls would be 40s; a handful of them is the whole margin here.
+    test "$fired" -lt 25 ||
+        fail "budget counted polls, not seconds: ${fired}s for a 1s budget"
 )
 
 # --- exit status and output of a healthy test pass straight through ----------
@@ -127,6 +169,84 @@ saw_budget 45 "an explicit budget"
 HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/95_budget.test" >"$out" 2>&1
 saw_budget 0 "a disabled guard"
 
+# --- a test may raise the budget, never lower it ----------------------------
+# 269's header sweep is n^2 compiles, real work that outlasts the wedge budget on an
+# emulated host; anything else asking would be disarming the guard.
+raiser() { # raiser <asked for>
+    # shellcheck disable=SC2016 # the fixture has to read the variable, not us
+    printf '# TEST_TIMEOUT_AT_LEAST: %s\necho "budget=${HTTRACK_TEST_TIMEOUT-unset}"\n' \
+        "$1" >"$tmp/97_raise.test"
+}
+# 0900 must not read as octal, and a value past intmax must not reach `test`, which
+# errors on it rather than comparing and would leave the guard unarmed.
+for want in 900:900 5:600 garbage:600 0900:900 99999999999999999999:600 ' 900':600; do
+    raiser "${want%%:*}"
+    HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1
+    saw_budget "${want##*:}" "a test asking for '${want%%:*}'"
+done
+raiser 900
+HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1
+saw_budget 0 "a raise under a disabled guard"
+# Read from the header only, or a test's own data would be one: 151 writes this line.
+window() { # window <lines of padding> <budget wanted>
+    {
+        i=0
+        while test "$i" -lt "$1"; do
+            i=$((i + 1))
+            echo "# pad $i"
+        done
+        cat "$tmp/97_raise.test"
+    } >"$tmp/97_deep.test"
+    HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/97_deep.test" >"$out" 2>&1
+    saw_budget "$2" "a raise on line $(($1 + 1))"
+}
+raiser 900
+window 39 900 # the last line the header reaches
+window 40 600
+
+# Enforced, not merely exported: the number the guard uses is the one it kills on.
+printf '# TEST_TIMEOUT_AT_LEAST: 900\nsleep 4\necho "outlived the default"\n' \
+    >"$tmp/97_raise.test"
+rc=0
+HTTRACK_TEST_TIMEOUT=2 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1 || rc=$?
+test "$rc" -eq 0 || fail "a 4s test that raised the budget to 900 reported $rc"
+grep -q 'outlived the default' "$out" || fail "the raised budget killed the test anyway"
+printf '# TEST_TIMEOUT_AT_LEAST: 1\nsleep 3\necho "not shrunk"\n' >"$tmp/97_raise.test"
+rc=0
+HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1 || rc=$?
+test "$rc" -eq 0 || fail "a 3s test asking for a 1s budget reported $rc"
+grep -q 'not shrunk' "$out" || fail "the header shrank the budget and killed the test"
+
+# --- one budget parser, and what is left of it ------------------------------
+secs() { # secs <value in the environment> <seconds it must read as>
+    local got
+    got=$(HTTRACK_TEST_TIMEOUT=$1 bash -c '. "$1"; budget_secs' _ "${testdir}/testlib.sh" 2>&1)
+    test "$got" = "$2" || fail "budget_secs read '$1' as '$got', want $2"
+}
+secs 45 45
+secs 0900 900 # decimal, or $((...)) and test disagree on the same string
+secs garbage 600
+secs 99999999999999999999 600 # past intmax, where test errors instead of comparing
+secs 0 0
+
+# budget_left hands a child what is left of it, and keeps 0 meaning "no guard".
+# shellcheck disable=SC2016 # the fixture has to call the helper, not us
+printf '. "%s"\nsleep 2\necho "left=$(budget_left) at=$SECONDS"\n' "${testdir}/testlib.sh" \
+    >"$tmp/98_left.test"
+# Exact, against the clock the child itself read: a tolerance would pass a wrong epoch.
+HTTRACK_TEST_TIMEOUT=60 bash "$driver" "$tmp/98_left.test" >"$out" 2>&1
+got=$(sed -n 's/^left=\([0-9][0-9]*\) .*/\1/p' "$out")
+at=$(sed -n 's/^left=[0-9][0-9]* at=\([0-9][0-9]*\)$/\1/p' "$out")
+case "$got$at" in '' | *[!0-9]*) fail "a 60s budget printed '$(cat "$out")'" ;; esac
+test "$got" -eq "$((60 - at))" ||
+    fail "a 60s budget left $got with $at gone, want $((60 - at))"
+HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/98_left.test" >"$out" 2>&1
+grep -q '^left=0 ' "$out" || fail "a disabled guard left '$(cat "$out")', want 0"
+# Never 0 on an exhausted budget: a child would read that as the guard being off.
+HTTRACK_TEST_TIMEOUT=1 bash -c '. "$1"; sleep 2; echo "left=$(budget_left)"' \
+    _ "${testdir}/testlib.sh" >"$out" 2>&1
+grep -qx 'left=1' "$out" || fail "an exhausted budget left '$(cat "$out")', want 1"
+
 # --- a test too slow to finish skips instead of being killed ----------------
 # hppa spends ~150s on one configure run, and 124 takes the build down where 77
 # does not.

tests/151_bash-shell-validate.test

@@ -8,6 +8,8 @@ set -euo pipefail
 
 # shellcheck source=tests/testlib.sh
 . "$(dirname "$0")/testlib.sh"
+# shellcheck source=tests/proclib.sh
+. "$(dirname "$0")/proclib.sh"
 
 sh=${BASH_SHELL:-}
 test -n "$sh" || {
@@ -51,11 +53,21 @@ chmod 755 "$tmp/fakebin/bash"
 mkfifo "$tmp/fifo"
 chmod 755 "$tmp/fifo"
 
+# Sampled rather than polled per second: the size read is a fork, and an emulated
+# host pays for it. SILENCE clears the slowest single configure probe there.
+SAMPLE=5
+SILENCE=${HTTRACK_CONFIGURE_SILENCE:-120}
+RESERVE=15 # what killing the run and skipping still needs of the budget
+
 n=0
 cases=16 # reject/accept calls below; pinned again once they have all run
 status=0
 log=
 rundir=
+took=0
+# What run() launches, so the checks below can hand it a child that hangs or one that
+# only crawls; nothing else may override it.
+configure_cmd=(bash "$tmp/src/configure" --disable-https)
 run() { # run <label> <env argument>...
     local label=$1 began=$SECONDS
     shift
@@ -65,23 +77,56 @@ run() { # run <label> <env argument>...
     status=0
     # Capped: configure executes the candidate, and a hang wedges "make check" with no output
     # at all. Polled, not a backgrounded "sleep" watchdog, which outlives the run it guards.
-    (cd "$rundir" && env "$@" bash "$tmp/src/configure" --disable-https) \
+    local had_m=
+    case "$-" in *m*) had_m=1 ;; esac
+    # Own process group, so the kills below reach what configure spawned: bash 3.2 keeps
+    # the subshell it runs in, and killing that alone leaves the child running (macOS).
+    set -m
+    (cd "$rundir" && env "$@" "${configure_cmd[@]}") \
         >"$rundir/log" 2>&1 &
-    local pid=$! waited=0
-    while test "$waited" -lt 300 && kill -0 "$pid" 2>/dev/null; do
+    local pid=$! waited=0 quiet=0 size=0 now left
+    test -n "$had_m" || set +m
+    # A hang is silence, not slowness: configure writes a line per probe,
+    # but hppa's emulated run can take longer overall than a runner's whole budget (#1146).
+    while kill -0 "$pid" 2>/dev/null; do
         sleep 1
         waited=$((waited + 1))
+        test "$((waited % SAMPLE))" -eq 0 || continue
+        now=$(wc -c <"$rundir/log")
+        if test "$now" -gt "$size"; then
+            size=$now
+            quiet=0
+        else
+            quiet=$((quiet + SAMPLE))
+        fi
+        test "$quiet" -lt "$SILENCE" || {
+            kill_tree "$pid"
+            echo "configure wrote nothing for ${quiet}s of ${waited}s for $label" >&2
+            tail -5 "$rundir/log" >&2
+            exit 1
+        }
+        # Still writing but out of time: skip, where the harness would kill the whole test
+        # and take the build down with it. Only while writing, and only with a full silence
+        # window still affordable, or a hang would reach this before the check above fires
+        # and a wedge would report a skip. 0 is the guard off.
+        left=$(budget_left)
+        if test "$quiet" -eq 0 && test "$left" -ne 0 &&
+            test "$left" -le "$((RESERVE + SILENCE))"; then
+            kill_tree "$pid"
+            echo "$label was still configuring ${waited}s in and the budget is out; skipping" >&2
+            exit 77
+        fi
     done
-    if kill -0 "$pid" 2>/dev/null; then
-        kill -9 "$pid" 2>/dev/null
-        echo "configure did not return within ${waited}s for $label" >&2
-        tail -5 "$rundir/log" >&2
-        exit 1
-    fi
     wait "$pid" || status=$?
     log=$(cat "$rundir/log")
+    took=$((SECONDS - began))
     echo "run $n ($label): exit $status"
-    skip_if_out_of_budget "$((cases - n))" "$((SECONDS - began))"
+}
+
+# Pace here rather than in run(), which returns with the answer still unjudged: a
+# skip between the two would bury a configure that answered wrongly.
+paced() {
+    skip_if_out_of_budget "$((cases - n))" "$took"
 }
 
 reject() { # reject <label> <expected message> <env argument>...
@@ -97,6 +142,7 @@ reject() { # reject <label> <expected message> <env argument>...
         tail -5 <<<"$log" >&2
         exit 1
     }
+    paced
 }
 
 # accept <label> <expected $(BASH_SHELL), "" for any> <expected message, "" for none> <env argument>...
@@ -125,8 +171,69 @@ accept() {
             exit 1
         }
     fi
+    paced
 }
 
+# --- what run() does to a child that hangs, and to one that is merely slow -------
+# Driven through configure_cmd, since the real configure can do neither on demand.
+probe() { # probe <run number> <seconds of budget left> <command>...
+    local want_n=$1 left=$2 rc=0
+    shift 2
+    (
+        # shellcheck disable=SC2030 # the isolation is the point: the real count is next door
+        n=$want_n SAMPLE=1 SILENCE=2
+        # shellcheck disable=SC2030,SC2031 # likewise: the budget here is the probe's own
+        export HTTRACK_TEST_TIMEOUT=$((SECONDS + left))
+        configure_cmd=("$@")
+        run probe
+    ) >"$tmp/probe.log" 2>&1 || rc=$?
+    echo "$rc"
+}
+# A wedge must fail even with the budget gone, or #922 comes back as a skip.
+rc=$(probe 90 6 sleep 999)
+test "$rc" -eq 1 || fail "a silent configure with 6s of budget reported $rc, want 1"
+grep -q 'wrote nothing' "$tmp/probe.log" || fail "the hang was not named: $(cat "$tmp/probe.log")"
+# Slow but talking is the emulated buildd, and a skip there beats the harness kill.
+rc=$(probe 91 6 bash -c 'while :; do echo tick; sleep 1; done')
+test "$rc" -eq 77 || fail "a slow but writing configure with 6s of budget reported $rc, want 77"
+# The kill has to reach what configure spawned. bash 3.2 keeps the subshell around the
+# child, so killing that alone leaves a live configure behind: it outlives "make check"
+# and holds the CI step open to its own timeout, with the suite reporting no failure.
+rc=$(probe 92 6 bash -c 'sleep 987 & wait')
+test "$rc" -eq 1 || fail "a silent configure with a child of its own reported $rc, want 1"
+sleep 1
+! ps_snapshot | grep -q '[s]leep 987' || fail "the killed run left its child running"
+# The pacer must not fire before the case is judged: run() returns with the verdict
+# still unread, and a skip there would bury a configure that answered wrongly. Through
+# the real run(), since a stub cannot see a pacer left inside the one it replaced.
+verdict() { # verdict <accept|reject> <run number> <status the child exits with>
+    local rc=0
+    (
+        # shellcheck disable=SC2030,SC2031 # the isolation is the point: the real run is next door
+        # Cases still to come, or the pacer this is looking for would decline to fire.
+        n=$2 cases=$(($2 + 5)) SAMPLE=1
+        # Spent by the time the run ends, so a pacer anywhere after it would fire.
+        # shellcheck disable=SC2030,SC2031 # likewise: the budget here is the probe's own
+        export HTTRACK_TEST_TIMEOUT=$((SECONDS + 4))
+        configure_cmd=(bash -c "sleep 3; exit $3")
+        # Their arities differ, and an extra argument would reach run() as an env
+        # assignment: the child would then fail to exec and answer the wrong question.
+        case "$1" in
+        accept) accept probe-verdict '' '' ;;
+        *) reject probe-verdict '' ;;
+        esac
+    ) >/dev/null 2>&1 || rc=$?
+    test "$rc" -eq 1 || fail "$1 of a wrong answer with the budget spent reported $rc, want 1"
+}
+verdict accept 80 1 # configure rejected what it must accept
+verdict reject 81 0 # configure accepted what it must reject
+# The probes ran in subshells, so the real cases below start from a clean count.
+cases=16
+n=0
+status=0
+log=
+took=0
+
 # The four that configure to completion run first. A reject stops at the
 # BASH_SHELL check and costs a fraction of one, and the pacer projects the step it
 # just timed: behind the cheap ones it read far too low and 151 met the harness

tests/269_install-headers-order.test

@@ -6,6 +6,10 @@
 # a break needing three headers, or a macro the consumer defined first, is out
 # of reach here. The sweep is shared with the MSVC job, which has no automake to
 # install with and so stages the same list out of DevIncludes_DATA (#1153).
+#
+# n^2 compiles is real work, not a wedge: emulated, it needs more than the suite's
+# default budget, and the sweep paces itself against whatever is left of this one.
+# TEST_TIMEOUT_AT_LEAST: 900
 
 set -euo pipefail
 
@@ -69,7 +73,10 @@ done
 
 sweep_argv=(--headers-dir "$tmp/include/httrack" --cc "${CC:-cc}" --cxx "$cxx")
 [ "${#cpp_argv[@]}" -eq 0 ] || sweep_argv+=(-- "${cpp_argv[@]}")
-bash "$testdir/install-headers-sweep.sh" "${sweep_argv[@]}" ||
-    fail "installed headers do not survive every include order"
+rc=0
+bash "$testdir/install-headers-sweep.sh" --budget "$(budget_left)" "${sweep_argv[@]}" || rc=$?
+# 77 is the sweep giving up on a host too slow to finish it, not a broken header.
+[ "$rc" -ne 77 ] || exit 77
+[ "$rc" -eq 0 ] || fail "installed headers do not survive every include order"
 
 exit 0

tests/install-headers-sweep.sh

@@ -14,7 +14,8 @@ set -euo pipefail
 
 usage() {
     echo "usage: ${0##*/} {--srcdir DIR [--builddir DIR] | --headers-dir DIR}" \
-        "[--backend cl|cc] [--cc CMD] [--cxx CMD] [--self-test] [-- CPPFLAGS...]" >&2
+        "[--backend cl|cc] [--cc CMD] [--cxx CMD] [--budget SECONDS] [--self-test]" \
+        "[-- CPPFLAGS...]" >&2
     exit 1
 }
 
@@ -26,6 +27,7 @@ cc_cmd=""
 cxx_cmd=""
 cxx_set=0
 selftest=0
+budget=
 extra=()
 while [ $# -gt 0 ]; do
     case $1 in
@@ -33,6 +35,10 @@ while [ $# -gt 0 ]; do
         selftest=1
         shift
         ;;
+    --budget)
+        budget=${2-}
+        shift 2 || usage
+        ;;
     --srcdir)
         srcdir=${2-}
         shift 2 || usage
@@ -237,15 +243,45 @@ fi
 
 began=$SECONDS
 bad=0
+# Sliced only for a caller that gave a budget: one call per batch cannot be given up on,
+# and an emulated compiler needs more time for it than the harness allows a test (#1146).
+# Unsliced elsewhere, so the Windows job keeps paying one compiler spawn per batch.
+if [ -n "$budget" ] && [ "$budget" -gt 0 ]; then
+    export HTTRACK_TEST_TIMEOUT=$budget
+    slices=8
+else
+    slices=1
+fi
+slice=$(((${#units[@]} + slices - 1) / slices))
+# From the slice size, not from $slices: they differ whenever the units do not divide
+# evenly, and a step count that outlives the loop leaves the pacer projecting forever.
+per=$(((${#units[@]} + slice - 1) / slice))
+left=$((${#langs[@]} * ${#modes[@]} * per))
+swept=0
 for lang in "${langs[@]}"; do
     for mode in "${modes[@]}"; do
-        compile "$lang" "$mode" "${units[@]}" || {
-            head -40 "$sweep_log" >&2
-            echo "the headers do not compile as $lang standalone and pairwise ($mode)" >&2
-            bad=1
-        }
+        i=0
+        while [ "$i" -lt "${#units[@]}" ]; do
+            step=$SECONDS
+            chunk=("${units[@]:i:slice}")
+            swept=$((swept + ${#chunk[@]}))
+            compile "$lang" "$mode" "${chunk[@]}" || {
+                head -40 "$sweep_log" >&2
+                echo "the headers do not compile as $lang standalone and pairwise ($mode)" >&2
+                bad=1
+            }
+            i=$((i + slice))
+            left=$((left - 1))
+            # Only while nothing has failed: a skip past a real break would bury it.
+            [ "$bad" -ne 0 ] || [ -z "$budget" ] ||
+                skip_if_out_of_budget "$left" "$((SECONDS - step))"
+        done
     done
 done
+# What reached the compiler, not what was generated: a slice loop that steps past a unit
+# would otherwise report the full set and pass.
+want=$((${#langs[@]} * ${#modes[@]} * ${#units[@]}))
+[ "$swept" -eq "$want" ] || fail "compiled $swept units of $want, the slicing lost some"
 echo "swept $n headers standalone and pairwise x ${#modes[@]} bytecode modes x ${langs[*]}" \
     "= $((${#modes[@]} * ${#langs[@]} * ${#units[@]})) units in $((SECONDS - began))s with $backend"
 [ "$bad" -eq 0 ] || exit 1

tests/test-timeout.sh

@@ -22,19 +22,39 @@ testdir=$(cd "$(dirname "$0")" && pwd)
 # (CRAWL_DEADLINE, 180s a pass) -- budget below that and a slow-but-legitimate
 # run would be killed. The slowest healthy test measures 39s. A non-numeric or
 # absurd value falls back; 0 disables the guard, for use under a debugger.
-budget=${HTTRACK_TEST_TIMEOUT:-600}
-case "$budget" in
-'' | *[!0-9]*) budget=600 ;;
-esac
+budget=$(budget_secs)
+
+# The test script is the last argument; automake passes no others today.
+for path in "$@"; do :; done
+name=$(basename "$path")
+
+# A test whose work legitimately outlasts the wedge budget says so in its header
+# (269 sweeps n^2 compiles and paces itself inside it). The name carries the rule the
+# reader cannot see: it raises the budget, so no test can disarm the guard. Read with
+# the shell to keep it off the per-test fork bill, and bounded, since bash's `test`
+# errors rather than compares past intmax and would leave the guard unarmed.
+if test "$budget" -gt 0 && test -r "$path"; then
+    read_lines=0
+    while test "$read_lines" -lt 40 && IFS= read -r line; do
+        read_lines=$((read_lines + 1))
+        case "$line" in
+        '# TEST_TIMEOUT_AT_LEAST: '*)
+            want=${line#'# TEST_TIMEOUT_AT_LEAST: '}
+            case "$want" in
+            '' | *[!0-9]* | ???????*) ;;
+            *) test "$((10#$want))" -le "$budget" || budget=$((10#$want)) ;;
+            esac
+            break
+            ;;
+        esac
+    done <"$path"
+fi
+
 # Exported so a test can pace itself against the same number (skip_if_out_of_budget)
 # instead of being killed halfway.
 export HTTRACK_TEST_TIMEOUT="$budget"
 test "$budget" -gt 0 || exec "$BASH" "$@"
 
-# The test script is the last argument; automake passes no others today.
-for name in "$@"; do :; done
-name=$(basename "$name")
-
 # Give the test its own TMPDIR, so the hang dump can salvage exactly this test's
 # crawl logs instead of racing (and deleting) a sibling's under "make check -j".
 tmproot=${TMPDIR:-/tmp}

tests/testlib.sh

@@ -539,16 +539,39 @@ EOF
 # one step is slower than its neighbours. It asks an ordering of the callers
 # instead, expensive steps first, so no step left can outrun the reserve the one
 # before it set (#1146).
-skip_if_out_of_budget() { # skip_if_out_of_budget <steps left> <seconds the last took>
-    local budget=${HTTRACK_TEST_TIMEOUT:-600} need=$(($2 + $2 / 2))
+# The budget test-timeout.sh enforces, in seconds, 0 being the guard off. The one
+# parser: a value bash arithmetic or test would choke on falls back to the default,
+# and a leading zero would otherwise read as octal in one place and decimal in the next.
+budget_secs() {
+    local budget=${HTTRACK_TEST_TIMEOUT:-600}
+    case "$budget" in '' | *[!0-9]* | ???????*) budget=600 ;; esac
+    echo "$((10#$budget))"
+}
 
-    case "$budget" in '' | *[!0-9]*) budget=600 ;; esac
+skip_if_out_of_budget() { # skip_if_out_of_budget <steps left> <seconds the last took>
+    local budget need=$(($2 + $2 / 2))
+
+    budget=$(budget_secs)
     test "$1" -gt 0 && test "$budget" -gt 0 || return 0
     test "$((SECONDS + need))" -ge "$budget" || return 0
     echo "$1 steps left, the last took ${2}s and the budget is ${budget}s; skipping" >&2
     exit 77
 }
 
+# Seconds left of the budget, for a child pacing itself against it (269 hands it to
+# the sweep). Never below 1 unless the guard is off, when it stays 0.
+budget_left() {
+    local budget left
+    budget=$(budget_secs)
+    test "$budget" -gt 0 || {
+        echo 0
+        return 0
+    }
+    left=$((budget - SECONDS))
+    test "$left" -ge 1 || left=1
+    echo "$left"
+}
+
 # Collect a killed job, giving up after REAP_GRACE seconds. kill_tree can fail to
 # reap a native Windows descendant -- the very case these watchdogs exist for --
 # and a bare `wait` then blocks the watchdog itself forever, so the timeout it was

tools/emulated-suite.sh

@@ -48,7 +48,12 @@ cd /bld
 bash "${GITHUB_WORKSPACE:-/src}/configure"
 make -j"$(nproc)"
 # The buildd's own invocation, so a failure here is the one it would report.
-make check -j"$(nproc)"
+rc=0
+make check -j"$(nproc)" || rc=$?
+# Always, not only where automake prints it: this leg exists to say what an emulated
+# host does. A paced-out skip must not read as coverage with no reason given.
+cat tests/test-suite.log || true
+test "$rc" -eq 0 || exit "$rc"
 
 # make check exits 0 for an all-SKIP run, and this leg skips a lot by design, so
 # a container that quietly lost a dependency would report a green covering