ci: harden the webhttrack smoke and drop the leaky watchdog

Review found two issues. The content check matched only the HTTrack brand
string, which is a literal in every page header, so a truncated/degraded
template served 200 would still pass; also require the step-2 form action.
The hard watchdog subshell was un-redirected, so its orphaned sleep kept the
CI step stdout open ~35s after every run; the poll loop plus teardown already
bound the run, so drop the watchdog (SIGKILL webhttrack if it ignores TERM).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/workflows/ci.yml

@@ -135,23 +135,50 @@ jobs:
       - name: Build
         run: make -j"$(sysctl -n hw.ncpu)"
 
-      - name: Add loopback aliases (macOS lacks 127.0.0.2/.3)
-        # 19_local-connect-fallback needs the dead 127.0.0.2/.3 to refuse
-        # instantly like Linux; alias them onto lo0 so they don't stall to timeout.
-        run: |
-          set -euo pipefail
-          sudo ifconfig lo0 alias 127.0.0.2 up
-          sudo ifconfig lo0 alias 127.0.0.3 up
-
       - name: Test
+        # bigcrawl's sustained -c8 crawl drops fetches on macOS's loopback when
+        # it competes with other crawls, flaking its exact file count (the #527
+        # macOS drop). Run everything else in parallel, then bigcrawl alone (its
+        # serial-safe condition). Linux tolerates the full parallel run.
         run: |
           jobs=$(( $(sysctl -n hw.ncpu) * 2 )); [ "$jobs" -le 16 ] || jobs=16
-          make check -j"$jobs"
+          rest=$(cd tests && ls *.test | grep -v '^36_local-bigcrawl\.test$' | tr '\n' ' ')
+          make check -j"$jobs" TESTS="$rest"
+          make check TESTS=36_local-bigcrawl.test
 
       - name: Print the test log on failure
         if: failure()
         run: cat tests/test-suite.log 2>/dev/null || true
 
+  # Runtime smoke of the WebHTTrack launcher on macOS: it carries a Darwin-only
+  # browser path (open -W) and nothing else exercises htsserver. Install into a
+  # temp prefix, then check webhttrack brings up htsserver and serves the UI.
+  webhttrack-macos:
+    name: webhttrack smoke (macOS arm64)
+    runs-on: macos-14
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          submodules: recursive
+
+      - name: Install build dependencies
+        run: |
+          set -euo pipefail
+          brew install autoconf automake libtool autoconf-archive
+
+      - name: Build and install into a temp prefix
+        run: |
+          set -euo pipefail
+          ssl="$(brew --prefix openssl@3)"
+          autoreconf -fi
+          ./configure CPPFLAGS="-I${ssl}/include" LDFLAGS="-L${ssl}/lib" \
+            --prefix="$RUNNER_TEMP/inst"
+          make -j"$(sysctl -n hw.ncpu)"
+          make install
+
+      - name: Smoke-test webhttrack
+        run: bash tests/webhttrack-smoke.sh "$RUNNER_TEMP/inst"
+
   # Portability/hardening: 32-bit (i386) build on the x86-64 runner via multilib
   # -- no extra hardware. Exercises the 32-bit size_t/pointer ABI, where size
   # and bounds math can truncate or wrap in ways 64-bit never reveals (the axis

AGENTS.md

@@ -14,10 +14,10 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   automatically; only a test slower than the current longest raises the floor.
   On a few-core Linux box, `-j` at 2x the core count is faster still: the tests
   spend much of their wall time asleep (server trickles, httrack self-pacing),
-  so an idle core covers a sleeping one. CI uses `min(2*cores, 16)` on every
-  platform, macOS included: the test server raises its listen backlog
-  (`request_queue_size`) so macOS/BSD don't drop connections under a parallel
-  `-c16` bigcrawl the way Python's default backlog of 5 did.
+  so an idle core covers a sleeping one. CI uses `min(2*cores, 16)`. macOS runs
+  36_local-bigcrawl alone in a second pass: its sustained `-c8` crawl overloads
+  the macOS loopback when it competes with other crawls and flakes its exact
+  file count (Linux tolerates the full parallel run).
   Or run `sh build.sh` to do bootstrap + configure + make in one shot.
 
 ## Hard invariants

tests/36_local-bigcrawl.test

@@ -53,4 +53,4 @@ bash "$top_srcdir/tests/local-crawl.sh" --rerun \
     --log-found ', no files updated' \
     --max-mirror-bytes 700000 \
     --min-mirror-bytes 500000 \
-    httrack 'BASEURL/big/index.html' --retries=0 -c16 -%c100 -A100000000
+    httrack 'BASEURL/big/index.html' --retries=0 -c8 -%c100 -A100000000

tests/webhttrack-smoke.sh

@@ -0,0 +1,99 @@
+#!/bin/bash
+# Smoke-test an installed webhttrack: launch it with a stub browser and assert
+# htsserver comes up and serves the web UI. Arg: the install prefix.
+set -euo pipefail
+
+prefix="${1:?usage: webhttrack-smoke.sh <install-prefix>}"
+wht="$prefix/bin/webhttrack"
+test -x "$wht" || {
+    echo "no webhttrack at $wht" >&2
+    exit 1
+}
+
+work="$(mktemp -d)"
+# webhttrack backgrounds htsserver, which outlives it; reap any stray one (scoped
+# to this prefix) so a lingering server can never hold the CI step open.
+trap 'pkill -f "$prefix/bin/htsserver" 2>/dev/null || true; rm -rf "$work"' EXIT
+export HOME="$work/home"
+mkdir -p "$HOME/websites"
+marker="$work/marker"
+
+stubdir="$work/bin"
+mkdir -p "$stubdir"
+
+# On Darwin webhttrack hardcodes "open -W", which launches a real GUI browser and
+# blocks headless. Shadow uname so it takes the generic path and picks the stub
+# browser below; htsserver and webhttrack's path resolution still run for real.
+cat >"$stubdir/uname" <<'EOF'
+#!/bin/bash
+[ "${1:-}" = "-s" ] && {
+    echo Linux
+    exit 0
+}
+exec /usr/bin/uname "$@"
+EOF
+chmod +x "$stubdir/uname"
+
+# Stub browser: webhttrack tries its browser-name list in order and runs the
+# first it finds, so shadow the first entry, "x-www-browser". It fetches the
+# server URL and records PASS only for the working UI: the brand string AND the
+# step-2 form action, which a truncated/degraded template page would lack (the
+# bare title alone is not enough). htsserver only lives until webhttrack exits,
+# so the check has to happen here.
+# -a: the UI is served ISO-8859-1, so grep must not treat it as binary.
+cat >"$stubdir/x-www-browser" <<EOF
+#!/bin/bash
+echo "stub browser invoked with: \$1" >&2
+if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html; then
+    echo PASS >"$marker"
+else
+    echo "FAIL: unexpected response from \$1" >"$marker"
+fi
+EOF
+chmod +x "$stubdir/x-www-browser"
+export PATH="$stubdir:$prefix/bin:$PATH"
+
+echo "launching webhttrack"
+"$wht" </dev/null >"$work/webhttrack.log" 2>&1 &
+whpid=$!
+
+# Bounded poll for the marker (macOS has no timeout(1)); teardown below kills
+# webhttrack and reaps htsserver, so the run is bounded without a watchdog.
+for i in $(seq 1 45); do
+    test -f "$marker" && {
+        echo "marker written after ${i}s"
+        break
+    }
+    kill -0 "$whpid" 2>/dev/null || {
+        echo "webhttrack exited on its own after ${i}s"
+        break
+    }
+    sleep 1
+done
+
+# Reap webhttrack and the htsserver it spawned. Confirm death with a bounded poll
+# (not a blocking wait, which could hang on macOS); SIGKILL if it ignores TERM.
+echo "tearing down"
+kill "$whpid" 2>/dev/null || true
+if pkill -f "$prefix/bin/htsserver" 2>/dev/null; then
+    echo "reaped a lingering htsserver"
+else
+    echo "no lingering htsserver"
+fi
+for _ in $(seq 1 10); do
+    kill -0 "$whpid" 2>/dev/null || break
+    sleep 1
+done
+kill -9 "$whpid" 2>/dev/null || true
+
+echo "--- webhttrack.log ---"
+cat "$work/webhttrack.log" 2>/dev/null || true
+echo "--- end ---"
+echo "marker=[$(cat "$marker" 2>/dev/null || echo NONE)]"
+
+if test "$(cat "$marker" 2>/dev/null || true)" = PASS; then
+    echo "webhttrack smoke: PASS"
+else
+    echo "webhttrack smoke: FAIL" >&2
+    exit 1
+fi