ci: alias 127.0.0.2/.3 on macOS lo0 for the connect-fallback test

macOS configures only 127.0.0.1 as loopback, so 19_local-connect-fallback's dead 127.0.0.2/.3 addresses have no fast-refuse path: the connect intermittently stalls to the timeout and logs "Connect Time Out", which the zero-errors assertion rejects (flaked ~1/6 on the macOS runner). #531's --timeout/--retries bound the stall but not the spurious error. Aliasing the addresses onto lo0 makes them refuse instantly like Linux, so the fallback runs cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Un-isolate bigcrawl on macOS and bump it to -c16 (#539 )
2026-07-13 04:09:56 +03:00 · 2026-07-12 12:53:24 +02:00 · 2026-07-12 12:24:42 +02:00 · 2026-07-12 11:38:20 +02:00 · 2026-07-12 09:44:40 +02:00
4 changed files with 20 additions and 13 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -135,16 +135,18 @@ jobs:
      - name: Build
        run: make -j"$(sysctl -n hw.ncpu)"

+      - name: Add loopback aliases (macOS lacks 127.0.0.2/.3)
+        # 19_local-connect-fallback needs the dead 127.0.0.2/.3 to refuse
+        # instantly like Linux; alias them onto lo0 so they don't stall to timeout.
+        run: |
+          set -euo pipefail
+          sudo ifconfig lo0 alias 127.0.0.2 up
+          sudo ifconfig lo0 alias 127.0.0.3 up
+
      - name: Test
-        # bigcrawl's sustained -c8 crawl drops fetches on macOS's loopback when
-        # it competes with other crawls, flaking its exact file count (the #527
-        # macOS drop). Run everything else in parallel, then bigcrawl alone (its
-        # serial-safe condition). Linux tolerates the full parallel run.
        run: |
          jobs=$(( $(sysctl -n hw.ncpu) * 2 )); [ "$jobs" -le 16 ] || jobs=16
-          rest=$(cd tests && ls *.test | grep -v '^36_local-bigcrawl\.test$' | tr '\n' ' ')
-          make check -j"$jobs" TESTS="$rest"
-          make check TESTS=36_local-bigcrawl.test
+          make check -j"$jobs"

      - name: Print the test log on failure
        if: failure()
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -14,10 +14,10 @@ the operational checklist: toolchain, invariants, and how to ship a change.
  automatically; only a test slower than the current longest raises the floor.
  On a few-core Linux box, `-j` at 2x the core count is faster still: the tests
  spend much of their wall time asleep (server trickles, httrack self-pacing),
-  so an idle core covers a sleeping one. CI uses `min(2*cores, 16)`. macOS runs
-  36_local-bigcrawl alone in a second pass: its sustained `-c8` crawl overloads
-  the macOS loopback when it competes with other crawls and flakes its exact
-  file count (Linux tolerates the full parallel run).
+  so an idle core covers a sleeping one. CI uses `min(2*cores, 16)` on every
+  platform, macOS included: the test server raises its listen backlog
+  (`request_queue_size`) so macOS/BSD don't drop connections under a parallel
+  `-c16` bigcrawl the way Python's default backlog of 5 did.
  Or run `sh build.sh` to do bootstrap + configure + make in one shot.

 ## Hard invariants
--- a/tests/36_local-bigcrawl.test
+++ b/tests/36_local-bigcrawl.test
@@ -53,4 +53,4 @@ bash "$top_srcdir/tests/local-crawl.sh" --rerun \
    --log-found ', no files updated' \
    --max-mirror-bytes 700000 \
    --min-mirror-bytes 500000 \
-    httrack 'BASEURL/big/index.html' --retries=0 -c8 -%c100 -A100000000
+    httrack 'BASEURL/big/index.html' --retries=0 -c16 -%c100 -A100000000
--- a/tests/local-server.py
+++ b/tests/local-server.py
@@ -1645,7 +1645,12 @@ def main():
    def factory(*a, **kw):
        return Handler(*a, directory=root, **kw)

-    httpd = ThreadingHTTPServer((args.bind, 0), factory)
+    # macOS/BSD drop SYNs when the listen backlog overflows (Linux is lenient);
+    # raise it from Python's default 5 so a busy -c8 crawl can't lose fetches.
+    class BacklogHTTPServer(ThreadingHTTPServer):
+        request_queue_size = 128
+
+    httpd = BacklogHTTPServer((args.bind, 0), factory)

    if args.tls:
        import ssl
Author	SHA1	Message	Date
Xavier Roche	b571ac8616	ci: alias 127.0.0.2/.3 on macOS lo0 for the connect-fallback test macOS configures only 127.0.0.1 as loopback, so 19_local-connect-fallback's dead 127.0.0.2/.3 addresses have no fast-refuse path: the connect intermittently stalls to the timeout and logs "Connect Time Out", which the zero-errors assertion rejects (flaked ~1/6 on the macOS runner). #531's --timeout/--retries bound the stall but not the spurious error. Aliasing the addresses onto lo0 makes them refuse instantly like Linux, so the fallback runs cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>	2026-07-12 12:53:24 +02:00
Xavier Roche	b5f28f6eb5	Un-isolate bigcrawl on macOS and bump it to -c16 (#539 ) PR #538's larger listen backlog fixed the real cause of the macOS bigcrawl file-count flake: Python's default 5-slot accept queue overflowed under parallel CPU contention, and macOS drops SYNs on overflow. With that gone, macOS no longer needs the second-pass "run bigcrawl alone" workaround, so it runs the full suite in parallel like Linux. The crawl can also push more connections now: -c8 to -c16 shaves ~14% off its wall time (33.4s to 28.6s locally; -%c100 caps the rest) and stresses the engine harder. 128 backlog slots leave ample room for 16 connections. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:24:42 +02:00
Xavier Roche	3d65bb2e3b	Raise the test server's listen backlog so -c8 bigcrawl stops flaking on macOS (#538 ) local-server.py used ThreadingHTTPServer's default request_queue_size of 5. macOS/BSD drop SYNs once the accept queue overflows; Linux is lenient. Under a parallel make check the single-threaded accept loop is starved for CPU, drains the 5-deep queue slowly, and an -c8 bigcrawl burst overflows it. The 36_local-bigcrawl file count then flaked (e.g. 361 -> 283) because --retries=0 turns each dropped connection into a lost file. Raise the backlog to 128 for ample headroom; -c8 is unchanged. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 11:38:20 +02:00
Xavier Roche	68897b09ce	Re-fetch a self-redirect cookie wall instead of dropping it (#537 ) * Re-fetch a self-redirect cookie wall instead of dropping it (#15) A page that 302-redirects to itself only to set a cookie (a consent or session "cookie wall") was lost: the self-redirect guard treated the loop as a crazy server and never re-issued the request, so the real content behind the cookie was never fetched. The redirect's Set-Cookie is already folded into the shared jar before the guard runs, so a re-issue would carry it. In the delayed-type loop (hts_wait_delayed) snapshot the jar before the request; when a self-redirect changed it, evict the cached fast-header for that URL and re-fetch once with the new cookie. Termination is bounded: the jar stops changing once the cookie is satisfied (real walls resolve in two requests), and the existing loops<7 cap backstops a server that mints a fresh cookie on every hit. Under the default HARD delayed-type mode every URL routes through this loop, so both unknown-ext (wall.php) and known-ext (wall.html) walls are covered. A wall reached only under -%N0/1 still takes the hts_mirror_check_moved path, which cannot see the Set-Cookie without an ABI change; left as a known limitation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Test the cookie-wall give-up and retry-cap paths (#15) The happy-path test only drove a wall that sets its cookie on the first hit, so an engine that dropped the jar-changed gate and retried every self-redirect would still pass. Add two cases that pin the gate and the bound: - cookiewall3: a self-redirect that sets no cookie must give up at once (asserts the "loop to same filename" guard fires and the page is not mirrored). A dropped gate would retry and skip that log line. - cookiewall4: a self-redirect that mints a fresh cookie every hit must stop at the loops<7 cap, not spin (asserts the crawl terminates, takes the retry path, and does not mirror the wall). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Scope the cookie-wall retry to this URL's own cookies (#15) The retry trigger compared the whole shared cookie jar before and after the request. Because the jar is one process-wide store mutated by every in-flight fetch, a concurrent slot's Set-Cookie landing mid-loop could flip the compare and force a needless (though bounded) re-fetch of an unrelated self-redirect. Compare instead the Cookie header THIS url would send, built for its own host/path. A Set-Cookie from another host no longer trips the retry; a change in this URL's own outgoing cookies (its self-redirect setting one) still does, which is exactly the wall we want to re-fetch. No jar-wide snapshot, and the same natural termination: once the cookie stabilizes the header stops changing. Reuses append_cookie_header via http_cookie_header (renamed from the _selftest-suffixed wrapper, now that engine code calls it too). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 09:44:40 +02:00