Un-isolate bigcrawl on macOS and bump it to -c16

PR #538's larger listen backlog fixed the real cause of the macOS bigcrawl
file-count flake: Python's default 5-slot accept queue overflowed under parallel
CPU contention, and macOS drops SYNs on overflow. With that gone, macOS no
longer needs the second-pass "run bigcrawl alone" workaround, so it runs the
full suite in parallel like Linux.

The crawl can also push more connections now: -c8 to -c16 shaves ~14% off its
wall time (33.4s to 28.6s locally; -%c100 caps the rest) and stresses the engine
harder. 128 backlog slots leave ample room for 16 connections.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/workflows/ci.yml

@@ -136,15 +136,9 @@ jobs:
         run: make -j"$(sysctl -n hw.ncpu)"
 
       - name: Test
-        # bigcrawl's sustained -c8 crawl drops fetches on macOS's loopback when
-        # it competes with other crawls, flaking its exact file count (the #527
-        # macOS drop). Run everything else in parallel, then bigcrawl alone (its
-        # serial-safe condition). Linux tolerates the full parallel run.
         run: |
           jobs=$(( $(sysctl -n hw.ncpu) * 2 )); [ "$jobs" -le 16 ] || jobs=16
-          rest=$(cd tests && ls *.test | grep -v '^36_local-bigcrawl\.test$' | tr '\n' ' ')
-          make check -j"$jobs" TESTS="$rest"
-          make check TESTS=36_local-bigcrawl.test
+          make check -j"$jobs"
 
       - name: Print the test log on failure
         if: failure()

AGENTS.md

@@ -14,10 +14,10 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   automatically; only a test slower than the current longest raises the floor.
   On a few-core Linux box, `-j` at 2x the core count is faster still: the tests
   spend much of their wall time asleep (server trickles, httrack self-pacing),
-  so an idle core covers a sleeping one. CI uses `min(2*cores, 16)`. macOS runs
-  36_local-bigcrawl alone in a second pass: its sustained `-c8` crawl overloads
-  the macOS loopback when it competes with other crawls and flakes its exact
-  file count (Linux tolerates the full parallel run).
+  so an idle core covers a sleeping one. CI uses `min(2*cores, 16)` on every
+  platform, macOS included: the test server raises its listen backlog
+  (`request_queue_size`) so macOS/BSD don't drop connections under a parallel
+  `-c16` bigcrawl the way Python's default backlog of 5 did.
   Or run `sh build.sh` to do bootstrap + configure + make in one shot.
 
 ## Hard invariants

tests/36_local-bigcrawl.test

@@ -53,4 +53,4 @@ bash "$top_srcdir/tests/local-crawl.sh" --rerun \
     --log-found ', no files updated' \
     --max-mirror-bytes 700000 \
     --min-mirror-bytes 500000 \
-    httrack 'BASEURL/big/index.html' --retries=0 -c8 -%c100 -A100000000
+    httrack 'BASEURL/big/index.html' --retries=0 -c16 -%c100 -A100000000