* tests: run the loopback crawl suite on Windows
The ~40 *_local-* tests crawl the bundled Python server: the real TLS
handshake, cache, and file writer, none of which any Windows check covered.
Three things stopped them running there. python3 is python.exe on Windows;
MSYS hands out /d/a/... paths a native python.exe cannot resolve (and arg
rewriting is off, so nothing fixes them up); and Python's text layer turns the
"PORT <n>" discovery line into CRLF, so the \r landed in the parsed port.
Factor the python lookup and the path conversion into tests/testlib.sh, fix the
newline at the source in the three servers, and skip the --file-mode assertion
on Windows, where the engine does not chmod. Behaviour on POSIX is unchanged:
the full suite still passes 97/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: share the server shutdown, and make a skip fail the Windows run
stop_server moves to testlib.sh: the MSYS "a signal cannot reach a native
python.exe, only -9 lands" knowledge was in one of the nine places that kill a
python server. Every step is "|| true" because the callers run under set -e and
reaping a server we just signalled makes wait return 143.
Nothing is expected to skip on Windows, so treat a skip as a failure: the pass
floor alone left slack for exactly the tests that can silently gate themselves
off (TLS, the content codings, socks5, connect-fallback). Add the local proxy
crawl, which the glob was missing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: resolve the server path before backgrounding the server
The port-discovery poll raced the server it had just started: the native path
conversion sat inside the backgrounded command, so the child ran two forks
(command -v cygpath, cygpath) before applying its ">server.log" redirect, and
on Windows the parent's first "head" reached the file first. head then failed,
and under set -e that killed the test silently.
Resolve the paths in the parent, and create the log before the launch so the
first poll cannot lose the race. nativepath keys off the platform, not off
cygpath happening to be on PATH.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: skip the connect-fallback crawl on Windows
19_local-connect-fallback fails there because the engine never falls back to
the next address: Winsock reports a failed connect in select()'s exception set
rather than as writable, and the exception loop fails the slot before
back_connect_next() is reached. Skip it pending the engine fix (#579).
Pin the expected skips instead of counting them, so a gate that silently turns
some other test off still fails the run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: trim the comments added by the Windows port
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: say why the crange resume lost its file
48_local-crange-memresume fails on Windows with a bare "blob.bin missing", and
the crawl log it would have to explain that lives in the tmpdir the test wipes.
Dump the engine's errors and the mirror on that branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: dump the crawl log when the crange resume loses its file
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: check the resume pass survived, don't assume it
48_local-crange-memresume ran pass 2 and printed "terminated" whatever came
back, so a crashed engine read as a clean run. It fails on Windows with an
empty log and an empty mirror, which is what that blind spot looks like.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: dump both passes' state for the Windows crange failure
Temporary: pass-1 mirror + log and pass-2 log/mirror, to see why the resume
mirrors and logs nothing on Windows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: skip the crange memory-resume crawl on Windows
48_local-crange-memresume needs a graceful pass-1 interrupt so the cache is
clean when pass 2 resumes; MSYS can only hard-kill a native exe, and the
engine's restart-whole path after an unusable 206 then fails on the repaired
cache (#581). Skip on Windows pending that fix.
Keep the pass-2 exit-code check the investigation added: the test printed
"terminated" whatever came back, so a crashed engine read as a clean run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the CONNECT-tunnel change, from an adversarial review (the proxy
response is hostile input: a malicious or MITM proxy controls every byte).
- Bound the response read so a proxy cannot stall the single-threaded back_wait
crawl: proxy_getline now fails on an over-long line instead of consuming it
forever, the header drain is capped at 64 lines, and the send loop gives up
rather than spin against a socket that reports writable but never accepts.
- Size `authority` to hold any url_adr host (HTS_URLMAXSIZE*2) so an oversized
hostname can't trip the abort-on-overflow buff helpers; grow `req` to match.
- Reject control bytes in the CONNECT authority as a local backstop; today the
CR/LF defense lives entirely upstream (escape_remove_control / header-line
splitting).
- Test: the origin now records the headers it receives, and the test asserts
Proxy-Authorization never reaches the origin through the tunnel (the previous
assertions couldn't see a leak). Added a flooding-proxy scenario that proves
the crawl terminates instead of hanging on an unbounded response.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
httrack opened https connections straight to the origin even when a proxy
was configured, so --proxy was silently ignored for https and the crawler
used the real IP. http_xfopen bypassed the proxy for any https:// URL,
because the absolute-URI proxy form it uses for http cannot carry https.
Connect to the proxy instead and, once the TCP connection is up, open an
HTTP CONNECT tunnel (http_proxy_tunnel) before the TLS handshake, so TLS
runs end-to-end with the origin. Proxy credentials now ride the CONNECT
request rather than the tunneled GET, where they would leak to the origin.
The exchange is a bounded blocking read inside the back_wait connect path:
no new async state, no struct/ABI change (the helpers stay visibility-hidden).
Verified end-to-end by 13_crawl_proxy_https.test: it crawls a local
self-signed https origin through a logging CONNECT proxy and asserts the
proxy saw the CONNECT and that credentials ride it. The assertion fails on
the pre-fix bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
