mirror of
https://github.com/xroche/httrack.git
synced 2026-07-23 01:01:41 +03:00
1fa0184468
* Tunnel crawls through a SOCKS5 proxy (RFC 1928/1929) -P socks5://[user:pass@]host[:port] now negotiates a SOCKS5 tunnel to the origin instead of speaking HTTP to the proxy. The handshake runs on the raw socket right after connect, so it covers plain http as well as https (TLS then runs end-to-end over the tunnel, no CONNECT); the origin request stays origin-form, and the proxy credentials ride the RFC 1929 sub-negotiation rather than a Proxy-Authorization header. The origin hostname is sent as-is (ATYP=domain): the proxy resolves it, like curl's socks5h, which is what makes .onion and split-horizon names work and keeps the crawl from leaking DNS. ftp:// links fail under a socks proxy rather than silently dial the origin direct. Reply frames are drained exactly per ATYP, since the socket is shared with the origin stream: one byte too few and the leftovers corrupt the first TLS record. -#test=socks5 asserts that byte count against scripted replies, plus the truncated/hostile cases; 52_local-socks5.test crawls http and https through a local SOCKS5 server that only answers for a .invalid name, so a locally-resolved origin could not reach it. Closes #563 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Don't re-run the SOCKS5 handshake on a reused keep-alive socket back_wait re-enters the connect-completed block for a keep-alive socket revived by back_trylive (plain http; https is covered by the ssl_con guard). The socket is already tunneled, so re-sending the SOCKS greeting injected \x05\x01\x00... into the established origin stream and corrupted every request after the first. Gate the handshake on !keep_alive, which back_connxfr carries onto the revived slot and which is zero on a fresh connect. 52_local-socks5.test now crawls a keep-alive origin (advertising max>1) with -c1: the whole plain-http crawl rides one tunnel, exactly one handshake, every subpage intact. Without the gate the reused stream desyncs and the crawl hangs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Use scheme-less origins in the SOCKS5 self-test (CodeQL cpp/non-https-url) The scripted handshake strips the scheme and takes the TLS flag as a parameter, so the "http://origin.test" fixtures were inert cleartext-URL literals that tripped CodeQL. Pass the bare authority instead; the parsing is identical and the real scheme handling is covered end-to-end by 52_local-socks5.test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
10 lines
251 B
Bash
10 lines
251 B
Bash
#!/bin/bash
|
|
#
|
|
|
|
set -euo pipefail
|
|
|
|
# The SOCKS5 handshake framing and credential split, driven against scripted
|
|
# server replies (frame draining, oversize rejects, RFC 1929 fields).
|
|
|
|
httrack -O /dev/null '-#test=socks5' | grep -q "socks5 self-test OK"
|