mirror of
https://github.com/xroche/httrack.git
synced 2026-07-17 06:10:42 +03:00
Tunnel plain HTTP through a CONNECT-only proxy (connect:// scheme)
HTTPS already reaches the origin through a CONNECT tunnel (#85), but plain HTTP still used the absolute-URI request form, which a CONNECT-only proxy like Tor's HTTPTunnelPort rejects. Adds a `connect://` proxy scheme (`-P connect://host:port`) that drives the proxy with CONNECT for every request: httrack opens a tunnel to the origin (port 80 for http, 443 for https), then sends the request origin-form with no Proxy-Authorization, the same way https does. It can't be the default (CONNECT-to-:80 breaks proxies that only allow CONNECT on 443) and auto-detection is ruled out, so the mode rides the proxy scheme like socks5:// does. That keeps it ABI- and cache-safe (the scheme lives in proxy.name, no new struct field) and drops into the webhttrack GUI as one more entry beside HTTP and SOCKS5. The tooltip was reworded and retranslated across the lang/*.txt files. Test 57 fronts a plain-http origin with a CONNECT-only proxy (it 501s anything but CONNECT) and checks the request arrives origin-form, that dropping connect:// gets the classic form refused, that credentials ride the CONNECT and never reach the origin, that a portless origin tunnels to :80, and that a flooding proxy does not hang. Closes #564.
This commit is contained in:
Xavier Roche
@@ -103,6 +103,7 @@ ${LANG_PROXYTYPE}
|
||||
title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info(' '); return true"
|
||||
>
|
||||
<option value=""${ztest:proxytype: selected:}>HTTP</option>
|
||||
<option value="2"${test:proxytype::: selected}>HTTP (CONNECT tunnel)</option>
|
||||
<option value="1"${test:proxytype:: selected}>SOCKS5</option>
|
||||
</select>
|
||||
<br><br>
|
||||
|
||||
@@ -192,7 +192,7 @@ ${do:end-if}
|
||||
${test:logtype:::--extra-log:--debug-log}
|
||||
${test:index:--index=0:}
|
||||
${test:index2:--search-index=0:--search-index}
|
||||
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype:socks5\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
|
||||
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
|
||||
${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
|
||||
</textarea>
|
||||
|
||||
|
||||
2
lang.def
2
lang.def
@@ -1009,7 +1009,7 @@ A fatal error has occurred during this mirror
|
||||
LANG_PROXYTYPE
|
||||
Proxy type:
|
||||
LANG_PROXYTYPETIP
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
LANG_COOKIEFILE
|
||||
Load cookies from file:
|
||||
LANG_COOKIEFILETIP
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Ôàòàëíà ãðåøêà ïðè ñúçäàâàíåòî íà òîçè îãëåäàëåí ñàéò
|
||||
Proxy type:
|
||||
Òèï íà ïðîêñè:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Ïðîòîêîë íà ïðîêñè: HTTP èëè SOCKS5 (ñòàíäàðòíèÿò ïîðò íà SOCKS5 å 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Ïðîòîêîë íà ïðîêñè. HTTP: ñòàíäàðòíî ïðîêñè. HTTP (CONNECT òóíåë): èçïðàùà âñÿêà çàÿâêà ïðåç CONNECT òóíåë, çà ïðîêñè ïîääúðæàùè ñàìî CONNECT êàòî HTTPTunnelPort íà Tor. SOCKS5: ïîðò ïî ïîäðàçáèðàíå 1080.
|
||||
Load cookies from file:
|
||||
Çàðåæäàíå íà 'áèñêâèòêè' îò ôàéë:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Ha ocurrido un error fatal durante esta copia
|
||||
Proxy type:
|
||||
Tipo de proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protocolo del proxy: HTTP o SOCKS5 (el puerto SOCKS5 predeterminado es 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protocolo del proxy. HTTP: proxy estándar. HTTP (túnel CONNECT): envía cada petición a través de un túnel CONNECT, para proxys que solo admiten CONNECT como el HTTPTunnelPort de Tor. SOCKS5: puerto predeterminado 1080.
|
||||
Load cookies from file:
|
||||
Cargar cookies desde un archivo:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Typ proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protokol proxy: HTTP nebo SOCKS5 (výchozí port SOCKS5 je 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protokol proxy. HTTP: standardní proxy. HTTP (tunel CONNECT): odešle každý požadavek pøes tunel CONNECT, pro proxy podporující jen CONNECT jako HTTPTunnelPort v Toru. SOCKS5: výchozí port 1080.
|
||||
Load cookies from file:
|
||||
Naèíst cookies ze souboru:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
這鏡像發生了不可回復的錯誤
|
||||
Proxy type:
|
||||
proxy 類型:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
proxy 協定: HTTP 或 SOCKS5 (SOCKS5 的預設連接埠為 1080)。
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
代理協定。HTTP：標準代理。HTTP（CONNECT 隧道）：透過 CONNECT 隧道傳送每個請求，用於僅支援 CONNECT 的代理，例如 Tor 的 HTTPTunnelPort。SOCKS5：預設連接埠 1080。
|
||||
Load cookies from file:
|
||||
從檔案載入 cookies:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
代理类型:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
代理协议: HTTP 或 SOCKS5 (SOCKS5 的默认端口为 1080)。
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
代理协议。HTTP：标准代理。HTTP（CONNECT 隧道）：通过 CONNECT 隧道发送每个请求，用于仅支持 CONNECT 的代理，例如 Tor 的 HTTPTunnelPort。SOCKS5：默认端口 1080。
|
||||
Load cookies from file:
|
||||
从文件加载 cookies:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -932,8 +932,8 @@ A fatal error has occurred during this mirror
|
||||
Tijekom ovog zrcaljenja je nastala fatalna pogreška
|
||||
Proxy type:
|
||||
Vrsta posrednika:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protokol posrednika: HTTP ili SOCKS5 (zadani port za SOCKS5 je 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protokol posrednika. HTTP: standardni posrednik. HTTP (CONNECT tunel): ¹alje svaki zahtjev kroz CONNECT tunel, za posrednike koji podr¾avaju samo CONNECT poput Torovog HTTPTunnelPorta. SOCKS5: zadani port 1080.
|
||||
Load cookies from file:
|
||||
Uèitati kolaèiæe iz datoteke:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -978,8 +978,8 @@ HTTrack: could not save profile for '%s'!
|
||||
HTTrack: kunne ikke gemme profil for '%s'!
|
||||
Proxy type:
|
||||
Proxytype:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxyprotokol: HTTP eller SOCKS5 (SOCKS5 bruger som standard port 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxyprotokol. HTTP: standardproxy. HTTP (CONNECT-tunnel): sender hver forespørgsel gennem en CONNECT-tunnel, til proxyer der kun understøtter CONNECT som Tors HTTPTunnelPort. SOCKS5: standardport 1080.
|
||||
Load cookies from file:
|
||||
Indlæs cookies fra fil:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Fataler Fehler während der Webseiten-Kopie
|
||||
Proxy type:
|
||||
Proxy-Typ:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy-Protokoll: HTTP oder SOCKS5 (Standardport von SOCKS5 ist 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxy-Protokoll. HTTP: normaler Proxy. HTTP (CONNECT-Tunnel): sendet jede Anfrage durch einen CONNECT-Tunnel, für Proxys, die nur CONNECT unterstützen, wie Tors HTTPTunnelPort. SOCKS5: Standardport 1080.
|
||||
Load cookies from file:
|
||||
Cookies aus Datei laden:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Proxy tüüp:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy protokoll: HTTP või SOCKS5 (SOCKS5 vaikeport on 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Puhverserveri protokoll. HTTP: tavaline puhverserver. HTTP (CONNECT-tunnel): saadab iga päringu läbi CONNECT-tunneli, ainult CONNECT-it toetavate puhverserverite jaoks nagu Tori HTTPTunnelPort. SOCKS5: vaikeport 1080.
|
||||
Load cookies from file:
|
||||
Lae küpsised failist:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -978,8 +978,8 @@ HTTrack: could not save profile for '%s'!
|
||||
HTTrack: could not save profile for '%s'!
|
||||
Proxy type:
|
||||
Proxy type:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Load cookies from file:
|
||||
Load cookies from file:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -933,8 +933,8 @@ T
|
||||
|
||||
Proxy type:
|
||||
Välityspalvelimen tyyppi:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Välityspalvelimen protokolla: HTTP tai SOCKS5 (SOCKS5:n oletusportti on 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Välityspalvelimen protokolla. HTTP: tavallinen välityspalvelin. HTTP (CONNECT-tunneli): lähettää jokaisen pyynnön CONNECT-tunnelin kautta, vain CONNECT-yhteyksiä tukeville välityspalvelimille kuten Torin HTTPTunnelPort. SOCKS5: oletusportti 1080.
|
||||
Load cookies from file:
|
||||
Lataa evästeet tiedostosta:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -978,8 +978,8 @@ Default referer URL
|
||||
Champ referer par défaut
|
||||
Proxy type:
|
||||
Type de proxy :
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protocole du proxy : HTTP ou SOCKS5 (le port SOCKS5 par défaut est 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protocole du proxy. HTTP : proxy standard. HTTP (tunnel CONNECT) : envoie chaque requête via un tunnel CONNECT, pour les proxys acceptant uniquement CONNECT comme le HTTPTunnelPort de Tor. SOCKS5 : port par défaut 1080.
|
||||
Load cookies from file:
|
||||
Charger les cookies depuis un fichier :
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -932,8 +932,8 @@ A fatal error has occurred during this mirror
|
||||
Ένα καταστροφικό σφάλμα προκλήθηκε κατά την αντιγραφή αυτού του τόπου
|
||||
Proxy type:
|
||||
Τύπος proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Πρωτόκολλο proxy: HTTP ή SOCKS5 (η προεπιλεγμένη θύρα του SOCKS5 είναι 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Πρωτόκολλο διαμεσολαβητή. HTTP: τυπικός διαμεσολαβητής. HTTP (σήραγγα CONNECT): στέλνει κάθε αίτημα μέσω σήραγγας CONNECT, για διαμεσολαβητές που υποστηρίζουν μόνο CONNECT όπως το HTTPTunnelPort του Tor. SOCKS5: προεπιλεγμένη θύρα 1080.
|
||||
Load cookies from file:
|
||||
Φόρτωση cookies από αρχείο:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Si è verificato un errore fatale durante la copia
|
||||
Proxy type:
|
||||
Tipo di proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protocollo del proxy: HTTP o SOCKS5 (la porta SOCKS5 predefinita è 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protocollo del proxy. HTTP: proxy standard. HTTP (tunnel CONNECT): invia ogni richiesta attraverso un tunnel CONNECT, per proxy che supportano solo CONNECT come l'HTTPTunnelPort di Tor. SOCKS5: porta predefinita 1080.
|
||||
Load cookies from file:
|
||||
Carica i cookie da un file:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
プロキシの種類:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
プロキシのプロトコル: HTTP または SOCKS5 (SOCKS5 の既定ポートは 1080)。
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
プロキシのプロトコル。HTTP: 標準のプロキシ。HTTP (CONNECT トンネル): すべてのリクエストを CONNECT トンネル経由で送信します。Tor の HTTPTunnelPort のような CONNECT のみのプロキシ用です。SOCKS5: 既定ポート 1080。
|
||||
Load cookies from file:
|
||||
クッキーをファイルから読み込む:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Íàñòàíà ôàòàëíà ãðåøêà ïðè îâî¼ mirror
|
||||
Proxy type:
|
||||
ÂØß ÝÐ ßàÞÚáØ:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
¿àÞâÞÚÞÛ ÝÐ ßàÞÚáØ: HTTP ØÛØ SOCKS5 (áâÐÝÔÐàÔÝÐâÐ ßÞàâÐ ÝÐ SOCKS5 Õ 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
¿àÞâÞÚÞÛ ÝÐ ßàÞÚáØ. HTTP: áâÐÝÔÐàÔÕÝ ßàÞÚáØ. HTTP (âãÝÕÛ CONNECT): ÓÞ ØáßàÐüÐ áÕÚÞÕ ÑÐàÐúÕ ßàÕÚã âãÝÕÛ CONNECT, ×Ð ßàÞÚáØ èâÞ ßÞÔÔàÖãÒÐÐâ áÐÜÞ CONNECT ÚÐÚÞ HTTPTunnelPort ÝÐ Tor. SOCKS5: áâÐÝÔÐàÔÝÐ ßÞàâÐ 1080.
|
||||
Load cookies from file:
|
||||
²çØâÐø ÚÞÛÐçØúÐ ÞÔ ÔÐâÞâÕÚÐ:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Végzetes hiba történt a tükrözés közben
|
||||
Proxy type:
|
||||
Proxy típusa:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy protokoll: HTTP vagy SOCKS5 (a SOCKS5 alapértelmezett portja 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxy protokoll. HTTP: szabványos proxy. HTTP (CONNECT alagút): minden kérést CONNECT alagúton keresztül küld, a csak CONNECT-et támogató proxykhoz, mint a Tor HTTPTunnelPortja. SOCKS5: alapértelmezett port 1080.
|
||||
Load cookies from file:
|
||||
Sütik betöltése fájlból:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Een fatale fout is opgetreden tijdens deze spiegeling
|
||||
Proxy type:
|
||||
Proxytype:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxyprotocol: HTTP of SOCKS5 (standaardpoort van SOCKS5 is 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxyprotocol. HTTP: standaardproxy. HTTP (CONNECT-tunnel): stuurt elke aanvraag via een CONNECT-tunnel, voor proxy's die alleen CONNECT ondersteunen zoals Tors HTTPTunnelPort. SOCKS5: standaardpoort 1080.
|
||||
Load cookies from file:
|
||||
Cookies uit bestand laden:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Proxytype:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxyprotokoll: HTTP eller SOCKS5 (SOCKS5 bruker som standard port 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxyprotokoll. HTTP: standard proxy. HTTP (CONNECT-tunnel): sender hver forespørsel gjennom en CONNECT-tunnel, for proxyer som bare støtter CONNECT som Tors HTTPTunnelPort. SOCKS5: standardport 1080.
|
||||
Load cookies from file:
|
||||
Last inn cookies fra fil:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Podczas tworzenia lustra wydarzyl sie fatalny blad.
|
||||
Proxy type:
|
||||
Typ proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protokó³ proxy: HTTP lub SOCKS5 (domy¶lny port SOCKS5 to 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protokó³ proxy. HTTP: standardowe proxy. HTTP (tunel CONNECT): wysy³a ka¿de ¿±danie przez tunel CONNECT, dla proxy obs³uguj±cych tylko CONNECT, jak HTTPTunnelPort w Torze. SOCKS5: port domy¶lny 1080.
|
||||
Load cookies from file:
|
||||
Wczytaj ciasteczka z pliku:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -978,8 +978,8 @@ HTTrack: could not save profile for '%s'!
|
||||
HTTrack: não foi possível salvar o perfil para '%s'!
|
||||
Proxy type:
|
||||
Tipo de proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protocolo do proxy: HTTP ou SOCKS5 (a porta SOCKS5 padrão é 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protocolo do proxy. HTTP: proxy padrão. HTTP (túnel CONNECT): envia cada requisição através de um túnel CONNECT, para proxies que só suportam CONNECT como o HTTPTunnelPort do Tor. SOCKS5: porta padrão 1080.
|
||||
Load cookies from file:
|
||||
Carregar cookies de um arquivo:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Tipo de proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protocolo do proxy: HTTP ou SOCKS5 (a porta SOCKS5 predefinida é 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protocolo do proxy. HTTP: proxy padrão. HTTP (túnel CONNECT): envia cada pedido através de um túnel CONNECT, para proxies que só suportam CONNECT como o HTTPTunnelPort do Tor. SOCKS5: porta predefinida 1080.
|
||||
Load cookies from file:
|
||||
Carregar cookies de um ficheiro:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
A survenit o eroare fatală în timpul acestei clonări.
|
||||
Proxy type:
|
||||
Tip proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protocol proxy: HTTP sau SOCKS5 (portul SOCKS5 implicit este 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protocol proxy. HTTP: proxy standard. HTTP (tunel CONNECT): trimite fiecare cerere printr-un tunel CONNECT, pentru proxy care accepta doar CONNECT precum HTTPTunnelPort al lui Tor. SOCKS5: port implicit 1080.
|
||||
Load cookies from file:
|
||||
Încarca cookies dintr-un fisier:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Âî âðåìÿ òåêóùåé çàêà÷êè ïðîèçîøëà ôàòàëüíàÿ îøèáêà
|
||||
Proxy type:
|
||||
Òèï ïðîêñè:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Ïðîòîêîë ïðîêñè: HTTP èëè SOCKS5 (ïî óìîë÷àíèþ SOCKS5 èñïîëüçóåò ïîðò 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Ïðîòîêîë ïðîêñè. HTTP: îáû÷íûé ïðîêñè. HTTP (òóííåëü CONNECT): îòïðàâëÿåò êàæäûé çàïðîñ ÷åðåç òóííåëü CONNECT, äëÿ ïðîêñè, ïîääåðæèâàþùèõ òîëüêî CONNECT, íàïðèìåð HTTPTunnelPort â Tor. SOCKS5: ïîðò ïî óìîë÷àíèþ 1080.
|
||||
Load cookies from file:
|
||||
Çàãðóçèòü cookies èç ôàéëà:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Typ proxy:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protokol proxy: HTTP alebo SOCKS5 (predvolený port SOCKS5 je 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protokol proxy. HTTP: ¹tandardné proxy. HTTP (tunel CONNECT): odo¹le ka¾dú po¾iadavku cez tunel CONNECT, pre proxy podporujúce len CONNECT ako HTTPTunnelPort v Tore. SOCKS5: predvolený port 1080.
|
||||
Load cookies from file:
|
||||
Naèíta» cookies zo súboru:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Vrsta proxyja:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Protokol proxy: HTTP ali SOCKS5 (privzeta vrata SOCKS5 so 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Protokol proxy. HTTP: standardni proxy. HTTP (tunel CONNECT): poslje vsako zahtevo skozi tunel CONNECT, za posrednike, ki podpirajo samo CONNECT, kot je HTTPTunnelPort v Toru. SOCKS5: privzeta vrata 1080.
|
||||
Load cookies from file:
|
||||
Nalozi piskotke iz datoteke:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Proxytyp:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxyprotokoll: HTTP eller SOCKS5 (SOCKS5 använder som standard port 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxyprotokoll. HTTP: standardproxy. HTTP (CONNECT-tunnel): skickar varje begäran genom en CONNECT-tunnel, för proxyservrar som bara stöder CONNECT som Tors HTTPTunnelPort. SOCKS5: standardport 1080.
|
||||
Load cookies from file:
|
||||
Läs in cookies från fil:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Bu yansýlama iþlemi sýrasýnda ölümcül bir hata oluþtu
|
||||
Proxy type:
|
||||
Proxy türü:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proxy protokolü: HTTP veya SOCKS5 (SOCKS5 varsayýlan portu 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proxy protokolü. HTTP: standart proxy. HTTP (CONNECT tüneli): her isteði bir CONNECT tüneli üzerinden gönderir; Tor'un HTTPTunnelPort gibi yalnýzca CONNECT destekleyen proxy'ler için. SOCKS5: varsayýlan baðlantý noktasý 1080.
|
||||
Load cookies from file:
|
||||
Çerezleri dosyadan yükle:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
|
||||
Proxy type:
|
||||
Òèï ïðîêñ³:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Ïðîòîêîë ïðîêñ³: HTTP àáî SOCKS5 (çà çàìîâ÷óâàííÿì SOCKS5 âèêîðèñòîâóº ïîðò 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Ïðîòîêîë ïðîêñ³. HTTP: ñòàíäàðòíèé ïðîêñ³. HTTP (òóíåëü CONNECT): íàäñèëຠêîæåí çàïèò ÷åðåç òóíåëü CONNECT, äëÿ ïðîêñ³, ùî ï³äòðèìóþòü ëèøå CONNECT, ÿê-îò HTTPTunnelPort ó Tor. SOCKS5: ïîðò çà çàìîâ÷óâàííÿì 1080.
|
||||
Load cookies from file:
|
||||
Çàâàíòàæèòè cookies ç ôàéëó:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -930,8 +930,8 @@ A fatal error has occurred during this mirror
|
||||
Joriy ko’chirish vaqtida jiddiy xatolik yuz berdi
|
||||
Proxy type:
|
||||
Proksi turi:
|
||||
Proxy protocol: HTTP, or SOCKS5 (SOCKS5 default port is 1080).
|
||||
Proksi protokoli: HTTP yoki SOCKS5 (SOCKS5 ning standart porti 1080).
|
||||
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
|
||||
Proksi protokoli. HTTP: oddiy proksi. HTTP (CONNECT tuneli): har bir so'rovni CONNECT tuneli orqali yuboradi; faqat CONNECT'ni qo'llab-quvvatlaydigan proksilar uchun, masalan Tor HTTPTunnelPort. SOCKS5: standart port 1080.
|
||||
Load cookies from file:
|
||||
Fayldan cookies yuklamoq:
|
||||
Preload cookies from a Netscape cookies.txt file before crawling.
|
||||
|
||||
@@ -135,7 +135,7 @@ continue an interrupted mirror using the cache (\-\-continue)
|
||||
mirror ALL links located in the first level pages (mirror links) (\-\-mirrorlinks)
|
||||
.SS Proxy options:
|
||||
.IP \-P
|
||||
proxy use (\-P [socks5://][user:pass@]proxy:port) (\-\-proxy <param>)
|
||||
proxy use (\-P [socks5://|connect://][user:pass@]proxy:port) (\-\-proxy <param>)
|
||||
.IP \-%f
|
||||
*use proxy for ftp (f0 don't use) (\-\-httpproxy\-ftp[=N])
|
||||
.IP \-%b
|
||||
|
||||
@@ -2757,6 +2757,28 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
}
|
||||
}
|
||||
|
||||
// plain http tunneled through a CONNECT-only proxy (#564)
|
||||
if (back[i].r.req.proxy.active &&
|
||||
hts_proxy_is_connect(back[i].r.req.proxy.name) &&
|
||||
!back[i].r.keep_alive
|
||||
#if HTS_USEOPENSSL
|
||||
&& !back[i].r.ssl
|
||||
#endif
|
||||
) {
|
||||
const int timeout = back[i].timeout > 0 ? back[i].timeout : 30;
|
||||
|
||||
if (!http_proxy_tunnel(opt, &back[i].r, back[i].url_adr, timeout)) {
|
||||
if (!strnotempty(back[i].r.msg))
|
||||
strcpybuff(back[i].r.msg, "proxy CONNECT failed");
|
||||
deletehttp(&back[i].r);
|
||||
back[i].r.soc = INVALID_SOCKET;
|
||||
back[i].r.statuscode = STATUSCODE_NON_FATAL;
|
||||
back[i].status = STATUS_READY;
|
||||
back_set_finished(sback, i);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
#if HTS_USEOPENSSL
|
||||
/* SSL mode */
|
||||
if (back[i].r.ssl) {
|
||||
|
||||
@@ -477,7 +477,7 @@ void help(const char *app, int more) {
|
||||
(" Y mirror ALL links located in the first level pages (mirror links)");
|
||||
infomsg("");
|
||||
infomsg("Proxy options:");
|
||||
infomsg(" P proxy use (-P [socks5://][user:pass@]proxy:port)");
|
||||
infomsg(" P proxy use (-P [socks5://|connect://][user:pass@]proxy:port)");
|
||||
infomsg(" %f *use proxy for ftp (f0 don't use)");
|
||||
infomsg(" %b use this local hostname to make/send requests (-%b hostname)");
|
||||
infomsg("");
|
||||
|
||||
20
src/htslib.c
20
src/htslib.c
@@ -947,9 +947,10 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
|
||||
/* widths bound method[256], url[HTS_URLMAXSIZE*2], protocol[256] */
|
||||
if (sscanf(line, "%255s %2047s %255s", method, url, protocol) == 3) {
|
||||
size_t ret;
|
||||
// absolute-URI for an http proxy; a socks tunnel takes origin-form
|
||||
// http proxy: absolute-URI; socks/CONNECT tunnel: origin-form
|
||||
if (retour->req.proxy.active &&
|
||||
!hts_proxy_is_socks(retour->req.proxy.name)) {
|
||||
!hts_proxy_is_socks(retour->req.proxy.name) &&
|
||||
!hts_proxy_is_connect(retour->req.proxy.name)) {
|
||||
print_buffer(&bstr,
|
||||
"%s http://%s%s %s\r\n", method, adr, url,
|
||||
protocol);
|
||||
@@ -985,9 +986,10 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
|
||||
print_buffer(&bstr, "HEAD ");
|
||||
}
|
||||
|
||||
// an http proxy needs an absolute URI; a socks tunnel does not
|
||||
// an http proxy needs an absolute URI; a socks or CONNECT tunnel does not
|
||||
if (retour->req.proxy.active &&
|
||||
!hts_proxy_is_socks(retour->req.proxy.name) &&
|
||||
!hts_proxy_is_connect(retour->req.proxy.name) &&
|
||||
(strncmp(adr, "https://", 8) != 0)) {
|
||||
if (!link_has_authority(adr)) { // default http
|
||||
#if HDEBUG
|
||||
@@ -1032,10 +1034,10 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
|
||||
if (xsend)
|
||||
print_buffer(&bstr, "%s", xsend); // éventuelles autres lignes
|
||||
|
||||
// for https, auth rides the CONNECT (the tunneled GET would leak it); for
|
||||
// socks, the handshake
|
||||
// https/connect://: auth rides the CONNECT; socks: the handshake
|
||||
if (retour->req.proxy.active &&
|
||||
!hts_proxy_is_socks(retour->req.proxy.name) &&
|
||||
!hts_proxy_is_connect(retour->req.proxy.name) &&
|
||||
strncmp(adr, "https://", 8) != 0) {
|
||||
if (link_has_authorization(retour->req.proxy.name)) { // et hop, authentification proxy!
|
||||
const char *a = jump_identification_const(retour->req.proxy.name);
|
||||
@@ -3717,6 +3719,8 @@ const char *jump_protocol_const(const char *source) {
|
||||
source += p;
|
||||
else if ((p = strfield(source, "socks5:")))
|
||||
source += p;
|
||||
else if ((p = strfield(source, "connect:")))
|
||||
source += p;
|
||||
// net_path
|
||||
if (strncmp(source, "//", 2) == 0)
|
||||
source += 2;
|
||||
@@ -3732,6 +3736,12 @@ hts_boolean hts_proxy_is_socks(const char *name) {
|
||||
: HTS_FALSE;
|
||||
}
|
||||
|
||||
hts_boolean hts_proxy_is_connect(const char *name) {
|
||||
if (name == NULL)
|
||||
return HTS_FALSE;
|
||||
return strfield(name, "connect:") ? HTS_TRUE : HTS_FALSE;
|
||||
}
|
||||
|
||||
// default proxy port for a -P argument, keyed on the scheme
|
||||
static int proxy_default_port(const char *arg) {
|
||||
return hts_proxy_is_socks(arg) ? 1080 : 8080;
|
||||
|
||||
@@ -210,6 +210,9 @@ int check_writeinput_t(T_SOC soc, int timeout);
|
||||
/* TRUE if this -P proxy name (which keeps its scheme) is a SOCKS5 proxy. */
|
||||
hts_boolean hts_proxy_is_socks(const char *name);
|
||||
|
||||
/* TRUE if this -P proxy name is a "connect://" CONNECT-only proxy (#564). */
|
||||
hts_boolean hts_proxy_is_connect(const char *name);
|
||||
|
||||
void treathead(t_cookie * cookie, const char *adr, const char *fil, htsblk * retour,
|
||||
char *rcvd);
|
||||
void treatfirstline(htsblk * retour, const char *rcvd);
|
||||
|
||||
@@ -90,12 +90,15 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
|
||||
if (soc == INVALID_SOCKET)
|
||||
return 0;
|
||||
|
||||
// CONNECT needs an explicit host:port; default the https port
|
||||
// CONNECT needs an explicit host:port; default :80 for http, :443 for https
|
||||
authority[0] = '\0';
|
||||
if (portsep != NULL)
|
||||
strlcatbuff(authority, host, sizeof(authority)); // already host:port
|
||||
else
|
||||
snprintf(authority, sizeof(authority), "%s:%d", host, 443);
|
||||
else {
|
||||
const int defport = (strncmp(adr, "https://", 8) == 0) ? 443 : 80;
|
||||
|
||||
snprintf(authority, sizeof(authority), "%s:%d", host, defport);
|
||||
}
|
||||
|
||||
// backstop: never let a stray CR/LF in the host smuggle a second line into
|
||||
// the CONNECT request (the host is already sanitized upstream)
|
||||
@@ -132,7 +135,7 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
|
||||
}
|
||||
strlcatbuff(req, H_CRLF, sizeof(req)); // end of request headers
|
||||
|
||||
// raw send: ssl is set, so sendc() would route to TLS
|
||||
// raw send(): sendc() would route to TLS when ssl is set (https tunnel)
|
||||
{
|
||||
const char *p = req;
|
||||
size_t remain = strlen(req);
|
||||
|
||||
@@ -1365,9 +1365,12 @@ static int st_proxyurl(httrackp *opt, int argc, char **argv) {
|
||||
return 1;
|
||||
}
|
||||
hts_parse_proxy(argv[0], name, sizeof(name), &port);
|
||||
// host= is what the connect actually resolves (scheme + user:pass stripped)
|
||||
printf("name=%s port=%d host=%s\n", name, port,
|
||||
jump_identification_const(name));
|
||||
// host= is the resolved host (scheme/userinfo stripped); kind= the transport
|
||||
printf("name=%s port=%d host=%s kind=%s\n", name, port,
|
||||
jump_identification_const(name),
|
||||
hts_proxy_is_socks(name) ? "socks"
|
||||
: hts_proxy_is_connect(name) ? "connect"
|
||||
: "http");
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
@@ -5,40 +5,51 @@ set -euo pipefail
|
||||
|
||||
# hts_parse_proxy splits a -P argument "[scheme://][user:pass@]host[:port]" into
|
||||
# the proxy host string and port. -#test=proxyurl <arg> prints
|
||||
# "name=.. port=.. host=..", where host= is what the connect resolves.
|
||||
# "name=.. port=.. host=.. kind=..", where host= is what the connect resolves and
|
||||
# kind= is the transport the scheme selects (http, socks or connect).
|
||||
|
||||
p() {
|
||||
test "$(httrack -O /dev/null -#test=proxyurl "$1")" == "$2" || exit 1
|
||||
}
|
||||
|
||||
# bare host, with and without an explicit port
|
||||
p 'proxy.example.com:8080' 'name=proxy.example.com port=8080 host=proxy.example.com'
|
||||
p 'proxy.example.com' 'name=proxy.example.com port=8080 host=proxy.example.com'
|
||||
p 'proxy.example.com:8080' 'name=proxy.example.com port=8080 host=proxy.example.com kind=http'
|
||||
p 'proxy.example.com' 'name=proxy.example.com port=8080 host=proxy.example.com kind=http'
|
||||
|
||||
# user:pass@ is kept in name (for proxy auth) and stripped from host
|
||||
p 'user:pass@proxy:3128' 'name=user:pass@proxy port=3128 host=proxy'
|
||||
p 'user:pass@proxy' 'name=user:pass@proxy port=8080 host=proxy'
|
||||
p 'user:pass@proxy:3128' 'name=user:pass@proxy port=3128 host=proxy kind=http'
|
||||
p 'user:pass@proxy' 'name=user:pass@proxy port=8080 host=proxy kind=http'
|
||||
|
||||
# a scheme colon must not be read as a port: http://host with no port used to
|
||||
# parse to name=http with a garbage port
|
||||
p 'http://proxy:8080' 'name=http://proxy port=8080 host=proxy'
|
||||
p 'http://proxy' 'name=http://proxy port=8080 host=proxy'
|
||||
p 'http://proxy:8080' 'name=http://proxy port=8080 host=proxy kind=http'
|
||||
p 'http://proxy' 'name=http://proxy port=8080 host=proxy kind=http'
|
||||
|
||||
# socks5/socks5h default to 1080 and resolve to the bare host
|
||||
p 'socks5://host:1080' 'name=socks5://host port=1080 host=host'
|
||||
p 'socks5://host' 'name=socks5://host port=1080 host=host'
|
||||
p 'socks5h://host' 'name=socks5h://host port=1080 host=host'
|
||||
p 'socks5://host:1080' 'name=socks5://host port=1080 host=host kind=socks'
|
||||
p 'socks5://host' 'name=socks5://host port=1080 host=host kind=socks'
|
||||
p 'socks5h://host' 'name=socks5h://host port=1080 host=host kind=socks'
|
||||
# explicit port wins over the 1080 default, with and without userinfo
|
||||
p 'socks5://host:9050' 'name=socks5://host port=9050 host=host'
|
||||
p 'socks5://user:pass@host:9050' 'name=socks5://user:pass@host port=9050 host=host'
|
||||
p 'socks5://host:9050' 'name=socks5://host port=9050 host=host kind=socks'
|
||||
p 'socks5://user:pass@host:9050' 'name=socks5://user:pass@host port=9050 host=host kind=socks'
|
||||
|
||||
# connect:// is an http proxy driven with CONNECT for every request (#564); it
|
||||
# defaults to the http 8080 port and resolves to the bare host
|
||||
p 'connect://host' 'name=connect://host port=8080 host=host kind=connect'
|
||||
p 'connect://host:8082' 'name=connect://host port=8082 host=host kind=connect'
|
||||
p 'connect://user:pass@host:8082' 'name=connect://user:pass@host port=8082 host=host kind=connect'
|
||||
# only the scheme selects connect mode: a host literally named "connect" is a
|
||||
# plain http proxy, not a tunnel
|
||||
p 'connect:8080' 'name=connect port=8080 host=connect kind=http'
|
||||
p 'connect.example.com:3128' 'name=connect.example.com port=3128 host=connect.example.com kind=http'
|
||||
|
||||
# the split is byte-transparent: percent-escapes stay encoded (decoded later at
|
||||
# auth time), only structural ASCII ':'/'@' delimit, and the last '@' ends the
|
||||
# userinfo. So %40/%3A and UTF-8 bytes never mis-split host or port.
|
||||
p 'socks5://user:p%40ss@host:1080' 'name=socks5://user:p%40ss@host port=1080 host=host'
|
||||
p 'socks5://us%3Aer:pass@host:1080' 'name=socks5://us%3Aer:pass@host port=1080 host=host'
|
||||
p 'socks5://a@b:c@host:1080' 'name=socks5://a@b:c@host port=1080 host=host'
|
||||
p 'socks5://naïve:pass@héllo:1080' 'name=socks5://naïve:pass@héllo port=1080 host=héllo'
|
||||
p 'socks5://user:p%40ss@host:1080' 'name=socks5://user:p%40ss@host port=1080 host=host kind=socks'
|
||||
p 'socks5://us%3Aer:pass@host:1080' 'name=socks5://us%3Aer:pass@host port=1080 host=host kind=socks'
|
||||
p 'socks5://a@b:c@host:1080' 'name=socks5://a@b:c@host port=1080 host=host kind=socks'
|
||||
p 'socks5://naïve:pass@héllo:1080' 'name=socks5://naïve:pass@héllo port=1080 host=héllo kind=socks'
|
||||
|
||||
# a lone ':' must not underflow the backward scan
|
||||
p ':' 'name= port=8080 host='
|
||||
p ':' 'name= port=8080 host= kind=http'
|
||||
|
||||
168
tests/57_local-proxy-connect.test
Normal file
168
tests/57_local-proxy-connect.test
Normal file
@@ -0,0 +1,168 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Issue #564: plain http through a CONNECT-only proxy (tor HTTPTunnelPort). With
|
||||
# a "connect://" proxy httrack must open a CONNECT tunnel to the origin and send
|
||||
# the request origin-form, like https already does, instead of the absolute-URI
|
||||
# form a CONNECT-only proxy rejects. Fully local: a plain-http origin behind a
|
||||
# proxy that answers CONNECT and 501s everything else.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "$top_srcdir/tests/testlib.sh"
|
||||
|
||||
python=$(find_python) || python=
|
||||
if test -z "$python"; then
|
||||
echo "python3 missing, skipping"
|
||||
exit 77
|
||||
fi
|
||||
|
||||
server=$(nativepath "$top_srcdir/tests/proxy-connect-server.py")
|
||||
tmpdir=$(mktemp -d)
|
||||
pids=
|
||||
|
||||
cleanup() {
|
||||
for pid in $pids; do
|
||||
stop_server "$pid"
|
||||
done
|
||||
rm -rf "$tmpdir"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# start_server <logdir> <mode>: launches a proxy+origin pair, sets $origin_port
|
||||
# and $proxy_port from its announced ephemeral ports.
|
||||
start_server() {
|
||||
local dir="$1" mode="$2" ports
|
||||
mkdir -p "$dir"
|
||||
ports="$dir/ports.txt"
|
||||
dir_native=$(nativepath "$dir")
|
||||
: >"$ports"
|
||||
"$python" "$server" "$dir_native" "$mode" \
|
||||
>"$ports" 2>"$dir/server.err" &
|
||||
pids="$pids $!"
|
||||
for _ in $(seq 1 100); do
|
||||
grep -q "^ready" "$ports" 2>/dev/null && break
|
||||
sleep 0.1
|
||||
done
|
||||
grep -q "^ready" "$ports" 2>/dev/null || {
|
||||
echo "server ($mode) did not start" >&2
|
||||
cat "$dir/server.err" >&2
|
||||
exit 1
|
||||
}
|
||||
origin_port=$(awk '/^ORIGIN/{print $2}' "$ports")
|
||||
proxy_port=$(awk '/^PROXY/{print $2}' "$ports")
|
||||
}
|
||||
|
||||
# Kill httrack after a deadline so a hang surfaces as $HANG_RC, not a stalled job
|
||||
# (a portable stand-in for `timeout`, which macOS lacks).
|
||||
HANG_RC=137 # 128 + SIGKILL
|
||||
run_crawl() {
|
||||
local out="$1" proxy="$2" url="${4:-http://127.0.0.1:${3}/}"
|
||||
rm -rf "$out"
|
||||
httrack "$url" --proxy "$proxy" \
|
||||
-O "$out" -r1 -s0 --timeout=10 >"$out.log" 2>&1 &
|
||||
local pid=$!
|
||||
(sleep 60 && kill -9 "$pid" 2>/dev/null) &
|
||||
local guard=$!
|
||||
local rc=0
|
||||
wait "$pid" 2>/dev/null || rc=$?
|
||||
kill "$guard" 2>/dev/null || true
|
||||
wait "$guard" 2>/dev/null || true
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
# --- working proxy ----------------------------------------------------------
|
||||
ok="$tmpdir/ok"
|
||||
start_server "$ok" ok
|
||||
|
||||
# 1. page fetched through a CONNECT tunnel, request delivered origin-form
|
||||
run_crawl "$ok/out" "connect://127.0.0.1:${proxy_port}" "$origin_port"
|
||||
grep -rq "ORIGIN-PAGE-564" "$ok/out" || {
|
||||
echo "FAIL: origin page not downloaded through the CONNECT tunnel" >&2
|
||||
cat "$ok/out.log" >&2
|
||||
exit 1
|
||||
}
|
||||
grep -q "^CONNECT 127.0.0.1:${origin_port} " "$ok/proxy.log" || {
|
||||
echo "FAIL: proxy never received a CONNECT for the plain-http origin" >&2
|
||||
cat "$ok/proxy.log" >&2
|
||||
exit 1
|
||||
}
|
||||
grep -q "^GET / HTTP/1" "$ok/origin-headers.log" || {
|
||||
echo "FAIL: origin did not see an origin-form request line" >&2
|
||||
cat "$ok/origin-headers.log" >&2
|
||||
exit 1
|
||||
}
|
||||
if grep -q "^GET http://" "$ok/origin-headers.log"; then
|
||||
echo "FAIL: absolute-URI request leaked through the tunnel to the origin" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: plain http tunneled origin-form through CONNECT"
|
||||
|
||||
# 2. default CONNECT port follows the origin scheme: a portless http origin
|
||||
# tunnels to :80 (not the https :443). Nothing serves :80 here so the fetch
|
||||
# fails; only the CONNECT target port is asserted. Reverting to a hardcoded 443
|
||||
# makes this log ":443" and fail.
|
||||
: >"$ok/proxy.log"
|
||||
run_crawl "$ok/out_port" "connect://127.0.0.1:${proxy_port}" "" "http://127.0.0.1/" || true
|
||||
grep -q "^CONNECT 127.0.0.1:80 " "$ok/proxy.log" || {
|
||||
echo "FAIL: portless http origin did not CONNECT to the default port 80" >&2
|
||||
cat "$ok/proxy.log" >&2
|
||||
exit 1
|
||||
}
|
||||
echo "OK: portless http origin tunnels to the default port 80"
|
||||
|
||||
# 3. teeth: without connect://, httrack sends the absolute-URI form, which the
|
||||
# CONNECT-only proxy rejects (501) -> nothing fetched. Proves the scheme is what
|
||||
# flips the behavior and that the proxy really is CONNECT-only.
|
||||
: >"$ok/proxy.log"
|
||||
: >"$ok/origin-headers.log"
|
||||
run_crawl "$ok/out_plain" "127.0.0.1:${proxy_port}" "$origin_port" || true
|
||||
grep -rq "ORIGIN-PAGE-564" "$ok/out_plain" && {
|
||||
echo "FAIL: absolute-URI form unexpectedly fetched through a CONNECT-only proxy" >&2
|
||||
exit 1
|
||||
}
|
||||
grep -q "^GET http://127.0.0.1:${origin_port}/" "$ok/proxy.log" || {
|
||||
echo "FAIL: a plain proxy should have received an absolute-URI GET" >&2
|
||||
cat "$ok/proxy.log" >&2
|
||||
exit 1
|
||||
}
|
||||
echo "OK: absolute-URI form (no connect://) rejected by the CONNECT-only proxy"
|
||||
|
||||
# 4. authenticated proxy: creds ride the CONNECT, never reach the origin
|
||||
: >"$ok/proxy.log"
|
||||
: >"$ok/origin-headers.log"
|
||||
run_crawl "$ok/out2" "connect://user:secret@127.0.0.1:${proxy_port}" "$origin_port"
|
||||
grep -rq "ORIGIN-PAGE-564" "$ok/out2" || {
|
||||
echo "FAIL: origin page not downloaded through the authenticated proxy" >&2
|
||||
exit 1
|
||||
}
|
||||
got=$(awk '/^AUTH Basic /{print $3}' "$ok/proxy.log" | head -1)
|
||||
# base64("user:secret"); compared literally to stay portable (no base64 -d)
|
||||
test "$got" == "dXNlcjpzZWNyZXQ=" || {
|
||||
echo "FAIL: Proxy-Authorization not carried on CONNECT (got '$got')" >&2
|
||||
cat "$ok/proxy.log" >&2
|
||||
exit 1
|
||||
}
|
||||
if grep -qi "proxy-authorization" "$ok/origin-headers.log"; then
|
||||
echo "FAIL: proxy credentials leaked to the origin through the tunnel" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: proxy credentials carried on CONNECT, not leaked to origin"
|
||||
|
||||
# --- hostile proxy ----------------------------------------------------------
|
||||
# A proxy that answers 200 then streams headers forever must not hang the crawl.
|
||||
flood="$tmpdir/flood"
|
||||
start_server "$flood" flood
|
||||
rc=0
|
||||
run_crawl "$flood/out" "connect://127.0.0.1:${proxy_port}" "$origin_port" || rc=$?
|
||||
test "$rc" -ne "$HANG_RC" || {
|
||||
echo "FAIL: crawl hung on a flooding proxy (bounded read missing)" >&2
|
||||
exit 1
|
||||
}
|
||||
grep -rq "ORIGIN-PAGE-564" "$flood/out" 2>/dev/null && {
|
||||
echo "FAIL: flooding proxy unexpectedly served the page" >&2
|
||||
exit 1
|
||||
}
|
||||
echo "OK: bounded proxy response, no hang on a flooding proxy"
|
||||
@@ -2,7 +2,7 @@
|
||||
# explicitly: automake does not expand wildcards in EXTRA_DIST, so a glob would
|
||||
# silently drop it from the dist tarball and break "make distcheck".
|
||||
EXTRA_DIST = $(TESTS) crawl-test.sh run-all-tests.sh check-network.sh \
|
||||
proxy-https-server.py socks5-server.py \
|
||||
proxy-https-server.py socks5-server.py proxy-connect-server.py \
|
||||
local-crawl.sh local-server.py testlib.sh server.crt server.key \
|
||||
server-root/simple/basic.html server-root/simple/link.html \
|
||||
server-root/stripquery/index.html server-root/stripquery/a.html \
|
||||
@@ -133,6 +133,7 @@ TESTS = \
|
||||
53_local-proxytrack-cache-corrupt.test \
|
||||
54_local-update-truncate-purge.test \
|
||||
55_local-chunked.test \
|
||||
56_local-proxy-noleak.test
|
||||
56_local-proxy-noleak.test \
|
||||
57_local-proxy-connect.test
|
||||
|
||||
CLEANFILES = check-network_sh.cache
|
||||
|
||||
152
tests/proxy-connect-server.py
Normal file
152
tests/proxy-connect-server.py
Normal file
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Local CONNECT-only proxy + plain-HTTP origin for the issue #564 test.
|
||||
|
||||
Models a tor HTTPTunnelPort: an HTTP proxy that honours CONNECT and rejects
|
||||
every other method with 501 (the classic absolute-URI form a normal HTTP proxy
|
||||
accepts). Fronting it is a plain-HTTP origin. The proxy logs each request line
|
||||
(and any Proxy-Authorization); the origin logs its request line and headers.
|
||||
That lets the test assert a plain-http crawl tunneled through CONNECT, arrived
|
||||
origin-form (not "GET http://host/..."), and never leaked proxy credentials.
|
||||
|
||||
Proxy modes (argv[2], default "ok"):
|
||||
ok - honour CONNECT and tunnel to the origin
|
||||
flood - answer 200 then stream headers forever with no blank line, to exercise
|
||||
the client's bound on the proxy response (must not hang the crawl)
|
||||
|
||||
Usage: proxy-connect-server.py <logdir> [mode]
|
||||
Prints "ORIGIN <port>", "PROXY <port>", then "ready" (one per line) on stdout.
|
||||
"""
|
||||
import http.server
|
||||
import os
|
||||
import socket
|
||||
import socketserver
|
||||
import sys
|
||||
import threading
|
||||
|
||||
ORIGIN_BODY = b"<html><body>ORIGIN-PAGE-564</body></html>"
|
||||
PROXY_LOG = "proxy.log"
|
||||
ORIGIN_LOG = "origin-headers.log"
|
||||
|
||||
|
||||
def make_origin(logdir):
|
||||
class Origin(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
with open(os.path.join(logdir, ORIGIN_LOG), "a") as handle:
|
||||
handle.write(self.requestline + "\n")
|
||||
for key in self.headers.keys():
|
||||
handle.write(key + "\n")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html")
|
||||
self.send_header("Content-Length", str(len(ORIGIN_BODY)))
|
||||
self.end_headers()
|
||||
self.wfile.write(ORIGIN_BODY)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
return Origin
|
||||
|
||||
|
||||
def start_origin(logdir):
|
||||
httpd = socketserver.TCPServer(("127.0.0.1", 0), make_origin(logdir))
|
||||
port = httpd.socket.getsockname()[1]
|
||||
threading.Thread(target=httpd.serve_forever, daemon=True).start()
|
||||
return port
|
||||
|
||||
|
||||
def pipe(src, dst):
|
||||
try:
|
||||
while True:
|
||||
data = src.recv(65536)
|
||||
if not data:
|
||||
break
|
||||
dst.sendall(data)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
for sock in (src, dst):
|
||||
try:
|
||||
sock.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def handle_client(conn, logdir, mode):
|
||||
rfile = conn.makefile("rb")
|
||||
request_line = rfile.readline().decode("latin-1").strip()
|
||||
auth = None
|
||||
while True:
|
||||
line = rfile.readline().decode("latin-1")
|
||||
if line in ("\r\n", "\n", ""):
|
||||
break
|
||||
key, _, value = line.partition(":")
|
||||
if key.strip().lower() == "proxy-authorization":
|
||||
auth = value.strip()
|
||||
with open(os.path.join(logdir, PROXY_LOG), "a") as handle:
|
||||
handle.write(request_line + "\n")
|
||||
if auth is not None:
|
||||
handle.write("AUTH " + auth + "\n")
|
||||
parts = request_line.split()
|
||||
# CONNECT-only: reject the classic absolute-URI form a normal proxy accepts
|
||||
if not (len(parts) >= 2 and parts[0] == "CONNECT"):
|
||||
conn.sendall(b"HTTP/1.0 501 Not Implemented\r\n\r\n")
|
||||
conn.close()
|
||||
return
|
||||
if mode == "flood":
|
||||
# 200, then an endless header stream with no terminating blank line: the
|
||||
# client must bound this and give up, not hang.
|
||||
try:
|
||||
conn.sendall(b"HTTP/1.0 200 Connection established\r\n")
|
||||
while True:
|
||||
conn.sendall(b"X-Pad: 0123456789\r\n")
|
||||
except OSError:
|
||||
pass
|
||||
conn.close()
|
||||
return
|
||||
host, _, port = parts[1].partition(":")
|
||||
try:
|
||||
upstream = socket.create_connection((host, int(port or 80)))
|
||||
except OSError:
|
||||
conn.sendall(b"HTTP/1.0 502 Bad Gateway\r\n\r\n")
|
||||
conn.close()
|
||||
return
|
||||
conn.sendall(b"HTTP/1.0 200 Connection established\r\n\r\n")
|
||||
threading.Thread(target=pipe, args=(conn, upstream), daemon=True).start()
|
||||
pipe(upstream, conn)
|
||||
|
||||
|
||||
def start_proxy(logdir, mode):
|
||||
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
srv.bind(("127.0.0.1", 0))
|
||||
srv.listen(16)
|
||||
port = srv.getsockname()[1]
|
||||
|
||||
def serve():
|
||||
while True:
|
||||
conn, _ = srv.accept()
|
||||
threading.Thread(
|
||||
target=handle_client, args=(conn, logdir, mode), daemon=True
|
||||
).start()
|
||||
|
||||
threading.Thread(target=serve, daemon=True).start()
|
||||
return port
|
||||
|
||||
|
||||
def main():
|
||||
logdir = sys.argv[1]
|
||||
mode = sys.argv[2] if len(sys.argv) > 2 else "ok"
|
||||
for name in (PROXY_LOG, ORIGIN_LOG):
|
||||
open(os.path.join(logdir, name), "w").close()
|
||||
origin_port = start_origin(logdir)
|
||||
proxy_port = start_proxy(logdir, mode)
|
||||
# Keep the port lines the caller parses LF: Windows would emit \r\n.
|
||||
sys.stdout.reconfigure(newline="\n")
|
||||
print("ORIGIN %d" % origin_port, flush=True)
|
||||
print("PROXY %d" % proxy_port, flush=True)
|
||||
print("ready", flush=True)
|
||||
threading.Event().wait()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user