Compare commits

..

6 Commits

Author SHA1 Message Date
Xavier Roche
419827736b proxytrack: MSVC spells the timezone global _timezone
store.c's getGMT reads the POSIX global, which the UCRT only exposes under the
underscore name, so proxytrack did not compile on MSVC. Extend the existing BSD
branch rather than reach for timegm: the arithmetic here is unchanged everywhere
else, and this is a build port, not a fix for its DST handling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 10:39:05 +02:00
Xavier Roche
9fa1dc16bf Link the CLI against Ws2_32
httrack.c's main() calls WSAStartup and WSACleanup itself, so the import library
is needed here as well, not just in the projects that do their own networking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 10:29:00 +02:00
Xavier Roche
a38773e010 Fix a stray closing tag in the new project files, and gate XML in CI
The three .vcxproj I added carried a trailing </content> line, so MSBuild refused
to load them (MSB4025) and nothing built. Strip it.

The failure cost a full Windows CI round to learn something a parser answers in
milliseconds, so check every src/*.vcxproj for well-formedness in the lint job,
which runs on Linux. Verified against the malformed file it would have caught.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 10:18:06 +02:00
Xavier Roche
4067ab55c8 ci: give the CRT check a negative control
The assertion had only ever been observed passing, which says nothing: a detector
that matches anything would pass too. Run it first against cmd.exe, which links
the old msvcrt and no vcruntime, so a pattern that cannot come back empty fails
the job instead of silently blessing a mismatched CRT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 10:06:26 +02:00
Xavier Roche
59ba7f6c07 ci: make the CRT check actually able to fail, and pin the machine type
The check used -notmatch against dumpbin's output. On an array those operators
filter rather than test, so it returned the (many) lines without vcruntime in
them, which is truthy, and the step threw whatever the binaries imported. Pull
the vcruntime name out of each binary instead, require both to have one, and
compare them: the property under test is that the two sides share a CRT, not
that each has one.

Assert the machine type in the same pass. Test-Path would happily accept an x64
binary sitting in the Win32 output directory.

httrack.vcxproj gets the vcpkg triplet the other projects carry; without it
applocal deployment does not drop libhttrack's OpenSSL and zlib DLLs beside
httrack.exe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 10:05:36 +02:00
Xavier Roche
024f18fd94 Port the CLI, webhttrack and proxytrack to VS2022, and retire the VS2008 files
libhttrack got a .vcxproj in #554; the other three projects were left on VS2008,
so WinHTTrack.sln still could not be opened as a whole. Recreate them as v143,
x86+x64, /MD, and drop the hardcoded C:\Dev and C:\temp paths: output now lands in
src/$(Platform)/$(Configuration) like libhttrack, and zlib comes from vcpkg. httrack
and webhttrack carry a ProjectReference to libhttrack; proxytrack is standalone (it
borrows engine headers but links none of it, hts_effective_mime being a macro).

The legacy .vcproj files go, along with the VC6 .dsp/.dsw beside them: v143 cannot
open either, and EXTRA_DIST now ships the .vcxproj instead, so a release tarball
still carries a project that opens.

CI builds all four and checks the binaries land where WinHTTrack expects. It also
asserts, via dumpbin /dependents, that the DLL and the exe import the same
vcruntime140.dll: a CRT split across that boundary links cleanly and crashes on the
first cross-heap free, and nothing else in either build would catch it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 00:05:33 +02:00
43 changed files with 318 additions and 2633 deletions

3
.gitattributes vendored
View File

@@ -1,3 +0,0 @@
# Resource scripts are Windows tooling input and must stay CRLF: the engine has no
# encoding guard, so an autocrlf checkout could otherwise flatten them silently.
*.rc text eol=crlf

View File

@@ -41,7 +41,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev
- name: Configure
run: |
@@ -50,9 +50,6 @@ jobs:
# committed generated files already let a plain checkout build.
autoreconf -fi
./configure
# a missing decoder would silently drop the coding from Accept-Encoding
grep -q "define HTS_USEBROTLI 1" config.h
grep -q "define HTS_USEZSTD 1" config.h
- name: Build
run: make -j"$(nproc)"
@@ -85,7 +82,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev
- name: Configure
run: |
@@ -126,18 +123,14 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
brew install autoconf automake libtool autoconf-archive brotli zstd
brew install autoconf automake libtool autoconf-archive
- name: Configure
run: |
set -euo pipefail
ssl="$(brew --prefix openssl@3)"
brewp="$(brew --prefix)"
autoreconf -fi
./configure CPPFLAGS="-I${ssl}/include -I${brewp}/include" \
LDFLAGS="-L${ssl}/lib -L${brewp}/lib"
grep -q "define HTS_USEBROTLI 1" config.h
grep -q "define HTS_USEZSTD 1" config.h
./configure CPPFLAGS="-I${ssl}/include" LDFLAGS="-L${ssl}/lib"
- name: Build
run: make -j"$(sysctl -n hw.ncpu)"
@@ -173,16 +166,14 @@ jobs:
- name: Install build dependencies
run: |
set -euo pipefail
brew install autoconf automake libtool autoconf-archive brotli zstd
brew install autoconf automake libtool autoconf-archive
- name: Build and install into a temp prefix
run: |
set -euo pipefail
ssl="$(brew --prefix openssl@3)"
brewp="$(brew --prefix)"
autoreconf -fi
./configure CPPFLAGS="-I${ssl}/include -I${brewp}/include" \
LDFLAGS="-L${ssl}/lib -L${brewp}/lib" \
./configure CPPFLAGS="-I${ssl}/include" LDFLAGS="-L${ssl}/lib" \
--prefix="$RUNNER_TEMP/inst"
make -j"$(sysctl -n hw.ncpu)"
make install
@@ -210,8 +201,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential gcc-multilib autoconf automake libtool \
autoconf-archive zlib1g-dev:i386 libssl-dev:i386 \
libbrotli-dev:i386 libzstd-dev:i386
autoconf-archive zlib1g-dev:i386 libssl-dev:i386
- name: Configure
run: |
@@ -252,7 +242,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev
- name: Configure (sanitized)
run: |
@@ -354,7 +344,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev
- name: Configure (fuzzers, static)
run: |
@@ -434,7 +424,7 @@ jobs:
apt-get install -y --no-install-recommends \
ca-certificates git \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
zlib1g-dev libssl-dev \
debhelper devscripts lintian fakeroot
- uses: actions/checkout@v6
@@ -478,7 +468,7 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev
- name: distcheck
run: |

View File

@@ -119,23 +119,6 @@ jobs:
throw "CRT mismatch across the boundary: libhttrack.dll=$a httrack.exe=$b"
}
# The engine self-tests exercise the codecs the DLL was linked with: a
# missing brotli/zstd decoder shows up here, not just at link time.
- name: Run the content-coding self-tests
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
$exe = "src\${{ matrix.platform }}\${{ matrix.configuration }}\httrack.exe"
$tmp = New-Item -ItemType Directory -Path (Join-Path $env:RUNNER_TEMP "st")
foreach ($t in @("acceptencoding", "contentcodings")) {
$out = & $exe -O NUL "-#test=$t" $tmp.FullName run
if ($LASTEXITCODE -ne 0) { throw "$t failed" }
if (-not ($out | Select-String -SimpleMatch "$t self-test OK")) {
throw "$t did not report OK: $out"
}
Write-Host ($out -join "`n")
}
- name: Upload MSBuild logs
if: always()
uses: actions/upload-artifact@v4

View File

@@ -1 +0,0 @@
/home/roche/git/httrack-works/CLAUDE.httrack.local.md

View File

@@ -173,10 +173,6 @@ AC_CHECK_HEADERS([execinfo.h])
### zlib
CHECK_ZLIB()
### brotli and zstd content codings (optional)
CHECK_BROTLI()
CHECK_ZSTD()
### OpenSSL is explicitly enabled/disabled ?
AC_MSG_CHECKING(whether to enable https support)
AC_ARG_ENABLE([https],

2
debian/control vendored
View File

@@ -2,7 +2,7 @@ Source: httrack
Section: web
Maintainer: Xavier Roche <roche@httrack.com>
Standards-Version: 4.7.4
Build-Depends: debhelper-compat (= 14), autoconf, autoconf-archive, automake, libtool, zlib1g-dev, libssl-dev, libbrotli-dev, libzstd-dev
Build-Depends: debhelper-compat (= 14), autoconf, autoconf-archive, automake, libtool, zlib1g-dev, libssl-dev
Rules-Requires-Root: no
Homepage: http://www.httrack.com
Vcs-Git: https://github.com/xroche/httrack.git

View File

@@ -1,66 +0,0 @@
dnl @synopsis CHECK_BROTLI()
dnl @synopsis CHECK_ZSTD()
dnl
dnl Look for libbrotlidec and libzstd, the decoders for the "br" and "zstd"
dnl HTTP content codings. Both are optional: a missing library only drops the
dnl coding from Accept-Encoding. --with-brotli=DIR / --with-zstd=DIR point at a
dnl prefix; --without-brotli / --without-zstd disable the coding; the default
dnl is to use the library when it is found.
dnl
dnl Define HTS_USEBROTLI / HTS_USEZSTD to 1 or 0, and substitute BROTLI_LIBS /
dnl ZSTD_LIBS.
dnl CHECK_CODEC(name, NAME, header, library, symbol, companion-libs)
AC_DEFUN([CHECK_CODEC], [
AC_ARG_WITH([$1],
[AS_HELP_STRING([--with-$1@<:@=DIR@:>@],[Enable the $1 content coding @<:@default=auto@:>@])],
[codec_want=$withval], [codec_want=auto])
$2_LIBS=""
codec_have=no
if test "$codec_want" != "no"; then
codec_old_cppflags="$CPPFLAGS"
codec_old_ldflags="$LDFLAGS"
codec_dir=no
if test "$codec_want" != "yes" -a "$codec_want" != "auto"; then
codec_dir=yes
# An explicit prefix is authoritative: if the header is not under
# it, error rather than silently pick a system copy.
if test ! -f "$codec_want/include/$3"; then
AC_MSG_ERROR([$1 requested at $codec_want but $codec_want/include/$3 is missing])
fi
CPPFLAGS="$CPPFLAGS -I$codec_want/include"
LDFLAGS="$LDFLAGS -L$codec_want/lib"
fi
AC_CHECK_HEADER([$3], [codec_h=yes], [codec_h=no])
AC_CHECK_LIB([$4], [$5], [codec_lib=yes], [codec_lib=no], [$6])
CPPFLAGS="$codec_old_cppflags"
LDFLAGS="$codec_old_ldflags"
if test "$codec_h" = "yes" -a "$codec_lib" = "yes"; then
codec_have=yes
$2_LIBS="-l$4 $6"
if test "$codec_dir" = "yes"; then
$2_LIBS="-L$codec_want/lib -l$4 $6"
# The rest of configure still needs the header path.
CPPFLAGS="$CPPFLAGS -I$codec_want/include"
fi
elif test "$codec_want" != "auto"; then
AC_MSG_ERROR([$1 requested but lib$4/$3 not found])
fi
fi
AC_MSG_CHECKING([whether to enable the $1 content coding])
AC_MSG_RESULT([$codec_have])
if test "$codec_have" = "yes"; then
AC_DEFINE([HTS_USE$2], [1], [Define to 1 to decode the $1 content coding])
else
AC_DEFINE([HTS_USE$2], [0], [Define to 1 to decode the $1 content coding])
fi
AC_SUBST([$2_LIBS])
AC_SUBST([$2_ENABLED], [$codec_have])
])
dnl brotlidec needs brotlicommon (the static dictionary) for a fully-static link.
AC_DEFUN([CHECK_BROTLI],
[CHECK_CODEC([brotli], [BROTLI], [brotli/decode.h], [brotlidec], [BrotliDecoderDecompressStream], [-lbrotlicommon])])
AC_DEFUN([CHECK_ZSTD],
[CHECK_CODEC([zstd], [ZSTD], [zstd.h], [zstd], [ZSTD_decompressStream], [])])

View File

@@ -3,7 +3,7 @@
.\"
.\" This file is generated by man/makeman.sh; do not edit by hand.
.\" SPDX-License-Identifier: GPL-3.0-or-later
.TH httrack 1 "14 July 2026" "httrack website copier"
.TH httrack 1 "12 July 2026" "httrack website copier"
.SH NAME
httrack \- offline browser : copy websites to a local directory
.SH SYNOPSIS
@@ -135,7 +135,7 @@ continue an interrupted mirror using the cache (\-\-continue)
mirror ALL links located in the first level pages (mirror links) (\-\-mirrorlinks)
.SS Proxy options:
.IP \-P
proxy use (\-P [socks5://][user:pass@]proxy:port) (\-\-proxy <param>)
proxy use (\-P proxy:port or \-P user:pass@proxy:port) (\-\-proxy <param>)
.IP \-%f
*use proxy for ftp (f0 don't use) (\-\-httpproxy\-ftp[=N])
.IP \-%b

View File

@@ -61,7 +61,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htshelp.c htslib.c htscoremain.c \
htsname.c htsrobots.c htstools.c htswizard.c \
htsalias.c htsthread.c htsindex.c htsbauth.c \
htsmd5.c htscodec.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
htsmd5.c htszlib.c htswrap.c htsconcat.c \
htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
md5.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
@@ -72,20 +72,17 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htshelp.h htsindex.h htslib.h htsmd5.h \
htsmodules.h htsname.h htsnet.h htssniff.h \
htsopt.h htsrobots.h htsthread.h \
htstools.h htswizard.h htswrap.h htscodec.h htsproxy.h htszlib.h \
htstools.h htswizard.h htswrap.h htszlib.h \
htsstrings.h htsarrays.h httrack-library.h \
htscharset.h punycode.h htsencoding.h \
htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \
md5.h coucal/murmurhash3.h \
minizip/crypt.h minizip/ioapi.h minizip/mztools.h minizip/unzip.h minizip/zip.h
libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS) $(OPENSSL_LIBS) $(DL_LIBS) $(SOCKET_LIBS) $(ICONV_LIBS)
libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(OPENSSL_LIBS) $(DL_LIBS) $(SOCKET_LIBS) $(ICONV_LIBS)
libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
EXTRA_DIST = httrack.h webhttrack \
version.rc \
libhttrack.rc \
httrack.rc \
coucal/murmurhash3.h.diff \
coucal/murmurhash3.h.orig \
minizip/iowin32.c \

View File

@@ -44,8 +44,6 @@ Please visit our Website: http://www.httrack.com
#include "htsback.h"
#include "htsftp.h"
#include "htscodec.h"
#include "htsproxy.h"
#if HTS_USEZLIB
#include "htszlib.h"
#else
@@ -558,14 +556,6 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
return 0;
}
/* Move src onto dst; RENAME does not clobber an existing target on Windows. */
static hts_boolean replace_file(const char *src, const char *dst) {
if (RENAME(src, dst) == 0)
return HTS_TRUE;
(void) UNLINK(dst);
return RENAME(src, dst) == 0 ? HTS_TRUE : HTS_FALSE;
}
/* Commit or restore a re-fetch backup (#77 follow-up): a re-fetch over an
existing file moved the good copy to back->tmpfile before truncating url_sav.
commit keeps the new file and drops the backup; else restore it so an aborted
@@ -581,9 +571,10 @@ static void back_finalize_backup(httrackp *opt, lien_back *const back,
fclose(back->r.out);
back->r.out = NULL;
}
/* On failure keep the backup: an orphaned temp beats losing the good copy.
*/
if (!replace_file(back->tmpfile, back->url_sav))
(void) UNLINK(back->url_sav); /* drop the failed partial */
/* On rename failure keep the backup: it still holds the good copy, so
losing it would be worse than an orphaned temp. */
if (RENAME(back->tmpfile, back->url_sav) != 0)
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"could not restore %s; previous copy kept as %s",
back->url_sav, back->tmpfile);
@@ -636,7 +627,8 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
if (!back[p].testmode) { // not test mode
const char *state = "unknown";
/* Undo the content coding */
/* décompression */
#if HTS_USEZLIB
if (back[p].r.compressed) {
if (back[p].r.size > 0) {
// stats
@@ -682,67 +674,28 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
// décompression
if (back[p].tmpfile != NULL) {
if (back[p].url_sav[0]) {
const hts_codec codec =
hts_codec_parse(back[p].r.contentencoding);
/* Never decode over url_sav: a failed decode would destroy the
copy an --update re-fetch is supposed to refresh (#557). */
char BIGSTK unpacked[HTS_URLMAXSIZE * 2 + 4]; // room for ".u"
LLint size;
snprintf(unpacked, sizeof(unpacked), "%s.u", back[p].url_sav);
if ((size = hts_codec_unpack(codec, back[p].tmpfile,
unpacked)) >= 0) {
file_notify(opt, back[p].url_adr, back[p].url_fil,
back[p].url_sav, 1, 1, back[p].r.notmodified);
filecreateempty(&opt->state.strc, back[p].url_sav); // filenote & co
if ((size = hts_zunpack(back[p].tmpfile, back[p].url_sav)) >= 0) {
back[p].r.size = back[p].r.totalsize = size;
// fichier -> mémoire
if (!back[p].r.is_write) {
// fichier -> mémoire ; le fichier est écrit plus tard
deleteaddr(&back[p].r);
back[p].r.adr = readfile_utf8(unpacked);
back[p].r.adr = readfile_utf8(back[p].url_sav);
if (!back[p].r.adr) {
back[p].r.statuscode = STATUSCODE_INVALID;
strcpybuff(back[p].r.msg,
"Read error when decompressing");
}
UNLINK(unpacked);
} else if (replace_file(unpacked, back[p].url_sav)) {
/* The temp bypassed filecreate(), which is what chmods. */
#ifndef _WIN32
chmod(back[p].url_sav, HTS_ACCESS_FILE);
#endif
file_notify(opt, back[p].url_adr, back[p].url_fil,
back[p].url_sav, 1, 1, back[p].r.notmodified);
filenote(&opt->state.strc, back[p].url_sav, NULL);
} else {
back[p].r.statuscode = STATUSCODE_INVALID;
strcpybuff(back[p].r.msg,
"Write error when decompressing (can not rename "
"the temporary file)");
/* Keep the decoded body: the failed replace may have
removed the previous copy, leaving this as the only one.
*/
hts_log_print(
opt, LOG_WARNING | LOG_ERRNO,
"could not replace %s; decoded copy kept as %s",
back[p].url_sav, unpacked);
UNLINK(back[p].url_sav);
}
} else {
back[p].r.statuscode = STATUSCODE_INVALID;
snprintf(back[p].r.msg, sizeof(back[p].r.msg),
codec == HTS_CODEC_UNSUPPORTED
? "Unsupported Content-Encoding (%s)"
: "Error when decompressing (%s)",
back[p].r.contentencoding);
/* Drop the undecoded body so the writer can't commit the
coded bytes as the page; url_sav is left untouched. */
if (!back[p].r.is_write)
deleteaddr(&back[p].r);
UNLINK(unpacked);
strcpybuff(back[p].r.msg, "Error when decompressing");
}
/* A failed decode keeps the previously-mirrored copy: note it,
or the update purge (in old.lst, absent from new.lst) would
delete what we just took care not to overwrite. */
if (back[p].r.statuscode == STATUSCODE_INVALID &&
fexist_utf8(back[p].url_sav))
filenote(&opt->state.strc, back[p].url_sav, NULL);
}
/* ensure that no remaining temporary file exists */
unlink(back[p].tmpfile);
@@ -755,6 +708,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
// unflag
}
}
#endif
/* Body fully received: keep the freshly written url_sav, drop the
backup of the previous copy. */
back_finalize_backup(opt, &back[p], HTS_TRUE);
@@ -853,7 +807,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
else
strcatbuff(flags, "-");
if (back[p].r.compressed)
strcatbuff(flags, "Z"); // content coding
strcatbuff(flags, "Z"); // gzip
else
strcatbuff(flags, "-");
/* Err I had to split these.. */
@@ -862,7 +816,9 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
fprintf(cache->txt, LLintP, (LLint) back[p].r.totalsize);
fprintf(cache->txt, "\t%s\t", flags);
}
#if HTS_USEZLIB
back[p].r.compressed = 0;
#endif
if (back[p].r.statuscode == HTTP_OK) {
if (back[p].r.size >= 0) {
@@ -2112,17 +2068,6 @@ int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char
"error: forbidden test with ftp link for back_add");
return -1; // erreur pas de test permis
}
// the ftp client dials the origin itself: over socks that would bypass
// the proxy, so fail the link rather than leak the connection (#563)
if (back[p].r.req.proxy.active &&
hts_proxy_is_socks(back[p].r.req.proxy.name)) {
back[p].r.statuscode = STATUSCODE_NON_FATAL;
strcpybuff(back[p].r.msg,
"ftp:// is not supported over a SOCKS proxy");
back[p].status = STATUS_READY;
back_set_finished(sback, p);
return 0;
}
if (!(back[p].r.req.proxy.active && opt->ftp_proxy)) { // connexion directe, gérée en thread
FTPDownloadStruct *str =
(FTPDownloadStruct *) malloc(sizeof(FTPDownloadStruct));
@@ -2725,40 +2670,13 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
}
busy_state = 1;
// socks5: tunnel to the origin before anything is written, for http
// as well as https. Skip on a reused keep-alive socket (already
// tunneled) and on the post-TLS re-entry (ssl_con set) (#563).
if (back[i].r.req.proxy.active &&
hts_proxy_is_socks(back[i].r.req.proxy.name) &&
!back[i].r.keep_alive
#if HTS_USEOPENSSL
&& back[i].r.ssl_con == NULL
#endif
) {
const int timeout = back[i].timeout > 0 ? back[i].timeout : 30;
if (!socks5_handshake(opt, &back[i].r, back[i].url_adr, timeout)) {
if (!strnotempty(back[i].r.msg))
strcpybuff(back[i].r.msg, "SOCKS5 handshake failed");
deletehttp(&back[i].r);
back[i].r.soc = INVALID_SOCKET;
back[i].r.statuscode = STATUSCODE_NON_FATAL;
back[i].status = STATUS_READY;
back_set_finished(sback, i);
continue;
}
}
#if HTS_USEOPENSSL
/* SSL mode */
if (back[i].r.ssl) {
int tunnel_ok = 1;
// https via an http proxy: CONNECT-tunnel before TLS (#85); socks
// already carries the origin connection
if (back[i].r.req.proxy.active &&
!hts_proxy_is_socks(back[i].r.req.proxy.name) &&
back[i].r.ssl_con == NULL) {
// https via proxy: CONNECT-tunnel before TLS (#85)
if (back[i].r.req.proxy.active && back[i].r.ssl_con == NULL) {
const int timeout = back[i].timeout > 0 ? back[i].timeout : 30;
tunnel_ok =
@@ -3023,13 +2941,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
int last_errno = 0;
back[i].r.is_write = 1; // écrire
/* a .gz/.br/.zst saved under its own coding stays
packed on disk */
if (back[i].r.compressed &&
!hts_codec_is_archive_ext(
hts_codec_parse(back[i].r.contentencoding),
get_ext(catbuff, sizeof(catbuff),
back[i].url_sav))) {
/* .gz are *NOT* depacked!! */
strfield(get_ext(catbuff, sizeof(catbuff), back[i].url_sav), "gz") == 0
&& strfield(get_ext(catbuff, sizeof(catbuff), back[i].url_sav), "tgz") == 0
) {
if (create_back_tmpfile(opt, &back[i], "z") ==
0) {
assertf(back[i].tmpfile != NULL);

View File

@@ -232,10 +232,11 @@ static int disk_fallback_selftest(httrackp *opt) {
static const char body[] = "BINARY-on-disk-body-0123456789-no-trailing-nul";
const size_t body_len = sizeof(body) - 1;
/* a DOS-ified X-Save loses the path_html_utf8 prefix the reader matches on,
and gets re-rooted as a pre-3.40 relative path; fconv() only on access */
concat(save, sizeof(save), StringBuff(opt->path_html_utf8),
"example.com/blob.bin");
/* X-Save must start with path_html_utf8 so the reader resolves it verbatim
(otherwise it re-roots it as a pre-3.40 relative path); then the body we
create at fconv(save) is exactly where cache_readex looks for it. */
fconcat(save, sizeof(save), StringBuff(opt->path_html_utf8),
"example.com/blob.bin");
/* write only the header (X-In-Cache: 0); the body stays on disk */
selftest_open_for_write(&cache, opt);
@@ -1252,9 +1253,8 @@ static void corrupt_build_disk(httrackp *opt) {
memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
remove(reconcile_st_path(opt, "hts-cache/new.zip"));
remove(reconcile_st_path(opt, "hts-cache/old.zip"));
/* X-Save stored verbatim: see disk_fallback_selftest */
concat(save, sizeof(save), StringBuff(opt->path_html_utf8),
CORRUPT_ADR "/victim.bin");
fconcat(save, sizeof(save), StringBuff(opt->path_html_utf8),
CORRUPT_ADR "/victim.bin");
selftest_open_for_write(&cache, opt);
store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
"OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,

View File

@@ -1,366 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: HTTP content codings (gzip/deflate, brotli, zstd) */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
/* Internal engine bytecode */
#define HTS_INTERNAL_BYTECODE
#include "htsbase.h"
#include "htscore.h"
#include "htscodec.h"
#include "htszlib.h"
#if HTS_USEBROTLI
#include <brotli/decode.h>
#endif
#if HTS_USEZSTD
#include <zstd.h>
/* 8 MB, the window an HTTP zstd decoder must honor (RFC 9659); libzstd
defaults to 128 MB. */
#define HTS_ZSTD_WINDOWLOG_MAX 23
#endif
/* Decoded-size budget, bounding a bomb: brotli and zstd reach a million to one.
Deflate cannot pass 1032x, so it only meets this at the 2 GiB ceiling. */
#define HTS_CODEC_MAX_RATIO 4096
#define HTS_CODEC_MIN_MAXOUT (1024 * 1024)
LLint hts_codec_maxout(LLint in_size) {
LLint maxout;
if (in_size <= 0 || in_size > INT_MAX / HTS_CODEC_MAX_RATIO)
maxout = INT_MAX;
else
maxout = in_size * HTS_CODEC_MAX_RATIO;
if (maxout < HTS_CODEC_MIN_MAXOUT)
maxout = HTS_CODEC_MIN_MAXOUT;
return maxout;
}
hts_codec hts_codec_parse(const char *encoding) {
if (encoding == NULL || encoding[0] == '\0' ||
strfield2(encoding, "identity"))
return HTS_CODEC_IDENTITY;
if (strfield2(encoding, "gzip") || strfield2(encoding, "x-gzip") ||
strfield2(encoding, "deflate") || strfield2(encoding, "x-deflate"))
return HTS_USEZLIB ? HTS_CODEC_DEFLATE : HTS_CODEC_UNSUPPORTED;
if (strfield2(encoding, "br"))
return HTS_USEBROTLI ? HTS_CODEC_BROTLI : HTS_CODEC_UNSUPPORTED;
if (strfield2(encoding, "zstd"))
return HTS_USEZSTD ? HTS_CODEC_ZSTD : HTS_CODEC_UNSUPPORTED;
if (strfield2(encoding, "compress") || strfield2(encoding, "x-compress"))
return HTS_CODEC_UNSUPPORTED; /* LZW: never advertised, no decoder here */
/* Not a coding at all: broken servers put charsets and the like here, and
the plain body they sent must survive it. */
return HTS_CODEC_IDENTITY;
}
#if HTS_USEBROTLI
#define HTS_AE_BROTLI ", br"
#else
#define HTS_AE_BROTLI ""
#endif
#if HTS_USEZSTD
#define HTS_AE_ZSTD ", zstd"
#else
#define HTS_AE_ZSTD ""
#endif
const char *hts_acceptencoding(hts_boolean compressible, hts_boolean secure) {
if (!compressible)
return "identity";
#if HTS_USEZLIB
/* br and zstd over TLS only, as browsers do: a cleartext intermediary that
rewrites a coding it can not read would corrupt the mirror. */
if (secure)
return "gzip, deflate" HTS_AE_BROTLI HTS_AE_ZSTD ", identity;q=0.9";
return "gzip, deflate, identity;q=0.9";
#else
(void) secure;
return "identity";
#endif
}
hts_boolean hts_codec_is_archive_ext(hts_codec codec, const char *ext) {
if (ext == NULL || ext[0] == '\0')
return HTS_FALSE;
switch (codec) {
case HTS_CODEC_DEFLATE:
return strfield2(ext, "gz") || strfield2(ext, "tgz") ? HTS_TRUE : HTS_FALSE;
case HTS_CODEC_BROTLI:
return strfield2(ext, "br") ? HTS_TRUE : HTS_FALSE;
case HTS_CODEC_ZSTD:
return strfield2(ext, "zst") || strfield2(ext, "tzst") ? HTS_TRUE
: HTS_FALSE;
default:
return HTS_FALSE;
}
}
#if HTS_USEBROTLI || HTS_USEZSTD
/* Append produced bytes to out under the decoded-size budget, advancing *total.
HTS_FALSE on a short write or once the budget is exceeded (a bomb); the
over-budget bytes are never written. */
static hts_boolean codec_sink(FILE *out, const void *buf, size_t produced,
LLint *total, LLint maxout) {
if (produced == 0)
return HTS_TRUE;
*total += (LLint) produced;
if (*total > maxout)
return HTS_FALSE;
return fwrite(buf, 1, produced, out) == produced ? HTS_TRUE : HTS_FALSE;
}
#endif
#if HTS_USEBROTLI
static int codec_unpack_brotli(FILE *in, FILE *out, LLint maxout) {
BrotliDecoderState *const state =
BrotliDecoderCreateInstance(NULL, NULL, NULL);
hts_boolean ok = HTS_TRUE, done = HTS_FALSE;
LLint total = 0;
if (state == NULL)
return -1;
while (ok && !done) {
unsigned char BIGSTK inbuf[8192];
size_t avail_in = fread(inbuf, 1, sizeof(inbuf), in);
const uint8_t *next_in = inbuf;
if (avail_in == 0)
break; /* EOF; a stream still hungry for input is truncated */
for (;;) {
unsigned char BIGSTK outbuf[8192];
uint8_t *next_out = outbuf;
size_t avail_out = sizeof(outbuf);
size_t produced;
BrotliDecoderResult res;
res = BrotliDecoderDecompressStream(state, &avail_in, &next_in,
&avail_out, &next_out, NULL);
produced = sizeof(outbuf) - avail_out;
if (!codec_sink(out, outbuf, produced, &total, maxout)) {
ok = HTS_FALSE;
break;
}
if (res == BROTLI_DECODER_RESULT_ERROR) {
ok = HTS_FALSE;
break;
}
if (res == BROTLI_DECODER_RESULT_SUCCESS) {
done = HTS_TRUE;
break;
}
if (res == BROTLI_DECODER_RESULT_NEEDS_MORE_INPUT)
break;
}
if (ferror(in))
ok = HTS_FALSE;
}
BrotliDecoderDestroyInstance(state);
return (ok && done && !ferror(in)) ? (int) total : -1;
}
static size_t codec_head_brotli(const void *in, size_t in_len, void *out,
size_t out_len) {
BrotliDecoderState *const state =
BrotliDecoderCreateInstance(NULL, NULL, NULL);
const uint8_t *next_in = (const uint8_t *) in;
uint8_t *next_out = (uint8_t *) out;
size_t avail_in = in_len, avail_out = out_len;
BrotliDecoderResult res;
if (state == NULL)
return 0;
res = BrotliDecoderDecompressStream(state, &avail_in, &next_in, &avail_out,
&next_out, NULL);
BrotliDecoderDestroyInstance(state);
return (res != BROTLI_DECODER_RESULT_ERROR) ? out_len - avail_out : 0;
}
#endif
#if HTS_USEZSTD
static int codec_unpack_zstd(FILE *in, FILE *out, LLint maxout) {
ZSTD_DStream *const zds = ZSTD_createDStream();
hts_boolean ok = HTS_TRUE, eof = HTS_FALSE;
size_t zret = 1; /* 0 once a frame is fully flushed */
LLint total = 0;
if (zds == NULL)
return -1;
if (ZSTD_isError(ZSTD_DCtx_setParameter(zds, ZSTD_d_windowLogMax,
HTS_ZSTD_WINDOWLOG_MAX))) {
ZSTD_freeDStream(zds);
return -1;
}
while (ok && !eof) {
unsigned char BIGSTK inbuf[8192];
ZSTD_inBuffer input;
input.src = inbuf;
input.size = fread(inbuf, 1, sizeof(inbuf), in);
input.pos = 0;
if (input.size == 0) {
eof = HTS_TRUE;
break;
}
while (input.pos < input.size) {
unsigned char BIGSTK outbuf[8192];
ZSTD_outBuffer output;
output.dst = outbuf;
output.size = sizeof(outbuf);
output.pos = 0;
zret = ZSTD_decompressStream(zds, &output, &input);
if (ZSTD_isError(zret)) {
ok = HTS_FALSE;
break;
}
if (!codec_sink(out, outbuf, output.pos, &total, maxout)) {
ok = HTS_FALSE;
break;
}
}
if (ferror(in))
ok = HTS_FALSE;
}
ZSTD_freeDStream(zds);
/* zret != 0 at EOF: the last frame never completed */
return (ok && zret == 0 && !ferror(in)) ? (int) total : -1;
}
static size_t codec_head_zstd(const void *in, size_t in_len, void *out,
size_t out_len) {
ZSTD_DStream *const zds = ZSTD_createDStream();
ZSTD_inBuffer input;
ZSTD_outBuffer output;
size_t zret;
if (zds == NULL)
return 0;
if (ZSTD_isError(ZSTD_DCtx_setParameter(zds, ZSTD_d_windowLogMax,
HTS_ZSTD_WINDOWLOG_MAX))) {
ZSTD_freeDStream(zds);
return 0;
}
input.src = in;
input.size = in_len;
input.pos = 0;
output.dst = out;
output.size = out_len;
output.pos = 0;
zret = ZSTD_decompressStream(zds, &output, &input);
ZSTD_freeDStream(zds);
return ZSTD_isError(zret) ? 0 : output.pos;
}
#endif
LLint hts_codec_coded_size(FILE *in) {
long size;
if (fseek(in, 0, SEEK_END) != 0)
return -1;
size = ftell(in);
if (size < 0 || fseek(in, 0, SEEK_SET) != 0)
return -1;
return (LLint) size;
}
int hts_codec_unpack(hts_codec codec, const char *filename,
const char *newfile) {
if (filename == NULL || newfile == NULL || !filename[0] || !newfile[0])
return -1;
switch (codec) {
case HTS_CODEC_DEFLATE:
#if HTS_USEZLIB
return hts_zunpack(filename, newfile);
#else
return -1;
#endif
case HTS_CODEC_BROTLI:
case HTS_CODEC_ZSTD:
break;
default: /* IDENTITY never reaches the unpacker; UNSUPPORTED must fail */
return -1;
}
#if HTS_USEBROTLI || HTS_USEZSTD
{
char catbuff[CATBUFF_SIZE];
FILE *out, *in = FOPEN(fconv(catbuff, sizeof(catbuff), filename), "rb");
LLint maxout;
int ret = -1;
if (in == NULL)
return -1;
maxout = hts_codec_maxout(hts_codec_coded_size(in));
out = FOPEN(fconv(catbuff, sizeof(catbuff), newfile), "wb");
if (out == NULL) {
fclose(in);
return -1;
}
#if HTS_USEBROTLI
if (codec == HTS_CODEC_BROTLI)
ret = codec_unpack_brotli(in, out, maxout);
#endif
#if HTS_USEZSTD
if (codec == HTS_CODEC_ZSTD)
ret = codec_unpack_zstd(in, out, maxout);
#endif
fclose(out);
fclose(in);
return ret;
}
#else
return -1;
#endif
}
size_t hts_codec_head(hts_codec codec, const void *in, size_t in_len, void *out,
size_t out_len) {
if (in == NULL || in_len == 0 || out == NULL || out_len == 0)
return 0;
switch (codec) {
#if HTS_USEZLIB
case HTS_CODEC_DEFLATE:
return hts_zhead(in, in_len, out, out_len);
#endif
#if HTS_USEBROTLI
case HTS_CODEC_BROTLI:
return codec_head_brotli(in, in_len, out, out_len);
#endif
#if HTS_USEZSTD
case HTS_CODEC_ZSTD:
return codec_head_zstd(in, in_len, out, out_len);
#endif
default:
return 0;
}
}

View File

@@ -1,79 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: HTTP content codings (gzip/deflate, brotli, zstd) */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTS_DEFCODEC
#define HTS_DEFCODEC
#include "htsglobal.h"
/* Content codings we may meet in a Content-Encoding header. */
typedef enum {
HTS_CODEC_IDENTITY = 0, /**< no coding, or a token we treat as none */
HTS_CODEC_DEFLATE, /**< gzip, x-gzip, deflate, x-deflate */
HTS_CODEC_BROTLI, /**< br */
HTS_CODEC_ZSTD, /**< zstd */
HTS_CODEC_UNSUPPORTED /**< a real coding we can not decode */
} hts_codec;
#ifdef HTS_INTERNAL_BYTECODE
/* Codec for a Content-Encoding token. A known coding we can not decode is
UNSUPPORTED; an unrecognized token is IDENTITY (servers do emit junk). */
extern hts_codec hts_codec_parse(const char *encoding);
/* Accept-Encoding value to advertise; secure (TLS) also offers br and zstd. */
extern const char *hts_acceptencoding(hts_boolean compressible,
hts_boolean secure);
/* HTS_TRUE if ext is the codec's own archive extension: a foo.gz served as
Content-Encoding: gzip stays packed on disk. */
extern hts_boolean hts_codec_is_archive_ext(hts_codec codec, const char *ext);
/* Decode filename into newfile. Returns the decoded size, or -1 on a truncated
or corrupt stream, an unsupported coding, a bomb over the budget, or I/O. */
extern int hts_codec_unpack(hts_codec codec, const char *filename,
const char *newfile);
/* Decode up to out_len leading bytes of a coded body for the MIME sniffer;
0 when nothing could be decoded. */
extern size_t hts_codec_head(hts_codec codec, const void *in, size_t in_len,
void *out, size_t out_len);
/* Ceiling on the decoded size of a body of in_size coded bytes. */
extern LLint hts_codec_maxout(LLint in_size);
/* Coded size of an open stream, left rewound; -1 if unknown. */
extern LLint hts_codec_coded_size(FILE *in);
#endif
#endif

View File

@@ -2253,12 +2253,21 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
htsmain_free();
return -1;
} else {
char BIGSTK pname[HTS_URLMAXSIZE * 2];
char *a;
na++;
opt->proxy.active = 1;
hts_parse_proxy(argv[na], pname, sizeof(pname), &opt->proxy.port);
StringCopy(opt->proxy.name, pname);
// Rechercher MAIS en partant de la fin à cause de user:pass@proxy:port
a = argv[na] + strlen(argv[na]) - 1;
while((a > argv[na]) && (*a != ':') && (*a != '@'))
a--;
if (*a == ':') { // un port est présent, <proxy>:port
sscanf(a + 1, "%d", &opt->proxy.port);
StringCopyN(opt->proxy.name, argv[na], (int) (a - argv[na]));
} else { // <proxy>
opt->proxy.port = 8080;
StringCopy(opt->proxy.name, argv[na]);
}
}
break;
case 'F': // user-agent field

View File

@@ -140,15 +140,6 @@ Please visit our Website: http://www.httrack.com
#define HTS_USEZLIB 1
#endif
// brotli and zstd content codings; off unless the build opted in (configure,
// or the Visual Studio projects, which link the vcpkg libraries)
#ifndef HTS_USEBROTLI
#define HTS_USEBROTLI 0
#endif
#ifndef HTS_USEZSTD
#define HTS_USEZSTD 0
#endif
#ifndef HTS_INET6
#define HTS_INET6 0
#endif

View File

@@ -477,7 +477,7 @@ void help(const char *app, int more) {
(" Y mirror ALL links located in the first level pages (mirror links)");
infomsg("");
infomsg("Proxy options:");
infomsg(" P proxy use (-P [socks5://][user:pass@]proxy:port)");
infomsg(" P proxy use (-P proxy:port or -P user:pass@proxy:port)");
infomsg(" %f *use proxy for ftp (f0 don't use)");
infomsg(" %b use this local hostname to make/send requests (-%b hostname)");
infomsg("");

View File

@@ -48,7 +48,6 @@ Please visit our Website: http://www.httrack.com
#include "htsmodules.h"
#include "htscharset.h"
#include "htsencoding.h"
#include "htscodec.h"
#ifdef _WIN32
#include <direct.h>
@@ -649,6 +648,165 @@ T_SOC http_fopen(httrackp * opt, const char *adr, const char *fil, htsblk * reto
return http_xfopen(opt, 0, 1, 1, NULL, adr, fil, retour);
}
// Read a CRLF line from a non-blocking socket (waits up to timeout per recv).
// Returns the line length (0 = empty), or -1 on timeout/EOF/error.
static int proxy_getline(T_SOC soc, char *s, int max, int timeout) {
int j = 0;
for (;;) {
unsigned char ch;
int n;
if (!check_readinput_t(soc, timeout))
return -1; // timed out waiting for data
n = (int) recv(soc, &ch, 1, 0);
if (n == 1) {
if (ch == 13) // CR
continue;
if (ch == 10) // LF: end of line
break;
if (j >= max - 1)
return -1; // line too long: bound the read against a hostile proxy
s[j++] = (char) ch;
} else if (n == 0) {
return -1; // connection closed
} else {
#ifdef _WIN32
if (WSAGetLastError() == WSAEWOULDBLOCK)
continue;
#else
if (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK)
continue;
#endif
return -1;
}
}
s[j] = '\0';
return j;
}
int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
int timeout) {
const T_SOC soc = retour->soc;
const char *const host = jump_identification_const(adr); // host[:port]
const char *const portsep = jump_toport_const(adr); // ":port" or NULL
char BIGSTK authority[HTS_URLMAXSIZE * 2];
char BIGSTK req[HTS_URLMAXSIZE * 4 + 1100];
char line[1024];
int code;
if (soc == INVALID_SOCKET)
return 0;
// CONNECT needs an explicit host:port; default the https port
authority[0] = '\0';
if (portsep != NULL)
strlcatbuff(authority, host, sizeof(authority)); // already host:port
else
snprintf(authority, sizeof(authority), "%s:%d", host, 443);
// backstop: never let a stray CR/LF in the host smuggle a second line into
// the CONNECT request (the host is already sanitized upstream)
{
const char *c;
for (c = authority; *c != '\0'; c++) {
if ((unsigned char) *c < ' ') {
strcpybuff(retour->msg, "proxy CONNECT: invalid host");
return 0;
}
}
}
snprintf(req, sizeof(req), "CONNECT %s HTTP/1.0" H_CRLF "Host: %s" H_CRLF,
authority, authority);
// creds go on the CONNECT, not the tunneled origin request
if (link_has_authorization(retour->req.proxy.name)) {
const char *a = jump_identification_const(retour->req.proxy.name);
const char *astart = jump_protocol_const(retour->req.proxy.name);
char autorisation[1100];
char user_pass[256];
autorisation[0] = user_pass[0] = '\0';
strncatbuff(user_pass, astart, (int) (a - astart) - 1);
strcpybuff(user_pass, unescape_http(OPT_GET_BUFF(opt),
OPT_GET_BUFF_SIZE(opt), user_pass));
code64((unsigned char *) user_pass, (int) strlen(user_pass),
(unsigned char *) autorisation, 0);
strlcatbuff(req, "Proxy-Authorization: Basic ", sizeof(req));
strlcatbuff(req, autorisation, sizeof(req));
strlcatbuff(req, H_CRLF, sizeof(req));
}
strlcatbuff(req, H_CRLF, sizeof(req)); // end of request headers
// raw send: ssl is set, so sendc() would route to TLS
{
const char *p = req;
size_t remain = strlen(req);
int stalls = 0;
while (remain > 0) {
const int n = (int) send(soc, p, (int) remain, 0);
if (n > 0) {
p += n;
remain -= (size_t) n;
stalls = 0;
} else {
#ifdef _WIN32
const int wouldblock = (WSAGetLastError() == WSAEWOULDBLOCK);
#else
const int wouldblock =
(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR);
#endif
// don't spin forever on a fatal error or an unwritable socket
if (!wouldblock || !check_writeinput_t(soc, timeout) ||
++stalls > 100) {
strcpybuff(retour->msg, "proxy CONNECT: write error");
return 0;
}
}
}
}
// proxy status line: "HTTP/1.x <code> ..."
if (proxy_getline(soc, line, sizeof(line), timeout) < 0) {
strcpybuff(retour->msg, "proxy CONNECT: no response");
return 0;
}
if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
code = 0;
if (code < 200 || code >= 300) {
snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
return 0;
}
// drain headers to the blank line; cap the count so a flooding proxy can't
// stall the crawl
{
int headers = 0;
for (;;) {
const int n = proxy_getline(soc, line, sizeof(line), timeout);
if (n < 0) {
strcpybuff(retour->msg, "proxy CONNECT: truncated response");
return 0;
}
if (n == 0)
break; // blank line: tunnel ready
if (++headers > 64) {
strcpybuff(retour->msg, "proxy CONNECT: too many response headers");
return 0;
}
}
}
return 1;
}
// ouverture d'une liaison http, envoi d'une requète
// mode: 0 GET 1 HEAD [2 POST]
// treat: traiter header?
@@ -947,9 +1105,8 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
/* widths bound method[256], url[HTS_URLMAXSIZE*2], protocol[256] */
if (sscanf(line, "%255s %2047s %255s", method, url, protocol) == 3) {
size_t ret;
// absolute-URI for an http proxy; a socks tunnel takes origin-form
if (retour->req.proxy.active &&
!hts_proxy_is_socks(retour->req.proxy.name)) {
// selon que l'on a ou pas un proxy
if (retour->req.proxy.active) {
print_buffer(&bstr,
"%s http://%s%s %s\r\n", method, adr, url,
protocol);
@@ -985,10 +1142,8 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
print_buffer(&bstr, "HEAD ");
}
// an http proxy needs an absolute URI; a socks tunnel does not
if (retour->req.proxy.active &&
!hts_proxy_is_socks(retour->req.proxy.name) &&
(strncmp(adr, "https://", 8) != 0)) {
// si on gère un proxy, il faut une Absolute URI: on ajoute avant http://www.adr.dom
if (retour->req.proxy.active && (strncmp(adr, "https://", 8) != 0)) {
if (!link_has_authority(adr)) { // default http
#if HDEBUG
printf("Proxy Use: for %s%s proxy %d port %d\n", adr, fil,
@@ -1032,11 +1187,8 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
if (xsend)
print_buffer(&bstr, "%s", xsend); // éventuelles autres lignes
// for https, auth rides the CONNECT (the tunneled GET would leak it); for
// socks, the handshake
if (retour->req.proxy.active &&
!hts_proxy_is_socks(retour->req.proxy.name) &&
strncmp(adr, "https://", 8) != 0) {
// for https, auth rides the CONNECT (the tunneled GET would leak it)
if (retour->req.proxy.active && strncmp(adr, "https://", 8) != 0) {
if (link_has_authorization(retour->req.proxy.name)) { // et hop, authentification proxy!
const char *a = jump_identification_const(retour->req.proxy.name);
const char *astart = jump_protocol_const(retour->req.proxy.name);
@@ -1127,16 +1279,11 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
// Compression accepted ?
if (retour->req.http11) {
hts_boolean compressible = HTS_FALSE;
hts_boolean secure = HTS_FALSE;
#if HTS_USEZLIB
compressible = (!retour->req.range_used && !retour->req.nocompression);
#endif
#if HTS_USEOPENSSL
secure = retour->ssl ? HTS_TRUE : HTS_FALSE;
#endif
print_buffer(&bstr, "Accept-Encoding: %s" H_CRLF,
hts_acceptencoding(compressible, secure));
hts_acceptencoding(compressible));
}
/* Authentification */
@@ -1521,10 +1668,22 @@ void treathead(t_cookie * cookie, const char *adr, const char *fil, htsblk * ret
strcpybuff(retour->contentencoding, tempo);
else
retour->contentencoding[0] = '\0';
/* A coding to undo (or one we can not undo, which must fail the fetch
rather than save the coded bytes as the page) */
if (hts_codec_parse(retour->contentencoding) != HTS_CODEC_IDENTITY)
retour->compressed = 1;
#if HTS_USEZLIB
/* Check known encodings */
if (retour->contentencoding[0]) {
if ((strfield2(retour->contentencoding, "gzip"))
|| (strfield2(retour->contentencoding, "x-gzip"))
/*
|| (strfield2(retour->contentencoding, "compress"))
|| (strfield2(retour->contentencoding, "x-compress"))
*/
|| (strfield2(retour->contentencoding, "deflate"))
|| (strfield2(retour->contentencoding, "x-deflate"))
) {
retour->compressed = 1;
}
}
#endif
}
} else if ((p = strfield(rcvd, "Location:")) != 0) {
if (retour) {
@@ -3713,10 +3872,6 @@ const char *jump_protocol_const(const char *source) {
source += p;
else if ((p = strfield(source, "file:")))
source += p;
else if ((p = strfield(source, "socks5h:")))
source += p;
else if ((p = strfield(source, "socks5:")))
source += p;
// net_path
if (strncmp(source, "//", 2) == 0)
source += 2;
@@ -3725,48 +3880,6 @@ const char *jump_protocol_const(const char *source) {
DECLARE_NON_CONST_VERSION(jump_protocol)
hts_boolean hts_proxy_is_socks(const char *name) {
if (name == NULL)
return HTS_FALSE;
return (strfield(name, "socks5h:") || strfield(name, "socks5:")) ? HTS_TRUE
: HTS_FALSE;
}
// default proxy port for a -P argument, keyed on the scheme
static int proxy_default_port(const char *arg) {
return hts_proxy_is_socks(arg) ? 1080 : 8080;
}
void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port) {
const char *authority = strstr(arg, "://");
const char *a;
size_t namelen;
if (name_size == 0)
return;
// scan back to the port ':' (or userinfo '@'), but never past the authority,
// so a scheme's own colon is not read as a port separator; inspect a[-1] from
// one-past-end so no pointer below the string is ever formed
authority = (authority != NULL) ? authority + 3 : arg;
a = arg + strlen(arg);
while (a > authority && a[-1] != ':' && a[-1] != '@')
a--;
if (a > authority && a[-1] == ':') {
int p = -1;
sscanf(a, "%d", &p);
*port = (p > 0) ? p : proxy_default_port(arg);
namelen = (size_t) (a - 1 - arg);
} else {
*port = proxy_default_port(arg);
namelen = strlen(arg);
}
if (namelen >= name_size) // arg is user-controlled: truncate, don't overflow
namelen = name_size - 1;
memcpy(name, arg, namelen);
name[namelen] = '\0';
}
// codage base 64 a vers b
void code64(unsigned char *a, int size_a, unsigned char *b, int crlf) {
int i1 = 0, i2 = 0, i3 = 0, i4 = 0;
@@ -4252,6 +4365,11 @@ HTSEXT_API void get_httptype(httrackp *opt, char *s, const char *fil,
(void) get_httptype_sized(opt, s, HTS_MIMETYPE_SIZE, fil, flag);
}
/* Advertised Accept-Encoding; gzip and deflate both decode via hts_zunpack */
const char *hts_acceptencoding(hts_boolean compressible) {
return compressible ? "gzip, deflate, identity;q=0.9" : "identity";
}
// get type of fil (php)
// s: buffer (text/html) or NULL
// return: 1 if known by user

View File

@@ -207,9 +207,15 @@ int check_readinput(htsblk * r);
int check_readinput_t(T_SOC soc, int timeout);
int check_writeinput_t(T_SOC soc, int timeout);
/* TRUE if this -P proxy name (which keeps its scheme) is a SOCKS5 proxy. */
hts_boolean hts_proxy_is_socks(const char *name);
/* Open an HTTP CONNECT tunnel through the active proxy for an https request:
`retour->soc` must already be TCP-connected to the proxy, and `adr` is the
origin authority (url_adr, e.g. "https://host:port"). Sends the CONNECT
request (with Proxy-Authorization when the proxy carries credentials) and
reads the proxy's status line, so the caller's TLS handshake then runs
end-to-end with the origin. Blocks up to `timeout` seconds. Returns 1 on a
2xx tunnel, 0 on failure (retour->msg/statuscode set). */
int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
int timeout);
void treathead(t_cookie * cookie, const char *adr, const char *fil, htsblk * retour,
char *rcvd);
void treatfirstline(htsblk * retour, const char *rcvd);
@@ -277,18 +283,15 @@ int ishttperror(int err);
int get_userhttptype(httrackp * opt, char *s, const char *fil);
int give_mimext(char *s, size_t ssize, const char *st);
/* Advertised Accept-Encoding value (no header name/CRLF); see htslib.c. */
const char *hts_acceptencoding(hts_boolean compressible);
int may_bogus_multiple(httrackp * opt, const char *mime, const char *filename);
int may_unknown2(httrackp * opt, const char *mime, const char *filename);
const char *strrchr_limit(const char *s, char c, const char *limit);
char *jump_protocol(char *source);
const char *jump_protocol_const(const char *source);
/* Split a -P proxy argument "[scheme://][user:pass@]host[:port]" into the proxy
host string (scheme and any user:pass kept, for later stripping and auth),
written NUL-terminated into name[name_size] (truncated to fit), and the port
in *port. The port defaults by scheme: 1080 for socks5/socks5h, else 8080. */
void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port);
void code64(unsigned char *a, int size_a, unsigned char *b, int crlf);
#define copychar(catbuff,a) concat(catbuff,(a),NULL)

View File

@@ -42,7 +42,6 @@ Please visit our Website: http://www.httrack.com
#include "htscharset.h"
#include "htsencoding.h"
#include "htssniff.h"
#include "htscodec.h"
#if HTS_USEZLIB
#include "htszlib.h"
#endif
@@ -152,6 +151,29 @@ typedef struct sniff_src {
const char *prev_save; /* previous run's save name (cache X-Save) */
} sniff_src;
#if HTS_USEZLIB
/* Inflate the head of a gzip/zlib stream; 0 when undecodable. */
static size_t sniff_inflate_head(const void *in, size_t in_len, void *out,
size_t out_len) {
z_stream zs;
size_t n = 0;
int err;
memset(&zs, 0, sizeof(zs));
if (inflateInit2(&zs, 47) != Z_OK) /* 47: gzip or zlib, autodetected */
return 0;
zs.next_in = (const Bytef *) in;
zs.avail_in = (uInt) in_len;
zs.next_out = (Bytef *) out;
zs.avail_out = (uInt) out_len;
err = inflate(&zs, Z_SYNC_FLUSH);
if (err == Z_OK || err == Z_STREAM_END || err == Z_BUF_ERROR)
n = out_len - zs.avail_out;
inflateEnd(&zs);
return n;
}
#endif
static size_t sniff_read_head(const char *path, void *buf, size_t len) {
char catbuff[CATBUFF_SIZE];
FILE *const fp = FOPEN(fconv(catbuff, sizeof(catbuff), path), "rb");
@@ -183,12 +205,16 @@ static size_t sniff_slot_head(const lien_back *slot, void *buf, size_t len) {
n = sniff_read_head(slot->tmpfile, buf, len);
}
if (n > 0 && r->compressed) {
#if HTS_USEZLIB
unsigned char raw[HTS_SNIFF_LEN];
if (n > sizeof(raw))
n = sizeof(raw);
memcpy(raw, buf, n);
n = hts_codec_head(hts_codec_parse(r->contentencoding), raw, n, buf, len);
n = sniff_inflate_head(raw, n, buf, len);
#else
n = 0;
#endif
}
return n;
}

View File

@@ -328,8 +328,8 @@ typedef enum hts_wizard {
typedef enum hts_robots {
HTS_ROBOTS_NEVER = 0, /**< ignore robots rules */
HTS_ROBOTS_SOMETIMES = 1, /**< partial obedience, set by -s */
HTS_ROBOTS_ALWAYS = 2, /**< obey robots rules (default) */
HTS_ROBOTS_SOMETIMES = 1, /**< partial obedience (default) */
HTS_ROBOTS_ALWAYS = 2, /**< obey robots rules */
HTS_ROBOTS_ALWAYS_STRICT = 3 /**< obey even strict rules */
} hts_robots;
#endif

View File

@@ -1,539 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: Proxy tunneling (HTTP CONNECT, SOCKS5) */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
/* Internal engine bytecode */
#define HTS_INTERNAL_BYTECODE
#include "htscore.h"
#include "htslib.h"
#include "htsproxy.h"
#include <string.h>
// Read a CRLF line from a non-blocking socket (waits up to timeout per recv).
// Returns the line length (0 = empty), or -1 on timeout/EOF/error.
static int proxy_getline(T_SOC soc, char *s, int max, int timeout) {
int j = 0;
for (;;) {
unsigned char ch;
int n;
if (!check_readinput_t(soc, timeout))
return -1; // timed out waiting for data
n = (int) recv(soc, &ch, 1, 0);
if (n == 1) {
if (ch == 13) // CR
continue;
if (ch == 10) // LF: end of line
break;
if (j >= max - 1)
return -1; // line too long: bound the read against a hostile proxy
s[j++] = (char) ch;
} else if (n == 0) {
return -1; // connection closed
} else {
#ifdef _WIN32
if (WSAGetLastError() == WSAEWOULDBLOCK)
continue;
#else
if (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK)
continue;
#endif
return -1;
}
}
s[j] = '\0';
return j;
}
int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
int timeout) {
const T_SOC soc = retour->soc;
const char *const host = jump_identification_const(adr); // host[:port]
const char *const portsep = jump_toport_const(adr); // ":port" or NULL
char BIGSTK authority[HTS_URLMAXSIZE * 2];
char BIGSTK req[HTS_URLMAXSIZE * 4 + 1100];
char line[1024];
int code;
if (soc == INVALID_SOCKET)
return 0;
// CONNECT needs an explicit host:port; default the https port
authority[0] = '\0';
if (portsep != NULL)
strlcatbuff(authority, host, sizeof(authority)); // already host:port
else
snprintf(authority, sizeof(authority), "%s:%d", host, 443);
// backstop: never let a stray CR/LF in the host smuggle a second line into
// the CONNECT request (the host is already sanitized upstream)
{
const char *c;
for (c = authority; *c != '\0'; c++) {
if ((unsigned char) *c < ' ') {
strcpybuff(retour->msg, "proxy CONNECT: invalid host");
return 0;
}
}
}
snprintf(req, sizeof(req), "CONNECT %s HTTP/1.0" H_CRLF "Host: %s" H_CRLF,
authority, authority);
// creds go on the CONNECT, not the tunneled origin request
if (link_has_authorization(retour->req.proxy.name)) {
const char *a = jump_identification_const(retour->req.proxy.name);
const char *astart = jump_protocol_const(retour->req.proxy.name);
char autorisation[1100];
char user_pass[256];
autorisation[0] = user_pass[0] = '\0';
strncatbuff(user_pass, astart, (int) (a - astart) - 1);
strcpybuff(user_pass, unescape_http(OPT_GET_BUFF(opt),
OPT_GET_BUFF_SIZE(opt), user_pass));
code64((unsigned char *) user_pass, (int) strlen(user_pass),
(unsigned char *) autorisation, 0);
strlcatbuff(req, "Proxy-Authorization: Basic ", sizeof(req));
strlcatbuff(req, autorisation, sizeof(req));
strlcatbuff(req, H_CRLF, sizeof(req));
}
strlcatbuff(req, H_CRLF, sizeof(req)); // end of request headers
// raw send: ssl is set, so sendc() would route to TLS
{
const char *p = req;
size_t remain = strlen(req);
int stalls = 0;
while (remain > 0) {
const int n = (int) send(soc, p, (int) remain, 0);
if (n > 0) {
p += n;
remain -= (size_t) n;
stalls = 0;
} else {
#ifdef _WIN32
const int wouldblock = (WSAGetLastError() == WSAEWOULDBLOCK);
#else
const int wouldblock =
(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR);
#endif
// don't spin forever on a fatal error or an unwritable socket
if (!wouldblock || !check_writeinput_t(soc, timeout) ||
++stalls > 100) {
strcpybuff(retour->msg, "proxy CONNECT: write error");
return 0;
}
}
}
}
// proxy status line: "HTTP/1.x <code> ..."
if (proxy_getline(soc, line, sizeof(line), timeout) < 0) {
strcpybuff(retour->msg, "proxy CONNECT: no response");
return 0;
}
if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
code = 0;
if (code < 200 || code >= 300) {
snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
return 0;
}
// drain headers to the blank line; cap the count so a flooding proxy can't
// stall the crawl
{
int headers = 0;
for (;;) {
const int n = proxy_getline(soc, line, sizeof(line), timeout);
if (n < 0) {
strcpybuff(retour->msg, "proxy CONNECT: truncated response");
return 0;
}
if (n == 0)
break; // blank line: tunnel ready
if (++headers > 64) {
strcpybuff(retour->msg, "proxy CONNECT: too many response headers");
return 0;
}
}
}
return 1;
}
/* SOCKS5 client (RFC 1928, RFC 1929 auth), hostname mode only: the proxy
resolves the origin name (remote DNS, curl's socks5h). The stream is the
proxy socket, or a scripted buffer under -#test=socks5. */
#define SOCKS5_VERSION 0x05
#define SOCKS5_AUTH_VERSION 0x01
#define SOCKS5_METHOD_NONE 0x00
#define SOCKS5_METHOD_USERPASS 0x02
#define SOCKS5_METHOD_NOACCEPTABLE 0xFF
#define SOCKS5_CMD_CONNECT 0x01
#define SOCKS5_ATYP_IPV4 0x01
#define SOCKS5_ATYP_DOMAIN 0x03
#define SOCKS5_ATYP_IPV6 0x04
#define SOCKS5_MAXFIELD 255 /* one length byte: host, user and password */
typedef struct socks5_stream {
T_SOC soc; /* INVALID_SOCKET when scripted (self-test) */
int timeout;
socks5_test_io *io;
} socks5_stream;
static int socks5_fail(char *msg, size_t msgsize, const char *text) {
if (msgsize != 0)
strlcpybuff(msg, text, msgsize);
return 0;
}
/* Read exactly n bytes, or fail: SOCKS frames are not self-delimiting, so a
short read would desync the stream shared with the origin traffic. */
static int socks5_read_n(socks5_stream *st, unsigned char *buf, size_t n) {
size_t got = 0;
int stalls = 0;
if (st->soc == INVALID_SOCKET) { /* scripted */
socks5_test_io *const io = st->io;
if (n > io->reply_len - io->consumed)
return 0;
memcpy(buf, io->reply + io->consumed, n);
io->consumed += n;
return 1;
}
while (got < n) {
const int r = (int) recv(st->soc, (char *) buf + got, (int) (n - got), 0);
if (r > 0) {
got += (size_t) r;
stalls = 0;
} else if (r == 0) {
return 0; // proxy closed mid-frame
} else {
#ifdef _WIN32
const int wouldblock = (WSAGetLastError() == WSAEWOULDBLOCK);
#else
const int wouldblock =
(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR);
#endif
if (!wouldblock || !check_readinput_t(st->soc, st->timeout) ||
++stalls > 100)
return 0;
}
}
return 1;
}
static int socks5_write_all(socks5_stream *st, const unsigned char *buf,
size_t len) {
size_t remain = len;
int stalls = 0;
if (st->soc == INVALID_SOCKET) { /* scripted */
socks5_test_io *const io = st->io;
if (len > sizeof(io->sent) - io->sent_len)
return 0;
memcpy(io->sent + io->sent_len, buf, len);
io->sent_len += len;
return 1;
}
while (remain > 0) {
// raw send: the socket is still plain here, sendc() would route to TLS
const int n = (int) send(st->soc, (const char *) buf, (int) remain, 0);
if (n > 0) {
buf += n;
remain -= (size_t) n;
stalls = 0;
} else {
#ifdef _WIN32
const int wouldblock = (WSAGetLastError() == WSAEWOULDBLOCK);
#else
const int wouldblock =
(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR);
#endif
if (!wouldblock || !check_writeinput_t(st->soc, st->timeout) ||
++stalls > 100)
return 0;
}
}
return 1;
}
static const char *socks5_rep_message(unsigned char rep) {
switch (rep) {
case 0x01:
return "general failure";
case 0x02:
return "connection not allowed by ruleset";
case 0x03:
return "network unreachable";
case 0x04:
return "host unreachable";
case 0x05:
return "connection refused";
case 0x06:
return "TTL expired";
case 0x07:
return "command not supported";
case 0x08:
return "address type not supported";
default:
return "unknown error";
}
}
/* The reply's BND.ADDR is variable-length: consume exactly what ATYP says, or
the leftover bytes prepend to the first origin (or TLS) read. */
static int socks5_read_reply(socks5_stream *st, char *msg, size_t msgsize) {
unsigned char head[4];
unsigned char addr[SOCKS5_MAXFIELD + 2];
size_t addrlen;
if (!socks5_read_n(st, head, sizeof(head)))
return socks5_fail(msg, msgsize, "SOCKS5: no reply from proxy");
if (head[0] != SOCKS5_VERSION)
return socks5_fail(msg, msgsize, "SOCKS5: bad version in reply");
if (head[1] != 0x00) {
snprintf(msg, msgsize, "SOCKS5 connect failed: %s",
socks5_rep_message(head[1]));
return 0;
}
switch (head[3]) {
case SOCKS5_ATYP_IPV4:
addrlen = 4;
break;
case SOCKS5_ATYP_IPV6:
addrlen = 16;
break;
case SOCKS5_ATYP_DOMAIN: {
unsigned char len;
if (!socks5_read_n(st, &len, 1))
return socks5_fail(msg, msgsize, "SOCKS5: truncated reply");
addrlen = len; // <= 255, always fits addr[]
break;
}
default: // unknown length: the stream cannot be resynchronized
return socks5_fail(msg, msgsize, "SOCKS5: unknown address type in reply");
}
if (addrlen != 0 && !socks5_read_n(st, addr, addrlen))
return socks5_fail(msg, msgsize, "SOCKS5: truncated reply address");
if (!socks5_read_n(st, addr, 2)) // BND.PORT, unused
return socks5_fail(msg, msgsize, "SOCKS5: truncated reply port");
return 1;
}
static int socks5_negotiate(socks5_stream *st, const char *host, size_t hostlen,
int port, const char *user, size_t userlen,
const char *pass, size_t passlen, int want_auth,
char *msg, size_t msgsize) {
unsigned char frame[3 + SOCKS5_MAXFIELD + SOCKS5_MAXFIELD];
unsigned char rep[2];
/* greeting */
frame[0] = SOCKS5_VERSION;
frame[1] = (unsigned char) (want_auth ? 2 : 1);
frame[2] = SOCKS5_METHOD_NONE;
frame[3] = SOCKS5_METHOD_USERPASS;
if (!socks5_write_all(st, frame, (size_t) 2 + frame[1]))
return socks5_fail(msg, msgsize, "SOCKS5: write error");
if (!socks5_read_n(st, rep, sizeof(rep)))
return socks5_fail(msg, msgsize, "SOCKS5: no method reply from proxy");
if (rep[0] != SOCKS5_VERSION)
return socks5_fail(msg, msgsize, "SOCKS5: bad version in method reply");
switch (rep[1]) {
case SOCKS5_METHOD_NONE:
break;
case SOCKS5_METHOD_USERPASS:
if (!want_auth)
return socks5_fail(msg, msgsize,
"SOCKS5: proxy requires authentication, none given");
/* RFC 1929 sub-negotiation; its version byte is 0x01, not 0x05 */
frame[0] = SOCKS5_AUTH_VERSION;
frame[1] = (unsigned char) userlen;
memcpy(frame + 2, user, userlen);
frame[2 + userlen] = (unsigned char) passlen;
memcpy(frame + 3 + userlen, pass, passlen);
if (!socks5_write_all(st, frame, 3 + userlen + passlen))
return socks5_fail(msg, msgsize, "SOCKS5: write error");
if (!socks5_read_n(st, rep, sizeof(rep)))
return socks5_fail(msg, msgsize, "SOCKS5: no authentication reply");
if (rep[1] != 0x00)
return socks5_fail(msg, msgsize, "SOCKS5: authentication failed");
break;
case SOCKS5_METHOD_NOACCEPTABLE:
return socks5_fail(
msg, msgsize,
"SOCKS5: proxy accepts no authentication method we offer");
default:
return socks5_fail(msg, msgsize,
"SOCKS5: proxy selected an unknown method");
}
/* CONNECT to the origin, by name */
frame[0] = SOCKS5_VERSION;
frame[1] = SOCKS5_CMD_CONNECT;
frame[2] = 0x00; // RSV
frame[3] = SOCKS5_ATYP_DOMAIN;
frame[4] = (unsigned char) hostlen;
memcpy(frame + 5, host, hostlen);
frame[5 + hostlen] = (unsigned char) (port >> 8);
frame[6 + hostlen] = (unsigned char) (port & 0xFF);
if (!socks5_write_all(st, frame, 7 + hostlen))
return socks5_fail(msg, msgsize, "SOCKS5: write error");
return socks5_read_reply(st, msg, msgsize);
}
/* Decode the proxy userinfo into the two RFC 1929 fields. Split on the first
colon of the still-escaped string, so a %3A stays inside the username. */
static int socks5_credentials(httrackp *opt, const char *proxy_name, char *user,
size_t user_size, size_t *userlen, char *pass,
size_t pass_size, size_t *passlen, char *msg,
size_t msgsize) {
const char *const a = jump_identification_const(proxy_name); // past the '@'
const char *const astart = jump_protocol_const(proxy_name);
char userinfo[1024];
char *colon;
if (a <= astart || (size_t) (a - astart) - 1 >= sizeof(userinfo))
return socks5_fail(msg, msgsize, "SOCKS5: credentials too long");
userinfo[0] = '\0';
strncatbuff(userinfo, astart, (int) (a - astart) - 1);
colon = strchr(userinfo, ':');
if (colon != NULL)
*colon++ = '\0';
strlcpybuff(
user, unescape_http(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), userinfo),
user_size);
strlcpybuff(pass,
colon != NULL ? unescape_http(OPT_GET_BUFF(opt),
OPT_GET_BUFF_SIZE(opt), colon)
: "",
pass_size);
*userlen = strlen(user);
*passlen = strlen(pass);
// reject, never truncate: a clipped secret would authenticate as another one
if (*userlen > SOCKS5_MAXFIELD || *passlen > SOCKS5_MAXFIELD)
return socks5_fail(msg, msgsize, "SOCKS5: credentials too long");
if (*userlen == 0)
return socks5_fail(msg, msgsize, "SOCKS5: empty proxy username");
return 1;
}
static int socks5_handshake_stream(httrackp *opt, socks5_stream *st,
const char *adr, const char *proxy_name,
int ssl, char *msg, size_t msgsize) {
const char *const host = jump_identification_const(adr);
const char *const portsep = jump_toport_const(adr);
const size_t hostlen =
portsep != NULL ? (size_t) (portsep - host) : strlen(host);
// sized for the whole userinfo: decoding shrinks, so an over-long credential
// lands intact and is rejected below rather than silently clipped
char user[1024];
char pass[1024];
size_t userlen = 0, passlen = 0;
int want_auth = 0;
int port = ssl ? 443 : 80;
size_t i;
if (hostlen == 0 || hostlen > SOCKS5_MAXFIELD)
return socks5_fail(msg, msgsize, "SOCKS5: invalid origin host");
if (host[0] == '[') // ATYP=domain cannot carry an IPv6 literal
return socks5_fail(msg, msgsize,
"SOCKS5: IPv6 literal origin is not supported");
for (i = 0; i < hostlen; i++) {
if ((unsigned char) host[i] < ' ')
return socks5_fail(msg, msgsize, "SOCKS5: invalid origin host");
}
if (portsep != NULL) {
int p = -1;
sscanf(portsep + 1, "%d", &p);
if (p <= 0 || p > 65535)
return socks5_fail(msg, msgsize, "SOCKS5: invalid origin port");
port = p;
}
if (link_has_authorization(proxy_name)) {
if (!socks5_credentials(opt, proxy_name, user, sizeof(user), &userlen, pass,
sizeof(pass), &passlen, msg, msgsize))
return 0;
want_auth = 1;
}
return socks5_negotiate(st, host, hostlen, port, user, userlen, pass, passlen,
want_auth, msg, msgsize);
}
int socks5_handshake(httrackp *opt, htsblk *retour, const char *adr,
int timeout) {
socks5_stream st;
#if HTS_USEOPENSSL
const int ssl = retour->ssl;
#else
const int ssl = 0;
#endif
if (retour->soc == INVALID_SOCKET)
return 0;
st.soc = retour->soc;
st.timeout = timeout;
st.io = NULL;
return socks5_handshake_stream(opt, &st, adr, retour->req.proxy.name, ssl,
retour->msg, sizeof(retour->msg));
}
int socks5_handshake_scripted(httrackp *opt, const char *adr,
const char *proxy_name, socks5_test_io *io) {
socks5_stream st;
st.soc = INVALID_SOCKET;
st.timeout = 0;
st.io = io;
io->consumed = 0;
io->sent_len = 0;
io->msg[0] = '\0';
return socks5_handshake_stream(opt, &st, adr, proxy_name, 0, io->msg,
sizeof(io->msg));
}

View File

@@ -1,82 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: Proxy tunneling (HTTP CONNECT, SOCKS5) .h */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTS_DEFPROXY
#define HTS_DEFPROXY
#include "htsglobal.h"
/* Forward definitions */
#ifndef HTS_DEF_FWSTRUCT_httrackp
#define HTS_DEF_FWSTRUCT_httrackp
typedef struct httrackp httrackp;
#endif
#ifndef HTS_DEF_FWSTRUCT_htsblk
#define HTS_DEF_FWSTRUCT_htsblk
typedef struct htsblk htsblk;
#endif
/* Open an HTTP CONNECT tunnel through the active proxy for an https request:
`retour->soc` must already be TCP-connected to the proxy, and `adr` is the
origin authority (url_adr, e.g. "https://host:port"). Sends the CONNECT
request (with Proxy-Authorization when the proxy carries credentials) and
reads the proxy's status line, so the caller's TLS handshake then runs
end-to-end with the origin. Blocks up to `timeout` seconds. Returns 1 on a
2xx tunnel, 0 on failure (retour->msg/statuscode set). */
int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
int timeout);
/* SOCKS5 (RFC 1928, RFC 1929 auth) handshake on the raw proxy socket, before
any TLS: `retour->soc` must be TCP-connected to the proxy and `adr` is the
origin (url_adr). The origin name is sent verbatim (ATYP=domain), so the
proxy resolves it; a bracketed IPv6 literal origin is rejected. Blocks up to
`timeout` seconds. Returns 1 with the stream positioned on the first origin
byte, 0 on failure (retour->msg set). */
int socks5_handshake(httrackp *opt, htsblk *retour, const char *adr,
int timeout);
/* Self-test hook (-#test=socks5): run the handshake against `reply`, a scripted
server byte stream, instead of a socket. `sent` captures the client frames,
`consumed` the reply bytes eaten (the frame-desync guard). */
typedef struct socks5_test_io {
const unsigned char *reply;
size_t reply_len;
size_t consumed;
unsigned char sent[1024];
size_t sent_len;
char msg[256];
} socks5_test_io;
int socks5_handshake_scripted(httrackp *opt, const char *adr,
const char *proxy_name, socks5_test_io *io);
#endif

View File

@@ -54,14 +54,9 @@ Please visit our Website: http://www.httrack.com
#include "htsftp.h"
#include "htsmd5.h"
#include "htssniff.h"
#include "htscodec.h"
#include "htsproxy.h"
#if HTS_USEZLIB
#include "htszlib.h"
#endif
#if HTS_USEZSTD
#include <zstd.h>
#endif
#include "coucal/coucal.h"
#include <ctype.h>
@@ -1309,162 +1304,6 @@ static int st_identurl(httrackp *opt, int argc, char **argv) {
return 0;
}
static int st_proxyurl(httrackp *opt, int argc, char **argv) {
char BIGSTK name[HTS_URLMAXSIZE * 2];
int port = -1;
(void) opt;
if (argc < 1) {
fprintf(stderr, "proxyurl: needs a proxy argument\n");
return 1;
}
hts_parse_proxy(argv[0], name, sizeof(name), &port);
// host= is what the connect actually resolves (scheme + user:pass stripped)
printf("name=%s port=%d host=%s\n", name, port,
jump_identification_const(name));
return 0;
}
/* Scripted SOCKS5 server: no-auth method reply, then a success CONNECT reply
carrying the given ATYP address, then a sentinel origin byte the handshake
must leave unread (an off-by-one in the reply framing eats it). */
static size_t socks5_reply(unsigned char *buf, unsigned char atyp,
const unsigned char *addr, size_t addrlen) {
size_t n = 0;
buf[n++] = 0x05;
buf[n++] = 0x00; /* method: no authentication */
buf[n++] = 0x05;
buf[n++] = 0x00; /* REP: succeeded */
buf[n++] = 0x00;
buf[n++] = atyp;
memcpy(buf + n, addr, addrlen);
n += addrlen;
buf[n++] = 0x1f;
buf[n++] = 0x90; /* BND.PORT */
buf[n++] = 0xAA; /* sentinel */
return n;
}
#define SOCKS5_SENTINEL_LEFT(io, len) ((io).consumed == (len) - 1)
static int st_socks5(httrackp *opt, int argc, char **argv) {
static const unsigned char v4[4] = {127, 0, 0, 1};
static const unsigned char v6[16] = {0};
static const unsigned char domain[6] = {5, 'p', 'r', 'o', 'x', 'y'};
const char *const proxy = "socks5://127.0.0.1";
unsigned char script[64];
socks5_test_io io;
size_t len;
(void) argc;
(void) argv;
/* each reply address type is drained exactly, sentinel untouched */
len = socks5_reply(script, 0x01, v4, sizeof(v4));
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 1);
assertf(SOCKS5_SENTINEL_LEFT(io, len));
/* the greeting offers no-auth only, and the origin goes out by name, port 80
*/
assertf(io.sent_len == 3 + 7 + 11);
assertf(memcmp(io.sent, "\x05\x01\x00", 3) == 0);
assertf(memcmp(io.sent + 3, "\x05\x01\x00\x03\x0borigin.test\x00\x50", 18) ==
0);
len = socks5_reply(script, 0x04, v6, sizeof(v6));
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 1);
assertf(SOCKS5_SENTINEL_LEFT(io, len));
len = socks5_reply(script, 0x03, domain, sizeof(domain));
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 1);
assertf(SOCKS5_SENTINEL_LEFT(io, len));
/* an unknown address type has no known length: fail, never guess */
len = socks5_reply(script, 0x02, v4, sizeof(v4));
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 0);
/* truncated frames (here: no BND.PORT) fail instead of over-reading */
len = socks5_reply(script, 0x01, v4, sizeof(v4)) - 3;
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 0);
/* explicit origin port is encoded big-endian (8443 = 0x20fb) */
len = socks5_reply(script, 0x01, v4, sizeof(v4));
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test:8443", proxy, &io) == 1);
assertf(memcmp(io.sent + io.sent_len - 2, "\x20\xfb", 2) == 0);
/* credentials: split on the first colon of the escaped userinfo, so %3a stays
inside the username and a colon in the password is not a delimiter */
{
static const unsigned char auth_script[] = {
0x05, 0x02, /* method: user/pass */
0x01, 0x00, /* auth: success */
0x05, 0x00, 0x00, 0x01, /* REP: succeeded, ATYP ipv4 */
127, 0, 0, 1, 0x1f, 0x90, 0xAA};
io.reply = auth_script;
io.reply_len = sizeof(auth_script);
assertf(socks5_handshake_scripted(opt, "origin.test",
"socks5://us%3aer:p:ass@127.0.0.1",
&io) == 1);
assertf(SOCKS5_SENTINEL_LEFT(io, sizeof(auth_script)));
assertf(memcmp(io.sent, "\x05\x02\x00\x02", 4) == 0);
assertf(memcmp(io.sent + 4, "\x01\x05us:er\x05p:ass", 13) == 0);
}
/* a proxy demanding auth we cannot provide, and one refusing every method */
io.reply = (const unsigned char *) "\x05\x02";
io.reply_len = 2;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 0);
io.reply = (const unsigned char *) "\x05\xff";
io.reply_len = 2;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 0);
/* a refused CONNECT is an error, not a tunnel */
io.reply = (const unsigned char
*) "\x05\x00\x05\x05\x00\x01\x7f\x00\x00\x01\x1f\x90";
io.reply_len = 12;
assertf(socks5_handshake_scripted(opt, "origin.test", proxy, &io) == 0);
/* over-long host or credentials are rejected before anything is sent */
{
char host[512];
char name[1024];
size_t i;
host[0] = '\0';
for (i = 0; i < 256; i++)
strcatbuff(host, "a");
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, host, proxy, &io) == 0);
assertf(io.sent_len == 0);
strcpybuff(name, "socks5://user:");
for (i = 0; i < 256; i++)
strcatbuff(name, "p");
strcatbuff(name, "@127.0.0.1");
io.reply = script;
io.reply_len = len;
assertf(socks5_handshake_scripted(opt, "origin.test", name, &io) == 0);
assertf(io.sent_len == 0);
}
printf("socks5 self-test OK\n");
return 0;
}
/* Regression for the one-byte fil[] overflow: a 2047-byte hostless "?"-URL used
to abort in strncat_safe_ when the missing leading '/' pushed fil to 2048. */
static int st_identabs(httrackp *opt, int argc, char **argv) {
@@ -2466,7 +2305,6 @@ static int ae_write_collision(const char *path, const unsigned char *src,
freet(buf);
return ok ? 0 : 1;
}
#endif
/* Write src[0..len) to path as-is; 0 on success. */
static int ae_write_raw(const char *path, const unsigned char *src,
@@ -2500,22 +2338,17 @@ static int ae_check_decoded(const char *path, const unsigned char *expect,
fclose(f);
return (off == len) ? 0 : 1;
}
#endif
/* Accept-Encoding (#450): advertise gzip+deflate; both decode (hts_zunpack) */
static int st_acceptencoding(httrackp *opt, int argc, char **argv) {
const char *off = hts_acceptencoding(HTS_FALSE, HTS_TRUE);
const char *on = hts_acceptencoding(HTS_TRUE, HTS_FALSE);
const char *tls = hts_acceptencoding(HTS_TRUE, HTS_TRUE);
const char *off = hts_acceptencoding(HTS_FALSE);
const char *on = hts_acceptencoding(HTS_TRUE);
(void) opt;
assertf(strcmp(off, "identity") == 0);
assertf(strstr(on, "gzip") != NULL);
assertf(strstr(on, "deflate") != NULL); /* fails on the old gzip-only list */
/* br and zstd ride on TLS only, so a cleartext proxy can not be handed a
coding it may try to rewrite */
assertf(strstr(on, "br") == NULL && strstr(on, "zstd") == NULL);
assertf((strstr(tls, ", br") != NULL) == (HTS_USEBROTLI != 0));
assertf((strstr(tls, "zstd") != NULL) == (HTS_USEZSTD != 0));
#if HTS_USEZLIB
if (argc >= 1) {
static const int windowBits[] = {16 + MAX_WBITS, MAX_WBITS, -MAX_WBITS};
@@ -2603,115 +2436,6 @@ static int st_acceptencoding(httrackp *opt, int argc, char **argv) {
return 0;
}
#if HTS_USEBROTLI
/* No brotli encoder is linked, so the coded bytes are canned: brotli quality 9
over cc_text, and over 4 MiB of 'A' (14 bytes, ~300000x). */
static const unsigned char cc_br_text[] = {
0x1b, 0x43, 0x00, 0x00, 0x44, 0xdd, 0x96, 0xea, 0xe8, 0x22, 0xdd, 0x90,
0xa4, 0x1b, 0x8a, 0xf7, 0x47, 0x0e, 0xc2, 0xc5, 0x3d, 0x09, 0x1b, 0x70,
0xe0, 0x1e, 0x60, 0xa0, 0x8b, 0xcc, 0xbe, 0xcb, 0xb0, 0x31, 0x76, 0x9e,
0xcf, 0x6e, 0x41, 0xb5, 0xe8, 0x2e, 0x56, 0x78, 0x08, 0x1b, 0xfa, 0x08,
0x8a, 0x50, 0x83, 0x4e, 0x62, 0x7f, 0xbf, 0x05, 0xf2, 0x22, 0x8f, 0xdf,
0x28, 0xdc, 0x9f, 0xa9, 0x90, 0x50, 0x37, 0x62, 0x56, 0x4f, 0xa8};
static const unsigned char cc_br_bomb[] = {0x9b, 0xff, 0xff, 0x3f, 0x00,
0x24, 0x82, 0xe2, 0xb1, 0x40,
0x72, 0xef, 0x7f, 0x00};
#endif
static const unsigned char cc_text[] =
"content codings: HTTrack decodes gzip, brotli and zstd bodies alike.";
/* Content codings: br and zstd decode, junk tokens stay identity, a coding we
can not undo fails the fetch, and a bomb never lands on disk. */
static int st_contentcodings(httrackp *opt, int argc, char **argv) {
const size_t tlen = sizeof(cc_text) - 1;
char inpath[HTS_URLMAXSIZE], outpath[HTS_URLMAXSIZE];
(void) opt;
assertf(hts_codec_parse("gzip") == HTS_CODEC_DEFLATE);
assertf(hts_codec_parse("x-deflate") == HTS_CODEC_DEFLATE);
assertf(hts_codec_parse("") == HTS_CODEC_IDENTITY);
assertf(hts_codec_parse("identity") == HTS_CODEC_IDENTITY);
/* servers do label plain bodies with junk; the page must survive that */
assertf(hts_codec_parse("utf-8") == HTS_CODEC_IDENTITY);
/* a real coding with no decoder here: fail, never save the coded bytes */
assertf(hts_codec_parse("compress") == HTS_CODEC_UNSUPPORTED);
assertf(hts_codec_parse("br") ==
(HTS_USEBROTLI ? HTS_CODEC_BROTLI : HTS_CODEC_UNSUPPORTED));
assertf(hts_codec_parse("zstd") ==
(HTS_USEZSTD ? HTS_CODEC_ZSTD : HTS_CODEC_UNSUPPORTED));
assertf(hts_codec_is_archive_ext(HTS_CODEC_DEFLATE, "tgz"));
assertf(!hts_codec_is_archive_ext(HTS_CODEC_DEFLATE, "html"));
assertf(hts_codec_is_archive_ext(HTS_CODEC_BROTLI, "br"));
assertf(hts_codec_is_archive_ext(HTS_CODEC_ZSTD, "zst"));
/* decoded-size budget: 4096x, floor 1 MiB, ceiling INT_MAX */
assertf(hts_codec_maxout(1) == 1024 * 1024);
assertf(hts_codec_maxout(1024) == 4096 * 1024);
assertf(hts_codec_maxout(1024 * 1024) == INT_MAX);
if (argc >= 1) {
snprintf(inpath, sizeof(inpath), "%s/cc-in", argv[0]);
snprintf(outpath, sizeof(outpath), "%s/cc-out", argv[0]);
#if HTS_USEBROTLI
{
unsigned char head[16];
assertf(ae_write_raw(inpath, cc_br_text, sizeof(cc_br_text)) == 0);
assertf(hts_codec_unpack(HTS_CODEC_BROTLI, inpath, outpath) ==
(int) tlen);
assertf(ae_check_decoded(outpath, cc_text, tlen) == 0);
assertf(hts_codec_head(HTS_CODEC_BROTLI, cc_br_text, sizeof(cc_br_text),
head, sizeof(head)) == sizeof(head));
assertf(memcmp(head, cc_text, sizeof(head)) == 0);
/* truncated: must fail, not fall back to a verbatim copy */
assertf(ae_write_raw(inpath, cc_br_text, sizeof(cc_br_text) - 4) == 0);
assertf(hts_codec_unpack(HTS_CODEC_BROTLI, inpath, outpath) < 0);
/* cc_br_bomb is a valid stream that expands to 4 MiB; the budget, not a
corrupt frame, is what must reject it. */
assertf((LLint) (4 * 1024 * 1024) >
hts_codec_maxout((LLint) sizeof(cc_br_bomb)));
assertf(ae_write_raw(inpath, cc_br_bomb, sizeof(cc_br_bomb)) == 0);
assertf(hts_codec_unpack(HTS_CODEC_BROTLI, inpath, outpath) < 0);
}
#endif
#if HTS_USEZSTD
{
const size_t bomblen = 4 * 1024 * 1024;
const size_t bound = ZSTD_compressBound(bomblen);
unsigned char *bomb = malloct(bomblen);
unsigned char *zbuf = malloct(bound);
unsigned char head[16];
size_t zlen;
assertf(bomb != NULL && zbuf != NULL);
zlen = ZSTD_compress(zbuf, bound, cc_text, tlen, 6);
assertf(!ZSTD_isError(zlen));
assertf(ae_write_raw(inpath, zbuf, zlen) == 0);
assertf(hts_codec_unpack(HTS_CODEC_ZSTD, inpath, outpath) == (int) tlen);
assertf(ae_check_decoded(outpath, cc_text, tlen) == 0);
assertf(hts_codec_head(HTS_CODEC_ZSTD, zbuf, zlen, head, sizeof(head)) ==
sizeof(head));
assertf(memcmp(head, cc_text, sizeof(head)) == 0);
assertf(ae_write_raw(inpath, zbuf, zlen - 4) == 0);
assertf(hts_codec_unpack(HTS_CODEC_ZSTD, inpath, outpath) < 0);
memset(bomb, 'A', bomblen);
zlen = ZSTD_compress(zbuf, bound, bomb, bomblen, 6);
assertf(!ZSTD_isError(zlen));
/* the fixture must really be past the budget, whatever the ratio is */
assertf((LLint) bomblen > hts_codec_maxout((LLint) zlen));
assertf(ae_write_raw(inpath, zbuf, zlen) == 0);
assertf(hts_codec_unpack(HTS_CODEC_ZSTD, inpath, outpath) < 0);
freet(bomb);
freet(zbuf);
}
#endif
/* a coding we can not undo yields no file at all */
assertf(hts_codec_unpack(HTS_CODEC_UNSUPPORTED, inpath, outpath) < 0);
}
printf("contentcodings self-test OK\n");
return 0;
}
/* Each call parses `txt` under a fresh host, then checkrobots() for `path`. */
static int rb_decide(robots_wizard *r, const char *txt, const char *path) {
static int n = 0;
@@ -2913,10 +2637,6 @@ static const struct selftest_entry {
{"resolve", "<link> <adr> <fil>", "resolve a link against an origin",
st_resolve},
{"identurl", "<url>", "split an absolute URL into (adr, fil)", st_identurl},
{"proxyurl", "<proxy-arg>", "parse a -P proxy URL into host/port",
st_proxyurl},
{"socks5", "", "SOCKS5 handshake framing and credential self-test",
st_socks5},
{"identabs", "", "ident_url_absolute one-byte fil[] overflow self-test",
st_identabs},
{"header", "<raw-header-line> ...", "response header-line parsing",
@@ -2957,9 +2677,6 @@ static const struct selftest_entry {
{"status", "", "HTTP status code -> reason phrase self-test", st_status},
{"acceptencoding", "[dir]",
"Accept-Encoding advertises gzip+deflate, both decode", st_acceptencoding},
{"contentcodings", "[dir]",
"brotli and zstd bodies decode; bombs and unknown codings are refused",
st_contentcodings},
{"robots", "", "robots.txt RFC 9309 Allow/Disallow precedence self-test",
st_robots},
#ifndef _WIN32

View File

@@ -37,7 +37,6 @@ Please visit our Website: http://www.httrack.com
/* specific definitions */
#include "htsbase.h"
#include "htscore.h"
#include "htscodec.h"
#include "htszlib.h"
#if HTS_USEZLIB
@@ -47,8 +46,13 @@ Please visit our Website: http://www.httrack.com
#include "htszlib.h"
*/
/*
Unpack file into a new file (gzip, zlib RFC1950 or raw deflate RFC1951).
A body provably in no deflate framing is copied verbatim (identity).
Return value: size of the new file, or -1 if an error occurred
*/
/* Note: utf-8 */
int hts_zunpack(const char *filename, const char *newfile) {
int hts_zunpack(char *filename, char *newfile) {
int ret = -1;
if (filename != NULL && newfile != NULL && filename[0] && newfile[0]) {
@@ -57,7 +61,6 @@ int hts_zunpack(const char *filename, const char *newfile) {
if (in != NULL) {
unsigned char BIGSTK inbuf[8192];
const LLint maxout = hts_codec_maxout(hts_codec_coded_size(in));
size_t navail = fread(inbuf, 1, sizeof(inbuf), in);
/* gzip/zlib headers -> +32 windowBits; else raw deflate (RFC1951) */
const hts_boolean wrapped =
@@ -70,10 +73,9 @@ int hts_zunpack(const char *filename, const char *newfile) {
deflate framing at all. env_error: local I/O or memory failure. */
hts_boolean not_deflate = HTS_FALSE;
hts_boolean env_error = HTS_FALSE;
hts_boolean bomb = HTS_FALSE;
/* deflate is ambiguous; on failure retry with the other windowBits */
for (attempt = 0; attempt < 2 && ret < 0 && !bomb; attempt++) {
for (attempt = 0; attempt < 2 && ret < 0; attempt++) {
const int windowBits =
(attempt == 0 ? wrapped : !wrapped) ? (32 + MAX_WBITS) : -MAX_WBITS;
FILE *fpout;
@@ -100,7 +102,7 @@ int hts_zunpack(const char *filename, const char *newfile) {
}
{
hts_boolean ok = HTS_TRUE;
LLint size = 0;
int size = 0;
int zerr = Z_OK;
/* chunked inflate; first chunk in inbuf, single member */
@@ -124,33 +126,27 @@ int hts_zunpack(const char *filename, const char *newfile) {
break;
}
produced = sizeof(outbuf) - strm.avail_out;
size += (LLint) produced;
if (size > maxout) { /* decompression bomb: no retry, no copy */
bomb = HTS_TRUE;
ok = HTS_FALSE;
break;
}
if (produced > 0 &&
fwrite(outbuf, 1, produced, fpout) != produced) {
env_error = HTS_TRUE;
ok = HTS_FALSE;
break;
}
size += (int) produced;
} while (strm.avail_out == 0);
if (!ok || zerr == Z_STREAM_END)
break;
navail = fread(inbuf, 1, sizeof(inbuf), in);
} while (navail > 0);
if (ok && zerr == Z_STREAM_END)
ret = (int) size;
ret = size;
}
inflateEnd(&strm);
fclose(fpout);
}
/* keep a mislabeled identity body verbatim only when provably not
deflate; truncation or a local failure must keep failing (#47) */
if (ret < 0 && !bomb && !wrapped && not_deflate && !env_error &&
!ferror(in)) {
if (ret < 0 && !wrapped && not_deflate && !env_error && !ferror(in)) {
FILE *const fpout =
FOPEN(fconv(catbuff, sizeof(catbuff), newfile), "wb");
@@ -176,25 +172,6 @@ int hts_zunpack(const char *filename, const char *newfile) {
return ret;
}
size_t hts_zhead(const void *in, size_t in_len, void *out, size_t out_len) {
z_stream zs;
size_t n = 0;
int err;
memset(&zs, 0, sizeof(zs));
if (inflateInit2(&zs, 47) != Z_OK) /* 47: gzip or zlib, autodetected */
return 0;
zs.next_in = (const Bytef *) in;
zs.avail_in = (uInt) in_len;
zs.next_out = (Bytef *) out;
zs.avail_out = (uInt) out_len;
err = inflate(&zs, Z_SYNC_FLUSH);
if (err == Z_OK || err == Z_STREAM_END || err == Z_BUF_ERROR)
n = out_len - zs.avail_out;
inflateEnd(&zs);
return n;
}
int hts_extract_meta(const char *path) {
char catbuff[CATBUFF_SIZE];
unzFile zFile = unzOpen(fconcat(catbuff, sizeof(catbuff), path, "hts-cache/new.zip"));

View File

@@ -44,13 +44,7 @@ Please visit our Website: http://www.httrack.com
/* Library internal definictions */
#ifdef HTS_INTERNAL_BYTECODE
/* Inflate filename into newfile (gzip, zlib or raw deflate); decoded size, or
-1 on a corrupt/truncated stream, an over-budget body, or an I/O failure. A
body provably in no deflate framing is copied verbatim. */
extern int hts_zunpack(const char *filename, const char *newfile);
/* Inflate the head of a gzip/zlib stream, out_len max; 0 if undecodable */
extern size_t hts_zhead(const void *in, size_t in_len, void *out,
size_t out_len);
extern int hts_zunpack(char *filename, char *newfile);
extern int hts_extract_meta(const char *path);
extern const char *hts_get_zerror(int err);
#endif

View File

@@ -1,5 +0,0 @@
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"

View File

@@ -108,9 +108,6 @@
<Project>{e76ad871-54c1-45e8-a657-6117adeffb46}</Project>
</ProjectReference>
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="httrack.rc" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>

View File

@@ -1,5 +0,0 @@
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"

View File

@@ -59,17 +59,15 @@
<ItemDefinitionGroup>
<ClCompile>
<!-- LIBHTTRACK_EXPORTS turns HTSEXT_API into __declspec(dllexport); ZLIB_DLL
imports zlib from its DLL. HTS_USEBROTLI/HTS_USEZSTD enable the br and
zstd content codings. Windows 7 floor, matching WinHTTrack. -->
<PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
imports zlib from its DLL. Windows 7 floor, matching WinHTTrack. -->
<PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<AdditionalIncludeDirectories>$(MSBuildThisFileDirectory);$(MSBuildThisFileDirectory)coucal;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
<WarningLevel>Level3</WarningLevel>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
<CompileAs>CompileAsC</CompileAs>
</ClCompile>
<Link>
<!-- vcpkg auto-links libssl/libcrypto/zlib/brotli/zstd; only the OS import
lib is explicit. -->
<!-- vcpkg auto-links libssl/libcrypto/zlib; only the OS import lib is explicit. -->
<AdditionalDependencies>Ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
<GenerateDebugInformation>true</GenerateDebugInformation>
</Link>
@@ -109,7 +107,6 @@
<ClCompile Include="htscache_selftest.c" />
<ClCompile Include="htscatchurl.c" />
<ClCompile Include="htscharset.c" />
<ClCompile Include="htscodec.c" />
<ClCompile Include="htsconcat.c" />
<ClCompile Include="htscore.c" />
<ClCompile Include="htscoremain.c" />
@@ -125,7 +122,6 @@
<ClCompile Include="htsmodules.c" />
<ClCompile Include="htsname.c" />
<ClCompile Include="htsparse.c" />
<ClCompile Include="htsproxy.c" />
<ClCompile Include="htsrobots.c" />
<ClCompile Include="htsselftest.c" />
<ClCompile Include="htssniff.c" />
@@ -142,9 +138,6 @@
<ClCompile Include="minizip\zip.c" />
<ClCompile Include="punycode.c" />
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="libhttrack.rc" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>

View File

@@ -1,12 +1,9 @@
{
"$schema":
"https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"name": "libhttrack",
"version-string": "3.49",
"dependencies": [
"brotli",
"openssl",
"zlib",
"zstd"
"zlib"
]
}

View File

@@ -1,50 +0,0 @@
// Version resource for libhttrack.dll and httrack.exe. Signing enforces that every
// binary in a release names the same product and version, so both need one.
// Spelled out here because a VERSIONINFO cannot take a version string apart;
// tests/01_engine-version-macros.test fails if this drifts from htsglobal.h.
// The per-binary parts come from libhttrack.rc / httrack.rc.
#include <winver.h>
#ifndef VER_FILE_DESCRIPTION
#define VER_FILE_DESCRIPTION "HTTrack Website Copier"
#endif
#ifndef VER_ORIGINAL_FILENAME
#define VER_ORIGINAL_FILENAME ""
#endif
#ifndef VER_FILETYPE
#define VER_FILETYPE VFT_APP
#endif
VS_VERSION_INFO VERSIONINFO
FILEVERSION 3, 49, 12, 0
PRODUCTVERSION 3, 49, 12, 0
FILEFLAGSMASK VS_FFI_FILEFLAGSMASK
#ifdef _DEBUG
FILEFLAGS VS_FF_DEBUG
#else
FILEFLAGS 0x0L
#endif
FILEOS VOS_NT_WINDOWS32
FILETYPE VER_FILETYPE
FILESUBTYPE VFT2_UNKNOWN
BEGIN
BLOCK "StringFileInfo"
BEGIN
BLOCK "040904b0" // U.S. English, Unicode
BEGIN
VALUE "CompanyName", "Xavier Roche"
VALUE "FileDescription", VER_FILE_DESCRIPTION
VALUE "FileVersion", "3.49.12"
VALUE "InternalName", VER_ORIGINAL_FILENAME
VALUE "LegalCopyright", "Copyright (C) 1998-2026 Xavier Roche and other contributors. GNU GPL v3 or later."
VALUE "OriginalFilename", VER_ORIGINAL_FILENAME
VALUE "ProductName", "HTTrack Website Copier"
VALUE "ProductVersion", "3.49-12"
END
END
BLOCK "VarFileInfo"
BEGIN
VALUE "Translation", 0x409, 1200
END
END

View File

@@ -1,44 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# hts_parse_proxy splits a -P argument "[scheme://][user:pass@]host[:port]" into
# the proxy host string and port. -#test=proxyurl <arg> prints
# "name=.. port=.. host=..", where host= is what the connect resolves.
p() {
test "$(httrack -O /dev/null -#test=proxyurl "$1")" == "$2" || exit 1
}
# bare host, with and without an explicit port
p 'proxy.example.com:8080' 'name=proxy.example.com port=8080 host=proxy.example.com'
p 'proxy.example.com' 'name=proxy.example.com port=8080 host=proxy.example.com'
# user:pass@ is kept in name (for proxy auth) and stripped from host
p 'user:pass@proxy:3128' 'name=user:pass@proxy port=3128 host=proxy'
p 'user:pass@proxy' 'name=user:pass@proxy port=8080 host=proxy'
# a scheme colon must not be read as a port: http://host with no port used to
# parse to name=http with a garbage port
p 'http://proxy:8080' 'name=http://proxy port=8080 host=proxy'
p 'http://proxy' 'name=http://proxy port=8080 host=proxy'
# socks5/socks5h default to 1080 and resolve to the bare host
p 'socks5://host:1080' 'name=socks5://host port=1080 host=host'
p 'socks5://host' 'name=socks5://host port=1080 host=host'
p 'socks5h://host' 'name=socks5h://host port=1080 host=host'
# explicit port wins over the 1080 default, with and without userinfo
p 'socks5://host:9050' 'name=socks5://host port=9050 host=host'
p 'socks5://user:pass@host:9050' 'name=socks5://user:pass@host port=9050 host=host'
# the split is byte-transparent: percent-escapes stay encoded (decoded later at
# auth time), only structural ASCII ':'/'@' delimit, and the last '@' ends the
# userinfo. So %40/%3A and UTF-8 bytes never mis-split host or port.
p 'socks5://user:p%40ss@host:1080' 'name=socks5://user:p%40ss@host port=1080 host=host'
p 'socks5://us%3Aer:pass@host:1080' 'name=socks5://us%3Aer:pass@host port=1080 host=host'
p 'socks5://a@b:c@host:1080' 'name=socks5://a@b:c@host port=1080 host=host'
p 'socks5://naïve:pass@héllo:1080' 'name=socks5://naïve:pass@héllo port=1080 host=héllo'
# a lone ':' must not underflow the backward scan
p ':' 'name= port=8080 host='

View File

@@ -35,8 +35,7 @@ chk image/avif hex:0000001C6674797061766966 "$yes"
chk image/avif hex:0000001C6674797068656963 "$no" # heic brand is not avif
chk image/heic hex:0000001C6674797068656963 "$yes"
chk image/svg+xml '<svg xmlns="x">' "$yes"
# BOM+ws skip; hex, as Windows argv cannot carry the raw BOM through the ANSI codepage
chk image/svg+xml hex:EFBBBF20203C3F786D6C2076657273696F6E3D22312E30223F3E "$yes"
chk image/svg+xml $'\xef\xbb\xbf <?xml version="1.0"?>' "$yes" # BOM+ws skip
# audio / video
chk audio/mpeg 'ID3xxx' "$yes"

View File

@@ -1,9 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# The SOCKS5 handshake framing and credential split, driven against scripted
# server replies (frame draining, oversize rejects, RFC 1929 fields).
httrack -O /dev/null '-#test=socks5' | grep -q "socks5 self-test OK"

View File

@@ -1,64 +0,0 @@
#!/bin/bash
#
# version.rc repeats the version that htsglobal.h declares. Signing enforces that
# every binary in a release reports the same one, so a drift fails the signing
# request on release day rather than the build. Assert the two agree.
set -euo pipefail
src="${top_srcdir:-..}/src"
h="$src/htsglobal.h"
rc="$src/version.rc"
for f in "$h" "$rc"; do
[ -f "$f" ] || {
echo "cannot find $f"
exit 1
}
done
# 3.49-12 (display) and 3.49.12 (dotted).
version=$(sed -n 's/^#define HTTRACK_VERSION[[:space:]][[:space:]]*"\([^"]*\)".*/\1/p' "$h")
versionid=$(sed -n 's/^#define HTTRACK_VERSIONID[[:space:]][[:space:]]*"\([^"]*\)".*/\1/p' "$h")
if [ -z "$version" ] || [ -z "$versionid" ]; then
echo "could not read the version from $h"
exit 1
fi
# The same version, as version.rc states it.
fileversion=$(sed -n 's/^[[:space:]]*FILEVERSION[[:space:]][[:space:]]*\(.*\)$/\1/p' "$rc" | tr -d ' \r')
productversion=$(sed -n 's/^[[:space:]]*PRODUCTVERSION[[:space:]][[:space:]]*\(.*\)$/\1/p' "$rc" | tr -d ' \r')
rc_fileversion=$(sed -n 's/.*VALUE "FileVersion",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
rc_productversion=$(sed -n 's/.*VALUE "ProductVersion",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
rc_productname=$(sed -n 's/.*VALUE "ProductName",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
# 3.49.12 -> 3,49,12,0
expected_numeric="$(echo "$versionid" | tr '.' ','),0"
fail=0
check() { # what expected actual
if [ "$2" != "$3" ]; then
echo "version.rc $1 is \"$3\", but htsglobal.h says it should be \"$2\""
fail=1
fi
}
check FILEVERSION "$expected_numeric" "$fileversion"
check PRODUCTVERSION "$expected_numeric" "$productversion"
check FileVersion "$versionid" "$rc_fileversion"
check ProductVersion "$version" "$rc_productversion"
# Signing pins the product name too.
check ProductName "HTTrack Website Copier" "$rc_productname"
[ "$fail" -eq 0 ] || exit 1
# And the shipped binary must agree with the header it was built from.
out=$(httrack --version)
case "$out" in
*"$version"*) ;;
*)
echo "httrack --version says \"$out\", which does not mention $version"
exit 1
;;
esac
echo "version resource agrees with htsglobal.h: $version ($expected_numeric)"

View File

@@ -1,11 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# brotli/zstd decode, unknown codings, and the decoded-size budget.
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=contentcodings "$dir" run |
grep -q "contentcodings self-test OK"

View File

@@ -1,38 +0,0 @@
#!/bin/bash
#
# br and zstd on the wire: both decode, a junk coding leaves the body alone, an
# undecodable one fails the fetch, and br/zstd are advertised over TLS only.
set -euo pipefail
: "${top_srcdir:=..}"
# Both decoders are needed for the wire fixtures; the codec-off matrix is
# covered by the fenced 01_zlib-contentcodings self-test.
if test "${BROTLI_ENABLED:-yes}" != "yes" || test "${ZSTD_ENABLED:-yes}" != "yes"; then
echo "brotli/zstd not both compiled, skipping"
exit 77
fi
# bad.html (in memory) and bin.dat (direct-to-disk) both carry an undecodable
# coding: the fetch fails and neither leaves coded bytes on disk.
bash "$top_srcdir/tests/local-crawl.sh" --errors 2 --files 5 \
--file-matches 'codec/br.html' 'coded body' \
--file-matches 'codec/zstd.html' 'coded body' \
--file-matches 'codec/junk.html' 'junk coding' \
--file-matches 'codec/ae.html' 'AE=gzip, deflate,' \
--file-not-matches 'codec/ae.html' ' br' \
--file-not-matches 'codec/ae.html' 'zstd' \
--not-found 'codec/bad.html' \
--not-found 'codec/bin.dat' \
--log-found 'Unsupported Content-Encoding' \
httrack 'BASEURL/codec/index.html'
if test "${HTTPS_SUPPORT:-}" == "no"; then
echo "no https support compiled, skipping the TLS leg"
exit 0
fi
bash "$top_srcdir/tests/local-crawl.sh" --tls --errors 2 --files 5 \
--file-matches 'codec/ae.html' 'AE=gzip, deflate, br, zstd,' \
httrack 'BASEURL/codec/index.html'

View File

@@ -1,38 +0,0 @@
#!/bin/bash
#
# A coded re-fetch that fails to decode must not destroy the mirrored file (#557).
# Pass 1 mirrors from valid gzip; pass 2 (--update) serves an undecodable body for
# mem.html (in-memory), disk.bin (direct-to-disk) and unsup.html (a coding we have
# no decoder for). All three must keep their pass-1 content. fresh.html and
# freshdisk.bin decode on both passes: they are the controls that a good update
# still lands, in memory and direct-to-disk (the latter renaming the decoded temp
# over an existing file). Unfixed, the decode target is the mirror itself, so the
# first three are lost.
set -euo pipefail
: "${top_srcdir:=..}"
# A restrictive umask makes the decoded direct-to-disk file's mode observable:
# it is committed by renaming a temp, so it needs the chmod filecreate() does.
umask 077
bash "$top_srcdir/tests/local-crawl.sh" \
--rerun-args '--update' \
--errors 3 \
--log-found 'decompressing.*upcodec/mem\.html' \
--log-found 'decompressing.*upcodec/disk\.bin' \
--log-found 'Unsupported Content-Encoding.*upcodec/unsup\.html' \
--file-matches 'upcodec/mem.html' 'MIRRORED-MEM-V1' \
--file-matches 'upcodec/disk.bin' 'MIRRORED-DISK-V1' \
--file-min-bytes 'upcodec/disk.bin' 32768 \
--file-mode 'upcodec/disk.bin' 644 \
--file-matches 'upcodec/unsup.html' 'MIRRORED-UNSUP-V1' \
--file-matches 'upcodec/fresh.html' 'FRESH-V2' \
--file-not-matches 'upcodec/fresh.html' 'FRESH-V1' \
--file-mode 'upcodec/fresh.html' 644 \
--file-matches 'upcodec/freshdisk.bin' 'FRESHDISK-V2' \
--file-not-matches 'upcodec/freshdisk.bin' 'FRESHDISK-V1' \
--file-min-bytes 'upcodec/freshdisk.bin' 32768 \
--file-mode 'upcodec/freshdisk.bin' 644 \
httrack 'BASEURL/upcodec/index.html'

View File

@@ -1,211 +0,0 @@
#!/bin/bash
#
# Issue #563: a SOCKS5 proxy must carry both http and https, resolve the origin
# name REMOTELY (ATYP=domain, never a local lookup), and authenticate per RFC
# 1929 without leaking the credentials to the origin.
set -euo pipefail
: "${top_srcdir:=..}"
if test "${HTTPS_SUPPORT:-}" == "no"; then
echo "no https support compiled, skipping"
exit 77
fi
if ! command -v python3 >/dev/null 2>&1 || ! command -v openssl >/dev/null 2>&1; then
echo "python3/openssl missing, skipping"
exit 77
fi
# a .invalid name never resolves (RFC 6761): reaching the page at all proves the
# proxy did the DNS
host="socks-origin.invalid"
server="$top_srcdir/tests/socks5-server.py"
tmpdir=$(mktemp -d)
pids=
cleanup() {
for pid in $pids; do
kill "$pid" 2>/dev/null || true
done
rm -rf "$tmpdir"
}
trap cleanup EXIT
openssl req -x509 -newkey rsa:2048 -keyout "$tmpdir/key.pem" \
-out "$tmpdir/cert.pem" -days 2 -nodes -subj "/CN=127.0.0.1" \
>/dev/null 2>&1
cat "$tmpdir/key.pem" "$tmpdir/cert.pem" >"$tmpdir/both.pem"
# start_server <logdir> <mode>: sets $tls_port/$http_port/$socks_port
start_server() {
local dir="$1" mode="$2" ports
mkdir -p "$dir"
ports="$dir/ports.txt"
python3 "$server" "$tmpdir/both.pem" "$dir" "$mode" \
>"$ports" 2>"$dir/server.err" &
pids="$pids $!"
for _ in $(seq 1 100); do
grep -q "^ready" "$ports" 2>/dev/null && break
sleep 0.1
done
grep -q "^ready" "$ports" 2>/dev/null || {
echo "server ($mode) did not start" >&2
cat "$dir/server.err" >&2
exit 1
}
tls_port=$(awk '/^TLS/{print $2}' "$ports")
http_port=$(awk '/^HTTP/{print $2}' "$ports")
socks_port=$(awk '/^SOCKS/{print $2}' "$ports")
}
# kill a hung httrack so a missing read bound surfaces as $HANG_RC instead of
# stalling the job (macOS has no `timeout`)
HANG_RC=137 # 128 + SIGKILL
run_crawl() {
local out="$1" url="$2" proxy="$3"
shift 3
local extra="$*"
test -n "$extra" || extra="-r1 -s0"
rm -rf "$out"
# shellcheck disable=SC2086
httrack "$url" --proxy "$proxy" \
-O "$out" $extra --timeout=10 >"$out.log" 2>&1 &
local pid=$!
(sleep 60 && kill -9 "$pid" 2>/dev/null) &
local guard=$!
local rc=0
wait "$pid" 2>/dev/null || rc=$?
kill "$guard" 2>/dev/null || true
wait "$guard" 2>/dev/null || true
return "$rc"
}
# --- https over socks, remote DNS -------------------------------------------
ok="$tmpdir/ok"
start_server "$ok" ok
run_crawl "$ok/out" "https://${host}:${tls_port}/" "socks5://127.0.0.1:${socks_port}"
grep -rq "ORIGIN-PAGE-563" "$ok/out" || {
echo "FAIL: origin page not downloaded through the socks proxy" >&2
cat "$ok/out.log" "$ok/server.err" >&2
exit 1
}
grep -q "^ATYP domain ${host}\$" "$ok/socks.log" || {
echo "FAIL: proxy did not receive ATYP=domain ${host} (no remote DNS)" >&2
cat "$ok/socks.log" >&2
exit 1
}
# a client that resolved locally would have sent an address, not a name
grep -q "^ATYP ipv4" "$ok/socks.log" && {
echo "FAIL: httrack resolved locally and sent an IPv4 to the proxy" >&2
cat "$ok/socks.log" >&2
exit 1
}
echo "OK: https tunneled through socks5 with remote DNS"
# --- plain http over socks --------------------------------------------------
: >"$ok/origin-headers.log" # the https crawl above also logged a request line
run_crawl "$ok/outh" "http://${host}:${http_port}/" "socks5://127.0.0.1:${socks_port}"
grep -rq "ORIGIN-PAGE-563" "$ok/outh" || {
echo "FAIL: plain http not tunneled through the socks proxy" >&2
cat "$ok/outh.log" "$ok/socks.log" >&2
exit 1
}
# the tunneled request is origin-form, as on a direct connection: the http-proxy
# absolute URI would reach the origin as "GET http://host/"
grep -q "^REQUEST GET / " "$ok/origin-headers.log" || {
echo "FAIL: tunneled request is not origin-form" >&2
cat "$ok/origin-headers.log" >&2
exit 1
}
grep -q "^REQUEST GET http" "$ok/origin-headers.log" && {
echo "FAIL: absolute-URI request sent through the socks tunnel" >&2
cat "$ok/origin-headers.log" >&2
exit 1
}
echo "OK: plain http tunneled through socks5"
# --- keep-alive socket reuse must NOT re-run the handshake ------------------
# A reused keep-alive socket is already tunneled; re-injecting the SOCKS
# greeting corrupts the origin stream. -c1 + the origin's max>1 advert makes
# httrack serve the whole plain-http crawl over one tunnel: exactly one
# handshake, every subpage intact.
ka="$tmpdir/ka"
start_server "$ka" ok
run_crawl "$ka/out" "http://${host}:${http_port}/" "socks5://127.0.0.1:${socks_port}" -r3 -c1
missing=0
for i in 1 2 3 4 5; do
grep -rq "SUBPAGE-563 /p${i}.html" "$ka/out" || missing=1
done
test "$missing" -eq 0 || {
echo "FAIL: a keep-alive subpage was lost (re-handshake corrupted the reused socket)" >&2
cat "$ka/out.log" "$ka/socks.log" >&2
exit 1
}
handshakes=$(grep -c "^CMD " "$ka/socks.log" || true)
test "$handshakes" -eq 1 || {
echo "FAIL: expected 1 SOCKS handshake for the reused socket, got $handshakes" >&2
cat "$ka/socks.log" >&2
exit 1
}
echo "OK: keep-alive socket reused over one socks tunnel, no re-handshake"
# --- authenticated socks (RFC 1929) -----------------------------------------
# over plain http: that is the request an http proxy would decorate with
# Proxy-Authorization, so the no-leak check below has teeth
auth="$tmpdir/auth"
start_server "$auth" ok
run_crawl "$auth/out" "http://${host}:${http_port}/" \
"socks5://user:secret@127.0.0.1:${socks_port}"
grep -rq "ORIGIN-PAGE-563" "$auth/out" || {
echo "FAIL: origin not downloaded through the authenticated socks proxy" >&2
cat "$auth/out.log" "$auth/socks.log" >&2
exit 1
}
grep -q "^METHOD userpass\$" "$auth/socks.log" || {
echo "FAIL: user/pass method not negotiated (auth ignored)" >&2
cat "$auth/socks.log" >&2
exit 1
}
grep -q "^AUTH user secret\$" "$auth/socks.log" || {
echo "FAIL: wrong socks credentials sent" >&2
cat "$auth/socks.log" >&2
exit 1
}
grep -qi "secret\|proxy-authorization" "$auth/origin-headers.log" && {
echo "FAIL: socks credentials leaked into the origin request" >&2
cat "$auth/origin-headers.log" >&2
exit 1
}
echo "OK: socks5 user/pass negotiated, creds not leaked to origin"
# --- hostile: refusing proxy ------------------------------------------------
ref="$tmpdir/refuse"
start_server "$ref" refuse
rc=0
run_crawl "$ref/out" "https://${host}:${tls_port}/" "socks5://127.0.0.1:${socks_port}" || rc=$?
test "$rc" -ne "$HANG_RC" || {
echo "FAIL: crawl hung on a refusing socks proxy" >&2
exit 1
}
grep -rq "ORIGIN-PAGE-563" "$ref/out" 2>/dev/null && {
echo "FAIL: refusing proxy unexpectedly served the page" >&2
exit 1
}
echo "OK: refusing socks proxy fails cleanly, no hang"
# --- hostile: truncated handshake reply -------------------------------------
trunc="$tmpdir/trunc"
start_server "$trunc" truncate
rc=0
run_crawl "$trunc/out" "https://${host}:${tls_port}/" "socks5://127.0.0.1:${socks_port}" || rc=$?
test "$rc" -ne "$HANG_RC" || {
echo "FAIL: crawl hung on a truncated socks reply (bounded read missing)" >&2
exit 1
}
grep -rq "ORIGIN-PAGE-563" "$trunc/out" 2>/dev/null && {
echo "FAIL: truncated-reply proxy unexpectedly served the page" >&2
exit 1
}
echo "OK: bounded socks handshake, no hang on a truncated reply"

View File

@@ -2,7 +2,7 @@
# explicitly: automake does not expand wildcards in EXTRA_DIST, so a glob would
# silently drop it from the dist tarball and break "make distcheck".
EXTRA_DIST = $(TESTS) crawl-test.sh run-all-tests.sh check-network.sh \
proxy-https-server.py socks5-server.py \
proxy-https-server.py \
local-crawl.sh local-server.py server.crt server.key \
server-root/simple/basic.html server-root/simple/link.html \
server-root/stripquery/index.html server-root/stripquery/a.html \
@@ -15,8 +15,6 @@ TESTS_ENVIRONMENT += PATH=$(top_builddir)/src$(PATH_SEPARATOR)$$PATH
### TESTS_ENVIRONMENT += $(SHLIBPATH_VAR)="$(top_builddir)/src/$(LT_CV_OBJDIR)$${$(SHLIBPATH_VAR):+$(PATH_SEPARATOR)}$$$(SHLIBPATH_VAR)"
TESTS_ENVIRONMENT += ONLINE_UNIT_TESTS=$(ONLINE_UNIT_TESTS)
TESTS_ENVIRONMENT += HTTPS_SUPPORT=$(HTTPS_SUPPORT)
TESTS_ENVIRONMENT += BROTLI_ENABLED=$(BROTLI_ENABLED)
TESTS_ENVIRONMENT += ZSTD_ENABLED=$(ZSTD_ENABLED)
TESTS_ENVIRONMENT += V6_SUPPORT=$(V6_SUPPORT)
TESTS_ENVIRONMENT += top_srcdir=$(top_srcdir)
@@ -46,8 +44,6 @@ TESTS = \
01_engine-header.test \
01_engine-idna.test \
01_engine-identurl.test \
01_engine-proxyurl.test \
01_engine-socks5.test \
01_engine-identabs.test \
01_engine-escape-room.test \
01_engine-inplace-escape.test \
@@ -70,10 +66,8 @@ TESTS = \
01_engine-urlhack.test \
01_engine-unescape-bounds.test \
01_engine-useragent.test \
01_engine-version-macros.test \
01_engine-xfread.test \
01_zlib-acceptencoding.test \
01_zlib-contentcodings.test \
01_zlib-cache.test \
01_zlib-cache-corrupt.test \
01_zlib-cache-legacy.test \
@@ -125,9 +119,6 @@ TESTS = \
46_local-update-304-resume.test \
47_local-crange-overflow.test \
48_local-crange-memresume.test \
49_local-cookiewall.test \
50_local-contentcodings.test \
51_local-update-codec.test \
52_local-socks5.test
49_local-cookiewall.test
CLEANFILES = check-network_sh.cache

View File

@@ -17,7 +17,7 @@
# --errors N --errors-content N --files N --found PATH ... --directory PATH ... \
# --log-found REGEX ... --log-not-found REGEX ... \
# --file-matches PATH REGEX ... --file-not-matches PATH REGEX ... \
# --file-min-bytes PATH N --file-mode PATH OCTAL --max-mirror-bytes N \
# --file-min-bytes PATH N --max-mirror-bytes N \
# httrack BASEURL/some/path [httrack-args...]
# --errors counts every "Error:" log line; --errors-content drops transient
# network failures (codes -2..-6) that flake on busy loopback under -c8.
@@ -26,7 +26,6 @@
# --file-matches/--file-not-matches grep (ERE) a mirrored file (PATH under the
# host root), to assert rewritten link/content survived the crawl.
# --file-min-bytes asserts a mirrored file (PATH) is at least N bytes.
# --file-mode asserts its octal permissions (e.g. 644); POSIX hosts only.
# --rerun-args runs a second pass (same server and mirror dir) with the given
# extra httrack args appended, e.g. an --update run under a cap.
# --cookie writes a Netscape cookies.txt (scoped to the discovered host:port,
@@ -142,7 +141,7 @@ while test "$pos" -lt "$nargs"; do
audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
pos=$((pos + 1))
;;
--file-matches | --file-not-matches | --file-min-bytes | --file-mode)
--file-matches | --file-not-matches | --file-min-bytes)
audit+=("${args[$pos]}" "${args[$((pos + 1))]}" "${args[$((pos + 2))]}")
pos=$((pos + 2))
;;
@@ -320,10 +319,9 @@ done
test -n "$hostroot" || die "could not find host root under $out"
debug "host root: $hostroot"
# No crawl, even a cancelled one, may leave engine temporaries: .delayed (#107,
# #483), or the .z/.u content-coding temps (#557).
info "checking for leftover engine temporaries"
leftovers=$(find "$out" \( -name '*.delayed' -o -name '*.z' -o -name '*.u' \) 2>/dev/null | head -5)
# No crawl, even a cancelled one, may leave .delayed temporaries (#107, #483).
info "checking for leftover .delayed files"
leftovers=$(find "$out" -name '*.delayed' 2>/dev/null | head -5)
if test -z "$leftovers"; then result "OK"; else
result "leftover: $leftovers"
exit 1
@@ -438,17 +436,6 @@ while test "$i" -lt "${#audit[@]}"; do
exit 1
fi
;;
--file-mode)
path="${audit[$((i + 1))]}"
i=$((i + 2))
mode=$(stat -c '%a' "${hostroot}/${path}" 2>/dev/null ||
stat -f '%Lp' "${hostroot}/${path}" 2>/dev/null)
info "checking ${path} mode ${mode:-none} is ${audit[$i]}"
if test "$mode" = "${audit[$i]}"; then result "OK"; else
result "wrong mode"
exit 1
fi
;;
esac
i=$((i + 1))
done

View File

@@ -14,7 +14,6 @@ stdlib only (http.server + ssl) -- no new build or runtime dependency.
"""
import argparse
import base64
import gzip
import hashlib
import os
@@ -634,149 +633,6 @@ class Handler(SimpleHTTPRequestHandler):
extra_headers=[("Content-Encoding", "gzip")],
)
# --- content codings ---------------------------------------------------
# Canned br/zstd bodies (no brotli/zstd module in the stdlib): both decode
# to CODEC_BODY. Regenerate with the brotli/zstd CLIs over that string.
CODEC_BODY = (
b"<html><head><title>codec</title></head>"
b"<body><p>coded body</p></body></html>"
)
CODEC_BR = base64.b64decode(
"G0sAAAQccqSBBfJlUvOccsDeitqC9CbHwENWiptQj5aExP0mBjjVgy2DF17olLzLo2T2Eg=="
)
CODEC_ZSTD = base64.b64decode(
"KLUv/SBMFQIAFAM8aHRtbD48aGVhZD48dGl0bGU+Y29kZWM8LzwvYm9keT48"
"cGQgPC9wPjwvL2h0bWw+BQA7p8QMDNQ1PgcWhjkG"
)
def route_codec_index(self):
self.send_html(
'\t<a href="br.html">br</a>\n'
'\t<a href="zstd.html">zstd</a>\n'
'\t<a href="junk.html">junk</a>\n'
'\t<a href="bad.html">bad</a>\n'
'\t<a href="bin.dat">bin</a>\n'
'\t<a href="ae.html">ae</a>\n'
)
def route_codec_br(self):
self.send_raw(
self.CODEC_BR, "text/html", extra_headers=[("Content-Encoding", "br")]
)
def route_codec_zstd(self):
self.send_raw(
self.CODEC_ZSTD, "text/html", extra_headers=[("Content-Encoding", "zstd")]
)
# Junk token on a plain body: the page must survive (broken servers do this)
def route_codec_junk(self):
self.send_raw(
b"<html><body><p>junk coding</p></body></html>",
"text/html",
extra_headers=[("Content-Encoding", "utf-8")],
)
# A real coding we have no decoder for: the fetch must fail rather than
# save the coded bytes as the page.
def route_codec_bad(self):
self.send_raw(
b"<html><body><p>never decoded</p></body></html>",
"text/html",
extra_headers=[("Content-Encoding", "compress")],
)
# Same, on a non-HTML body: this takes the direct-to-disk (is_write) branch,
# a different discard path than the in-memory bad.html above.
def route_codec_bin(self):
self.send_raw(
b"\x00\x01\x02 CODED-BINARY-MUST-NOT-LAND \xff\xfe" * 8,
"application/octet-stream",
extra_headers=[("Content-Encoding", "compress")],
)
# --- coded re-fetch that fails to decode (#557) -------------------------
# Pass 1 mirrors each file from a valid gzip body; pass 2 (--update) serves
# a body that cannot be decoded. The previously-mirrored copy must survive.
# fresh.html is the control: its pass-2 body decodes, so it must be updated.
UPCODEC_SEEN = {}
def upcodec_pass(self):
"""1 on the first body fetch of this path, 2 on the next ones. HEADs
don't count, so a stray one can't shift which pass gets the bad body."""
if self.command == "HEAD":
return 1
seen = Handler.UPCODEC_SEEN.get(self.path, 0) + 1
Handler.UPCODEC_SEEN[self.path] = seen
return seen
@staticmethod
def gzipped(body):
return gzip.compress(body)
@staticmethod
def bad_gzip(body):
"""A gzip stream whose deflate payload is mangled: inflate fails partway
through, after some plausible output has already been produced."""
raw = bytearray(gzip.compress(body))
raw[20:40] = b"\xff" * 20
return bytes(raw[:-4])
def send_coded(self, body, content_type, coding="gzip"):
self.send_raw(body, content_type, extra_headers=[("Content-Encoding", coding)])
def route_upcodec_index(self):
self.send_html(
'\t<a href="mem.html">mem</a>\n'
'\t<a href="disk.bin">disk</a>\n'
'\t<a href="unsup.html">unsup</a>\n'
'\t<a href="fresh.html">fresh</a>\n'
'\t<a href="freshdisk.bin">freshdisk</a>\n'
)
MEM_V1 = b"<html><body><p>MIRRORED-MEM-V1</p></body></html>"
DISK_V1 = b"MIRRORED-DISK-V1\n" + b"\x00\x01\x02\xff" * 8192
UNSUP_V1 = b"<html><body><p>MIRRORED-UNSUP-V1</p></body></html>"
def route_upcodec_mem(self):
if self.upcodec_pass() == 1:
self.send_coded(self.gzipped(self.MEM_V1), "text/html")
else:
self.send_coded(self.bad_gzip(self.MEM_V1), "text/html")
def route_upcodec_disk(self):
if self.upcodec_pass() == 1:
self.send_coded(self.gzipped(self.DISK_V1), "application/octet-stream")
else:
self.send_coded(self.bad_gzip(self.DISK_V1), "application/octet-stream")
# Pass 2 switches to a coding we have no decoder for.
def route_upcodec_unsup(self):
if self.upcodec_pass() == 1:
self.send_coded(self.gzipped(self.UNSUP_V1), "text/html")
else:
self.send_coded(self.UNSUP_V1, "text/html", coding="compress")
def route_upcodec_fresh(self):
pass1 = self.upcodec_pass() == 1
body = b"<html><body><p>FRESH-V%d</p></body></html>" % (1 if pass1 else 2)
self.send_coded(self.gzipped(body), "text/html")
# Same, direct-to-disk: the update pass decodes, so the temp is renamed over
# an existing mirror file.
def route_upcodec_freshdisk(self):
pass1 = self.upcodec_pass() == 1
body = b"FRESHDISK-V%d\n" % (1 if pass1 else 2) + b"\x03\x02\x01\xfe" * 8192
self.send_coded(self.gzipped(body), "application/octet-stream")
# Echo what httrack advertised, so a crawl can assert the header.
def route_codec_ae(self):
self.send_raw(
b"<html><body><p>AE=%s</p></body></html>"
% self.headers.get("Accept-Encoding", "").encode(),
"text/html",
)
# --- MIME-type exclusion abort (issue #58) -----------------------------
# A -mime:application/pdf filter must abort the transfer once the header
# arrives, not download the whole body and discard it.
@@ -1459,19 +1315,6 @@ class Handler(SimpleHTTPRequestHandler):
"/gated/index.php": route_gated_index,
"/gated/secret.php": route_gated_secret,
"/robots.txt": route_robots,
"/codec/index.html": route_codec_index,
"/codec/br.html": route_codec_br,
"/codec/zstd.html": route_codec_zstd,
"/codec/junk.html": route_codec_junk,
"/codec/bad.html": route_codec_bad,
"/codec/bin.dat": route_codec_bin,
"/codec/ae.html": route_codec_ae,
"/upcodec/index.html": route_upcodec_index,
"/upcodec/mem.html": route_upcodec_mem,
"/upcodec/disk.bin": route_upcodec_disk,
"/upcodec/unsup.html": route_upcodec_unsup,
"/upcodec/fresh.html": route_upcodec_fresh,
"/upcodec/freshdisk.bin": route_upcodec_freshdisk,
"/types/index.html": route_types_index,
"/types/control.php": route_types,
"/types/photo.png": route_types,

View File

@@ -1,228 +0,0 @@
#!/usr/bin/env python3
"""Local SOCKS5 proxy (RFC 1928/1929) + TLS and plain HTTP origins for #563.
Logs the auth method, any user/pass, and the CONNECT ATYP + literal domain, so
the test proves httrack sent a hostname (remote DNS) and negotiated auth.
Modes (argv[3]): ok | refuse (REP=5) | truncate (partial reply, must not hang).
Usage: socks5-server.py <cert.pem> <logdir> [mode]
Prints "TLS <port>", "HTTP <port>", "SOCKS <port>", "ready" on stdout.
"""
import http.server
import os
import socket
import socketserver
import ssl
import struct
import sys
import threading
# The one name the proxy answers for; a .invalid TLD never resolves (RFC 6761),
# so a locally-resolving client could not reach us -- success proves remote DNS.
REMOTE_HOST = b"socks-origin.invalid"
# index links the subpages so an -r3 crawl reuses one keep-alive socket
LINKS = "".join('<a href="/p%d.html">%d</a>' % (i, i) for i in range(1, 6))
ORIGIN_BODY = ("<html><body>ORIGIN-PAGE-563 " + LINKS + "</body></html>").encode()
SOCKS_LOG = "socks.log"
ORIGIN_LOG = "origin-headers.log"
def make_origin(logdir):
class Origin(http.server.BaseHTTPRequestHandler):
# HTTP/1.1 + a max>1 advertisement lets httrack keep the socket alive
protocol_version = "HTTP/1.1"
def do_GET(self):
with open(os.path.join(logdir, ORIGIN_LOG), "a") as handle:
handle.write(
"REQUEST %s %s %s\n"
% (self.command, self.path, self.request_version)
)
for key in self.headers.keys():
handle.write(key + ": " + self.headers[key] + "\n")
if "p" in self.path and ".html" in self.path:
body = (
b"<html><body>SUBPAGE-563 " + self.path.encode() + b"</body></html>"
)
else:
body = ORIGIN_BODY
self.send_response(200)
self.send_header("Content-Type", "text/html")
self.send_header("Content-Length", str(len(body)))
self.send_header("Keep-Alive", "timeout=15, max=100")
self.send_header("Connection", "keep-alive")
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
return Origin
class ThreadingTCPServer(socketserver.ThreadingTCPServer):
allow_reuse_address = True
daemon_threads = True
def start_origin(logdir, certfile=None):
httpd = ThreadingTCPServer(("127.0.0.1", 0), make_origin(logdir))
if certfile is not None:
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain(certfile)
httpd.socket = ctx.wrap_socket(httpd.socket, server_side=True)
port = httpd.socket.getsockname()[1]
threading.Thread(target=httpd.serve_forever, daemon=True).start()
return port
def recvn(conn, n):
buf = b""
while len(buf) < n:
chunk = conn.recv(n - len(buf))
if not chunk:
raise OSError("short read")
buf += chunk
return buf
def pipe(src, dst):
try:
while True:
data = src.recv(65536)
if not data:
break
dst.sendall(data)
except OSError:
pass
finally:
for sock in (src, dst):
try:
sock.shutdown(socket.SHUT_RDWR)
except OSError:
pass
def log(logdir, line):
with open(os.path.join(logdir, SOCKS_LOG), "a") as handle:
handle.write(line + "\n")
def negotiate_auth(conn, logdir):
"""RFC 1928 greeting + RFC 1929 sub-negotiation. Returns True to proceed."""
ver, nmethods = recvn(conn, 2)
if ver != 0x05:
return False
methods = recvn(conn, nmethods)
if 0x02 in methods: # prefer user/pass so the auth test exercises RFC 1929
conn.sendall(b"\x05\x02")
(subver,) = recvn(conn, 1)
(ulen,) = recvn(conn, 1)
uname = recvn(conn, ulen)
(plen,) = recvn(conn, 1)
passwd = recvn(conn, plen)
log(logdir, "METHOD userpass")
log(logdir, "AUTH %s %s" % (uname.decode(), passwd.decode()))
conn.sendall(b"\x01\x00") # RFC 1929 success
elif 0x00 in methods:
conn.sendall(b"\x05\x00")
log(logdir, "METHOD noauth")
else:
conn.sendall(b"\x05\xff") # no acceptable method
return False
return True
def read_request(conn, logdir):
"""RFC 1928 CONNECT. Logs CMD, ATYP, domain/addr, port. Returns dest port."""
ver, cmd, _rsv, atyp = recvn(conn, 4)
log(logdir, "CMD %d" % cmd)
if atyp == 0x01:
addr = socket.inet_ntoa(recvn(conn, 4))
log(logdir, "ATYP ipv4 %s" % addr)
elif atyp == 0x03:
(dlen,) = recvn(conn, 1)
domain = recvn(conn, dlen)
log(logdir, "ATYP domain %s" % domain.decode())
elif atyp == 0x04:
recvn(conn, 16)
log(logdir, "ATYP ipv6")
domain = b""
else:
return None, None
(port,) = struct.unpack(">H", recvn(conn, 2))
log(logdir, "PORT %d" % port)
name = domain if atyp == 0x03 else b""
return name, port
def reply(conn, rep):
conn.sendall(bytes([0x05, rep, 0x00, 0x01]) + b"\x00\x00\x00\x00" + b"\x00\x00")
def handle_socks(conn, logdir, mode):
try:
if not negotiate_auth(conn, logdir):
conn.close()
return
if mode == "truncate":
# one byte of the 10-byte reply, then hold the socket: the client
# must bound this handshake read and give up, not hang.
conn.sendall(b"\x05")
threading.Event().wait()
return
name, port = read_request(conn, logdir)
if mode == "refuse" or name != REMOTE_HOST or port is None:
reply(conn, 0x05) # connection refused
conn.close()
return
try:
upstream = socket.create_connection(("127.0.0.1", port))
except OSError:
reply(conn, 0x05)
conn.close()
return
reply(conn, 0x00) # succeeded
threading.Thread(target=pipe, args=(conn, upstream), daemon=True).start()
pipe(upstream, conn)
except OSError:
try:
conn.close()
except OSError:
pass
def start_socks(logdir, mode):
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", 0))
srv.listen(16)
port = srv.getsockname()[1]
def serve():
while True:
conn, _ = srv.accept()
threading.Thread(
target=handle_socks, args=(conn, logdir, mode), daemon=True
).start()
threading.Thread(target=serve, daemon=True).start()
return port
def main():
certfile, logdir = sys.argv[1], sys.argv[2]
mode = sys.argv[3] if len(sys.argv) > 3 else "ok"
for name in (SOCKS_LOG, ORIGIN_LOG):
open(os.path.join(logdir, name), "w").close()
tls_port = start_origin(logdir, certfile)
http_port = start_origin(logdir, None)
socks_port = start_socks(logdir, mode)
print("TLS %d" % tls_port, flush=True)
print("HTTP %d" % http_port, flush=True)
print("SOCKS %d" % socks_port, flush=True)
print("ready", flush=True)
threading.Event().wait()
if __name__ == "__main__":
main()