Compare commits

..

4 Commits

Author SHA1 Message Date
Xavier Roche
0a468c617f Factor the budget-enforced write shared by the brotli and zstd decoders
The two streaming decoders duplicated the decoded-size budget check, the
security-critical part of each loop. Pull it into one codec_sink helper so the
bomb ceiling lives in a single place. Behavior-preserving; the codec self-test
(exact decoded bytes, truncation and bomb rejection) proves equivalence.

Also compile the streaming-decode body only when a streaming codec is built,
which drops an unused-variable warning in the --without-brotli --without-zstd
build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 18:35:25 +02:00
Xavier Roche
baaf8f993f tests: probe the direct-to-disk decode-failure discard, and assert absence
The wire test only exercised the in-memory discard path (bad.html is HTML).
Add a non-HTML body under an unsupported coding (bin.dat) so the is_write
branch is covered too, and assert both are absent from the mirror rather than
grepping a file that the fix now removes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 18:23:16 +02:00
Xavier Roche
72bec8c9c9 Fix copyright year on the new codec files (2026, not 1998)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 18:08:37 +02:00
Xavier Roche
9e4f961025 Decode the br and zstd content codings, and bound every decode
httrack advertised gzip and deflate only, so it took the oldest coding on
offer while every browser negotiates brotli (46% of CDN traffic) or zstd.
Worse, the response side had no codec identity at all: Content-Encoding was
collapsed into a boolean and hts_zunpack guessed the framing from the body
bytes, so a br or zstd body sent unsolicited (mis-keyed Vary caches do this)
fell through the identity path and was saved as the page, coded bytes and all.

Content-Encoding now maps to a codec (htscodec.c), and the decode dispatches on
it: brotli and zstd get streaming decoders, a known coding we cannot undo fails
the fetch instead of saving garbage, and an unrecognized token still means
identity, because servers do put charsets in that header. br and zstd are
advertised over TLS only, as browsers do.

On any decompression failure the undecoded body is now dropped rather than left
in memory for the writer to commit as the page. That closes the coded-bytes-as-
page hole for the new unsupported-coding path, and with it a latent leak on the
gzip path, where a body that failed to inflate (a truncated stream) was written
to disk verbatim.

Every decode is now bounded: 4096x the coded body, floor 1 MiB, ceiling INT_MAX.
Nothing capped the decoded size before, which was survivable while deflate was
the only codec (it cannot pass 1032x) but not with brotli and zstd, which reach
a million to one. The zlib accumulator also moves to LLint, so a body over 2 GiB
fails instead of overflowing an int.

libbrotlidec and libzstd are optional at configure time and required on Windows,
where libhttrack.dll links them from vcpkg and CI runs the self-tests to prove
the decoders are in.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 17:51:07 +02:00
10 changed files with 2 additions and 139 deletions

3
.gitattributes vendored
View File

@@ -1,3 +0,0 @@
# Resource scripts are Windows tooling input and must stay CRLF: the engine has no
# encoding guard, so an autocrlf checkout could otherwise flatten them silently.
*.rc text eol=crlf

View File

@@ -83,9 +83,6 @@ libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
EXTRA_DIST = httrack.h webhttrack \
version.rc \
libhttrack.rc \
httrack.rc \
coucal/murmurhash3.h.diff \
coucal/murmurhash3.h.orig \
minizip/iowin32.c \

View File

@@ -328,8 +328,8 @@ typedef enum hts_wizard {
typedef enum hts_robots {
HTS_ROBOTS_NEVER = 0, /**< ignore robots rules */
HTS_ROBOTS_SOMETIMES = 1, /**< partial obedience, set by -s */
HTS_ROBOTS_ALWAYS = 2, /**< obey robots rules (default) */
HTS_ROBOTS_SOMETIMES = 1, /**< partial obedience (default) */
HTS_ROBOTS_ALWAYS = 2, /**< obey robots rules */
HTS_ROBOTS_ALWAYS_STRICT = 3 /**< obey even strict rules */
} hts_robots;
#endif

View File

@@ -1,5 +0,0 @@
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"

View File

@@ -108,9 +108,6 @@
<Project>{e76ad871-54c1-45e8-a657-6117adeffb46}</Project>
</ProjectReference>
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="httrack.rc" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>

View File

@@ -1,5 +0,0 @@
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"

View File

@@ -141,9 +141,6 @@
<ClCompile Include="minizip\zip.c" />
<ClCompile Include="punycode.c" />
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="libhttrack.rc" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>

View File

@@ -1,50 +0,0 @@
// Version resource for libhttrack.dll and httrack.exe. Signing enforces that every
// binary in a release names the same product and version, so both need one.
// Spelled out here because a VERSIONINFO cannot take a version string apart;
// tests/01_engine-version-macros.test fails if this drifts from htsglobal.h.
// The per-binary parts come from libhttrack.rc / httrack.rc.
#include <winver.h>
#ifndef VER_FILE_DESCRIPTION
#define VER_FILE_DESCRIPTION "HTTrack Website Copier"
#endif
#ifndef VER_ORIGINAL_FILENAME
#define VER_ORIGINAL_FILENAME ""
#endif
#ifndef VER_FILETYPE
#define VER_FILETYPE VFT_APP
#endif
VS_VERSION_INFO VERSIONINFO
FILEVERSION 3, 49, 12, 0
PRODUCTVERSION 3, 49, 12, 0
FILEFLAGSMASK VS_FFI_FILEFLAGSMASK
#ifdef _DEBUG
FILEFLAGS VS_FF_DEBUG
#else
FILEFLAGS 0x0L
#endif
FILEOS VOS_NT_WINDOWS32
FILETYPE VER_FILETYPE
FILESUBTYPE VFT2_UNKNOWN
BEGIN
BLOCK "StringFileInfo"
BEGIN
BLOCK "040904b0" // U.S. English, Unicode
BEGIN
VALUE "CompanyName", "Xavier Roche"
VALUE "FileDescription", VER_FILE_DESCRIPTION
VALUE "FileVersion", "3.49.12"
VALUE "InternalName", VER_ORIGINAL_FILENAME
VALUE "LegalCopyright", "Copyright (C) 1998-2026 Xavier Roche and other contributors. GNU GPL v3 or later."
VALUE "OriginalFilename", VER_ORIGINAL_FILENAME
VALUE "ProductName", "HTTrack Website Copier"
VALUE "ProductVersion", "3.49-12"
END
END
BLOCK "VarFileInfo"
BEGIN
VALUE "Translation", 0x409, 1200
END
END

View File

@@ -1,64 +0,0 @@
#!/bin/bash
#
# version.rc repeats the version that htsglobal.h declares. Signing enforces that
# every binary in a release reports the same one, so a drift fails the signing
# request on release day rather than the build. Assert the two agree.
set -euo pipefail
src="${top_srcdir:-..}/src"
h="$src/htsglobal.h"
rc="$src/version.rc"
for f in "$h" "$rc"; do
[ -f "$f" ] || {
echo "cannot find $f"
exit 1
}
done
# 3.49-12 (display) and 3.49.12 (dotted).
version=$(sed -n 's/^#define HTTRACK_VERSION[[:space:]][[:space:]]*"\([^"]*\)".*/\1/p' "$h")
versionid=$(sed -n 's/^#define HTTRACK_VERSIONID[[:space:]][[:space:]]*"\([^"]*\)".*/\1/p' "$h")
if [ -z "$version" ] || [ -z "$versionid" ]; then
echo "could not read the version from $h"
exit 1
fi
# The same version, as version.rc states it.
fileversion=$(sed -n 's/^[[:space:]]*FILEVERSION[[:space:]][[:space:]]*\(.*\)$/\1/p' "$rc" | tr -d ' \r')
productversion=$(sed -n 's/^[[:space:]]*PRODUCTVERSION[[:space:]][[:space:]]*\(.*\)$/\1/p' "$rc" | tr -d ' \r')
rc_fileversion=$(sed -n 's/.*VALUE "FileVersion",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
rc_productversion=$(sed -n 's/.*VALUE "ProductVersion",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
rc_productname=$(sed -n 's/.*VALUE "ProductName",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
# 3.49.12 -> 3,49,12,0
expected_numeric="$(echo "$versionid" | tr '.' ','),0"
fail=0
check() { # what expected actual
if [ "$2" != "$3" ]; then
echo "version.rc $1 is \"$3\", but htsglobal.h says it should be \"$2\""
fail=1
fi
}
check FILEVERSION "$expected_numeric" "$fileversion"
check PRODUCTVERSION "$expected_numeric" "$productversion"
check FileVersion "$versionid" "$rc_fileversion"
check ProductVersion "$version" "$rc_productversion"
# Signing pins the product name too.
check ProductName "HTTrack Website Copier" "$rc_productname"
[ "$fail" -eq 0 ] || exit 1
# And the shipped binary must agree with the header it was built from.
out=$(httrack --version)
case "$out" in
*"$version"*) ;;
*)
echo "httrack --version says \"$out\", which does not mention $version"
exit 1
;;
esac
echo "version resource agrees with htsglobal.h: $version ($expected_numeric)"

View File

@@ -68,7 +68,6 @@ TESTS = \
01_engine-urlhack.test \
01_engine-unescape-bounds.test \
01_engine-useragent.test \
01_engine-version-macros.test \
01_engine-xfread.test \
01_zlib-acceptencoding.test \
01_zlib-contentcodings.test \