Compare commits

..

5 Commits

Author SHA1 Message Date
Xavier Roche
c2c391e8af Store the .rc blobs LF-normalized, as .gitattributes declares
.gitattributes marks *.rc as "text eol=crlf": the blob is LF, the checkout is
CRLF. The two resource scripts added in #558 went in with CRLF blobs, so git
renormalizes them on every add and every checkout reports them as modified.
Renormalized with git add --renormalize; version.rc was already correct. The
working-tree files stay CRLF, which is what the Windows toolchain reads.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 09:00:51 +02:00
Xavier Roche
9ec5a3a067 Decode brotli and zstd content codings (#556)
* Decode the br and zstd content codings, and bound every decode

httrack advertised gzip and deflate only, so it took the oldest coding on
offer while every browser negotiates brotli (46% of CDN traffic) or zstd.
Worse, the response side had no codec identity at all: Content-Encoding was
collapsed into a boolean and hts_zunpack guessed the framing from the body
bytes, so a br or zstd body sent unsolicited (mis-keyed Vary caches do this)
fell through the identity path and was saved as the page, coded bytes and all.

Content-Encoding now maps to a codec (htscodec.c), and the decode dispatches on
it: brotli and zstd get streaming decoders, a known coding we cannot undo fails
the fetch instead of saving garbage, and an unrecognized token still means
identity, because servers do put charsets in that header. br and zstd are
advertised over TLS only, as browsers do.

On any decompression failure the undecoded body is now dropped rather than left
in memory for the writer to commit as the page. That closes the coded-bytes-as-
page hole for the new unsupported-coding path, and with it a latent leak on the
gzip path, where a body that failed to inflate (a truncated stream) was written
to disk verbatim.

Every decode is now bounded: 4096x the coded body, floor 1 MiB, ceiling INT_MAX.
Nothing capped the decoded size before, which was survivable while deflate was
the only codec (it cannot pass 1032x) but not with brotli and zstd, which reach
a million to one. The zlib accumulator also moves to LLint, so a body over 2 GiB
fails instead of overflowing an int.

libbrotlidec and libzstd are optional at configure time and required on Windows,
where libhttrack.dll links them from vcpkg and CI runs the self-tests to prove
the decoders are in.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix copyright year on the new codec files (2026, not 1998)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: probe the direct-to-disk decode-failure discard, and assert absence

The wire test only exercised the in-memory discard path (bad.html is HTML).
Add a non-HTML body under an unsupported coding (bin.dat) so the is_write
branch is covered too, and assert both are absent from the mirror rather than
grepping a file that the fix now removes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Factor the budget-enforced write shared by the brotli and zstd decoders

The two streaming decoders duplicated the decoded-size budget check, the
security-critical part of each loop. Pull it into one codec_sink helper so the
bomb ceiling lives in a single place. Behavior-preserving; the codec self-test
(exact decoded bytes, truncation and bomb rejection) proves equivalence.

Also compile the streaming-decode body only when a streaming codec is built,
which drops an unused-variable warning in the --without-brotli --without-zstd
build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 08:26:20 +02:00
Xavier Roche
cd38e6e68d Merge pull request #559 from xroche/fix/robots-default-comment
Put the robots.txt "(default)" label on the option that is the default
2026-07-13 20:51:23 +02:00
Xavier Roche
e692e56592 Put the robots.txt "(default)" label on the option that is the default
The enum said HTS_ROBOTS_SOMETIMES was the default. It is not: hts_init_opt
sets opt->robots = HTS_ROBOTS_ALWAYS (htslib.c:5990), and SOMETIMES is what
the -s option selects when the user asks for it (htscoremain.c:1234).

So HTTrack obeys robots.txt fully by default, and the header said the opposite.
Anyone reading it to answer "does this crawler respect robots.txt" got the
wrong answer from the one place they would look.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-13 20:37:03 +02:00
Xavier Roche
7d9d4ed890 Give the Windows binaries a version resource (#558)
libhttrack.dll and httrack.exe have never carried a VERSIONINFO: no product
name, no version, nothing. Right-click either one on Windows and Properties
has no Details worth reading.

That becomes a hard problem now that we are signing them. SignPath enforce
file metadata restrictions: every binary in a release must name the same
product and the same product version, and a binary that names nothing cannot
satisfy the rule. It fails the signing request, on release day, rather than
the build.

So add src/version.rc, and two wrappers that set the parts which differ per
binary. The version is spelled out there as numbers, because a VERSIONINFO
needs 3,49,12,0 and cannot take a string apart, and because htsglobal.h drags
in C declarations that a resource compiler has no business parsing.

That leaves the version written twice, which is how a version resource quietly
starts lying about which release it is. tests/01_engine-version-macros.test
fails the build if version.rc and htsglobal.h ever disagree, on any of the four
values, and on the product name too. I checked that it fires rather than
assuming: drift the numbers and it fails, restore them and it passes.

The product name is HTTrack Website Copier, on all of them. WinHTTrack is the
name of the Windows GUI, not of the command line program or of the engine.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 20:20:02 +02:00
10 changed files with 139 additions and 2 deletions

3
.gitattributes vendored Normal file
View File

@@ -0,0 +1,3 @@
# Resource scripts are Windows tooling input and must stay CRLF: the engine has no
# encoding guard, so an autocrlf checkout could otherwise flatten them silently.
*.rc text eol=crlf

View File

@@ -83,6 +83,9 @@ libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
EXTRA_DIST = httrack.h webhttrack \
version.rc \
libhttrack.rc \
httrack.rc \
coucal/murmurhash3.h.diff \
coucal/murmurhash3.h.orig \
minizip/iowin32.c \

View File

@@ -328,8 +328,8 @@ typedef enum hts_wizard {
typedef enum hts_robots {
HTS_ROBOTS_NEVER = 0, /**< ignore robots rules */
HTS_ROBOTS_SOMETIMES = 1, /**< partial obedience (default) */
HTS_ROBOTS_ALWAYS = 2, /**< obey robots rules */
HTS_ROBOTS_SOMETIMES = 1, /**< partial obedience, set by -s */
HTS_ROBOTS_ALWAYS = 2, /**< obey robots rules (default) */
HTS_ROBOTS_ALWAYS_STRICT = 3 /**< obey even strict rules */
} hts_robots;
#endif

5
src/httrack.rc Normal file
View File

@@ -0,0 +1,5 @@
// Version resource for httrack.exe, the command line program. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier (command line)"
#define VER_ORIGINAL_FILENAME "httrack.exe"
#define VER_FILETYPE VFT_APP
#include "version.rc"

View File

@@ -108,6 +108,9 @@
<Project>{e76ad871-54c1-45e8-a657-6117adeffb46}</Project>
</ProjectReference>
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="httrack.rc" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>

5
src/libhttrack.rc Normal file
View File

@@ -0,0 +1,5 @@
// Version resource for libhttrack.dll. See version.rc.
#define VER_FILE_DESCRIPTION "HTTrack Website Copier engine"
#define VER_ORIGINAL_FILENAME "libhttrack.dll"
#define VER_FILETYPE VFT_DLL
#include "version.rc"

View File

@@ -141,6 +141,9 @@
<ClCompile Include="minizip\zip.c" />
<ClCompile Include="punycode.c" />
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="libhttrack.rc" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
</Project>

50
src/version.rc Normal file
View File

@@ -0,0 +1,50 @@
// Version resource for libhttrack.dll and httrack.exe. Signing enforces that every
// binary in a release names the same product and version, so both need one.
// Spelled out here because a VERSIONINFO cannot take a version string apart;
// tests/01_engine-version-macros.test fails if this drifts from htsglobal.h.
// The per-binary parts come from libhttrack.rc / httrack.rc.
#include <winver.h>
#ifndef VER_FILE_DESCRIPTION
#define VER_FILE_DESCRIPTION "HTTrack Website Copier"
#endif
#ifndef VER_ORIGINAL_FILENAME
#define VER_ORIGINAL_FILENAME ""
#endif
#ifndef VER_FILETYPE
#define VER_FILETYPE VFT_APP
#endif
VS_VERSION_INFO VERSIONINFO
FILEVERSION 3, 49, 12, 0
PRODUCTVERSION 3, 49, 12, 0
FILEFLAGSMASK VS_FFI_FILEFLAGSMASK
#ifdef _DEBUG
FILEFLAGS VS_FF_DEBUG
#else
FILEFLAGS 0x0L
#endif
FILEOS VOS_NT_WINDOWS32
FILETYPE VER_FILETYPE
FILESUBTYPE VFT2_UNKNOWN
BEGIN
BLOCK "StringFileInfo"
BEGIN
BLOCK "040904b0" // U.S. English, Unicode
BEGIN
VALUE "CompanyName", "Xavier Roche"
VALUE "FileDescription", VER_FILE_DESCRIPTION
VALUE "FileVersion", "3.49.12"
VALUE "InternalName", VER_ORIGINAL_FILENAME
VALUE "LegalCopyright", "Copyright (C) 1998-2026 Xavier Roche and other contributors. GNU GPL v3 or later."
VALUE "OriginalFilename", VER_ORIGINAL_FILENAME
VALUE "ProductName", "HTTrack Website Copier"
VALUE "ProductVersion", "3.49-12"
END
END
BLOCK "VarFileInfo"
BEGIN
VALUE "Translation", 0x409, 1200
END
END

View File

@@ -0,0 +1,64 @@
#!/bin/bash
#
# version.rc repeats the version that htsglobal.h declares. Signing enforces that
# every binary in a release reports the same one, so a drift fails the signing
# request on release day rather than the build. Assert the two agree.
set -euo pipefail
src="${top_srcdir:-..}/src"
h="$src/htsglobal.h"
rc="$src/version.rc"
for f in "$h" "$rc"; do
[ -f "$f" ] || {
echo "cannot find $f"
exit 1
}
done
# 3.49-12 (display) and 3.49.12 (dotted).
version=$(sed -n 's/^#define HTTRACK_VERSION[[:space:]][[:space:]]*"\([^"]*\)".*/\1/p' "$h")
versionid=$(sed -n 's/^#define HTTRACK_VERSIONID[[:space:]][[:space:]]*"\([^"]*\)".*/\1/p' "$h")
if [ -z "$version" ] || [ -z "$versionid" ]; then
echo "could not read the version from $h"
exit 1
fi
# The same version, as version.rc states it.
fileversion=$(sed -n 's/^[[:space:]]*FILEVERSION[[:space:]][[:space:]]*\(.*\)$/\1/p' "$rc" | tr -d ' \r')
productversion=$(sed -n 's/^[[:space:]]*PRODUCTVERSION[[:space:]][[:space:]]*\(.*\)$/\1/p' "$rc" | tr -d ' \r')
rc_fileversion=$(sed -n 's/.*VALUE "FileVersion",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
rc_productversion=$(sed -n 's/.*VALUE "ProductVersion",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
rc_productname=$(sed -n 's/.*VALUE "ProductName",[[:space:]]*"\([^"]*\)".*/\1/p' "$rc")
# 3.49.12 -> 3,49,12,0
expected_numeric="$(echo "$versionid" | tr '.' ','),0"
fail=0
check() { # what expected actual
if [ "$2" != "$3" ]; then
echo "version.rc $1 is \"$3\", but htsglobal.h says it should be \"$2\""
fail=1
fi
}
check FILEVERSION "$expected_numeric" "$fileversion"
check PRODUCTVERSION "$expected_numeric" "$productversion"
check FileVersion "$versionid" "$rc_fileversion"
check ProductVersion "$version" "$rc_productversion"
# Signing pins the product name too.
check ProductName "HTTrack Website Copier" "$rc_productname"
[ "$fail" -eq 0 ] || exit 1
# And the shipped binary must agree with the header it was built from.
out=$(httrack --version)
case "$out" in
*"$version"*) ;;
*)
echo "httrack --version says \"$out\", which does not mention $version"
exit 1
;;
esac
echo "version resource agrees with htsglobal.h: $version ($expected_numeric)"

View File

@@ -68,6 +68,7 @@ TESTS = \
01_engine-urlhack.test \
01_engine-unescape-bounds.test \
01_engine-useragent.test \
01_engine-version-macros.test \
01_engine-xfread.test \
01_zlib-acceptencoding.test \
01_zlib-contentcodings.test \