9 Commits

Author SHA1 Message Date
Xavier Roche
a2a20ef684 Fix what Fedora's build patches around, and pin the installed file set
Fedora ships httrack with no patches, but its spec worked around three things
that were ours to fix, and its build broke twice in three weeks on changes to
our installed file set.

The metainfo declared a stock icon that is not one (stock names come from the
freedesktop Icon Naming Specification), which appstream-util rejects and which
also suppressed the merge from WebHTTrack.desktop that supplies the real icon.
libtest/readme.txt carried the pre-2005 FSF postal address. CI and
210_appstream-metainfo.test now run both validators.

225_install-manifest.test pins the installed file set, the class that broke
their 3.49.13 and 3.49.18 builds. It found a real bug on its first outing: the
.desktop files installed to $(prefix)/share/applications, so any --datadir
override put them where no desktop environment looks.
2026-08-06 10:50:56 +00:00
Xavier Roche
bfe0a4a7f7 The 16 px icon antialiases every stem to grey (#1031)
* Draw the small application icons on the pixel grid

The 16 px icon was the vector master downscaled, so every stem edge landed
on a fraction of a pixel: 74 colours in a 16x16 tile, four of them black.
The small sizes are now built in a viewBox equal to their pixel size, with
the outline coordinates rounded to whole pixels and no downscale after, so
the raster carries the field and the ink and nothing between them.

Still generated from the same Jost* master; the generator change is
httrack-works 90ceb7e2.

Test 228 counts the shipped raster's colours through a stdlib PNG reader,
since the suite cannot assume Pillow or ImageMagick on a build host.

Closes #938

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Tighten the new test's comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 11:33:03 +02:00
Xavier Roche
03c30e17ce The installed socket headers still need -D_POSIX_C_SOURCE under -std=c99 (#1028)
* The installed socket headers still need -D_POSIX_C_SOURCE under -std=c99

htsnet.h and htsopt.h were the last installed headers a strict-ISO consumer
had to prefix with -D_POSIX_C_SOURCE, and tests/206 carried an exception for
exactly those two. All of it came from one inline body: SOCaddr_inetntoa_
calls getnameinfo() and NI_NUMERICHOST, and no libc declares either to a
__STRICT_ANSI__ translation unit.

The body moves to a new src/htsnet.c and the exception in 206 goes away, so
all fourteen installed headers now compile under -std=c99 and -std=c11
against the strict-mode libc.

This turns a static inline into an exported symbol, an ABI addition rather
than a break, so VERSION_INFO gets a revision-only bump to 3:11:0 and the
soname and the Debian package stay put. proxytrack does not link libhttrack,
so it compiles htsnet.c in directly under -DHTS_NO_LIBHTTRACK, which leaves
the export marker bare there because MSVC rejects a dllimport definition.

Closes #1001

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin 206's header count to DevIncludes_DATA and unhide the strnlen check

The >= 10 floor let a header vanish from DevIncludes_DATA with 206 still
green, so #1001 could regress to "not installed" unnoticed: derive the
expected count from the list itself and name htsnet.h and htsopt.h beside
htssafe.h, since a dropped entry moves both sides of the count.

The "htssafe_strnlen_ is not libc's" preprocessor check sat inside the
gate asking whether a binary can run in TMPDIR, so a noexec TMPDIR or a
cross build skipped a check that needs no execution. Hoist it out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 08:59:27 +00:00
Xavier Roche
1a45e0237b An over-long FTP path or host name aborts the process instead of failing the link (#1029)
* FTP: refuse an over-long command instead of aborting on the copy

An FTP path or host name that survives the upstream clamps reached an
aborting strcpybuff and killed the process instead of failing the link.
The path case needs 1024 bytes exactly and nobody could reproduce it;
the host case is trivial, since the URL host was copied into a 256-byte
buffer with no check at all.

Every FTP control line now shares one FTP_LINE_SIZE, ftp_command()
reports a command that did not fit, and an over-long path or host fails
the transfer rather than being clipped into a request for another file.
send_line() also dropped the CRLF off a maximal command, and the
SIZE/MDTM probes reused the same builder.

Closes #1019

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the FTP control-line capacity and probe the host gate at its edge

The tests passed the fix without pinning it. Six mutants survived: widening
line_retr past the line[] it is copied into (that is #1019 verbatim), a stray
write one byte past ftp_command()'s bound, a host gate widened to reject every
host, and one moved off _adr[256] so a 256-byte host aborts again.

ftp_command_line() replaces the size argument at every build site with a
compile-time check that the buffer is FTP_LINE_SIZE, and the copy destination
is asserted equal to line_retr. The self-test now compares the whole poisoned
tail instead of one canary byte, and the crawl leg drives 255/256/257-byte
hosts plus a loopback outsider, so a widened gate no longer reads as a pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:51:33 +02:00
Xavier Roche
2a44dc1728 doc: note the --changes example is one captured run (#1027)
generator and date are genuine per-run fields (htschanges.c stamps
HTTRACK_VERSION and the current time on every write), so dropping them would
misrepresent the payload. Label the sample instead, so a reader does not
compare it against their own output as if it should match.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:51:25 +02:00
Xavier Roche
23e182281c The suite watchdog cannot see the engine on an emulated buildd (#1026)
* The suite watchdog cannot see the engine on an emulated buildd

Under qemu-user the process the kernel reports is the binfmt interpreter and
the engine is only its first argument, so the name matchers find nothing to
name or signal. Skip an interpreter before reading the command, gated on the
interpreter's own name: matching the argument unconditionally would take any
wrapper (strace, timeout) for the engine and signal that instead.

request_engine_backtraces only sends SIGABRT on Linux, so 105_suite-timeout
now asserts the death there and the "no stack mechanism known" report on the
platforms that get none, which is what Hurd's "GNU" uname takes.

Closes #1025

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Only shift past a real emulator, and cover the branch Hurd takes

qemu-img and its friends match "qemu-<word>" but take an image, not a
program, so shifting past them read a disk path as the process name --
into a list that is fed to kill. Exclude them, anchor the -binfmt form,
and give each a control row.

No CI leg reaches the no-stack-mechanism branch, which is the one the
Hurd buildds take, so drive it through a uname shim: it has to report
itself and leave the engine unsignalled.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 06:55:55 +00:00
Xavier Roche
3426ea35ea Release 3.49.18 (#1024)
* Release 3.49.18

Source-only release so packagers can pick up the test-suite build fixes:
3.49.17-1 was FTBFS on armhf, powerpc, hppa and loong64, and the suite
failed on a Fedora build root with no ps. The engine is untouched, so
VERSION_INFO moves by revision alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Wrap the 3.49.18 changelog entry under 80 columns

lintian's debian-changelog-line-too-long fires on two lines of the new
entry, and the CI gate runs --fail-on=error,warning. The sid perl
transition that had been masking this gate has cleared, so it is live
again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Sharpen the 3.49-18 release notes

The #1021 line named the symptom and implied the remedy was a louder
failure; the fix is a /proc fallback, so the diagnostics work. Add the
internal catch-all the 3.49-17 block carries, so the cross-arch matrix
appears to the upstream audience and not only in debian/changelog.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:43:50 +00:00
Xavier Roche
0274e016af Four architectures cannot build the altstack test shim, and nothing before the buildd sees it (#1023)
* Fix the four 3.49.17 buildd failures and cross-compile for those arches in CI

The LD_PRELOAD altstack shim did not build on Debian's 32-bit time64
architectures, and 183 read a frame count no unwinder there can produce.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Install the cross libc, and keep a trace-less report a failure

Review found the frame floor took a report naming no frame at all for a
weak unwinder, and the cross jobs had no target headers.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Give the cross jobs a libssl-free configure and assert what they built

No ports architecture has a cross libssl, and an empty TESTS= would have
passed having built nothing.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Add deb-src in place: a second stanza collides on the keyring spelling

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Extract zlib unpatched: Debian's arch patches need Debian's rules

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Turn off zlib's s390x vector CRC: the .dfsg repack drops its sources

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Drop zlib's s390x vx object too, not just its define

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Trim the raw_mmap comment

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the raw_mmap comment inside clang-format's width

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:43:51 +00:00
Xavier Roche
22fcf9ffff The suite watchdog dumps nothing on a host without ps (#1022)
* The suite watchdog dumps nothing on a host without ps

Fedora's build root ships no procps, so every process list in the hang
diagnostics came back empty: the guard still fired, but named no process
and could not signal the wedged engine for a stack. Read /proc directly
when ps is missing, and say so when neither source is available.

Closes #1021

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Harden the no-ps fallback and its test

Review follow-ups: a ps that exits 0 while listing nothing (hidepid, a
locked-down container) now falls back too, whitespace inside an argv can
no longer shift the columns the consumers match on, a failed /proc open
stops leaking to stderr, and a bogus CLK_TCK can no longer abort the
shell. The test leg pins the header line, the column order and the group
filter, and drives request_engine_backtraces itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* kill -0 succeeds on a zombie, so the SIGABRT check failed in a container

The fake engine is reparented once its subshell exits, and a container's
pid 1 does not always reap it, so the signalled process stays visible to
kill -0. Read its state from /proc instead: gone or Z both prove the
signal landed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:16:14 +00:00
34 changed files with 1131 additions and 173 deletions

View File

@@ -41,7 +41,8 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential clang autoconf automake libtool autoconf-archive \
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev
zlib1g-dev libssl-dev libbrotli-dev libzstd-dev \
appstream appstream-util
- name: Configure
run: |
@@ -715,7 +716,8 @@ jobs:
sudo apt-get update
# noble ships shfmt 3.8.0 (universe), matching the pinned local dev
# version; use it rather than fetching a release binary from github.com.
sudo apt-get install -y --no-install-recommends shellcheck shfmt appstream
sudo apt-get install -y --no-install-recommends shellcheck shfmt appstream \
appstream-util
shfmt --version
appstreamcli --version
@@ -742,6 +744,8 @@ jobs:
set -euo pipefail
appstreamcli validate --no-net --explain \
html/server/div/com.httrack.WebHTTrack.metainfo.xml
appstream-util validate-relax --nonet \
html/server/div/com.httrack.WebHTTrack.metainfo.xml
# Check clang-format on CHANGED LINES ONLY. The engine predates clang-format
# (it was shaped by an old Visual Studio formatter) and does not round-trip,

116
.github/workflows/cross-arch.yml vendored Normal file
View File

@@ -0,0 +1,116 @@
# Cross-compiles for the Debian architectures no GitHub runner exists for.
# 3.49.17 failed on four of them at once and every failure was a compile or
# assemble error that x86-64 and arm64 cannot produce, so the buildd was the
# first thing to see the code.
name: Cross-arch
on:
push:
branches: [master]
pull_request:
workflow_dispatch:
# Least privilege: the workflow only needs to read the repo.
permissions:
contents: read
# Cancel superseded runs on the same branch or PR.
concurrency:
group: cross-arch-${{ github.ref }}
cancel-in-progress: true
jobs:
cross:
name: cross (${{ matrix.arch }})
runs-on: ubuntu-24.04
# sid, not stable: the point is to predict what the sid buildds will say.
container: debian:sid
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
# arch selects the hardening flags, triplet is the toolchain prefix.
# The first four are 3.49.17's failures.
- { arch: armhf, triplet: arm-linux-gnueabihf }
- { arch: powerpc, triplet: powerpc-linux-gnu }
- { arch: hppa, triplet: hppa-linux-gnu }
- { arch: loong64, triplet: loongarch64-linux-gnu }
- { arch: sh4, triplet: sh4-linux-gnu }
- { arch: m68k, triplet: m68k-linux-gnu }
- { arch: sparc64, triplet: sparc64-linux-gnu }
- { arch: riscv64, triplet: riscv64-linux-gnu }
- { arch: s390x, triplet: s390x-linux-gnu }
env:
DEB_HOST_ARCH: ${{ matrix.arch }}
TRIPLET: ${{ matrix.triplet }}
ZPREFIX: /tmp/zlib-${{ matrix.arch }}
steps:
- name: Install the toolchain
run: |
set -euo pipefail
# deb-src, so zlib below comes from the archive and not a pinned URL.
# In place: a second stanza has to repeat Signed-By, and apt rejects
# the same source carrying two spellings of the keyring.
sed -i 's/^Types: deb$/Types: deb deb-src/' \
/etc/apt/sources.list.d/debian.sources
apt-get update
apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool autoconf-archive \
dpkg-dev git ca-certificates \
"gcc-$TRIPLET" "libc6-dev-$DEB_HOST_ARCH-cross"
- uses: actions/checkout@v7
with:
submodules: recursive
- name: Cross-build zlib
run: |
set -euo pipefail
# Mandatory, and no ports arch has a zlib1g-dev:<arch> to
# multiarch-install. -fPIC because libhttrack.so links it.
mkdir -p /tmp/zsrc && cd /tmp/zsrc
# Unpatched: Debian's arch-specific patches expect its own rules to
# build them (s390x's vector CRC pulls a header they add), and we only
# need something to link -lz against.
apt-get source --download-only zlib1g
dpkg-source --skip-patches -x ./*.dsc src
cd src
CHOST="$TRIPLET" CC="$TRIPLET-gcc" CFLAGS="-O2 -fPIC" \
./configure --prefix="$ZPREFIX" --static
# configure detects s390x's vector CRC, whose sources live in the
# contrib/ the .dfsg repack drops: drop the define and the object it
# would build. Nothing here needs a fast CRC.
sed -i -E 's/(-DHAVE_S390X_VX|crc32_vx\.l?o)//g' Makefile
make -j"$(nproc)"
make install
- name: Configure
run: |
set -euo pipefail
autoreconf -fi
mkdir -p /tmp/bld && cd /tmp/bld
# The buildd's own per-arch flags: -D_FILE_OFFSET_BITS=64 is the shim
# hazard, and -fstack-clash-protection is not offered everywhere.
eval "$(dpkg-buildflags --export=sh)"
# A cross AC_TRY_RUN answers "cross", which would swap in the bundled
# snprintf that no buildd ever builds. Every glibc passes these.
# --disable-https: no ports arch has a cross libssl, so the TLS paths
# are the one part of the tree this matrix does not compile.
"$GITHUB_WORKSPACE/configure" --host="$TRIPLET" \
--build="$(dpkg-architecture -qDEB_BUILD_GNU_TYPE)" --disable-https \
ac_cv_have_working_snprintf=yes ac_cv_have_working_vsnprintf=yes \
CPPFLAGS="${CPPFLAGS:-} -I$ZPREFIX/include" \
LDFLAGS="${LDFLAGS:-} -L$ZPREFIX/lib"
- name: Build
run: make -C /tmp/bld -j"$(nproc)"
- name: Build the test artifacts
# Nothing runs here, but the shims are where three of the four 3.49.17
# failures were: they only build under `check`. Asserted, because an
# empty TESTS= would otherwise pass having built nothing.
run: |
set -euo pipefail
make -C /tmp/bld -j"$(nproc)" check TESTS=
test -f /tmp/bld/tests/.libs/libaltstackprobe.so

View File

@@ -1,6 +1,6 @@
AC_PREREQ([2.71])
AC_INIT([httrack], [3.49.17], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_INIT([httrack], [3.49.18], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_COPYRIGHT([
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998-2015 Xavier Roche and other contributors
@@ -29,6 +29,9 @@ AC_CONFIG_SRCDIR(src/httrack.c)
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_HEADERS(config.h)
AM_INIT_AUTOMAKE([subdir-objects])
# 3:11:0: revision-only bump. #1001 moved SOCaddr_inetntoa_ out of htsnet.h, adding
# an export where callers had an inline; nothing changed or went away.
# 3:10:0: revision-only bump; only the version macro moved, the engine is untouched.
# 3:9:0: revision-only bump. #991 and #1005 each added an export
# (hts_set_thread_hooks, escape_control_url); nothing changed or went away.
# 3:8:0: revision-only bump, no ABI change.
@@ -37,7 +40,7 @@ AM_INIT_AUTOMAKE([subdir-objects])
# moves nothing). Soname stays .so.3: HTTrackQt is the only consumer of the installed
# headers, so a libhttrack4 rename isn't worth it.
# (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
VERSION_INFO="3:9:0"
VERSION_INFO="3:11:0"
AM_MAINTAINER_MODE
AC_USE_SYSTEM_EXTENSIONS

12
debian/changelog vendored
View File

@@ -1,3 +1,15 @@
httrack (3.49.18-1) unstable; urgency=medium
* New upstream release, fixing the 3.49.17-1 build failures on armhf,
powerpc, hppa and loong64: the test suite's mmap interposer did not
compile on the first three, and the alternate-stack test insisted on a
longer backtrace than loong64's unwinder produces. Nothing in the engine
changed. Upstream now cross-builds the test shims for the ports
architectures on every push, so the next such break surfaces before the
buildds do.
-- Xavier Roche <xavier@debian.org> Wed, 05 Aug 2026 23:00:24 +0200
httrack (3.49.17-1) unstable; urgency=medium
* New upstream release: security fixes in the FTP and HTTP request paths and

View File

@@ -4,6 +4,11 @@ HTTrack Website Copier release history:
This file lists all changes and fixes that have been made for HTTrack
3.49-18
+ Fixed: the 3.49.17 package build failed on four architectures: the test suite's mmap interposer did not compile on armhf, powerpc or hppa, and the alternate-stack test rejected loong64's shorter backtrace (#1023)
+ Fixed: the test suite failed on a host with no ps command, such as a Fedora build root; its hang diagnostics now read /proc directly (#1021)
+ Changed: multiple internal test and CI improvements, including a cross-compile matrix covering the Debian ports architectures
3.49-17
+ New: a pkg-config file, libhttrack.pc, ships with the development headers (#1018)
+ Fixed: a crawled page could read the WebHTTrack session id and drive the control panel; the id is now unguessable and cross-origin commands are refused (#877)

View File

@@ -15,7 +15,7 @@ WebIcon64x64dir = $(datadir)/icons/hicolor/64x64/apps
WebIcon128x128dir = $(datadir)/icons/hicolor/128x128/apps
WebIcon256x256dir = $(datadir)/icons/hicolor/256x256/apps
WebIconScalabledir = $(datadir)/icons/hicolor/scalable/apps
VFolderEntrydir = $(prefix)/share/applications
VFolderEntrydir = $(datadir)/applications
MetaInfodir = $(datadir)/metainfo
# Wildcards are globbed against $(srcdir): a bare "*.html" is resolved against

View File

@@ -155,6 +155,8 @@ rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
<h3 id="example">Example</h3>
One captured run; <tt>generator</tt> and <tt>date</tt> will read differently in yours.
<pre>
{
"schema": 1,

Binary file not shown.

Before

Width:  |  Height:  |  Size: 447 B

After

Width:  |  Height:  |  Size: 98 B

View File

@@ -26,7 +26,6 @@
</ul>
</description>
<launchable type="desktop-id">WebHTTrack.desktop</launchable>
<icon type="stock">httrack</icon>
<categories>
<category>Network</category>
</categories>
@@ -51,6 +50,13 @@
<content_rating type="oars-1.1"/>
<!-- Newest first; tests/01_engine-version-macros.test enforces it. -->
<releases>
<release version="3.49.18" date="2026-08-05">
<description>
<ul>
<li>The program itself is unchanged: this release only fixes building HTTrack from source on some Linux distributions and processors</li>
</ul>
</description>
</release>
<release version="3.49.17" date="2026-08-05">
<description>
<ul>

View File

@@ -1,37 +1,24 @@
/* XPM */
static char *httrack16x16[] = {
/* columns rows colors chars-per-pixel */
"16 16 15 1 ",
" c #0D0D10",
". c #191921",
"X c #2A2A38",
"o c #363647",
"O c #424257",
"+ c #53536E",
"@ c #717197",
"# c #7E7EA8",
"$ c #8585B1",
"% c #9898CB",
"& c #9999CC",
"* c #9A9ACE",
"= c #9696C9",
"- c #9C9CD0",
"; c #9999CC",
"16 16 2 1 ",
" c #040404",
". c #9999CC",
/* pixels */
"&&&&&&&&&%%&&&&&",
"&&&&&&&&%%%%&&&&",
"&&&&&&&&%%%%&&&&",
"&%%&&&%%%%%%%%%&",
"%+o$&%+o@ooooo+%",
"%o $-%X @X. .O%",
"%o @$$X $=@ o$=&",
"%o . $-# o&&&",
"%o oOo. $-# o%&&",
"%o $-%X $-# o%&&",
"%X $&%X $-# o%&&",
"%+o$&%+o$&$o+%&&",
"%%%&&&%%&&&%%&&&",
"%%&&&&&&&&&&&&&&",
"%&&&&&&&&&&&&&&&",
"&&&&&&&&&&&&&&&&"
"................",
"................",
"................",
"................",
". ... . .",
". ... . .",
". ... ... ...",
". ... ...",
". ... ...",
". ... ... ...",
". ... ... ...",
". ... ... ...",
"................",
"................",
"................",
"................"
};

View File

@@ -29,8 +29,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
along with this program. If not, see <http://www.gnu.org/licenses/>.
========================================================================
MAKEFILE PROJECT : libtest Project Overview

View File

@@ -43,7 +43,8 @@ htsserver_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS) libhttrack.la
proxytrack_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS)
httrack_CFLAGS = $(AM_CFLAGS) $(CFLAGS_PIE)
proxytrack_CFLAGS = $(AM_CFLAGS) $(CFLAGS_PIE) -DNO_MALLOCT -DZLIB_CONST -DHTS_INTHASH_USES_MD5
# -DHTS_NO_LIBHTTRACK: see htsnet.h's HTSNET_API guard.
proxytrack_CFLAGS = $(AM_CFLAGS) $(CFLAGS_PIE) -DNO_MALLOCT -DZLIB_CONST -DHTS_INTHASH_USES_MD5 -DHTS_NO_LIBHTTRACK
htsserver_CFLAGS = $(AM_CFLAGS) $(CFLAGS_PIE) -DZLIB_CONST -DHTS_INTHASH_USES_MD5
# @RPATH_ORIGIN_LDFLAGS@ makes a copied tree find libhttrack next to it (#906),
@@ -59,7 +60,7 @@ htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
htsurlport.c htsurlport.h
proxytrack_SOURCES = proxy/main.c \
proxy/proxytrack.c proxy/store.c \
htsurlport.c htsurlport.h \
htsurlport.c htsurlport.h htsnet.c \
coucal/coucal.c htsmd5.c md5.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c
@@ -80,7 +81,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htsname.c htsrobots.c htstools.c htswizard.c \
htsalias.c htsthread.c htsindex.c htsbauth.c \
htscrashtest.c \
htsmd5.c htscodec.c htswarc.c htschanges.c htssinglefile.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
htsmd5.c htsnet.c htscodec.c htswarc.c htschanges.c htssinglefile.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
md5.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \

View File

@@ -150,14 +150,21 @@ void ftp_split_userpass(const char *src, const char *end, char *user,
}
/* Build "<verb> <path>" (see htsftp.h). */
void ftp_command(char *line, size_t line_size, const char *verb,
const char *path) {
hts_boolean ftp_command(char *line, size_t line_size, const char *verb,
const char *path) {
int n;
/* A leading '-' would reach a server that shells out to ls as a flag. */
if (path[0] == '-' || strchr(path, ' ') != NULL ||
strchr(path, '\"') != NULL || strchr(path, '\'') != NULL)
snprintf(line, line_size, "%s \"%s\"", verb, path);
n = snprintf(line, line_size, "%s \"%s\"", verb, path);
else
snprintf(line, line_size, "%s %s", verb, path);
n = snprintf(line, line_size, "%s %s", verb, path);
if (n < 0 || (size_t) n >= line_size) {
line[0] = '\0'; // fail safe for a caller that ignores the result
return HTS_FALSE;
}
return HTS_TRUE;
}
/* MDTM reply "213 YYYYMMDDHHMMSS[.frac]" (RFC 3659, UTC) into tm_time. */
@@ -211,7 +218,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
httrackp *opt = pStruct->pOpt;
char user[256] = "anonymous";
char pass[256] = "user@";
char line_retr[2048];
char line_retr[FTP_LINE_SIZE];
int port = 21;
#if FTP_PASV
@@ -262,13 +269,19 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
if (strnotempty(a)) {
const size_t len_a =
strlen(unescape_http(ftp_path, sizeof(ftp_path), a));
hts_boolean fits;
if (len_a > 0 &&
ftp_path[len_a - 1] == '/') { /* obviously a directory listing */
transfer_list = 1;
ftp_command(line_retr, sizeof(line_retr), "LIST -A", ftp_path);
fits = ftp_command_line(line_retr, "LIST -A", ftp_path);
} else {
ftp_command(line_retr, sizeof(line_retr), "RETR", ftp_path);
fits = ftp_command_line(line_retr, "RETR", ftp_path);
}
if (!fits) {
strcpybuff(back->r.msg, "FTP path too long");
back->r.statuscode = STATUSCODE_INVALID;
_HALT_FTP return 0;
}
} else {
transfer_list = 1;
@@ -298,6 +311,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
SOCaddr server;
char *a;
char _adr[256];
size_t adr_len;
const char *error = "unknown error";
_adr[0] = '\0';
@@ -315,9 +329,16 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
_HALT_FTP return 0;
}
strncatbuff(_adr, adr, (int) (a - adr));
adr_len = (size_t) (a - adr);
} else
strcpybuff(_adr, adr);
adr_len = strlen(adr);
// no resolvable name is this long, and clipping would query another host
if (adr_len >= sizeof(_adr)) {
htsblk_failf(&back->r, "Host name too long");
back->r.statuscode = STATUSCODE_INVALID;
_HALT_FTP return 0;
}
strncatbuff(_adr, adr, (int) adr_len);
// récupérer adresse résolue
strcpybuff(back->info, "host name");
@@ -356,7 +377,10 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
_CHECK_HALT_FTP;
{
char BIGSTK line[1024];
char BIGSTK line[FTP_LINE_SIZE];
/* line_retr is copied here verbatim; a narrower line[] would clip it. */
HTS_COMPILE_ASSERT(sizeof(line) == sizeof(line_retr));
// envoi du login
@@ -514,9 +538,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
#endif
// SIZE
if (back->r.statuscode != -1) {
if (!transfer_list) {
ftp_command(line, sizeof(line), "SIZE", ftp_path);
// a clipped probe would size and date a different file
if (!transfer_list && ftp_command_line(line, "SIZE", ftp_path)) {
// SIZE?
strcpybuff(back->info, "size");
send_line(soc_ctl, line);
@@ -537,22 +560,24 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
// MDTM?
ftp_command(line, sizeof(line), "MDTM", ftp_path);
strcpybuff(back->info, "mdtm");
send_line(soc_ctl, line);
get_ftp_line(soc_ctl, line, sizeof(line), timeout);
_CHECK_HALT_FTP;
if (ftp_parse_mdtm(line, &remote_tm)) {
char date[256];
if (ftp_command_line(line, "MDTM", ftp_path)) {
strcpybuff(back->info, "mdtm");
send_line(soc_ctl, line);
get_ftp_line(soc_ctl, line, sizeof(line), timeout);
_CHECK_HALT_FTP;
if (ftp_parse_mdtm(line, &remote_tm)) {
char date[256];
time_rfc822(date, &remote_tm);
/* Stamp the mirror as the HTTP path does, so a later pass
compares server-clock times instead of crossing clocks. */
back->r.lastmodified[0] = '\0';
strlncatbuff(back->r.lastmodified, date,
sizeof(back->r.lastmodified),
sizeof(back->r.lastmodified) - 1);
remote_mtime = timegm(&remote_tm);
time_rfc822(date, &remote_tm);
/* Stamp the mirror as the HTTP path does, so a later pass
compares server-clock times instead of crossing clocks.
*/
back->r.lastmodified[0] = '\0';
strlncatbuff(back->r.lastmodified, date,
sizeof(back->r.lastmodified),
sizeof(back->r.lastmodified) - 1);
remote_mtime = timegm(&remote_tm);
}
}
/* Only over a copy back_add() judged partial: on --update every
@@ -570,7 +595,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
rest_understood = 1;
} // else never mind
}
} // sinon tant pis
} // sinon tant pis
}
}
#if FTP_PASV
@@ -613,7 +638,9 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
SOCaddr_initport(server, port_pasv);
if (connect(soc_dat, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
strcpybuff(back->info, "retr");
strcpybuff(line, line_retr);
// clip, never abort: this line is built from a crawled URL
line[0] = '\0';
strlncatbuff(line, line_retr, sizeof(line), sizeof(line) - 1);
send_line(soc_ctl, line);
get_ftp_line(soc_ctl, line, sizeof(line), timeout);
_CHECK_HALT_FTP;
@@ -660,7 +687,9 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
_CHECK_HALT_FTP;
if (line[0] == '2') { // ok
strcpybuff(back->info, "retr");
strcpybuff(line, line_retr);
// clip, never abort: this line is built from a crawled URL
line[0] = '\0';
strlncatbuff(line, line_retr, sizeof(line), sizeof(line) - 1);
send_line(soc_ctl, line);
get_ftp_line(soc_ctl, line, sizeof(line), timeout);
_CHECK_HALT_FTP;
@@ -922,7 +951,8 @@ FILE *dd = NULL;
// routines de réception/émission
// 0 = ERROR
int send_line(T_SOC soc, const char *data) {
char BIGSTK line[1024];
char BIGSTK line[FTP_LINE_SIZE + 2]; // room for the CRLF of a maximal command
int n;
// backstop: the driver fails earlier, but no injected byte reaches the wire
if (!hts_is_control_free(data))
@@ -942,7 +972,10 @@ int send_line(T_SOC soc, const char *data) {
printf("---> %s", data);
fflush(stdout);
#endif
snprintf(line, sizeof(line), "%s\x0d\x0a", data);
// an unterminated command would blend into whatever the server reads next
n = snprintf(line, sizeof(line), "%s\x0d\x0a", data);
if (n < 0 || n >= (int) sizeof(line))
return 0;
if (check_socket_connect(soc) != 1) {
#if FTP_DEBUG
printf("!SOC WRITE ERROR\n");

View File

@@ -59,6 +59,9 @@ struct FTPDownloadStruct {
/* Library internal definictions */
#ifdef HTS_INTERNAL_BYTECODE
/* Capacity of every FTP control-line buffer; send_line() adds the CRLF. */
#define FTP_LINE_SIZE 1024
#if USE_BEGINTHREAD
void launch_ftp(FTPDownloadStruct * params);
void back_launch_ftp(void *pP);
@@ -75,11 +78,17 @@ int get_ftp_line(T_SOC soc, char *line, size_t line_size, int timeout);
Both sizes must be nonzero. */
void ftp_split_userpass(const char *src, const char *end, char *user,
size_t user_size, char *pass, size_t pass_size);
/* Build "<verb> <path>" into line[line_size], truncating to fit. The path is
quoted whenever a bare one would give the server a second token; it must
already have been screened for control bytes. */
void ftp_command(char *line, size_t line_size, const char *verb,
const char *path);
/* Build "<verb> <path>" into line[line_size]. The path is quoted whenever a
bare one would give the server a second token; it must already have been
screened for control bytes. Returns HTS_FALSE and empties line when the
command does not fit, as a clipped one would name a different file. */
hts_boolean ftp_command(char *line, size_t line_size, const char *verb,
const char *path);
/* ftp_command() into a control line of the one capacity every FTP buffer has;
anything narrower fails the build rather than refusing a path that fits. */
#define ftp_command_line(line, verb, path) \
(HTS_COMPILE_ASSERT(sizeof(line) == FTP_LINE_SIZE), \
ftp_command((line), sizeof(line), (verb), (path)))
T_SOC get_datasocket(char *to_send, size_t to_send_size);
int stop_ftp(lien_back * back);
char *linejmp(char *line);

View File

@@ -43,8 +43,8 @@ Please visit our Website: http://www.httrack.com
configure.ac, decoupled from these). VERSION is the display form, VERSIONID
the dotted numeric form, AFF_VERSION the short form shown in footers,
LIB_VERSION the data/cache format generation. */
#define HTTRACK_VERSION "3.49-17"
#define HTTRACK_VERSIONID "3.49.17"
#define HTTRACK_VERSION "3.49-18"
#define HTTRACK_VERSIONID "3.49.18"
#define HTTRACK_AFF_VERSION "3.x"
#define HTTRACK_LIB_VERSION "2.0"
@@ -273,6 +273,9 @@ typedef int hts_tristate;
/* True when A is a non-NULL, non-empty string. */
#define strnotempty(A) (((A) != NULL && (A)[0] != '\0'))
/* Compile-time check, usable as an expression. */
#define HTS_COMPILE_ASSERT(cond) ((void) sizeof(char[(cond) ? 1 : -1]))
/* 'inline' where the dialect supports it (C++), nothing in plain C. */
#ifdef __cplusplus
#define HTS_INLINE inline

52
src/htsnet.c Normal file
View File

@@ -0,0 +1,52 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 2026 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: Out-of-line htsnet.h helpers, kept here so the installed */
/* header needs nothing beyond strict ISO C */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#include "htsnet.h"
HTSNET_API void SOCaddr_inetntoa_(char *namebuf, size_t namebuflen,
SOCaddr *const ss, const char *file,
const int line) {
assertf_(namebuf != NULL, file, line);
assertf_(ss != NULL, file, line);
if (getnameinfo(&ss->m_addr.sa, sizeof(ss->m_addr), namebuf, namebuflen, NULL,
0, NI_NUMERICHOST) == 0) {
/* remove scope id(s) */
char *const pos = strchr(namebuf, '%');
if (pos != NULL) {
*pos = '\0';
}
} else {
namebuf[0] = '\0';
}
}

View File

@@ -273,25 +273,20 @@ static HTS_UNUSED socklen_t SOCaddr_copyaddr_(SOCaddr *const server,
__LINE__); \
} while (0)
/** Write the numeric (dotted/colon) host of ss into namebuf (capacity
namebuflen), scope id stripped. On failure namebuf becomes "". */
static HTS_UNUSED void SOCaddr_inetntoa_(char *namebuf, size_t namebuflen,
SOCaddr *const ss, const char *file,
const int line) {
assertf_(namebuf != NULL, file, line);
assertf_(ss != NULL, file, line);
/* proxytrack compiles htsnet.c in rather than linking the library, and MSVC
rejects a dllimport definition. */
#ifdef HTS_NO_LIBHTTRACK
#define HTSNET_API
#else
#define HTSNET_API HTSEXT_API
#endif
if (getnameinfo(&ss->m_addr.sa, sizeof(ss->m_addr), namebuf, namebuflen, NULL,
0, NI_NUMERICHOST) == 0) {
/* remove scope id(s) */
char *const pos = strchr(namebuf, '%');
if (pos != NULL) {
*pos = '\0';
}
} else {
namebuf[0] = '\0';
}
}
/** Write the numeric (dotted/colon) host of ss into namebuf (capacity
namebuflen), scope id stripped. On failure namebuf becomes "". Out of line:
getnameinfo() isn't declared to a strict-ISO translation unit (#1001). */
HTSNET_API void SOCaddr_inetntoa_(char *namebuf, size_t namebuflen,
SOCaddr *const ss, const char *file,
const int line);
/** Numeric host of ss into namebuf (capacity namebuflen); "" on failure. */
#define SOCaddr_inetntoa(namebuf, namebuflen, ss) \

View File

@@ -4540,6 +4540,75 @@ static int st_ftpuser(httrackp *opt, int argc, char **argv) {
return 0;
}
/* Both quoting forms at two capacities: the quoted form is two bytes wider
(#1019). */
static int st_ftpcmdlen(httrackp *opt, int argc, char **argv) {
static const size_t caps[] = {32, FTP_LINE_SIZE};
char BIGSTK buf[FTP_LINE_SIZE + 32];
char BIGSTK poison[FTP_LINE_SIZE + 32];
char BIGSTK path[FTP_LINE_SIZE + 2];
char BIGSTK wire[FTP_LINE_SIZE * 2];
size_t c, got = 0;
T_SOC sv[2];
(void) opt;
(void) argc;
(void) argv;
memset(poison, '#', sizeof(poison));
for (c = 0; c < sizeof(caps) / sizeof(caps[0]); c++) {
const size_t cap = caps[c];
int quoted;
for (quoted = 0; quoted <= 1; quoted++) {
const size_t verb = 5 + 2 * (size_t) quoted; /* "RETR " plus quotes */
const size_t fit = cap - 1 - verb; /* longest path still fitting */
size_t len;
for (len = fit - 1; len <= fit + 1; len++) {
memset(path, 'p', len);
path[len] = '\0';
if (quoted)
path[0] = ' '; /* any of these forces the quoted form */
memcpy(buf, poison, sizeof(buf)); /* a zero canary would hide a NUL */
if (len > fit) {
assertf(ftp_command(buf, cap, "RETR", path) == HTS_FALSE);
assertf(buf[0] == '\0'); /* fail-safe for an ignored result */
} else {
assertf(ftp_command(buf, cap, "RETR", path) == HTS_TRUE);
assertf(strlen(buf) == verb + len);
assertf(strncmp(buf, "RETR ", 5) == 0);
assertf(buf[verb + len - 1] == (quoted ? '\"' : 'p'));
}
/* the whole tail: one canary byte misses a write just past it */
assertf(memcmp(buf + cap, poison, sizeof(buf) - cap) == 0);
}
}
}
/* send_line() adds the CRLF and drops, rather than truncates, an over-length
line. */
memset(path, 'q', FTP_LINE_SIZE);
path[FTP_LINE_SIZE] = '\0';
assertf(st_socketpair(sv) == 0);
assertf(send_line(sv[0], path) == 0); /* one byte too long: never sent */
path[FTP_LINE_SIZE - 1] = '\0';
assertf(send_line(sv[0], path) != 0);
deletesoc(sv[0]);
for (;;) {
const int n = (int) recv(sv[1], wire + got, (int) (sizeof(wire) - got), 0);
if (n <= 0)
break;
got += (size_t) n;
}
deletesoc(sv[1]);
assertf(got == FTP_LINE_SIZE + 1); /* the maximal command alone */
assertf(memcmp(wire, path, FTP_LINE_SIZE - 1) == 0);
assertf(memcmp(wire + FTP_LINE_SIZE - 1, "\r\n", 2) == 0);
printf("ftp-cmdlen self-test OK (%d bytes sent)\n", (int) got);
return 0;
}
/* send_line() must drop a command line carrying a control byte (#1010). */
static int st_ftpctrl(httrackp *opt, int argc, char **argv) {
/* Verb and URL path as run_launch_ftp() hands them over, then the line the
@@ -8120,6 +8189,8 @@ static const struct selftest_entry {
{"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser},
{"ftp-ctrlchars", "", "send_line rejects a control byte in an FTP command",
st_ftpctrl},
{"ftp-cmdlen", "",
"an FTP command too long for its control line is refused", st_ftpcmdlen},
{"warc", "<dir>", "WARC/1.1 writer: framing, digests, revisit dedup",
st_warc},
{"warc-trunc", "<dir>", "WARC-Truncated on a cap-truncated body",

View File

@@ -126,6 +126,7 @@
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="htsnet.c" />
<ClCompile Include="htsmodules.c" />
<ClCompile Include="htsname.c" />
<ClCompile Include="htsparse.c" />

View File

@@ -57,7 +57,7 @@
<ClCompile>
<!-- Matches proxytrack_CFLAGS in Makefile.am. Standalone: it does not link
libhttrack, it only borrows headers (hts_effective_mime is a macro). -->
<PreprocessorDefinitions>WIN32;_CONSOLE;_MBCS;NO_MALLOCT;ZLIB_CONST;HTS_INTHASH_USES_MD5;ZLIB_DLL;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<PreprocessorDefinitions>WIN32;_CONSOLE;_MBCS;NO_MALLOCT;HTS_NO_LIBHTTRACK;ZLIB_CONST;HTS_INTHASH_USES_MD5;ZLIB_DLL;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<AdditionalIncludeDirectories>$(MSBuildThisFileDirectory);$(MSBuildThisFileDirectory)coucal;$(MSBuildThisFileDirectory)proxy;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
<WarningLevel>Level3</WarningLevel>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
@@ -100,6 +100,7 @@
<ClCompile Include="proxy\proxytrack.c" />
<ClCompile Include="proxy\store.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="htsnet.c" />
<ClCompile Include="coucal\coucal.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="md5.c" />

View File

@@ -17,8 +17,8 @@
#endif
VS_VERSION_INFO VERSIONINFO
FILEVERSION 3, 49, 17, 0
PRODUCTVERSION 3, 49, 17, 0
FILEVERSION 3, 49, 18, 0
PRODUCTVERSION 3, 49, 18, 0
FILEFLAGSMASK VS_FFI_FILEFLAGSMASK
#ifdef _DEBUG
FILEFLAGS VS_FF_DEBUG
@@ -35,12 +35,12 @@ BEGIN
BEGIN
VALUE "CompanyName", "Xavier Roche"
VALUE "FileDescription", VER_FILE_DESCRIPTION
VALUE "FileVersion", "3.49.17"
VALUE "FileVersion", "3.49.18"
VALUE "InternalName", VER_ORIGINAL_FILENAME
VALUE "LegalCopyright", "Copyright (C) 1998-2026 Xavier Roche and other contributors. GNU GPL v3 or later."
VALUE "OriginalFilename", VER_ORIGINAL_FILENAME
VALUE "ProductName", "HTTrack Website Copier"
VALUE "ProductVersion", "3.49-17"
VALUE "ProductVersion", "3.49-18"
END
END
BLOCK "VarFileInfo"

View File

@@ -65,7 +65,7 @@ grep -q 'slow but healthy' "$out" || fail "healthy test output lost"
# The killed tree must really be gone, or the next test inherits its ports.
sleep 1
! grep -q "$tmp/90_wedged.test" <<<"$(ps -A -o args 2>/dev/null)" ||
! grep -q "$tmp/90_wedged.test" <<<"$(ps_snapshot)" ||
fail "the wedged test survived the guard"
# --- the budget is wall clock, not a count of poll iterations ----------------
@@ -139,6 +139,121 @@ HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/94_pacer.test" >"$out" 2>&1 || rc=
test "$rc" -eq 0 || fail "a 60s step under a 600s budget reported $rc, want 0"
grep -q 'the pacer let it through' "$out" || fail "the pacer skipped a test that fits"
# --- the process lists survive a host with no ps ----------------------------
# Fedora's build root ships no procps, and the guard then named nothing (#1021).
if ! is_windows && test -r /proc/self/stat; then
mkdir -p "$tmp/nops"
printf '#!/bin/sh\nexit 127\n' >"$tmp/nops/ps"
chmod +x "$tmp/nops/ps"
# Both lists match on the basename alone, so any long-lived binary will do.
ln -sf "$(command -v sleep)" "$tmp/httrack"
# Started from a subshell so ppid and pgid differ: swapping those two columns
# is invisible whenever the test shell happens to lead its own group.
(
"$tmp/httrack" 60 &
echo $! >"$tmp/fake.pid"
)
# Its own group, to prove the filter keeps an outsider out.
(
set -m
"$tmp/httrack" 60 &
echo $! >"$tmp/outside.pid"
)
fake=$(cat "$tmp/fake.pid")
outside=$(cat "$tmp/outside.pid")
trap 'set +e; kill "$fake" "$outside" 2>/dev/null; rm -rf "$tmp"' EXIT
# ppid and pgid of $1: the oracle the listing is checked against.
procfields() { awk '{ sub(/.*\) /, ""); print $2, $3 }' "/proc/$1/stat"; }
procstate() { awk '{ sub(/.*\) /, ""); print $1 }' "/proc/$1/stat" 2>/dev/null || echo gone; }
read -r fppid pgid _ <<<"$(procfields "$fake")"
read -r _ opgid _ <<<"$(procfields "$outside")"
test "$fppid" != "$pgid" || fail "the fake shares ppid and pgid ($pgid)"
test "$opgid" != "$pgid" || fail "the outsider landed in the group under test"
(
PATH="$tmp/nops:$PATH"
! ps -A >/dev/null 2>&1 || fail "the ps shim did not take"
snap=$(list_stray_processes "$pgid" group)
# Line 1 is the header every consumer drops; a row there would be lost.
case "$(head -n1 <<<"$snap")" in
[0-9]*) fail "the no-ps listing has no header: $snap" ;;
esac
! grep -qE "^$outside " <<<"$snap" ||
fail "no-ps group list reported pid $outside, in another group"
read -r _ rppid rpgid _ <<<"$(grep -E "^$fake " <<<"$snap")"
test "$rppid $rpgid" = "$fppid $pgid" ||
fail "no-ps row for $fake reads ppid/pgid '$rppid $rpgid', want '$fppid $pgid'"
engines=$(list_engine_pids "$pgid")
grep -qx "$fake" <<<"$engines" || fail "no-ps engine list lost pid $fake"
! grep -qx "$outside" <<<"$engines" ||
fail "no-ps engine list reached outside the group (pid $outside)"
# A platform with no stack mechanism must say so and leave the engine
# alone. Shimmed, because no CI leg is one: the Hurd branch below runs
# only on the buildds that failed for want of it.
# shellcheck disable=SC2016 # the shim reads them, not us
printf '#!/bin/sh\ncase "$1" in -s) echo GNU;; *) exec %s "$@";; esac\n' \
"$(command -v uname)" >"$tmp/nops/uname"
chmod +x "$tmp/nops/uname"
grep -q "no stack mechanism known for GNU" <<<"$(request_engine_backtraces "$pgid")" ||
fail "a platform with no stack mechanism reported nothing"
case "$(procstate "$fake")" in
Z | gone) fail "the no-mechanism branch signalled pid $fake" ;;
esac
rm -f "$tmp/nops/uname"
# The chain, not just its input: this is what reported nothing on Fedora.
stacks=$(request_engine_backtraces "$pgid")
! grep -q 'no engine process left' <<<"$stacks" ||
fail "no-ps stack request found no engine to signal: $stacks"
# Only Linux signals; elsewhere (Hurd's uname reports "GNU") the chain
# owes the reader its no-mechanism report instead of silence.
if test "$(uname -s)" = Linux; then
# Gone, or a zombie: the fake is reparented, and a container's pid 1
# does not always reap. kill -0 succeeds on a zombie, proving nothing.
case "$(procstate "$fake")" in
Z | gone) ;;
*) fail "the SIGABRT never reached pid $fake" ;;
esac
else
grep -q 'no stack mechanism known' <<<"$stacks" ||
fail "no stack mechanism was reported on $(uname -s): $stacks"
fi
# With neither source the dump must say so, not print an empty section.
# shellcheck disable=SC2317 # reached through ps_snapshot
proc_snapshot() { return 1; }
grep -q 'no process list' <<<"$(list_stray_processes "$pgid" group)" ||
fail "a host with no ps and no /proc produced no notice"
)
kill "$fake" "$outside" 2>/dev/null || true
fi
# --- an emulated buildd hides the engine behind its binfmt interpreter -------
# Synthetic rows: no runner here has qemu-user, and the column shift is the point.
if ! is_windows; then
(
# shellcheck disable=SC2317 # reached through the two matchers below
ps_snapshot() {
cat <<'EOF'
PID PPID PGID ELAPSED S COMMAND
11 1 40 12 S /usr/libexec/qemu-binfmt/hppa-binfmt-P /bld/src/httrack -q http://h/
12 1 40 12 S /usr/bin/qemu-hppa-static /usr/bin/python3 /t/local-server.py 8080
13 1 40 12 S /bld/src/httrack -q http://h/
14 1 40 12 S /usr/bin/strace /bld/src/httrack -q http://h/
15 1 40 12 S /usr/bin/qemu-img convert /srv/local-server.py out.raw
16 1 40 12 S /usr/bin/qemu-nbd /mnt/httrack
17 1 40 12 S /opt/my-custom-binfmt /mnt/httrack
EOF
}
# Rows 14 to 17 are the controls: shifting past anything but an emulator
# would take a wrapper, or a disk image, for the engine and kill it.
engines=$(list_engine_pids 40)
test "$engines" = $'11\n13' ||
fail "emulated engine list is '$engines', want pids 11 and 13"
named=$(list_stray_processes 0 named | awk 'NR > 1 { print $1 }')
test "$named" = $'11\n12\n13' ||
fail "emulated stray list is '$named', want pids 11, 12 and 13"
)
fi
# --- a wedged crawl yields a symbolized engine stack ------------------------
# The whole point of the dump: name the frame the engine is stuck in. Windows has
# neither half (MSYS signals do not reach a native httrack.exe, and that build has

View File

@@ -63,11 +63,24 @@ grep -q "^Caught signal 11$" <<<"$worker" || fail "-#c=threadstack: no 'Caught s
# worker's, which is what had no alternate stack.
worker_frames=$(frame_count "$worker")
plain_frames=$(frame_count "$plain")
test "$worker_frames" -ge 100 ||
fail "-#c=threadstack: $worker_frames frames, expected a runaway recursion"
test "$plain_frames" -lt 100 ||
fail "-#c=segv: $plain_frames frames, the threshold no longer discriminates"
# Naming no frame at all is the unwinder failing outright, not a weak one, and
# the floor below would take it for loong64. hts_print_backtrace() says which.
if grep -q "No stack trace available" <<<"$worker"; then
fail "-#c=threadstack: the report carries no stack trace at all"
fi
# armhf and loong64 cannot unwind past the frame that faulted on the guard page
# (180 skips on the same floor); the release trace below still judges the fix.
if [ "$worker_frames" -ge 1 ] && [ "$worker_frames" -lt 6 ]; then
echo "-#c=threadstack: $worker_frames frames, too few for this unwinder to" \
"show the recursion" >&2
elif [ "$worker_frames" -lt 100 ]; then
fail "-#c=threadstack: $worker_frames frames, expected a runaway recursion"
fi
# Everything above only proves the stack gets installed: the crashing worker
# never returns, so the release hook never runs there. -#test=threadwait spawns
# workers that do return, and the syscall trace says what each did with its

View File

@@ -88,32 +88,36 @@ env MAKEFLAGS= MFLAGS= "$make" -C "$abs_top_builddir/src" install-DevIncludesDAT
}
headers=("$tmp/include/httrack"/*.h)
[ "${#headers[@]}" -ge 10 ] || fail "only ${#headers[@]} headers installed, the list cannot be right"
[ -f "$tmp/include/httrack/htssafe.h" ] || fail "htssafe.h was not installed"
# No libc hands getnameinfo() to a strict-ISO unit, so a consumer of the two
# socket headers must ask for POSIX. Prefer 2001: 2008 also un-hides strnlen,
# which would stop those two covering that half of #972.
posix_argv=(-D_POSIX_C_SOURCE=200112L)
printf '#include <httrack/htsnet.h>\n' >"$tmp/tu.c"
if ! "${cc_argv[@]}" "${cpp_argv[@]}" "${posix_argv[@]}" "-std=${stds[0]}" \
-fsyntax-only "$tmp/tu.c" 2>/dev/null; then
fail "htsnet.h needs POSIX.1-2008, which un-hides strnlen and would leave that half of #972 uncovered"
fi
# Derived from DevIncludes_DATA so a new header needs no edit here. A header
# dropped from that list moves both sides, hence the named checks below.
declared=$(awk '/^DevIncludes_DATA[[:space:]]*=/ { inlist = 1 }
inlist {
last = ($0 !~ /\\$/)
sub(/^[^=]*=/, "")
gsub(/\\/, " ")
for (i = 1; i <= NF; i++)
if ($i ~ /\.h$/) n++
if (last) exit
}
END { print n + 0 }' "${abs_top_srcdir:?}/src/Makefile.am")
[ "$declared" -ge 10 ] || fail "read only $declared headers out of DevIncludes_DATA, the parse is wrong"
[ "${#headers[@]}" -eq "$declared" ] ||
fail "${#headers[@]} headers installed, DevIncludes_DATA declares $declared"
for h in htssafe.h htsnet.h htsopt.h; do
[ -f "$tmp/include/httrack/$h" ] || fail "$h was not installed"
done
# No -D_POSIX_C_SOURCE anywhere below: every installed header, htsnet.h and
# htsopt.h included, must compile against the strict-ISO libc alone (#1001).
count=0
bad=0
for h in "${headers[@]}"; do
b=$(basename "$h")
argv=("${cpp_argv[@]}")
case "$b" in
htsnet.h | htsopt.h) argv+=("${posix_argv[@]}") ;;
esac
printf '#include <httrack/%s>\n' "$b" >"$tmp/tu.c"
for std in "${stds[@]}"; do
for mode in -UHTS_INTERNAL_BYTECODE -DHTS_INTERNAL_BYTECODE; do
count=$((count + 1))
if ! "${cc_argv[@]}" "${argv[@]}" "-std=$std" "$mode" -fsyntax-only "$tmp/tu.c" 2>"$tmp/cc.log"; then
if ! "${cc_argv[@]}" "${cpp_argv[@]}" "-std=$std" "$mode" -fsyntax-only "$tmp/tu.c" 2>"$tmp/cc.log"; then
echo "$b does not compile under -std=$std ($mode):" >&2
head -5 "$tmp/cc.log" >&2
bad=1
@@ -122,6 +126,7 @@ for h in "${headers[@]}"; do
done
done
echo "compiled ${#headers[@]} headers x ${#stds[@]} std x 2 bytecode modes = $count units" >&2
[ "$count" -eq $((${#headers[@]} * ${#stds[@]} * 2)) ] || fail "the compile loop skipped units"
[ "$bad" -eq 0 ] || fail "installed headers do not compile in strict-ISO mode"
# A macro body is only compiled where it expands, so the include-only loop above
@@ -178,19 +183,23 @@ int main(void) {
return 0;
}
EOF
# Vacuous if the gate handed us libc's strnlen. Match that branch, not the
# loop, or a mutated loop reads as a lost fallback instead. Needs no runnable
# binary, so it stays outside the gate below.
for std in "${stds[@]}"; do
pp=$("${cc_argv[@]}" "${cpp_argv[@]}" "-std=$std" -E "$tmp/strnlen.c") ||
fail "cannot preprocess the htssafe_strnlen_ probe under -std=$std"
if grep -q 'return strnlen' <<<"$pp"; then
fail "-std=$std left htssafe_strnlen_ on libc, the differential proves nothing"
fi
done
# A noexec TMPDIR or a cross-compiler would red the run for the wrong reason.
printf 'int main(void) { return 0; }\n' >"$tmp/tu.c"
if ! ("${cc_argv[@]}" "${cpp_argv[@]}" -o "$tmp/tu" "$tmp/tu.c" 2>/dev/null && "$tmp/tu"); then
echo "cannot run a binary built in $tmp, skipping the htssafe_strnlen_ differential" >&2
else
for std in "${stds[@]}"; do
# Vacuous if the gate handed us libc's strnlen. Match that branch, not
# the loop, or a mutated loop reads as a lost fallback instead.
pp=$("${cc_argv[@]}" "${cpp_argv[@]}" "-std=$std" -E "$tmp/strnlen.c") ||
fail "cannot preprocess the htssafe_strnlen_ probe under -std=$std"
if grep -q 'return strnlen' <<<"$pp"; then
fail "-std=$std left htssafe_strnlen_ on libc, the check below proves nothing"
fi
"${cc_argv[@]}" "${cpp_argv[@]}" "-std=$std" -o "$tmp/strnlen" "$tmp/strnlen.c" 2>"$tmp/cc.log" || {
head -5 "$tmp/cc.log" >&2
fail "the htssafe_strnlen_ probe does not build under -std=$std"
@@ -221,9 +230,6 @@ else
for h in "${headers[@]}"; do
b=$(basename "$h")
argv=("${base_cpp_argv[@]}" -Wall -Wextra -Werror)
case "$b" in
htsnet.h | htsopt.h) argv+=("${posix_argv[@]}") ;;
esac
printf '#include <httrack/%s>\nint main() { return 0; }\n' "$b" >"$tmp/tu.cpp"
for std in "${cxx_stds[@]}"; do
for mode in -UHTS_INTERNAL_BYTECODE -DHTS_INTERNAL_BYTECODE; do

View File

@@ -6,14 +6,34 @@
set -euo pipefail
srcdir="${abs_top_srcdir:?not run under make check}"
metainfo="${srcdir}/html/server/div/com.httrack.WebHTTrack.metainfo.xml"
command -v appstreamcli >/dev/null 2>&1 || {
echo "appstreamcli not installed; skipping" >&2
# --no-net/--nonet: validate the file, not the reachability of the screenshot
# host. Fedora's build gates on appstream-util, stricter about icons and
# metadata than appstreamcli.
validate() {
case $1 in
appstreamcli) appstreamcli validate --no-net --explain "$2" ;;
appstream-util) appstream-util validate-relax --nonet "$2" ;;
*)
echo "FAIL: no validator named $1" >&2
exit 1
;;
esac
}
tools=()
for tool in appstreamcli appstream-util; do
command -v "${tool}" >/dev/null 2>&1 && tools+=("${tool}")
done
test ${#tools[@]} -gt 0 || {
echo "no appstream validator installed; skipping" >&2
exit 77
}
test -f "${metainfo}" || {
echo "FAIL: ${metainfo} not found" >&2
# html/Makefile.am installs these by glob; a second one must not ship unchecked.
metainfos=("${srcdir}"/html/server/div/*.metainfo.xml)
test -f "${metainfos[0]}" || {
echo "FAIL: no metainfo under ${srcdir}/html/server/div" >&2
exit 1
}
@@ -23,21 +43,18 @@ work=$(mktemp -d "${TMPDIR:-/tmp}/appstream.XXXXXX") || {
}
trap 'set +e; rm -rf "${work}"' EXIT
# --no-net: validate the file, not the reachability of the screenshot host.
validate() { appstreamcli validate --no-net --explain "$1"; }
validate "${metainfo}" || {
echo "FAIL: metainfo does not validate" >&2
exit 1
}
# Positive control: an appstreamcli that never fails would pass the above.
grep -v '<id>com.httrack.WebHTTrack</id>' "${metainfo}" >"${work}/noid.xml"
if validate "${work}/noid.xml" >"${work}/noid.log" 2>&1; then
echo "FAIL: metainfo without <id> validated; the check proves nothing" >&2
exit 1
fi
grep -q component-id-missing "${work}/noid.log" ||
echo "note: id-less copy failed for another reason: $(tail -3 "${work}/noid.log")" >&2
echo "appstream metainfo: valid"
for metainfo in "${metainfos[@]}"; do
# Positive control: a validator that never fails would pass the check below.
sed '/<id>/d' "${metainfo}" >"${work}/noid.xml"
for tool in "${tools[@]}"; do
validate "${tool}" "${metainfo}" || {
echo "FAIL: ${metainfo##*/} does not validate under ${tool}" >&2
exit 1
}
if validate "${tool}" "${work}/noid.xml" >"${work}/noid.log" 2>&1; then
echo "FAIL: ${tool} validated a metainfo with no <id>; it proves nothing" >&2
exit 1
fi
done
echo "appstream metainfo: ${metainfo##*/} valid (${tools[*]})"
done

View File

@@ -0,0 +1,42 @@
#!/bin/bash
#
# A too-long FTP command must be refused, never truncated or aborted (#1019).
set -euo pipefail
fail() {
echo "FAIL: $*" >&2
exit 1
}
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_ftpcmdlen.XXXXXX") || exit 1
trap 'set +e; rm -rf "${tmpdir}"' EXIT
trap 'exit 1' HUP INT QUIT TERM
out=$(httrack -O "${tmpdir}/st" "-#test=ftp-cmdlen" run 2>&1) ||
fail "self-test exited non-zero: ${out}"
grep -q "ftp-cmdlen self-test OK" <<<"${out}" || fail "unexpected output: ${out}"
# The reachable half: the copy into _adr[256] aborted on a long host.
rep() { awk -v n="$1" 'BEGIN { while (i++ < n) printf "a" }'; }
probe_host() {
rm -rf "${tmpdir}/host"
httrack "ftp://$1/f.txt" -O "${tmpdir}/host" -q >/dev/null 2>&1 ||
fail "a ${#1}-byte FTP host name crashed the engine"
cat "${tmpdir}/host/hts-log.txt"
}
for n in 256 257 300; do
log=$(probe_host "$(rep "${n}")")
grep -q "Host name too long" <<<"${log}" ||
fail "a ${n}-byte host was not rejected: ${log}"
done
# 255 still fits _adr[256]; 127.0.0.1:1 is the outsider a widened gate swallows.
log=$(probe_host "$(rep 255)")
grep -q "Unable to get server's address" <<<"${log}" ||
fail "a 255-byte host did not reach the resolver: ${log}"
log=$(probe_host "127.0.0.1:1")
grep -q "Unable to connect to the server" <<<"${log}" ||
fail "a normal host did not reach the connect: ${log}"

99
tests/225_install-manifest.test Executable file
View File

@@ -0,0 +1,99 @@
#!/bin/bash
#
# Distro packages (Fedora %files, debian/*.files) name every installed file, so a
# file appearing or vanishing here breaks their build and not ours; it broke
# Fedora's twice in three weeks.
set -euo pipefail
: "${abs_top_builddir:?not run under make check}"
manifest="${abs_top_srcdir:?not run under make check}/tests/install-manifest.txt"
fail() {
echo "FAIL: $*" >&2
exit 1
}
# macOS names the library .dylib, and Windows has no distro packaging to protect.
test "$(uname -s)" = Linux || {
echo "install layout is pinned on Linux only; skipping" >&2
exit 77
}
test -r "${manifest}" || fail "no ${manifest}"
# The docs are dropped below, so a docdir holding datadir would drop the pinned
# data files with them and leave nothing to compare.
docdir=${CONFIGURED_DOCDIR:?not run under make check}
case "${CONFIGURED_DATADIR:?not run under make check}" in
"${docdir%/}" | "${docdir%/}"/*)
echo "docdir contains datadir; skipping" >&2
exit 77
;;
esac
work=$(mktemp -d "${TMPDIR:-/tmp}/manifest.XXXXXX") || {
echo "no tmpdir" >&2
exit 1
}
trap 'set +e; rm -rf "${work}"' EXIT
stage=${work}/stage
env -u MAKEFLAGS -u MAKELEVEL "${MAKE:-make}" -C "${abs_top_builddir}" \
install DESTDIR="${stage}" >"${work}/install.log" 2>&1 || {
cat "${work}/install.log" >&2
fail "make install DESTDIR=${stage}"
}
# Fold the configured directories back to tokens so the pin survives any --prefix
# or --libdir. Longest path first, so a nested directory beats the one it sits in
# whatever the layout puts where.
: >"${work}/dirs"
for var in DOCDIR HTMLDIR MANDIR INCLUDEDIR LIBDIR BINDIR DATADIR; do
name=CONFIGURED_${var}
path=${!name:-}
test -n "${path}" || fail "${name} not set"
path=${path%/}
printf '%s\t%s\t%s\n' "${#path}" "${path}" "${var}" >>"${work}/dirs"
done
LC_ALL=C sort -rn "${work}/dirs" -o "${work}/dirs"
paths=() tokens=()
while IFS=$'\t' read -r _ path var; do
paths+=("${path}")
tokens+=("${var}")
done <"${work}/dirs"
# Matched literally: a directory holding a regex metacharacter would silently
# fold nothing if this were a pattern.
fold() {
local line=$1 i
for i in "${!paths[@]}"; do
case ${line} in
"${paths[i]}"/*)
printf '@%s@/%s\n' "${tokens[i]}" "${line#"${paths[i]}"/}"
return
;;
esac
done
printf '%s\n' "${line}"
}
# Dropped: the docs, since where they sit under docdir is a --htmldir choice each
# distro makes its own way and no package names them one by one; the .a/.la
# libtool bookkeeping no distro ships; and the library revision, which moves
# every release. The soname digit stays: a bump renames Debian's package.
(cd "${stage}" && find . \( -type f -o -type l \) | sed 's|^\.||') |
while IFS= read -r line; do fold "${line}"; done |
sed -e '/^@DOCDIR@\//d' -e '/^@HTMLDIR@\//d' \
-e '/\.la$/d' -e '/\.a$/d' \
-e 's|\(\.so\.[0-9][0-9]*\)\.[0-9][0-9]*\.[0-9][0-9]*$|\1.@revision@|' |
LC_ALL=C sort >"${work}/actual"
# An install that staged nothing matches nothing; say that, not a 300-line diff.
grep -qx '@BINDIR@/httrack' "${work}/actual" || fail "nothing staged under ${stage}"
LC_ALL=C sort "${manifest}" >"${work}/expected"
diff -u "${work}/expected" "${work}/actual" ||
fail "the installed file set changed; update tests/install-manifest.txt and say so in the release notes -- Fedora's %files and debian/*.files name these files"
echo "install manifest: $(wc -l <"${work}/actual") files"

View File

@@ -0,0 +1,46 @@
#!/bin/bash
#
# Issue #938: downscaling the vector master to 16 px antialiased every stem to
# grey. The small raster is drawn on the pixel grid, so it must stay flat.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
srcdir="${abs_top_srcdir:?not run under make check}"
div="${srcdir}/html/server/div"
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
fail() {
echo "FAIL: $*" >&2
exit 1
}
colors() {
"${python}" "${testdir}/png-colors.py" "${div}/$1/httrack.png" ||
fail "$1: cannot read the icon"
}
small=$(colors 16x16)
n=$(wc -l <<<"${small}")
test "${n}" -le 4 || fail "16x16 icon has ${n} colours, the stems are antialiased"
# Both flats present: an all-field tile and an all-ink one are flat too.
for flat in '#9999cc' '#040404'; do
grep -q " ${flat}\$" <<<"${small}" || fail "16x16 icon has no ${flat} pixel"
done
# A hairline and a near-solid block both satisfy every count above.
ink=$(sed -n "s/ #040404\$//p" <<<"${small}")
if test "${ink}" -lt 26 || test "${ink}" -gt 128; then
fail "16x16 icon inks ${ink} of 256 pixels"
fi
# Control: a counter blind to antialiasing would pass every assertion above.
big=$(wc -l <<<"$(colors 48x48)")
test "${big}" -gt 16 || fail "48x48 icon has ${big} colours, the counter is blind"
echo "16x16 icon: ${n} colours, ${ink}/256 ink; 48x48: ${big} colours"

View File

@@ -8,14 +8,15 @@ EXTRA_DIST = $(TESTS) renamefail.c threadattrfail.c nobacktrace.c altstackprobe.
proxy-https-server.py socks5-server.py proxy-connect-server.py \
proxytestlib.py tls-stall-server.py warc-validate.py wacz-validate.py \
header-injection-server.py header-injection-check.py \
pty-resize.py test-timeout.sh \
pty-resize.py test-timeout.sh png-colors.py \
local-crawl.sh local-server.py ftp-server.py testlib.sh \
ci-windows-suite.sh \
server.crt server.key \
server-root/simple/basic.html server-root/simple/link.html \
server-root/stripquery/index.html server-root/stripquery/a.html \
server-root/fraglink/index.html server-root/fraglink/target.html \
fixtures/cache-golden/hts-cache/new.zip
fixtures/cache-golden/hts-cache/new.zip \
install-manifest.txt
TESTS_ENVIRONMENT =
TESTS_ENVIRONMENT += PATH=$(top_builddir)/src$(PATH_SEPARATOR)$$PATH
@@ -34,6 +35,10 @@ TESTS_ENVIRONMENT += abs_top_builddir=$(abs_top_builddir)
TESTS_ENVIRONMENT += CONFIGURED_DATADIR=$(datadir)
TESTS_ENVIRONMENT += CONFIGURED_LIBDIR=$(libdir)
TESTS_ENVIRONMENT += CONFIGURED_BINDIR=$(bindir)
# 225_install-manifest.test folds these back out of the staged paths.
TESTS_ENVIRONMENT += CONFIGURED_MANDIR=$(mandir)
TESTS_ENVIRONMENT += CONFIGURED_DOCDIR=$(docdir)
TESTS_ENVIRONMENT += CONFIGURED_HTMLDIR=$(htmldir)
TESTS_ENVIRONMENT += ORIGIN_RPATH=$(ORIGIN_RPATH)
# 222_pkgconfig-consumer.test asserts the staged .pc against these, and both
# sides of the gate deciding whether it hands a consumer an rpath.

View File

@@ -14,6 +14,11 @@
#define _GNU_SOURCE
/* The largefile redirect renames this file's mmap() to mmap64, colliding with
its own mmap64(). _TIME_BITS rides on _FILE_OFFSET_BITS, so it goes too. */
#undef _FILE_OFFSET_BITS
#undef _TIME_BITS
#include <dlfcn.h>
#include <fcntl.h>
#include <signal.h>
@@ -187,6 +192,19 @@ static void *trace_mapped(void *sp) {
return sp;
}
/* The fallback when dlsym() failed, so it has to be right: 32-bit arches have
no SYS_mmap and i386's wants an argument block, so prefer SYS_mmap2, whose
offset is in 4096-byte units whatever the page size. */
static void *raw_mmap(void *addr, size_t len, int prot, int flags, int fd,
off_t off) {
#ifdef SYS_mmap2
return (void *) syscall(SYS_mmap2, addr, len, prot, flags, fd,
(unsigned long) (off / 4096));
#else
return (void *) syscall(SYS_mmap, addr, len, prot, flags, fd, off);
#endif
}
SHIM_EXPORT void *mmap(void *addr, size_t len, int prot, int flags, int fd,
off_t off);
@@ -194,10 +212,9 @@ SHIM_EXPORT void *mmap(void *addr, size_t len, int prot, int flags, int fd,
between, so the thread's last mapping is the whole of the check. */
SHIM_EXPORT void *mmap(void *addr, size_t len, int prot, int flags, int fd,
off_t off) {
return trace_mapped(
real_mmap != NULL
? real_mmap(addr, len, prot, flags, fd, off)
: (void *) syscall(SYS_mmap, addr, len, prot, flags, fd, off));
return trace_mapped(real_mmap != NULL
? real_mmap(addr, len, prot, flags, fd, off)
: raw_mmap(addr, len, prot, flags, fd, off));
}
/* glibc only, and the name that matters: _FILE_OFFSET_BITS=64 redirects the

123
tests/install-manifest.txt Normal file
View File

@@ -0,0 +1,123 @@
@BINDIR@/htsserver
@BINDIR@/httrack
@BINDIR@/proxytrack
@BINDIR@/webhttrack
@DATADIR@/applications/WebHTTrack-Websites.desktop
@DATADIR@/applications/WebHTTrack.desktop
@DATADIR@/httrack/html
@DATADIR@/httrack/lang.def
@DATADIR@/httrack/lang.indexes
@DATADIR@/httrack/lang/Bulgarian.txt
@DATADIR@/httrack/lang/Castellano.txt
@DATADIR@/httrack/lang/Cesky.txt
@DATADIR@/httrack/lang/Chinese-BIG5.txt
@DATADIR@/httrack/lang/Chinese-Simplified.txt
@DATADIR@/httrack/lang/Croatian.txt
@DATADIR@/httrack/lang/Dansk.txt
@DATADIR@/httrack/lang/Deutsch.txt
@DATADIR@/httrack/lang/Eesti.txt
@DATADIR@/httrack/lang/English.txt
@DATADIR@/httrack/lang/Finnish.txt
@DATADIR@/httrack/lang/Francais.txt
@DATADIR@/httrack/lang/Greek.txt
@DATADIR@/httrack/lang/Italiano.txt
@DATADIR@/httrack/lang/Japanese.txt
@DATADIR@/httrack/lang/Macedonian.txt
@DATADIR@/httrack/lang/Magyar.txt
@DATADIR@/httrack/lang/Nederlands.txt
@DATADIR@/httrack/lang/Norsk.txt
@DATADIR@/httrack/lang/Polski.txt
@DATADIR@/httrack/lang/Portugues-Brasil.txt
@DATADIR@/httrack/lang/Portugues.txt
@DATADIR@/httrack/lang/Romanian.txt
@DATADIR@/httrack/lang/Russian.txt
@DATADIR@/httrack/lang/Slovak.txt
@DATADIR@/httrack/lang/Slovenian.txt
@DATADIR@/httrack/lang/Svenska.txt
@DATADIR@/httrack/lang/Turkish.txt
@DATADIR@/httrack/lang/Ukrainian.txt
@DATADIR@/httrack/lang/Uzbek.txt
@DATADIR@/httrack/libtest/callbacks-example-baselinks.c
@DATADIR@/httrack/libtest/callbacks-example-changecontent.c
@DATADIR@/httrack/libtest/callbacks-example-contentfilter.c
@DATADIR@/httrack/libtest/callbacks-example-displayheader.c
@DATADIR@/httrack/libtest/callbacks-example-filename.c
@DATADIR@/httrack/libtest/callbacks-example-filename2.c
@DATADIR@/httrack/libtest/callbacks-example-filenameiisbug.c
@DATADIR@/httrack/libtest/callbacks-example-listlinks.c
@DATADIR@/httrack/libtest/callbacks-example-log.c
@DATADIR@/httrack/libtest/callbacks-example-simple.c
@DATADIR@/httrack/libtest/example-main.c
@DATADIR@/httrack/libtest/example-main.h
@DATADIR@/httrack/libtest/readme.txt
@DATADIR@/httrack/templates/index-body.html
@DATADIR@/httrack/templates/index-footer.html
@DATADIR@/httrack/templates/index-header.html
@DATADIR@/httrack/templates/topindex-body.html
@DATADIR@/httrack/templates/topindex-bodycat.html
@DATADIR@/httrack/templates/topindex-footer.html
@DATADIR@/httrack/templates/topindex-header.html
@DATADIR@/icons/hicolor/128x128/apps/httrack.png
@DATADIR@/icons/hicolor/16x16/apps/httrack.png
@DATADIR@/icons/hicolor/256x256/apps/httrack.png
@DATADIR@/icons/hicolor/32x32/apps/httrack.png
@DATADIR@/icons/hicolor/48x48/apps/httrack.png
@DATADIR@/icons/hicolor/64x64/apps/httrack.png
@DATADIR@/icons/hicolor/scalable/apps/httrack.svg
@DATADIR@/metainfo/com.httrack.WebHTTrack.metainfo.xml
@DATADIR@/pixmaps/httrack.xpm
@DATADIR@/pixmaps/httrack16x16.xpm
@DATADIR@/pixmaps/httrack32x32.xpm
@DATADIR@/pixmaps/httrack48x48.xpm
@INCLUDEDIR@/httrack/config.h
@INCLUDEDIR@/httrack/htsarrays.h
@INCLUDEDIR@/httrack/htsbasenet.h
@INCLUDEDIR@/httrack/htsbauth.h
@INCLUDEDIR@/httrack/htsconfig.h
@INCLUDEDIR@/httrack/htsdefines.h
@INCLUDEDIR@/httrack/htsglobal.h
@INCLUDEDIR@/httrack/htsmodules.h
@INCLUDEDIR@/httrack/htsnet.h
@INCLUDEDIR@/httrack/htsopt.h
@INCLUDEDIR@/httrack/htssafe.h
@INCLUDEDIR@/httrack/htsstrings.h
@INCLUDEDIR@/httrack/htswrap.h
@INCLUDEDIR@/httrack/httrack-library.h
@LIBDIR@/httrack/libbaselinks.so
@LIBDIR@/httrack/libbaselinks.so.1
@LIBDIR@/httrack/libbaselinks.so.1.@revision@
@LIBDIR@/httrack/libchangecontent.so
@LIBDIR@/httrack/libchangecontent.so.1
@LIBDIR@/httrack/libchangecontent.so.1.@revision@
@LIBDIR@/httrack/libcontentfilter.so
@LIBDIR@/httrack/libcontentfilter.so.1
@LIBDIR@/httrack/libcontentfilter.so.1.@revision@
@LIBDIR@/httrack/libdisplayheader.so
@LIBDIR@/httrack/libdisplayheader.so.1
@LIBDIR@/httrack/libdisplayheader.so.1.@revision@
@LIBDIR@/httrack/libfilename.so
@LIBDIR@/httrack/libfilename.so.1
@LIBDIR@/httrack/libfilename.so.1.@revision@
@LIBDIR@/httrack/libfilename2.so
@LIBDIR@/httrack/libfilename2.so.1
@LIBDIR@/httrack/libfilename2.so.1.@revision@
@LIBDIR@/httrack/libfilenameiisbug.so
@LIBDIR@/httrack/libfilenameiisbug.so.1
@LIBDIR@/httrack/libfilenameiisbug.so.1.@revision@
@LIBDIR@/httrack/liblistlinks.so
@LIBDIR@/httrack/liblistlinks.so.1
@LIBDIR@/httrack/liblistlinks.so.1.@revision@
@LIBDIR@/httrack/liblog.so
@LIBDIR@/httrack/liblog.so.1
@LIBDIR@/httrack/liblog.so.1.@revision@
@LIBDIR@/httrack/libsimple.so
@LIBDIR@/httrack/libsimple.so.1
@LIBDIR@/httrack/libsimple.so.1.@revision@
@LIBDIR@/libhttrack.so
@LIBDIR@/libhttrack.so.3
@LIBDIR@/libhttrack.so.3.@revision@
@LIBDIR@/pkgconfig/libhttrack.pc
@MANDIR@/man1/htsserver.1
@MANDIR@/man1/httrack.1
@MANDIR@/man1/proxytrack.1
@MANDIR@/man1/webhttrack.1

107
tests/png-colors.py Executable file
View File

@@ -0,0 +1,107 @@
#!/usr/bin/env python3
"""Print a PNG's distinct colours as "count #rrggbb", commonest first.
Usage: png-colors.py FILE
Stdlib only: the suite cannot assume Pillow or ImageMagick on a build host.
"""
import struct
import sys
import zlib
CHANNELS = {0: 1, 2: 3, 3: 1, 4: 2, 6: 4}
def chunks(blob):
off = 8
while off < len(blob):
(n,) = struct.unpack(">I", blob[off : off + 4])
yield blob[off + 4 : off + 8], blob[off + 8 : off + 8 + n]
off += 12 + n
def unfilter(raw, width, height, bpp, stride):
"""Undo the per-scanline filters; returns the concatenated raw scanlines."""
out, prev, off = bytearray(), bytearray(stride), 0
for _ in range(height):
ftype, line = raw[off], bytearray(raw[off + 1 : off + 1 + stride])
off += 1 + stride
for i in range(stride):
a = line[i - bpp] if i >= bpp else 0
b = prev[i]
c = prev[i - bpp] if i >= bpp else 0
if ftype == 1:
line[i] = (line[i] + a) & 0xFF
elif ftype == 2:
line[i] = (line[i] + b) & 0xFF
elif ftype == 3:
line[i] = (line[i] + (a + b) // 2) & 0xFF
elif ftype == 4:
p = a + b - c
pa, pb, pc = abs(p - a), abs(p - b), abs(p - c)
line[i] = (
line[i] + (a if pa <= pb and pa <= pc else b if pb <= pc else c)
) & 0xFF
elif ftype != 0:
raise SystemExit("unknown filter %d" % ftype)
out += line
prev = line
return out
def samples(row, depth, count):
"""The first `count` samples of one unfiltered scanline."""
if depth == 8:
return list(row[:count])
if depth == 16:
return [row[i * 2] << 8 | row[i * 2 + 1] for i in range(count)]
per, mask = 8 // depth, (1 << depth) - 1
return [(row[i // per] >> (8 - depth * (i % per + 1))) & mask for i in range(count)]
def colors(path):
blob = open(path, "rb").read()
if blob[:8] != b"\x89PNG\r\n\x1a\n":
raise SystemExit("%s: not a PNG" % path)
plte, idat = b"", b""
for kind, data in chunks(blob):
if kind == b"IHDR":
width, height, depth, ctype, _, _, interlace = struct.unpack(
">IIBBBBB", data
)
elif kind == b"PLTE":
plte = data
elif kind == b"IDAT":
idat += data
if interlace:
raise SystemExit("%s: interlaced" % path)
nchan = CHANNELS[ctype]
stride = (width * nchan * depth + 7) // 8
bpp = max(1, nchan * depth // 8)
raw = unfilter(zlib.decompress(idat), width, height, bpp, stride)
tally = {}
for y in range(height):
row = raw[y * stride : (y + 1) * stride]
vals = samples(row, depth, width * nchan)
for x in range(width):
px = tuple(vals[x * nchan : (x + 1) * nchan])
if ctype == 3:
px = tuple(plte[px[0] * 3 : px[0] * 3 + 3])
elif ctype in (0, 4):
px = (px[0], px[0], px[0])
tally[px[:3]] = tally.get(px[:3], 0) + 1
return tally
def main():
if len(sys.argv) != 2:
raise SystemExit(__doc__)
tally = colors(sys.argv[1])
for px, n in sorted(tally.items(), key=lambda kv: (-kv[1], kv[0])):
print("%d #%02x%02x%02x" % ((n,) + px))
if __name__ == "__main__":
main()

View File

@@ -176,6 +176,70 @@ kill_tree() {
ENGINE_EXE_RE='^(lt-)?(httrack|proxytrack|htsserver|webhttrack)([.]exe)?$'
FIXTURE_SERVER_RE='^(local-server|proxy-https-server|proxy-connect-server|socks5-server|tls-stall-server)[.]py$'
# awk prologue for the matchers below: under qemu-user the kernel reports the
# binfmt interpreter as the command, so the name we match on lands one column
# right (Debian's hppa buildd emulates).
# shellcheck disable=SC2016 # awk fields, not shell expansions
AWK_PROC_NAMES='
function basen(s) { sub(/.*[\/\\]/, "", s); return s }
function qemushift( n) {
n = basen($6)
# qemu-img and friends take an image, not a program, and this list is fed to
# kill: shifting past them would read a disk path as the process name.
if (n ~ /^qemu-(img|nbd|io|ga|edid|keymap)$/) return 0
return n ~ /^qemu-[[:alnum:]_]+(-static)?$|^[[:alnum:]_]+-binfmt(-[[:upper:]]+)?$/ ? 1 : 0
}'
# Every process as "PID PPID PGID ELAPSED S COMMAND", header first. A Fedora
# build root has no procps, and an empty list reads as "nothing running" (#1021).
ps_snapshot() {
local snap
# POSIX keywords, so the ps route holds on macOS too; a ps that lists nothing
# (hidepid, a locked-down container) is no better than an absent one.
if snap=$(ps -A -o pid,ppid,pgid,etime,state,args 2>/dev/null |
awk 'NR > 1 { rows++ } { print } END { exit rows ? 0 : 1 }'); then
printf '%s\n' "$snap"
return 0
fi
proc_snapshot && return 0
# Every consumer drops line 1, so the notice rides in the header's place.
printf 'no process list: this host has neither ps nor a readable /proc\n'
return 1
}
# The same six columns out of /proc, in bash alone; elapsed as plain seconds.
proc_snapshot() {
local d stat rest arg args comm hz uptime
local -a f
local ws # spelled out of line: bash 3.2 fails to parse $'..' inside ${v//p/r}
ws=$' \t\n\v\f\r'
test -r /proc/self/stat || return 1
hz=$(getconf CLK_TCK 2>/dev/null) || hz=
# A zero or non-numeric tick would abort the shell in the division below.
case "$hz" in '' | 0 | *[!0-9]*) hz=100 ;; esac
read -r uptime _ </proc/uptime 2>/dev/null || return 1
printf 'PID PPID PGID ELAPSED S COMMAND\n'
for d in /proc/[0-9]*; do
# Braced: a failed open reports to the caller's stderr, not the redirect.
{ read -r stat <"$d/stat"; } 2>/dev/null || continue
# comm may hold spaces and parens; every field past it is numeric.
rest=${stat##*') '}
comm=${stat#*(}
comm=${comm%)*}
read -ra f <<<"$rest" || continue
test "${#f[@]}" -ge 20 || continue # short read: the process is going away
args=
# Whitespace inside an argv would split the row or shift the columns the
# consumers match on, which ps avoids by mapping those bytes away.
{ while IFS= read -r -d '' arg; do
args="${args:+$args }${arg//["$ws"]/ }"
done <"$d/cmdline"; } 2>/dev/null || true
printf '%s %s %s %s %s %s\n' "${d#/proc/}" "${f[1]}" "${f[2]}" \
"$(((${uptime%%.*} * hz - f[19]) / hz))" \
"${f[0]}" "${args:-[$comm]}"
done
}
# List processes a hung test may have left running, one per line. $1 is the test's
# process group; $2 selects "group" (that group's members, whatever their name),
# "others" (engine and fixture processes outside it, which under "make check -j"
@@ -190,15 +254,16 @@ list_stray_processes() {
test "$mode" != others || return 0
tasklist 2>/dev/null | grep -Ei 'httrack|proxytrack|htsserver|python' || true
else
# `args` and `state` are POSIX ps keywords, so this holds on macOS too.
# Fields 6 and 7 are the command and its first argument (the interpreter
# and its script, for the Python fixtures).
ps -A -o pid,ppid,pgid,etime,state,args 2>/dev/null |
awk -v pg="$pgid" -v mode="$mode" -v eng="$ENGINE_EXE_RE" -v srv="$FIXTURE_SERVER_RE" '
ps_snapshot |
awk -v pg="$pgid" -v mode="$mode" -v eng="$ENGINE_EXE_RE" -v srv="$FIXTURE_SERVER_RE" \
"$AWK_PROC_NAMES"'
NR == 1 { print; next }
{ ingroup = (pg > 0 && $3 == pg)
c = $6; sub(/.*[\/\\]/, "", c)
s = $7; sub(/.*[\/\\]/, "", s)
q = qemushift()
c = basen($(6 + q))
s = basen($(7 + q))
named = (c ~ eng || s ~ srv)
if (mode == "group" ? ingroup : \
mode == "named" ? named : (named && !ingroup)) print }' || true
@@ -239,9 +304,9 @@ reap_leftover_processes() {
list_engine_pids() {
local pgid=${1:-0}
test "$pgid" -gt 0 2>/dev/null || return 0
ps -A -o pid,pgid,args 2>/dev/null |
awk -v pg="$pgid" -v eng="$ENGINE_EXE_RE" \
'NR > 1 && $2 == pg { c = $3; sub(/.*[\/\\]/, "", c); if (c ~ eng) print $1 }'
ps_snapshot |
awk -v pg="$pgid" -v eng="$ENGINE_EXE_RE" "$AWK_PROC_NAMES"'
NR > 1 && $3 == pg { if (basen($(6 + qemushift())) ~ eng) print $1 }'
}
# Ask the wedged test's engine processes for a stack. What is obtainable differs

View File

@@ -259,3 +259,6 @@ TESTS += 216_engine-ftp-ctrlchars.test
TESTS += 221_local-ftp-ctrlchars.test
TESTS += 218_crash-nopie-frames.test
TESTS += 222_pkgconfig-consumer.test
TESTS += 228_icon-small-flat.test
TESTS += 224_engine-ftp-cmdlen.test
TESTS += 225_install-manifest.test