Compare commits

..

2 Commits

Author SHA1 Message Date
Xavier Roche
8e2e4e7d34 Tighten the LLint typedef comments
Two one-line blocks instead of a four-line header; drop the change-narration
("no per-platform ladder"). Keep the load-bearing note that LLintP carries its
own '%'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-11 15:01:17 +02:00
Xavier Roche
42c5e71f36 Type LLint/TStamp as int64_t, drop the per-platform boilerplate
The wide-integer typedefs picked long/long long/__int64 per platform and
carried a matching printf-format ladder plus the HTS_LONGLONG capability
macro. That is what let x32 pick a 32-bit long for a "64-bit" LLint (#524).

<stdint.h>/<inttypes.h> answer both questions directly: int64_t is exactly
64-bit signed with defined wrap, and "%" PRId64 is its conversion. LLint
stays signed because -1 is a size/range sentinel across the engine (a
uint64_t would silently break every "< 0" check). The names LLint, TStamp,
and the LLintP macro are kept, so the ~70 call sites are untouched.

HTS_LONGLONG is now dead (a 64-bit type is a hard C99 dependency here:
md5.h includes <stdint.h> and the installed httrack-library.h includes
<inttypes.h>), so its detection blocks and htslib.c's #ifdef around the
GiB/TiB/PiB formatting go too; those branches are always valid now.

No ABI change: int64_t is `long` on LP64, so httrackp/htsblk stay
byte-identical and the exported symbol set is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-11 14:46:27 +02:00
5 changed files with 12 additions and 53 deletions

View File

@@ -55,9 +55,7 @@ jobs:
run: make -j"$(nproc)"
- name: Test
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
run: make check
- name: Print the test log on failure
if: failure()
@@ -101,8 +99,7 @@ jobs:
sudo find /usr/bin /usr/local/bin -maxdepth 1 -name 'python3*' \
-exec mv {} {}.hidden \;
! command -v python3
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
make check
- name: Print the test log on failure
if: failure()
@@ -136,15 +133,7 @@ jobs:
run: make -j"$(sysctl -n hw.ncpu)"
- name: Test
# bigcrawl's sustained -c8 crawl drops fetches on macOS's loopback when
# it competes with other crawls, flaking its exact file count (the #527
# macOS drop). Run everything else in parallel, then bigcrawl alone (its
# serial-safe condition). Linux tolerates the full parallel run.
run: |
jobs=$(( $(sysctl -n hw.ncpu) * 2 )); [ "$jobs" -le 16 ] || jobs=16
rest=$(cd tests && ls *.test | grep -v '^36_local-bigcrawl\.test$' | tr '\n' ' ')
make check -j"$jobs" TESTS="$rest"
make check TESTS=36_local-bigcrawl.test
run: make check
- name: Print the test log on failure
if: failure()
@@ -182,9 +171,7 @@ jobs:
run: make -j"$(nproc)"
- name: Test
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
run: make check
- name: Print the test log on failure
if: failure()
@@ -239,9 +226,7 @@ jobs:
env:
ASAN_OPTIONS: detect_leaks=0:abort_on_error=1:halt_on_error=1:strict_string_checks=1:malloc_fill_byte=202:max_malloc_fill_size=2147483647:free_fill_byte=203:max_free_fill_size=2147483647
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
run: make check
- name: Print the test log on failure
if: failure()
@@ -287,8 +272,7 @@ jobs:
# 01_engine-* only; zlib-dependent self-tests are named 01_zlib-* and
# skipped here (uninstrumented libz floods MSan with false positives).
tests="$(cd tests && ls 01_engine-*.test | tr '\n' ' ')"
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs" TESTS="$tests"
make check TESTS="$tests"
- name: Print the test log on failure
if: failure()
@@ -363,9 +347,7 @@ jobs:
run: make -j"$(nproc)"
- name: Test
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
run: make check
- name: Print the test log on failure
if: failure()

View File

@@ -6,19 +6,8 @@ the operational checklist: toolchain, invariants, and how to ship a change.
## Build & test
- Fresh clone first: `git submodule update --init src/coucal`
- `./bootstrap` (regenerates `configure` via `autoreconf`; needs autoconf,
automake, libtool), then `bash configure && make -j"$(nproc)" && make check
-j"$(nproc)"`. Always pass `-j` to `make check`: the suite runs under
automake's parallel harness and each crawl test binds its own ephemeral-port
server, so `-j` never contends and a multi-minute serial run drops to
seconds. A new `.test` added to `$(TESTS)` is scheduled onto a free worker
automatically; only a test slower than the current longest raises the floor.
On a few-core Linux box, `-j` at 2x the core count is faster still: the tests
spend much of their wall time asleep (server trickles, httrack self-pacing),
so an idle core covers a sleeping one. CI uses `min(2*cores, 16)`. macOS runs
36_local-bigcrawl alone in a second pass: its sustained `-c8` crawl overloads
the macOS loopback when it competes with other crawls and flakes its exact
file count (Linux tolerates the full parallel run).
Or run `sh build.sh` to do bootstrap + configure + make in one shot.
automake, libtool), then `bash configure && make && make check`. Or run
`sh build.sh` to do bootstrap + configure + make in one shot.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every

View File

@@ -9,7 +9,7 @@ set -euo pipefail
: "${top_srcdir:=..}"
bash "$top_srcdir/tests/local-crawl.sh" --rerun \
--errors-content 4 --files 361 \
--errors 4 --files 361 \
--found 'big/p/95.html' \
--found 'big/a/d1/d2/d3/d4/d5/d6/d7/d8/deep.png' \
--found 'big/a/f2-2x.png' \

View File

@@ -8,6 +8,4 @@ set -euo pipefail
bash "$top_srcdir/tests/local-crawl.sh" --rerun \
--found 'errmask/keep.dat' \
--file-min-bytes 'errmask/keep.dat' 1024 \
--file-matches 'errmask/keep.dat' '^KEEP' \
--file-not-matches 'errmask/keep.dat' 'error 403' \
httrack 'BASEURL/errmask/index.html'

View File

@@ -14,13 +14,11 @@
# Usage:
# bash local-crawl.sh [--tls] [--root DIR] [--cookie NAME=VALUE ...] \
# [--rerun-args 'ARGS'] \
# --errors N --errors-content N --files N --found PATH ... --directory PATH ... \
# --errors N --files N --found PATH ... --directory PATH ... \
# --log-found REGEX ... --log-not-found REGEX ... \
# --file-matches PATH REGEX ... --file-not-matches PATH REGEX ... \
# --file-min-bytes PATH N --max-mirror-bytes N \
# httrack BASEURL/some/path [httrack-args...]
# --errors counts every "Error:" log line; --errors-content drops transient
# network failures (codes -2..-6) that flake on busy loopback under -c8.
# --log-found/--log-not-found grep (ERE) the crawl's hts-log.txt.
# --max/--min-mirror-bytes bound the mirrored content bytes (host root).
# --file-matches/--file-not-matches grep (ERE) a mirrored file (PATH under the
@@ -133,7 +131,7 @@ while test "$pos" -lt "$nargs"; do
pos=$((pos + 1))
rerun_args="${args[$pos]}"
;;
--errors | --errors-content | --files)
--errors | --files)
audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
pos=$((pos + 1))
;;
@@ -336,14 +334,6 @@ while test "$i" -lt "${#audit[@]}"; do
assert_equals "checking errors" "${audit[$i]}" \
"$(grep -iEc "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt")"
;;
--errors-content)
i=$((i + 1))
total=$(grep -icE "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt")
# transient network failures (statuscode -2..-6) flake on busy loopback;
# the code parens are followed by " at link" or " after N retries at link"
transient=$(grep -cE '\(-[2-6]\) (at link|after )' "${out}/hts-log.txt" || true)
assert_equals "checking content errors" "${audit[$i]}" "$((total - transient))"
;;
--files)
i=$((i + 1))
nFiles=$(grep -E "^HTTrack Website Copier/[^ ]* mirror complete in " "${out}/hts-log.txt" |