Compare commits

...

4 Commits

Author SHA1 Message Date
Xavier Roche
88a9b5e879 Correct stale facts in the HTML help pages
- faq.html: the SOCKS question answered "Not yet!" — SOCKS5 and HTTP
  CONNECT proxies have shipped (-P socks5://|connect://); rewrite the
  answer and fix the "If there"/"Is there" typo.
- cache.html: minizip emits a ZIP64 central directory automatically for
  archives over 4GiB or 65535 entries, so drop the "(not currently
  supported)" claim and keep the real limit (a single entry of 4GiB+).
- filters.html: two -mime:video/* example rows described "application/"
  instead of the video rule; fix them, and "priority is it is declared
  later" -> "if it is declared later".
- plug.html: compile-breaking fprintf(stder, ...) in the sample module.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-21 13:58:48 +02:00
dependabot[bot]
86bc02d8da Bump github.com/microsoft/vcpkg from master to 2026.06.24 in /src (#644)
Bumps [github.com/microsoft/vcpkg](https://github.com/microsoft/vcpkg) from master to 2026.06.24. This release includes the previously tagged commit.
- [Release notes](https://github.com/microsoft/vcpkg/releases)
- [Commits](4bca8fd865...cd61e1e26a)

---
updated-dependencies:
- dependency-name: github.com/microsoft/vcpkg
  dependency-version: 2026.06.24
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-21 08:25:34 +02:00
Xavier Roche
e675d60301 Pin vcpkg builtin-baseline so shipped OpenSSL is reproducible (#643)
src/vcpkg.json had no builtin-baseline, so vcpkg resolved OpenSSL/zlib/brotli/zstd in Classic mode from whatever ports tree the build runner happened to sit at: the crypto shipped next to libhttrack.dll was a property of the runner image, not of any commit. Pin the baseline (resolves OpenSSL 3.6.3, zlib 1.3.2#1, what the Windows build already ships) so it becomes deterministic, and add a Dependabot vcpkg entry so the baseline advances on a schedule instead of freezing into a stale, CVE-bearing OpenSSL. The Windows workflow now fetches the pinned baseline before building, read back from the manifest so Dependabot bumps need no workflow edit.

Closes #642

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 07:43:59 +02:00
Xavier Roche
1c8b93d9e7 Document httrack C conventions and test-harness gotchas in AGENTS.md (#641)
Fold repo-specific conventions from private notes into the tracked
guide: a C conventions section (htssafe.h *t allocators, HTSEXT_API as
the exported ABI surface, Windows-breakable/POSIX-stable ABI split),
the concrete Latin-1 file list behind the byte-safe-edits rule, and
three test gotchas (register NN_*.test in TESTS, installed-binary PATH
shadowing, set -e in new .test scripts).

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 19:37:11 +02:00
8 changed files with 62 additions and 16 deletions

8
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,8 @@
version: 2
updates:
# Bumps builtin-baseline in src/vcpkg.json so the pinned OpenSSL/zlib can't rot.
# windows-build.yml validates each bump; the pin makes what ships reproducible.
- package-ecosystem: vcpkg
directory: /src
schedule:
interval: weekly

View File

@@ -44,6 +44,20 @@ jobs:
shell: pwsh
run: vcpkg integrate install
# The runner image's vcpkg checkout is pinned to some commit; our manifest's
# builtin-baseline is usually newer, so `git show <baseline>:versions/...`
# fails until that commit is local. Fetch exactly the pinned baseline (read
# from the manifest, so Dependabot bumps need no workflow edit).
- name: Fetch the pinned vcpkg baseline
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
$root = if ($env:VCPKG_INSTALLATION_ROOT) { $env:VCPKG_INSTALLATION_ROOT } else { "C:\vcpkg" }
$baseline = (Get-Content src/vcpkg.json -Raw | ConvertFrom-Json).'builtin-baseline'
if (-not $baseline) { throw "no builtin-baseline in src/vcpkg.json" }
git -C $root fetch --no-tags origin $baseline
if ($LASTEXITCODE -ne 0) { throw "could not fetch vcpkg baseline $baseline" }
# httrack and webhttrack carry a ProjectReference to libhttrack, so building
# them builds it first; proxytrack is standalone.
- name: Build

View File

@@ -19,6 +19,13 @@ the operational checklist: toolchain, invariants, and how to ship a change.
(`request_queue_size`) so macOS/BSD don't drop connections under a parallel
`-c16` bigcrawl the way Python's default backlog of 5 did.
Or run `sh build.sh` to do bootstrap + configure + make in one shot.
- A `tests/NN_*.test` runs only if listed in `tests/Makefile.am`'s `TESTS`; an
unregistered file is silently skipped.
- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every
@@ -31,15 +38,32 @@ the operational checklist: toolchain, invariants, and how to ship a change.
- **Format only changed lines** with `git clang-format` (clang-format 19). Never
reformat untouched code: the engine was formatted by an old tool and won't
round-trip.
- **Byte-safe edits.** Files with raw high bytes are ISO-8859-1 (French
comments). Edit them byte-wise (`perl -0pi`, `sed`), not through a tool that
re-encodes to UTF-8 and corrupts them.
- **Byte-safe edits.** A few tracked files carry raw ISO-8859-1 high bytes
(French comments): `src/htsconcat.c`, `lang/*.txt`, `html/contact.html`, and
the `fuzz/corpus/*` vectors. Edit those byte-wise (`perl -0pi`, `sed`), not
through a tool that re-encodes to UTF-8 and corrupts them. The rest of the tree
is UTF-8 and safe to edit normally.
## Security (HTTrack parses hostile input off the network)
- Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
`untrusted < limit - controlled` — never `controlled + untrusted < limit`,
which can wrap and pass.
## C conventions
- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
`malloct`/`calloct`/`realloct`/`freet`/`strdupt`, in test and selftest code
too. `freet` NULLs its (lvalue) argument and tolerates NULL; `calloct(n, sz)`
keeps calloc's arg order. Only exception: storing or calling a libc symbol
itself (e.g. a resolver-backend function pointer).
- **Exported API is `HTSEXT_API`.** Everything else is hidden by
`-fvisibility=hidden` and free to change (check with `nm -D --defined-only
libhttrack.so`). Touching an installed-header struct (see `DevIncludes_DATA` in
`src/Makefile.am`) or an exported signature is an ABI break — flag and discuss,
bump the soname, and prefer keeping the old entry point beside a new one.
- **Windows ABI is free to break, POSIX is not.** The Windows DLL ships next to
the exe with no soname contract, so a `_WIN32`-only ABI change needs no
deprecation dance; POSIX/ELF keeps the flag-discuss-bump rules.
## Code & prose
- Be terse. Comment the why, in English; translate French comments you touch.
- Strip AI tells from prose (em-dash overuse, rule-of-three, filler, vague

View File

@@ -173,7 +173,7 @@ There are also specific issues regarding this format:
<ul>
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory must be present <b>(not currently supported)</b></li>
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
</ul>
<br />

View File

@@ -263,7 +263,7 @@ clear language!
<li><a href="#QM8">Can HTTrack generates HP-UX or ISO9660 compatible files?</a><br></li>
<li><a href="#QM9">If there any SOCKS support?</a><br></li>
<li><a href="#QM9">Is there any SOCKS support?</a><br></li>
<li><a href="#QM10">What's this hts-cache directory? Can I remove it?</a><br></li>
@@ -835,8 +835,8 @@ A: <em>Yes. Use user:password@your_proxy_name as your proxy name (example: <tt>s
<br><br><a NAME="QM8">Q: <strong>Can HTTrack generates HP-UX or ISO9660 compatible files?</strong></a><br>
A: <em>Yes. See the build options (-N, or see the WinHTTrack options)</em>
<br><br><a NAME="QM9">Q: <strong>If there any SOCKS support?</strong></a><br>
A: <em>Not yet!</em>
<br><br><a NAME="QM9">Q: <strong>Is there any SOCKS support?</strong></a><br>
A: <em>Yes. HTTrack supports SOCKS5 and HTTP CONNECT proxies: give the proxy with a scheme prefix, e.g. <tt>-P socks5://host:port</tt> or <tt>-P connect://host:port</tt> (prefix <tt>user:pass@</tt> before the host for authenticated proxies).</em>
<br><br><a NAME="QM10">Q: <strong>What's this hts-cache directory? Can I remove it?</strong></a><br>
A: <em>NO if you want to update the site, because this directory is used by HTTrack for this purpose.

View File

@@ -189,7 +189,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Filters are analyzed by HTTrack from the first filter to the last one. The complete URL
name is compared to filters defined by the user or added automatically by HTTrack. <br><br>
A scan rule has an higher priority is it is declared later - hierarchy is important: <br>
A scan rule has a higher priority if it is declared later - hierarchy is important: <br>
<br>
<table BORDER="1" CELLPADDING="2">
@@ -307,7 +307,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Filters are analyzed by HTTrack from the first filter to the last one. The sizes
are compared against scan rules defined by the user.<br><br>
A scan rule has an higher priority is it is declared later - hierarchy is important.<br>
A scan rule has a higher priority if it is declared later - hierarchy is important.<br>
Note: scan rules based on size can be mixed with regular URL patterns<br>
@@ -361,7 +361,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<br>
Filters are analyzed by HTTrack from the first filter to the last one. The complete MIME
type is compared against scan rules defined by the user.<br><br>
A scan rule has an higher priority is it is declared later - hierarchy is important<br>
A scan rule has a higher priority if it is declared later - hierarchy is important<br>
Note: scan rules based on MIME types can <b>NOT</b> be mixed with regular URL patterns or size patterns within the same rule, but you can use both of them in distinct ones<br>
@@ -387,12 +387,12 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
<tr>
<td nowrap><tt>-mime:video/*</tt></td>
<td>This will refuse all video links that were already scheduled for download
(i.e. all other 'application/' link download will be aborted)</td>
(i.e. the download will be aborted)</td>
</tr>
<tr>
<td nowrap><tt>-mime:video/* -mime:audio/*</tt></td>
<td>This will refuse all audio and video links that were already scheduled for download
(i.e. all other 'application/' link download will be aborted)</td>
(i.e. the download will be aborted)</td>
</tr>
<tr>
<td nowrap><tt>-mime:*/* +mime:text/html +mime:image/*</tt></td>

View File

@@ -286,7 +286,7 @@ module exit point
the function name and prototype MUST match this prototype
*/
EXTERNAL_FUNCTION int hts_unplug(httrackp *opt) {
fprintf(stder, "Module unplugged");
fprintf(stderr, "Module unplugged");
return 1; /* success */
}

View File

@@ -1,12 +1,12 @@
{
"$schema":
"https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"name": "libhttrack",
"version-string": "3.49",
"builtin-baseline": "cd61e1e26a038e82d6550a3ebbe0fbbfe7da78e3",
"dependencies": [
"brotli",
"openssl",
"zlib",
"zstd"
]
}
}