mirror of
https://github.com/xroche/httrack.git
synced 2026-07-21 16:20:05 +03:00
Compare commits
4 Commits
docs-c-con
...
doc-correc
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
88a9b5e879 | ||
|
|
86bc02d8da | ||
|
|
e675d60301 | ||
|
|
1c8b93d9e7 |
8
.github/dependabot.yml
vendored
Normal file
8
.github/dependabot.yml
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
version: 2
|
||||
updates:
|
||||
# Bumps builtin-baseline in src/vcpkg.json so the pinned OpenSSL/zlib can't rot.
|
||||
# windows-build.yml validates each bump; the pin makes what ships reproducible.
|
||||
- package-ecosystem: vcpkg
|
||||
directory: /src
|
||||
schedule:
|
||||
interval: weekly
|
||||
14
.github/workflows/windows-build.yml
vendored
14
.github/workflows/windows-build.yml
vendored
@@ -44,6 +44,20 @@ jobs:
|
||||
shell: pwsh
|
||||
run: vcpkg integrate install
|
||||
|
||||
# The runner image's vcpkg checkout is pinned to some commit; our manifest's
|
||||
# builtin-baseline is usually newer, so `git show <baseline>:versions/...`
|
||||
# fails until that commit is local. Fetch exactly the pinned baseline (read
|
||||
# from the manifest, so Dependabot bumps need no workflow edit).
|
||||
- name: Fetch the pinned vcpkg baseline
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$root = if ($env:VCPKG_INSTALLATION_ROOT) { $env:VCPKG_INSTALLATION_ROOT } else { "C:\vcpkg" }
|
||||
$baseline = (Get-Content src/vcpkg.json -Raw | ConvertFrom-Json).'builtin-baseline'
|
||||
if (-not $baseline) { throw "no builtin-baseline in src/vcpkg.json" }
|
||||
git -C $root fetch --no-tags origin $baseline
|
||||
if ($LASTEXITCODE -ne 0) { throw "could not fetch vcpkg baseline $baseline" }
|
||||
|
||||
# httrack and webhttrack carry a ProjectReference to libhttrack, so building
|
||||
# them builds it first; proxytrack is standalone.
|
||||
- name: Build
|
||||
|
||||
30
AGENTS.md
30
AGENTS.md
@@ -19,6 +19,13 @@ the operational checklist: toolchain, invariants, and how to ship a change.
|
||||
(`request_queue_size`) so macOS/BSD don't drop connections under a parallel
|
||||
`-c16` bigcrawl the way Python's default backlog of 5 did.
|
||||
Or run `sh build.sh` to do bootstrap + configure + make in one shot.
|
||||
- A `tests/NN_*.test` runs only if listed in `tests/Makefile.am`'s `TESTS`; an
|
||||
unregistered file is silently skipped.
|
||||
- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
|
||||
not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
|
||||
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
|
||||
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
|
||||
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
|
||||
|
||||
## Hard invariants
|
||||
- **Generated autotools files are NOT in git.** `configure`, every
|
||||
@@ -31,15 +38,32 @@ the operational checklist: toolchain, invariants, and how to ship a change.
|
||||
- **Format only changed lines** with `git clang-format` (clang-format 19). Never
|
||||
reformat untouched code: the engine was formatted by an old tool and won't
|
||||
round-trip.
|
||||
- **Byte-safe edits.** Files with raw high bytes are ISO-8859-1 (French
|
||||
comments). Edit them byte-wise (`perl -0pi`, `sed`), not through a tool that
|
||||
re-encodes to UTF-8 and corrupts them.
|
||||
- **Byte-safe edits.** A few tracked files carry raw ISO-8859-1 high bytes
|
||||
(French comments): `src/htsconcat.c`, `lang/*.txt`, `html/contact.html`, and
|
||||
the `fuzz/corpus/*` vectors. Edit those byte-wise (`perl -0pi`, `sed`), not
|
||||
through a tool that re-encodes to UTF-8 and corrupts them. The rest of the tree
|
||||
is UTF-8 and safe to edit normally.
|
||||
|
||||
## Security (HTTrack parses hostile input off the network)
|
||||
- Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
|
||||
`untrusted < limit - controlled` — never `controlled + untrusted < limit`,
|
||||
which can wrap and pass.
|
||||
|
||||
## C conventions
|
||||
- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
|
||||
`malloct`/`calloct`/`realloct`/`freet`/`strdupt`, in test and selftest code
|
||||
too. `freet` NULLs its (lvalue) argument and tolerates NULL; `calloct(n, sz)`
|
||||
keeps calloc's arg order. Only exception: storing or calling a libc symbol
|
||||
itself (e.g. a resolver-backend function pointer).
|
||||
- **Exported API is `HTSEXT_API`.** Everything else is hidden by
|
||||
`-fvisibility=hidden` and free to change (check with `nm -D --defined-only
|
||||
libhttrack.so`). Touching an installed-header struct (see `DevIncludes_DATA` in
|
||||
`src/Makefile.am`) or an exported signature is an ABI break — flag and discuss,
|
||||
bump the soname, and prefer keeping the old entry point beside a new one.
|
||||
- **Windows ABI is free to break, POSIX is not.** The Windows DLL ships next to
|
||||
the exe with no soname contract, so a `_WIN32`-only ABI change needs no
|
||||
deprecation dance; POSIX/ELF keeps the flag-discuss-bump rules.
|
||||
|
||||
## Code & prose
|
||||
- Be terse. Comment the why, in English; translate French comments you touch.
|
||||
- Strip AI tells from prose (em-dash overuse, rule-of-three, filler, vague
|
||||
|
||||
@@ -173,7 +173,7 @@ There are also specific issues regarding this format:
|
||||
<ul>
|
||||
<li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
|
||||
<li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
|
||||
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory must be present <b>(not currently supported)</b></li>
|
||||
<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
|
||||
</ul>
|
||||
|
||||
<br />
|
||||
|
||||
@@ -263,7 +263,7 @@ clear language!
|
||||
|
||||
<li><a href="#QM8">Can HTTrack generates HP-UX or ISO9660 compatible files?</a><br></li>
|
||||
|
||||
<li><a href="#QM9">If there any SOCKS support?</a><br></li>
|
||||
<li><a href="#QM9">Is there any SOCKS support?</a><br></li>
|
||||
|
||||
<li><a href="#QM10">What's this hts-cache directory? Can I remove it?</a><br></li>
|
||||
|
||||
@@ -835,8 +835,8 @@ A: <em>Yes. Use user:password@your_proxy_name as your proxy name (example: <tt>s
|
||||
<br><br><a NAME="QM8">Q: <strong>Can HTTrack generates HP-UX or ISO9660 compatible files?</strong></a><br>
|
||||
A: <em>Yes. See the build options (-N, or see the WinHTTrack options)</em>
|
||||
|
||||
<br><br><a NAME="QM9">Q: <strong>If there any SOCKS support?</strong></a><br>
|
||||
A: <em>Not yet!</em>
|
||||
<br><br><a NAME="QM9">Q: <strong>Is there any SOCKS support?</strong></a><br>
|
||||
A: <em>Yes. HTTrack supports SOCKS5 and HTTP CONNECT proxies: give the proxy with a scheme prefix, e.g. <tt>-P socks5://host:port</tt> or <tt>-P connect://host:port</tt> (prefix <tt>user:pass@</tt> before the host for authenticated proxies).</em>
|
||||
|
||||
<br><br><a NAME="QM10">Q: <strong>What's this hts-cache directory? Can I remove it?</strong></a><br>
|
||||
A: <em>NO if you want to update the site, because this directory is used by HTTrack for this purpose.
|
||||
|
||||
@@ -189,7 +189,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
<br>
|
||||
Filters are analyzed by HTTrack from the first filter to the last one. The complete URL
|
||||
name is compared to filters defined by the user or added automatically by HTTrack. <br><br>
|
||||
A scan rule has an higher priority is it is declared later - hierarchy is important: <br>
|
||||
A scan rule has a higher priority if it is declared later - hierarchy is important: <br>
|
||||
|
||||
<br>
|
||||
<table BORDER="1" CELLPADDING="2">
|
||||
@@ -307,7 +307,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
<br>
|
||||
Filters are analyzed by HTTrack from the first filter to the last one. The sizes
|
||||
are compared against scan rules defined by the user.<br><br>
|
||||
A scan rule has an higher priority is it is declared later - hierarchy is important.<br>
|
||||
A scan rule has a higher priority if it is declared later - hierarchy is important.<br>
|
||||
|
||||
Note: scan rules based on size can be mixed with regular URL patterns<br>
|
||||
|
||||
@@ -361,7 +361,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
<br>
|
||||
Filters are analyzed by HTTrack from the first filter to the last one. The complete MIME
|
||||
type is compared against scan rules defined by the user.<br><br>
|
||||
A scan rule has an higher priority is it is declared later - hierarchy is important<br>
|
||||
A scan rule has a higher priority if it is declared later - hierarchy is important<br>
|
||||
|
||||
Note: scan rules based on MIME types can <b>NOT</b> be mixed with regular URL patterns or size patterns within the same rule, but you can use both of them in distinct ones<br>
|
||||
|
||||
@@ -387,12 +387,12 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
<tr>
|
||||
<td nowrap><tt>-mime:video/*</tt></td>
|
||||
<td>This will refuse all video links that were already scheduled for download
|
||||
(i.e. all other 'application/' link download will be aborted)</td>
|
||||
(i.e. the download will be aborted)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td nowrap><tt>-mime:video/* -mime:audio/*</tt></td>
|
||||
<td>This will refuse all audio and video links that were already scheduled for download
|
||||
(i.e. all other 'application/' link download will be aborted)</td>
|
||||
(i.e. the download will be aborted)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td nowrap><tt>-mime:*/* +mime:text/html +mime:image/*</tt></td>
|
||||
|
||||
@@ -286,7 +286,7 @@ module exit point
|
||||
the function name and prototype MUST match this prototype
|
||||
*/
|
||||
EXTERNAL_FUNCTION int hts_unplug(httrackp *opt) {
|
||||
fprintf(stder, "Module unplugged");
|
||||
fprintf(stderr, "Module unplugged");
|
||||
|
||||
return 1; /* success */
|
||||
}
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"$schema":
|
||||
"https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
|
||||
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
|
||||
"name": "libhttrack",
|
||||
"version-string": "3.49",
|
||||
"builtin-baseline": "cd61e1e26a038e82d6550a3ebbe0fbbfe7da78e3",
|
||||
"dependencies": [
|
||||
"brotli",
|
||||
"openssl",
|
||||
"zlib",
|
||||
"zstd"
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user