Range-check --warc-max-size and regenerate the man page

Replace the unchecked sscanf on the --warc-max-size argument with a
strtoll parse that rejects non-numeric, negative, and overflowing values,
leaving the default 0 (single archive) intact. The man page and its HTML
render were already regenerated for the option's help line in the feature
commit, so this only tightens the parse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/dependabot.yml

@@ -0,0 +1,8 @@
+version: 2
+updates:
+  # Bumps builtin-baseline in src/vcpkg.json so the pinned OpenSSL/zlib can't rot.
+  # windows-build.yml validates each bump; the pin makes what ships reproducible.
+  - package-ecosystem: vcpkg
+    directory: /src
+    schedule:
+      interval: weekly

.github/workflows/ci.yml

@@ -618,3 +618,31 @@ jobs:
               echo "Fix locally with: git clang-format --binary clang-format-19 $base"
               exit 1 ;;
           esac
+
+  man-page-sync:
+    name: man page / html in sync
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      # html/httrack.man.html is groff-rendered from man/httrack.1 and committed.
+      # Rendering needs the full groff html device, so CI can't regenerate it;
+      # instead require the two to move together: a PR that touches httrack.1
+      # must also touch the html, catching the "regenerated roff, forgot html".
+      - name: httrack.1 changes must include html/httrack.man.html
+        run: |
+          set -euo pipefail
+          git fetch --no-tags origin \
+            "+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
+          base="origin/${{ github.base_ref }}"
+          changed="$(git diff --name-only "$base"...HEAD)"
+          has() { printf '%s\n' "$changed" | grep -qx "$1"; }
+          if has man/httrack.1 && ! has html/httrack.man.html; then
+            echo "::error::man/httrack.1 changed but html/httrack.man.html did not."
+            echo "Regenerate it with: make -C man regen-man-html (needs the full groff package)."
+            exit 1
+          fi
+          echo "man/html sync OK."

.github/workflows/codeql.yml

@@ -55,6 +55,9 @@ jobs:
             query-filters:
               - exclude:
                   id: cpp/world-writable-file-creation
+              # Models auth-bypass-by-spoofing; httrack has no auth surface, its +/- crawl filter is a mirror boundary, not a security one.
+              - exclude:
+                  id: cpp/user-controlled-bypass
 
       # Manual build: CodeQL traces the compiler, so build exactly what ships.
       - name: Build

.github/workflows/windows-build.yml

@@ -44,6 +44,20 @@ jobs:
         shell: pwsh
         run: vcpkg integrate install
 
+      # The runner image's vcpkg checkout is pinned to some commit; our manifest's
+      # builtin-baseline is usually newer, so `git show <baseline>:versions/...`
+      # fails until that commit is local. Fetch exactly the pinned baseline (read
+      # from the manifest, so Dependabot bumps need no workflow edit).
+      - name: Fetch the pinned vcpkg baseline
+        shell: pwsh
+        run: |
+          $ErrorActionPreference = "Stop"
+          $root = if ($env:VCPKG_INSTALLATION_ROOT) { $env:VCPKG_INSTALLATION_ROOT } else { "C:\vcpkg" }
+          $baseline = (Get-Content src/vcpkg.json -Raw | ConvertFrom-Json).'builtin-baseline'
+          if (-not $baseline) { throw "no builtin-baseline in src/vcpkg.json" }
+          git -C $root fetch --no-tags origin $baseline
+          if ($LASTEXITCODE -ne 0) { throw "could not fetch vcpkg baseline $baseline" }
+
       # httrack and webhttrack carry a ProjectReference to libhttrack, so building
       # them builds it first; proxytrack is standalone.
       - name: Build
@@ -188,7 +202,8 @@ jobs:
 
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
-          expected_skips=" 48_local-crange-memresume.test 71_local-crange-repaircache.test"  # pending #581
+          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -19,6 +19,13 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   (`request_queue_size`) so macOS/BSD don't drop connections under a parallel
   `-c16` bigcrawl the way Python's default backlog of 5 did.
   Or run `sh build.sh` to do bootstrap + configure + make in one shot.
+- A `tests/NN_*.test` runs only if listed in `tests/Makefile.am`'s `TESTS`; an
+  unregistered file is silently skipped.
+- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
+  not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
+  check`, or `PATH="<bld>/src:$PATH"` for a manual run.
+- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
+  `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -31,15 +38,32 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - **Format only changed lines** with `git clang-format` (clang-format 19). Never
   reformat untouched code: the engine was formatted by an old tool and won't
   round-trip.
-- **Byte-safe edits.** Files with raw high bytes are ISO-8859-1 (French
-  comments). Edit them byte-wise (`perl -0pi`, `sed`), not through a tool that
-  re-encodes to UTF-8 and corrupts them.
+- **Byte-safe edits.** A few tracked files carry raw ISO-8859-1 high bytes
+  (French comments): `src/htsconcat.c`, `lang/*.txt`, `html/contact.html`, and
+  the `fuzz/corpus/*` vectors. Edit those byte-wise (`perl -0pi`, `sed`), not
+  through a tool that re-encodes to UTF-8 and corrupts them. The rest of the tree
+  is UTF-8 and safe to edit normally.
 
 ## Security (HTTrack parses hostile input off the network)
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
 
+## C conventions
+- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
+  `malloct`/`calloct`/`realloct`/`freet`/`strdupt`, in test and selftest code
+  too. `freet` NULLs its (lvalue) argument and tolerates NULL; `calloct(n, sz)`
+  keeps calloc's arg order. Only exception: storing or calling a libc symbol
+  itself (e.g. a resolver-backend function pointer).
+- **Exported API is `HTSEXT_API`.** Everything else is hidden by
+  `-fvisibility=hidden` and free to change (check with `nm -D --defined-only
+  libhttrack.so`). Touching an installed-header struct (see `DevIncludes_DATA` in
+  `src/Makefile.am`) or an exported signature is an ABI break — flag and discuss,
+  bump the soname, and prefer keeping the old entry point beside a new one.
+- **Windows ABI is free to break, POSIX is not.** The Windows DLL ships next to
+  the exe with no soname contract, so a `_WIN32`-only ABI change needs no
+  deprecation dance; POSIX/ELF keeps the flag-discuss-bump rules.
+
 ## Code & prose
 - Be terse. Comment the why, in English; translate French comments you touch.
 - Strip AI tells from prose (em-dash overuse, rule-of-three, filler, vague

html/abuse.html

@@ -217,7 +217,7 @@ school or
 shows. They might do that because they are connected through expensive modem connection,
 or because they would like to consult pages while travelling, or archive sites that may be
 removed
-one day, make some data mining, comiling information ("if only I could find this
+one day, make some data mining, compiling information ("if only I could find this
 website I saw one day..&quot;). <br>
 There are many good reasons to mirror websites, and this helps many good people.<br>
 As a webmaster, you might be interested to use such tools, too: test broken links, move a
@@ -229,7 +229,7 @@ test the webserver response and performances, index it..<br>
 Anyway, bandwidth abuse can be a problem. If your site is regularly &quot;clobbered&quot;
 by evil downloaders, you have <br>
 various solutions. You have radical solutions, and intermediate solutions. I strongly
-recomment not to use<br>
+recommend not to use<br>
 radical solutions, because of the previous remarks (good people often mirror websites).<br>
 <br>
 In general, for all solutions,<br>
@@ -244,7 +244,7 @@ or, to be extreme: if you unplug the wire, there will be no bandwidth abuse<br>
 Good: Will work with good people. Many good people just don't KNOW that they can slow down
 a network.<br>
 Bad: Will **only** work with good people<br>
-How to do: Obvious - place a note, a warning, an article, a draw, a poeme or whatever you
+How to do: Obvious - place a note, a warning, an article, a draw, a poem or whatever you
 want<br>
 <br>
 </li><li>Use &quot;robots.txt&quot; file<br>
@@ -266,7 +266,7 @@ Good: Efficient<br>
 Bad: Multiple users behind proxies will be slow down, not really easy to setup<br>
 How to do: Depends on webserver. Might be done with low-level IP rules (QoS)<br>
 <br>
-</li><li>Priorize small files, against large files<br>
+</li><li>Prioritize small files, against large files<br>
 Good: Efficient if large files are the cause of abuse<br>
 Bad: Not always efficient<br>
 How to do: Depends on the webserver<br>
@@ -283,7 +283,7 @@ How to do: Use routine QoS (fair queuing), or webserver options<br>
 <br>
 </li><li>Use technical tricks (like javascript) to hide URLs<br>
 Good: Efficient<br>
-Bad: The most efficient tricks will also cause your website to he heavy, and not
+Bad: The most efficient tricks will also cause your website to be heavy, and not
 user-friendly (and therefore less attractive, even for surfing users). Remember: clients
 or visitors might want to consult offline your website. Advanced users will also be still
 able to note the URLs and catch them. Will not work on non-javascript browsers. It will
@@ -335,7 +335,7 @@ Example: Use things like
 </li><li>Use technical tricks to temporarily ban IPs<br>
 Good: Efficient<br>
 Bad: Radical (your site will only be available online for all users), not easy to setup<br>
-How to to: Create fake links with &quot;killing&quot; targets<br>
+How to do: Create fake links with &quot;killing&quot; targets<br>
 Example: Use things like &lt;a href=&quot;killme.cgi&quot;&gt;&lt;nothing&gt;&lt;/a&gt;
 (again an example in php:)<br>
 <tt>
@@ -417,7 +417,7 @@ Example:<br>
 </li><li>Another one is to create images of emails<br>
 Good: Efficient, does not require javascript<br>
 Bad: There is still the problem of the link (mailto:), images are bigger than text, and it can cause problems for blind people (a good solution is use an ALT attribute with the email written like "smith at mycompany dot com")<br>
-How to do: Not so obvious of you do not want to create images by yourself<br>
+How to do: Not so obvious if you do not want to create images by yourself<br>
 Example: (php, Unix)<br>
 
 <tt>
@@ -491,7 +491,7 @@ echo <br>
 </li><li>You can also create temporary email aliases, each week, for all users<br>
 Good: Efficient, and you can give your real email in your reply-to address<br>
 Bad: Temporary emails<br>
-How to do: Not so hard todo<br>
+How to do: Not so hard to do<br>
 Example: (script &amp; php, Unix)<br>
 
 <tt>
@@ -579,7 +579,7 @@ And then, put the email address in your pages through:
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/addurl.html

@@ -144,7 +144,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/android.html

@@ -0,0 +1,223 @@
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+
+<head>
+	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+	<meta name="description" content="How to mirror a website with HTTrack on Android: install the app, create a project, enter the address, run the mirror, and browse the result on your device." />
+	<meta name="keywords" content="httrack, HTTrack, android, offline browser, web mirror utility, website mirroring, mobile, Google Play" />
+	<title>HTTrack on Android</title>
+
+	<style type="text/css">
+	<!--
+
+body {
+	margin: 0;  padding: 0;  margin-bottom: 15px;  margin-top: 8px;
+	background: #77b;
+}
+body, td {
+	font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
+	}
+
+#subTitle {
+	background: #000;  color: #fff;  padding: 4px;  font-weight: bold;
+	}
+
+#siteNavigation a, #siteNavigation .current {
+	font-weight: bold;  color: #448;
+	}
+#siteNavigation a:link    { text-decoration: none; }
+#siteNavigation a:visited { text-decoration: none; }
+
+#siteNavigation .current { background-color: #ccd; }
+
+#siteNavigation a:hover   { text-decoration: none;  background-color: #fff;  color: #000; }
+#siteNavigation a:active  { text-decoration: none;  background-color: #ccc; }
+
+
+a:link    { text-decoration: underline;  color: #00f; }
+a:visited { text-decoration: underline;  color: #000; }
+a:hover   { text-decoration: underline;  color: #c00; }
+a:active  { text-decoration: underline; }
+
+#pageContent {
+	clear: both;
+	border-bottom: 6px solid #000;
+	padding: 10px;  padding-top: 20px;
+	line-height: 1.65em;
+	background-image: url(images/bg_rings.gif);
+	background-repeat: no-repeat;
+	background-position: top right;
+	}
+
+#pageContent, #siteNavigation {
+	background-color: #ccd;
+	}
+
+
+.imgLeft  { float: left;   margin-right: 10px;  margin-bottom: 10px; }
+.imgRight { float: right;  margin-left: 10px;   margin-bottom: 10px; }
+
+hr { height: 1px;  color: #000;  background-color: #000;  margin-bottom: 15px; }
+
+h1 { margin: 0;  font-weight: bold;  font-size: 2em; }
+h2 { margin: 0;  font-weight: bold;  font-size: 1.6em; }
+h3 { margin: 0;  font-weight: bold;  font-size: 1.3em; }
+h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
+
+.blak { background-color: #000; }
+.hide { display: none; }
+.tableWidth { min-width: 400px; }
+
+.tblRegular       { border-collapse: collapse; }
+.tblRegular td    { padding: 6px;  background-image: url(fade.gif);  border: 2px solid #99c; }
+.tblHeaderColor, .tblHeaderColor td { background: #99c; }
+.tblNoBorder td   { border: 0; }
+
+
+// -->
+</style>
+
+</head>
+
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+	<tr>
+	<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
+	<tr>
+	<td id="subTitle">Open Source offline browser</td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+<tr class="blak">
+<td>
+	<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
+	<tr>
+	<td colspan="6">
+		<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
+		<tr>
+		<td id="pageContent">
+<!-- ==================== End prologue ==================== -->
+
+<h2 align="center"><em>HTTrack on Android</em></h2>
+
+<br>
+
+<p>HTTrack downloads a website to your device so you can read it offline. The
+Android app runs the same mirroring engine as the desktop version behind a touch
+interface. It needs Android 7.0 or later, and is available on
+<a href="https://play.google.com/store/apps/details?id=com.httrack.android">Google Play</a>.</p>
+
+<p>The steps below follow one mirror from start to finish.</p>
+
+<br>
+
+<ol>
+
+  <li><b>Grant storage access</b></li>
+  <br><small>On first launch the app asks for permission to store mirrors on your
+  device. The prompt reads <em>"Allow HTTrack Website Copier to access photos,
+  media, and files on your device?"</em>. Tap <b>ALLOW</b>: without it the app
+  cannot save the downloaded files.</small>
+  <br><br><center><img src="img/android_permission.png" width="320" alt="First-run storage permission dialog" border="0"></center>
+  <br><small>If you have used an older release, a second prompt offers to bring its
+  mirrors into the app. Tap <b>Import</b> to move them, or <b>Not now</b> to
+  skip.</small>
+  <br><br><center><img src="img/android_import.png" width="320" alt="Import mirrors from an older version" border="0"></center>
+<br><br>
+
+  <li><b>Start</b></li>
+  <br><small>The welcome screen shows the engine version at the bottom. Tap
+  <b>Next</b> to create a project, or <b>Browse sites</b> to open a mirror you
+  already made.</small>
+  <br><br><center><img src="img/android_startup.png" width="320" alt="Welcome screen" border="0"></center>
+<br><br>
+
+  <li><b>Name the project</b></li>
+  <br><small>Give the project a name, and optionally a category to group related
+  mirrors. <b>Base path</b> shows where the files will be written. Tap
+  <b>Next</b>.</small>
+  <br><br><center><img src="img/android_project.png" width="320" alt="Project name, category, and base path" border="0"></center>
+<br><br>
+
+  <li><b>Enter the address</b></li>
+  <br><small>Type the site address. If a project of the same name already exists,
+  pick <b>Continue interrupted download</b> or <b>Update existing download</b>.
+  Tap <b>Options</b> to adjust the crawl, or <b>Start</b> to begin.</small>
+  <br><br><center><img src="img/android_url.png" width="320" alt="Web address and download action" border="0"></center>
+<br><br>
+
+  <li><b>Options (optional)</b></li>
+  <br><small>The <b>Options</b> screen holds the crawl settings, grouped into
+  eleven tabs. It uses the same profile format as the desktop version, so a tab you
+  know from WinHTTrack behaves the same way here.</small>
+  <br><br><center><img src="img/android_options.png" width="320" alt="Options tab list" border="0"></center>
+  <br><small>The tabs are:</small>
+  <ul>
+    <li><small><b>Scan Rules</b>: wildcard filters for the addresses to keep or skip.</small></li>
+    <li><small><b>Limits</b>: caps on depth, size, time, speed, and number of connections.</small></li>
+    <li><small><b>Flow Control</b>: simultaneous connections, timeouts, and retries.</small></li>
+    <li><small><b>Links</b>: how far to follow links, and which extra files to fetch.</small></li>
+    <li><small><b>Build</b>: how the saved files and folders are named.</small></li>
+    <li><small><b>Browser ID</b>: the user-agent, language, and headers sent to servers.</small></li>
+    <li><small><b>Spider</b>: cookies, robots.txt handling, and request behavior.</small></li>
+    <li><small><b>Proxy address</b>: proxy host and port.</small></li>
+    <li><small><b>Log, Index, Cache</b>: log files, the search index, and the update cache.</small></li>
+    <li><small><b>Type/MIME associations</b>: map file extensions to MIME types.</small></li>
+    <li><small><b>Experts Only</b>: scan mode, how far the crawl may travel, and link rewriting.</small></li>
+  </ul>
+  <br><center><img src="img/android_scanrules.png" width="320" alt="Scan Rules tab" border="0"></center>
+  <br><center><img src="img/android_experts.png" width="320" alt="Experts Only tab" border="0"></center>
+  <br><small>The <a href="step9.html">desktop option reference</a> describes every
+  setting in full. A few desktop options are missing or fixed on Android: the proxy
+  tab has a host and port but no login and password, the download folder is fixed
+  inside the app's storage so there is no output-path setting, and the MIME table
+  holds up to eight entries.</small>
+<br><br>
+
+  <li><b>Run the mirror</b></li>
+  <br><small>HTTrack downloads the site and reports live figures: bytes saved,
+  links scanned, transfer rate, and errors. Tap <b>Abort</b> to stop early; a
+  partial mirror can be resumed later.</small>
+  <br><br><center><img src="img/android_progress.png" width="320" alt="Crawl in progress with live statistics" border="0"></center>
+<br><br>
+
+  <li><b>Browse the result</b></li>
+  <br><small>When the crawl finishes the app reports <b>Success</b> and the mirror
+  location. Tap <b>Browse Mirrored Website</b> to read the copy in your browser,
+  <b>View log</b> to see what happened, or <b>New project</b> to start
+  again.</small>
+  <br><br><center><img src="img/android_finished.png" width="320" alt="Mirror finished, success" border="0"></center>
+<br><br>
+
+  <li><b>Where the files are</b></li>
+  <br><small>Mirrors are written under <tt>/storage/emulated/0/HTTrack/Websites</tt>,
+  in a folder named after the project. That folder lives in your device's shared
+  storage, so a file manager or a USB connection can reach it. Opening
+  <tt>index.html</tt> there browses a mirror without the app.</small>
+
+</ol>
+
+<br><br><br><br>
+<p align="right">Back to <a href="index.html">Home</a></p>
+
+<!-- ==================== Start epilogue ==================== -->
+		</td>
+		</tr>
+		</table>
+	</td>
+	</tr>
+	</table>
+</td>
+</tr>
+</table>
+
+<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
+	<tr>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	</tr>
+</table>
+
+</body>
+
+</html>

html/cache.html

@@ -173,7 +173,7 @@ There are also specific issues regarding this format:
 <ul>
 <li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
 <li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
-<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory must be present <b>(not currently supported)</b></li>
+<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
 </ul>
 
 <br />
@@ -282,7 +282,7 @@ Libraries should generally handle this peculiar format, however.
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/cmddoc.html

@@ -1,155 +0,0 @@
-<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
-
-<head>
-	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
-	<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
-	<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local  site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
-	<title>HTTrack Website Copier - Offline Browser</title>
-
-	<style type="text/css">
-	<!--
-
-body {
-	margin: 0;  padding: 0;  margin-bottom: 15px;  margin-top: 8px;
-	background: #77b;
-}
-body, td {
-	font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
-	}
-
-#subTitle {
-	background: #000;  color: #fff;  padding: 4px;  font-weight: bold; 
-	}
-
-#siteNavigation a, #siteNavigation .current {
-	font-weight: bold;  color: #448;
-	}
-#siteNavigation a:link    { text-decoration: none; }
-#siteNavigation a:visited { text-decoration: none; }
-
-#siteNavigation .current { background-color: #ccd; }
-
-#siteNavigation a:hover   { text-decoration: none;  background-color: #fff;  color: #000; }
-#siteNavigation a:active  { text-decoration: none;  background-color: #ccc; }
-
-
-a:link    { text-decoration: underline;  color: #00f; }
-a:visited { text-decoration: underline;  color: #000; }
-a:hover   { text-decoration: underline;  color: #c00; }
-a:active  { text-decoration: underline; }
-
-#pageContent {
-	clear: both;
-	border-bottom: 6px solid #000;
-	padding: 10px;  padding-top: 20px;
-	line-height: 1.65em;
-	background-image: url(images/bg_rings.gif);
-	background-repeat: no-repeat;
-	background-position: top right;
-	}
-
-#pageContent, #siteNavigation {
-	background-color: #ccd;
-	}
-
-
-.imgLeft  { float: left;   margin-right: 10px;  margin-bottom: 10px; }
-.imgRight { float: right;  margin-left: 10px;   margin-bottom: 10px; }
-
-hr { height: 1px;  color: #000;  background-color: #000;  margin-bottom: 15px; }
-
-h1 { margin: 0;  font-weight: bold;  font-size: 2em; }
-h2 { margin: 0;  font-weight: bold;  font-size: 1.6em; }
-h3 { margin: 0;  font-weight: bold;  font-size: 1.3em; }
-h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
-
-.blak { background-color: #000; }
-.hide { display: none; }
-.tableWidth { min-width: 400px; }
-
-.tblRegular       { border-collapse: collapse; }
-.tblRegular td    { padding: 6px;  background-image: url(fade.gif);  border: 2px solid #99c; }
-.tblHeaderColor, .tblHeaderColor td { background: #99c; }
-.tblNoBorder td   { border: 0; }
-
-
-// -->
-</style>
-
-</head>
-
-<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
-	<tr>
-	<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
-	</tr>
-</table>
-<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
-	<tr>
-	<td id="subTitle">Open Source offline browser</td>
-	</tr>
-</table>
-<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
-<tr class="blak">
-<td>
-	<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
-	<tr>
-	<td colspan="6"> 
-		<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
-		<tr> 
-		<td id="pageContent"> 
-<!-- ==================== End prologue ==================== -->
-
-<h2 align="center"><em>Command-Line Documentation</em></h2>
-
-<br>
-
-The command-line version
-
-<ul>
-  <li><a href="options.html">Command line Options</a></li>
-  <br>List of all powerful command line options<br><br>
-  <li>How to use httrack command-line version:</li>
-  <ul>
-    <li>Open a shell window</li>
-      <br>
-      <br>
-    <li>Type in <tt>httrack</tt> (or the complete path to the httrack executable)</li>
-      <br><small><tt>httrack</tt></small>
-      <br>
-      <br>
-    <li>Add the URLs, separated by a blank space</li>
-      <br><small><tt>httrack www.example.com/foo/</tt></small>
-      <br>
-      <br>
-    <li>If you need, add some options (see the <a href="options.html">option list</a>)</li>
-      <br><small><tt>httrack www.example.com/foo/ -O "/webs" -N4 -P proxy.myhost.com:3128</tt></small>
-      <br>
-      <br>
-    <li>Launch the command line, and wait until the mirror is finishing</li>
-      <br><small>You can (especially on the Unix release) press ^C to stop the mirror or put httrack in background</small>
-      <br>
-      <br>
-  </ul>
-</ul>
-
-<!-- ==================== Start epilogue ==================== -->
-		</td>
-		</tr>
-		</table>
-	</td>
-	</tr>
-	</table>
-</td>
-</tr>
-</table>
-
-<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
-	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
-	</tr>
-</table>
-
-</body>
-
-</html>
-

html/cmdguide.html

@@ -0,0 +1,506 @@
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+
+<head>
+	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+	<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
+	<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local  site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
+	<title>HTTrack Website Copier - Offline Browser</title>
+
+	<style type="text/css">
+	<!--
+
+body {
+	margin: 0;  padding: 0;  margin-bottom: 15px;  margin-top: 8px;
+	background: #77b;
+}
+body, td {
+	font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
+	}
+
+#subTitle {
+	background: #000;  color: #fff;  padding: 4px;  font-weight: bold; 
+	}
+
+#siteNavigation a, #siteNavigation .current {
+	font-weight: bold;  color: #448;
+	}
+#siteNavigation a:link    { text-decoration: none; }
+#siteNavigation a:visited { text-decoration: none; }
+
+#siteNavigation .current { background-color: #ccd; }
+
+#siteNavigation a:hover   { text-decoration: none;  background-color: #fff;  color: #000; }
+#siteNavigation a:active  { text-decoration: none;  background-color: #ccc; }
+
+a:link    { text-decoration: underline;  color: #00f; }
+a:visited { text-decoration: underline;  color: #000; }
+a:hover   { text-decoration: underline;  color: #c00; }
+a:active  { text-decoration: underline; }
+
+#pageContent {
+	clear: both;
+	border-bottom: 6px solid #000;
+	padding: 10px;  padding-top: 20px;
+	line-height: 1.65em;
+	background-image: url(images/bg_rings.gif);
+	background-repeat: no-repeat;
+	background-position: top right;
+	}
+
+#pageContent, #siteNavigation {
+	background-color: #ccd;
+	}
+
+.imgLeft  { float: left;   margin-right: 10px;  margin-bottom: 10px; }
+.imgRight { float: right;  margin-left: 10px;   margin-bottom: 10px; }
+
+hr { height: 1px;  color: #000;  background-color: #000;  margin-bottom: 15px; }
+
+h1 { margin: 0;  font-weight: bold;  font-size: 2em; }
+h2 { margin: 0;  font-weight: bold;  font-size: 1.6em; }
+h3 { margin: 0;  font-weight: bold;  font-size: 1.3em; }
+h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
+
+.blak { background-color: #000; }
+.hide { display: none; }
+.tableWidth { min-width: 400px; }
+
+.tblRegular       { border-collapse: collapse; }
+.tblRegular td    { padding: 6px;  background-image: url(fade.gif);  border: 2px solid #99c; }
+.tblHeaderColor, .tblHeaderColor td { background: #99c; }
+.tblNoBorder td   { border: 0; }
+
+// -->
+</style>
+
+</head>
+
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+	<tr>
+	<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
+	<tr>
+	<td id="subTitle">Open Source offline browser</td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+<tr class="blak">
+<td>
+	<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
+	<tr>
+	<td colspan="6"> 
+		<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
+		<tr> 
+		<td id="pageContent"> 
+<!-- ==================== End prologue ==================== -->
+
+<h2 align="center"><em>Command-Line Guide</em></h2>
+
+<p>This is a task-oriented guide to the <tt>httrack</tt> command line: how to do the
+things people actually ask for, and the handful of defaults that surprise
+newcomers. It sits on top of the
+<a href="httrack.man.html">generated manual page</a>, which lists every option in
+full. When you want the exhaustive detail for a flag, that page is the reference;
+this one is the map.</p>
+
+<p>Two habits before anything else. First, HTTrack has its own options: they are not
+wget or curl flags, so reach for the tables here rather than guessing. Second,
+when a mirror does something you did not expect, the answer is almost always in
+the log. Every project writes <tt>hts-log.txt</tt> (and <tt>hts-err.txt</tt>) into
+its output directory, and those files name every URL that was refused, redirected,
+or filtered out. Read them first.</p>
+
+<h4>On this page</h4>
+<ul class="tblNoBorder">
+  <li><a href="#quickstart">1. Quick start</a></li>
+  <li><a href="#scope">2. Scope: how far the crawl reaches</a></li>
+  <li><a href="#filters">3. Filters and scan rules</a></li>
+  <li><a href="#limits">4. Limits and politeness</a></li>
+  <li><a href="#names">5. File names and types</a></li>
+  <li><a href="#links">6. Links and page building</a></li>
+  <li><a href="#identity">7. Identity, cookies and login</a></li>
+  <li><a href="#proxy">8. Proxy and network</a></li>
+  <li><a href="#update">9. Update and cache</a></li>
+  <li><a href="#experts">10. Experts and scripting</a></li>
+  <li><a href="#recipes">11. Recipes</a></li>
+</ul>
+
+<h3 id="quickstart">1. Quick start</h3>
+
+<p>A mirror is one command: a start URL and an output directory.</p>
+
+<p><tt>httrack https://example.com/ --path mydir</tt></p>
+
+<p>With no other options HTTrack mirrors that site, stays on the same host, follows
+links to any depth, rebuilds them to browse offline, and stores everything under
+<tt>mydir</tt>. The same directory also holds the log files and the
+<tt>hts-cache/</tt> folder that makes a later update or resume possible.</p>
+
+<p>Two defaults are worth knowing up front, because both catch people out:</p>
+<ul>
+  <li>HTTrack throttles itself to about <b>100 KB/s</b> even when you pass no rate
+  option. If a mirror feels slow, that is why. See
+  <a href="#limits">Limits</a> for how to lift it.</li>
+  <li>The download proceeds as a well-behaved robot: it identifies itself as
+  <tt>HTTrack</tt>, obeys <tt>robots.txt</tt>, and sends a Referer with each
+  request. A site that blocks that behavior needs the levers in
+  <a href="#identity">Identity</a>, not brute force.</li>
+</ul>
+
+<h3 id="scope">2. Scope: how far the crawl reaches</h3>
+
+<p>Scope decides which links HTTrack is even willing to follow, before any filter
+you write. Get this right and most "it downloaded too much" or "it only grabbed
+the index" problems disappear.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--depth (-r)</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
+  <tr><td><tt>--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
+  <tr><td><tt>--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
+  <tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
+  <tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
+  <tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
+</table>
+
+<p>The single most common surprise is "only the home page came down." That is
+usually not a scope option at all: it is an off-host redirect. A start URL of
+<tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
+lands you on a different host, and same-host scope stops the crawl there. Start
+from the final URL, or add a filter that re-admits the real host (see
+<a href="#filters">Filters</a>). The log will show the redirect.</p>
+
+<p><tt>-n</tt> is the fix for pages that render locally without their images or
+stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note
+that its embedded-asset handling (following <tt>img</tt>, <tt>link</tt>,
+<tt>script</tt>, <tt>style</tt> and HTML5 <tt>source</tt>/<tt>track</tt> targets
+past the normal depth and filter limits) applies only when <tt>-n</tt> is on; it
+is not automatic. It can also over-fetch by dragging in a whole external host from
+a single link, in which case name the assets you want with a filter instead.</p>
+
+<h3 id="filters">3. Filters and scan rules</h3>
+
+<p>Filters are the number-one source of confusion, and also the tool that solves
+most scope problems once you understand them. A filter is a rule that accepts
+(<tt>+</tt>) or rejects (<tt>-</tt>) URLs by pattern. The sign is mandatory:
+<tt>+pattern</tt> adds, <tt>-pattern</tt> removes, and a bare pattern is an error.</p>
+
+<p>The rules that matter:</p>
+<ul>
+  <li><b>Last match wins.</b> Rules are applied in order and the last one that
+  matches a URL decides its fate. Order your rules from general to specific.</li>
+  <li><b>Wildcards.</b> <tt>*</tt> matches any run of characters;
+  <tt>*[a-z]</tt>, <tt>*[0-9]</tt> and similar classes match sets. So
+  <tt>+*.pdf</tt> means "any URL ending in .pdf".</li>
+  <li><b>Whitelisting.</b> To keep one site and nothing else, deny everything then
+  re-admit the host: <tt>"-*" "+example.com/*"</tt>. A lone <tt>+</tt> rule only
+  adds to the default scope; it never restricts.</li>
+  <li><b>Size rules.</b> <tt>*[&gt;100000]</tt> and <tt>*[&lt;1000]</tt> filter by
+  byte size. Because size is only known once the transfer starts, an oversize file
+  is fetched partway and then aborted, not skipped for free.</li>
+  <li><b>mime: rules.</b> A rule like <tt>-mime:video/*</tt> matches the
+  <tt>Content-Type</tt>. That type is only known after the response headers arrive,
+  so a mime rule <b>cannot stop a request</b>; it can only abort the body. Use a
+  URL pattern when you want to avoid the fetch entirely.</li>
+</ul>
+
+<p>Quote your filters. Shells treat <tt>*</tt>, <tt>[</tt> and sometimes <tt>+</tt>
+specially, so wrap each rule in quotes as shown above. The full pattern language,
+with tables for wildcards, size and mime, is in
+<a href="filters.html">the filters page</a>, and the
+<a href="faq.html">FAQ</a> has a worked tutorial.</p>
+
+<p><b>robots.txt.</b> By default HTTrack obeys <tt>robots.txt</tt> (<tt>-s2</tt>).
+<tt>-s0</tt> ignores it entirely, <tt>-s1</tt> obeys it but lets one of your
+<tt>+</tt> filters override a disallow for a URL you explicitly asked for. Note
+that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
+options will not help there. That is an
+<a href="#identity">identity</a> problem.</p>
+
+<h4>Filter wildcards</h4>
+<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
+bracket forms match narrower sets. The full table, with size and mime rules, is on
+<a href="filters.html">the filters page</a>.</p>
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
+  <tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> &mdash; any URL ending <tt>.pdf</tt></td></tr>
+  <tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> &mdash; a directory-index page</td></tr>
+  <tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
+  <tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
+  <tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
+  <tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
+  <tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
+  <tr><td><tt>*[&lt;NN]</tt>, <tt>*[&gt;NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[&lt;5]</tt> skips GIFs under 5&nbsp;KB</td></tr>
+  <tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
+</table>
+
+<h3 id="limits">4. Limits and politeness</h3>
+
+<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
+accident, and the <a href="abuse.html">abuse page</a> is worth a read. The limits
+below let you go faster when you own the target, and slower when you do not.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--max-rate (-A)</tt></td><td>Maximum transfer rate in bytes/sec. <b>The default is about 100 KB/s even without this flag.</b> Raise it to go faster.</td></tr>
+  <tr><td><tt>--sockets (-c)</tt></td><td>Number of parallel connections (default 4). <tt>--tiny</tt>, <tt>--wide</tt> and <tt>--ultrawide</tt> are presets.</td></tr>
+  <tr><td><tt>--connection-per-second (-%c)</tt></td><td>New connections opened per second (default 5).</td></tr>
+  <tr><td><tt>--max-size (-M)</tt></td><td>Stop after N bytes <b>received from the network</b> across the whole mirror (this counts what was transferred, not what was saved).</td></tr>
+  <tr><td><tt>--max-time (-E)</tt></td><td>Stop after N seconds of wall-clock time.</td></tr>
+  <tr><td><tt>--max-files (-m)</tt></td><td>Per-file size caps.</td></tr>
+  <tr><td><tt>--timeout (-T), --retries (-R), --min-rate (-J), --host-control (-H)</tt></td><td>Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.</td></tr>
+  <tr><td><tt>--max-pause (-G), --pause (-%G)</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
+</table>
+
+<p><b>The security clamps.</b> To keep an accidental typo from turning into a flood,
+HTTrack silently caps a few values: at most 8 connections (<tt>-c</tt>), at most
+10 MB/s (<tt>-A</tt>), and at most 5 new connections per second (<tt>-%c</tt>).
+Ask for more and you get the ceiling, quietly. The single flag
+<tt>--disable-security-limits</tt> lifts all three (the short form <tt>-%!</tt>
+also works, but the bare <tt>!</tt> is awkward to type safely in a shell). Use it
+only against infrastructure you are allowed to load that hard.</p>
+
+<h3 id="names">5. File names and types</h3>
+
+<p>Where local files land, and what they are called, is controlled by the naming
+options. This is the second-biggest source of "why did it do that" questions,
+usually about a URL like <tt>/article?id=42</tt> or a <tt>.php</tt> page that is
+really HTML.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
+  <tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
+  <tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
+  <tr><td><tt>--delayed-type-check (-%N), --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
+  <tr><td><tt>--include-query-string (-%q), --strip-query (-%g)</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
+</table>
+
+<p>The <tt>-N</tt> presets are built from modular arithmetic on the name fields, so
+undocumented number combinations often "work" by accident. If you care about the
+exact layout, use an explicit template (the <tt>%h %p %n %q %t</tt> placeholders)
+rather than a magic number, and check the result on a small crawl first.</p>
+
+<p>A dynamic page served as <tt>.php</tt> or <tt>.asp</tt> that is actually HTML is
+the classic case: without help it can be saved with an extension a browser will
+not open locally. <tt>--assume php=text/html</tt> fixes both the extension and the
+naming.</p>
+
+<h3 id="links">6. Links and page building</h3>
+
+<p>After a page is downloaded, HTTrack parses it for more links and rewrites the
+ones it kept so the local copy browses offline. These options tune both halves.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--keep-links (-K)</tt></td><td>How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare <tt>-K</tt> keeps <b>absolute</b> URLs, and <tt>-K0</tt> is the <b>relative</b> default. <tt>-K3</tt> keeps absolute URIs, <tt>-K4</tt> keeps the original links.</td></tr>
+  <tr><td><tt>--replace-external (-x), --generate-errors (-o)</tt></td><td>Replace external links with an error page, and generate an error page for links that failed.</td></tr>
+  <tr><td><tt>--preserve (-%p), --disable-passwords (-%x)</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
+  <tr><td><tt>--extended-parsing (-%P), --parse-java (-j)</tt></td><td>Aggressive link discovery, and how much script content is parsed for links.</td></tr>
+  <tr><td><tt>--mime-html (-%M)</tt></td><td>Save the whole mirror as a single MIME-encapsulated <tt>.mht</tt> archive (<tt>index.mht</tt>).</td></tr>
+  <tr><td><tt>--index (-I), --build-top-index (-%i), --search-index (-%I)</tt></td><td>Build a per-mirror index, a top index across projects, and a searchable keyword index.</td></tr>
+</table>
+
+<p>HTTrack finds links by parsing HTML and CSS. It does not run JavaScript, so any
+URL a page builds at runtime in script (a lazy-loaded image, a
+JavaScript-assembled path) is invisible to the crawler and will be missing from
+the mirror. There is no flag that fixes this; the asset has to appear in the
+static HTML or CSS to be found. <tt>-%P</tt> widens discovery for links that are
+present but awkwardly formatted, not for links that do not exist until script
+runs.</p>
+
+<h3 id="identity">7. Identity, cookies and login</h3>
+
+<p>By default HTTrack is an honest robot: it sends a <tt>User-Agent</tt> of
+<tt>HTTrack</tt>, a Referer with each link (which reveals the crawl path to the
+server), and obeys robots. Plenty of sites filter exactly that profile. These
+options control what HTTrack says about itself.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--user-agent (-F)</tt></td><td>The <tt>User-Agent</tt>. Set a browser string to get past crawler blocks; <tt>--user-agent ""</tt> sends none.</td></tr>
+  <tr><td><tt>--referer (-%R), --from (-%E), --language (-%l), --accept (-%a)</tt></td><td>Referer, From, Accept-Language and Accept headers.</td></tr>
+  <tr><td><tt>--headers (-%X)</tt></td><td>Add raw header lines to every request.</td></tr>
+  <tr><td><tt>--footer (-%F)</tt></td><td>A footer written into saved pages (on disk, not a network header). See <b>Footer fields</b> below.</td></tr>
+  <tr><td><tt>--cookies (-b), --cookies-file (-%K)</tt></td><td>Accept cookies, and preload a Netscape <tt>cookies.txt</tt>.</td></tr>
+</table>
+
+<p><b>Footer fields.</b> A footer with no <tt>%s</tt> may reference named fields:
+<tt>{addr}</tt>, <tt>{path}</tt>, <tt>{url}</tt>, <tt>{date}</tt> (mirror time),
+<tt>{lastmodified}</tt> (the page's Last-Modified), <tt>{version}</tt>,
+<tt>{mime}</tt>, <tt>{charset}</tt>, <tt>{status}</tt> and <tt>{size}</tt>; write
+<tt>{{</tt> or <tt>}}</tt> for a literal brace. A footer that contains <tt>%s</tt>
+keeps the older positional form (host, path, date in that order). Example:
+<tt>-%F "&lt;!-- Mirrored from {url} on {date} --&gt;"</tt>.</p>
+
+<p><b>Login.</b> For HTTP Basic auth, put the credentials in the URL:
+<tt>http://user:pass@host/</tt>. An <tt>@</tt> inside the username must be written
+<tt>%40</tt>. Only Basic is supported, not Digest.</p>
+
+<p>For cookie or form logins, the simplest path is to log in with a browser, export
+its <tt>cookies.txt</tt>, and drop that file in the project directory so HTTrack
+sends the session cookie. For a form that needs a POST, <tt>--catchurl</tt> can
+capture the exact request your browser sends and replay it. A few cookie caveats
+to know: expiry is ignored, there is a silent cap of about 8 cookies sent per
+request, and <tt>-b0</tt> disables cookies and the reuse of Basic credentials
+across links at the same time.</p>
+
+<h3 id="proxy">8. Proxy and network</h3>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
+  <tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
+  <tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
+  <tr><td><tt>--http-10 (-%h), --keep-alive (-%k), --disable-compression (-%z)</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
+  <tr><td><tt>--bind (-%b), --tolerant (-%B)</tt></td><td>Bind to a local address, and accept technically-bogus responses some servers send.</td></tr>
+</table>
+
+<p>Two network facts worth stating plainly. Over SOCKS5, HTTrack always resolves
+host names at the proxy (remote DNS) for both <tt>socks5://</tt> and
+<tt>socks5h://</tt>, so your local resolver is never consulted. And HTTrack does
+not verify TLS certificates: HTTPS gives you an encrypted transport, but not an
+authenticated one. That is a deliberate choice for a mirroring tool, not a bug,
+but it is worth knowing if you are relying on it for trust.</p>
+
+<h3 id="update">9. Update and cache</h3>
+
+<p>Every project keeps a cache under <tt>hts-cache/</tt>. It records every URL that
+was fetched, together with the options you used, and it is what makes resuming and
+updating possible. It is not a size-limited scratch area you can delete: throw it
+away and you lose the ability to continue or update the mirror.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--continue</tt></td><td>Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.</td></tr>
+  <tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
+  <tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
+  <tr><td><tt>--cache (-C)</tt></td><td>Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.</td></tr>
+  <tr><td><tt>--urlhack (-%u), --keep-www-prefix (-%j), --keep-double-slashes (-%o), --keep-query-order (-%y), --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
+  <tr><td><tt>--debug-cache (-#C), --repair-cache (-#R), --clean</tt></td><td>Inspect the cache, repair its ZIP, and erase cache plus logs.</td></tr>
+</table>
+
+<p><b>The purge trap.</b> An <tt>--update</tt> run rebuilds the list of files the
+mirror should contain, then deletes any previously-mirrored file that is not on the
+new list. This is what keeps a mirror in sync with a shrinking site, but it means a
+partial or interrupted update can delete files you meant to keep. If an update might
+not complete cleanly, add <tt>-X0</tt> to protect the existing tree, and expect
+dynamic pages to look "changed" on every run. See the
+<a href="cache.html">cache page</a> for the details.</p>
+
+<h3 id="experts">10. Experts and scripting</h3>
+
+<p>HTTrack is also a scriptable fetch-and-scan tool. These options turn off the
+mirror behavior and expose the engine.</p>
+
+<table class="tblRegular tableWidth" border="0">
+  <tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
+  <tr><td><tt>--get URL</tt></td><td>Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.</td></tr>
+  <tr><td><tt>--spider --testlinks --skeleton</tt></td><td>Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.</td></tr>
+  <tr><td><tt>--userdef-cmd (-V)</tt></td><td>Run a shell command on each downloaded file; <tt>$0</tt> is the file path. Good for on-the-fly processing.</td></tr>
+  <tr><td><tt>--callback (-%W)</tt></td><td>Load an external callback module to hook the engine.</td></tr>
+  <tr><td><tt>--do-not-log (-Q), --quiet (-q), --verbose (-v), --file-log (-f), --extra-log (-z), --debug-log (-Z)</tt></td><td>Logging: quiet, no questions, verbose on screen, and the various log-to-file levels.</td></tr>
+</table>
+
+<p>The <a href="dev.html">developer page</a> covers the callback API and batch use
+in more depth.</p>
+
+<h3 id="recipes">11. Recipes</h3>
+
+<p>Copy-ready command lines for the tasks people ask about most. Each has the one
+gotcha that trips it up.</p>
+
+<h4>Mirror one site, nothing off-host</h4>
+<p><tt>httrack https://example.com/ --path mydir</tt><br>
+<small>Same-host is already the default. The usual failure is a
+<tt>www.</tt>-to-apex or <tt>http</tt>-to-<tt>https</tt> redirect that moves you off
+host and stops the crawl; start from the final URL, or add
+<tt>"+finalhost/*"</tt>.</small></p>
+
+<h4>One level of links only</h4>
+<p><tt>httrack https://example.com/ --depth=2 --path mydir</tt><br>
+<small>Depth counts the start page as level 1, so one level out is <tt>--depth=2</tt>.</small></p>
+
+<h4>This site only, deny everything else</h4>
+<p><tt>httrack https://example.com/ "-*" "+example.com/*" --path mydir</tt><br>
+<small>You must deny-all first; a lone <tt>+</tt> only adds, and the last matching
+rule wins.</small></p>
+
+<h4>Download the PDFs on a site</h4>
+<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
+<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
+<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
+links and keeps only PDFs reachable from the front page. Instead admit the HTML as
+scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
+depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
+level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
+archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
+included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
+or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
+
+<h4>Keep page requisites, including off-host images</h4>
+<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>
+<small><tt>--near</tt> can over-fetch by pulling in an entire external host from one
+link; if it does, name the assets with <tt>"+cdn.example.com/*"</tt> instead.</small></p>
+
+<h4>Resume an interrupted crawl</h4>
+<p><tt>httrack --continue --path mydir</tt><br>
+<small>Needs an intact <tt>hts-cache/</tt>. Deleting it loses the URL list and your
+options.</small></p>
+
+<h4>Update a mirror, downloading only what changed</h4>
+<p><tt>httrack --update --path mydir</tt><br>
+<small>Purge deletes any previously-mirrored file not seen this run; add
+<tt>--purge-old=0</tt> to protect the tree against a partial update.</small></p>
+
+<h4>Reach a section behind a login cookie</h4>
+<p><small>Export your browser's <tt>cookies.txt</tt> into the project directory,
+then:</small><br>
+<tt>httrack https://example.com/members/ --path mydir</tt><br>
+<small>The file must be Netscape <tt>cookies.txt</tt> in the project folder;
+<tt>--cookies=0</tt> would disable cookies and Basic-auth reuse together.</small></p>
+
+<h4>Get past a crawler-UA block</h4>
+<p><tt>httrack https://example.com/ --user-agent "Mozilla/5.0 (Windows NT 10.0)" --path mydir</tt><br>
+<small>A 403 is server-side, not robots: change the User-Agent, and add
+<tt>--http-10</tt> for fragile CGI. Do not reach for <tt>--robots=0</tt>.</small></p>
+
+<h4>Full-speed mirror on your own server</h4>
+<p><tt>httrack https://example.com/ --max-rate=2000000 --sockets=8 --disable-security-limits --path mydir</tt><br>
+<small>The default throttles to about 100 KB/s. <tt>--disable-security-limits</tt>
+lifts the built-in caps; use it only against infrastructure you are allowed to
+load.</small></p>
+
+<h4>HTTrack as a fetch tool</h4>
+<p><tt>httrack --get https://host/file.bin --path tmp</tt><br>
+<small><tt>--get</tt> fetches one file with cache, index, depth, cookies and robots
+all disabled.</small></p>
+
+<p><br>For the complete, always-current option list, see
+<a href="httrack.man.html">the manual page</a>. For the filter language, see
+<a href="filters.html">filters</a>; for the cache and updates, see
+<a href="cache.html">cache</a>.</p>
+
+<!-- ==================== Start epilogue ==================== -->
+		</td>
+		</tr>
+		</table>
+	</td>
+	</tr>
+	</table>
+</td>
+</tr>
+</table>
+
+<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
+	<tr>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	</tr>
+</table>
+
+</body>
+
+</html>
+

html/contact.html

@@ -243,7 +243,7 @@ roche at httrack dot com (Xavier ROCHE)<br>
 <br><hr><br>
 <br>
 	This program is covered by the GNU General Public License.<br>
-     	HTTrack/HTTrack Website Copier is Copyright (C) 1998-2007 Xavier Roche and other contributors
+     	HTTrack/HTTrack Website Copier is Copyright (C) 1998-2026 Xavier Roche and other contributors
 <br>
 
 <!-- ==================== Start epilogue ==================== -->
@@ -259,7 +259,7 @@ roche at httrack dot com (Xavier ROCHE)<br>
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/dev.html

@@ -147,7 +147,7 @@ This page describes the HTTrack cache format.
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/faq.html

@@ -3,7 +3,7 @@
 <head>
 	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
 	<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
-	<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local  site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows vista, windows seven, windows 8, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
+	<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local  site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, linux, windows, macos, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
 	<title>HTTrack Website Copier - Offline Browser</title>
 
 	<style type="text/css">
@@ -135,9 +135,7 @@ clear language!
 
 <li><a href="#QG2">Where can I find French/other languages documentation?</a><br></li>
 
-<li><a href="#QG3b">Is HTTrack working on Windows Vista/Windows Seven/Windows 8 ?</a><br></li>
-
-<li><a href="#QG3">Is HTTrack working on Windows 95/98 ?</a><br></li>
+<li><a href="#QG3">Which systems does HTTrack run on?</a><br></li>
 
 <li><a href="#QG4">What's the difference between HTTrack, WinHTTrack and WebHTTrack?</a><br></li>
 
@@ -263,7 +261,7 @@ clear language!
 
 <li><a href="#QM8">Can HTTrack generates HP-UX or ISO9660 compatible files?</a><br></li>
 
-<li><a href="#QM9">If there any SOCKS support?</a><br></li>
+<li><a href="#QM9">Is there any SOCKS support?</a><br></li>
 
 <li><a href="#QM10">What's this hts-cache directory? Can I remove it?</a><br></li>
 
@@ -320,10 +318,10 @@ This can easily done by using filters: go to the Option panel, select the 'Scan
 <tt>+www.example.com/gallery/trees/*<br>
 +www.example.com/photos/*</tt><br>
 <br>
-This means "accept all links begining with <tt>www.example.com/gallery/trees/</tt> and <tt>www.example.com/photos/</tt>" 
+This means "accept all links beginning with <tt>www.example.com/gallery/trees/</tt> and <tt>www.example.com/photos/</tt>" 
 - the <tt>+</tt> means "accept" and the final <tt>*</tt> means "any character will match after the previous ones".
 Remember the <tt>*.doc</tt> or <tt>*.zip</tt> encountered when you want to select all files from a certain type on your computer: 
-it is almost the same here, except the begining "+"<br>
+it is almost the same here, except the beginning "+"<br>
 <br>
 Now, we might want to exclude all links in <tt>www.example.com/gallery/trees/hugetrees/</tt>, because with the previous filter,
 we accepted too many files. Here again, you can add a filter rule to refuse these links. Modify the previous filters to:<br>
@@ -331,8 +329,8 @@ we accepted too many files. Here again, you can add a filter rule to refuse thes
 +www.example.com/photos/*<br>
 -www.example.com/gallery/trees/hugetrees/*</tt><br>
 <br>
-You have noticed the <tt>-</tt> in the begining of the third rule: this means "refuse links matching the rule" 
-; and the rule is "any files begining with <tt>www.example.com/gallery/trees/hugetrees/</tt><br>
+You have noticed the <tt>-</tt> in the beginning of the third rule: this means "refuse links matching the rule" 
+; and the rule is "any files beginning with <tt>www.example.com/gallery/trees/hugetrees/</tt><br>
 
 Voila! With these three rules, you have precisely defined what you wanted to capture.<br>
 <br>
@@ -361,12 +359,12 @@ You can freely download it, without paying any fees, copy it to your friends, an
 There are NO official/authorized resellers, because HTTrack is <b>NOT</b> a commercial product.
 But you can be charged for duplication fees, or any other services (example: software CDroms or shareware collections, or fees for maintenance), 
 but you should have been informed that the software was free software/GPL, and you <b><u>MUST</u></b> have received a copy of the GNU General Public License.
-Otherwise this is dishonnest and unfair (ie. selling httrack on ebay without telling that it was a free software is a scam).
+Otherwise this is dishonest and unfair (ie. selling httrack on ebay without telling that it was a free software is a scam).
 </em>
 
 <br><br><a NAME="QG0b">Q: <strong>Are there any risks of viruses with this software?</strong></a><br>
 A: <em>For the software itself: 
-All official releases (at httrack.com) are checked against all known viruses, and the packaging process is also checked. Archives are stored on Un*x servers, not really concerned by viruses. It has been reported, however, that some rogue freeware sites are embedding free softwares and freewares inside badware installers. Always download httrack from the main site (www.httrack.com), and never from an untrusted source!<br>
+All official releases (at httrack.com) are checked against all known viruses, and the packaging process is also checked. Archives are stored on Un*x servers, not really concerned by viruses. It has been reported, however, that some rogue freeware sites are embedding free software and freeware inside badware installers. Always download httrack from the main site (www.httrack.com), and never from an untrusted source!<br>
 For files you are downloading on the WWW using HTTrack: You may encounter websites which were corrupted by viruses, and downloading data on these websites might be dangerous if you execute downloaded executables, or if embedded pages contain infected material (as dangerous as if using a regular Browser). Always ensure that websites you are crawling are safe.
  (Note: remember that using an antivirus software is a good idea once you are connected to the Internet)</em>
 
@@ -376,11 +374,8 @@ A: <em>That's right. You can, however, install WinHTTrack on your own machine, a
 <br><br><a NAME="QG2">Q: <strong>Where can I find French/other languages documentation?</strong></a><br>
 A: <em>Windows interface is available on several languages, but not yet the documentation!</em>
 
-<br><br><a NAME="QG3b">Q: <strong>Is HTTrack working on Windows Vista/Windows Seven/Windows 8 ?</strong></a><br>
-A: <em>Yes, it does</em>
-
-<br><br><a NAME="QG3">Q: <strong>Is HTTrack working on Windows 95/98 ?</strong></a><br>
-A: <em>No, not anymore. You may try to pick an older release (such as 3.33)</em>
+<br><br><a NAME="QG3">Q: <strong>Which systems does HTTrack run on?</strong></a><br>
+A: <em>HTTrack runs on current Windows, Linux and other Unix-like systems, and macOS. Very old platforms such as Windows 95/98 are no longer supported; you may try an older release (such as 3.33) on those.</em>
 
 <br><br><a NAME="QG4">Q: <strong>What's the difference between HTTrack, WinHTTrack and WebHTTrack?</strong></a><br>
 A: <em>WinHTTrack is the Windows GUI release of HTTrack (with a native graphic shell) and WebHTTrack is the Linux/Posix release of HTTrack (with an html graphic shell)</em>
@@ -394,7 +389,7 @@ A: <em>It should. The <tt>configure.ac</tt> may be modified in some cases, howev
 <br><br><a NAME="QG7">Q: <strong>I use HTTrack for professional purpose. What about restrictions/license fee?</strong></a><br>
 A: <em>HTTrack is covered by the GNU General Public License (GPL). There is no restrictions using HTTrack for professional purpose, 
 except if you develop a software which uses HTTrack components (parts of the source, or any other component).
-See the <tt>license.txt</tt> file for more information</em>. See also the next question regarding copyright issues when reditributing downloaded material.
+See the <tt>license.txt</tt> file for more information</em>. See also the next question regarding copyright issues when redistributing downloaded material.
 
 <br><br><a NAME="QG7b">Q: <strong>Is there any license royalties for distributing a mirror made with HTTrack?</strong></a><br>
 A: <em>On the HTTrack side, no. However, sharing, publishing or reusing copyrighted material downloaded from a site requires the authorization of the copyright holders, and possibly paying royalty fees. Always ask the authorization before creating a mirror of a site, even if the site appears to be royalty-free and/or without copyright notice.</em>
@@ -415,7 +410,7 @@ There are several reasons (and solutions) for a mirror to fail. Reading the log
 
 <ul>
  <li>Links within the site refers to external links, or links located in another (or upper) directories, not captured by default - the use of filters is generally THE solution, as this is one of the powerful option in HTTrack. <u>See the above questions/answers</u>.</li>
- <li>Website <a href="#Q1b1">'robots.txt' rules</a> forbide access to several website parts - you can disable them, but only with great care!</li>
+ <li>Website <a href="#Q1b1">'robots.txt' rules</a> forbid access to several website parts - you can disable them, but only with great care!</li>
  <li>HTTrack is filtered (by its default User-agent IDentity) - you can change the Browser User-Agent identity to an anonymous one (MSIE, Netscape..) - here again, use this option with care, as this measure might have been put to avoid some bandwidth abuse (see also the <a href="abuse.html">abuse faq</a>!)</li>
 </ul>
 
@@ -457,14 +452,14 @@ A: <em>Yes, HTTrack does support (since 3.20 release) ipv6 sites, using A/AAAA e
 A: <em>Check the build options (you may have selected user-defined structure with wrong parameters!)</em>
 
 <br><br><a NAME="QT5">Q: <strong>When capturing real audio/video links (.ram), I only get a shortcut!</a></strong></a></br>
-A: <em>Yes, but .ra/.rm associated file should be captured together - except if rtsp:// protocol is used (not supported by HTTrack yet), or if proper filters are needed</em>
+A: <em>The .ra/.rm associated file can be captured together with the shortcut, if proper filters are set. Streaming protocols such as rtsp:// are out of scope: HTTrack is an HTTP/FTP mirror and does not capture rtsp streams.</em>
 
 <br><br><a NAME="QT6">Q: <strong>Using user:password@address is not working!</a></strong></a></br>
 A: <em>Again, first check the <tt>hts-log.txt</tt> and <tt>hts-err.txt</tt> error log files - this can give you precious information<br>
 The site may have a different authentication scheme - form based authentication, for example.
 In this case, use the URL capture features of HTTrack, it might work.
 <br>Note: If your username and/or password contains a '<tt>@</tt>' character, you may have to replace all '<tt>@</tt>' 
-occurences by '<tt>%40</tt>' so that it can work, such as in <tt>user%40domain.com:foobar@www.foo.com/auth/. 
+occurrences by '<tt>%40</tt>' so that it can work, such as in <tt>user%40domain.com:foobar@www.foo.com/auth/. 
 You may have to do the same for all "special" characters like spaces (%20), quotes (%22)..</tt>
 </em>
 <br><br>
@@ -520,7 +515,7 @@ These rules, stored in a file called robots.txt, are given by the website, to sp
 - for example, /cgi-bin or large images files.
 They are followed by default by HTTrack, as it is advised. Therefore, you may miss some files that would have been downloaded without
 these rules - check in your logs if it is the case:<br>
-<tt>Info: Note: due to www.foobar.com remote robots.txt rules, links begining with these path will be forbidden: /cgi-bin/,/images/ (see in the options to disable this)
+<tt>Info: Note: due to www.foobar.com remote robots.txt rules, links beginning with these path will be forbidden: /cgi-bin/,/images/ (see in the options to disable this)
 </tt>
 <br>
 If you want to disable them, just change the corresponding option in the option list! (but only disable this option with great care, 
@@ -540,7 +535,7 @@ HTTrack must find one. Therefore, two index.html will be produced, one with the
 <br>
 It might be a good idea to consider that http://www.foobar.com/ and http://www.foobar.com/index.html are the same links, to avoid
 duplicate files, isn't it?
-NO, because the top index (/) can refer to ANY filename, and if index.html is generally the default name, index.htm can be choosen,
+NO, because the top index (/) can refer to ANY filename, and if index.html is generally the default name, index.htm can be chosen,
 or index.php3, mydog.jpg, or anything you may imagine. (some webmasters are really crazy)
 <br>
 <br>
@@ -595,8 +590,7 @@ A: <em>Simply use the <tt>--assume dat=application/x-zip</tt> option
 A: <em>You may need cookies! Cookies are specific data (for example, your username or password) that are sent to your browser once
 you have logged in certain sites so that you only have to log-in once. For example, after having entered your username in a website, you can
 view pages and articles, and the next time you will go to this site, you will not have to re-enter your username/password.<br>
-To "merge" your personnal cookies to an HTTrack project, just copy the cookies.txt file from your Netscape folder (or the cookies located into the Temporary Internet Files folder for IE)
-into your project folder (or even the HTTrack folder)
+To supply your own cookies to an HTTrack project, put a Netscape-format <tt>cookies.txt</tt> file in your project folder, or point HTTrack at one with the <tt>--cookies-file</tt> option. You can export your browser's session cookies to that format with a browser extension.
 </em>
 <br>
 <br>
@@ -661,7 +655,7 @@ Shell version, skip some slow files, too.</em><br>
 
 </a><a NAME="Q3b">Q: <strong>I want to update a site, but it's taking too much time! What's happening?</strong><br>
 A: <em>First, HTTrack always tries to minimize the download flow by interrogating the server about the
-file changes. But, because HTTrack has to rescan all files from the begining to rebuild the local site structure, 
+file changes. But, because HTTrack has to rescan all files from the beginning to rebuild the local site structure, 
 it can take some time. 
 Besides, some servers are not very smart and always consider that they get newer files, forcing HTTrack to reload them,
 even if no changes have been made!
@@ -740,7 +734,7 @@ retransferred.</em><br>
 </a><a NAME="Q7">Q: <strong>I just want to retrieve all ZIP files or other files in a web
 site/in a page. How do I do it?</strong><br>
 A: <em>You can use different methods. You can use the 'get files near a link' option if
-files are in a foreign domain. You can use, too, a filter adress: adding <tt>+*.zip</tt>
+files are in a foreign domain. You can use, too, a filter address: adding <tt>+*.zip</tt>
 in the URL list (or in the filter list) will accept all ZIP files, even if these files are
 outside the address. <br>
 Example : <tt>httrack www.example.com/someaddress.html +*.zip</tt> will allow
@@ -835,8 +829,8 @@ A: <em>Yes. Use user:password@your_proxy_name as your proxy name (example: <tt>s
 <br><br><a NAME="QM8">Q: <strong>Can HTTrack generates HP-UX or ISO9660 compatible files?</strong></a><br>
 A: <em>Yes. See the build options (-N, or see the WinHTTrack options)</em>
 
-<br><br><a NAME="QM9">Q: <strong>If there any SOCKS support?</strong></a><br>
-A: <em>Not yet!</em>
+<br><br><a NAME="QM9">Q: <strong>Is there any SOCKS support?</strong></a><br>
+A: <em>Yes. HTTrack supports SOCKS5 and HTTP CONNECT proxies: give the proxy with a scheme prefix, e.g. <tt>-P socks5://host:port</tt> or <tt>-P connect://host:port</tt> (prefix <tt>user:pass@</tt> before the host for authenticated proxies).</em>
 
 <br><br><a NAME="QM10">Q: <strong>What's this hts-cache directory? Can I remove it?</strong></a><br>
 A: <em>NO if you want to update the site, because this directory is used by HTTrack for this purpose. 
@@ -931,7 +925,7 @@ A: <em>Feel free to <a href="contact.html">contact us</a>!
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/filters.html

@@ -111,7 +111,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     starts links, the default mode is to mirror these links - i.e. if one of your start page is
     www.example.com/test/index.html, all links starting with www.example.com/test/ will be
     accepted. But links directly in www.example.com/.. will not be accepted, however, because
-    they are in a higher strcuture. This prevent HTTrack from mirroring the whole site. (All
+    they are in a higher structure. This prevent HTTrack from mirroring the whole site. (All
     files in structure levels equal or lower than the primary links will be retrieved.)<br>
     </i>
     <br>
@@ -127,13 +127,15 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     an authorization filter, like <b><tt>+*.gif</tt></b>. The pattern is a plus (this one: <b><tt>+</tt></b>),
     followed by a pattern composed of letters and wildcards (this one: <b><tt>*</tt></b>). 
     <br><br>
-    To forbide a family of links, define
+    To forbid a family of links, define
     an authorization filter, like <b><tt>-*.gif</tt></b>. The pattern is a dash (this one: <b><tt>-</tt></b>),
     followed by a the same kind of pattern as for the authorization filter.
     <br><br>
     Example: +*.gif will accept all files finished by .gif<br>
     Example: -*.gif will refuse all files finished by .gif<br>
     <br>
+    To see which rule accepted or blocked a given URL, run HTTrack with the <b><tt>--why</tt></b> (<b><tt>-%Y</tt></b>) option, described in <a href="httrack.man.html#OPTIONS">the manual page</a>.<br>
+    <br>
     
 <p>    
     <h4>Scan rules based on size (e.g. accept or refuse files bigger/smaller than a certain size)</h4>
@@ -143,7 +145,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     size to ensure that you won't reach a defined limit.
     
     Example: You may want to accept all files on the domain www.example.com, using '+www.example.com/*', 
-    including gif files inside this domain and outside (eternal images), but not take to large images, 
+    including gif files inside this domain and outside (external images), but not take to large images, 
     or too small ones (thumbnails)<br>
     Excluding gif images smaller than 5KB and images larger than 100KB is therefore a good option;
     +www.example.com +*.gif -*.gif*[<5] -*.gif*[>100] 
@@ -189,7 +191,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     <br>
     Filters are analyzed by HTTrack from the first filter to the last one. The complete URL
     name is compared to filters defined by the user or added automatically by HTTrack. <br><br>
-    A scan rule has an higher priority is it is declared later - hierarchy is important: <br>
+    A scan rule has a higher priority if it is declared later - hierarchy is important: <br>
 
     <br>
     <table BORDER="1" CELLPADDING="2">
@@ -307,7 +309,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     <br>
     Filters are analyzed by HTTrack from the first filter to the last one. The sizes
     are compared against scan rules defined by the user.<br><br>
-    A scan rule has an higher priority is it is declared later - hierarchy is important.<br>
+    A scan rule has a higher priority if it is declared later - hierarchy is important.<br>
     
     Note: scan rules based on size can be mixed with regular URL patterns<br>
 
@@ -361,7 +363,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     <br>
     Filters are analyzed by HTTrack from the first filter to the last one. The complete MIME
     type is compared against scan rules defined by the user.<br><br>
-    A scan rule has an higher priority is it is declared later - hierarchy is important<br>
+    A scan rule has a higher priority if it is declared later - hierarchy is important<br>
 
     Note: scan rules based on MIME types can <b>NOT</b> be mixed with regular URL patterns or size patterns within the same rule, but you can use both of them in distinct ones<br>
 
@@ -387,12 +389,12 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
       <tr>
         <td nowrap><tt>-mime:video/*</tt></td>
         <td>This will refuse all video links that were already scheduled for download
-        (i.e. all other 'application/' link download will be aborted)</td>
+        (i.e. the download will be aborted)</td>
       </tr>
       <tr>
         <td nowrap><tt>-mime:video/* -mime:audio/*</tt></td>
         <td>This will refuse all audio and video links that were already scheduled for download
-        (i.e. all other 'application/' link download will be aborted)</td>
+        (i.e. the download will be aborted)</td>
       </tr>
       <tr>
         <td nowrap><tt>-mime:*/* +mime:text/html +mime:image/*</tt></td>
@@ -466,7 +468,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/index.html

@@ -109,12 +109,21 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
   <li>How to Use</li>
   <ul>
     <li><a href="shelldoc.html">WinHTTrack/WebHTTrack (GUI version for Windows or Linux/Unix)</a></li>
-    <br>HTTrack GUI documentation, with step-by-step example, for the Windows release (WinHTTrack) and the Linux/Unix relese (WebHTTrack)<br>
+    <br>HTTrack GUI documentation, with step-by-step example, for the Windows release (WinHTTrack) and the Linux/Unix release (WebHTTrack)<br>
+    <br>
+    <li><a href="android.html">HTTrack on Android</a></li>
+    <br>Step-by-step guide for the Android app<br>
+    <br>
+    <li><a href="httrack.man.html">Command-line version</a></li>
+    <br>How to run HTTrack from the command line, and where to find every option<br>
+    <br>
+    <li><a href="cmdguide.html">Command-line Guide</a></li>
+    <br>A task-oriented guide to the command line: common tasks, recipes, and the defaults that surprise newcomers<br>
     <br>
     <li><a href="fcguide.html">HTTrack Users Guide By Fred Cohen</a></li>
     <br>A tutorial that describes all command-line options, for Linux and Windows users<br>
     <br>
-    <li><a href="dev.html">Developper/Programming</a></li>
+    <li><a href="dev.html">Developer/Programming</a></li>
     <br>How to use HTTrack in batch files, and how to use the library<br>
   </ul>
   <br>
@@ -142,7 +151,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/library.html

@@ -125,7 +125,7 @@ You may also want to check the <tt>httrack.c</tt> and <tt>httrack.h<tt> files to
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/options.html

@@ -101,242 +101,22 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <h2 align="center"><em>Options</em></h2>
 
+<p><strong>Note:</strong> this is a legacy option list and may be out of date.
+For the authoritative, always-current reference see <a href="httrack.man.html">the httrack manual page</a>,
+and for a task-oriented walkthrough see the <a href="cmdguide.html">Command-line Guide</a>.</p>
+
 <ul>
   <li>Filters: <a href="filters.html">how to use them</a></li>
   <br><small>Here you can find informations on filters: how to accept all gif files in a mirror, for example</small>
   <br><br>
-  <li>List of options</li>
-</ul>
+  </ul>
 
-<tt>
-<pre>
+<p>The complete, current list of command-line options is generated from the
+program itself, in <a href="httrack.man.html">the httrack manual page</a>.</p>
 
-w  mirror with automatic wizard
-This is the default scanning option, the engine automatically scans links according to the default options, and filters defined. It does not prompt a message when a "foreign" link is reached.
+<p>WinHTTrack and WebHTTrack offer the same options through their graphical option
+panels, described in the <a href="step9.html">step-by-step guide</a>.</p>
 
-W  semi-automatic mirror with help-wizard (asks questions)
-This option lets the engine ask the user if a link must be mirrored or not, when a new web has been found.
-
-g  just get files (saved in the current directory)
-This option forces the engine not to scan the files indicated - i.e. the engine only gets the files indicated.
-
-i  continue an interrupted mirror using the cache
-This option indicates to the engine that a mirror must be updated or continued.
-
-rN  recurse get with limited link depth of N
-This option sets the maximum recurse level. Default is infinite (the engine "knows" that it should not go out of current domain)
-
-a   stay on the same address
-This is the default primary scanning option, the engine does not go out of domains without permissions (filters, for example)
-
-d   stay on the same principal domain
-This option lets the engine go on all sites that exist on the same principal domain.
-Example: a link located at www.example.com that goes to members.example.com will be followed.
-
-l   stay on the same location (.com, etc.)
-This option lets the engine go on all sites that exist on the same location.
-Example: a link located at www.example.com that goes to www.anyotherweb.com will be followed.
-Warning: this is a potentially dangerous option, limit the recurse depth with r option.
-
-e   go everywhere on the web
-This option lets the engine go on any sites.
-Example: a link located at www.example.com that goes to www.anyotherweb.org will be followed.
-Warning: this is a potentially dangerous option, limit the recurse depth with r option.
-
-n   get non-html files 'near' an html file (ex: an image located outside)
-This option lets the engine catch all files that have references on a page, but that exist outside the web site.
-Example: List of ZIP files links on a page.
-
-t   test all URLs (even forbidden ones)
-This option lets the engine test all links that are not caught.
-Example: to test broken links in a site
-
-x   replace external html links by error pages
-This option tells the engine to rewrite all links not taken into warning pages.
-Example: to browse offline a site, and to warn people that they must be online if they click to external links.
-
-sN  follow robots.txt and meta robots tags
-This option sets the way the engine treats "robots.txt" files. This file is often set by webmasters to avoir cgi-bin directories, or other irrevelant pages.
-Values: 
-  s0  Do not take robots.txt rules
-  s1  Follow rules, if compatible with internal filters
-  s2  Always follow site's rules
-
-bN  accept cookies in cookies.txt
-This option activates or unactivates the cookie
-  b0 do not accept cookies
-  b1 accept cookies
-
-S   stay on the same directory
-This option asks the engine to stay on the same folder level.
-Example: A link in /index.html that points to /sub/other.html will not be followed
-
-D   can only go down into subdirs
-This is the default option, the engine can go everywhere on the same directoy, or in lower structures
-
-U   can only go to upper directories
-This option asks the engine to stay on the same folder level or in upper structures
-
-B   can both go up&down into the directory structure
-This option lets the engine to go in any directory level
-
-Y   mirror ALL links located in the first level pages (mirror links)
-This option is activated for the links typed in the command line
-Example: if you have a list of web sites in www.asitelist.com/index.html, then all these sites will be mirrored
-
-NN  name conversion type (0 *original structure 1,2,3 html/data in one directory)
-  N0 Site-structure (default)
-  N1 Html in web/, images/other files in web/images/
-  N2 Html in web/html, images/other in web/images
-  N3 Html in web/,  images/other in web/
-  N4 Html in web/, images/other in web/xxx, where xxx is the file extension (all gif will be placed onto web/gif, for example)
-  N5 Images/other in web/xxx and Html in web/html
-
-  N99 All files in web/, with random names (gadget !)
-
-  N100 Site-structure, without www.domain.xxx/
-  N101 Identical to N1 except that "web" is replaced by the site's name
-  N102 Identical to N2 except that "web" is replaced by the site's name
-  N103 Identical to N3 except that "web" is replaced by the site's name
-  N104 Identical to N4 except that "web" is replaced by the site's name
-  N105 Identical to N5 except that "web" is replaced by the site's name
-  N199 Identical to N99 except that "web" is replaced by the site's name
-
-  N1001 Identical to N1 except that there is no "web" directory
-  N1002 Identical to N2 except that there is no "web" directory
-  N1003 Identical to N3 except that there is no "web" directory (option set for g option)
-  N1004 Identical to N4 except that there is no "web" directory
-  N1005 Identical to N5 except that there is no "web" directory
-  N1099 Identical to N99 except that there is no "web" directory
-
-LN  long names
-  L0 Filenames and directory names are limited to 8 characters + 3 for extension
-  L1 No restrictions (default)
-
-K   keep original links (e.g. http://www.adr/link) (K0 *relative link)
-This option has only been kept for compatibility reasons
-
-pN  priority mode:
-  p0 just scan, don't save anything (for checking links)
-  p1 save only html files
-  p2 save only non html files
-  p3 save all files
-  p7 get html files before, then treat other files
-
-cN  number of multiple connections (*c8)
-Set the numer of multiple simultaneous connections
-
-O   path for mirror/logfiles+cache (-O path_mirror[,path_cache_and_logfiles])
-This option define the path for mirror and log files
-Example: -P "/user/webs","/user/logs"
-
-P   proxy use (-P proxy:port or -P user:pass@proxy:port)
-This option define the proxy used in this mirror
-Example: -P proxy.myhost.com:8080
-
-F   user-agent field (-F \"user-agent name\
-This option define the user-agent field
-Example: -F "Mozilla/4.5 (compatible; HTTrack 1.2x; Windows 98)"
-
-mN maximum file length for a non-html file
-This option define the maximum size for non-html files
-Example: -m100000
-
-mN,N'  for non html (N) and html (N')
-This option define the maximum size for non-html files and html-files
-Example: -m100000,250000
-
-MN maximum overall size that can be uploaded/scanned
-This option define the maximum amount of bytes that can be downloaded
-Example: -M1000000
-
-EN maximum mirror time in seconds (60=1 minute, 3600=1 hour)
-This option define the maximum time that the mirror can last
-Example: -E3600
-
-AN maximum transfer rate in bytes/seconds (1000=1kb/s max)
-This option define the maximum transfer rate
-Example: -A2000
-
-GN pause transfer if N bytes reached, and wait until lock file is deleted
-This option asks the engine to pause every time N bytes have been transferred, and restarts when the lock file "hts-pause.lock" is being deleted
-Example: -G20000000
-
-u check document type if unknown (cgi,asp..)
-This option define the way the engine checks the file type
-  u0 do not check
-  u1 check but /
-  u2 check always
-
-RN number of retries, in case of timeout or non-fatal errors (*R0)
-This option sets the maximum number of tries that can be processed for a file
-
-o *generate output html file in case of error (404..) (o0 don't generate)
-This option define whether the engine has to generate html output file or not if an error occurred
-
-TN timeout, number of seconds after a non-responding link is shutdown
-This option define the timeout
-Example: -T120
-
-JN traffic jam control, minimum transfert rate (bytes/seconds) tolerated for a link
-This option define the minimum transfer rate
-Example: -J200
-
-HN host is abandoned if: 0=never, 1=timeout, 2=slow, 3=timeout or slow
-This option define whether the engine has to abandon a host if a timeout/"too slow" error occurred
-
-&P extended parsing, attempt to parse all links (even in unknown tags or Javascript)
-This option activates the extended parsing, that attempt to find links in unknown Html code/javascript
-
-j *parse scripts (j0 don't parse)
-This option defines whether the engine has to parse scripts or not to catch included files
-
-I *make an index (I0 don't make)
-This option define whether the engine has to generate an index.html on the top directory
-
-X *delete old files after update (X0 keep delete)
-This option define whether the engine has to delete locally, after an update, files that have been deleted in the remote mirror, or that have been excluded
-
-C *create/use a cache for updates and retries (C0 no cache)
-This option define whether the engine has to generate a cache for retries and updates or not
-
-k  store all files in cache (not useful if files on disk)
-This option define whether the engine has to store all files in cache or not
-
-V execute system command after each files ($0 is the filename: -V \"rm \\$0\
-This option lets the engine execute a command for each file saved on disk
-
-q  quiet mode (no questions)
-Do not ask questions (for example, for confirm an option)
-
-Q  log quiet mode (no log)
-Do not generate log files
-
-v  verbose screen mode
-Log files are printed in the screen
-
-f *log file mode
-Log files are generated into two log files
-
-z  extra infos log
-Add more informations on log files
-
-Z  debug log
-Add debug informations on log files
-
-
---mirror  <URLs> *make a mirror of site(s) 
---get <URLs>  get the files indicated, do not seek other URLs
---mirrorlinks <URLs>  test links in pages (identical to -Y)
---testlinks   <URLs>  test links in pages
---spider  <URLs>  spider site(s), to test links (reports Errors & Warnings)
---update  <URLs>  update a mirror, without confirmation
---skeleton<URLs>  make a mirror, but gets only html files
-
---http10  force http/1.0 requests when possible
-
-</pre>
-</tt>
 
 <!-- ==================== Start epilogue ==================== -->
 
@@ -352,7 +132,7 @@ Add debug informations on log files
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/overview.html

@@ -145,7 +145,7 @@ downloads. HTTrack is fully configurable, and has an integrated help system.
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/plug.html

@@ -286,7 +286,7 @@ module exit point
 the function name and prototype MUST match this prototype
 */
 EXTERNAL_FUNCTION int hts_unplug(httrackp *opt) {
-  fprintf(stder, "Module unplugged");
+  fprintf(stderr, "Module unplugged");
 
   return 1;  /* success */
 }
@@ -331,7 +331,7 @@ httrack --wrapper mylibrary,myparameter-string http://www.example.com
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/plug_330.html

@@ -204,7 +204,7 @@ Below additional function names that can be defined inside the optional libhttra
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/scripting.html

@@ -250,7 +250,7 @@ Script example:
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/server/about.html

@@ -92,7 +92,7 @@ ${LANG_K3} : ${HTTRACK_WEB}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/addurl.html

@@ -159,7 +159,7 @@ ${do:end-if}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/error.html

@@ -72,7 +72,7 @@ ${error}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/file.html

@@ -96,7 +96,7 @@ ${do:loadhash}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/finished.html

@@ -143,7 +143,7 @@ ${path}/${projname}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/help.html

@@ -103,7 +103,7 @@ ${do:end-if}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/index.html

@@ -133,7 +133,7 @@ ${LANG_THANKYOU}!
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option1.html

@@ -158,7 +158,7 @@ ${do:end-if}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option10.html

@@ -160,7 +160,7 @@ ${LANG_IOPT10}:
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option11.html

@@ -250,7 +250,7 @@ ${LANG_W3}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option2.html

@@ -176,7 +176,7 @@ ${listid:build:LISTDEF_3}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option2b.html

@@ -140,7 +140,7 @@ ${do:output-mode:}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option3.html

@@ -191,7 +191,7 @@ ${listid:travel3:LISTDEF_11}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option4.html

@@ -184,7 +184,7 @@ ${LANG_I46}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option5.html

@@ -228,7 +228,7 @@ ${LANG_I64b}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option6.html

@@ -156,7 +156,7 @@ ${LANG_I43b}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option7.html

@@ -148,7 +148,7 @@ ${LANG_B13}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option8.html

@@ -212,7 +212,7 @@ ${LANG_STRIPQUERY}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/option9.html

@@ -166,7 +166,7 @@ ${listid:logtype:LISTDEF_9}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/refresh.html

@@ -204,7 +204,7 @@ ${LANG_H20} ${info.currentjob}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/step2.html

@@ -283,7 +283,7 @@ ${do:end-if:}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/step3.html

@@ -95,7 +95,7 @@ ${do:end-if}
 
 <table border="0" width="100%">
 <tr><td width="90%">
-	<h2 align="center"><em>Select URLs</em></h2>
+	<h2 align="center"><em>${LANG_G44}</em></h2>
 </td>
 ${/* show help only if available */}
 ${do:if-file-exists:html/index.html}
@@ -196,7 +196,7 @@ ${do:output-mode:}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/server/step4.html

@@ -77,7 +77,7 @@ ${do:end-if}
 
 <table border="0" width="100%">
 <tr><td width="90%">
-	<h2 align="center"><em>Start</em></h2>
+	<h2 align="center"><em>${LANG_J9}</em></h2>
 </td>
 ${/* show help only if available */}
 ${do:if-file-exists:html/index.html}
@@ -324,7 +324,7 @@ ${do:output-mode:}
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
+	<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
 	</tr>
 </table>
 

html/shelldoc.html

@@ -141,7 +141,7 @@ You may encounter minor differences (in the display, or in various options) betw
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step.html

@@ -128,7 +128,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step1.html

@@ -143,7 +143,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step2.html

@@ -157,7 +157,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step3.html

@@ -151,7 +151,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step4.html

@@ -128,7 +128,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step5.html

@@ -127,7 +127,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9.html

@@ -144,7 +144,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt1.html

@@ -145,7 +145,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt10.html

@@ -150,7 +150,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt11.html

@@ -182,7 +182,7 @@ In this case, HTTrack won't check the type, because it has learned that "foo" is
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt2.html

@@ -181,7 +181,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt3.html

@@ -145,7 +145,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt4.html

@@ -176,7 +176,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt5.html

@@ -165,7 +165,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt6.html

@@ -162,7 +162,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt7.html

@@ -151,7 +151,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt8.html

@@ -141,7 +141,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

html/step9_opt9.html

@@ -156,7 +156,7 @@ h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
 
 <table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
 	<tr>
-	<td id="footer"><small>&copy; 2007 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
 	</tr>
 </table>
 

man/Makefile.am

@@ -16,6 +16,11 @@ regen-man: makeman.sh $(top_builddir)/src/httrack$(EXEEXT)
 
 # Render html/httrack.man.html from httrack.1. Needs the groff html device
 # (Debian: full "groff" package, not "groff-base"). Run by hand: make -C man regen-man-html
+# Strip groff's version-stamp and creation-date comments so the committed file
+# doesn't churn across groff versions or rebuilds; the ci man-page-sync guard
+# only requires this file to change whenever httrack.1 does.
 regen-man-html: httrack.1
-	groff -t -man -Thtml $(srcdir)/httrack.1 > $(top_srcdir)/html/httrack.man.html
+	groff -t -man -Thtml $(srcdir)/httrack.1 \
+		| sed -e 's/groff version [0-9][0-9.]*/groff/' -e '/^<!-- CreationDate:/d' \
+		> $(top_srcdir)/html/httrack.man.html
 .PHONY: regen-man-html

man/httrack.1

@@ -3,7 +3,7 @@
 .\"
 .\" This file is generated by man/makeman.sh; do not edit by hand.
 .\" SPDX-License-Identifier: GPL-3.0-or-later
-.TH httrack 1 "16 July 2026" "httrack website copier"
+.TH httrack 1 "22 July 2026" "httrack website copier"
 .SH NAME
 httrack \- offline browser : copy websites to a local directory
 .SH SYNOPSIS
@@ -37,8 +37,10 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-%L, \-\-list\fR ]
 [ \fB\-%S, \-\-urllist\fR ]
 [ \fB\-NN, \-\-structure[=N]\fR ]
+[ \fB\-%N, \-\-delayed\-type\-check\fR ]
 [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
 [ \fB\-%M, \-\-mime\-html\fR ]
+[ \fB\-%r, \-\-warc\fR ]
 [ \fB\-LN, \-\-long\-names[=N]\fR ]
 [ \fB\-KN, \-\-keep\-links[=N]\fR ]
 [ \fB\-x, \-\-replace\-external\fR ]
@@ -57,6 +59,7 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-sN, \-\-robots[=N]\fR ]
 [ \fB\-%h, \-\-http\-10\fR ]
 [ \fB\-%k, \-\-keep\-alive\fR ]
+[ \fB\-%z, \-\-disable\-compression\fR ]
 [ \fB\-%B, \-\-tolerant\fR ]
 [ \fB\-%s, \-\-updatehack\fR ]
 [ \fB\-%u, \-\-urlhack\fR ]
@@ -97,6 +100,7 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-%!, \-\-disable\-security\-limits\fR ]
 [ \fB\-V, \-\-userdef\-cmd\fR ]
 [ \fB\-%W, \-\-callback\fR ]
+[ \fB\-y, \-\-background\-on\-suspend\fR ]
 [ \fB\-K, \-\-keep\-links[=N]\fR ]
 .SH DESCRIPTION
 .B httrack
@@ -156,14 +160,14 @@ maximum mirror time in seconds (60=1 minute, 3600=1 hour) (\-\-max\-time[=N])
 .IP \-AN
 maximum transfer rate in bytes/seconds (1000=1KB/s max) (\-\-max\-rate[=N])
 .IP \-%cN
-maximum number of connections/seconds (*%c10) (\-\-connection\-per\-second[=N])
+maximum number of connections/seconds (*%c5) (\-\-connection\-per\-second[=N])
 .IP \-%G
 random pause of MIN[:MAX] seconds between files (e.g. %G5:10) (\-\-pause <param>)
 .IP \-GN
 pause transfer if N bytes reached, and wait until lock file is deleted (\-\-max\-pause[=N])
 .SS Flow control:
 .IP \-cN
-number of multiple connections (*c8) (\-\-sockets[=N])
+number of multiple connections (*c4) (\-\-sockets[=N])
 .IP \-TN
 timeout, number of seconds after a non\-responding link is shutdown; also bounds host name resolution (\-\-timeout[=N])
 .IP \-RN
@@ -189,11 +193,15 @@ structure type (0 *original structure, 1+: see below) (\-\-structure[=N])
 .br
 or user defined structure (\-N "%h%p/%n%q.%t")
 .IP \-%N
-delayed type check, don't make any link test but wait for files download to start instead (experimental) (%N0 don't use, %N1 use for unknown extensions, * %N2 always use)
+delayed type check, don't make any link test but wait for files download to start instead (experimental) (%N0 don't use, %N1 use for unknown extensions, * %N2 always use) (\-\-delayed\-type\-check)
 .IP \-%D
 cached delayed type check, don't wait for remote type during updates, to speedup them (%D0 wait, * %D1 don't wait) (\-\-cached\-delayed\-type\-check)
 .IP \-%M
 generate a RFC MIME\-encapsulated full\-archive (.mht) (\-\-mime\-html)
+.IP \-%r
+write an ISO\-28500 WARC/1.1 archive; \-\-warc\-file NAME sets the output name, \-\-warc\-max\-size N rotates segments past N bytes (\-\-warc)
+.IP \-%t
+keep the original file extension, don't rewrite it from the MIME type (%t0 rewrite)
 .IP \-LN
 long names (L1 *long names / L0 8\-3 conversion / L2 ISO9660 compatible) (\-\-long\-names[=N])
 .IP \-KN
@@ -231,6 +239,8 @@ follow robots.txt and meta robots tags (0=never,1=sometimes,* 2=always, 3=always
 force HTTP/1.0 requests (reduce update features, only for old servers or proxies) (\-\-http\-10)
 .IP \-%k
 use keep\-alive if possible, greately reducing latency for small files and test requests (%k0 don't use) (\-\-keep\-alive)
+.IP \-%z
+do not request compressed content (%z0 request) (\-\-disable\-compression)
 .IP \-%B
 tolerant requests (accept bogus responses on some servers, but not standard!) (\-\-tolerant)
 .IP \-%s
@@ -257,7 +267,7 @@ default referer field sent in HTTP headers (\-\-referer <param>)
 .IP \-%E
 from email address sent in HTTP headers (\-\-from <param>)
 .IP \-%F
-footer string in Html code (\-%F "Mirrored [from host %s [file %s [at %s]]]" (\-\-footer <param>)
+footer string in Html code (\-%F "Mirrored from {url} on {date}"; fields {addr} {path} {url} {date} {lastmodified} {version} {mime} {charset} {status} {size}, or legacy %s) (\-\-footer <param>)
 .IP \-%l
 preferred language (\-%l "fr, en, jp, *" (\-\-language <param>)
 .IP \-%a
@@ -325,8 +335,6 @@ go everywhere on the web (\-\-go\-everywhere)
 .IP \-%H
 debug HTTP headers in logfile (\-\-debug\-headers)
 .SS Guru options: (do NOT use if possible)
-.IP \-#X
-*use optimized engine (limited memory boundary checks) (\-\-fast\-engine)
 .IP \-#test
 list engine self\-tests (run one with \-#test=NAME [args])
 .IP \-#C
@@ -351,8 +359,6 @@ maximum number of links (\-#L1000000) (\-\-advanced\-maxlinks[=N])
 display ugly progress information (\-\-advanced\-progressinfo)
 .IP \-#P
 catch URL (\-\-catch\-url)
-.IP \-#R
-old FTP routines (debug) (\-\-repair\-cache)
 .IP \-#T
 generate transfer ops. log every minutes (\-\-debug\-xfrstats)
 .IP \-#u
@@ -371,6 +377,8 @@ USE IT WITH EXTREME CARE
 execute system command after each files ($0 is the filename: \-V "rm \\$0") (\-\-userdef\-cmd <param>)
 .IP \-%W
 use an external library function as a wrapper (\-%W myfoo.so[,myparameters]) (\-\-callback <param>)
+.IP \-y
+go to background when suspended (y0 don't) (\-\-background\-on\-suspend)
 .SS Details: Option N
 .IP \-N0
 Site\-structure (default)

src/Makefile.am

@@ -63,7 +63,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
@@ -74,7 +74,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \

src/htsalias.c

@@ -114,6 +114,10 @@ const char *hts_optalias[][4] = {
    "strip [host/pattern=]key1,key2,... from URLs"},
   {"cookies-file", "-%K", "param1",
    "load extra cookies from a Netscape cookies.txt"},
+  {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
+  {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
+  {"warc-max-size", "-%rs", "param1",
+   "rotate the WARC archive once a segment passes N bytes (0: single file)"},
   {"why", "-%Y", "param1",
    "explain which filter rule accepts or rejects a URL, then exit"},
   {"pause", "-%G", "param1",

src/htsback.c

@@ -37,6 +37,7 @@ Please visit our Website: http://www.httrack.com
 /* specific definitions */
 #include "htsnet.h"
 #include "htscore.h"
+#include "htswarc.h"
 #include "htsthread.h"
 #include <time.h>
 /* END specific definitions */
@@ -979,6 +980,10 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
         // status finished callback
         RUN_CALLBACK1(opt, xfrstatus, &back[p]);
 
+        // WARC archive of the transaction (request + response/revisit)
+        if (StringNotEmpty(opt->warc_file))
+          warc_write_backtransaction(opt, &back[p]);
+
         return 0;
       } else {                  // testmode
         if (back[p].r.statuscode / 100 >= 3) {  /* Store 3XX, 4XX, 5XX test response codes, but NOT 2XX */
@@ -1055,6 +1060,9 @@ void back_copy_static(const lien_back * src, lien_back * dst) {
   dst->r.soc = INVALID_SOCKET;
   dst->r.adr = NULL;
   dst->r.headers = NULL;
+  dst->r.warc_reqhdr = NULL;
+  dst->r.warc_resphdr = NULL;
+  dst->r.warc_truncated = 0;
   dst->r.out = NULL;
   dst->r.location = dst->location_buffer;
   dst->r.fp = NULL;
@@ -1118,6 +1126,9 @@ int back_unserialize(FILE * fp, lien_back ** dst) {
     (*dst)->chunk_adr = NULL;
     (*dst)->r.adr = NULL;
     (*dst)->r.out = NULL;
+    (*dst)->r.warc_reqhdr = NULL;
+    (*dst)->r.warc_resphdr = NULL;
+    (*dst)->r.warc_truncated = 0;
     (*dst)->r.location = (*dst)->location_buffer;
     (*dst)->r.fp = NULL;
     (*dst)->r.soc = INVALID_SOCKET;
@@ -1585,6 +1596,7 @@ int back_clear_entry(lien_back * back) {
       freet(back->r.headers);
       back->r.headers = NULL;
     }
+    warc_free_request(&back->r);
     // Tout nettoyer
     memset(back, 0, sizeof(lien_back));
     back->r.soc = INVALID_SOCKET;
@@ -1617,14 +1629,15 @@ int back_add_if_not_exists(struct_back * sback, httrackp * opt,
   back_clean(opt, cache, sback);        /* first cleanup the backlog to ensure that we have some entry left */
   if (!back_exist(sback, opt, adr, fil, save)) {
     return back_add(sback, opt, cache, adr, fil, save, referer_adr, referer_fil,
-                    test);
+                    test, HTS_FALSE);
   }
   return 0;
 }
 
-int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char *adr,
-             const char *fil, const char *save, const char *referer_adr, const char *referer_fil,
-             int test) {
+int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
+             const char *adr, const char *fil, const char *save,
+             const char *referer_adr, const char *referer_fil, int test,
+             hts_boolean refetch_whole) {
   lien_back *const back = sback->lnk;
   const int back_max = sback->count;
   int p = 0;
@@ -1701,6 +1714,12 @@ int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char
     else if (strcmp(back[p].url_sav, BACK_ADD_TEST2) == 0)      // test en GET
       back[p].head_request = 2; // test en get
 
+    /* Forced whole refetch (#581): drop the stale temp-ref and skip the resume
+       branches below, so a surviving partial can't Range-loop. */
+    if (refetch_whole) {
+      url_savename_refname_remove(opt, adr, fil);
+    }
+
     /* Stop requested - abort backing */
     /* For update mode: second check after cache lookup not to lose all previous cache data ! */
     if (opt->state.stop && !opt->is_update) {
@@ -1956,8 +1975,9 @@ int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char
         }
       }
       /* Not in cache ; maybe in temporary cache ? Warning: non-movable
-         "url_sav" */
-      else if (back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
+         "url_sav" (skipped on a forced whole refetch, #581) */
+      else if (!refetch_whole &&
+               back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
         const LLint file_size = fsize_utf8(itemback->url_sav);
 
         /* Found file on disk */
@@ -1991,8 +2011,9 @@ int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char
         freet(itemback);        /* delete item */
         itemback = NULL;
       }
-      /* Not in cache or temporary cache ; found on disk ? (hack) */
-      else if (fexist_utf8(save)) {
+      /* Not in cache or temporary cache ; found on disk ? (hack)
+         (skipped on a forced whole refetch, #581) */
+      else if (!refetch_whole && fexist_utf8(save)) {
         const LLint sz = fsize_utf8(save);
 
         // Bon, là il est possible que le fichier ait été partiellement transféré
@@ -2500,6 +2521,19 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
 
     for (i = 0; i < (unsigned int) back_max; i++) {
       if (back[i].status > 0 && back[i].status < STATUS_FTP_TRANSFER) {
+        /* A cap-truncated body is deliberate, not broken: archive what arrived
+           with WARC-Truncated before the abort overwrites the slot's real 2xx
+           status. HTTrack still treats the slot as incomplete afterwards. */
+        if (StringNotEmpty(opt->warc_file) && back[i].r.statuscode > 0 &&
+            back[i].r.warc_resphdr != NULL && back[i].r.size > 0 &&
+            !(back[i].r.is_write && IS_DELAYED_EXT(back[i].url_sav))) {
+          if (back[i].r.is_write && back[i].r.out != NULL)
+            fflush(back[i].r.out);
+          back[i].r.warc_truncated = (limit == HTS_MIRROR_LIMIT_SIZE)
+                                         ? WARC_TRUNC_LENGTH
+                                         : WARC_TRUNC_TIME;
+          warc_write_backtransaction(opt, &back[i]);
+        }
         if (back[i].r.soc != INVALID_SOCKET) {
           deletehttp(&back[i].r);
         }
@@ -3622,6 +3656,10 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                     deleteaddr(&back[i].r);
                     back[i].r.headers = block;
                   }
+                  // Stash the raw response headers for WARC (deletehttp frees
+                  // r.headers when the socket closes, before back_finalize)
+                  if (StringNotEmpty(opt->warc_file))
+                    warc_stash_response(&back[i].r, back[i].r.headers);
 
                   /* 
                      Status code and header-response hacks
@@ -3847,6 +3885,8 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                     deletehttp(&back[i].r);
                     back[i].r.soc = INVALID_SOCKET;
                     back[i].r.statuscode = STATUSCODE_NON_FATAL;
+                    back[i].r.refetch_wholefile =
+                        HTS_TRUE; // retry whole, no Range (#581)
                     strcpybuff(back[i].r.msg,
                                "Bogus 304 on resume, restarting");
                     back[i].status = STATUS_READY;
@@ -4118,6 +4158,9 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                         }
                         back[i].r.soc = INVALID_SOCKET;
                         back[i].r.statuscode = STATUSCODE_NON_FATAL;
+                        // the resume was rejected: the retry must GET the whole
+                        // file, never re-Range a surviving partial/ref (#581)
+                        back[i].r.refetch_wholefile = HTS_TRUE;
                         if (strnotempty(back[i].r.msg))
                           strcpybuff(back[i].r.msg,
                                      "Error attempting to solve status 206 (partial file)");

src/htsback.h

@@ -83,9 +83,12 @@ HTS_INLINE int back_exist(struct_back * sback, httrackp * opt, const char *adr,
                           const char *fil, const char *sav);
 int back_nsoc(const struct_back * sback);
 int back_nsoc_overall(const struct_back * sback);
-int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char *adr,
-             const char *fil, const char *save, const char *referer_adr, const char *referer_fil,
-             int test);
+/* refetch_whole: force a whole-file GET, ignoring any partial/temp-ref resume
+   (set when a prior 206 was rejected as unusable, #581). */
+int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
+             const char *adr, const char *fil, const char *save,
+             const char *referer_adr, const char *referer_fil, int test,
+             hts_boolean refetch_whole);
 int back_add_if_not_exists(struct_back * sback, httrackp * opt,
                            cache_back * cache, const char *adr, const char *fil, const char *save,
                            const char *referer_adr, const char *referer_fil, int test);

src/htscache.c

@@ -856,7 +856,41 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
 // si save==null alors test unqiquement
 static int hts_rename(httrackp * opt, const char *a, const char *b) {
   hts_log_print(opt, LOG_DEBUG, "Cache: rename %s -> %s (%p %p)", a, b, a, b);
-  return rename(a, b);
+  return RENAME(a, b);
+}
+
+/* Open the cache ZIP via hts_fopen_utf8 so a non-ASCII path_log isn't mangled
+   to ANSI (#630); 64-bit funcs keep multi-GB caches whole on Windows LLP64. */
+static voidpf ZCALLBACK hts_zip_fopen_utf8(voidpf opaque, const void *filename,
+                                           int mode) {
+  const char *mode_fopen = NULL;
+
+  (void) opaque;
+  if ((mode & ZLIB_FILEFUNC_MODE_READWRITEFILTER) == ZLIB_FILEFUNC_MODE_READ)
+    mode_fopen = "rb";
+  else if (mode & ZLIB_FILEFUNC_MODE_EXISTING)
+    mode_fopen = "r+b";
+  else if (mode & ZLIB_FILEFUNC_MODE_CREATE)
+    mode_fopen = "wb";
+  if (filename == NULL || mode_fopen == NULL)
+    return NULL;
+  return (voidpf) FOPEN((const char *) filename, mode_fopen);
+}
+
+static unzFile hts_unzOpen_utf8(const char *path) {
+  zlib_filefunc64_def ff;
+
+  fill_fopen64_filefunc(&ff);
+  ff.zopen64_file = hts_zip_fopen_utf8;
+  return unzOpen2_64(path, &ff);
+}
+
+static zipFile hts_zipOpen_utf8(const char *path, int append) {
+  zlib_filefunc64_def ff;
+
+  fill_fopen64_filefunc(&ff);
+  ff.zopen64_file = hts_zip_fopen_utf8;
+  return zipOpen2_64(path, append, NULL, &ff);
 }
 
 /* Pathname of a file inside the mirror dir (rotating concat buffer). */
@@ -874,9 +908,9 @@ static char *reconcile_path(httrackp *opt, const char *name) {
 /* Replace the new-generation file by the old one, when the old one exists. */
 static void reconcile_promote(httrackp *opt, const char *oldname,
                               const char *newname) {
-  if (fexist(reconcile_path(opt, oldname))) {
-    remove(reconcile_path(opt, newname));
-    rename(reconcile_path(opt, oldname), reconcile_path(opt, newname));
+  if (fexist_utf8(reconcile_path(opt, oldname))) {
+    UNLINK(reconcile_path(opt, newname));
+    RENAME(reconcile_path(opt, oldname), reconcile_path(opt, newname));
   }
 }
 
@@ -885,7 +919,7 @@ void hts_cache_reconcile(httrackp *opt, hts_cache_reconcile_mode mode) {
   case CACHE_RECONCILE_PROMOTE:
     /* Previous run rotated new.* to old.* then died before writing: promote
        the old generation back, whichever format it uses. */
-    if (!fexist(reconcile_path(opt, "hts-cache/new.zip")))
+    if (!fexist_utf8(reconcile_path(opt, "hts-cache/new.zip")))
       reconcile_promote(opt, "hts-cache/old.zip", "hts-cache/new.zip");
     break;
   case CACHE_RECONCILE_INTERRUPTED:
@@ -894,16 +928,17 @@ void hts_cache_reconcile(httrackp *opt, hts_cache_reconcile_mode mode) {
        is -1 for a missing file, which would spuriously pass the "< TINY" test
        and overwrite a solid old generation that PROMOTE/ROLLBACK should keep.
      */
-    if (!opt->cache || !fexist(reconcile_path(opt, "hts-in_progress.lock")))
+    if (!opt->cache ||
+        !fexist_utf8(reconcile_path(opt, "hts-in_progress.lock")))
       break;
-    if (fexist(reconcile_path(opt, "hts-cache/new.zip")) &&
-        fexist(reconcile_path(opt, "hts-cache/old.zip")) &&
-        fsize(reconcile_path(opt, "hts-cache/new.zip")) <
+    if (fexist_utf8(reconcile_path(opt, "hts-cache/new.zip")) &&
+        fexist_utf8(reconcile_path(opt, "hts-cache/old.zip")) &&
+        fsize_utf8(reconcile_path(opt, "hts-cache/new.zip")) <
             CACHE_RECONCILE_NEW_TINY &&
-        fsize(reconcile_path(opt, "hts-cache/old.zip")) >
+        fsize_utf8(reconcile_path(opt, "hts-cache/old.zip")) >
             CACHE_RECONCILE_OLD_SOLID &&
-        fsize(reconcile_path(opt, "hts-cache/old.zip")) >
-            fsize(reconcile_path(opt, "hts-cache/new.zip")))
+        fsize_utf8(reconcile_path(opt, "hts-cache/old.zip")) >
+            fsize_utf8(reconcile_path(opt, "hts-cache/new.zip")))
       reconcile_promote(opt, "hts-cache/old.zip", "hts-cache/new.zip");
     break;
   case CACHE_RECONCILE_ROLLBACK:
@@ -940,24 +975,26 @@ void cache_init(cache_back * cache, httrackp * opt) {
 #endif
     if (!cache->ro) {
 #ifdef _WIN32
-      mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache"));
+      /* Windows mkdir takes no mode; use the UTF-8 wrapper for #630. */
+      MKDIR(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                    StringBuff(opt->path_log), "hts-cache"));
 #else
-      mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache"),
+      /* keep the cache dir 0700, not MKDIR's 0755. */
+      mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                    StringBuff(opt->path_log), "hts-cache"),
             HTS_PROTECT_FOLDER);
 #endif
-      if ((fexist(fconcat(
+      if ((fexist_utf8(fconcat(
               OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
               StringBuff(opt->path_log),
               "hts-cache/new.zip")))) { // a previous cache exists.. rename it
         /* Remove OLD cache */
-        if (fexist
-            (fconcat
-             (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-              "hts-cache/old.zip"))) {
-          if (remove
-              (fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                "hts-cache/old.zip")) != 0) {
+        if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                                StringBuff(opt->path_log),
+                                "hts-cache/old.zip"))) {
+          if (UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                             StringBuff(opt->path_log), "hts-cache/old.zip")) !=
+              0) {
             hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
                           "Cache: error while moving previous cache");
           }
@@ -980,33 +1017,27 @@ void cache_init(cache_back * cache, httrackp * opt) {
       hts_log_print(opt, LOG_DEBUG, "Cache: no cache found");
     }
     hts_log_print(opt, LOG_DEBUG, "Cache: size %d",
-                  (int)
-                  fsize(fconcat
-                        (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                         "hts-cache/old.zip")));
+                  (int) fsize_utf8(
+                      fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                              StringBuff(opt->path_log), "hts-cache/old.zip")));
 
     // charger index cache précédent
-    if ((!cache->ro
-         &&
-         fsize(fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                "hts-cache/old.zip")) > 0)
-        || (cache->ro
-            &&
-            fsize(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                   "hts-cache/new.zip")) > 0)
-      ) {
+    if ((!cache->ro &&
+         fsize_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                            StringBuff(opt->path_log), "hts-cache/old.zip")) >
+             0) ||
+        (cache->ro &&
+         fsize_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                            StringBuff(opt->path_log), "hts-cache/new.zip")) >
+             0)) {
       if (!cache->ro) {
-        cache->zipInput =
-          unzOpen(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                   "hts-cache/old.zip"));
+        cache->zipInput = hts_unzOpen_utf8(
+            fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                    StringBuff(opt->path_log), "hts-cache/old.zip"));
       } else {
-        cache->zipInput =
-          unzOpen(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                   "hts-cache/new.zip"));
+        cache->zipInput = hts_unzOpen_utf8(
+            fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                    StringBuff(opt->path_log), "hts-cache/new.zip"));
       }
 
       // Corrupted ZIP file ? Try to repair!
@@ -1026,6 +1057,8 @@ void cache_init(cache_back * cache, httrackp * opt) {
         }
         hts_log_print(opt, LOG_WARNING,
                       "Cache: damaged cache, trying to repair");
+        /* mztools has no UTF-8 hook, so repairing a corrupt cache under a
+           non-ASCII path_log fails cleanly (re-crawl), never forks a twin. */
         if (unzRepair
             (name,
              fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
@@ -1033,11 +1066,11 @@ void cache_init(cache_back * cache, httrackp * opt) {
                                                       StringBuff(opt->path_log),
                                                       "hts-cache/repair.tmp"),
              &repaired, &repairedBytes) == Z_OK) {
-          unlink(name);
-          rename(fconcat
-                 (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                  "hts-cache/repair.zip"), name);
-          cache->zipInput = unzOpen(name);
+          UNLINK(name);
+          RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                         StringBuff(opt->path_log), "hts-cache/repair.zip"),
+                 name);
+          cache->zipInput = hts_unzOpen_utf8(name);
           hts_log_print(opt, LOG_WARNING,
                         "Cache: %d bytes successfully recovered in %d entries",
                         (int) repairedBytes, (int) repaired);
@@ -1133,8 +1166,8 @@ void cache_init(cache_back * cache, httrackp * opt) {
                       "Cache: error trying to open the cache");
       }
 
-    } else if (fsize(reconcile_path(opt, "hts-cache/old.ndx")) > 0 ||
-               fsize(reconcile_path(opt, "hts-cache/new.ndx")) > 0) {
+    } else if (fsize_utf8(reconcile_path(opt, "hts-cache/old.ndx")) > 0 ||
+               fsize_utf8(reconcile_path(opt, "hts-cache/new.ndx")) > 0) {
       /* pre-3.31 (2003) .dat/.ndx cache: import support removed */
       hts_log_print(opt, LOG_ERROR,
                     "Cache: the pre-3.31 .dat/.ndx cache format is no longer "
@@ -1150,67 +1183,58 @@ void cache_init(cache_back * cache, httrackp * opt) {
 #endif
     if (!cache->ro) {
       // ouvrir caches actuels
-      structcheck(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache/"));
+      structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                               StringBuff(opt->path_log), "hts-cache/"));
 
       {
         /* Create ZIP file cache */
-        cache->zipOutput =
-          (void *)
-          zipOpen(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                   "hts-cache/new.zip"), 0);
+        cache->zipOutput = (void *) hts_zipOpen_utf8(
+            fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                    StringBuff(opt->path_log), "hts-cache/new.zip"),
+            0);
 
         if (cache->zipOutput != NULL) {
           // supprimer old.lst
-          if (fexist
-              (fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                "hts-cache/old.lst")))
-            remove(fconcat
-                   (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                    "hts-cache/old.lst"));
+          if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                                  StringBuff(opt->path_log),
+                                  "hts-cache/old.lst")))
+            UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                           StringBuff(opt->path_log), "hts-cache/old.lst"));
           // renommer
-          if (fexist
-              (fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                "hts-cache/new.lst")))
-            rename(fconcat
-                   (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                    "hts-cache/new.lst"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
-                                                  StringBuff(opt->path_log),
-                                                  "hts-cache/old.lst"));
+          if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                                  StringBuff(opt->path_log),
+                                  "hts-cache/new.lst")))
+            RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                           StringBuff(opt->path_log), "hts-cache/new.lst"),
+                   fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                           StringBuff(opt->path_log), "hts-cache/old.lst"));
           // ouvrir
           cache->lst =
-            fopen(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                   "hts-cache/new.lst"), "wb");
+              FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                            StringBuff(opt->path_log), "hts-cache/new.lst"),
+                    "wb");
           strcpybuff(opt->state.strc.path, StringBuff(opt->path_html));
           opt->state.strc.lst = cache->lst;
 
           // supprimer old.txt
-          if (fexist
-              (fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                "hts-cache/old.txt")))
-            remove(fconcat
-                   (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                    "hts-cache/old.txt"));
+          if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                                  StringBuff(opt->path_log),
+                                  "hts-cache/old.txt")))
+            UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                           StringBuff(opt->path_log), "hts-cache/old.txt"));
           // renommer
-          if (fexist
-              (fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                "hts-cache/new.txt")))
-            rename(fconcat
-                   (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                    "hts-cache/new.txt"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
-                                                  StringBuff(opt->path_log),
-                                                  "hts-cache/old.txt"));
+          if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                                  StringBuff(opt->path_log),
+                                  "hts-cache/new.txt")))
+            RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                           StringBuff(opt->path_log), "hts-cache/new.txt"),
+                   fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                           StringBuff(opt->path_log), "hts-cache/old.txt"));
           // ouvrir
           cache->txt =
-            fopen(fconcat
-                  (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                   "hts-cache/new.txt"), "wb");
+              FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                            StringBuff(opt->path_log), "hts-cache/new.txt"),
+                    "wb");
           if (cache->txt) {
             fprintf(cache->txt,
                     "date\tsize'/'remotesize\tflags(request:Update,Range state:File response:Modified,Chunked,gZipped)\t");

src/htscore.c

@@ -39,6 +39,7 @@ Please visit our Website: http://www.httrack.com
 
 /* File defs */
 #include "htscore.h"
+#include "htswarc.h"
 
 /* specific definitions */
 #include "htsbase.h"
@@ -2245,6 +2246,7 @@ int httpmirror(char *url1, httrackp * opt) {
 
   // ending
   usercommand(opt, 0, NULL, NULL, NULL, NULL);
+  warc_close_opt(opt);
 
   // désallocation mémoire & buffers
   XH_uninit;
@@ -3293,10 +3295,10 @@ int back_fill(struct_back * sback, httrackp * opt, cache_back * cache,
       if (ok) {
         if (!back_exist
             (sback, opt, heap(p)->adr, heap(p)->fil, heap(p)->sav)) {
-          if (back_add
-              (sback, opt, cache, heap(p)->adr, heap(p)->fil, heap(p)->sav,
-               heap(heap(p)->precedent)->adr, heap(heap(p)->precedent)->fil,
-               heap(p)->testmode) == -1) {
+          if (back_add(sback, opt, cache, heap(p)->adr, heap(p)->fil,
+                       heap(p)->sav, heap(heap(p)->precedent)->adr,
+                       heap(heap(p)->precedent)->fil, heap(p)->testmode,
+                       heap(p)->refetch_whole) == -1) {
             hts_log_print(opt, LOG_DEBUG,
                           "error: unable to add more links through back_add for back_fill");
 #if BDEBUG==1
@@ -3628,6 +3630,10 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
   if (StringNotEmpty(from->cookies_file))
     StringCopyS(to->cookies_file, from->cookies_file);
 
+  if (StringNotEmpty(from->warc_file))
+    StringCopyS(to->warc_file, from->warc_file);
+  to->warc_max_size = from->warc_max_size;
+
   if (from->pause_max_ms > 0) {
     to->pause_min_ms = from->pause_min_ms;
     to->pause_max_ms = from->pause_max_ms;

src/htscoremain.c

@@ -40,6 +40,7 @@ Please visit our Website: http://www.httrack.com
 #include "htscore.h"
 #include "htsdefines.h"
 #include "htsalias.h"
+#include "htswarc.h"
 #include "htsbauth.h"
 #include "htswrap.h"
 #include "htsmodules.h"
@@ -1055,6 +1056,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
               while(isdigit((unsigned char) *(com + 1)))
                 com++;
             }
+            break;
           case 'c':
             if (isdigit((unsigned char) *(com + 1))) {
               sscanf(com + 1, "%d", &opt->maxsoc);
@@ -1621,10 +1623,9 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                 if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                   HTS_PANIC_PRINTF
                     ("Option %F needs to be followed by a blank space, and a footer string");
-                  printf
-                    ("Example: -%%F \"<!-- Mirrored from %%s by HTTrack Website Copier/"
-                     HTTRACK_AFF_VERSION " " HTTRACK_AFF_AUTHORS
-                     ", %%s -->\"\n");
+                  printf("Example: -%%F \"<!-- Mirrored from {addr}{path} by "
+                         "HTTrack Website Copier/"
+                         "{version} " HTTRACK_AFF_AUTHORS ", {date} -->\"\n");
                   htsmain_free();
                   return -1;
                 } else {
@@ -1789,6 +1790,45 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->cookies_file, argv[na]);
                 }
                 break;
+              case 'r': // warc / warc-file: write an ISO-28500 WARC archive
+                if (*(com + 1) == 'f') { // --warc-file NAME: explicit basename
+                  com++;
+                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
+                    HTS_PANIC_PRINTF(
+                        "Option warc-file needs a blank space and a WARC name");
+                    htsmain_free();
+                    return -1;
+                  }
+                  na++;
+                  if (strlen(argv[na]) >= 1024) {
+                    HTS_PANIC_PRINTF("WARC file name too long");
+                    htsmain_free();
+                    return -1;
+                  }
+                  StringCopy(opt->warc_file, argv[na]);
+                } else if (*(com + 1) == 's') { // --warc-max-size N: rotation
+                  com++;
+                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
+                    HTS_PANIC_PRINTF(
+                        "Option warc-max-size needs a blank space and a size");
+                    htsmain_free();
+                    return -1;
+                  }
+                  na++;
+                  { // reject non-numeric/negative/overflow; keep default 0
+                    // (single file)
+                    char *end;
+                    LLint v;
+                    errno = 0;
+                    v = strtoll(argv[na], &end, 10);
+                    if (isdigit((unsigned char) argv[na][0]) && *end == '\0' &&
+                        errno != ERANGE)
+                      opt->warc_max_size = v;
+                  }
+                } else { // --warc: auto-named archive under the output dir
+                  StringCopy(opt->warc_file, WARC_AUTONAME);
+                }
+                break;
               case 'Y': // why: explain the filter verdict for a URL, no crawl
                 if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                   HTS_PANIC_PRINTF("Option why needs a blank space and a URL");

src/htshelp.c

@@ -77,17 +77,17 @@ void infomsg(const char *msg) {
           if (msg[2] != ' ') {
             if ((msg[3] == ' ') || (msg[4] == ' ')) {
               char cmd[32] = "-";
-              int p = 0;
+              int p;
 
-              while(cmd[p] == ' ')
-                p++;
-              sscanf(msg + p, "%s", cmd + strlen(cmd));
-              /* clears cN -> c */
-              if ((p = (int) strlen(cmd)) > 2)
-                if (cmd[p - 1] == 'N')
-                  cmd[p - 1] = '\0';
-              /* finds alias (if any) */
+              sscanf(msg, "%30s", cmd + strlen(cmd));
+              /* try the flag as-is, then strip a trailing N as the numeric-arg
+                 placeholder (cN -> c); this order keeps -%N from becoming -% */
               p = optreal_find(cmd);
+              if (p < 0 && (int) strlen(cmd) > 2 &&
+                  cmd[strlen(cmd) - 1] == 'N') {
+                cmd[strlen(cmd) - 1] = '\0';
+                p = optreal_find(cmd);
+              }
               if (p >= 0) {
                 /* fings type of parameter: number,param,param concatenated,single cmd */
                 if (strcmp(opttype_value(p), "param") == 0)
@@ -489,13 +489,13 @@ void help(const char *app, int more) {
   infomsg("  MN maximum overall size that can be uploaded/scanned");
   infomsg("  EN maximum mirror time in seconds (60=1 minute, 3600=1 hour)");
   infomsg("  AN maximum transfer rate in bytes/seconds (1000=1KB/s max)");
-  infomsg(" %cN maximum number of connections/seconds (*%c10)");
+  infomsg(" %cN maximum number of connections/seconds (*%c5)");
   infomsg(" %G  random pause of MIN[:MAX] seconds between files (e.g. %G5:10)");
   infomsg
     ("  GN pause transfer if N bytes reached, and wait until lock file is deleted");
   infomsg("");
   infomsg("Flow control:");
-  infomsg("  cN number of multiple connections (*c8)");
+  infomsg("  cN number of multiple connections (*c4)");
   infomsg("  TN timeout, number of seconds after a non-responding link is"
           " shutdown; also bounds host name resolution");
   infomsg
@@ -524,6 +524,10 @@ void help(const char *app, int more) {
   infomsg
     (" %D  cached delayed type check, don't wait for remote type during updates, to speedup them (%D0 wait, * %D1 don't wait)");
   infomsg(" %M  generate a RFC MIME-encapsulated full-archive (.mht)");
+  infomsg(" %r  write an ISO-28500 WARC/1.1 archive; --warc-file NAME sets the "
+          "output name, --warc-max-size N rotates segments past N bytes");
+  infomsg(" %t  keep the original file extension, don't rewrite it from the "
+          "MIME type (%t0 rewrite)");
   infomsg
     ("  LN long names (L1 *long names / L0 8-3 conversion / L2 ISO9660 compatible)");
   infomsg
@@ -554,6 +558,7 @@ void help(const char *app, int more) {
     (" %h  force HTTP/1.0 requests (reduce update features, only for old servers or proxies)");
   infomsg
     (" %k  use keep-alive if possible, greately reducing latency for small files and test requests (%k0 don't use)");
+  infomsg(" %z  do not request compressed content (%z0 request)");
   infomsg
     (" %B  tolerant requests (accept bogus responses on some servers, but not standard!)");
   infomsg
@@ -578,8 +583,10 @@ void help(const char *app, int more) {
     ("  F  user-agent field sent in HTTP headers (-F \"user-agent name\")");
   infomsg(" %R  default referer field sent in HTTP headers");
   infomsg(" %E  from email address sent in HTTP headers");
-  infomsg
-    (" %F  footer string in Html code (-%F \"Mirrored [from host %s [file %s [at %s]]]\"");
+  infomsg(
+      " %F  footer string in Html code (-%F \"Mirrored from {url} on "
+      "{date}\"; fields {addr} {path} {url} {date} {lastmodified} {version} "
+      "{mime} {charset} {status} {size}, or legacy %s)");
   infomsg(" %l  preferred language (-%l \"fr, en, jp, *\"");
   infomsg(" %a  accepted formats (-%a \"text/html,image/png;q=0.9,*/*;q=0.1\"");
   infomsg(" %X  additional HTTP header line (-%X \"X-Magic: 42\"");
@@ -620,7 +627,6 @@ void help(const char *app, int more) {
   infomsg(" %H  debug HTTP headers in logfile");
   infomsg("");
   infomsg("Guru options: (do NOT use if possible)");
-  infomsg(" #X *use optimized engine (limited memory boundary checks)");
   infomsg(" #test  list engine self-tests (run one with -#test=NAME [args])");
   infomsg(" #C  cache list (-#C '*.com/spider*.gif'");
   infomsg(" #R  cache repair (damaged cache)");
@@ -633,7 +639,6 @@ void help(const char *app, int more) {
   infomsg(" #L  maximum number of links (-#L1000000)");
   infomsg(" #p  display ugly progress information");
   infomsg(" #P  catch URL");
-  infomsg(" #R  old FTP routines (debug)");
   infomsg(" #T  generate transfer ops. log every minutes");
   infomsg(" #u  wait time");
   infomsg(" #Z  generate transfer rate statistics every minutes");
@@ -648,9 +653,9 @@ void help(const char *app, int more) {
   infomsg("Command-line specific options:");
   infomsg
     ("  V execute system command after each files ($0 is the filename: -V \"rm \\$0\")");
-  infomsg
-    (" %W use an external library function as a wrapper (-%W myfoo.so[,myparameters])");
-  /* infomsg(" %O do a chroot before setuid"); */
+  infomsg(" %W use an external library function as a wrapper (-%W "
+          "myfoo.so[,myparameters])");
+  infomsg("  y  go to background when suspended (y0 don't)");
   infomsg("");
   infomsg("Details: Option N");
   infomsg("  N0 Site-structure (default)");

src/htslib.c

@@ -36,6 +36,7 @@ Please visit our Website: http://www.httrack.com
 // Fichier librairie .c
 
 #include "htscore.h"
+#include "htswarc.h"
 
 /* specific definitions */
 #include "htsbase.h"
@@ -1201,6 +1202,11 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
   }                             // Fin test pas postfile
   //
 
+  // Stash the raw request for the WARC request record (freed at
+  // back_clear_entry)
+  if (StringNotEmpty(opt->warc_file))
+    warc_stash_request(retour, bstr.buffer);
+
   // Callback
   {
     int test_head =
@@ -2114,6 +2120,8 @@ htsblk http_test(httrackp * opt, const char *adr, const char *fil, char *loc) {
 #if HTS_DEBUG_CLOSESOCK
     DEBUG_W("http_test: deletehttp\n");
 #endif
+    // this probe's htsblk is discarded by callers, so free any WARC stash here
+    warc_free_request(&retour);
     deletehttp(&retour);
     retour.soc = INVALID_SOCKET;
   }
@@ -6010,6 +6018,8 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   StringCopy(opt->footer, HTS_DEFAULT_FOOTER);
   StringCopy(opt->strip_query, "");
   StringCopy(opt->cookies_file, "");
+  StringCopy(opt->warc_file, "");
+  opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
   opt->pause_max_ms = 0;
@@ -6161,6 +6171,7 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->strip_query);
     StringFree(opt->cookies_file);
     StringFree(opt->why_url);
+    StringFree(opt->warc_file);
 
     StringFree(opt->path_html);
     StringFree(opt->path_html_utf8);

src/htsname.c

@@ -615,9 +615,9 @@ int url_savename(lien_adrfilsave *const afs,
             strcpybuff(current.fil, fil_complete);
             // ajouter dans le backing le fichier en mode test
             // savename: rien car en mode test
-            if (back_add
-                (sback, opt, cache, current.adr, current.fil, BACK_ADD_TEST,
-                 referer_adr, referer_fil, 1) != -1) {
+            if (back_add(sback, opt, cache, current.adr, current.fil,
+                         BACK_ADD_TEST, referer_adr, referer_fil, 1,
+                         HTS_FALSE) != -1) {
               int b;
 
               b = back_index(opt, sback, current.adr, current.fil, BACK_ADD_TEST);
@@ -706,7 +706,10 @@ int url_savename(lien_adrfilsave *const afs,
                                 if (!hts_wait_available_socket(sback, opt,
                                                                cache, ptr))
                                   return -1;
-                                if (back_add(sback, opt, cache, moved.adr, moved.fil, methode, referer_adr, referer_fil, 1) != -1) {        // OK
+                                if (back_add(sback, opt, cache, moved.adr,
+                                             moved.fil, methode, referer_adr,
+                                             referer_fil, 1,
+                                             HTS_FALSE) != -1) { // OK
                                   hts_log_print(opt, LOG_DEBUG,
                                                 "(during prefetch) %s (%d) to link %s at %s%s",
                                                 back[b].r.msg,
@@ -725,7 +728,8 @@ int url_savename(lien_adrfilsave *const afs,
                                     has_been_moved = 1; // sinon ne pas forcer has_been_moved car non déplacé
                                   petits_tours++;
                                   //
-                                } else {        // sinon on fait rien et on s'en va.. (ftp etc)
+                                } else { // sinon on fait rien et on s'en va..
+                                         // (ftp etc)
                                   hts_log_print(opt, LOG_DEBUG,
                                                 "Warning: Savename redirect backing error at %s%s",
                                                 moved.adr, moved.fil);
@@ -796,7 +800,6 @@ int url_savename(lien_adrfilsave *const afs,
               hts_log_print(opt, LOG_ERROR,
                             "Unexpected savename backing error at %s%s", adr,
                             fil_complete);
-
             }
             // restaurer
             opt->state._hts_in_html_parsing = hihp;

src/htsopt.h

@@ -256,6 +256,7 @@ struct htsoptstate {
   unsigned int debug_state;
   unsigned int tmpnameid; /**< counter for temporary file names */
   int is_ended;           /**< mirror has finished */
+  void *warc;             /**< open WARC writer (warc_writer*), or NULL */
 };
 
 /* Library handles */
@@ -538,6 +539,10 @@ struct httrackp {
   int pause_max_ms; /**< inter-file pause upper bound, ms */
   String why_url;   /**< URL to diagnose (--why): print the deciding filter rule
                          and exit without crawling */
+  String warc_file; /**< WARC output: WARC_AUTONAME for --warc, or the
+                         --warc-file basename (appended at the tail: ABI) */
+  LLint warc_max_size; /**< --warc-max-size: rotate the archive past this many
+                            bytes (<=0: single file). Tail: ABI */
 };
 
 /* Running statistics for a mirror. */
@@ -658,6 +663,14 @@ struct htsblk {
   int debugid;           /**< connection debug id */
   /* */
   htsrequest req; /**< parameters used for the request */
+  /* Restart-whole signal: a resume this response rejected (unusable 206) must
+     retry with no Range, else a surviving partial/temp-ref loops (#581). */
+  hts_boolean refetch_wholefile;
+  char *warc_reqhdr; /**< stashed raw request header block for WARC (or NULL) */
+  char *
+      warc_resphdr; /**< stashed raw response header block for WARC (or NULL) */
+  int warc_truncated; /**< WARC-Truncated reason for a cap-truncated body
+                           (WARC_TRUNC_*, 0=none). Tail: ABI */
   /*char digest[32+2];   // md5 digest generated by the engine ("" if none) */
 };
 
@@ -681,6 +694,9 @@ struct lien_url {
   char link_import; /**< imported after a move; skip the usual up/down rules */
   int retry;    /**< remaining retries */
   int testmode; /**< test only: send just a HEAD */
+  hts_boolean
+      refetch_whole; /**< force a whole-file GET, ignoring any partial/temp-ref
+                   resume, so a rejected 206 can't loop (#581) */
 };
 
 /* A file being fetched in the background. */

src/htsparse.c

@@ -315,9 +315,18 @@ static void escape_url_parens(char *const s, const size_t size) {
   strlcpybuff(s, buff, size);
 }
 
-/* Strip a default ":80" from lien's authority in place. Any spelling that
-   range-parses to 80 (":80", ":080") is dropped by its matched length, not a
-   hardcoded 3 chars; a value that merely wraps to 80 as an int (#614) stays. */
+/* Default port for lien's scheme (case-insensitive), 80 if absent/unknown; so
+   schemeless and protocol-relative //host links default to 80 (known gap). */
+static int scheme_default_port(const char *lien) {
+  if (strfield(lien, "https:"))
+    return 443;
+  if (strfield(lien, "ftp:"))
+    return 21;
+  return 80;
+}
+
+/* Strip the scheme's own default port (80 http, 443 https, 21 ftp) from lien's
+   authority in place; :80 on https/ftp stays as a real port (#638, #614). */
 void hts_strip_default_port(char *lien, size_t size) {
   char *a;
 
@@ -339,7 +348,8 @@ void hts_strip_default_port(char *lien, size_t size) {
       b++;
     saved = *b;
     *b = '\0';
-    is_default = hts_parse_url_port(a + 1, &port) && port == 80;
+    is_default =
+        hts_parse_url_port(a + 1, &port) && port == scheme_default_port(lien);
     *b = saved;
     if (is_default) { // default port, strip it
       char BIGSTK tempo[HTS_URLMAXSIZE * 2];
@@ -782,17 +792,72 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                   char gmttime[256];
                   char BIGSTK safe_adr[HTS_URLMAXSIZE * 3 + 4];
                   char BIGSTK safe_fil[HTS_URLMAXSIZE * 3 + 4];
+                  char BIGSTK safe_url[HTS_URLMAXSIZE * 6 + 16];
+                  char safe_lastmod[sizeof(r->lastmodified) * 3 + 4];
+                  char safe_ctype[sizeof(r->contenttype) * 3 + 4];
+                  char safe_charset[sizeof(r->charset) * 3 + 4];
+                  char status_str[16];
+                  char size_str[32];
+                  // {url} scheme: jump_identification_const strips it for
+                  // http/https/ftp, so re-add it (bare host is http); for any
+                  // other scheme the host keeps its "scheme://" and we add
+                  // none.
+                  const char *const url_host =
+                      jump_identification_const(urladr());
+                  const char *const url_scheme =
+                      strstr(url_host, "://") != NULL  ? ""
+                      : strfield(urladr(), "https://") ? "https://"
+                      : strfield(urladr(), "ftp://")   ? "ftp://"
+                                                       : "http://";
 
-                  tempo[0] = '\0';
+                  // {addr}/{path} are the escaped host and remote path; {url}
+                  // prepends the scheme to them (credentials already stripped
+                  // by jump_identification_const, and the scheme is a safe
+                  // literal, so no re-escaping is needed).
+                  html_inline_safe(url_host, safe_adr, sizeof(safe_adr));
+                  html_inline_safe(urlfil(), safe_fil, sizeof(safe_fil));
+                  snprintf(safe_url, sizeof(safe_url), "%s%s%s", url_scheme,
+                           safe_adr, safe_fil);
+                  snprintf(status_str, sizeof(status_str), "%d", r->statuscode);
+                  snprintf(size_str, sizeof(size_str), LLintP, (LLint) r->size);
                   time_gmt_rfc822(gmttime);
-                  strcatbuff(tempo, eol);
-                  hts_template_format_str(tempo + strlen(tempo), sizeof(tempo) - strlen(tempo),
-                          StringBuff(opt->footer),
-                          html_inline_safe(jump_identification_const(urladr()), safe_adr, sizeof(safe_adr)),
-                          html_inline_safe(urlfil(), safe_fil, sizeof(safe_fil)), gmttime,
-                          HTTRACK_VERSIONID, /* EOF */ NULL);
-                  strcatbuff(tempo, eol);
-                  HT_ADD(tempo);
+
+                  {
+                    // Every network-derived string is html_inline_safe()'d: the
+                    // footer sits inside an HTML comment, so a value holding
+                    // "-->" would otherwise close it and inject markup (#165).
+                    // status/size are formatted integers and need no escaping.
+                    const hts_footer_field fields[] = {
+                        {"addr", safe_adr},
+                        {"path", safe_fil},
+                        {"url", safe_url},
+                        {"date", gmttime},
+                        {"lastmodified",
+                         html_inline_safe(r->lastmodified, safe_lastmod,
+                                          sizeof(safe_lastmod))},
+                        {"version", HTTRACK_VERSIONID},
+                        {"mime", html_inline_safe(r->contenttype, safe_ctype,
+                                                  sizeof(safe_ctype))},
+                        {"charset", html_inline_safe(r->charset, safe_charset,
+                                                     sizeof(safe_charset))},
+                        {"status", status_str},
+                        {"size", size_str},
+                    };
+
+                    tempo[0] = '\0';
+                    strcatbuff(tempo, eol);
+                    // hts_footer_format returns <0 on overflow, leaving tempo
+                    // unterminated; emitting it would abort in strcatbuff
+                    // below.
+                    if (hts_footer_format(tempo + strlen(tempo),
+                                          sizeof(tempo) - strlen(tempo),
+                                          StringBuff(opt->footer), fields,
+                                          sizeof(fields) / sizeof(fields[0])) >=
+                        0) {
+                      strcatbuff(tempo, eol);
+                      HT_ADD(tempo);
+                    }
+                  }
                 }
                 // Emit charset ?
                 if (emited_footer == 1 && strnotempty(r->charset)) {
@@ -3746,6 +3811,9 @@ int hts_mirror_check_moved(htsmoduleStruct * str,
             heap_top()->retry = heap(ptr)->retry - 1;     // moins 1 retry!
             heap_top()->premier = heap(ptr)->premier;
             heap_top()->precedent = heap(ptr)->precedent;
+            // a rejected resume (unusable 206) must refetch whole, no Range
+            // (#581)
+            heap_top()->refetch_whole = r->refetch_wholefile;
           } else {              // oups erreur, plus de mémoire!!
             return 0;
           }
@@ -3977,18 +4045,17 @@ int hts_mirror_wait_for_next_file(htsmoduleStruct * str,
 #if BDEBUG==1
     printf("crash backing: %s%s\n", heap(ptr)->adr, heap(ptr)->fil);
 #endif
-    if (back_add
-        (sback, opt, cache, urladr(), urlfil(), savename(),
-         heap(heap(ptr)->precedent)->adr, heap(heap(ptr)->precedent)->fil,
-         heap(ptr)->testmode) == -1) {
+    if (back_add(sback, opt, cache, urladr(), urlfil(), savename(),
+                 heap(heap(ptr)->precedent)->adr,
+                 heap(heap(ptr)->precedent)->fil, heap(ptr)->testmode,
+                 heap(ptr)->refetch_whole) == -1) {
       printf("PANIC! : Crash adding error, unexpected error found.. [%d]\n",
              __LINE__);
 #if BDEBUG==1
       printf("error while crash adding\n");
 #endif
-      hts_log_print(opt, LOG_ERROR, "Unexpected backing error for %s%s", urladr(),
-                    urlfil());
-
+      hts_log_print(opt, LOG_ERROR, "Unexpected backing error for %s%s",
+                    urladr(), urlfil());
     }
   }
 #if BDEBUG==1

src/htsselftest.c

@@ -57,6 +57,7 @@ Please visit our Website: http://www.httrack.com
 #include "htssniff.h"
 #include "htscodec.h"
 #include "htsproxy.h"
+#include "htswarc.h"
 #if HTS_USEZLIB
 #include "htszlib.h"
 #endif
@@ -971,6 +972,47 @@ static int st_entities(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+// -#test=footerfmt <template>: expand a -%F footer with fixed fields (drives
+// tests/01_engine-footerfmt.test). Also asserts the overflow/zero-size returns
+// the CLI cap keeps out of reach.
+static int st_footerfmt(httrackp *opt, int argc, char **argv) {
+  static const hts_footer_field fields[] = {
+      {"addr", "host.example"},
+      {"path", "/dir/page.html"},
+      {"url", "http://host.example/dir/page.html"},
+      {"date", "DATE"},
+      {"lastmodified", "LASTMOD"},
+      {"version", "VER"},
+      {"mime", "text/html"},
+      {"charset", "utf-8"},
+      {"status", "200"},
+      {"size", "1234"},
+  };
+  const size_t nfields = sizeof(fields) / sizeof(fields[0]);
+  char out[1024];
+  char tiny[4];
+
+  (void) opt;
+  // Overflow (named and legacy) and a zero-size buffer must return <0, never
+  // truncate silently or write out of bounds.
+  assertf(hts_footer_format(tiny, sizeof(tiny), "{addr}", fields, nfields) < 0);
+  assertf(hts_footer_format(tiny, sizeof(tiny), "a %s b", fields, nfields) < 0);
+  assertf(hts_footer_format(out, 0, "", fields, nfields) < 0);
+  // An empty template yields an empty, terminated string.
+  assertf(hts_footer_format(out, sizeof(out), "", fields, nfields) == 1 &&
+          out[0] == '\0');
+  if (argc < 1) {
+    fprintf(stderr, "footerfmt: needs a template\n");
+    return 1;
+  }
+  if (hts_footer_format(out, sizeof(out), argv[0], fields, nfields) < 0) {
+    fprintf(stderr, "footerfmt: overflow\n");
+    return 1;
+  }
+  printf("%s\n", out);
+  return 0;
+}
+
 /* The unescapers must reserve one byte for the trailing NUL: a 'max'-byte
    dest holding 'max' output chars pre-fix wrote dest[max] (1-byte OOB, caught
    by ASan). Both unescapeEntities and unescapeUrl share the guard. */
@@ -1260,6 +1302,13 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
   if (strcmp(StringBuff(to->cookies_file), "/tmp/jar.txt") != 0)
     err = 1;
 
+  /* warc_file: same String deep-copy path as cookies_file */
+  StringCopy(from->warc_file, "run.warc.gz");
+  StringCopy(to->warc_file, "");
+  copy_htsopt(from, to);
+  if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0)
+    err = 1;
+
   /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
   from->pause_min_ms = 5000;
   from->pause_max_ms = 10000;
@@ -1589,10 +1638,8 @@ static int st_identabs(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
-/* Default-port strip (#627): a genuine 80 (any spelling) is removed by its
-   matched length, host preserved; a non-80 port or one that only wraps to 80 as
-   a 32-bit int (#614) is left intact. Guards the old bug where ":080"/":0080"
-   dropped a hardcoded 3 chars and glued the leftover digits onto the host. */
+/* Default-port strip is scheme-aware (#638), overflow-safe (#614): a scheme's
+   own default (any spelling) is dropped, a real port stays; guards #627. */
 static int st_stripport(httrackp *opt, int argc, char **argv) {
   static const struct {
     const char *in, *out;
@@ -1606,6 +1653,13 @@ static int st_stripport(httrackp *opt, int argc, char **argv) {
       {"http://127.0.0.1:8080/x", "http://127.0.0.1:8080/x"},
       {"http://127.0.0.1:4294967376/x", "http://127.0.0.1:4294967376/x"},
       {"http://127.0.0.1/x", "http://127.0.0.1/x"},
+      {"https://127.0.0.1:443/x", "https://127.0.0.1/x"},
+      {"https://127.0.0.1:80/x", "https://127.0.0.1:80/x"},
+      // Scheme match is case-insensitive: HTTPS' default is 443, so :80 stays.
+      {"HTTPS://127.0.0.1:80/x", "HTTPS://127.0.0.1:80/x"},
+      {"ftp://127.0.0.1:21/x", "ftp://127.0.0.1/x"},
+      {"ftp://127.0.0.1:80/x", "ftp://127.0.0.1:80/x"},
+      {"http://127.0.0.1:443/x", "http://127.0.0.1:443/x"},
   };
 
   size_t k;
@@ -3210,6 +3264,498 @@ static int st_ftpuser(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* Bounded substring search (records carry NUL bytes; strstr won't do). */
+static const char *warc_memstr(const char *hay, const char *needle,
+                               size_t haylen, size_t nlen) {
+  if (nlen == 0 || haylen < nlen)
+    return NULL;
+  {
+    size_t i;
+    for (i = 0; i + nlen <= haylen; i++) {
+      if (memcmp(hay + i, needle, nlen) == 0)
+        return hay + i;
+    }
+  }
+  return NULL;
+}
+
+/* Slurp a whole file into a malloc'd buffer; sets *len. NULL on error. */
+static unsigned char *warc_slurp(const char *path, size_t *len) {
+  FILE *f = FOPEN(path, "rb");
+  unsigned char *buf;
+  long sz;
+  if (f == NULL)
+    return NULL;
+  if (fseek(f, 0, SEEK_END) != 0 || (sz = ftell(f)) < 0) {
+    fclose(f);
+    return NULL;
+  }
+  rewind(f);
+  buf = malloct((size_t) sz + 1);
+  if (buf == NULL) {
+    fclose(f);
+    return NULL;
+  }
+  *len = fread(buf, 1, (size_t) sz, f);
+  fclose(f);
+  return buf;
+}
+
+/* Inflate one gzip member at *in (limit end); returns the decompressed record
+   in a malloc'd buffer (*out_len), advancing *in past the member. NULL at end
+   or on error (*out_len distinguishes: 0 and NULL = clean end). */
+static unsigned char *warc_next_member(const unsigned char **in,
+                                       const unsigned char *end,
+                                       size_t *out_len) {
+  z_stream zs;
+  unsigned char *out = NULL;
+  size_t len = 0;
+  int zerr;
+  *out_len = 0;
+  if (*in >= end)
+    return NULL;
+  memset(&zs, 0, sizeof(zs));
+  if (inflateInit2(&zs, 15 + 32) != Z_OK)
+    return NULL;
+  zs.next_in = (const Bytef *) *in;
+  zs.avail_in = (uInt) (end - *in);
+  do {
+    unsigned char tmp[8192];
+    size_t got;
+    zs.next_out = tmp;
+    zs.avail_out = sizeof(tmp);
+    zerr = inflate(&zs, Z_NO_FLUSH);
+    if (zerr != Z_OK && zerr != Z_STREAM_END) {
+      freet(out);
+      inflateEnd(&zs);
+      return NULL;
+    }
+    got = sizeof(tmp) - zs.avail_out;
+    if (got > 0) {
+      unsigned char *n = realloct(out, len + got + 1);
+      if (n == NULL) {
+        freet(out);
+        inflateEnd(&zs);
+        return NULL;
+      }
+      out = n;
+      memcpy(out + len, tmp, got);
+      len += got;
+    }
+  } while (zerr != Z_STREAM_END);
+  *in = (const unsigned char *) zs.next_in; /* start of the next member */
+  inflateEnd(&zs);
+  if (out != NULL)
+    out[len] = '\0';
+  *out_len = len;
+  return out;
+}
+
+/* Feed a synthetic transaction and validate the resulting .warc.gz against the
+   WARC/1.1 spec: each record a self-standing gzip member starting WARC/1.,
+   Content-Length == block length, the \r\n\r\n trailer intact, the response
+   body round-trips, and the encoding headers are stripped (strategy B). */
+static int st_warc(httrackp *opt, int argc, char **argv) {
+  char path[HTS_URLMAXSIZE];
+  warc_writer *w;
+  unsigned char *data;
+  size_t data_len = 0;
+  const unsigned char *p, *end;
+  int err = 0, nrec = 0, nresp = 0, nreq = 0, nrevisit = 0, ninfo = 0;
+  int seen_a_body = 0, body_occurrences = 0, a2_bodyless = 0, nm_cl_ok = 0;
+  static const char a_body[] = "Hello, WARC!\n";
+
+  if (argc < 1) {
+    fprintf(stderr, "warc: needs a writable directory\n");
+    return 1;
+  }
+  fconcat(path, sizeof(path), argv[0], "warc-selftest.warc.gz");
+
+  w = warc_open(opt, path);
+  assertf(w != NULL);
+
+  /* 200 HTML: bogus Content-Length + gzip/chunked encodings must be stripped.
+   */
+  warc_write_transaction(
+      w, "http://test.local/a.html", "127.0.0.1",
+      "GET /a.html HTTP/1.1\r\nHost: test.local\r\n\r\n",
+      "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Encoding: "
+      "gzip\r\nTransfer-Encoding: chunked\r\nContent-Length: 999\r\n\r\n",
+      a_body, sizeof(a_body) - 1, NULL, 200, 0, 0);
+
+  /* 302 redirect: header-only, no body. */
+  warc_write_transaction(
+      w, "http://test.local/r", "127.0.0.1",
+      "GET /r HTTP/1.1\r\nHost: test.local\r\n\r\n",
+      "HTTP/1.1 302 Found\r\nLocation: http://test.local/a.html\r\n\r\n", NULL,
+      0, NULL, 302, 0, 0);
+
+  /* 200 binary, chunked coding on the wire (already de-chunked here). */
+  warc_write_transaction(
+      w, "http://test.local/b.bin", "127.0.0.1",
+      "GET /b.bin HTTP/1.1\r\nHost: test.local\r\n\r\n",
+      "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n"
+      "Transfer-Encoding: chunked\r\n\r\n",
+      "\x00\x01\x02\x03\x04", 5, NULL, 200, 0, 0);
+
+  /* 200 with a body shorter than the declared Content-Length (rewritten). */
+  warc_write_transaction(
+      w, "http://test.local/trunc", "127.0.0.1",
+      "GET /trunc HTTP/1.1\r\nHost: test.local\r\n\r\n",
+      "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: "
+      "100\r\n\r\n",
+      "short", 5, NULL, 200, 0, 0);
+
+  /* Same payload as a.html at a new URL: identical-payload-digest revisit
+     (OpenSSL builds only; a plain build writes a second full response). */
+  warc_write_transaction(w, "http://test.local/a2.html", "127.0.0.1",
+                         "GET /a2.html HTTP/1.1\r\nHost: test.local\r\n\r\n",
+                         "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n",
+                         a_body, sizeof(a_body) - 1, NULL, 200, 0, 0);
+
+  /* 304 revisit with an EMPTY response-header block: the block is just the
+     2-byte separator, so declared Content-Length must be exactly 2 (F3). */
+  warc_write_transaction(w, "http://test.local/nm", "127.0.0.1",
+                         "GET /nm HTTP/1.1\r\nHost: test.local\r\n\r\n", "",
+                         NULL, 0, NULL, 304, 1, 0);
+
+  warc_close(w);
+
+  data = warc_slurp(path, &data_len);
+  assertf(data != NULL);
+  p = data;
+  end = data + data_len;
+
+  while (p < end) {
+    size_t rlen = 0;
+    unsigned char *rec = warc_next_member(&p, end, &rlen);
+    const char *sep, *cl;
+    long long block_len = 0; /* 0 on a parse failure; err is already set */
+    size_t hdr_len;
+    if (rec == NULL) {
+      if (rlen == 0)
+        break; /* clean end */
+      err = 1;
+      break;
+    }
+    nrec++;
+    /* magic */
+    if (rlen < 8 || memcmp(rec, "WARC/1.", 7) != 0)
+      err = 1;
+    /* record header ends at the first blank line */
+    sep = warc_memstr((char *) rec, "\r\n\r\n", rlen, 4);
+    if (sep == NULL) {
+      err = 1;
+      freet(rec);
+      continue;
+    }
+    hdr_len = (size_t) ((const unsigned char *) sep - rec) + 4;
+    /* Content-Length must equal the actual block length */
+    cl = warc_memstr((char *) rec, "Content-Length:", hdr_len, 15);
+    if (cl == NULL || sscanf(cl + 15, "%lld", &block_len) != 1)
+      err = 1;
+    else {
+      if (hdr_len + (size_t) block_len + 4 != rlen)
+        err = 1; /* header + block + trailing CRLFCRLF */
+      else if (memcmp(rec + hdr_len + block_len, "\r\n\r\n", 4) != 0)
+        err = 1; /* trailer intact */
+    }
+    if (warc_memstr((char *) rec, "WARC-Type: warcinfo", hdr_len, 19) != NULL)
+      ninfo++;
+    if (warc_memstr((char *) rec, "WARC-Type: request", hdr_len, 18) != NULL)
+      nreq++;
+    if (warc_memstr((char *) rec, "WARC-Type: response", hdr_len, 19) != NULL)
+      nresp++;
+    if (warc_memstr((char *) rec, "WARC-Type: revisit", hdr_len, 18) != NULL)
+      nrevisit++;
+    /* F1: the full body must appear exactly once across the whole file (a
+       revisit must not re-embed it). */
+    if (warc_memstr((char *) rec, a_body, rlen, sizeof(a_body) - 1) != NULL)
+      body_occurrences++;
+    /* F1: the a2.html identical-payload-digest revisit carries no body. */
+    if (warc_memstr((char *) rec, "WARC-Target-URI: http://test.local/a2.html",
+                    hdr_len, 42) != NULL &&
+        warc_memstr((char *) rec, "WARC-Type: revisit", hdr_len, 18) != NULL)
+      a2_bodyless =
+          (warc_memstr((char *) rec, a_body, rlen, sizeof(a_body) - 1) == NULL);
+    /* F3: the empty-header 304 revisit block is exactly the 2-byte separator
+       (the request record shares this target URI, so match the revisit only).
+     */
+    if (warc_memstr((char *) rec, "WARC-Target-URI: http://test.local/nm",
+                    hdr_len, 37) != NULL &&
+        warc_memstr((char *) rec, "WARC-Type: revisit", hdr_len, 18) != NULL)
+      nm_cl_ok = (block_len == 2);
+    /* a.html response body must round-trip and carry no encoding headers */
+    if (warc_memstr((char *) rec, "WARC-Target-URI: http://test.local/a.html",
+                    hdr_len, 41) != NULL &&
+        warc_memstr((char *) rec, "msgtype=response", hdr_len, 16) != NULL) {
+      const char *bsep = warc_memstr((char *) rec + hdr_len, "\r\n\r\n",
+                                     (size_t) block_len, 4);
+      if (bsep == NULL)
+        err = 1;
+      else {
+        size_t bodyoff = (size_t) (bsep - (char *) rec) + 4;
+        size_t got = rlen - 4 - bodyoff; /* minus record trailer */
+        if (got != sizeof(a_body) - 1 ||
+            memcmp(rec + bodyoff, a_body, got) != 0)
+          err = 1;
+        seen_a_body = 1;
+      }
+      if (warc_memstr((char *) rec, "Content-Encoding", hdr_len + block_len,
+                      16) != NULL ||
+          warc_memstr((char *) rec, "Transfer-Encoding", hdr_len + block_len,
+                      17) != NULL)
+        err = 1;
+    }
+    freet(rec);
+  }
+  freet(data);
+
+  /* warcinfo + 6 transactions (response/revisit + request each) = 13 records.
+   */
+  if (ninfo != 1 || nreq != 6 || nrec != 13 || !seen_a_body || !nm_cl_ok)
+    err = 1;
+#if HTS_USEOPENSSL
+  /* a.html + b.bin + trunc + 302 are full responses; a2.html deduped to a
+     revisit (bodyless), nm is the 304 revisit; the body appears exactly once.
+   */
+  if (nrevisit != 2 || nresp != 4 || !a2_bodyless || body_occurrences != 1)
+    err = 1;
+#else
+  /* No digests: a2.html is a second full response, so the body appears twice
+     and only the 304 nm is a revisit. */
+  if (nrevisit != 1 || nresp != 5 || body_occurrences != 2)
+    err = 1;
+  (void) a2_bodyless; /* only meaningful with digests */
+#endif
+
+  printf("warc: %d records (%d response, %d request, %d revisit): %s\n", nrec,
+         nresp, nreq, nrevisit, err ? "FAIL" : "OK");
+  return err;
+}
+
+/* Parse a record's header/block split; sets *hdr_len and *block_len, returns 0
+   when Content-Length matches the actual block bytes, -1 otherwise. */
+static int warc_rec_split(const unsigned char *rec, size_t rlen,
+                          size_t *hdr_len, long long *block_len) {
+  const char *sep = warc_memstr((const char *) rec, "\r\n\r\n", rlen, 4);
+  const char *cl;
+  *block_len = 0;
+  if (sep == NULL)
+    return -1;
+  *hdr_len = (size_t) ((const unsigned char *) sep - rec) + 4;
+  cl = warc_memstr((const char *) rec, "Content-Length:", *hdr_len, 15);
+  if (cl == NULL || sscanf(cl + 15, "%lld", block_len) != 1 ||
+      *hdr_len + (size_t) *block_len + 4 != rlen)
+    return -1;
+  return 0;
+}
+
+/* A cap-truncated body is still archived, tagged WARC-Truncated (v1.1). Assert
+   the sole response carries "WARC-Truncated: length" and every record parses.
+ */
+static int st_warc_trunc(httrackp *opt, int argc, char **argv) {
+  char path[HTS_URLMAXSIZE];
+  warc_writer *w;
+  unsigned char *data;
+  size_t data_len = 0;
+  const unsigned char *p, *end;
+  int err = 0, found_trunc = 0, nresp = 0;
+  static const char body[] = "partial body bytes\n";
+
+  if (argc < 1) {
+    fprintf(stderr, "warc-trunc: needs a writable directory\n");
+    return 1;
+  }
+  fconcat(path, sizeof(path), argv[0], "warc-trunc.warc.gz");
+  w = warc_open(opt, path);
+  assertf(w != NULL);
+  warc_write_transaction(
+      w, "http://test.local/big.bin", "127.0.0.1",
+      "GET /big.bin HTTP/1.1\r\nHost: test.local\r\n\r\n",
+      "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n\r\n", body,
+      sizeof(body) - 1, NULL, 200, 0, WARC_TRUNC_LENGTH);
+  warc_close(w);
+
+  data = warc_slurp(path, &data_len);
+  assertf(data != NULL);
+  p = data;
+  end = data + data_len;
+  while (p < end) {
+    size_t rlen = 0, hdr_len = 0;
+    long long block_len = 0;
+    unsigned char *rec = warc_next_member(&p, end, &rlen);
+    if (rec == NULL) {
+      if (rlen != 0)
+        err = 1;
+      break;
+    }
+    if (warc_rec_split(rec, rlen, &hdr_len, &block_len) != 0)
+      err = 1;
+    else if (warc_memstr((char *) rec, "WARC-Type: response", hdr_len, 19) !=
+             NULL) {
+      nresp++;
+      if (warc_memstr((char *) rec, "WARC-Truncated: length", hdr_len, 22) !=
+          NULL)
+        found_trunc = 1;
+    }
+    freet(rec);
+  }
+  freet(data);
+  if (!found_trunc || nresp != 1)
+    err = 1;
+  printf("warc-trunc: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
+/* An ftp:// capture is ONE resource record: WARC-Type: resource, the payload's
+   own Content-Type, block == payload, and no request/response pair. */
+static int st_warc_ftp(httrackp *opt, int argc, char **argv) {
+  char path[HTS_URLMAXSIZE];
+  warc_writer *w;
+  unsigned char *data;
+  size_t data_len = 0;
+  const unsigned char *p, *end;
+  int err = 0, nresource = 0, nresp = 0, nreq = 0;
+  static const char body[] = "\x00\x01"
+                             "FTP payload"
+                             "\x02\x03";
+
+  if (argc < 1) {
+    fprintf(stderr, "warc-ftp: needs a writable directory\n");
+    return 1;
+  }
+  fconcat(path, sizeof(path), argv[0], "warc-ftp.warc.gz");
+  w = warc_open(opt, path);
+  assertf(w != NULL);
+  warc_write_resource(w, "ftp://ftp.local/file.bin", "127.0.0.1",
+                      "application/octet-stream", body, sizeof(body) - 1, NULL,
+                      0);
+  warc_close(w);
+
+  data = warc_slurp(path, &data_len);
+  assertf(data != NULL);
+  p = data;
+  end = data + data_len;
+  while (p < end) {
+    size_t rlen = 0, hdr_len = 0;
+    long long block_len = 0;
+    unsigned char *rec = warc_next_member(&p, end, &rlen);
+    if (rec == NULL) {
+      if (rlen != 0)
+        err = 1;
+      break;
+    }
+    if (warc_rec_split(rec, rlen, &hdr_len, &block_len) != 0)
+      err = 1;
+    if (warc_memstr((char *) rec, "WARC-Type: resource", hdr_len, 19) != NULL) {
+      nresource++;
+      if ((size_t) block_len != sizeof(body) - 1 ||
+          memcmp(rec + hdr_len, body, sizeof(body) - 1) != 0)
+        err = 1; /* block is the raw payload, no HTTP envelope */
+      if (warc_memstr((char *) rec, "WARC-Target-URI: ftp://ftp.local/file.bin",
+                      hdr_len, 41) == NULL ||
+          warc_memstr((char *) rec, "Content-Type: application/octet-stream",
+                      hdr_len, 38) == NULL)
+        err = 1;
+    }
+    if (warc_memstr((char *) rec, "WARC-Type: response", hdr_len, 19) != NULL)
+      nresp++;
+    if (warc_memstr((char *) rec, "WARC-Type: request", hdr_len, 18) != NULL)
+      nreq++;
+    freet(rec);
+  }
+  freet(data);
+  if (nresource != 1 || nresp != 0 || nreq != 0)
+    err = 1;
+  printf("warc-ftp: resource=%d response=%d request=%d: %s\n", nresource, nresp,
+         nreq, err ? "FAIL" : "OK");
+  return err;
+}
+
+/* --warc-max-size rotates into <base>-00000.warc.gz, -00001, ...; each segment
+   is independently valid and begins with its own warcinfo. */
+static int st_warc_rotate(httrackp *opt, int argc, char **argv) {
+  char path[HTS_URLMAXSIZE];
+  char seg[HTS_URLMAXSIZE];
+  warc_writer *w;
+  LLint saved_max;
+  unsigned char body[600];
+  unsigned int rng = 0x12345678u;
+  int err = 0, nseg = 0, i;
+  size_t j;
+
+  if (argc < 1) {
+    fprintf(stderr, "warc-rotate: needs a writable directory\n");
+    return 1;
+  }
+  for (j = 0; j < sizeof(body);
+       j++) { /* incompressible: gzip can't shrink it */
+    rng ^= rng << 13;
+    rng ^= rng >> 17;
+    rng ^= rng << 5;
+    body[j] = (unsigned char) (rng >> 24);
+  }
+  fconcat(path, sizeof(path), argv[0], "warc-rot.warc.gz");
+  saved_max = opt->warc_max_size;
+  opt->warc_max_size =
+      1000; /* a couple records per segment => several segments */
+  w = warc_open(opt, path);
+  assertf(w != NULL);
+  for (i = 0; i < 8; i++) {
+    char uri[64];
+    snprintf(uri, sizeof(uri), "http://test.local/f%d.bin", i);
+    warc_write_transaction(
+        w, uri, "127.0.0.1", "GET / HTTP/1.1\r\nHost: test.local\r\n\r\n",
+        "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n\r\n",
+        (const char *) body, sizeof(body), NULL, 200, 0, 0);
+  }
+  warc_close(w);
+  opt->warc_max_size = saved_max;
+
+  for (i = 0;; i++) {
+    char fname[64];
+    unsigned char *data;
+    size_t data_len = 0;
+    const unsigned char *p, *pend;
+    int first = 1;
+    snprintf(fname, sizeof(fname), "warc-rot-%05d.warc.gz", i);
+    fconcat(seg, sizeof(seg), argv[0], fname);
+    data = warc_slurp(seg, &data_len);
+    if (data == NULL)
+      break; /* past the last segment */
+    nseg++;
+    p = data;
+    pend = data + data_len;
+    while (p < pend) {
+      size_t rlen = 0, hdr_len = 0;
+      long long block_len = 0;
+      unsigned char *rec = warc_next_member(&p, pend, &rlen);
+      if (rec == NULL) {
+        if (rlen != 0)
+          err = 1;
+        break;
+      }
+      if (warc_rec_split(rec, rlen, &hdr_len, &block_len) != 0)
+        err = 1;
+      if (first) { /* each segment leads with its own warcinfo */
+        if (warc_memstr((char *) rec, "WARC-Type: warcinfo", hdr_len, 19) ==
+            NULL)
+          err = 1;
+        first = 0;
+      }
+      freet(rec);
+    }
+    freet(data);
+    if (first) /* empty segment */
+      err = 1;
+  }
+  if (nseg < 2)
+    err = 1;
+  printf("warc-rotate: %d segments: %s\n", nseg, err ? "FAIL" : "OK");
+  return err;
+}
+
 /* ------------------------------------------------------------ */
 /* Registry: name -> handler, with a usage hint and a one-line description. */
 /* ------------------------------------------------------------ */
@@ -3252,6 +3798,8 @@ static const struct selftest_entry {
     {"idna-decode", "<host>", "decode an IDNA/punycode hostname",
      st_idna_decode},
     {"entities", "<string> [encoding]", "unescape HTML entities", st_entities},
+    {"footerfmt", "<template>", "-%F footer positional/named expansion",
+     st_footerfmt},
     {"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
      st_unescape_bounds},
     {"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
@@ -3327,6 +3875,14 @@ static const struct selftest_entry {
     {"ftp-line", "", "get_ftp_line bounds a hostile FTP reply line",
      st_ftpline},
     {"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser},
+    {"warc", "<dir>", "WARC/1.1 writer: framing, digests, revisit dedup",
+     st_warc},
+    {"warc-trunc", "<dir>", "WARC-Truncated on a cap-truncated body",
+     st_warc_trunc},
+    {"warc-ftp", "<dir>", "ftp resource record (no HTTP envelope)",
+     st_warc_ftp},
+    {"warc-rotate", "<dir>", "--warc-max-size segment rotation",
+     st_warc_rotate},
 };
 
 static void list_selftests(void) {

src/htstools.c

@@ -754,35 +754,48 @@ typedef struct hts_template_format_buf {
   size_t offset;
 } hts_template_format_buf;
 
+// Bounded append to a template buffer (or FILE); returns -1 on overflow/error.
+static int htsfmt_putc(hts_template_format_buf *buf, char c) {
+  if (buf->fp != NULL) {
+    assertf(buf->buffer == NULL);
+    if (fputc(c, buf->fp) < 0)
+      return -1;
+  } else {
+    assertf(buf->buffer != NULL);
+    if (buf->offset + 1 < buf->size)
+      buf->buffer[buf->offset++] = c;
+    else
+      return -1;
+  }
+  return 0;
+}
+
+static int htsfmt_puts(hts_template_format_buf *buf, const char *s) {
+  size_t i;
+  assertf(s != NULL);
+  for (i = 0; s[i] != '\0'; i++) {
+    if (htsfmt_putc(buf, s[i]) < 0)
+      return -1;
+  }
+  return 0;
+}
+
 // note: upstream arg list MUST be NULL-terminated for safety
 // returns a negative value upon error
-static int hts_template_formatv(hts_template_format_buf *buf, 
+static int hts_template_formatv(hts_template_format_buf *buf,
                                 const char *format, va_list args) {
 #undef FPUTC
 #undef FPUTS
-#define FPUTC(C) do { \
-  if (buf->fp != NULL) { \
-    assertf(buf->buffer == NULL); \
-    if (fputc(C, buf->fp) < 0) { \
-      return -1; \
-    } \
-  } else { \
-    assertf(buf->buffer != NULL); \
-    if (buf->offset + 1 < buf->size) { \
-      buf->buffer[buf->offset++] = (C); \
-    } else { \
-      return -1; \
-    } \
-  } \
-} while(0)
-#define FPUTS(S) do { \
-  size_t i; \
-  const char *const str_ = (S); \
-  assertf(str_ != NULL); \
-  for(i = 0 ; str_[i] != '\0' ; i++) { \
-    FPUTC(str_[i]); \
-  } \
-} while(0)
+#define FPUTC(C)                                                               \
+  do {                                                                         \
+    if (htsfmt_putc(buf, (C)) < 0)                                             \
+      return -1;                                                               \
+  } while (0)
+#define FPUTS(S)                                                               \
+  do {                                                                         \
+    if (htsfmt_puts(buf, (S)) < 0)                                             \
+      return -1;                                                               \
+  } while (0)
 
   if (buf != NULL && format != NULL) {
     const char *arg_expanded[32];
@@ -859,6 +872,74 @@ int hts_template_format_str(char *buffer, size_t size, const char *format, ...)
   return success;
 }
 
+// Value of the named field, or "" if absent (never NULL, so callers can pass it
+// straight to a formatter).
+static const char *footer_field_value(const hts_footer_field *fields,
+                                      size_t nfields, const char *name) {
+  size_t j;
+  for (j = 0; j < nfields; j++) {
+    if (strcmp(fields[j].name, name) == 0)
+      return fields[j].value != NULL ? fields[j].value : "";
+  }
+  return "";
+}
+
+int hts_footer_format(char *buffer, size_t size, const char *footer,
+                      const hts_footer_field *fields, size_t nfields) {
+  hts_template_format_buf buf = {NULL, buffer, size, 0};
+  size_t i;
+
+  if (footer == NULL || buffer == NULL || size == 0)
+    return -1;
+  // %s keeps the legacy positional model, byte-for-byte for existing -%F
+  // strings: addr, path, date, version, looked up by name and
+  // order-independent.
+  if (strstr(footer, "%s") != NULL)
+    return hts_template_format_str(
+        buffer, size, footer, footer_field_value(fields, nfields, "addr"),
+        footer_field_value(fields, nfields, "path"),
+        footer_field_value(fields, nfields, "date"),
+        footer_field_value(fields, nfields, "version"), /* EOF */ NULL);
+  // "{{"/"}}" emit a literal brace; an unknown "{...}" is left verbatim so
+  // typos stay visible.
+  for (i = 0; footer[i] != '\0'; i++) {
+    const char c = footer[i];
+    if (c == '{' && footer[i + 1] == '{') {
+      if (htsfmt_putc(&buf, '{') < 0)
+        return -1;
+      i++;
+    } else if (c == '}' && footer[i + 1] == '}') {
+      if (htsfmt_putc(&buf, '}') < 0)
+        return -1;
+      i++;
+    } else if (c == '{') {
+      const char *const end = strchr(footer + i + 1, '}');
+      int matched = 0;
+      if (end != NULL) {
+        const size_t namelen = (size_t) (end - (footer + i + 1));
+        size_t j;
+        for (j = 0; j < nfields; j++) {
+          if (strlen(fields[j].name) == namelen &&
+              strncmp(fields[j].name, footer + i + 1, namelen) == 0) {
+            if (htsfmt_puts(&buf,
+                            fields[j].value != NULL ? fields[j].value : "") < 0)
+              return -1;
+            i += namelen + 1; // consume the name and its closing '}'
+            matched = 1;
+            break;
+          }
+        }
+      }
+      if (!matched && htsfmt_putc(&buf, '{') < 0)
+        return -1;
+    } else if (htsfmt_putc(&buf, c) < 0) {
+      return -1;
+    }
+  }
+  buffer[buf.offset] = '\0';
+  return 1;
+}
+
 /* Note: NOT utf-8 */
 HTSEXT_API int hts_buildtopindex(httrackp * opt, const char *path,
                                  const char *binpath) {

src/htstools.h

@@ -73,6 +73,20 @@ HTS_INLINE int rech_tageq_all(const char *adr, const char *s);
 int hts_template_format(FILE *const out, const char *format, ...);
 int hts_template_format_str(char *buffer, size_t size, const char *format, ...);
 
+// A footer named field and its already-context-escaped value.
+typedef struct hts_footer_field {
+  const char *name;
+  const char *value;
+} hts_footer_field;
+
+// Expand a footer template. A "%s" in it selects the legacy positional model,
+// consuming the "addr"/"path"/"date"/"version" fields in that order; otherwise
+// "{name}" is substituted from fields by name ("{{"/"}}" emit a literal brace,
+// an unknown "{...}" is left verbatim). Values must already be escaped for the
+// target context by the caller. Returns <0 on overflow.
+int hts_footer_format(char *buffer, size_t size, const char *footer,
+                      const hts_footer_field *fields, size_t nfields);
+
 #define rech_tageq(adr,s) \
   ( \
     ( (*((adr)-1)=='<') || (is_space(*((adr)-1))) ) ? \

src/htswarc.c

@@ -0,0 +1,995 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack WARC/1.1 output writer (ISO 28500). See warc.h.
+   Strategy B (see design): the response record stores the decoded body and a
+   normalized header block (Content-Encoding/Transfer-Encoding stripped,
+   Content-Length rewritten to the decoded length). Valid, replayable WARC. */
+/* ------------------------------------------------------------ */
+
+#define HTS_INTERNAL_BYTECODE
+
+#include "htswarc.h"
+
+#include "htscore.h"
+#include "htslib.h"
+#include "htstools.h"
+#include "htssafe.h"
+#include "htszlib.h"
+#include "coucal/coucal.h"
+
+#include <stdarg.h>
+#include <stdint.h>
+#include <time.h>
+
+#if HTS_USEOPENSSL
+#include <openssl/evp.h>
+#include <openssl/rand.h>
+#endif
+
+/* opt->state.warc value meaning "open failed once, do not retry". */
+#define WARC_DISABLED ((void *) ~(uintptr_t) 0)
+
+struct warc_writer {
+  FILE *f;
+  int gz;          /* 1: one gzip member per record; 0: raw */
+  uint64_t offset; /* running byte offset (member starts, for a future index) */
+  uint64_t counter; /* monotonic record counter */
+  uint64_t rng;     /* PRNG state for the UUID fallback */
+  char info_id[64]; /* warcinfo WARC-Record-ID, referenced by every record */
+  coucal seen;      /* base32 payload digest -> "uri\001date" (revisit dedup) */
+  /* --warc-max-size rotation: NAME-00000.warc.gz, -00001, ... (wget-style). */
+  uint64_t max_size;   /* rotate once a segment reaches this; 0: single file */
+  char *seg_base;      /* segment path without the .warc[.gz] suffix, or NULL */
+  const char *seg_ext; /* ".warc.gz" or ".warc" */
+  unsigned seg;        /* current segment number */
+  char *info_fields;   /* warcinfo body, re-emitted at each new segment */
+};
+
+const char *warc_truncated_reason(int code) {
+  switch (code) {
+  case WARC_TRUNC_LENGTH:
+    return "length";
+  case WARC_TRUNC_TIME:
+    return "time";
+  case WARC_TRUNC_DISCONNECT:
+    return "disconnect";
+  default:
+    return NULL;
+  }
+}
+
+/* ---- growable byte buffer (overflow-safe, project allocators) ---- */
+
+typedef struct {
+  char *data;
+  size_t len;
+  size_t cap;
+} wbuf;
+
+static void wbuf_free(wbuf *b) {
+  freet(b->data);
+  b->len = b->cap = 0;
+}
+
+/* Append n bytes; returns 0 on success, -1 on OOM/overflow. */
+static int wbuf_add(wbuf *b, const void *p, size_t n) {
+  if (n > (size_t) -1 - b->len)
+    return -1;
+  if (b->len + n > b->cap) {
+    size_t ncap = b->cap ? b->cap : 256;
+    char *nd;
+    while (ncap < b->len + n) {
+      if (ncap > (size_t) -1 / 2)
+        return -1;
+      ncap *= 2;
+    }
+    nd = realloct(b->data, ncap);
+    if (nd == NULL)
+      return -1;
+    b->data = nd;
+    b->cap = ncap;
+  }
+  memcpy(b->data + b->len, p, n);
+  b->len += n;
+  return 0;
+}
+
+static int wbuf_puts(wbuf *b, const char *s) {
+  return wbuf_add(b, s, strlen(s));
+}
+
+static int wbuf_printf(wbuf *b, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
+
+static int wbuf_printf(wbuf *b, const char *fmt, ...) {
+  char tmp[1024];
+  int n;
+  va_list ap;
+  va_start(ap, fmt);
+  n = vsnprintf(tmp, sizeof(tmp), fmt, ap);
+  va_end(ap);
+  if (n < 0 || (size_t) n >= sizeof(tmp))
+    return -1;
+  return wbuf_add(b, tmp, (size_t) n);
+}
+
+/* ---- gzip-per-record member writer (mirrors ae_write_packed) ---- */
+
+typedef struct {
+  warc_writer *w;
+  z_stream strm;
+  int active; /* deflate stream initialized */
+} member;
+
+static int member_begin(member *m, warc_writer *w) {
+  m->w = w;
+  m->active = 0;
+  if (w->gz) {
+    memset(&m->strm, 0, sizeof(m->strm));
+    /* windowBits=31 => full RFC1952 gzip member */
+    if (deflateInit2(&m->strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, 31, 8,
+                     Z_DEFAULT_STRATEGY) != Z_OK)
+      return -1;
+    m->active = 1;
+  }
+  return 0;
+}
+
+static int member_write(member *m, const void *p, size_t n) {
+  if (!m->w->gz)
+    return (n == 0 || fwrite(p, 1, n, m->w->f) == n) ? 0 : -1;
+  m->strm.next_in = (const Bytef *) p;
+  while (n > 0) {
+    unsigned char out[8192];
+    size_t got;
+    uInt chunk = (n > (uInt) -1) ? (uInt) -1 : (uInt) n;
+    m->strm.avail_in = chunk;
+    do {
+      m->strm.next_out = out;
+      m->strm.avail_out = sizeof(out);
+      if (deflate(&m->strm, Z_NO_FLUSH) != Z_OK)
+        return -1;
+      got = sizeof(out) - m->strm.avail_out;
+      if (got > 0 && fwrite(out, 1, got, m->w->f) != got)
+        return -1;
+    } while (m->strm.avail_out == 0);
+    n -= chunk;
+  }
+  return 0;
+}
+
+static int member_end(member *m) {
+  int rc = 0;
+  if (m->active) {
+    unsigned char out[8192];
+    int zerr;
+    m->strm.avail_in = 0;
+    do {
+      m->strm.next_out = out;
+      m->strm.avail_out = sizeof(out);
+      zerr = deflate(&m->strm, Z_FINISH);
+      {
+        size_t got = sizeof(out) - m->strm.avail_out;
+        if (got > 0 && fwrite(out, 1, got, m->w->f) != got)
+          rc = -1;
+      }
+    } while (zerr == Z_OK);
+    if (zerr != Z_STREAM_END)
+      rc = -1;
+    deflateEnd(&m->strm);
+    m->active = 0;
+  }
+  return rc;
+}
+
+/* ---- SHA-1 + Base32 (digests are OpenSSL-only; omitted otherwise) ---- */
+
+#if HTS_USEOPENSSL
+/* Streaming SHA-1 over the block (all regions) and the payload (body only). */
+typedef struct {
+  EVP_MD_CTX *block;
+  EVP_MD_CTX *payload;
+} digester;
+
+static void base32_20(const unsigned char in[20], char out[33]) {
+  static const char a[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567";
+  int i, o = 0;
+  uint64_t buf = 0;
+  int bits = 0;
+  for (i = 0; i < 20; i++) {
+    buf = (buf << 8) | in[i];
+    bits += 8;
+    while (bits >= 5) {
+      bits -= 5;
+      out[o++] = a[(buf >> bits) & 0x1F];
+    }
+  }
+  out[o] = '\0'; /* 20 bytes => exactly 32 base32 chars, no padding */
+}
+#endif
+
+/* Stream a block to a sink. When http_section, the HTTP header bytes (may be
+   empty) plus their terminating CRLF are emitted first; the separator is bound
+   to http_section, not to http_hdr being non-NULL, so an empty header still
+   emits (and is counted in) the 2-byte separator (F3). The on-disk body is
+   written as EXACTLY body_len octets — capped if the file grew, zero-padded if
+   it shrank — so the declared Content-Length always equals the bytes written
+   across every pass (F2). region 0=header, 1=body. Returns 0 on success. */
+typedef int (*warc_sink)(void *ctx, int region, const void *p, size_t n);
+
+static int stream_body_pad(warc_sink sink, void *ctx, size_t remaining) {
+  static const char zeros[4096] = {0};
+  while (remaining > 0) {
+    size_t chunk = (remaining < sizeof(zeros)) ? remaining : sizeof(zeros);
+    if (sink(ctx, 1, zeros, chunk) != 0)
+      return -1;
+    remaining -= chunk;
+  }
+  return 0;
+}
+
+static int stream_block(int http_section, const char *http_hdr,
+                        size_t http_hdr_len, int has_body, const char *body,
+                        size_t body_len, const char *body_path, warc_sink sink,
+                        void *ctx) {
+  if (http_section) {
+    if (http_hdr != NULL && http_hdr_len > 0 &&
+        sink(ctx, 0, http_hdr, http_hdr_len) != 0)
+      return -1;
+    if (sink(ctx, 0, "\r\n", 2) != 0)
+      return -1;
+  }
+  if (has_body) {
+    if (body != NULL) {
+      if (body_len > 0 && sink(ctx, 1, body, body_len) != 0)
+        return -1;
+    } else if (body_path != NULL) {
+      char catbuff[CATBUFF_SIZE];
+      size_t remaining = body_len;
+      FILE *fp = FOPEN(fconv(catbuff, sizeof(catbuff), body_path), "rb");
+      if (fp == NULL)
+        return -1;
+      while (remaining > 0) {
+        char b[32768];
+        size_t want = (remaining < sizeof(b)) ? remaining : sizeof(b);
+        size_t nl = fread(b, 1, want, fp);
+        if (nl == 0)
+          break; /* short file: pad below so written == declared */
+        if (sink(ctx, 1, b, nl) != 0) {
+          fclose(fp);
+          return -1;
+        }
+        remaining -= nl;
+      }
+      fclose(fp);
+      if (stream_body_pad(sink, ctx, remaining) != 0)
+        return -1;
+    }
+  }
+  return 0;
+}
+
+#if HTS_USEOPENSSL
+static int digest_sink(void *ctx, int region, const void *p, size_t n) {
+  digester *d = (digester *) ctx;
+  if (d->block != NULL && EVP_DigestUpdate(d->block, p, n) != 1)
+    return -1;
+  if (region == 1 && d->payload != NULL &&
+      EVP_DigestUpdate(d->payload, p, n) != 1)
+    return -1;
+  return 0;
+}
+#endif
+
+static int write_sink(void *ctx, int region, const void *p, size_t n) {
+  (void) region;
+  return member_write((member *) ctx, p, n);
+}
+
+/* Base32 SHA-1 of a transaction payload (body only), for revisit dedup.
+   Returns 1 and fills out[33] on success, 0 without OpenSSL or on error. */
+static int payload_digest_b32(const char *body, size_t body_len,
+                              const char *body_path, char out[33]) {
+#if HTS_USEOPENSSL
+  digester d;
+  unsigned char md[EVP_MAX_MD_SIZE];
+  unsigned int mdlen = 0;
+  int ok;
+  d.block = NULL;
+  d.payload = EVP_MD_CTX_new();
+  if (d.payload == NULL)
+    return 0;
+  if (EVP_DigestInit_ex(d.payload, EVP_sha1(), NULL) != 1) {
+    EVP_MD_CTX_free(d.payload);
+    return 0;
+  }
+  ok = stream_block(0, NULL, 0, 1, body, body_len, body_path, digest_sink,
+                    &d) == 0 &&
+       EVP_DigestFinal_ex(d.payload, md, &mdlen) == 1 && mdlen == 20;
+  EVP_MD_CTX_free(d.payload);
+  if (!ok)
+    return 0;
+  base32_20(md, out);
+  return 1;
+#else
+  (void) body;
+  (void) body_len;
+  (void) body_path;
+  (void) out;
+  return 0;
+#endif
+}
+
+/* ---- misc record helpers ---- */
+
+static void warc_fill_random(warc_writer *w, unsigned char *b, size_t n) {
+  size_t i;
+#if HTS_USEOPENSSL
+  if (n <= (size_t) 0x7fffffff && RAND_bytes(b, (int) n) == 1)
+    return;
+#endif
+  for (i = 0; i < n; i++) {
+    w->rng ^= w->rng << 13;
+    w->rng ^= w->rng >> 7;
+    w->rng ^= w->rng << 17;
+    b[i] = (unsigned char) (w->rng >> 24);
+  }
+}
+
+/* urn:uuid: v4-shaped record id (uniqueness within the run is what matters). */
+static void warc_make_id(warc_writer *w, char out[64]) {
+  unsigned char b[16];
+  w->counter++;
+  warc_fill_random(w, b, sizeof(b));
+  b[6] = (unsigned char) ((b[6] & 0x0F) | 0x40);
+  b[8] = (unsigned char) ((b[8] & 0x3F) | 0x80);
+  snprintf(out, 64,
+           "<urn:uuid:%02x%02x%02x%02x-%02x%02x-%02x%02x-%02x%02x-"
+           "%02x%02x%02x%02x%02x%02x>",
+           b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7], b[8], b[9], b[10],
+           b[11], b[12], b[13], b[14], b[15]);
+}
+
+static void warc_now_iso8601(char out[32]) {
+  time_t t = time(NULL);
+  struct tm tmv;
+#if defined(_WIN32)
+  struct tm *g = gmtime(&t);
+  if (g != NULL)
+    tmv = *g;
+  else
+    memset(&tmv, 0, sizeof(tmv));
+#else
+  if (gmtime_r(&t, &tmv) == NULL)
+    memset(&tmv, 0, sizeof(tmv));
+#endif
+  strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
+}
+
+/* Case-insensitive "does line start with name:" test. */
+static int header_is(const char *line, size_t line_len, const char *name) {
+  size_t nl = strlen(name);
+  if (line_len < nl + 1)
+    return 0;
+  if (strncasecmp(line, name, nl) != 0)
+    return 0;
+  return line[nl] == ':';
+}
+
+/* Build the normalized HTTP header block from raw resp_hdr into out (no
+   trailing CRLF terminator). Always drops Transfer-Encoding (hop-by-hop).
+   When set_cl>=0, also drops Content-Encoding and the original Content-Length
+   and appends "Content-Length: <set_cl>". Returns 0 on success. */
+static int normalize_http_headers(const char *resp_hdr, long long set_cl,
+                                  wbuf *out) {
+  const char *p = resp_hdr;
+  int first = 1;
+  if (resp_hdr == NULL)
+    return -1;
+  while (*p != '\0') {
+    const char *eol = strchr(p, '\n');
+    size_t len = (eol != NULL) ? (size_t) (eol - p) : strlen(p);
+    size_t raw = len;
+    if (len > 0 && p[len - 1] == '\r')
+      len--; /* strip CR; re-added as CRLF below */
+    if (len == 0)
+      break; /* blank line: end of headers */
+    if (first) {
+      first = 0; /* status line: keep verbatim */
+    } else if (header_is(p, len, "Transfer-Encoding")) {
+      goto next;
+    } else if (set_cl >= 0 && (header_is(p, len, "Content-Encoding") ||
+                               header_is(p, len, "Content-Length"))) {
+      goto next;
+    }
+    if (wbuf_add(out, p, len) != 0 || wbuf_add(out, "\r\n", 2) != 0)
+      return -1;
+  next:
+    if (eol == NULL)
+      break;
+    p = eol + 1;
+    (void) raw;
+  }
+  if (set_cl >= 0 && wbuf_printf(out, "Content-Length: %lld\r\n", set_cl) != 0)
+    return -1;
+  return 0;
+}
+
+/* Close the current segment and open the next; writes its warcinfo. */
+static int warc_rotate(warc_writer *w);
+
+/* Emit one full WARC record. When http_section, the block carries an HTTP
+   header block (http_hdr, possibly empty) + a CRLF separator; body follows when
+   has_body. block_len is derived here (single source of truth: separator and
+   payload are counted exactly as stream_block emits them), so a declared
+   Content-Length can never desync from the written bytes. The payload digest
+   (body-only) is passed in when already known. truncated is a WARC-Truncated
+   reason token or NULL. On success the record id is copied to out_id (may be
+   NULL). */
+static int warc_emit(warc_writer *w, const char *type, const char *content_type,
+                     const char *target_uri, const char *ip,
+                     const char *concurrent_to, const char *refers_uri,
+                     const char *refers_date, const char *profile,
+                     const char *payload_digest, const char *truncated,
+                     int http_section, const char *http_hdr,
+                     size_t http_hdr_len, int has_body, const char *body,
+                     size_t body_len, const char *body_path, char out_id[64]) {
+  wbuf hdr;
+  member m;
+  char id[64], date[32];
+  size_t sep = http_section ? 2 : 0;
+  size_t payload = has_body ? body_len : 0;
+  size_t block_len;
+  int rc = -1;
+#if HTS_USEOPENSSL
+  digester d;
+  unsigned char md[EVP_MAX_MD_SIZE];
+  unsigned int mdlen = 0;
+  char block_b32[33];
+  int have_block_digest = 0;
+#endif
+
+  /* Rotate to the next segment before this record when the current one is full;
+     never split a record, and never rotate a warcinfo (it opens a segment). */
+  if (w->max_size > 0 && w->seg_base != NULL && w->offset >= w->max_size &&
+      strcmp(type, "warcinfo") != 0) {
+    if (warc_rotate(w) != 0)
+      return -1;
+  }
+
+  /* F4: overflow-safe block length; http_hdr_len+sep is provably small. */
+  if (payload > (size_t) -1 - http_hdr_len - sep)
+    return -1;
+  block_len = http_hdr_len + sep + payload;
+
+  memset(&hdr, 0, sizeof(hdr));
+  warc_make_id(w, id);
+  warc_now_iso8601(date);
+
+#if HTS_USEOPENSSL
+  /* Block digest over the whole block, in one streaming pass. */
+  d.block = EVP_MD_CTX_new();
+  d.payload = NULL;
+  if (d.block != NULL && EVP_DigestInit_ex(d.block, EVP_sha1(), NULL) == 1 &&
+      stream_block(http_section, http_hdr, http_hdr_len, has_body, body,
+                   body_len, body_path, digest_sink, &d) == 0 &&
+      EVP_DigestFinal_ex(d.block, md, &mdlen) == 1 && mdlen == 20) {
+    base32_20(md, block_b32);
+    have_block_digest = 1;
+  }
+  if (d.block != NULL)
+    EVP_MD_CTX_free(d.block);
+#endif
+
+  if (wbuf_puts(&hdr, "WARC/1.1\r\n") != 0 ||
+      wbuf_printf(&hdr, "WARC-Type: %s\r\n", type) != 0 ||
+      wbuf_printf(&hdr, "WARC-Record-ID: %s\r\n", id) != 0 ||
+      wbuf_printf(&hdr, "WARC-Date: %s\r\n", date) != 0)
+    goto done;
+  if (content_type != NULL &&
+      wbuf_printf(&hdr, "Content-Type: %s\r\n", content_type) != 0)
+    goto done;
+  if (wbuf_printf(&hdr, "Content-Length: %llu\r\n",
+                  (unsigned long long) block_len) != 0)
+    goto done;
+  if (w->info_id[0] != '\0' && strcmp(type, "warcinfo") != 0 &&
+      wbuf_printf(&hdr, "WARC-Warcinfo-ID: %s\r\n", w->info_id) != 0)
+    goto done;
+  if (target_uri != NULL && target_uri[0] != '\0' &&
+      wbuf_printf(&hdr, "WARC-Target-URI: %s\r\n", target_uri) != 0)
+    goto done;
+  if (ip != NULL && ip[0] != '\0' &&
+      wbuf_printf(&hdr, "WARC-IP-Address: %s\r\n", ip) != 0)
+    goto done;
+  if (concurrent_to != NULL && concurrent_to[0] != '\0' &&
+      wbuf_printf(&hdr, "WARC-Concurrent-To: %s\r\n", concurrent_to) != 0)
+    goto done;
+  if (profile != NULL &&
+      wbuf_printf(&hdr, "WARC-Profile: %s\r\n", profile) != 0)
+    goto done;
+  if (refers_uri != NULL && refers_uri[0] != '\0' &&
+      wbuf_printf(&hdr, "WARC-Refers-To-Target-URI: %s\r\n", refers_uri) != 0)
+    goto done;
+  if (refers_date != NULL && refers_date[0] != '\0' &&
+      wbuf_printf(&hdr, "WARC-Refers-To-Date: %s\r\n", refers_date) != 0)
+    goto done;
+#if HTS_USEOPENSSL
+  if (have_block_digest &&
+      wbuf_printf(&hdr, "WARC-Block-Digest: sha1:%s\r\n", block_b32) != 0)
+    goto done;
+#endif
+  if (payload_digest != NULL && payload_digest[0] != '\0' &&
+      wbuf_printf(&hdr, "WARC-Payload-Digest: sha1:%s\r\n", payload_digest) !=
+          0)
+    goto done;
+  if (truncated != NULL &&
+      wbuf_printf(&hdr, "WARC-Truncated: %s\r\n", truncated) != 0)
+    goto done;
+  if (wbuf_puts(&hdr, "\r\n") != 0)
+    goto done;
+
+  if (member_begin(&m, w) != 0)
+    goto done;
+  if (member_write(&m, hdr.data, hdr.len) != 0 ||
+      stream_block(http_section, http_hdr, http_hdr_len, has_body, body,
+                   body_len, body_path, write_sink, &m) != 0 ||
+      member_write(&m, "\r\n\r\n", 4) != 0) {
+    member_end(&m);
+    goto done;
+  }
+  if (member_end(&m) != 0)
+    goto done;
+
+  {
+    long pos = ftell(w->f);
+    if (pos >= 0)
+      w->offset = (uint64_t) pos;
+  }
+  if (out_id != NULL)
+    strlcpybuff(out_id, id, 64);
+  rc = 0;
+done:
+  wbuf_free(&hdr);
+  return rc;
+}
+
+/* ---- segment rotation (--warc-max-size) ---- */
+
+/* Emit the warcinfo that heads a segment; sets w->info_id for its records. */
+static int warc_write_warcinfo_record(warc_writer *w) {
+  w->info_id[0] = '\0'; /* warcinfo itself carries no WARC-Warcinfo-ID */
+  return warc_emit(w, "warcinfo", "application/warc-fields", NULL, NULL, NULL,
+                   NULL, NULL, NULL, NULL, NULL, 0, NULL, 0, 1, w->info_fields,
+                   w->info_fields != NULL ? strlen(w->info_fields) : 0, NULL,
+                   w->info_id);
+}
+
+static int warc_rotate(warc_writer *w) {
+  char namebuf[HTS_URLMAXSIZE * 2];
+  char catbuff[CATBUFF_SIZE];
+  if (w->f != NULL) {
+    fclose(w->f);
+    w->f = NULL;
+  }
+  w->seg++;
+  snprintf(namebuf, sizeof(namebuf), "%s-%05u%s", w->seg_base, w->seg,
+           w->seg_ext);
+  w->f = FOPEN(fconv(catbuff, sizeof(catbuff), namebuf), "wb");
+  if (w->f == NULL)
+    return -1;
+  w->offset = 0;
+  return warc_write_warcinfo_record(w);
+}
+
+/* ---- request stash (engine hooks) ---- */
+
+void warc_stash_request(htsblk *r, const char *reqhdr) {
+  if (r == NULL)
+    return;
+  freet(r->warc_reqhdr);
+  if (reqhdr != NULL)
+    r->warc_reqhdr = strdupt(reqhdr);
+}
+
+void warc_stash_response(htsblk *r, const char *resphdr) {
+  if (r == NULL)
+    return;
+  freet(r->warc_resphdr);
+  if (resphdr != NULL)
+    r->warc_resphdr = strdupt(resphdr);
+}
+
+void warc_free_request(htsblk *r) {
+  if (r != NULL) {
+    freet(r->warc_reqhdr);
+    freet(r->warc_resphdr);
+  }
+}
+
+/* ---- open / close ---- */
+
+warc_writer *warc_open(httrackp *opt, const char *path) {
+  warc_writer *w;
+  char namebuf[HTS_URLMAXSIZE * 2];
+  char catbuff[CATBUFF_SIZE];
+  wbuf info;
+  const char *robots;
+  size_t plen;
+
+  if (path == NULL)
+    return NULL;
+
+  /* --warc with no name: <output>/httrack-<timestamp>.warc.gz */
+  if (strcmp(path, WARC_AUTONAME) == 0) {
+    char ts[32];
+    time_t t = time(NULL);
+    struct tm tmv;
+#if defined(_WIN32)
+    struct tm *g = gmtime(&t);
+    if (g != NULL)
+      tmv = *g;
+    else
+      memset(&tmv, 0, sizeof(tmv));
+#else
+    if (gmtime_r(&t, &tmv) == NULL)
+      memset(&tmv, 0, sizeof(tmv));
+#endif
+    strftime(ts, sizeof(ts), "%Y%m%d%H%M%S", &tmv);
+    snprintf(catbuff, sizeof(catbuff), "httrack-%s.warc.gz", ts);
+    path =
+        fconcat(namebuf, sizeof(namebuf), StringBuff(opt->path_html), catbuff);
+  } else {
+    /* --warc-file NAME: append .warc.gz unless already a .warc/.warc.gz name;
+       place bare basenames under the output directory (like the auto name). */
+    size_t l = strlen(path);
+    int has_warc = (l >= 5 && strcasecmp(path + l - 5, ".warc") == 0);
+    int has_gz = (l >= 3 && strcasecmp(path + l - 3, ".gz") == 0);
+    char named[HTS_URLMAXSIZE];
+    if (has_warc || has_gz)
+      strlcpybuff(named, path, sizeof(named));
+    else
+      snprintf(named, sizeof(named), "%s.warc.gz", path);
+    if (strchr(named, '/') == NULL && strchr(named, '\\') == NULL) {
+      path =
+          fconcat(namebuf, sizeof(namebuf), StringBuff(opt->path_html), named);
+    } else {
+      strlcpybuff(namebuf, named, sizeof(namebuf));
+      path = namebuf;
+    }
+  }
+
+  w = calloct(1, sizeof(*w));
+  if (w == NULL)
+    return NULL;
+  plen = strlen(path);
+  w->gz = (plen >= 3 && strcasecmp(path + plen - 3, ".gz") == 0);
+  w->rng = (uint64_t) time(NULL) ^ ((uint64_t) (uintptr_t) w << 16) ^
+           0x9e3779b97f4a7c15ULL;
+  w->seen = coucal_new(0);
+  if (w->seen != NULL)
+    coucal_value_is_malloc(w->seen, 1);
+  w->max_size = (opt->warc_max_size > 0) ? (uint64_t) opt->warc_max_size : 0;
+
+  /* Build the warcinfo body once; each segment re-emits it. */
+  robots = (opt->robots == HTS_ROBOTS_NEVER) ? "ignore" : "obey";
+  memset(&info, 0, sizeof(info));
+  if (wbuf_printf(&info,
+                  "software: HTTrack/%s (+https://www.httrack.com/)\r\n"
+                  "format: WARC file version 1.1\r\n"
+                  "conformsTo: http://iipc.github.io/warc-specifications/"
+                  "specifications/warc-format/warc-1.1/\r\n"
+                  "robots: %s\r\n",
+                  HTTRACK_VERSION, robots) != 0 ||
+      (StringNotEmpty(opt->path_html) &&
+       wbuf_printf(&info, "isPartOf: %s\r\n", StringBuff(opt->path_html)) !=
+           0) ||
+      wbuf_add(&info, "", 1) != 0) { /* NUL-terminate for info_fields */
+    wbuf_free(&info);
+    warc_close(w);
+    return NULL;
+  }
+  w->info_fields = strdupt(info.data);
+  wbuf_free(&info);
+  if (w->info_fields == NULL) {
+    warc_close(w);
+    return NULL;
+  }
+
+  /* Rotation on: the first segment is <base>-00000<ext> (wget-style); split the
+     resolved path into base + suffix so later segments reuse the base. */
+  if (w->max_size > 0) {
+    size_t l = strlen(path);
+    size_t baselen;
+    if (l >= 8 && strcasecmp(path + l - 8, ".warc.gz") == 0) {
+      baselen = l - 8;
+      w->seg_ext = ".warc.gz";
+    } else if (l >= 5 && strcasecmp(path + l - 5, ".warc") == 0) {
+      baselen = l - 5;
+      w->seg_ext = ".warc";
+    } else {
+      baselen = l;
+      w->seg_ext = w->gz ? ".warc.gz" : ".warc";
+    }
+    w->seg_base = malloct(baselen + 1);
+    if (w->seg_base == NULL) {
+      warc_close(w);
+      return NULL;
+    }
+    memcpy(w->seg_base, path, baselen);
+    w->seg_base[baselen] = '\0';
+    w->seg = 0;
+    snprintf(namebuf, sizeof(namebuf), "%s-%05u%s", w->seg_base, w->seg,
+             w->seg_ext);
+    path = namebuf;
+  }
+
+  w->f = FOPEN(fconv(catbuff, sizeof(catbuff), path), "wb");
+  if (w->f == NULL) {
+    warc_close(w);
+    return NULL;
+  }
+
+  if (warc_write_warcinfo_record(w) != 0) {
+    warc_close(w);
+    return NULL;
+  }
+  return w;
+}
+
+void warc_close(warc_writer *w) {
+  if (w == NULL)
+    return;
+  if (w->f != NULL)
+    fclose(w->f);
+  if (w->seen != NULL)
+    coucal_delete(&w->seen);
+  freet(w->seg_base);
+  freet(w->info_fields);
+  freet(w);
+}
+
+void warc_close_opt(httrackp *opt) {
+  if (opt->state.warc != NULL && opt->state.warc != WARC_DISABLED) {
+    warc_close((warc_writer *) opt->state.warc);
+  }
+  opt->state.warc = NULL;
+}
+
+/* ---- one transaction ---- */
+
+int warc_write_transaction(warc_writer *w, const char *target_uri,
+                           const char *ip, const char *req_hdr,
+                           const char *resp_hdr, const char *body,
+                           size_t body_len, const char *body_path,
+                           int statuscode, int is_update_unchanged,
+                           int truncated) {
+  wbuf http;
+  char resp_id[64];
+  char pdig[33];
+  int have_pdig;
+  int is_revisit = 0;
+  const char *profile = NULL;
+  const char *refers_uri = NULL;
+  const char *refers_date = NULL;
+  char refers_buf[HTS_URLMAXSIZE * 2 + 64];
+  int has_payload;
+  int emit_body;
+  int rc = -1;
+
+  if (resp_hdr == NULL)
+    return -1;
+
+  /* A payload exists (for digesting) unless this is a bodyless 304. */
+  has_payload = (body_len > 0 && (body != NULL || body_path != NULL) &&
+                 !is_update_unchanged);
+
+  /* Payload digest drives identical-payload-digest dedup (OpenSSL only). */
+  have_pdig =
+      has_payload ? payload_digest_b32(body, body_len, body_path, pdig) : 0;
+
+  if (is_update_unchanged) {
+    is_revisit = 1;
+    profile = "http://netpreserve.org/warc/1.1/revisit/server-not-modified";
+  } else if (have_pdig && w->seen != NULL) {
+    void *prev = NULL;
+    if (coucal_read_pvoid(w->seen, pdig, &prev) && prev != NULL) {
+      char *slot = (char *) prev;
+      char *sep = strchr(slot, '\001');
+      is_revisit = 1;
+      profile =
+          "http://netpreserve.org/warc/1.1/revisit/identical-payload-digest";
+      if (sep != NULL) {
+        size_t n = (size_t) (sep - slot);
+        if (n < sizeof(refers_buf)) {
+          memcpy(refers_buf, slot, n);
+          refers_buf[n] = '\0';
+          refers_uri = refers_buf;
+          refers_date = sep + 1;
+        }
+      }
+    }
+  }
+
+  /* Both revisit kinds (server-304 and identical-payload-digest) are bodyless;
+     only a full response carries the payload (F1). */
+  emit_body = has_payload && !is_revisit;
+
+  /* Normalize headers: full response rewrites Content-Length to the decoded
+     body length and strips Content-Encoding; a revisit keeps them (no body). */
+  memset(&http, 0, sizeof(http));
+  if (normalize_http_headers(resp_hdr, emit_body ? (long long) body_len : -1,
+                             &http) != 0) {
+    wbuf_free(&http);
+    return -1;
+  }
+
+  /* Response first: its id links the request via WARC-Concurrent-To. A revisit
+     is a deliberate dedup, not a truncation, so tag WARC-Truncated only on a
+     full (body-carrying) response. */
+  resp_id[0] = '\0';
+  if (warc_emit(w, is_revisit ? "revisit" : "response",
+                "application/http;msgtype=response", target_uri, ip, NULL,
+                refers_uri, refers_date, profile, have_pdig ? pdig : NULL,
+                emit_body ? warc_truncated_reason(truncated) : NULL, 1,
+                http.data, http.len, emit_body, body, body_len, body_path,
+                resp_id) != 0) {
+    wbuf_free(&http);
+    return -1;
+  }
+  wbuf_free(&http);
+
+  if (req_hdr != NULL && req_hdr[0] != '\0') {
+    size_t rlen = strlen(req_hdr);
+    if (warc_emit(w, "request", "application/http;msgtype=request", target_uri,
+                  NULL, resp_id, NULL, NULL, NULL, NULL, NULL, 0, NULL, 0, 1,
+                  req_hdr, rlen, NULL, NULL) != 0)
+      return -1;
+  }
+
+  /* Record this payload for later identical-payload-digest revisits. */
+  if (!is_revisit && have_pdig && w->seen != NULL && target_uri != NULL) {
+    char date[32];
+    char *slot;
+    size_t need;
+    warc_now_iso8601(date);
+    need = strlen(target_uri) + 1 + strlen(date) + 1;
+    slot = malloct(need);
+    if (slot != NULL) {
+      snprintf(slot, need, "%s\001%s", target_uri, date);
+      if (coucal_write_pvoid(w->seen, pdig, slot) == 0) {
+        /* replaced an existing entry: coucal freed the old value */
+      }
+    }
+  }
+
+  (void) statuscode;
+  rc = 0;
+  return rc;
+}
+
+/* ---- one non-HTTP capture ---- */
+
+int warc_write_resource(warc_writer *w, const char *target_uri, const char *ip,
+                        const char *content_type, const char *body,
+                        size_t body_len, const char *body_path, int truncated) {
+  char pdig[33];
+  int has_body = (body_len > 0 && (body != NULL || body_path != NULL));
+  int have_pdig =
+      has_body ? payload_digest_b32(body, body_len, body_path, pdig) : 0;
+  /* resource: the block is the raw payload, its own MIME is the record's
+     Content-Type, and there is no HTTP request/response envelope. */
+  return warc_emit(w, "resource",
+                   (content_type != NULL && content_type[0] != '\0')
+                       ? content_type
+                       : "application/octet-stream",
+                   target_uri, ip, NULL, NULL, NULL, NULL,
+                   have_pdig ? pdig : NULL, warc_truncated_reason(truncated), 0,
+                   NULL, 0, has_body, body, body_len, body_path, NULL);
+}
+
+/* ---- engine emit hook ---- */
+
+void warc_write_backtransaction(httrackp *opt, lien_back *back) {
+  warc_writer *w;
+  char uri[HTS_URLMAXSIZE * 4 + 16];
+  char ip[128];
+  const char *body;
+  size_t body_len;
+  const char *body_path;
+  const char *resp_hdr;
+  char synth[512];
+  int is_unchanged;
+  int is_ftp;
+
+  if (opt->state.warc == WARC_DISABLED)
+    return;
+  if (opt->state.warc == NULL) {
+    w = warc_open(opt, StringBuff(opt->warc_file));
+    if (w == NULL) {
+      opt->state.warc = WARC_DISABLED;
+      hts_log_print(opt, LOG_ERROR, "could not create WARC archive %s",
+                    StringBuff(opt->warc_file));
+      return;
+    }
+    opt->state.warc = w;
+  }
+  w = (warc_writer *) opt->state.warc;
+
+  if (back->r.statuscode <= 0)
+    return;
+
+  is_ftp = strfield(back->url_adr, "ftp://") != 0;
+  snprintf(uri, sizeof(uri), "%s%s%s",
+           link_has_authority(back->url_adr) ? "" : "http://", back->url_adr,
+           back->url_fil);
+
+  ip[0] = '\0';
+  SOCaddr_inetntoa(ip, sizeof(ip), back->r.address);
+
+  if (!back->r.is_write) {
+    body = back->r.adr;
+    body_len = (back->r.size > 0) ? (size_t) back->r.size : 0;
+    body_path = NULL;
+  } else {
+    LLint fs;
+    body = NULL;
+    body_path = back->url_sav;
+    fs = fsize_utf8(body_path);
+    /* F4: an on-disk body past size_t (LLP32/ILP32, >4GB) would wrap the length
+       used as Content-Length; drop the body rather than desync the record. */
+    if (fs > 0 && (uint64_t) fs <= (uint64_t) (size_t) -1)
+      body_len = (size_t) fs;
+    else
+      body_len = 0;
+  }
+
+  /* FTP has no HTTP envelope: one resource record carrying the payload. */
+  if (is_ftp) {
+    warc_write_resource(w, uri, ip, back->r.contenttype, body, body_len,
+                        body_path, back->r.warc_truncated);
+    return;
+  }
+
+  is_unchanged = (back->r.notmodified && opt->is_update) ? 1 : 0;
+
+  /* Prefer the stashed raw headers; synthesize a minimal status line for the
+     header-less (HTTP/0.9-style) responses that never carried a header block.
+   */
+  resp_hdr = back->r.warc_resphdr;
+  if (resp_hdr == NULL) {
+    snprintf(synth, sizeof(synth), "HTTP/1.1 %d %s\r\nContent-Type: %s\r\n\r\n",
+             back->r.statuscode, back->r.msg[0] ? back->r.msg : "OK",
+             back->r.contenttype[0] ? back->r.contenttype
+                                    : "application/octet-stream");
+    resp_hdr = synth;
+  }
+
+  warc_write_transaction(w, uri, ip, back->r.warc_reqhdr, resp_hdr, body,
+                         body_len, body_path, back->r.statuscode, is_unchanged,
+                         back->r.warc_truncated);
+}

src/htswarc.h

@@ -0,0 +1,118 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack WARC/1.1 output writer (ISO 28500). Internal, not installed.
+   All WARC record serialization, gzip-member framing, digests, UUID and
+   revisit dedup live here; the engine only stashes the request, frees it,
+   and calls one entry point per finished transaction. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_WARC_DEFH
+#define HTS_WARC_DEFH
+
+#include "htsopt.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* opt->warc_file sentinel: --warc with no argument => auto-name the archive
+   under the project's output directory at open time. */
+#define WARC_AUTONAME "\001auto"
+
+/* htsblk.warc_truncated / WARC-Truncated reason tokens (ISO 28500 sec 5.13).
+   A cap-truncated body is still archived, tagged with why it was cut short. */
+#define WARC_TRUNC_NONE 0
+#define WARC_TRUNC_LENGTH 1     /* hit a size cap (-M mirror / -m per-file) */
+#define WARC_TRUNC_TIME 2       /* hit the mirror time cap (-E/--max-time) */
+#define WARC_TRUNC_DISCONNECT 3 /* connection dropped mid-body */
+
+/* WARC-Truncated token for a warc_truncated code, or NULL for none. */
+const char *warc_truncated_reason(int code);
+
+typedef struct warc_writer warc_writer;
+
+/* Stash the raw request header block (bstr.buffer) on r for the later WARC
+   request record; frees any prior stash. No-op when reqhdr is NULL. */
+void warc_stash_request(htsblk *r, const char *reqhdr);
+
+/* Stash the raw response header block: deletehttp frees r.headers when the
+   socket closes, before back_finalize, so keep a WARC-owned copy. */
+void warc_stash_response(htsblk *r, const char *resphdr);
+
+/* Free both stashed header blocks (idempotent, NULL-safe). */
+void warc_free_request(htsblk *r);
+
+/* Emit the request + response (or revisit) records for one finished
+   transaction. Lazily opens the writer into opt->state.warc; a no-op (logged
+   once) if the archive cannot be created. */
+void warc_write_backtransaction(httrackp *opt, lien_back *back);
+
+/* Close and free the writer held in opt->state.warc, if any. */
+void warc_close_opt(httrackp *opt);
+
+/* --- Direct writer API (used by the hooks above and the self-test). --- */
+
+/* Create the archive at path (auto-named when path is WARC_AUTONAME), writing
+   the warcinfo record. .warc.gz => one gzip member per record; .warc => raw.
+   Returns NULL on failure. */
+warc_writer *warc_open(httrackp *opt, const char *path);
+
+/* Flush, close and free the writer (NULL-safe). */
+void warc_close(warc_writer *w);
+
+/* Write one transaction's request + response (or revisit) records.
+   target_uri:  absolute URL fetched.
+   ip:          numeric peer IP, or NULL/"" to omit.
+   req_hdr:     exact request header block sent, or NULL to skip the request.
+   resp_hdr:    raw received response header block (status line + headers).
+   body/body_len: decoded in-memory body, or NULL when on disk.
+   body_path:   file re-read for the body when body==NULL (may be NULL).
+   is_update_unchanged: nonzero for a 304 server-not-modified revisit.
+   truncated: a WARC_TRUNC_* reason to tag a cap-truncated body, else 0.
+   Returns 0 on success, -1 on error. */
+int warc_write_transaction(warc_writer *w, const char *target_uri,
+                           const char *ip, const char *req_hdr,
+                           const char *resp_hdr, const char *body,
+                           size_t body_len, const char *body_path,
+                           int statuscode, int is_update_unchanged,
+                           int truncated);
+
+/* Write one non-HTTP capture as a single WARC 'resource' record: the block is
+   the raw payload (no HTTP envelope), Content-Type is the payload's own MIME.
+   Used for ftp:// transfers. truncated is a WARC_TRUNC_* reason or 0.
+   Returns 0 on success, -1 on error. */
+int warc_write_resource(warc_writer *w, const char *target_uri, const char *ip,
+                        const char *content_type, const char *body,
+                        size_t body_len, const char *body_path, int truncated);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/libhttrack.vcxproj

@@ -135,6 +135,7 @@
     <ClCompile Include="htswizard.c" />
     <ClCompile Include="htswrap.c" />
     <ClCompile Include="htszlib.c" />
+    <ClCompile Include="htswarc.c" />
     <ClCompile Include="md5.c" />
     <ClCompile Include="minizip\ioapi.c" />
     <ClCompile Include="minizip\iowin32.c" />

src/minizip/mztools.c.diff

@@ -1,6 +1,6 @@
 --- mztools.c.orig	2024-01-27 14:07:18.636193212 +0100
 +++ mztools.c	2024-01-27 14:09:55.356620093 +0100
-@@ -10,6 +10,7 @@
+@@ -10,10 +10,11 @@
  #include <string.h>
  #include "zlib.h"
  #include "unzip.h"
@@ -8,6 +8,11 @@
  
  #define READ_8(adr)  ((unsigned char)*(adr))
  #define READ_16(adr) ( READ_8(adr) | (READ_8(adr+1) << 8) )
+-#define READ_32(adr) ( READ_16(adr) | (READ_16((adr)+2) << 16) )
++#define READ_32(adr) ((uLong) READ_16(adr) | ((uLong) READ_16((adr) + 2) << 16))
+ 
+ #define WRITE_8(buff, n) do { \
+   *((unsigned char*)(buff)) = (unsigned char) ((n) & 0xff); \
 @@ -141,8 +142,8 @@
          /* Central directory entry */
          {

src/vcpkg.json

@@ -1,12 +1,12 @@
 {
-  "$schema":
-      "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
+  "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
   "name": "libhttrack",
   "version-string": "3.49",
+  "builtin-baseline": "cd61e1e26a038e82d6550a3ebbe0fbbfe7da78e3",
   "dependencies": [
     "brotli",
     "openssl",
     "zlib",
     "zstd"
   ]
-}
+}

tests/01_engine-cmdline.test

@@ -155,4 +155,22 @@ accepted "$tmp/pre-bti" "#615: -%i before the URL broke the crawl"
 run_only "$tmp/pre-proto" "-@i2" "file://$tmp/index.html"
 accepted "$tmp/pre-proto" "#615: -@i2 before the URL broke the crawl"
 
+# -K must not reset the -c socket count (its case fell through into 'c'). maxsoc
+# is only observable via the ">8, limited to 8" warning: -c16 trips it, and would
+# stop tripping it if a trailing -K reset the count back to the default 4.
+warned() {
+    grep -q 'simultaneous connections limited to' "$1/hts-log.txt" ||
+        ! echo "FAIL: $2" || exit 1
+}
+not_warned() {
+    ! grep -q 'simultaneous connections limited to' "$1/hts-log.txt" ||
+        ! echo "FAIL: $2" || exit 1
+}
+run "$tmp/soc-c16" -c16
+warned "$tmp/soc-c16" "-c16 did not trip the socket-count warning (probe blind)"
+run "$tmp/soc-c16k" -c16 -K
+warned "$tmp/soc-c16k" "-K reset the -c socket count (fell through into 'c')"
+run "$tmp/soc-c8k" -c8 -K
+not_warned "$tmp/soc-c8k" "-c8 -K spuriously warned"
+
 exit 0

tests/01_engine-footer-overflow.test

@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Keep this POSIX-portable: the harness runs it via $(BASH), which is a plain
+# POSIX /bin/sh on some platforms (e.g. macOS), so avoid bashisms despite the
+# #!/bin/bash above.
+
+# A -%F footer whose expansion overflows the on-page buffer must be dropped, not
+# crash the crawl. Before the fix the unchecked hts_footer_format return left the
+# buffer unterminated and the next strcatbuff aborted (SIGABRT).
+
+set -eu
+
+# The overflow needs a path longer than Windows MAX_PATH (260), so both the
+# source tree and the mirror output blow past it there; run on POSIX only. The
+# fix itself is platform-independent and covered on Linux/macOS.
+case "$(uname -s 2>/dev/null)" in
+MINGW* | MSYS* | CYGWIN*) exit 77 ;;
+esac
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+# A few-hundred-char file path (kept well under the URL length limit) makes the
+# {path} field long.
+seg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+deep="$dir"
+i=0
+while [ "$i" -lt 6 ]; do
+    deep="$deep/$seg"
+    i=$((i + 1))
+done
+mkdir -p "$deep"
+printf '<html><body>hi</body></html>' >"$deep/index.html"
+
+# 40 {path} references (~240 chars, under the 254-char -%F cap); expanded against
+# a few-hundred-char path this far exceeds the per-page footer buffer.
+footer=""
+i=0
+while [ "$i" -lt 40 ]; do
+    footer="$footer{path}"
+    i=$((i + 1))
+done
+
+mir="$dir/mir"
+httrack "file://$deep/index.html" -O "$mir" -%F "$footer" -q -s0 -%v0 \
+    >/dev/null 2>&1 || {
+    echo "crawl failed/aborted (exit $?) on an oversized footer" >&2
+    exit 1
+}
+
+# The crawled page must exist (proves the URL wasn't rejected for length, so the
+# footer path ran). Look under file/, not $mir, to skip the makeindex top index.
+find "$mir/file" -name index.html | grep -q . || {
+    echo "page not mirrored; the oversized-footer path was not exercised" >&2
+    exit 1
+}

tests/01_engine-footerfmt.test

@@ -0,0 +1,42 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+# -%F footer expansion (hts_footer_format via -#test=footerfmt). Fixed fields:
+# addr=host.example path=/dir/page.html date=DATE version=VER.
+ftr() {
+    out="$(httrack -O /dev/null -#test=footerfmt "$1")"
+    test "$out" == "$2" || {
+        echo "FAIL: '$1' -> '$out' (want '$2')"
+        exit 1
+    }
+}
+
+# Legacy positional model: a template with %s keeps the historic addr/path/date
+# order byte-for-byte (backward compatibility).
+ftr '<!-- Mirrored from %s%s by X, %s -->' \
+    '<!-- Mirrored from host.example/dir/page.html by X, DATE -->'
+# Legacy arg overflow past the four fields yields the historic "???" marker.
+ftr 'a %s b %s c %s d %s e %s' 'a host.example b /dir/page.html c DATE d VER e ???'
+# Legacy %% is a literal percent; an unknown %x passes through verbatim.
+ftr '100%% %d %s' '100% %d host.example'
+
+# Named model (no %s): fields substitute by name, in any order or subset.
+ftr '{addr}{path} {version} {date}' 'host.example/dir/page.html VER DATE'
+ftr 'built {date}' 'built DATE'
+# A literal char abutting a field's '}' survives (guards the index advance).
+ftr '{date}!{addr}' 'DATE!host.example'
+# {{ and }} escape to a literal brace.
+ftr 'a {{ b }} {addr}' 'a { b } host.example'
+# Unknown, empty, or unterminated braces are emitted verbatim (typos stay visible).
+ftr '{bogus} {} {addr' '{bogus} {} {addr'
+# A literal percent is untouched in named mode (no positional interpretation).
+ftr '100% {path}' '100% /dir/page.html'
+# %s anywhere forces legacy mode, so a mixed template leaves {addr} literal.
+ftr '{addr} %s' '{addr} host.example'
+
+# The added named fields (url, lastmodified, mime, charset, status, size).
+ftr '{url}' 'http://host.example/dir/page.html'
+ftr 'src {lastmodified}, got {date}' 'src LASTMOD, got DATE'
+ftr '{mime}; {charset}; {status}; {size}' 'text/html; utf-8; 200; 1234'