Merge origin/master into feat/change-report

Conflict in tests/Makefile.am only: master added 88 and 92; the change-report
test keeps 93.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/workflows/windows-build.yml

@@ -225,8 +225,10 @@ jobs:
 
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
-          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581.
-          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test"
+          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
+          # webdav-mime needs a reapable background listener, which MSYS cannot give it;
+          # badmtime needs a filesystem that stores an mtime past gmtime's range.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -26,6 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
+- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
+  pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
+  and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
+  as "marker absent". Capture the reply, assert the status line it must carry
+  (an empty, truncated or redirected one is marker-free too), then match with a
+  here-string.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -48,6 +54,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
+- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
+  (`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
+  our own data, wrong for anything read back from a cache, a header or the
+  wire, where it trades a memory smash for a crash on malformed input. Clip
+  with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
+- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
+  a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
+  into the same buffer never appears on it. Before scoping a hardening pass off
+  compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
+  `\bstrcpy\s*\(`, `\bstrcat\s*\(`).
 
 ## C conventions
 - **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -84,6 +100,17 @@ Before pushing, and when reviewing others, don't skim for bugs:
   layout/ABI, cache/wire format, or a security path? A static or unit check
   isn't enough; exercise the wrong behavior at runtime. Claude Code:
   `/review-recipe`.
+- **Poison a canary, never compare it against zero.** Checking that a
+  neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
+  terminator writes — the exact bug the canary is there for. Fill it with a
+  non-zero byte, and prove it by killing both the stray-`'X'` and the
+  stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
+  lands inside the same struct.
+- **Overshoot every destination, not one.** A bounds test that oversizes a
+  single field cannot tell a per-field bound from a one-size-fits-all one, nor
+  from a fix that bounds that field and leaves its neighbours raw. Exercise
+  each destination the path touches, spanning at least two capacities, and
+  check what the code actually emits before writing the expected values.
 
 ## Commits
 - **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

configure.ac

@@ -66,10 +66,11 @@ AC_SUBST(LT_CV_OBJDIR,$lt_cv_objdir)
 AC_SUBST(VERSION_INFO)
 
 ### Default CFLAGS
+# No -Wdeclaration-after-statement: nothing sets -std=, so this builds as gnu17.
 DEFAULT_CFLAGS="-Wall -Wformat -Wformat-security \
 -Wmultichar -Wwrite-strings -Wcast-qual -Wcast-align \
 -Wstrict-prototypes -Wmissing-prototypes \
--Wmissing-declarations -Wdeclaration-after-statement \
+-Wmissing-declarations \
 -Wpointer-arith -Wsequence-point -Wnested-externs \
 -D_REENTRANT"
 AC_SUBST(DEFAULT_CFLAGS)
@@ -77,27 +78,30 @@ DEFAULT_LDFLAGS=""
 AC_SUBST(DEFAULT_LDFLAGS)
 
 ### Additional flags (if supported)
-AX_CHECK_COMPILE_FLAG([-Wparentheses], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wparentheses"])
-AX_CHECK_COMPILE_FLAG([-Winit-self], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Winit-self"])
-AX_CHECK_COMPILE_FLAG([-Wunused-but-set-parameter], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wunused-but-set-parameter"])
-AX_CHECK_COMPILE_FLAG([-Waddress], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Waddress"])
-AX_CHECK_COMPILE_FLAG([-Wuninitialized], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wuninitialized"])
-AX_CHECK_COMPILE_FLAG([-Wformat=2], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat=2"])
-AX_CHECK_COMPILE_FLAG([-Wformat-nonliteral], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat-nonliteral"])
-AX_CHECK_COMPILE_FLAG([-Wmissing-parameter-type], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wmissing-parameter-type"])
-AX_CHECK_COMPILE_FLAG([-Wold-style-definition], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wold-style-definition"])
-AX_CHECK_COMPILE_FLAG([-Wignored-qualifiers], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wignored-qualifiers"])
+# -Werror on probes: exit-status-only checks let clang's warn-on-unknown-flag through.
+AX_CHECK_COMPILE_FLAG([-Wparentheses], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wparentheses"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Winit-self], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Winit-self"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wunused-but-set-parameter], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wunused-but-set-parameter"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Waddress], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Waddress"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wuninitialized], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wuninitialized"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wformat=2], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat=2"], [], [-Werror])
+# -Wformat-nonliteral needs -Wformat in the probe or gcc rejects it as ignored.
+AX_CHECK_COMPILE_FLAG([-Wformat-nonliteral], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat-nonliteral"], [], [-Werror -Wformat])
+AX_CHECK_COMPILE_FLAG([-Wmissing-parameter-type], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wmissing-parameter-type"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wold-style-definition], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wold-style-definition"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wignored-qualifiers], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wignored-qualifiers"], [], [-Werror])
 # Make htssafe.h's pointer-dest 'warning' attribute a hard error in our build
 # (migration is at zero; a new char* dest is a regression). gcc/clang each take
 # only their own spelling; downstream keeps the plain warning, not a build break.
-AX_CHECK_COMPILE_FLAG([-Werror=attribute-warning], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=attribute-warning"])
-AX_CHECK_COMPILE_FLAG([-Werror=user-defined-warnings], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=user-defined-warnings"])
-AX_CHECK_COMPILE_FLAG([-fstrict-aliasing -Wstrict-aliasing], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstrict-aliasing -Wstrict-aliasing"])
+AX_CHECK_COMPILE_FLAG([-Werror=attribute-warning], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=attribute-warning"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Werror=user-defined-warnings], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=user-defined-warnings"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fstrict-aliasing -Wstrict-aliasing], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstrict-aliasing -Wstrict-aliasing"], [], [-Werror])
 AX_CHECK_COMPILE_FLAG([-fstack-protector-strong], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector-strong"],
-	[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"])])
-AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"])
-AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"])
-AX_CHECK_LINK_FLAG([-Wl,--discard-all], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--discard-all"])
+	[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"], [], [-Werror])], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"], [], [-Werror])
+# No --discard-all: it drops the local symbols naming every static function, so
+# a trace misattributes them to the nearest surviving global. Costs 0.6% size.
 AX_CHECK_LINK_FLAG([-Wl,--no-undefined], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--no-undefined"])
 AX_CHECK_LINK_FLAG([-Wl,-z,relro,-z,now], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,relro,-z,now"])
 AX_CHECK_LINK_FLAG([-Wl,-z,noexecstack], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,noexecstack"])
@@ -119,13 +123,13 @@ AC_SUBST([LIBC_FORCE_LINK])
 ### PIE
 CFLAGS_PIE=""
 LDFLAGS_PIE=""
-AX_CHECK_COMPILE_FLAG([-fpie -pie], [CFLAGS_PIE="-fpie -pie"])
+AX_CHECK_COMPILE_FLAG([-fpie], [CFLAGS_PIE="-fpie"], [], [-Werror])
 AX_CHECK_LINK_FLAG([-pie], [LDFLAGS_PIE="-pie"])
 AC_SUBST([CFLAGS_PIE])
 AC_SUBST([LDFLAGS_PIE])
 
-## Export all symbols for backtraces
-AX_CHECK_COMPILE_FLAG([-rdynamic], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -rdynamic"])
+# Ties a crash trace from a stripped build back to its separate debug symbols.
+AX_CHECK_LINK_FLAG([-Wl,--build-id], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--build-id"])
 
 ### Check for -fvisibility=hidden support
 gl_VISIBILITY

html/changes.html

@@ -0,0 +1,263 @@
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+
+<head>
+	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+	<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
+	<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local  site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
+	<title>HTTrack Website Copier - Change report format specification</title>
+
+	<style type="text/css">
+	<!--
+
+body {
+	margin: 0;  padding: 0;  margin-bottom: 15px;  margin-top: 8px;
+	background: #77b;
+}
+body, td {
+	font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
+	}
+
+#subTitle {
+	background: #000;  color: #fff;  padding: 4px;  font-weight: bold; 
+	}
+
+#siteNavigation a, #siteNavigation .current {
+	font-weight: bold;  color: #448;
+	}
+#siteNavigation a:link    { text-decoration: none; }
+#siteNavigation a:visited { text-decoration: none; }
+
+#siteNavigation .current { background-color: #ccd; }
+
+#siteNavigation a:hover   { text-decoration: none;  background-color: #fff;  color: #000; }
+#siteNavigation a:active  { text-decoration: none;  background-color: #ccc; }
+
+
+a:link    { text-decoration: underline;  color: #00f; }
+a:visited { text-decoration: underline;  color: #000; }
+a:hover   { text-decoration: underline;  color: #c00; }
+a:active  { text-decoration: underline; }
+
+#pageContent {
+	clear: both;
+	border-bottom: 6px solid #000;
+	padding: 10px;  padding-top: 20px;
+	line-height: 1.65em;
+	background-image: url(images/bg_rings.gif);
+	background-repeat: no-repeat;
+	background-position: top right;
+	}
+
+#pageContent, #siteNavigation {
+	background-color: #ccd;
+	}
+
+
+.imgLeft  { float: left;   margin-right: 10px;  margin-bottom: 10px; }
+.imgRight { float: right;  margin-left: 10px;   margin-bottom: 10px; }
+
+hr { height: 1px;  color: #000;  background-color: #000;  margin-bottom: 15px; }
+
+h1 { margin: 0;  font-weight: bold;  font-size: 2em; }
+h2 { margin: 0;  font-weight: bold;  font-size: 1.6em; }
+h3 { margin: 0;  font-weight: bold;  font-size: 1.3em; }
+h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
+
+.blak { background-color: #000; }
+.hide { display: none; }
+.tableWidth { min-width: 400px; }
+
+.tblRegular       { border-collapse: collapse; }
+.tblRegular td    { padding: 6px;  background-image: url(fade.gif);  border: 2px solid #99c; }
+.tblHeaderColor, .tblHeaderColor td { background: #99c; }
+.tblNoBorder td   { border: 0; }
+
+
+// -->
+</style>
+
+</head>
+
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+	<tr>
+	<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
+	<tr>
+	<td id="subTitle">Open Source offline browser</td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+<tr class="blak">
+<td>
+	<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
+	<tr>
+	<td colspan="6"> 
+		<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
+		<tr> 
+		<td id="pageContent"> 
+<!-- ==================== End prologue ==================== -->
+
+<h2 align="center"><em>Change report format specification</em></h2>
+
+<br />
+
+Run with <tt>--changes</tt> (<tt>-%d</tt>), HTTrack writes <tt>hts-changes.json</tt>
+in the project directory, next to <tt>hts-log.txt</tt>, describing what the crawl
+left new, changed, unchanged and gone compared to the previous mirror. The file is
+rewritten from scratch at the end of every run, and the log carries a one-line
+summary of the same counts.
+
+<br /><br />
+
+<h3>What "changed" means</h3>
+
+A resource is changed when its bytes differ, not when the server merely re-sent
+it. HTTrack compares the payload it just received against the copy the previous
+run left behind: for pages it parses, the previous payload comes from the cache
+(the file on disk carries the mirror footer and its crawl date, so its bytes
+differ on every run); for everything else, the mirrored file is the payload
+verbatim and is compared directly.
+
+<br /><br />
+
+Where no digest can be taken on either side, because the cache is disabled or
+the previous copy is gone, the report falls back to the transfer signal, and a
+server that answers 200 rather than 304 reads as changed. Keeping the cache on
+(the default) is what makes the report precise.
+
+<br /><br />
+
+<h3>With the cache off</h3>
+
+<tt>--cache=0</tt> costs the report more than the digest of a parsed page. The
+mirror's file index (<tt>hts-cache/new.lst</tt>) is what records which files a
+run produced, so without it there is no previous mirror to subtract from: nothing
+is reported <tt>gone</tt>, and whether the run is a first crawl cannot be decided
+at all, which <tt>first_crawl</tt> states as <tt>null</tt> rather than guess. What
+is on disk is still compared byte for byte, so the other three lists stay
+meaningful, except for the pages HTTrack parses: those have no cached payload to
+compare against and fall back to the transfer signal.
+
+<br /><br />
+
+<h3>Fields</h3>
+
+<ul>
+<li><tt>schema</tt>: format version, currently <tt>1</tt>. It is bumped only
+on an incompatible change; new fields may appear without one.</li>
+<li><tt>generator</tt>: the HTTrack build that wrote the file.</li>
+<li><tt>date</tt>: when the report was written, UTC, <tt>YYYY-MM-DDThh:mm:ssZ</tt>.</li>
+<li><tt>first_crawl</tt>: true when no index of a previous mirror
+(<tt>hts-cache/old.lst</tt>) was found, so there was nothing to compare against and
+everything is listed as new. Null when the run kept no index at all and the
+question cannot be answered (see above).</li>
+<li><tt>partial</tt>: true when the report ran out of memory and lists only
+part of the mirror.</li>
+<li><tt>purged</tt>: true when <tt>--purge-old</tt> was in effect, so the
+files under <tt>gone</tt> were also deleted from disk.</li>
+<li><tt>counts</tt>: the size of each of the four lists.</li>
+<li><tt>new</tt>, <tt>changed</tt>, <tt>unchanged</tt>, <tt>gone</tt>: the
+lists themselves. Every mirrored file appears in exactly one of them.</li>
+</ul>
+
+Each entry is an object:
+
+<ul>
+<li><tt>url</tt>: the absolute URL the file came from. Empty under
+<tt>gone</tt>: deletions are computed from the mirror's file index, which records
+paths, not URLs.</li>
+<li><tt>file</tt>: the path relative to the mirror root, with forward
+slashes. This is the entry's identity: a URL and a redirect that resolve to the
+same local file are one entry, not two.</li>
+<li><tt>size</tt>: the mirrored file's size in bytes, absent when the file
+is not on disk.</li>
+<li><tt>previous_size</tt>: under <tt>changed</tt> only, the size of the
+copy the previous run left.</li>
+</ul>
+
+<br />
+
+<h3>Encoding</h3>
+
+The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are
+not guaranteed to be valid UTF-8; any byte sequence that is not becomes
+U+FFFD (<tt>\ufffd</tt>), so the file always parses. Compare on <tt>file</tt>
+rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
+
+<br /><br />
+
+<h3>Example</h3>
+
+<pre>
+{
+  "schema": 1,
+  "generator": "HTTrack Website Copier/3.49-14",
+  "date": "2026-07-26T15:29:03Z",
+  "first_crawl": false,
+  "partial": false,
+  "purged": true,
+  "counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
+  "new": [
+    { "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
+  ],
+  "changed": [
+    { "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
+  ],
+  "unchanged": [
+    { "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
+  ],
+  "gone": [
+    { "url": "", "file": "example.com/c.html" }
+  ]
+}
+</pre>
+
+<br /><br />
+
+<h3>Notes</h3>
+
+<ul>
+<li>A file listed under <tt>gone</tt> is only deleted when <tt>--purge-old</tt> is
+on. Left in place it drops out of the mirror's index, so it is reported once and
+not again.</li>
+<li>A resource whose local file name changed since the previous mirror (a new
+MIME type, say) is reported as <tt>new</tt> under its new name; the old name is
+reported as <tt>gone</tt> only if the file is still on disk. The two entries are
+not paired.</li>
+<li>A resource this run tried and failed to transfer also drops out of the
+mirror's index, but its previous copy is untouched, so it is reported
+<tt>unchanged</tt>. Under <tt>--purge-old</tt> that copy is deleted anyway, and
+the report says <tt>gone</tt> to match.</li>
+<li>A run that transfers no data at all is rolled back: HTTrack restores the
+previous cache generation and leaves the previous report in place, so a lost
+connection does not overwrite a good report with an empty one.</li>
+<li>Content diffs, and keeping the previous copy of a changed page, are out of
+scope: both change what a mirror directory contains.</li>
+</ul>
+
+<br /><br />
+
+<!-- ==================== Start epilogue ==================== -->
+		</td>
+		</tr>
+		</table>
+	</td>
+	</tr>
+	</table>
+</td>
+</tr>
+</table>
+
+<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
+	<tr>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	</tr>
+</table>
+
+</body>
+
+</html>
+
+

html/cmdguide.html

@@ -482,6 +482,15 @@ add <tt>--warc-cdx</tt> for a sorted CDXJ index, or <tt>--wacz</tt> to bundle th
 archive, index and pages into one WACZ for replay tools such as
 replayweb.page.</small></p>
 
+<h4>See what a re-crawl changed</h4>
+<p><tt>httrack https://example.com/ --update --changes --path mydir</tt><br>
+<small>Writes <tt>hts-changes.json</tt> in the project folder, listing every
+mirrored file as new, changed, unchanged or gone, plus a one-line summary in the
+log. &quot;Changed&quot; means the bytes really differ: a server that answers 200
+with the same content it served last time lands in <tt>unchanged</tt>. Deletions
+are reported whether or not <tt>--purge-old</tt> is deleting them. The format is
+documented in <a href="changes.html">the change report specification</a>.</small></p>
+
 <h4>HTTrack as a fetch tool</h4>
 <p><tt>httrack --get https://host/file.bin --path tmp</tt><br>
 <small><tt>--get</tt> fetches one file with cache, index, depth, cookies and robots
@@ -490,7 +499,8 @@ all disabled.</small></p>
 <p><br>For the complete, always-current option list, see
 <a href="httrack.man.html">the manual page</a>. For the filter language, see
 <a href="filters.html">filters</a>; for the cache and updates, see
-<a href="cache.html">cache</a>.</p>
+<a href="cache.html">cache</a>; for the change report, see
+<a href="changes.html">changes</a>.</p>
 
 <!-- ==================== Start epilogue ==================== -->
 		</td>

html/dev.html

@@ -129,6 +129,9 @@ The library can be used to write graphical GUIs for httrack, or to run mirrors f
 <li><a href="cache.html">Cache format</a></li><br>
 HTTrack stores original HTML data and references to downloaded files in a cache, located in the hts-cache directory.
 This page describes the HTTrack cache format.
+<li><a href="changes.html">Change report format</a></li><br>
+With --changes, HTTrack writes hts-changes.json describing what the crawl left new, changed, unchanged and gone
+compared to the previous mirror. This page describes that file.
 </ul>
 
 

html/httrack.man.html

@@ -108,16 +108,17 @@ offline browser : copy websites to a local directory</p>
 --footer</b> ] [ <b>-%l, --language</b> ] [ <b>-%a,
 --accept</b> ] [ <b>-%X, --headers</b> ] [ <b>-C,
 --cache[=N]</b> ] [ <b>-k, --store-all-in-cache</b> ] [
-<b>-%r, --warc</b> ] [ <b>-%n, --do-not-recatch</b> ] [
-<b>-%v, --display</b> ] [ <b>-Q, --do-not-log</b> ] [ <b>-q,
---quiet</b> ] [ <b>-z, --extra-log</b> ] [ <b>-Z,
---debug-log</b> ] [ <b>-v, --verbose</b> ] [ <b>-f,
---file-log</b> ] [ <b>-f2, --single-log</b> ] [ <b>-I,
---index</b> ] [ <b>-%i, --build-top-index</b> ] [ <b>-%I,
---search-index</b> ] [ <b>-pN, --priority[=N]</b> ] [ <b>-S,
---stay-on-same-dir</b> ] [ <b>-D, --can-go-down</b> ] [
-<b>-U, --can-go-up</b> ] [ <b>-B, --can-go-up-and-down</b> ]
-[ <b>-a, --stay-on-same-address</b> ] [ <b>-d,
+<b>-%r, --warc</b> ] [ <b>-%d, --changes</b> ] [ <b>-%n,
+--do-not-recatch</b> ] [ <b>-%v, --display</b> ] [ <b>-Q,
+--do-not-log</b> ] [ <b>-q, --quiet</b> ] [ <b>-z,
+--extra-log</b> ] [ <b>-Z, --debug-log</b> ] [ <b>-v,
+--verbose</b> ] [ <b>-f, --file-log</b> ] [ <b>-f2,
+--single-log</b> ] [ <b>-I, --index</b> ] [ <b>-%i,
+--build-top-index</b> ] [ <b>-%I, --search-index</b> ] [
+<b>-pN, --priority[=N]</b> ] [ <b>-S, --stay-on-same-dir</b>
+] [ <b>-D, --can-go-down</b> ] [ <b>-U, --can-go-up</b> ] [
+<b>-B, --can-go-up-and-down</b> ] [ <b>-a,
+--stay-on-same-address</b> ] [ <b>-d,
 --stay-on-same-domain</b> ] [ <b>-l, --stay-on-same-tld</b>
 ] [ <b>-e, --go-everywhere</b> ] [ <b>-%H,
 --debug-headers</b> ] [ <b>-%!,
@@ -1143,6 +1144,19 @@ past N bytes, --warc-cdx also writes a sorted CDXJ index,
 <td width="4%">
 
 
+<p>-%d</p></td>
+<td width="5%"></td>
+<td width="82%">
+
+
+<p>write hts-changes.json listing what this crawl left new,
+changed, unchanged and gone compared to the previous mirror
+(--changes)</p> </td></tr>
+<tr valign="top" align="left">
+<td width="9%"></td>
+<td width="4%">
+
+
 <p>-%n</p></td>
 <td width="5%"></td>
 <td width="82%">

html/server/finished.html

@@ -97,9 +97,8 @@ ${do:end-if}
 </pre>
 
 ${LANG_G8} : 
-${do:output-mode:html-urlescaped}
-<a href="file://${path}/${projname}/" target="_new">
-${do:output-mode:}
+${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
+<a href="/website/index.html" target="_new">
 ${path}/${projname}
 </a></li>
 <ul>

html/server/option10.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ftpprox" value="">
+
 ${LANG_PROXYTYPE}
 <select name="proxytype"
 			title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

html/server/option2.html

@@ -98,6 +98,13 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="errpage" value="">
+<input type="hidden" name="external" value="">
+<input type="hidden" name="hidepwd" value="">
+<input type="hidden" name="hidequery" value="">
+<input type="hidden" name="nopurge" value="">
+
 ${LANG_I33}
 <br>
 <select name="build"

html/server/option2b.html

@@ -103,7 +103,7 @@ ${LANG_Q3}
 <tr><td>
 	<table width="100%">
 	<tr><td align="left">
-	<input type="submit" value="${LANG_OK]" 
+	<input type="submit" value="${LANG_OK}"
 ${do:output-mode:html-urlescaped}
 		onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
 ${do:output-mode:}

html/server/option3.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="windebug" value="">
+
 ${LANG_I40c}
 <br>
 

html/server/option4.html

@@ -98,6 +98,11 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ka" value="">
+<input type="hidden" name="remt" value="">
+<input type="hidden" name="rems" value="">
+
 <table border="0" width="100%" cellspacing="0">
 
 <tr><td>

html/server/option8.html

@@ -98,6 +98,17 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="cookies" value="">
+<input type="hidden" name="parsejava" value="">
+<input type="hidden" name="updhack" value="">
+<input type="hidden" name="urlhack" value="">
+<input type="hidden" name="keepwww" value="">
+<input type="hidden" name="keepslashes" value="">
+<input type="hidden" name="keepqueryorder" value="">
+<input type="hidden" name="toler" value="">
+<input type="hidden" name="http10" value="">
+
 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I58}

html/server/option9.html

@@ -98,6 +98,14 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="warc" value="">
+<input type="hidden" name="changes" value="">
+<input type="hidden" name="norecatch" value="">
+<input type="hidden" name="logf" value="">
+<input type="hidden" name="index" value="">
+<input type="hidden" name="index2" value="">
+
 <input type="checkbox" name="cache2" ${checked:cache2}
 			title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I61}
@@ -114,6 +122,11 @@ ${LANG_WARCFILE}
 >
 <br><br>
 
+<input type="checkbox" name="changes" ${checked:changes}
+			title='${html:LANG_CHANGESTIP}' onMouseOver="info('${html:LANG_CHANGESTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+> ${LANG_CHANGES}
+<br><br>
+
 <input type="checkbox" name="norecatch" ${checked:norecatch}
 			title='${html:LANG_I5b}' onMouseOver="info('${html:LANG_I5b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I34b}

html/server/step2.html

@@ -143,6 +143,7 @@ ${do:copy:StripQuery:stripquery}
 ${do:copy:StoreAllInCache:cache2}
 ${do:copy:Warc:warc}
 ${do:copy:WarcFile:warcfile}
+${do:copy:Changes:changes}
 ${do:copy:LogType:logtype}
 ${do:copy:UseHTTPProxyForFTP:ftpprox}
 ${do:copy:ProxyType:proxytype}

html/server/step4.html

@@ -114,16 +114,16 @@ ${do:end-if}
 
 ${/* Real commands and ini file generated below */}
 
-<!-- engine commandline -->
+<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
 ${do:output-mode:html}
 <textarea name="command" cols="50" rows="4" style="visibility:hidden">
 httrack \
 	--quiet \
 	--build-top-index \
 	${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
-	${urls}
-	${test:filelist:-%L "}${filelist}${test:filelist:"}
-	--path "${html:path}/${html:projname}" 
+	${unquoted:urls}
+	${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
+	--path "${arg:path}/${arg:projname}" 
 \
 	${test:parseall:--near}
 	${test:link:--test}
@@ -131,7 +131,7 @@ httrack \
 	${test:htmlfirst::--priority=7}
 \
 ${do:if-not-empty:BuildString}
-	--structure "${BuildString}"
+	--structure "${arg:BuildString}"
 ${do:end-if}
 	${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
 \
@@ -150,51 +150,53 @@ ${do:end-if}
 	${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
 	${test:windebug:--debug-headers}
 \
-	${test:connexion:--sockets=}${connexion}
+	${test:connexion:--sockets=}${unquoted:connexion}
 	${test:ka:--keep-alive}
-	${test:timeout:--timeout=}${timeout}
+	${test:timeout:--timeout=}${unquoted:timeout}
 	${test:remt:--host-control=1}
-	${test:retry:--retries=}${retry}
-	${test:rate:--min-rate=}${rate}
+	${test:retry:--retries=}${unquoted:retry}
+	${test:rate:--min-rate=}${unquoted:rate}
 	${test:rems:--host-control=2}
 \
-	${test:depth:--depth=}${depth}
-	${test:depth2:--ext-depth=}${depth2}
-	${test:maxhtml:--max-files=,}${maxhtml}
-	${test:othermax:--max-files=}${othermax}
-	${test:sizemax:--max-files=}${sizemax}
-	${test:pausebytes:--max-pause=}${pausebytes}
-	${test:maxtime:--max-time=}${maxtime}
-	${test:maxrate:--max-rate=}${maxrate}
-	${test:maxconn:--connection-per-second=}${maxconn}
-	${test:maxlinks:--advanced-maxlinks=}${maxlinks}
+	${test:depth:--depth=}${unquoted:depth}
+	${test:depth2:--ext-depth=}${unquoted:depth2}
+${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
+	${test:othermax:--max-files=}${unquoted:othermax}
+	${test:maxhtml:--max-files=,}${unquoted:maxhtml}
+	${test:sizemax:--max-size=}${unquoted:sizemax}
+	${test:pausebytes:--max-pause=}${unquoted:pausebytes}
+	${test:maxtime:--max-time=}${unquoted:maxtime}
+	${test:maxrate:--max-rate=}${unquoted:maxrate}
+	${test:maxconn:--connection-per-second=}${unquoted:maxconn}
+	${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
 \
-	--user-agent "${html:user}"
-	--footer "${html:footer}"
+	--user-agent "${arg:user}"
+	--footer "${arg:footer}"
 \
-	${url2}
+	${unquoted:url2}
 \
-	${test:cookies:--cookies=0:}
-	${test:parsejava:--parse-java=0:}
+	${ztest:cookies:--cookies=0:}
+	${ztest:parsejava:--parse-java=0:}
 	${test:updhack:--updatehack}
 	${test:urlhack:--urlhack=0:--urlhack}
 	${test:keepwww:--keep-www-prefix}
 	${test:keepslashes:--keep-double-slashes}
 	${test:keepqueryorder:--keep-query-order}
-	${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
-	${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
-	${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
+	${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
+	${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
+	${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
 	${test:toler:--tolerant}
 	${test:http10:--http-10}
 	${test:cache2:--store-all-in-cache}
 	${test:warc:--warc}
-	${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
+	${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
+	${test:changes:--changes}
 	${test:norecatch:--do-not-recatch}
 	${test:logf:--single-log}
 	${test:logtype:::--extra-log:--debug-log}
 	${test:index:--index=0:}
 	${test:index2:--search-index=0:--search-index}
-	${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
+	${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
 	${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
 </textarea>
 
@@ -213,7 +215,7 @@ ParseAll=${ztest:parseall:0:1}
 HTMLFirst=${ztest:htmlfirst:0:1}
 Cache=${ztest:cache:0:1}
 NoRecatch=${ztest:norecatch:0:1}
-Dos=${dos
+Dos=${dos}
 Index=${ztest:index:0:1}
 WordIndex=${ztest:index2:0:1}
 Log=${ztest:logf:0:1:2}
@@ -241,6 +243,7 @@ StripQuery=${stripquery}
 StoreAllInCache=${ztest:cache2:0:1}
 Warc=${ztest:warc:0:1}
 WarcFile=${warcfile}
+Changes=${ztest:changes:0:1}
 LogType=${logtype}
 UseHTTPProxyForFTP=${ztest:ftpprox:0:1}
 ProxyType=${proxytype}

lang.def

@@ -1042,3 +1042,7 @@ LANG_WARCFILE
 WARC archive name:
 LANG_WARCFILETIP
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+LANG_CHANGES
+Report what changed since the previous mirror
+LANG_CHANGESTIP
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.

lang/Bulgarian.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Èìå íà WARC àðõèâà:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåçàäúëæèòåëíî áàçîâî èìå çà WARC àðõèâà; îñòàâåòå ïðàçíî çà àâòîìàòè÷íî èìåíóâàíå â èçõîäíàòà äèðåêòîðèÿ.
+Report what changed since the previous mirror
+Îò÷åò çà ïðîìåíèòå ñïðÿìî ïðåäèøíîòî îãëåäàëî
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Çàïèñâàíå è íà hts-changes.json ñúñ ñïèñúê íà íîâèòå, ïðîìåíåíèòå, íåïðîìåíåíèòå è èç÷åçíàëèòå ôàéëîâå ñïðÿìî ïðåäèøíîòî îãëåäàëî.

lang/Castellano.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Nombre del archivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida.
+Report what changed since the previous mirror
+Informar de los cambios desde la copia anterior
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Escribir también hts-changes.json con la lista de lo que esta captura deja como nuevo, modificado, sin cambios o desaparecido respecto a la copia anterior.

lang/Cesky.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Název archivu WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Volitelný základní název archivu WARC; ponechte prázdné pro automatické pojmenování ve výstupním adresáøi.
+Report what changed since the previous mirror
+Hlásit, co se od pøedchozího zrcadlení zmìnilo
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapsat také hts-changes.json se seznamem toho, co je oproti pøedchozímu zrcadlení nové, zmìnìné, nezmìnìné nebo chybìjící.

lang/Chinese-BIG5.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC 封存檔名稱：
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC 封存檔的選用基本名稱；留空則於輸出目錄中自動命名。
+Report what changed since the previous mirror
+回報自上次鏡射以來的變更
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+同時寫入 hts-changes.json，列出這次擷取相對於上次鏡射的新增、變更、未變更與消失的項目。

lang/Chinese-Simplified.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC 归档名称：
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC 归档的可选基本名称；留空则在输出目录中自动命名。
+Report what changed since the previous mirror
+报告自上次镜像以来的变更
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+同时写入 hts-changes.json，列出本次抓取相对于上次镜像的新增、更改、未更改和消失的项目。

lang/Croatian.txt

@@ -966,3 +966,7 @@ WARC archive name:
 Naziv WARC arhive:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju.
+Report what changed since the previous mirror
+Prijavi ¹to se promijenilo od prethodnog zrcaljenja
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapi¹i i hts-changes.json s popisom onoga ¹to je u odnosu na prethodno zrcaljenje novo, promijenjeno, nepromijenjeno ili nestalo.

lang/Dansk.txt

@@ -1012,3 +1012,7 @@ WARC archive name:
 Navn på WARC-arkiv:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen.
+Report what changed since the previous mirror
+Rapportér hvad der er ændret siden den forrige spejling
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Skriv også hts-changes.json med en liste over, hvad denne gennemgang efterlader som nyt, ændret, uændret eller forsvundet i forhold til den forrige spejling.

lang/Deutsch.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Name des WARC-Archivs:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen.
+Report what changed since the previous mirror
+Melden, was sich seit der vorherigen Spiegelung geändert hat
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zusätzlich hts-changes.json schreiben, das auflistet, was dieser Durchlauf gegenüber der vorherigen Spiegelung als neu, geändert, unverändert oder verschwunden hinterlässt.

lang/Eesti.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC-arhiivi nimi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC-arhiivi valikuline põhinimi; jäta tühjaks, et see väljundkataloogis automaatselt nimetada.
+Report what changed since the previous mirror
+Teata, mis on eelmisest peegeldusest saadik muutunud
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Kirjuta ka hts-changes.json, mis loetleb, mis on võrreldes eelmise peegeldusega uus, muutunud, muutumatu või kadunud.

lang/English.txt

@@ -1012,3 +1012,7 @@ WARC archive name:
 WARC archive name:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+Report what changed since the previous mirror
+Report what changed since the previous mirror
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.

lang/Finnish.txt

@@ -966,3 +966,7 @@ WARC archive name:
 WARC-arkiston nimi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valinnainen WARC-arkiston perusnimi; jätä tyhjäksi, jotta se nimetään automaattisesti tulostehakemistoon.
+Report what changed since the previous mirror
+Raportoi, mikä on muuttunut edellisen peilauksen jälkeen
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Kirjoita myös hts-changes.json, joka luettelee, mikä on edelliseen peilaukseen verrattuna uutta, muuttunutta, muuttumatonta tai kadonnutta.

lang/Francais.txt

@@ -1012,3 +1012,7 @@ WARC archive name:
 Nom de l'archive WARC :
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.
+Report what changed since the previous mirror
+Signaler ce qui a changé depuis le miroir précédent
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Écrire aussi hts-changes.json, qui liste ce que ce crawl laisse nouveau, modifié, inchangé ou disparu par rapport au miroir précédent.

lang/Greek.txt

@@ -966,3 +966,7 @@ WARC archive name:
 Όνομα αρχείου WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Προαιρετικό βασικό όνομα για το αρχείο WARC. Αφήστε το κενό για αυτόματη ονομασία στον κατάλογο εξόδου.
+Report what changed since the previous mirror
+Αναφορά των αλλαγών από το προηγούμενο αντίγραφο
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Εγγραφή και του hts-changes.json, που παραθέτει τι είναι νέο, τι άλλαξε, τι έμεινε ίδιο και τι χάθηκε σε σχέση με το προηγούμενο αντίγραφο.

lang/Italiano.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Nome dell'archivio WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome di base facoltativo per l'archivio WARC; lascia vuoto per assegnarlo automaticamente nella directory di output.
+Report what changed since the previous mirror
+Segnala che cosa è cambiato dalla copia precedente
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Scrive anche hts-changes.json, che elenca ciò che questa scansione lascia come nuovo, modificato, invariato o scomparso rispetto alla copia precedente.

lang/Japanese.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC アーカイブ名:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC アーカイブの任意のベース名。空欄にすると出力ディレクトリ内で自動的に名前が付けられます。
+Report what changed since the previous mirror
+前回のミラーからの変更点を報告する
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+hts-changes.json も書き出し、前回のミラーと比べて新規、変更、変更なし、消滅となったものを一覧にします。

lang/Macedonian.txt

@@ -964,3 +964,7 @@ WARC archive name:
 ¸ÜÕ ÝÐ WARC ÐàåØÒÐâÐ:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 ¸×ÑÞàÝÞ ÞáÝÞÒÝÞ ØÜÕ ×Ð WARC ÐàåØÒÐâÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÐÒâÞÜÐâáÚÞ ØÜÕÝãÒÐúÕ ÒÞ Ø×ÛÕ×ÝØÞâ ÔØàÕÚâÞàØãÜ.
+Report what changed since the previous mirror
+¸×ÒÕáâØ èâÞ Õ ßàÞÜÕÝÕâÞ ÞÔ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+·ÐßØèØ Ø hts-changes.json áÞ áßØáÞÚ ÝÐ âÞÐ èâÞ Õ ÝÞÒÞ, ßàÞÜÕÝÕâÞ, ÝÕßàÞÜÕÝÕâÞ ØÛØ ØáçÕ×ÝÐâÞ ÒÞ ÞÔÝÞá ÝÐ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ.

lang/Magyar.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC archívum neve:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 A WARC archívum opcionális alapneve; hagyja üresen az automatikus elnevezéshez a kimeneti könyvtárban.
+Report what changed since the previous mirror
+Jelentés arról, mi változott az elõzõ tükrözés óta
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+A hts-changes.json fájl írása is, amely felsorolja, mi új, mi változott, mi maradt változatlan és mi tûnt el az elõzõ tükrözéshez képest.

lang/Nederlands.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Naam van WARC-archief:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optionele basisnaam voor het WARC-archief; laat leeg om het automatisch een naam te geven in de uitvoermap.
+Report what changed since the previous mirror
+Rapporteren wat er sinds de vorige spiegeling is gewijzigd
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Ook hts-changes.json schrijven met een overzicht van wat deze doorloop nieuw, gewijzigd, ongewijzigd of verdwenen laat ten opzichte van de vorige spiegeling.

lang/Norsk.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Navn på WARC-arkiv:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valgfritt basisnavn for WARC-arkivet; la feltet stå tomt for automatisk navngivning i utdatamappen.
+Report what changed since the previous mirror
+Rapporter hva som er endret siden forrige speiling
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Skriv også hts-changes.json som lister opp hva denne gjennomgangen etterlater som nytt, endret, uendret eller forsvunnet sammenlignet med forrige speiling.

lang/Polski.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Nazwa archiwum WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Opcjonalna nazwa bazowa archiwum WARC; pozostaw puste, aby nazwaæ je automatycznie w katalogu wyj¶ciowym.
+Report what changed since the previous mirror
+Zg³o¶, co zmieni³o siê od poprzedniej kopii
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapisz równie¿ plik hts-changes.json z list± tego, co w porównaniu z poprzedni± kopi± jest nowe, zmienione, niezmienione lub usuniête.

lang/Portugues-Brasil.txt

@@ -1012,3 +1012,7 @@ WARC archive name:
 Nome do arquivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome base opcional para o arquivo WARC; deixe em branco para nomeá-lo automaticamente no diretório de saída.
+Report what changed since the previous mirror
+Relatar o que mudou desde o espelhamento anterior
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Gravar também hts-changes.json listando o que esta captura deixa como novo, alterado, inalterado ou removido em relação ao espelhamento anterior.

lang/Portugues.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Nome do arquivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome base opcional para o arquivo WARC; deixe em branco para o nomear automaticamente no diretório de saída.
+Report what changed since the previous mirror
+Comunicar o que mudou desde o espelho anterior
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Escrever também hts-changes.json, que lista o que esta recolha deixa como novo, alterado, inalterado ou desaparecido em relação ao espelho anterior.

lang/Romanian.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Numele arhivei WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nume de baza optional pentru arhiva WARC; lasati gol pentru a-l denumi automat in directorul de iesire.
+Report what changed since the previous mirror
+Raporteaza ce s-a schimbat fata de copia anterioara
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Scrie si hts-changes.json, care listeaza ce este nou, modificat, nemodificat sau disparut fata de copia anterioara.

lang/Russian.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Èìÿ WARC-àðõèâà:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåîáÿçàòåëüíîå áàçîâîå èìÿ WARC-àðõèâà; îñòàâüòå ïóñòûì äëÿ àâòîìàòè÷åñêîãî èìåíîâàíèÿ â âûõîäíîì êàòàëîãå.
+Report what changed since the previous mirror
+Ñîîáùàòü, ÷òî èçìåíèëîñü ñ ïðåäûäóùåãî çåðêàëà
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Òàêæå çàïèñûâàòü hts-changes.json ñî ñïèñêîì òîãî, ÷òî ïî ñðàâíåíèþ ñ ïðåäûäóùèì çåðêàëîì ñòàëî íîâûì, èçìåí¸ííûì, íåèçìåí¸ííûì èëè èñ÷åçëî.

lang/Slovak.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Názov archívu WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Voliteµný základný názov archívu WARC; ponechajte prázdne pre automatické pomenovanie vo výstupnom adresári.
+Report what changed since the previous mirror
+Oznámi», èo sa od predchádzajúceho zrkadlenia zmenilo
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapísa» aj hts-changes.json so zoznamom toho, èo je oproti predchádzajúcemu zrkadleniu nové, zmenené, nezmenené alebo chýbajúce.

lang/Slovenian.txt

@@ -964,3 +964,7 @@ WARC archive name:
 Ime arhiva WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Neobvezno osnovno ime arhiva WARC; pustite prazno za samodejno poimenovanje v izhodni mapi.
+Report what changed since the previous mirror
+Porocaj, kaj se je spremenilo od prejsnjega zrcaljenja
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapisi tudi hts-changes.json s seznamom tega, kar je v primerjavi s prejsnjim zrcaljenjem novo, spremenjeno, nespremenjeno ali izginilo.

lang/Svenska.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC-arkivets namn:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valfritt basnamn för WARC-arkivet; lämna tomt för att namnge det automatiskt i utdatakatalogen.
+Report what changed since the previous mirror
+Rapportera vad som ändrats sedan föregående spegling
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Skriv även hts-changes.json som listar vad den här genomgången lämnar som nytt, ändrat, oförändrat eller försvunnet jämfört med föregående spegling.

lang/Turkish.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC arþivi adý:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC arþivi için isteðe baðlý temel ad; çýktý dizininde otomatik adlandýrma için boþ býrakýn.
+Report what changed since the previous mirror
+Önceki yansýmadan bu yana deðiþenleri bildir
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Ayrýca hts-changes.json yazarak bu taramanýn önceki yansýmaya göre neyi yeni, deðiþmiþ, deðiþmemiþ veya kaybolmuþ býraktýðýný listele.

lang/Ukrainian.txt

@@ -964,3 +964,7 @@ WARC archive name:
 ²ì'ÿ WARC-àðõ³âó:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåîáîâ'ÿçêîâà áàçîâà íàçâà WARC-àðõ³âó; çàëèøòå ïîðîæí³ì äëÿ àâòîìàòè÷íîãî íàéìåíóâàííÿ ó âèõ³äíîìó êàòàëîç³.
+Report what changed since the previous mirror
+Ïîâ³äîìëÿòè, ùî çì³íèëîñÿ ç ïîïåðåäíüîãî äçåðêàëà
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Òàêîæ çàïèñóâàòè hts-changes.json ç³ ñïèñêîì òîãî, ùî ïîð³âíÿíî ç ïîïåðåäí³ì äçåðêàëîì º íîâèì, çì³íåíèì, íåçì³íåíèì àáî çíèêëèì.

lang/Uzbek.txt

@@ -964,3 +964,7 @@ WARC archive name:
 WARC arxivi nomi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC arxivi uchun ixtiyoriy asosiy nom; chiqish katalogida avtomatik nomlash uchun bo'sh qoldiring.
+Report what changed since the previous mirror
+Oldingi nusxadan beri nima o’zgarganini xabar qilish
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+hts-changes.json ham yoziladi: unda ushbu yig’ish oldingi nusxaga nisbatan nimani yangi, o’zgargan, o’zgarmagan yoki yo’qolgan holda qoldirgani ro’yxati bo’ladi.

libtest/callbacks-example-filenameiisbug.c

@@ -71,7 +71,9 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
       for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
       if (iisBogus[i][j] == '\0'
           && (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
-        strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
+        /* j bytes matched, so j fit: copying j cannot overrun whatever the
+           table holds, and the tail must survive untouched */
+        memcpy(a, iisBogusReplace[i], (size_t) j);
         break;
       }
     }

man/httrack.1

@@ -3,7 +3,7 @@
 .\"
 .\" This file is generated by man/makeman.sh; do not edit by hand.
 .\" SPDX-License-Identifier: GPL-3.0-or-later
-.TH httrack 1 "23 July 2026" "httrack website copier"
+.TH httrack 1 "26 July 2026" "httrack website copier"
 .SH NAME
 httrack \- offline browser : copy websites to a local directory
 .SH SYNOPSIS
@@ -75,6 +75,7 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-C, \-\-cache[=N]\fR ]
 [ \fB\-k, \-\-store\-all\-in\-cache\fR ]
 [ \fB\-%r, \-\-warc\fR ]
+[ \fB\-%d, \-\-changes\fR ]
 [ \fB\-%n, \-\-do\-not\-recatch\fR ]
 [ \fB\-%v, \-\-display\fR ]
 [ \fB\-Q, \-\-do\-not\-log\fR ]
@@ -279,6 +280,8 @@ create/use a cache for updates and retries (C0 no cache,C1 cache is prioritary,*
 store all files in cache (not useful if files on disk) (\-\-store\-all\-in\-cache)
 .IP \-%r
 write an ISO\-28500 WARC/1.1 archive; \-\-warc\-file NAME sets the output name, \-\-warc\-max\-size N rotates segments past N bytes, \-\-warc\-cdx also writes a sorted CDXJ index, \-\-wacz packages it all as a WACZ file (\-\-warc)
+.IP \-%d
+write hts\-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror (\-\-changes)
 .IP \-%n
 do not re\-download locally erased files (\-\-do\-not\-recatch)
 .IP \-%v

src/Makefile.am

@@ -32,7 +32,9 @@ AM_LDFLAGS = \
 
 bin_PROGRAMS = proxytrack httrack htsserver
 
-httrack_LDADD = $(THREADS_LIBS) libhttrack.la
+httrack_SOURCES = httrack.c htsbacktrace.c htsbacktrace.h
+# $(DL_LIBS): dladdr() in the crash handler, still in libdl on pre-2.34 glibc.
+httrack_LDADD = $(THREADS_LIBS) $(DL_LIBS) libhttrack.la
 htsserver_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS) libhttrack.la
 proxytrack_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS)
 
@@ -46,7 +48,8 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
 
 lib_LTLIBRARIES = libhttrack.la
 
-htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
+htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
+	htscmdline.c htscmdline.h \
 	htsurlport.c htsurlport.h
 proxytrack_SOURCES = proxy/main.c \
 	proxy/proxytrack.c proxy/store.c \
@@ -60,21 +63,21 @@ whttrackrun_SCRIPTS = webhttrack
 libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscache_selftest.c htsdns_selftest.c htsselftest.c \
 	htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
-	htshelp.c htslib.c htsurlport.c htscoremain.c \
+	htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htschanges.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
 	hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
 	htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h  \
-	htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
+	htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
 	htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htschanges.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \
@@ -84,7 +87,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
 libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
 libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
 
-EXTRA_DIST = httrack.h webhttrack \
+EXTRA_DIST = httrack.h htsstats.h webhttrack \
 		version.rc \
 		libhttrack.rc \
 		httrack.rc \

src/htsalias.c

@@ -114,6 +114,8 @@ const char *hts_optalias[][4] = {
    "strip [host/pattern=]key1,key2,... from URLs"},
   {"cookies-file", "-%K", "param1",
    "load extra cookies from a Netscape cookies.txt"},
+  {"changes", "-%d", "single",
+   "write hts-changes.json: what this crawl changed vs. the previous mirror"},
   {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
   {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
   {"warc-max-size", "-%rs", "param1",

src/htsback.c

@@ -38,6 +38,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsnet.h"
 #include "htscore.h"
 #include "htswarc.h"
+#include "htschanges.h"
 #include "htsthread.h"
 #include <time.h>
 /* END specific definitions */
@@ -125,6 +126,22 @@ void back_free(struct_back ** sback) {
    above a normal handshake. The last candidate still gets the full timeout. */
 #define HTS_CONNECT_FALLBACK_TIMEOUT 10
 
+void back_read_ftp_result(FILE *fp, htsblk *r) {
+  size_t j = 0;
+
+  if (fscanf(fp, "%d ", &r->statuscode) != 1)
+    r->statuscode = STATUSCODE_INVALID;
+  // an external helper writes this file: stop at capacity, not at EOF
+  while (j + 1 < sizeof(r->msg)) {
+    const int c = fgetc(fp);
+
+    if (c == EOF)
+      break;
+    r->msg[j++] = (char) c;
+  }
+  r->msg[j] = '\0';
+}
+
 int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
                               int timeout) {
   int deadline;
@@ -541,9 +558,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
                                const char *ext) {
   // do not use tempnam() but a regular filename
   back->tmpfile_buffer[0] = '\0';
-  if (back->url_sav != NULL && back->url_sav[0] != '\0') {
-    snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
-             back->url_sav, ext);
+  if (back->url_sav[0] != '\0') {
+    /* same capacity as url_sav, so truncation drops the extension and aliases
+       the temp name onto the live file that back_finalize_backup() UNLINKs */
+    if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
+      hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
+                    back->url_sav);
+      back->tmpfile_buffer[0] = '\0';
+      return -1;
+    }
     back->tmpfile = back->tmpfile_buffer;
     if (structcheck(back->tmpfile) != 0) {
       hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
@@ -551,8 +574,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
       return -1;
     }
   } else {
-    snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
-             StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
+    /* truncation here would collide distinct tmpnameid's onto one name */
+    if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
+                     StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
+                     ext)) {
+      hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
+                    StringBuff(opt->path_html_utf8));
+      back->tmpfile_buffer[0] = '\0';
+      return -1;
+    }
     back->tmpfile = back->tmpfile_buffer;
   }
   /* OK */
@@ -699,6 +729,13 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
                 if ((size = hts_codec_unpack(codec, back[p].tmpfile,
                                              unpacked)) >= 0) {
                   back[p].r.size = back[p].r.totalsize = size;
+                  if (back[p].r.is_write) {
+                    /* Sample the previous copy now: the rename below replaces
+                       it, and file_notify() only fires once it is gone. */
+                    hts_changes_notify(
+                        opt, back[p].url_adr, back[p].url_fil, back[p].url_sav,
+                        HTS_TRUE, back[p].r.notmodified ? HTS_TRUE : HTS_FALSE);
+                  }
                   if (!back[p].r.is_write) {
                     // fichier -> mémoire ; le fichier est écrit plus tard
                     deleteaddr(&back[p].r);
@@ -887,8 +924,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
               HTS_STAT.stat_bytes += back[p].r.size;
               HTS_STAT.stat_files++;
               hts_log_print(opt, LOG_TRACE, "added file %s%s => %s",
-                            back[p].url_adr, back[p].url_fil,
-                            back[p].url_sav != NULL ? back[p].url_sav : "");
+                            back[p].url_adr, back[p].url_fil, back[p].url_sav);
             }
             if ((!back[p].r.notmodified) && (opt->is_update)) {
               HTS_STAT.stat_updated_files++;    // page modifiée
@@ -2302,26 +2338,24 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
               && slot_can_be_finalized(opt, &back[i]);
             int may_serialize = slot_can_be_cached_on_disk(&back[i]);
 
-            hts_log_print(opt, LOG_DEBUG,
-                          "back[%03d]: may_clean=%d, may_finalize_disk=%d, may_serialize=%d:"
-                          LF "\t"
-                          "finalized(%d), status(%d), locked(%d), delayed(%d), test(%d), "
-                          LF "\t"
-                          "statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), "
-                          LF "\t" "contenttype(%s), url(%s%s), save(%s)", i,
-                          may_clean, may_finalize, may_serialize,
-                          back[i].finalized, back[i].status, back[i].locked,
-                          IS_DELAYED_EXT(back[i].url_sav), back[i].testmode,
-                          back[i].r.statuscode, (int) back[i].r.size,
-                          back[i].r.is_write, may_be_hypertext_mime(opt,
-                                                                    back[i].r.
-                                                                    contenttype,
-                                                                    back[i].
-                                                                    url_fil),
-                          /* */
-                          back[i].r.contenttype, back[i].url_adr,
-                          back[i].url_fil,
-                          back[i].url_sav ? back[i].url_sav : "<null>");
+            hts_log_print(
+                opt, LOG_DEBUG,
+                "back[%03d]: may_clean=%d, may_finalize_disk=%d, "
+                "may_serialize=%d:" LF "\t"
+                "finalized(%d), status(%d), locked(%d), delayed(%d), "
+                "test(%d), " LF "\t"
+                "statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), " LF
+                "\t"
+                "contenttype(%s), url(%s%s), save(%s)",
+                i, may_clean, may_finalize, may_serialize, back[i].finalized,
+                back[i].status, back[i].locked, IS_DELAYED_EXT(back[i].url_sav),
+                back[i].testmode, back[i].r.statuscode, (int) back[i].r.size,
+                back[i].r.is_write,
+                may_be_hypertext_mime(opt, back[i].r.contenttype,
+                                      back[i].url_fil),
+                /* */
+                back[i].r.contenttype, back[i].url_adr, back[i].url_fil,
+                back[i].url_sav);
           }
         }
       }
@@ -2857,7 +2891,8 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               // new session
               back[i].r.ssl_con = SSL_new(openssl_ctx);
               if (back[i].r.ssl_con) {
-                const char* hostname = jump_protocol_const(back[i].url_adr);
+                /* non-const twin: the OpenSSL macro casts the qualifier away */
+                char *hostname = jump_protocol(back[i].url_adr);
                 // some servers expect the hostname on the clienthello (SNI TLS extension)
                 SSL_set_tlsext_host_name(back[i].r.ssl_con, hostname);
                 SSL_clear(back[i].r.ssl_con);
@@ -2949,7 +2984,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               back[i].r.msg[0] = '\0';
               strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
               if (!strnotempty(back[i].r.msg)) {
-                sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
+                htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
               }
               deletehttp(&back[i].r);
               back[i].r.soc = INVALID_SOCKET;
@@ -3022,16 +3057,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
             FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
                   "rb");
           if (fp) {
-            int j = 0;
-
-            fscanf(fp, "%d ", &(back[i].r.statuscode));
-            while(!feof(fp)) {
-              int c = fgetc(fp);
-
-              if (c != EOF)
-                back[i].r.msg[j++] = c;
-            }
-            back[i].r.msg[j++] = '\0';
+            back_read_ftp_result(fp, &back[i].r);
             fclose(fp);
             UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
             strcpybuff(fconcat
@@ -3332,10 +3358,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                   deleteaddr(&back[i].r);
                   if (back[i].r.size < back[i].r.totalsize)
                     back[i].r.statuscode = STATUSCODE_CONNERROR;        // recatch
-                  sprintf(back[i].r.msg,
-                          "Incorrect length (" LLintP " Bytes, " LLintP
-                          " expected)", (LLint) back[i].r.size,
-                          (LLint) back[i].r.totalsize);
+                  htsblk_failf(&back[i].r,
+                               "Incorrect length (" LLintP " Bytes, " LLintP
+                               " expected)",
+                               (LLint) back[i].r.size,
+                               (LLint) back[i].r.totalsize);
                 } else {
                   // Un warning suffira..
                   hts_log_print(opt, LOG_WARNING,

src/htsback.h

@@ -74,6 +74,10 @@ void back_free(struct_back ** sback);
 // backing
 #define BACK_ADD_TEST "(dummy)"
 #define BACK_ADD_TEST2 "(dummy2)"
+/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
+   clipping the message to r->msg. */
+void back_read_ftp_result(FILE *fp, htsblk *r);
+
 int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
                const char *sav);
 int back_available(const struct_back * sback);

src/htsbacktrace.c

@@ -0,0 +1,238 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: crash backtrace printer                                 */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+/* Before every header: glibc gates dladdr() on it. */
+#if defined(__linux) && !defined(_GNU_SOURCE)
+#define _GNU_SOURCE
+#endif
+
+#include "htsbacktrace.h"
+
+#include "htsglobal.h"
+
+#include <stdlib.h>
+#include <string.h>
+#ifdef _WIN32
+#include <io.h> /* write */
+#endif
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+#if (defined(__linux) && defined(HAVE_EXECINFO_H))
+#include <dlfcn.h>
+#include <errno.h>
+#include <execinfo.h>
+#include <signal.h>
+#include <time.h>
+#include <sys/wait.h>
+#define USES_BACKTRACE
+#endif
+
+#ifdef USES_BACKTRACE
+#define BT_MAX_FRAMES 64     /* frames we try to name */
+#define BT_MAX_MODULES 8     /* distinct modules, one child each */
+#define BT_HEX_SIZE 19       /* "0x" + 16 nibbles + NUL */
+#define BT_PATH_SIZE 1024    /* module path; longer is skipped */
+#define BT_WAIT_TICKS 300    /* 10ms ticks, shared: cap a slow child */
+#define BT_NO_SYMBOLIZER 127 /* child exit: execvp() found none */
+
+static hts_boolean symbolize_crash = HTS_TRUE;
+
+/* "0x"-prefixed hex: the handler must stay stdio-free. */
+static void print_hex(char *buffer, uintptr_t value) {
+  static const char digits[] = "0123456789abcdef";
+  size_t i = 2, a, b;
+
+  buffer[0] = '0';
+  buffer[1] = 'x';
+  do {
+    buffer[i++] = digits[value & 0xf];
+    value >>= 4;
+  } while (value != 0);
+  buffer[i] = '\0';
+  for (a = 2, b = i - 1; a < b; a++, b--) {
+    const char c = buffer[a];
+
+    buffer[a] = buffer[b];
+    buffer[b] = c;
+  }
+}
+
+/* HTS_FALSE if src does not fit: a truncated module path would point the
+   symbolizer at the wrong file. */
+static hts_boolean copy_bounded(char *dest, size_t size, const char *src) {
+  size_t i;
+
+  for (i = 0; i < size - 1 && src[i] != '\0'; i++) {
+    dest[i] = src[i];
+  }
+  dest[i] = '\0';
+  return src[i] == '\0' ? HTS_TRUE : HTS_FALSE;
+}
+
+/* Run the symbolizer on argv, output on fd, within *budget ticks. HTS_FALSE
+   only if none could be run at all; otherwise silent, the raw trace stands. */
+static hts_boolean spawn_symbolizer(char **argv, int fd, int *budget) {
+  const pid_t pid = fork();
+  int status = 0;
+
+  if (pid == -1)
+    return HTS_FALSE;
+  if (pid == 0) {
+    static char llvm_prog[] = "llvm-symbolizer";
+    static char llvm_opts[] = "-p";
+
+    dup2(fd, 1); /* both symbolizers write on stdout */
+    execvp(argv[0], argv);
+    argv[0] = llvm_prog; /* an LLVM-only install ships no addr2line */
+    argv[1] = llvm_opts;
+    execvp(argv[0], argv);
+    _exit(BT_NO_SYMBOLIZER);
+  }
+  for (; *budget > 0; (*budget)--) {
+    const struct timespec tick = {0, 10 * 1000 * 1000};
+    const pid_t reaped = waitpid(pid, &status, WNOHANG);
+
+    if (reaped == pid)
+      return WIFEXITED(status) && WEXITSTATUS(status) == BT_NO_SYMBOLIZER
+                 ? HTS_FALSE
+                 : HTS_TRUE;
+    if (reaped == -1 && errno != EINTR)
+      return HTS_TRUE;
+    nanosleep(&tick, NULL);
+  }
+  kill(pid, SIGKILL);
+  waitpid(pid, NULL, 0);
+  return HTS_TRUE;
+}
+
+/* Name the frames backtrace_symbols_fd() leaves as module+offset:
+   -fvisibility=hidden keeps them out of .dynsym, but DWARF has them. dladdr()
+   is not formally async-signal-safe; accepted, this path is already fatal. */
+static void symbolize_backtrace(void *const *stack, int size, int fd) {
+  static char prog[] = "addr2line";
+  static char opts[] = "-Cfipa";
+  static char dashe[] = "-e";
+  char hex[BT_MAX_FRAMES][BT_HEX_SIZE];
+  const void *base[BT_MAX_FRAMES];
+  const char *name[BT_MAX_FRAMES];
+  hts_boolean grouped[BT_MAX_FRAMES];
+  char module[BT_PATH_SIZE];
+  char *argv[4 + BT_MAX_FRAMES + 1];
+  int budget = BT_WAIT_TICKS;
+  int i, spawned;
+
+  if (size > BT_MAX_FRAMES)
+    size = BT_MAX_FRAMES;
+
+  for (i = 0; i < size; i++) {
+    Dl_info info;
+
+    grouped[i] = HTS_TRUE; /* skipped unless dladdr() places the frame */
+    if (dladdr(stack[i], &info) == 0 || info.dli_fname == NULL ||
+        info.dli_fname[0] == '\0')
+      continue;
+    base[i] = info.dli_fbase;
+    name[i] = info.dli_fname;
+    print_hex(hex[i], (uintptr_t) ((const char *) stack[i] -
+                                   (const char *) info.dli_fbase));
+    grouped[i] = HTS_FALSE;
+  }
+
+  /* One child per module: addr2line takes a single -e. Each frame is claimed
+     once, so argc cannot exceed argv[]. */
+  for (spawned = 0; spawned < BT_MAX_MODULES; spawned++) {
+    int first, j, argc = 0;
+
+    for (first = 0; first < size && grouped[first]; first++)
+      ;
+    if (first >= size)
+      break;
+    argv[argc++] = prog;
+    argv[argc++] = opts;
+    argv[argc++] = dashe;
+    argv[argc++] = module;
+    for (j = first; j < size; j++) {
+      if (grouped[j] || base[j] != base[first])
+        continue;
+      grouped[j] = HTS_TRUE;
+      argv[argc++] = hex[j];
+    }
+    argv[argc] = NULL;
+    /* access(): skip pseudo-modules like linux-vdso, which have no file and
+       would draw nothing but an addr2line complaint. */
+    if (copy_bounded(module, sizeof(module), name[first]) &&
+        access(module, R_OK) == 0) {
+      const size_t len = strlen(module);
+
+      /* addr2line -a prints offsets only: say which module they are in. */
+      (void) (write(fd, module, len) == (ssize_t) len);
+      (void) (write(fd, ":\n", 2) == 2);
+      if (!spawn_symbolizer(argv, fd, &budget))
+        break; /* no symbolizer: stop at one header */
+    }
+  }
+}
+#endif
+
+void hts_backtrace_init(void) {
+#ifdef USES_BACKTRACE
+  symbolize_crash =
+      getenv("HTTRACK_NO_SYMBOLIZE") == NULL ? HTS_TRUE : HTS_FALSE;
+#endif
+}
+
+void hts_print_backtrace(int fd) {
+#ifdef USES_BACKTRACE
+  void *stack[256];
+  const int size = backtrace(stack, sizeof(stack) / sizeof(stack[0]));
+
+  /* A fault inside the handler lands back here: symbolizing twice interleaves
+     two traces on fd and spends a second budget. */
+  static volatile sig_atomic_t entered = 0;
+
+  if (size != 0) {
+    backtrace_symbols_fd(stack, size, fd);
+    if (symbolize_crash && entered == 0) {
+      entered = 1;
+      symbolize_backtrace(stack, size, fd);
+      entered = 0;
+    }
+  }
+#else
+  const char msg[] = "No stack trace available on this OS :(\n";
+
+  if (write(fd, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
+    /* sorry GCC */
+  }
+#endif
+}

src/htsbacktrace.h

@@ -0,0 +1,45 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: crash backtrace printer                                 */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#ifndef HTSBACKTRACE_DEFH
+#define HTSBACKTRACE_DEFH
+
+/* Sample HTTRACK_NO_SYMBOLIZE before any crash: getenv() is not signal-safe.
+   Call once, from the process that installs the fatal-signal handlers. */
+void hts_backtrace_init(void);
+
+/* Write the calling thread's stack to fd, callable from a fatal signal handler:
+   raw frames first, then whatever an external symbolizer can name. Allocates
+   nothing; prints a one-line notice where the OS has no backtrace(). */
+void hts_print_backtrace(int fd);
+
+#endif

src/htscache.c

@@ -142,10 +142,12 @@ struct cache_back_zip_entry {
   int compressionMethod;
 };
 
+/* A corrupt cache can carry a field wider than ours; clipping it keeps the
+   entry, where aborting would take the crawl down. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      strlcpybuff(refvalue, value, refvalue_size);                             \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -629,8 +631,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
                 // File exists on disk with declared cache name (this is expected!)
                 if (fexist_utf8(fconv(catbuff, sizeof(catbuff), previous_save))) {       // un fichier existe déja
                   // Expected size ?
-                  const size_t fsize =
-                    fsize_utf8(fconv(catbuff, sizeof(catbuff), previous_save));
+                  const LLint fsize = fsize_utf8(
+                      fconv(catbuff, sizeof(catbuff), previous_save));
                   if (fsize == r.size) {
                     // Target name is the previous name, and the file looks good: nothing to do!
                     if (strcmp(previous_save, target_save) == 0) {
@@ -666,7 +668,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
                 // Suppose a broken mirror, with a file being renamed: OK
                 else if (fexist_utf8(fconv(catbuff, sizeof(catbuff), target_save))) {
                   // Expected size ?
-                  const size_t fsize = fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
+                  const LLint fsize =
+                      fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
 
                   if (fsize == r.size) {
                     // So far so good
@@ -1440,7 +1443,7 @@ int cache_brstr(char *adr, char *s, size_t s_size) {
 
 /* binput bounded to a NUL-terminated buffer: refuse to start a read at or
    past `end`, so a prior over-advance can't walk a cache-index parse OOB. */
-int cache_binput(char *adr, const char *end, char *s, int max) {
+int cache_binput(const char *adr, const char *end, char *s, int max) {
   if (adr >= end) {
     s[0] = '\0';
     return 0;

src/htscache.h

@@ -93,7 +93,7 @@ void cache_rstr(FILE *fp, char *s, size_t s_size);
 char *cache_rstr_addr(FILE * fp);
 int cache_brstr(char *adr, char *s, size_t s_size);
 /* binput over a NUL-terminated buffer, bounded: no read starts at/past end. */
-int cache_binput(char *adr, const char *end, char *s, int max);
+int cache_binput(const char *adr, const char *end, char *s, int max);
 int cache_brint(char *adr, int *i);
 void cache_rint(FILE * fp, int *i);
 void cache_rLLint(FILE * fp, LLint * i);

src/htscache_selftest.c

@@ -1195,6 +1195,12 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
 
 /* --- read-side corruption injection --------------------------------------- */
 
+/* 100 'A's: a placeholder header line long enough to be overwritten by a
+   forged, over-long X-StatusMessage of the same byte length. */
+#define CORRUPT_LONG_ETAG                                                      \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"                         \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+
 /* canary read back intact after each corruption; victim gets the byte surgery
  */
 #define CORRUPT_ADR "corrupt.example.com"
@@ -1219,6 +1225,24 @@ static void corrupt_build(httrackp *opt) {
   selftest_close(&cache);
 }
 
+/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
+static void corrupt_build_longetag(httrackp *opt) {
+  cache_back cache;
+
+  memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
+  memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
+  remove(reconcile_st_path(opt, "hts-cache/new.zip"));
+  remove(reconcile_st_path(opt, "hts-cache/old.zip"));
+  selftest_open_for_write(&cache, opt);
+  store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
+              "OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
+              strlen(corrupt_body_a));
+  store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
+              "OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
+              corrupt_body_b, strlen(corrupt_body_b));
+  selftest_close(&cache);
+}
+
 /* Like corrupt_build, but the victim carries a 20-char Etag whose header line
    is later overwritten with a forged oversized X-Size (same byte length). */
 static void corrupt_build_etag(httrackp *opt) {
@@ -1403,6 +1427,48 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
   return fail;
 }
 
+/* An over-long field from a foreign cache must clip, not abort: the entry
+   still reads, clipped to capacity, and the canary survives. */
+static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
+                                         size_t wantlastmod, const char *what) {
+  cache_back cache;
+  htsblk v, c;
+  char BIGSTK lv[HTS_URLMAXSIZE * 2];
+  char BIGSTK lc[HTS_URLMAXSIZE * 2];
+  int fail = 0;
+
+  selftest_open_for_read(&cache, opt);
+  lv[0] = lc[0] = '\0';
+  v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
+  if (v.statuscode != 200) {
+    fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
+            v.statuscode);
+    fail++;
+  }
+  if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
+    fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
+            (unsigned) strlen(v.msg), (unsigned) wantmsg);
+    fail++;
+  }
+  if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
+    fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
+            what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
+    fail++;
+  }
+  c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
+  if (c.statuscode != 200) {
+    fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
+            c.statuscode);
+    fail++;
+  }
+  if (v.adr != NULL)
+    freet(v.adr);
+  if (c.adr != NULL)
+    freet(c.adr);
+  selftest_close(&cache);
+  return fail;
+}
+
 /* One zip corruption case: build, patch, then check victim+canary in-session.
  */
 static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1439,6 +1505,31 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
   failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
                                     "garbled deflate stream");
 
+  /* A corrupt cache can hold a field wider than ours. Clipping keeps the
+     entry; aborting would take the crawl down. Overwrite the placeholder Etag
+     line in place, same byte length, so the zip offsets stay intact. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "X-StatusMessage: " /* 17 + 89 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures +=
+      corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
+                                    (size_t) -1, "over-long X-StatusMessage");
+
+  /* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
+     cannot satisfy both. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "Last-Modified: " /* 15 + 91 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures += corrupt_expect_victim_clipped(
+      opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
+      "over-long Last-Modified");
+
   /* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
      truncates negative in the (int) cast the malloc uses: a wraparound alloc.
      cache_add asserts size fits an int, so such a value only reaches the reader

src/htschanges.c

@@ -0,0 +1,679 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: htschanges.c subroutines:                              */
+/*       --changes: what this crawl changed vs. the previous    */
+/*       mirror (hts-changes.json)                              */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#define HTS_INTERNAL_BYTECODE
+
+#include "htschanges.h"
+
+#include "htscache.h"
+#include "htscharset.h"
+#include "htscore.h"
+#include "htslib.h"
+#include "htsmd5.h"
+#include "htssafe.h"
+#include "htsthread.h"
+#include "htstools.h"
+#include "coucal/coucal.h"
+#include "md5.h"
+
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+
+#define DIGEST_SIZE 16
+
+typedef struct hts_changes hts_changes;
+
+/* One mirrored file. The mirror-relative path is the key, not the URL: a
+   redirect and its target can share a save name, and counting that file twice
+   would put it in two buckets. */
+typedef struct {
+  char *url;               /* absolute URL, "" for an engine-generated file */
+  char *file;              /* mirror-relative path, as listed in new.lst */
+  hts_boolean rewritten;   /* the crawl wrote over the local copy */
+  hts_boolean not_updated; /* transfer signal: the server reported no change */
+  hts_boolean existed;     /* the file was part of the previous mirror */
+  hts_boolean on_disk;     /* a copy was on disk when the crawl first saw it */
+  hts_boolean listed_prev; /* the previous mirror's index listed this file */
+  hts_boolean has_prev;    /* prev_digest holds the previous payload's digest */
+  hts_boolean has_new;     /* new_digest was taken from the payload, not disk */
+  unsigned char prev_digest[DIGEST_SIZE];
+  unsigned char new_digest[DIGEST_SIZE];
+  LLint prev_size; /* -1 when unknown */
+  LLint size;      /* size after the crawl, -1 when unknown */
+  hts_change_bucket bucket;
+} changes_entry;
+
+struct hts_changes {
+  coucal index; /* mirror-relative path -> entry slot + 1 */
+  changes_entry *entries;
+  size_t count;
+  size_t capacity;
+  hts_boolean has_index; /* the run keeps a mirror index (the cache is on) */
+  hts_boolean old_index; /* the previous mirror's index was read */
+  hts_boolean overflow;  /* an allocation failed; the report is partial */
+  hts_boolean closed;    /* reported: late notifies are dropped, not recorded */
+};
+
+/* ------------------------------------------------------------ */
+/* JSON                                                          */
+/* ------------------------------------------------------------ */
+
+/* Length of the UTF-8 sequence led by c, or 0 if c cannot lead one. */
+static size_t utf8_lead_length(unsigned char c) {
+  if (c >= 0xc2 && c <= 0xdf)
+    return 2;
+  if (c >= 0xe0 && c <= 0xef)
+    return 3;
+  if (c >= 0xf0 && c <= 0xf4)
+    return 4;
+  return 0;
+}
+
+void hts_changes_json_string(String *out, const char *s) {
+  const unsigned char *p = (const unsigned char *) s;
+
+  StringAddchar(*out, '"');
+  while (*p != '\0') {
+    const unsigned char c = *p;
+
+    if (c == '"' || c == '\\') {
+      StringAddchar(*out, '\\');
+      StringAddchar(*out, (char) c);
+      p++;
+    } else if (c < 0x20 || c == 0x7f) {
+      char esc[8];
+
+      snprintf(esc, sizeof(esc), "\\u%04x", (unsigned) c);
+      StringCat(*out, esc);
+      p++;
+    } else if (c < 0x80) {
+      StringAddchar(*out, (char) c);
+      p++;
+    } else {
+      /* Non-ASCII: emit it verbatim only if it is a well-formed sequence, so
+         a legacy-charset URL cannot produce unparseable JSON. */
+      const size_t len = utf8_lead_length(c);
+
+      if (len != 0 && strnlen((const char *) p, len) == len &&
+          hts_isStringUTF8((const char *) p, len)) {
+        StringMemcat(*out, (const char *) p, len);
+        p += len;
+      } else {
+        StringCat(*out, "\\ufffd");
+        p++;
+      }
+    }
+  }
+  StringAddchar(*out, '"');
+}
+
+/* ------------------------------------------------------------ */
+/* Accumulator                                                   */
+/* ------------------------------------------------------------ */
+
+static hts_changes *changes_new(void) {
+  hts_changes *changes = calloct(1, sizeof(*changes));
+
+  if (changes == NULL)
+    return NULL;
+  changes->index = coucal_new(0);
+  if (changes->index == NULL) {
+    freet(changes);
+    return NULL;
+  }
+  return changes;
+}
+
+static void changes_free(hts_changes **pchanges) {
+  hts_changes *changes = *pchanges;
+
+  if (changes == NULL)
+    return;
+  if (changes->entries != NULL) {
+    size_t i;
+
+    for (i = 0; i < changes->count; i++) {
+      freet(changes->entries[i].url);
+      freet(changes->entries[i].file);
+    }
+    freet(changes->entries);
+  }
+  if (changes->index != NULL)
+    coucal_delete(&changes->index);
+  freet(changes);
+  *pchanges = NULL;
+}
+
+/* Slot of `file`, or -1 when it has not been recorded. */
+static intptr_t changes_find(const hts_changes *changes, const char *file) {
+  intptr_t slot = 0;
+
+  return coucal_read(changes->index, file, &slot) ? slot - 1 : -1;
+}
+
+/* Slot of `file`, appending a fresh entry when it is new; -1 if allocation
+   failed, which flags the report partial. */
+static intptr_t changes_slot(hts_changes *changes, const char *file,
+                             hts_boolean *is_new) {
+  intptr_t slot = 0;
+
+  *is_new = HTS_FALSE;
+  if (coucal_read(changes->index, file, &slot))
+    return slot - 1;
+
+  if (changes->count == changes->capacity) {
+    const size_t capacity = changes->capacity != 0 ? changes->capacity * 2 : 64;
+    changes_entry *const entries =
+        realloct(changes->entries, capacity * sizeof(*entries));
+
+    if (entries == NULL) {
+      changes->overflow = HTS_TRUE;
+      return -1;
+    }
+    changes->entries = entries;
+    changes->capacity = capacity;
+  }
+  slot = (intptr_t) changes->count;
+  memset(&changes->entries[slot], 0, sizeof(changes->entries[slot]));
+  changes->entries[slot].file = strdupt(file);
+  changes->entries[slot].prev_size = -1;
+  changes->entries[slot].size = -1;
+  if (changes->entries[slot].file == NULL) {
+    changes->overflow = HTS_TRUE;
+    return -1;
+  }
+  changes->count++;
+  coucal_write(changes->index, file, slot + 1);
+  *is_new = HTS_TRUE;
+  return slot;
+}
+
+/* MD5 of the file at `path`; not a security boundary. Never call it holding
+   changes_mutex: hashing a large file would stall every other connection.
+   HTS_FALSE if it cannot be read. */
+static hts_boolean digest_file(const char *path,
+                               unsigned char digest[DIGEST_SIZE]) {
+  const int endian = 1;
+  struct MD5Context ctx;
+  char BIGSTK buffer[32768];
+  FILE *fp = FOPEN(path, "rb");
+  size_t nread;
+
+  if (fp == NULL)
+    return HTS_FALSE;
+  MD5Init(&ctx, *((const char *) &endian));
+  while ((nread = fread(buffer, 1, sizeof(buffer), fp)) > 0)
+    MD5Update(&ctx, (const unsigned char *) buffer, (unsigned int) nread);
+  if (ferror(fp) != 0) {
+    fclose(fp);
+    return HTS_FALSE;
+  }
+  fclose(fp);
+  MD5Final(digest, &ctx);
+  return HTS_TRUE;
+}
+
+static void digest_mem(const char *buffer, size_t len,
+                       unsigned char digest[DIGEST_SIZE]) {
+  domd5mem(buffer, len, (char *) digest, 0);
+}
+
+hts_change_bucket hts_changes_classify(hts_boolean rewritten,
+                                       hts_boolean existed,
+                                       hts_boolean not_updated,
+                                       hts_boolean have_digests,
+                                       hts_boolean digests_equal) {
+  if (!rewritten)
+    return HTS_CHANGE_UNCHANGED; /* the crawl left the copy alone */
+  if (!existed)
+    return HTS_CHANGE_NEW;
+  if (have_digests)
+    return digests_equal ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED;
+  /* No digest to compare: a server with no validators answers 200 with the
+     same bytes, so this signal alone over-reports. */
+  return not_updated ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED;
+}
+
+/* ------------------------------------------------------------ */
+/* Engine hooks                                                  */
+/* ------------------------------------------------------------ */
+
+/* FTP transfers reach file_notify() from a thread the crawl never joins, so
+   every entry point below, report and teardown included, takes this lock. */
+static htsmutex changes_mutex = HTSMUTEX_INIT;
+
+/* The live accumulator, created on first use; NULL when --changes is off or
+   the report is already written. Call under the lock. */
+static hts_changes *changes_get(httrackp *opt) {
+  hts_changes *changes = (hts_changes *) opt->changes_state;
+
+  if (!opt->changes)
+    return NULL;
+  if (changes == NULL) {
+    changes = changes_new();
+    opt->changes_state = changes;
+  }
+  return changes != NULL && !changes->closed ? changes : NULL;
+}
+
+void hts_changes_notify(httrackp *opt, const char *adr, const char *fil,
+                        const char *save, hts_boolean rewritten,
+                        hts_boolean not_updated) {
+  hts_changes *changes;
+  char BIGSTK file[HTS_URLMAXSIZE * 2];
+  char BIGSTK url[HTS_URLMAXSIZE * 4 + 8]; /* holds adr + fil + a scheme */
+  unsigned char prev_digest[DIGEST_SIZE];
+  hts_boolean has_prev = HTS_FALSE;
+  hts_boolean is_new = HTS_FALSE;
+  intptr_t slot = -1;
+  LLint prev_size;
+
+  if (!opt->changes)
+    return;
+  /* Engine-generated scaffolding (the top index) carries no URL and is not a
+     mirrored resource. */
+  if (save == NULL || !strnotempty(save) || adr == NULL || fil == NULL ||
+      (!strnotempty(adr) && !strnotempty(fil)))
+    return;
+
+  hts_savename_listed(StringBuff(opt->path_html), save, file, sizeof(file));
+
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL) {
+    slot = changes_slot(changes, file, &is_new);
+    if (slot >= 0 && !is_new) {
+      /* Retry, or a second call site for the same file: the pre-run state was
+         already sampled and the copy on disk may no longer be it. */
+      changes->entries[slot].rewritten =
+          changes->entries[slot].rewritten || rewritten;
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+  if (slot < 0 || !is_new)
+    return;
+
+  url[0] = '\0';
+  if (!link_has_authority(adr))
+    strlcatbuff(url, "http://", sizeof(url));
+  strlcatbuff(url, adr, sizeof(url));
+  strlcatbuff(url, fil, sizeof(url));
+
+  /* Unlocked: hashing a large file would stall every other connection. Hash
+     it only when it is about to be overwritten, the last moment it exists. */
+  prev_size = fsize_utf8(save);
+  if (prev_size >= 0 && rewritten)
+    has_prev = digest_file(save, prev_digest);
+
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  /* The slot index is stable, entries being append-only; the array is not. */
+  if (changes != NULL && (size_t) slot < changes->count) {
+    changes_entry *const entry = &changes->entries[slot];
+
+    entry->url = strdupt(url);
+    if (entry->url == NULL)
+      changes->overflow = HTS_TRUE;
+    entry->rewritten = entry->rewritten || rewritten;
+    entry->not_updated = not_updated;
+    entry->prev_size = prev_size;
+    entry->on_disk = prev_size >= 0;
+    /* hts_changes_html() compares payloads, and its digests win. */
+    if (has_prev && !entry->has_new) {
+      entry->has_prev = HTS_TRUE;
+      memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE);
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r,
+                      const char *adr, const char *fil, const char *save) {
+  hts_changes *changes;
+  char BIGSTK file[HTS_URLMAXSIZE * 2];
+  char BIGSTK location[HTS_URLMAXSIZE * 2];
+  unsigned char new_digest[DIGEST_SIZE];
+  unsigned char prev_digest[DIGEST_SIZE];
+  hts_boolean has_prev = HTS_FALSE;
+  intptr_t slot;
+  htsblk prev;
+
+  /* Ahead of the digest and the cache read: off must cost nothing. */
+  if (!opt->changes)
+    return;
+  if (r->adr == NULL || r->size < 0 || save == NULL || !strnotempty(save))
+    return;
+
+  /* On disk this is the payload plus rewritten links and a footer dated by the
+     crawl, so it differs every run; compare payloads, the previous one being
+     the body the cache kept. */
+  digest_mem(r->adr, (size_t) r->size, new_digest);
+  prev = cache_read_ro(opt, cache, adr, fil, "", location);
+  if (HTTP_IS_OK(prev.statuscode) && prev.adr != NULL && prev.size >= 0) {
+    digest_mem(prev.adr, (size_t) prev.size, prev_digest);
+    has_prev = HTS_TRUE;
+  }
+  freet(prev.adr);
+
+  /* Only now take the lock and look the entry up: cache_read_ro() can itself
+     reach file_notify(), which would move the entries array. */
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  /* file_notify() records the entry; this call only refines it. */
+  if (changes != NULL) {
+    hts_savename_listed(StringBuff(opt->path_html), save, file, sizeof(file));
+    slot = changes_find(changes, file);
+    if (slot >= 0) {
+      changes_entry *const entry = &changes->entries[slot];
+
+      entry->has_new = HTS_TRUE;
+      memcpy(entry->new_digest, new_digest, DIGEST_SIZE);
+      entry->has_prev = has_prev;
+      if (has_prev)
+        memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE);
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept) {
+  hts_changes *changes;
+  hts_boolean is_new;
+  intptr_t slot;
+
+  if (!opt->changes || file == NULL || !strnotempty(file))
+    return;
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL) {
+    slot = changes_slot(changes, file, &is_new);
+    if (slot >= 0 && is_new) {
+      changes->entries[slot].url = strdupt("");
+      changes->entries[slot].listed_prev = HTS_TRUE;
+      /* A kept file leaves rewritten clear, which resolves to unchanged. */
+      if (!kept)
+        changes->entries[slot].bucket = HTS_CHANGE_GONE;
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_previous(httrackp *opt, const char *file) {
+  hts_changes *changes;
+  intptr_t slot;
+
+  if (!opt->changes || file == NULL)
+    return;
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL) {
+    changes->has_index = HTS_TRUE;
+    changes->old_index = HTS_TRUE;
+    slot = changes_find(changes, file);
+    if (slot >= 0)
+      changes->entries[slot].listed_prev = HTS_TRUE;
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_indexed(httrackp *opt) {
+  hts_changes *changes;
+
+  if (!opt->changes)
+    return;
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL)
+    changes->has_index = HTS_TRUE;
+  hts_mutexrelease(&changes_mutex);
+}
+
+/* ------------------------------------------------------------ */
+/* Report                                                        */
+/* ------------------------------------------------------------ */
+
+/* Assign each entry its final bucket from the bytes now on disk. */
+static void changes_resolve(hts_changes *changes, httrackp *opt) {
+  char catbuff[CATBUFF_SIZE];
+  size_t i;
+
+  for (i = 0; i < changes->count; i++) {
+    changes_entry *const entry = &changes->entries[i];
+    const char *path;
+    unsigned char digest[DIGEST_SIZE];
+    hts_boolean have_digests = HTS_FALSE;
+    hts_boolean digests_equal = HTS_FALSE;
+
+    if (entry->bucket == HTS_CHANGE_GONE)
+      continue;
+
+    /* The previous mirror's index is the authority on what was there before:
+       a partial left by this crawl's own failed attempt is on disk but was
+       never part of the previous mirror. Without an index, fall back to what
+       the first notify saw on disk. */
+    entry->existed = changes->old_index ? entry->listed_prev : entry->on_disk;
+    path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_html),
+                   entry->file);
+    entry->size = fsize_utf8(path);
+    if (entry->rewritten && entry->existed) {
+      /* The size shortcut only holds when both digests describe the file on
+         disk; has_new means they are payload digests, and a parsed page's
+         rendered size moves with its links and footer. */
+      if (!entry->has_new && entry->size >= 0 && entry->prev_size >= 0 &&
+          entry->size != entry->prev_size) {
+        have_digests = HTS_TRUE; /* different lengths: no need to hash */
+        digests_equal = HTS_FALSE;
+      } else if (entry->has_prev &&
+                 (entry->has_new
+                      ? (memcpy(digest, entry->new_digest, DIGEST_SIZE), 1)
+                      : digest_file(path, digest))) {
+        have_digests = HTS_TRUE;
+        digests_equal = memcmp(digest, entry->prev_digest, DIGEST_SIZE) == 0
+                            ? HTS_TRUE
+                            : HTS_FALSE;
+      }
+    }
+    entry->bucket =
+        hts_changes_classify(entry->rewritten, entry->existed,
+                             entry->not_updated, have_digests, digests_equal);
+  }
+}
+
+static const char *const bucket_names[HTS_CHANGE_BUCKETS] = {
+    "new", "changed", "unchanged", "gone"};
+
+/* Serialize the report. Call under changes_mutex. */
+static void changes_serialize(httrackp *opt, String *out) {
+  hts_changes *const changes = (hts_changes *) opt->changes_state;
+  size_t counts[HTS_CHANGE_BUCKETS];
+  char date[32];
+  char scratch[64];
+  int bucket;
+  size_t i;
+
+  StringClear(*out);
+  if (changes == NULL)
+    return;
+  changes_resolve(changes, opt);
+
+  memset(counts, 0, sizeof(counts));
+  for (i = 0; i < changes->count; i++)
+    counts[changes->entries[i].bucket]++;
+
+  hts_now_iso8601(date);
+  StringCat(*out, "{\n  \"schema\": ");
+  snprintf(scratch, sizeof(scratch), "%d", HTS_CHANGES_SCHEMA);
+  StringCat(*out, scratch);
+  StringCat(*out, ",\n  \"generator\": ");
+  hts_changes_json_string(out, "HTTrack Website Copier/" HTTRACK_VERSION);
+  StringCat(*out, ",\n  \"date\": ");
+  hts_changes_json_string(out, date);
+  StringCat(*out, ",\n  \"first_crawl\": ");
+  StringCat(*out, !changes->has_index
+                      ? "null"
+                      : (changes->old_index ? "false" : "true"));
+  StringCat(*out, ",\n  \"partial\": ");
+  StringCat(*out, changes->overflow ? "true" : "false");
+  StringCat(*out, ",\n  \"purged\": ");
+  StringCat(*out, opt->delete_old ? "true" : "false");
+
+  StringCat(*out, ",\n  \"counts\": {");
+  for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) {
+    StringCat(*out, bucket != 0 ? ", " : " ");
+    hts_changes_json_string(out, bucket_names[bucket]);
+    snprintf(scratch, sizeof(scratch), ": %d", (int) counts[bucket]);
+    StringCat(*out, scratch);
+  }
+  StringCat(*out, " }");
+
+  for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) {
+    hts_boolean first = HTS_TRUE;
+
+    StringCat(*out, ",\n  ");
+    hts_changes_json_string(out, bucket_names[bucket]);
+    StringCat(*out, ": [");
+    for (i = 0; i < changes->count; i++) {
+      const changes_entry *const entry = &changes->entries[i];
+
+      if (entry->bucket != bucket)
+        continue;
+      StringCat(*out, first ? "\n    { \"url\": " : ",\n    { \"url\": ");
+      first = HTS_FALSE;
+      hts_changes_json_string(out, entry->url != NULL ? entry->url : "");
+      StringCat(*out, ", \"file\": ");
+      hts_changes_json_string(out, entry->file);
+      if (entry->size >= 0) {
+        snprintf(scratch, sizeof(scratch), ", \"size\": " LLintP,
+                 (LLint) entry->size);
+        StringCat(*out, scratch);
+      }
+      if (bucket == HTS_CHANGE_CHANGED && entry->prev_size >= 0) {
+        snprintf(scratch, sizeof(scratch), ", \"previous_size\": " LLintP,
+                 (LLint) entry->prev_size);
+        StringCat(*out, scratch);
+      }
+      StringCat(*out, " }");
+    }
+    StringCat(*out, first ? "]" : "\n  ]");
+  }
+  StringCat(*out, "\n}\n");
+}
+
+void hts_changes_report(httrackp *opt, String *out) {
+  hts_mutexlock(&changes_mutex);
+  changes_serialize(opt, out);
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_close_opt(httrackp *opt) {
+  char catbuff[CATBUFF_SIZE];
+  const char *path;
+  String report = STRING_EMPTY;
+  size_t counts[HTS_CHANGE_BUCKETS];
+  size_t total;
+  hts_boolean has_index, old_index;
+  FILE *fp;
+
+  hts_mutexlock(&changes_mutex);
+  {
+    /* changes_get(), not the raw field: a crawl that mirrored nothing still
+       owes the user a report, and a stale one from the previous run must not
+       survive on disk as if it described this one. */
+    hts_changes *const changes = changes_get(opt);
+    size_t i;
+
+    if (changes == NULL) {
+      hts_mutexrelease(&changes_mutex);
+      return;
+    }
+    changes_serialize(opt, &report);
+    memset(counts, 0, sizeof(counts));
+    for (i = 0; i < changes->count; i++)
+      counts[changes->entries[i].bucket]++;
+    total = changes->count;
+    has_index = changes->has_index;
+    old_index = changes->old_index;
+    /* Sticky: whatever the crawl's stragglers do next is not in this report. */
+    changes->closed = HTS_TRUE;
+  }
+  hts_mutexrelease(&changes_mutex);
+
+  path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_log),
+                 HTS_CHANGES_FILE);
+  fp = FOPEN(path, "wb");
+  if (fp != NULL) {
+    const size_t len = StringLength(report);
+
+    if (len != 0 && fwrite(StringBuff(report), 1, len, fp) != len)
+      hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
+                    "Unable to write the change report %s", path);
+    fclose(fp);
+  } else {
+    hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
+                  "Unable to create the change report %s", path);
+  }
+
+  if (!has_index) {
+    hts_log_print(opt, LOG_NOTICE,
+                  "Change report: no mirror index (the cache is off), %d files "
+                  "mirrored, deletions not detected (%s)",
+                  (int) total, HTS_CHANGES_FILE);
+  } else if (!old_index) {
+    hts_log_print(opt, LOG_NOTICE,
+                  "Change report: first crawl, %d files mirrored, nothing to "
+                  "compare against (%s)",
+                  (int) total, HTS_CHANGES_FILE);
+  } else {
+    hts_log_print(opt, LOG_NOTICE,
+                  "Change report: %d new, %d changed, %d unchanged, %d gone "
+                  "(%s)",
+                  (int) counts[HTS_CHANGE_NEW],
+                  (int) counts[HTS_CHANGE_CHANGED],
+                  (int) counts[HTS_CHANGE_UNCHANGED],
+                  (int) counts[HTS_CHANGE_GONE], HTS_CHANGES_FILE);
+  }
+
+  StringFree(report);
+}
+
+void hts_changes_free_opt(httrackp *opt) {
+  hts_changes *changes;
+
+  hts_mutexlock(&changes_mutex);
+  changes = (hts_changes *) opt->changes_state;
+  changes_free(&changes);
+  opt->changes_state = NULL;
+  hts_mutexrelease(&changes_mutex);
+}

src/htschanges.h

@@ -0,0 +1,125 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack change report (--changes). Internal, not installed.
+   Accumulates what the crawl did to each mirrored file, compares the bytes
+   against the copy the previous run left behind, and writes
+   hts-changes.json next to the log. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_CHANGES_DEFH
+#define HTS_CHANGES_DEFH
+
+#include "htsopt.h"
+#include "htsstrings.h"
+
+#ifndef HTS_DEF_FWSTRUCT_cache_back
+#define HTS_DEF_FWSTRUCT_cache_back
+typedef struct cache_back cache_back;
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Report file name, written under the project's log directory. */
+#define HTS_CHANGES_FILE "hts-changes.json"
+
+/* Schema version carried by the report; bump on an incompatible change. */
+#define HTS_CHANGES_SCHEMA 1
+
+/* Which side of the comparison a mirrored file ended up on. */
+typedef enum {
+  HTS_CHANGE_NEW = 0,   /* no local copy before this run */
+  HTS_CHANGE_CHANGED,   /* rewritten, and the bytes differ */
+  HTS_CHANGE_UNCHANGED, /* the bytes are those of the previous mirror */
+  HTS_CHANGE_GONE,      /* in the previous mirror, absent from this one */
+  HTS_CHANGE_BUCKETS
+} hts_change_bucket;
+
+/* Record what this crawl is doing to the local file `save` (absolute path;
+   adr/fil form its URL, either may be empty for an engine-generated file).
+   `rewritten` means the copy on disk is being written over, `not_updated` is
+   the 200-versus-304 signal, used only when no digest can be taken. Only the
+   first call for a file samples the previous copy, so a retried or
+   twice-notified resource is counted once. No-op unless --changes is on. */
+void hts_changes_notify(httrackp *opt, const char *adr, const char *fil,
+                        const char *save, hts_boolean rewritten,
+                        hts_boolean not_updated);
+
+/* Refine the entry for a parsed HTML file: its local copy carries rewritten
+   links and a footer dated by the crawl, so it differs every run. Compares the
+   payload `r` holds against the body the previous run left in the cache. */
+void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r,
+                      const char *adr, const char *fil, const char *save);
+
+/* Record `file` (mirror-relative, as listed in new.lst) as listed by the
+   previous mirror's index and absent from this run's. `kept` means its local
+   copy survives the run because the crawl tried and failed to replace it: a
+   failed transfer is not a deletion, so it is reported unchanged, not gone. */
+void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept);
+
+/* Record that the previous mirror's index listed `file`. That index, not the
+   file's presence on disk, decides what counts as already mirrored. */
+void hts_changes_previous(httrackp *opt, const char *file);
+
+/* Record that this run keeps a mirror index. Without one (the cache is off)
+   deletions and first_crawl are undecidable, and the report says so. */
+void hts_changes_indexed(httrackp *opt);
+
+/* Resolve every entry against the bytes now on disk and serialize the report
+   into `out` (replaced). Exposed for the self-tests. */
+void hts_changes_report(httrackp *opt, String *out);
+
+/* Write the report and log a one-line summary. Idempotent, and seals the
+   accumulator: a late notify is dropped rather than starting a second one. */
+void hts_changes_close_opt(httrackp *opt);
+
+/* Drop the accumulator, for a run that never reached its end and to start the
+   next one clean. Null-safe and idempotent. */
+void hts_changes_free_opt(httrackp *opt);
+
+/* Bucket for one entry, from what was observed. No filesystem access:
+   `have_digests` says both sides could be hashed, `digests_equal` compares
+   them, and `not_updated` is the fallback signal when they could not. */
+hts_change_bucket hts_changes_classify(hts_boolean rewritten,
+                                       hts_boolean existed,
+                                       hts_boolean not_updated,
+                                       hts_boolean have_digests,
+                                       hts_boolean digests_equal);
+
+/* Append `s` to `out` as a quoted JSON string. Byte sequences that are not
+   valid UTF-8 become U+FFFD, so a mirror carrying legacy-charset URLs still
+   produces parseable JSON. */
+void hts_changes_json_string(String *out, const char *s);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/htscmdline.c

@@ -0,0 +1,95 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: command line splitter, shared by the engine and         */
+/*       htsserver                                               */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#include "htscmdline.h"
+
+#include "htssafe.h"
+
+#include <limits.h>
+#include <stdint.h>
+
+char **hts_split_cmdline(char *cmd, int *nargs) {
+  size_t nsep = 0;
+  size_t capacity;
+  size_t r;
+  size_t w;
+  int argc = 0;
+  hts_boolean quoted = HTS_FALSE;
+  char **argv;
+
+  *nargs = 0;
+
+  /* fold the other separators, so counting them sizes the vector exactly */
+  for (r = 0; cmd[r] != '\0'; r++) {
+    if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
+      cmd[r] = ' ';
+    }
+    if (cmd[r] == ' ') {
+      nsep++;
+    }
+  }
+
+  /* at most one argument per separator, plus the leading one and the NULL */
+  if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
+    return NULL;
+  }
+  capacity = nsep + 2;
+  argv = (char **) malloct(capacity * sizeof(char *));
+  if (argv == NULL) {
+    return NULL;
+  }
+
+  argv[argc++] = cmd;
+  for (r = 0, w = 0; cmd[r] != '\0';) {
+    if (quoted && cmd[r] == '\\' &&
+        (cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
+      r++;
+      cmd[w++] = cmd[r++];
+    } else if (cmd[r] == '\"') {
+      quoted = !quoted;
+      cmd[w++] = cmd[r++];
+    } else if (cmd[r] == ' ' && !quoted) {
+      cmd[w++] = '\0';
+      assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
+      argv[argc++] = cmd + w;
+      r++;
+    } else {
+      cmd[w++] = cmd[r++];
+    }
+  }
+  cmd[w] = '\0';
+  argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
+
+  *nargs = argc;
+  return argv;
+}

src/htscmdline.h

@@ -0,0 +1,45 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: command line splitter, shared by the engine and         */
+/*       htsserver                                               */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#ifndef HTSCMDLINE_DEFH
+#define HTSCMDLINE_DEFH
+
+#include "htsglobal.h"
+
+/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
+   argv[0] being the program name and quotes left for the engine to strip.
+   Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
+   entries), or NULL when it cannot be sized or allocated. */
+char **hts_split_cmdline(char *cmd, int *nargs);
+
+#endif

src/htscore.c

@@ -40,6 +40,7 @@ Please visit our Website: http://www.httrack.com
 /* File defs */
 #include "htscore.h"
 #include "htswarc.h"
+#include "htschanges.h"
 
 /* specific definitions */
 #include "htsbase.h"
@@ -447,13 +448,22 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
                           const char *fil) {
   if (!*makeindex_done) {
     if (*makeindex_fp) {
-      char BIGSTK tempo[1024];
+      /* sized off link_escaped below: at the old flat 1024 a long first link
+         produced a redirect to a clipped URL */
+      char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
       if (makeindex_links == 1) {
         char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
         escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
-        snprintf(tempo, sizeof(tempo),
-                 "<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
-                 link_escaped);
+        /* no redirect beats one pointing at a clipped URL */
+        if (!sprintfbuff(
+                tempo,
+                "<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
+                link_escaped)) {
+          hts_log_print(opt, LOG_WARNING,
+                        "index redirect omitted: first link too long (%s)",
+                        makeindex_firstlink);
+          tempo[0] = '\0';
+        }
       } else
         tempo[0] = '\0';
       hts_template_format(*makeindex_fp, template_footer,
@@ -479,6 +489,7 @@ void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r,
                           const char *adr, const char *fil, const char *save) {
   {
     file_notify(opt, adr, fil, save, 1, 1, r->notmodified);
+    hts_changes_html(opt, cache, r, adr, fil, save);
     *fp = filecreate(&opt->state.strc, save);
     if (*fp) {
       if (ht_len > 0 && fwrite(ht_buff, 1, ht_len, *fp) != ht_len) {
@@ -682,6 +693,9 @@ int httpmirror(char *url1, httrackp * opt) {
   // hash table
   opt->hash = &hash;
 
+  // a change report left by a previous crawl on this opt is not this one's
+  hts_changes_free_opt(opt);
+
   // initialize link heap
   hts_record_init(opt);
 
@@ -696,17 +710,19 @@ int httpmirror(char *url1, httrackp * opt) {
   // copier adresse(s) dans liste des adresses
   {
     char *a = url1;
-    int primary_len = 8192;
-
-    if (StringNotEmpty(opt->filelist)) {
-      primary_len += max(0, fsize_utf8(StringBuff(opt->filelist)) * 2);
-    }
-    primary_len += (int) strlen(url1) * 2;
+    const LLint list_sz = StringNotEmpty(opt->filelist)
+                              ? fsize_utf8(StringBuff(opt->filelist))
+                              : 0;
+    /* two bytes reserved per list byte; -1 makes an undoublable size refused */
+    const LLint list_room =
+        list_sz > 0 ? (list_sz <= INT64_MAX / 2 ? list_sz * 2 : -1) : 0;
+    const size_t primary_len =
+        llint_grow_size_t(8192 + strlen(url1) * 2, list_room, 0);
 
     // création de la première page, qui contient les liens de base à scanner
     // c'est plus propre et plus logique que d'entrer à la main les liens dans la pile
     // on bénéficie ainsi des vérifications et des tests du robot pour les liens "primaires"
-    primary = (char *) malloct(primary_len);
+    primary = primary_len != (size_t) -1 ? (char *) malloct(primary_len) : NULL;
     if (!primary) {
       printf("PANIC! : Not enough memory [%d]\n", __LINE__);
       XH_extuninit;
@@ -887,7 +903,7 @@ int httpmirror(char *url1, httrackp * opt) {
       }
 
       if (filelist_buff != NULL) {
-        int filelist_ptr = 0;
+        size_t filelist_ptr = 0;
         int n = 0;
         char BIGSTK line[HTS_URLMAXSIZE * 2];
 
@@ -2078,9 +2094,12 @@ int httpmirror(char *url1, httrackp * opt) {
   if (cache.lst) {
     fclose(cache.lst);
     cache.lst = opt->state.strc.lst = NULL;
-    if (opt->delete_old) {
+    /* old.lst minus new.lst is the set of files the previous mirror had and
+       this one does not. --changes reports it; only --purge-old acts on it. */
+    if (opt->delete_old || opt->changes) {
       FILE *old_lst, *new_lst;
 
+      hts_changes_indexed(opt);
       //
       opt->state._hts_in_html_parsing = 3;
       //
@@ -2106,21 +2125,37 @@ int httpmirror(char *url1, httrackp * opt) {
               int purge = 0;
 
               while(!feof(old_lst)) {
-                linput(old_lst, line, 1000);
-                if (!strstr(adr, line)) { // not found in the new list?
-                  char BIGSTK file[HTS_URLMAXSIZE * 2];
+                char BIGSTK file[HTS_URLMAXSIZE * 2];
 
-                  strcpybuff(file, StringBuff(opt->path_html));
-                  strcatbuff(file, line + 1);
-                  file[strlen(file) - 1] = '\0';
-                  if (fexist_utf8(file)) { // toujours sur disque: virer
-                    hts_log_print(opt, LOG_INFO, "Purging %s", file);
-                    UNLINK(file);
-                    purge = 1;
+                linput(old_lst, line, 1000);
+                if (!strnotempty(line))
+                  continue;
+                strcpybuff(file, StringBuff(opt->path_html));
+                strcatbuff(file, line + 1);
+                file[strlen(file) - 1] = '\0';
+                hts_changes_previous(opt, file + StringLength(opt->path_html));
+                if (!strstr(adr, line)) { // not found in the new list?
+                  if (fexist_utf8(file)) { // still on disk
+                    /* A link this crawl did try but never wrote (a transfer
+                       killed mid-flight) also drops out of new.lst. Unless it
+                       is about to be purged, its previous copy stands and the
+                       file is not gone. */
+                    const hts_boolean kept =
+                        !opt->delete_old &&
+                        hash_read(opt->hash, file, NULL,
+                                  HASH_STRUCT_FILENAME) >= 0;
+
+                    hts_changes_dropped(
+                        opt, file + StringLength(opt->path_html), kept);
+                    if (opt->delete_old) {
+                      hts_log_print(opt, LOG_INFO, "Purging %s", file);
+                      UNLINK(file);
+                      purge = 1;
+                    }
                   }
                 }
               }
-              {
+              if (opt->delete_old) { // emptied directories go with the files
                 fseek(old_lst, 0, SEEK_SET);
                 while(!feof(old_lst)) {
                   linput(old_lst, line, 1000);
@@ -2155,7 +2190,7 @@ int httpmirror(char *url1, httrackp * opt) {
                 }
               }
               //
-              if (!purge) {
+              if (opt->delete_old && !purge) {
                 hts_log_print(opt, LOG_INFO, "No files purged");
               }
             }
@@ -2245,6 +2280,7 @@ int httpmirror(char *url1, httrackp * opt) {
   // ending
   usercommand(opt, 0, NULL, NULL, NULL, NULL);
   warc_close_opt(opt);
+  hts_changes_close_opt(opt);
 
   // désallocation mémoire & buffers
   XH_uninit;
@@ -2870,6 +2906,17 @@ int filecreateempty(filenote_strc * strc, const char *filename) {
     return 0;
 }
 
+void hts_savename_listed(const char *root, const char *s, char *dest,
+                         size_t destsize) {
+  char catbuff[CATBUFF_SIZE];
+
+  strlcpybuff(dest, fslash(catbuff, sizeof(catbuff), s), destsize);
+  if (strnotempty(root) && strncmp(fslash(catbuff, sizeof(catbuff), root), dest,
+                                   strlen(root)) == 0) {
+    strlcpybuff(dest, s + strlen(root), destsize);
+  }
+}
+
 // noter fichier
 int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
   // gestion du fichier liste liste
@@ -2879,15 +2926,8 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
     return 0;
   } else if (strc->lst) {
     char BIGSTK savelst[HTS_URLMAXSIZE * 2];
-    char catbuff[CATBUFF_SIZE];
 
-    strcpybuff(savelst, fslash(catbuff, sizeof(catbuff), s));
-    // couper chemin?
-    if (strnotempty(strc->path)) {
-      if (strncmp(fslash(catbuff, sizeof(catbuff), strc->path), savelst, strlen(strc->path)) == 0) {     // couper
-        strcpybuff(savelst, s + strlen(strc->path));
-      }
-    }
+    hts_savename_listed(strc->path, s, savelst, sizeof(savelst));
     fprintf(strc->lst, "[%s]" LF, savelst);
     fflush(strc->lst);
   }
@@ -2897,6 +2937,9 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
 /* Note: utf-8 */
 void file_notify(httrackp * opt, const char *adr, const char *fil,
                  const char *save, int create, int modify, int not_updated) {
+  hts_changes_notify(opt, adr, fil, save,
+                     (create || modify) ? HTS_TRUE : HTS_FALSE,
+                     not_updated ? HTS_TRUE : HTS_FALSE);
   RUN_CALLBACK6(opt, filesave2, adr, fil, save, create, modify, not_updated);
 }
 
@@ -3631,6 +3674,7 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
   to->warc_max_size = from->warc_max_size;
   to->warc_cdx = from->warc_cdx;
   to->warc_wacz = from->warc_wacz;
+  to->changes = from->changes;
 
   if (from->pause_max_ms > 0) {
     to->pause_min_ms = from->pause_min_ms;

src/htscore.h

@@ -354,6 +354,12 @@ int filecreateempty(filenote_strc * strct, const char *filename);
 
 int filenote(filenote_strc * strct, const char *s, filecreate_params * params);
 
+/* Copy into dest (destsize bytes) the form under which the local path s is
+   listed in new.lst: forward slashes, with the mirror `root` stripped when s
+   sits under it. Also the change report's key, so the two must not drift. */
+void hts_savename_listed(const char *root, const char *s, char *dest,
+                         size_t destsize);
+
 void file_notify(httrackp * opt, const char *adr, const char *fil,
                  const char *save, int create, int modify, int wasupdated);
 

src/htscoremain.c

@@ -41,6 +41,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsdefines.h"
 #include "htsalias.h"
 #include "htswarc.h"
+#include "htschanges.h"
 #include "htsbauth.h"
 #include "htswrap.h"
 #include "htsmodules.h"
@@ -145,7 +146,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
   int argv_url = -1;            // ==0 : utiliser cache et doit.log
   char *argv_firsturl = NULL;   // utilisé pour nommage par défaut
   char *url = NULL;             // URLS séparées par un espace
-  int url_sz = 65535;
+  size_t url_sz = 65535;
 
   // the parametres
   int httrack_logmode = 3;      // ONE log file
@@ -224,21 +225,22 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
   /* create x_argvblk buffer for transformed command line */
   {
-    int current_size = 0;
-    int size;
+    size_t current_size = 0;
+    const LLint size = fsize("config");
+    size_t blk_size;
     int na;
 
     for(na = 0; na < argc; na++)
-      current_size += (int) (strlen(argv[na]) + 1);
-    if ((size = fsize("config")) > 0)
-      current_size += size;
-    x_argvblk = (char *) malloct(current_size + 32768);
+      current_size += strlen(argv[na]) + 1;
+    /* a huge file named "config" must saturate, not wrap, the capacity */
+    blk_size = llint_grow_size_t(current_size, size > 0 ? size : 0, 32768);
+    x_argvblk = blk_size != (size_t) -1 ? (char *) malloct(blk_size) : NULL;
     if (x_argvblk == NULL) {
       HTS_PANIC_PRINTF("Error, not enough memory");
       htsmain_free();
       return -1;
     }
-    x_argvblk_size = (size_t) (current_size + 32768);
+    x_argvblk_size = blk_size;
     x_argvblk[0] = '\0';
     x_ptr = 0;
 
@@ -1456,20 +1458,29 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                     FILE *fp = FOPEN(argv[na], "rb");
 
                     if (fp != NULL) {
-                      int cl = (int) strlen(url);
+                      size_t cl = strlen(url);
+                      const size_t fzs = llint_to_size_t(fz);
+                      const size_t capa = llint_grow_size_t(cl, fz, 8192);
 
-                      ensureUrlCapacity(url, url_sz, cl + fz + 8192);
+                      if (capa == (size_t) -1) {
+                        fclose(fp);
+                        HTS_PANIC_PRINTF("File url list too large");
+                        htsmain_free();
+                        return -1;
+                      }
+                      ensureUrlCapacity(url, url_sz, capa);
                       if (cl > 0) {     /* don't stick! (3.43) */
                         url[cl] = ' ';
                         cl++;
                       }
-                      if (fread(url + cl, 1, fz, fp) != fz) {
+                      if (fread(url + cl, 1, fzs, fp) != fzs) {
+                        fclose(fp);
                         HTS_PANIC_PRINTF("File url list could not be read");
                         htsmain_free();
                         return -1;
                       }
                       fclose(fp);
-                      *(url + cl + fz) = '\0';
+                      *(url + cl + fzs) = '\0';
                     }
                   }
                 }
@@ -1737,6 +1748,13 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->cookies_file, argv[na]);
                 }
                 break;
+              case 'd': // --changes: report what this crawl changed
+                opt->changes = HTS_TRUE;
+                if (*(com + 1) == '0') {
+                  opt->changes = HTS_FALSE;
+                  com++;
+                }
+                break;
               case 'r': // warc / warc-file: write an ISO-28500 WARC archive
                 if (*(com + 1) == 'f') { // --warc-file NAME: explicit basename
                   com++;
@@ -2296,8 +2314,8 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
       } else {                  // URL/filters
         char catbuff[CATBUFF_SIZE];
-        const int urlSize = (int) strlen(argv[na]);
-        const int capa = (int) (strlen(url) + urlSize + 32);
+        const size_t urlSize = strlen(argv[na]);
+        const size_t capa = strlen(url) + urlSize + 32;
 
         assertf(urlSize < HTS_URLMAXSIZE);
         if (urlSize < HTS_URLMAXSIZE) {

src/htsftp.c

@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
       // folding a nonsense port into 1..65535 fetches one the link never named;
       // an empty "host:" just means the default (#614)
       if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
-        snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
+        htsblk_failf(&back->r, "Invalid port: %s", a + 1);
         back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
         _HALT_FTP return 0;
       }
@@ -262,8 +262,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
     // récupérer adresse résolue
     strcpybuff(back->info, "host name");
     if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
-      snprintf(back->r.msg, sizeof(back->r.msg),
-               "Unable to get server's address: %s", error);
+      htsblk_failf(&back->r, "Unable to get server's address: %s", error);
       back->r.statuscode = STATUSCODE_NON_FATAL;
       _HALT_FTP return 0;
     }
@@ -332,18 +331,15 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
             }
 
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }
         } else {
-          snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
-                   linejmp(line));
+          htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
           back->r.statuscode = STATUSCODE_INVALID;
         }
       } else {
-        snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
-                 linejmp(line));
+        htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
         back->r.statuscode = STATUSCODE_INVALID;
       }
 
@@ -410,8 +406,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
             }
             // -- fin analyse de l'adresse IP et du port --
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         } else {
@@ -442,13 +437,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 }
               }
             } else {
-              snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
-                       linejmp(line));
+              htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
               back->r.statuscode = STATUSCODE_INVALID;
             }
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         }
@@ -554,8 +547,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                   deletesoc(soc_dat);
                   soc_dat = INVALID_SOCKET;
                   //
-                  snprintf(back->r.msg, sizeof(back->r.msg),
-                           "RETR command error: %s", linejmp(line));
+                  htsblk_failf(&back->r, "RETR command error: %s",
+                               linejmp(line));
                   back->r.statuscode = STATUSCODE_INVALID;
                 }               // sinon on est prêts
               } else {
@@ -573,13 +566,12 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
               back->r.statuscode = STATUSCODE_INVALID;
             }                   // sinon on est prêts
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg),
-                     "Unable to resolve IP %s: %s", adr_ip, error);
+            htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
+                         error);
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         } else {
-          snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
-                   linejmp(line));
+          htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
           back->r.statuscode = STATUSCODE_INVALID;
         }                       // sinon on est prêts
 #else
@@ -603,13 +595,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 back->r.statuscode = STATUSCODE_INVALID;
               }
             } else {
-              snprintf(back->r.msg, sizeof(back->r.msg),
-                       "RETR command error: %s", linejmp(line));
+              htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
               back->r.statuscode = STATUSCODE_INVALID;
             }
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }
 #ifdef _WIN32
@@ -652,8 +642,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 len = 0;        // fin
                 break;
               case 0:
-                snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
-                         timeout);
+                htsblk_failf(&back->r, "Time out (%d)", timeout);
                 back->r.statuscode = STATUSCODE_INVALID;
                 len = 0;        // fin
                 break;
@@ -716,8 +705,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 strcpybuff(back->r.msg, "OK");
                 back->r.statuscode = HTTP_OK;
               } else {
-                snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
-                         linejmp(line));
+                htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
                 back->r.statuscode = STATUSCODE_INVALID;
               }
             } else {

src/htsglobal.h

@@ -72,7 +72,8 @@ Please visit our Website: http://www.httrack.com
    HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
    static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
    compiler type-checks the format string at argument index fmt against the
-   varargs starting at arg. */
+   varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
+   error signal, so dropping it is a bug; a (void) cast does not silence it. */
 #ifndef HTS_UNUSED
 #ifdef __GNUC__
 #define HTS_UNUSED __attribute__((unused))
@@ -80,10 +81,13 @@ Please visit our Website: http://www.httrack.com
 #define HTS_STATIC static __attribute__((unused))
 
 #define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
+
+#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
 #else
 #define HTS_UNUSED
 #define HTS_STATIC static
 #define HTS_PRINTF_FUN(fmt, arg)
+#define HTS_CHECK_RESULT
 #endif
 #endif
 
@@ -336,16 +340,20 @@ typedef int INTsys;
 #endif
 
 /* Socket-handle type. An unsigned integer wide enough for a Windows SOCKET;
-   a plain int file descriptor on POSIX. */
+   a plain int file descriptor on POSIX. T_SOCP is its printf conversion,
+   '%' included: unsigned __int64 on Win64 must not be printed with "%d". */
 #ifdef _WIN32
 #if defined(_WIN64)
 
 typedef unsigned __int64 T_SOC;
+#define T_SOCP "%" PRIu64
 #else
 typedef unsigned __int32 T_SOC;
+#define T_SOCP "%" PRIu32
 #endif
 #else
 typedef int T_SOC;
+#define T_SOCP "%d"
 #endif
 
 /* Buffer size for a printed network address (IPv4 or IPv6, NUL included). */

src/htshelp.c

@@ -124,7 +124,9 @@ typedef struct help_wizard_buffers {
   char stropt[2048];        // options
   char stropt2[2048];       // options longues
   char strwild[2048];       // wildcards
-  char cmd[4096];
+  /* holds all four of the above plus separators: at 4096 a long answer set
+     clipped the filters off the command line */
+  char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
   char str[256];
   char *argv[256];
 } help_wizard_buffers;
@@ -156,9 +158,7 @@ void help_wizard(httrackp * opt) {
   char *a;
 
   //
-  if (urls == NULL || mainpath == NULL || projname == NULL || stropt == NULL
-      || stropt2 == NULL || strwild == NULL || cmd == NULL || str == NULL
-      || argv == NULL) {
+  if (buffers == NULL) {
     fprintf(stderr, "* memory exhausted in %s, line %d\n", __FILE__, __LINE__);
     return;
   }
@@ -251,6 +251,7 @@ void help_wizard(httrackp * opt) {
       strcatbuff(stropt2, "--update ");
       break;
     case 0:
+      freet(buffers);
       return;
       break;
     }
@@ -309,14 +310,23 @@ void help_wizard(httrackp * opt) {
       printf("\n");
       if (strlen(stropt) == 1)
         stropt[0] = '\0';       // aucune
-      snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
+      /* the tail is the filter list, and cmd is split into the argv handed to
+         hts_main() below: a clipped line would silently widen the crawl */
+      if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
+        printf("* command line too long (%d bytes max)\n",
+               (int) sizeof(cmd) - 1);
+        freet(buffers);
+        return;
+      }
       printf("---> Wizard command line: httrack %s\n\n", cmd);
       printf("Ready to launch the mirror? (Y/n) :");
       fflush(stdout);
       linput(stdin, str, 250);
       if (strnotempty(str)) {
-        if (!((str[0] == 'y') || (str[0] == 'Y')))
+        if (!((str[0] == 'y') || (str[0] == 'Y'))) {
+          freet(buffers);
           return;
+        }
       }
       printf("\n");
 
@@ -340,7 +350,7 @@ void help_wizard(httrackp * opt) {
   }
 
   /* Free buffers */
-  free(buffers);
+  freet(buffers);
 #undef urls
 #undef mainpath
 #undef projname
@@ -423,7 +433,8 @@ void help_catchurl(const char *dest_path) {
       }
       // former URL!
       {
-        char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
+        /* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
+        char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
 
         inplace_escape_check_url(dest, sizeof(dest));
         snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);
@@ -597,6 +608,8 @@ void help(const char *app, int more) {
           "output name, --warc-max-size N rotates segments past N bytes, "
           "--warc-cdx also writes a sorted CDXJ index, --wacz packages it all "
           "as a WACZ file");
+  infomsg(" %d  write hts-changes.json listing what this crawl left new, "
+          "changed, unchanged and gone compared to the previous mirror");
   infomsg(" %n  do not re-download locally erased files");
   infomsg
     (" %v  display on screen filenames downloaded (in realtime) - * %v1 short version - %v2 full animation");

src/htslib.c

@@ -37,6 +37,7 @@ Please visit our Website: http://www.httrack.com
 
 #include "htscore.h"
 #include "htswarc.h"
+#include "htschanges.h"
 
 /* specific definitions */
 #include "htsbase.h"
@@ -704,18 +705,16 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
   /* Check for errors */
   if (soc == INVALID_SOCKET) {
     if (retour) {
-      if (retour->msg) {
-        if (!strnotempty(retour->msg)) {
+      if (!strnotempty(retour->msg)) {
 #ifdef _WIN32
-          int last_errno = WSAGetLastError();
+        int last_errno = WSAGetLastError();
 
-          sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
+        htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
 #else
-          int last_errno = errno;
+        int last_errno = errno;
 
-          sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
+        htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
 #endif
-        }
       }
     }
   }
@@ -2207,7 +2206,7 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #if DEBUG
       printf("erreur gethostbyname\n");
 #endif
-      if (retour && retour->msg) {
+      if (retour != NULL) {
 #ifdef _WIN32
         snprintf(retour->msg, sizeof(retour->msg),
                  "Unable to get server's address: %s", error);
@@ -2238,17 +2237,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
     DEBUG_W("socket()=%d\n" _(int) soc);
 #endif
     if (soc == INVALID_SOCKET) {
-      if (retour && retour->msg) {
+      if (retour != NULL) {
 #ifdef _WIN32
         int last_errno = WSAGetLastError();
 
-        sprintf(retour->msg, "Unable to create a socket: %s",
-                strerror(last_errno));
+        htsblk_failf(retour, "Unable to create a socket: %s",
+                     strerror(last_errno));
 #else
         int last_errno = errno;
 
-        sprintf(retour->msg, "Unable to create a socket: %s",
-                strerror(last_errno));
+        htsblk_failf(retour, "Unable to create a socket: %s",
+                     strerror(last_errno));
 #endif
       }
       return INVALID_SOCKET;    // erreur création socket impossible
@@ -2262,17 +2261,8 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
                              &bind_addr, &error) == NULL
           || bind(soc, &SOCaddr_sockaddr(bind_addr), 
                   SOCaddr_size(bind_addr)) != 0) {
-        if (retour && retour->msg) {
-#ifdef _WIN32
-          snprintf(retour->msg, sizeof(retour->msg),
-                   "Unable to bind the specificied server address: %s",
-                   error);
-#else
-          snprintf(retour->msg, sizeof(retour->msg),
-                   "Unable to bind the specificied server address: %s",
-                   error);
-#endif
-        }
+        snprintf(retour->msg, sizeof(retour->msg),
+                 "Unable to bind the specificied server address: %s", error);
         deletesoc(soc);
         return INVALID_SOCKET;
       }
@@ -2319,17 +2309,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #if HDEBUG
         printf("unable to connect!\n");
 #endif
-        if (retour != NULL && retour->msg) {
+        if (retour != NULL) {
 #ifdef _WIN32
           const int last_errno = WSAGetLastError();
 
-          sprintf(retour->msg, "Unable to connect to the server: %s",
-                  strerror(last_errno));
+          htsblk_failf(retour, "Unable to connect to the server: %s",
+                       strerror(last_errno));
 #else
           const int last_errno = errno;
 
-          sprintf(retour->msg, "Unable to connect to the server: %s",
-                  strerror(last_errno));
+          htsblk_failf(retour, "Unable to connect to the server: %s",
+                       strerror(last_errno));
 #endif
         }
         /* Close the socket and notify the error!!! */
@@ -2598,13 +2588,15 @@ void deletesoc(T_SOC soc) {
     if (closesocket(soc) != 0) {
       int err = WSAGetLastError();
 
-      fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
+      fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
+              strerror(err));
     }
 #else
     if (close(soc) != 0) {
       const int err = errno;
 
-      fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
+      fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
+              strerror(err));
     }
 #endif
 #if HTS_WIDE_DEBUG
@@ -2707,6 +2699,24 @@ void time_gmt_rfc822(char *s) {
   time_rfc822(s, A);
 }
 
+void hts_now_iso8601(char out[32]) {
+  time_t t = time(NULL);
+  struct tm tmv;
+
+#if defined(_WIN32)
+  struct tm *g = gmtime(&t);
+
+  if (g != NULL)
+    tmv = *g;
+  else
+    memset(&tmv, 0, sizeof(tmv));
+#else
+  if (gmtime_r(&t, &tmv) == NULL)
+    memset(&tmv, 0, sizeof(tmv));
+#endif
+  strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
+}
+
 // heure actuelle, format rfc (taille buffer 256o)
 void time_local_rfc822(char *s) {
   time_t tt;
@@ -3017,7 +3027,7 @@ int finput(T_SOC fd, char *s, int max) {
 }
 
 // Like linput, but in memory (optimized)
-int binput(char *buff, char *s, int max) {
+int binput(const char *buff, char *s, int max) {
   int count = 0;
   int destCount = 0;
 
@@ -6020,6 +6030,8 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   StringCopy(opt->cookies_file, "");
   StringCopy(opt->warc_file, "");
   opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
+  opt->changes = HTS_FALSE;
+  opt->changes_state = NULL;
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
   opt->pause_max_ms = 0;
@@ -6173,6 +6185,8 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->why_url);
     StringFree(opt->warc_file);
 
+    hts_changes_free_opt(opt);
+
     StringFree(opt->path_html);
     StringFree(opt->path_html_utf8);
     StringFree(opt->path_log);

src/htslib.h

@@ -199,6 +199,11 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
    etc.). */
 T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
                    int waitconnect, int addr_index, int *addr_count);
+/* Clips the formatted failure reason into r->msg, which also round-trips
+   through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
+#define htsblk_failf(R, ...)                                                   \
+  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
+
 HTS_INLINE void deletehttp(htsblk * r);
 HTS_INLINE int deleteaddr(htsblk * r);
 HTS_INLINE void deletesoc(T_SOC soc);
@@ -253,6 +258,10 @@ void sec2str(char *s, TStamp t);
 
 void time_gmt_rfc822(char *s);
 void time_local_rfc822(char *s);
+
+/* Current UTC time as "YYYY-MM-DDThh:mm:ssZ". */
+void hts_now_iso8601(char out[32]);
+
 struct tm *convert_time_rfc822(struct tm *buffer, const char *s);
 int set_filetime(const char *file, struct tm *tm_time);
 int set_filetime_rfc822(const char *file, const char *date);
@@ -262,7 +271,7 @@ HTS_INLINE void time_rfc822_local(char *s, struct tm *A);
 
 HTS_INLINE int sendc(htsblk * r, const char *s);
 int finput(T_SOC fd, char *s, int max);
-int binput(char *buff, char *s, int max);
+int binput(const char *buff, char *s, int max);
 int linput(FILE * fp, char *s, int max);
 int linputsoc(T_SOC soc, char *s, int max);
 int linputsoc_t(T_SOC soc, char *s, int max, int timeout);
@@ -607,6 +616,21 @@ static HTS_UNUSED size_t llint_to_size_t(LLint o) {
   }
 }
 
+/* Capacity for @p used bytes plus @p extra more plus @p slack spare;
+   (size_t) -1 if the total exceeds (size_t) -2 or @p extra is negative
+   (llint_to_size_t() would map that to a huge valid-looking size). */
+static HTS_UNUSED size_t llint_grow_size_t(size_t used, LLint extra,
+                                           size_t slack) {
+  const size_t max = (size_t) -2; /* (size_t) -1 is the error value */
+  const size_t e = extra >= 0 ? llint_to_size_t(extra) : (size_t) -1;
+
+  if (e == (size_t) -1 || used > max || slack > max - used ||
+      e > max - used - slack) {
+    return (size_t) -1;
+  }
+  return used + e + slack;
+}
+
 /* dirent() compatibility */
 #ifdef _WIN32
 /* Holds a UTF-8 d_name: MAX_PATH (260) UTF-16 units expand to <=3 bytes each.

src/htsopt.h

@@ -547,6 +547,10 @@ struct httrackp {
                              archive. Tail: ABI */
   hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip
                               (implies --warc + --warc-cdx). Tail: ABI */
+  hts_boolean changes;   /**< --changes: report what this crawl changed against
+                              the previous mirror. Tail: ABI */
+  void *changes_state;   /**< live change-report accumulator (hts_changes*),
+                              engine-owned. Tail: ABI */
 };
 
 /* Running statistics for a mirror. */

src/htsparse.c

@@ -4583,7 +4583,7 @@ int hts_wait_delayed(htsmoduleStruct * str, lien_adrfilsave *afs,
             /* seen as in error */
             in_error = back[b].r.statuscode;
             in_error_msg[0] = 0;
-            strncat(in_error_msg, back[b].r.msg, sizeof(in_error_msg) - 1);
+            strncatbuff(in_error_msg, back[b].r.msg, sizeof(in_error_msg) - 1);
             in_error_size = back[b].r.totalsize;
             /* don't break, even with "don't take error pages" switch, because we need to process the slot anyway (and cache the error) */
           }

src/htsproxy.c

@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
   if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
     code = 0;
   if (code < 200 || code >= 300) {
-    snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
-             strnotempty(line) ? line : "(no status)");
+    htsblk_failf(retour, "proxy CONNECT refused: %s",
+                 strnotempty(line) ? line : "(no status)");
     return 0;
   }
 

src/htssafe.h

@@ -33,6 +33,7 @@ Please visit our Website: http://www.httrack.com
 #ifndef HTSSAFE_DEFH
 #define HTSSAFE_DEFH
 
+#include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -116,6 +117,15 @@ static HTS_UNUSED void abortf_(const char *exp, const char *file, int line) {
 #endif
 #define HTS_IS_NOT_CHAR_BUFFER(VAR) (!HTS_IS_CHAR_BUFFER(VAR))
 
+/* Source capacity for the buff() family, (size_t)-1 when unknown; sizeof of the
+   TYPE keeps a decayed operand ("buf + 1") off -Wsizeof-array-decay. */
+#if (defined(__GNUC__) && !defined(__cplusplus))
+#define HTS_SIZEOF_SRC_(B)                                                     \
+  (HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(__typeof__(B)))
+#else
+#define HTS_SIZEOF_SRC_(B) (HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B))
+#endif
+
 /* Compile-time checks. */
 static HTS_UNUSED void htssafe_compile_time_check_(void) {
   char array[32];
@@ -205,19 +215,17 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #if (defined(__GNUC__) && !defined(__cplusplus))
 
 #define strncatbuff(A, B, N)                                                   \
-  __builtin_choose_expr(                                                       \
-      HTS_IS_CHAR_BUFFER(A),                                                   \
-      strncat_safe_(A, sizeof(A), B,                                           \
-                    HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N,    \
-                    "overflow while appending '" #B "' to '" #A "'", __FILE__, \
-                    __LINE__),                                                 \
-      strncatbuff_ptr_((A), (B), (N)))
+  __builtin_choose_expr(HTS_IS_CHAR_BUFFER(A),                                 \
+                        strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), N,  \
+                                      "overflow while appending '" #B          \
+                                      "' to '" #A "'",                         \
+                                      __FILE__, __LINE__),                     \
+                        strncatbuff_ptr_((A), (B), (N)))
 #else
 #define strncatbuff(A, B, N)                                                   \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strncat(A, B, N)                                                      \
-       : strncat_safe_(A, sizeof(A), B,                                        \
-                       HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N, \
+       : strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), N,                 \
                        "overflow while appending '" #B "' to '" #A "'",        \
                        __FILE__, __LINE__))
 #endif
@@ -232,9 +240,7 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #define strcatbuff(A, B)                                                       \
   __builtin_choose_expr(                                                       \
       HTS_IS_CHAR_BUFFER(A),                                                   \
-      strncat_safe_(A, sizeof(A), B,                                           \
-                    HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),       \
-                    (size_t) -1,                                               \
+      strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), (size_t) -1,          \
                     "overflow while appending '" #B "' to '" #A "'", __FILE__, \
                     __LINE__),                                                 \
       strcatbuff_ptr_((A), (B)))
@@ -242,9 +248,7 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #define strcatbuff(A, B)                                                       \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strcat(A, B)                                                          \
-       : strncat_safe_(A, sizeof(A), B,                                        \
-                       HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),    \
-                       (size_t) -1,                                            \
+       : strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), (size_t) -1,       \
                        "overflow while appending '" #B "' to '" #A "'",        \
                        __FILE__, __LINE__))
 #endif
@@ -257,19 +261,17 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #if (defined(__GNUC__) && !defined(__cplusplus))
 
 #define strcpybuff(A, B)                                                       \
-  __builtin_choose_expr(                                                       \
-      HTS_IS_CHAR_BUFFER(A),                                                   \
-      strcpy_safe_(A, sizeof(A), B,                                            \
-                   HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),        \
-                   "overflow while copying '" #B "' to '" #A "'", __FILE__,    \
-                   __LINE__),                                                  \
-      strcpybuff_ptr_((A), (B)))
+  __builtin_choose_expr(HTS_IS_CHAR_BUFFER(A),                                 \
+                        strcpy_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B),      \
+                                     "overflow while copying '" #B "' to '" #A \
+                                     "'",                                      \
+                                     __FILE__, __LINE__),                      \
+                        strcpybuff_ptr_((A), (B)))
 #else
 #define strcpybuff(A, B)                                                       \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strcpy(A, B)                                                          \
-       : strcpy_safe_(A, sizeof(A), B,                                         \
-                      HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),     \
+       : strcpy_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B),                     \
                       "overflow while copying '" #B "' to '" #A "'", __FILE__, \
                       __LINE__))
 #endif
@@ -286,24 +288,24 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
  * Append characters of "B" to "A", "A" having a maximum capacity of "S".
  */
 #define strlcatbuff(A, B, S)                                                   \
-  strncat_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),  \
-                (size_t) -1, "overflow while appending '" #B "' to '" #A "'",  \
-                __FILE__, __LINE__)
+  strncat_safe_(A, S, B, HTS_SIZEOF_SRC_(B), (size_t) -1,                      \
+                "overflow while appending '" #B "' to '" #A "'", __FILE__,     \
+                __LINE__)
 
 /**
  * Append at most "N" characters of "B" to "A", "A" having a maximum capacity
  * of "S".
  */
 #define strlncatbuff(A, B, S, N)                                               \
-  strncat_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),  \
-                N, "overflow while appending '" #B "' to '" #A "'", __FILE__,  \
+  strncat_safe_(A, S, B, HTS_SIZEOF_SRC_(B), N,                                \
+                "overflow while appending '" #B "' to '" #A "'", __FILE__,     \
                 __LINE__)
 
 /**
  * Copy characters of "B" to "A", "A" having a maximum capacity of "S".
  */
 #define strlcpybuff(A, B, S)                                                   \
-  strcpy_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),   \
+  strcpy_safe_(A, S, B, HTS_SIZEOF_SRC_(B),                                    \
                "overflow while copying '" #B "' to '" #A "'", __FILE__,        \
                __LINE__)
 
@@ -422,7 +424,9 @@ static HTS_INLINE HTS_UNUSED htsbuff htsbuff_ptr_(char *buf, size_t cap) {
  */
 static HTS_INLINE HTS_UNUSED void htsbuff_catn(htsbuff *b, const char *s,
                                                size_t n) {
-  const size_t add = strnlen(s, n);
+  /* the (size_t)-1 "no limit" sentinel would reach strnlen as a bound past
+     PTRDIFF_MAX */
+  const size_t add = n != (size_t) -1 ? strnlen(s, n) : strlen(s);
   /* Overflow-safe: keep the (potentially huge) 'add' alone on one side. The
      maintained invariant len < cap makes 'cap - len' >= 1 (no underflow), so
      'add < cap - len' cannot wrap the way 'len + add < cap' could. */
@@ -456,6 +460,85 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
   return b->buf;
 }
 
+/**
+ * Copy src into dest (capacity size, NUL included), truncating to fit and
+ * always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
+ * value read back from a cache, a header or the wire, where refusing the whole
+ * record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
+ * callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
+ */
+static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
+                                                     const char *src) {
+  size_t len, copy;
+
+  assertf(dest != NULL && src != NULL && size != 0);
+  len = strlen(src);
+  copy = len < size ? len : size - 1;
+  memcpy(dest, src, copy);
+  dest[copy] = '\0';
+  return copy == len ? HTS_TRUE : HTS_FALSE;
+}
+
+/**
+ * Callers that deliberately ignore truncation use this instead of
+ * slprintfbuff(), so it is not HTS_CHECK_RESULT.
+ */
+static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
+    vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
+  int ret;
+
+  assertf(dest != NULL && size != 0);
+  /* a vsnprintf failing outright may write nothing at all, leaving whatever
+     the caller had on the stack for it to publish */
+  dest[0] = '\0';
+  ret = vsnprintf(dest, size, fmt, args);
+  /* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
+  dest[size - 1] = '\0';
+  return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
+}
+
+/**
+ * Formatted print into dest (capacity size, NUL included), truncating to fit
+ * and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
+ * result is the only truncation signal, so it must be acted on. Unlike
+ * strcpybuff() it never aborts, so it suits text built from remote input.
+ */
+static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
+    slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
+  va_list args;
+  hts_boolean ret;
+
+  va_start(args, fmt);
+  ret = vslprintfbuff(dest, size, fmt, args);
+  va_end(args);
+  return ret;
+}
+
+/**
+ * slprintfbuff() for diagnostics quoting remote or client text, which are
+ * meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
+ * cast on slprintfbuff() is no substitute, GCC warns through it.
+ */
+static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
+    char *dest, size_t size, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  (void) vslprintfbuff(dest, size, fmt, args);
+  va_end(args);
+}
+
+/**
+ * slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
+ * On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
+ */
+#if (defined(__GNUC__) && !defined(__cplusplus))
+#define sprintfbuff(ARR, ...)                                                  \
+  slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
+#else
+#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
+#endif
+
 /* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
    added bounds checking. freet() also NULLs the freed pointer and tolerates
    NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

src/htsselftest.c

@@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsback.h"
 #include "htsdefines.h"
 #include "htslib.h"
 #include "htsalias.h"
@@ -51,6 +52,7 @@ Please visit our Website: http://www.httrack.com
 #include "htscache_selftest.h"
 #include "htsdns_selftest.h"
 #include "htscharset.h"
+#include "htscmdline.h"
 #include "htsencoding.h"
 #include "htsftp.h"
 #include "htsmd5.h"
@@ -58,6 +60,7 @@ Please visit our Website: http://www.httrack.com
 #include "htscodec.h"
 #include "htsproxy.h"
 #include "htswarc.h"
+#include "htschanges.h"
 #if HTS_USEZLIB
 #include "htszlib.h"
 #endif
@@ -455,6 +458,12 @@ static void basic_selftests(void) {
     // link one level up -> a "../" prefix
     assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
     assertf(strcmp(s, "../a.html") == 0);
+    // an empty current path: the trim used to walk off the front of it, which
+    // "?x" reaches too because the query pre-pass hands on the part before it
+    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
+    assertf(strcmp(s, "dir/page.html") == 0);
+    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
+    assertf(strcmp(s, "dir/page.html") == 0);
   }
 }
 
@@ -481,6 +490,28 @@ static int string_safety_selftests(void) {
   if (strcmp(buf, "abcd") != 0)
     return 1;
 
+  /* A decayed source has no known capacity, so the whole tail must land; a
+     sizeof(char*) capacity would abort here instead. */
+  {
+    char src[32] = "0123456789abcdefghij";
+    char dst[32];
+
+    strcpybuff(dst, src + 1);
+    if (strcmp(dst, "123456789abcdefghij") != 0)
+      return 1;
+  }
+
+  /* Truncating append: stops at N without aborting, what the status-message
+     call sites rely on. */
+  {
+    char dst[10]; /* never sizeof(char*), or MSVC reads it as a pointer */
+
+    dst[0] = '\0';
+    strncatbuff(dst, "abcdefghijkl", sizeof(dst) - 1);
+    if (strcmp(dst, "abcdefghi") != 0)
+      return 1;
+  }
+
   /* strlcpybuff: explicit-capacity copy into a pointer destination, the form
      the migration moves toward */
   {
@@ -546,6 +577,214 @@ static int string_safety_selftests(void) {
       return 1;
   }
 
+  /* sprintfbuff: truncate-and-report. Must never abort (its callers format
+     remote banners) nor write past the array, which the canary catches. */
+  {
+    struct {
+      char dst[8];
+      char canary[8];
+    } s;
+
+    const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
+
+    /* repoison before every call, or an implementation that measures first and
+       writes nothing "passes" the truncating cases on the previous content */
+#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
+
+    memset(&s, '#', sizeof(s));
+    if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
+      return 1;
+
+    /* exact fit: 7 characters plus the NUL */
+    POISON_DST();
+    if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
+      return 1;
+
+    /* one over, then far over: truncated to the prefix, terminated, reported */
+    POISON_DST();
+    if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
+      return 1;
+    POISON_DST();
+    if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
+      return 1;
+
+    /* explicit-capacity form, down to the degenerate size 1 */
+    {
+      char *const p = s.dst;
+
+      POISON_DST();
+      if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
+        return 1;
+      POISON_DST();
+      if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
+        return 1;
+    }
+#undef POISON_DST
+
+    if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
+      return 1;
+  }
+
+  /* strclipbuff: truncate-and-report, never abort. Same canary shape; the
+     destination is poisoned before every call so a case cannot pass on the
+     previous one's bytes. */
+  {
+    struct {
+      char dst[8];
+      char canary[8];
+    } s;
+
+    memset(&s, '#', sizeof(s));
+#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
+
+    /* well under capacity: no padding, nothing eaten off the end */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "abc") || strcmp(s.dst, "abc") != 0)
+      return 1;
+
+    /* exact fit: capacity - 1 characters plus the NUL */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "1234567") ||
+        strcmp(s.dst, "1234567") != 0)
+      return 1;
+
+    /* one over, then far over: clipped, terminated, reported. The expected
+       bytes differ from the case above, so a write-nothing implementation
+       cannot pass on the leftovers. */
+    POISON_DST();
+    if (strclipbuff(s.dst, sizeof(s.dst), "abcdefgh") ||
+        strcmp(s.dst, "abcdefg") != 0)
+      return 1;
+    POISON_DST();
+    if (strclipbuff(s.dst, sizeof(s.dst), "0123456789abcdef") ||
+        strcmp(s.dst, "0123456") != 0)
+      return 1;
+
+    /* degenerate capacity 1: only the NUL fits. Capacity 2 pins the boundary
+       between that and the sizes above: one character plus the NUL. */
+    POISON_DST();
+    if (strclipbuff(s.dst, 1, "x") || s.dst[0] != '\0')
+      return 1;
+    POISON_DST();
+    if (strclipbuff(s.dst, 2, "yz") || strcmp(s.dst, "y") != 0)
+      return 1;
+
+    /* the empty string fits any non-zero capacity */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "") || s.dst[0] != '\0')
+      return 1;
+
+    /* a byte over 0x7f must not end the copy early */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "\xff\xfe") ||
+        strcmp(s.dst, "\xff\xfe") != 0)
+      return 1;
+#undef POISON_DST
+
+    if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
+      return 1;
+  }
+
+  /* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
+     touches nothing else in the block */
+  {
+    htsblk r;
+    char expect[sizeof(r.msg)];
+    char big[4 * sizeof(r.msg)];
+
+    /* contenttype abuts msg, so a one-past-the-end store lands in it rather
+       than in padding. Poison it: a stray NUL is invisible against zeroes,
+       and a stray NUL is exactly what an off-by-one terminator writes. */
+#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
+
+    memset(&r, 0, sizeof(r));
+    memset(r.contenttype, '#', sizeof(r.contenttype));
+    r.statuscode = 1234;
+
+    memset(r.msg, '#', sizeof(r.msg));
+    htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
+    if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
+      return 1;
+
+    /* exact fit: capacity - 1 characters plus the NUL */
+    memset(expect, 'y', sizeof(expect) - 1);
+    expect[sizeof(expect) - 1] = '\0';
+    memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
+    memset(r.msg, '#', sizeof(r.msg));
+    htsblk_failf(&r, "%s", expect);
+    if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
+      return 1;
+
+    /* far over: the expected bytes differ from the cases above, so writing
+       nothing cannot pass on the leftovers */
+    memset(big, 'z', sizeof(big) - 1);
+    big[sizeof(big) - 1] = '\0';
+    memset(expect, 'z', sizeof(expect) - 1);
+    expect[sizeof(expect) - 1] = '\0';
+    memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
+
+    memset(r.msg, '#', sizeof(r.msg));
+    htsblk_failf(&r, "Bad user name: %s", big);
+    if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
+      return 1;
+#undef NEIGHBOURS_INTACT
+  }
+
+  /* back_read_ftp_result: the helper's result file is external input, so an
+     over-long message must stop at msg[]'s capacity */
+  {
+    htsblk r;
+    size_t k;
+
+    /* poisoned so a short message cannot pass on leftovers, and so a stray
+       NUL past msg[] is visible in the neighbour */
+#define FTP_RESULT_CASE(BODY)                                                  \
+  do {                                                                         \
+    FILE *fp_ = tmpfile();                                                     \
+                                                                               \
+    if (fp_ == NULL)                                                           \
+      return 1;                                                                \
+    BODY;                                                                      \
+    rewind(fp_);                                                               \
+    memset(&r, 0, sizeof(r));                                                  \
+    memset(r.msg, '#', sizeof(r.msg));                                         \
+    memset(r.contenttype, '#', sizeof(r.contenttype));                         \
+    back_read_ftp_result(fp_, &r);                                             \
+    fclose(fp_);                                                               \
+    if (r.contenttype[0] != '#')                                               \
+      return 1;                                                                \
+  } while (0)
+
+    /* over capacity: clipped to 79 payload bytes plus the NUL */
+    FTP_RESULT_CASE({
+      fprintf(fp_, "226 ");
+      for (k = 0; k < 4 * sizeof(r.msg); k++)
+        fputc('q', fp_);
+    });
+    if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
+      return 1;
+    for (k = 0; k < sizeof(r.msg) - 1; k++) {
+      if (r.msg[k] != 'q')
+        return 1;
+    }
+
+    /* well under capacity: nothing padded, nothing eaten off the end */
+    FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
+    if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
+      return 1;
+
+    /* a byte over 0x7f must not read as EOF and cut the message short */
+    FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
+    if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
+      return 1;
+
+    /* unparseable status: the message still loads, the code reports failure */
+    FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
+    if (r.statuscode != STATUSCODE_INVALID)
+      return 1;
+#undef FTP_RESULT_CASE
+  }
+
   /* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
      n_eval == 2 on a double-eval macro. */
   {
@@ -1104,6 +1343,128 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+// hts_split_cmdline(): the vector must grow with the argument count, and a
+// quote inside a value must not end the argument and hand -V to the parser.
+static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
+  char line[512];
+  char **args;
+  int nargs = 0;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+  // control: every separator splits, and argv[0] is the program name
+  strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(args != NULL && nargs == 6);
+  assertf(args[nargs] == NULL); // callers may walk to the terminator
+  assertf(strcmp(args[0], "httrack") == 0);
+  assertf(strcmp(args[1], "http://x/") == 0);
+  assertf(strcmp(args[2], "--quiet") == 0);
+  assertf(strcmp(args[3], "-c8") == 0);
+  assertf(strcmp(args[4], "-O") == 0);
+  assertf(strcmp(args[5], "out") == 0);
+  freet(args);
+
+  // the template pads with whitespace: empty arguments are kept (the engine
+  // skips them), so the count is one per separator
+  strcpybuff(line, "httrack  --quiet");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3 && args[1][0] == '\0');
+  assertf(strcmp(args[2], "--quiet") == 0);
+  freet(args);
+
+  // a quoted run keeps both its spaces and its quotes: the engine unquotes
+  strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 4);
+  assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
+  assertf(strcmp(args[3], "-c8") == 0);
+  freet(args);
+
+  // an escaped quote is a literal quote, not the end of the argument: the
+  // engine strips only the outer pair
+  strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 4);
+  assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
+  assertf(strcmp(args[3], "-c8") == 0);
+  freet(args);
+
+  // \\ is a literal backslash, so a Windows path survives
+  strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3);
+  assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
+  freet(args);
+
+  // outside a quoted run a backslash is literal: the url and wildcard-filter
+  // fields, which the wizard cannot quote, read as before
+  strcpybuff(line, "httrack -*\\** +*.png");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3);
+  assertf(strcmp(args[1], "-*\\**") == 0);
+  assertf(strcmp(args[2], "+*.png") == 0);
+  freet(args);
+
+  // a quoted run leaves slots unused, so the terminator has to be written and
+  // not inherited: size the vector from a full line first, so freeing it hands
+  // the same chunk back with stale pointers in those slots
+  strcpybuff(line, "httrack a b c d e");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 6);
+  freet(args);
+  strcpybuff(line, "httrack \"a b c d e\"");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 2);
+  assertf(args[nargs] == NULL);
+  freet(args);
+
+  // an unterminated quote protects the rest of the line, as one argument
+  strcpybuff(line, "httrack --footer \"unbalanced -V x");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3);
+  assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
+  freet(args);
+
+  // past the 1024 entries the vector used to hold: distinct arguments, so a
+  // write beyond the allocation cannot read back as the expected parse
+  {
+    const int n = 2000;
+    const size_t size = 16 * (size_t) n + 16;
+    char *big = malloct(size);
+    size_t pos = 0;
+    int i;
+
+    assertf(big != NULL);
+    pos = (size_t) snprintf(big, size, "httrack");
+    assertf(pos < size);
+    for (i = 0; i < n; i++) {
+      // snprintf returns what it wanted to write, so accumulating it blind
+      // would let the next size argument wrap
+      const int len = snprintf(big + pos, size - pos, " a%d", i);
+
+      assertf(len > 0 && (size_t) len < size - pos);
+      pos += (size_t) len;
+    }
+    args = hts_split_cmdline(big, &nargs);
+    assertf(args != NULL && nargs == n + 1);
+    assertf(args[nargs] == NULL);
+    for (i = 0; i < n; i++) {
+      char expect[16];
+
+      snprintf(expect, sizeof(expect), "a%d", i);
+      assertf(strcmp(args[i + 1], expect) == 0);
+    }
+    freet(args);
+    freet(big);
+  }
+
+  printf("cmdline-split self-test OK\n");
+  return 0;
+}
+
 static int st_hashtable(httrackp *opt, int argc, char **argv) {
   char *snum;
   unsigned long count = 0;
@@ -1225,7 +1586,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
       size_t i;
       for (i = bench[loop].offset; i < (size_t) count;
            i += bench[loop].modulus) {
-        int result;
+        int result = 0; /* no final else: an unknown type reports failure */
         FMT();
         if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
           size_t k;
@@ -1299,6 +1660,14 @@ static int st_strsafe(httrackp *opt, int argc, char **argv) {
       htsbuff b = htsbuff_array(small);
 
       htsbuff_cat(&b, src);
+    } else if (strcmp(argv[0], "overflow-src") == 0) {
+      /* Array source with no NUL: its capacity still comes from sizeof(), so
+         the bounded strlen aborts rather than running off the array. */
+      char nonul[6]; /* never sizeof(char*), per the note above */
+      char big[64];
+
+      memset(nonul, src[0], sizeof(nonul));
+      strcpybuff(big, nonul);
     } else {
       strcpybuff(small, src);
     }
@@ -1372,6 +1741,12 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
   if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0)
     err = 1;
 
+  from->changes = HTS_TRUE;
+  to->changes = HTS_FALSE;
+  copy_htsopt(from, to);
+  if (to->changes != HTS_TRUE)
+    err = 1;
+
   /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
   from->pause_min_ms = 5000;
   from->pause_max_ms = 10000;
@@ -1993,6 +2368,77 @@ static int st_fsize(httrackp *opt, int argc, char **argv) {
   return rc;
 }
 
+/* 4GB+100KB wraps to ~108KB through an int, and needs 33 unsigned bits. A
+   macro, not a static const: MSVC's C mode (/TC) rejects a const object
+   used inside another object's static initializer below (C2099). */
+#define HTS_ST_GROWSIZE_OVER32 (4LL * 1024 * 1024 * 1024 + 100 * 1024)
+
+/* llint_grow_size_t() sizes the buffer holding a whole -%S list file: the
+   result must be the exact 64-bit sum or a clean refusal, never a short one. */
+static int st_growsize(httrackp *opt, int argc, char **argv) {
+  enum { REFUSE, ACCEPT, WIDTH };
+
+  static const struct {
+    size_t used;
+    LLint extra;
+    size_t slack;
+    int want;
+  } cases[] = {
+      {0, 0, 0, ACCEPT},
+      {10, 100, 8192, ACCEPT},
+      {(size_t) -2 - 8, 4, 4, ACCEPT}, /* exact fit, no room to spare */
+      {(size_t) -2, 0, 0, ACCEPT},     /* largest representable capacity */
+      {0, -1, 0, REFUSE},              /* fsize() failure */
+      /* -1 already maps to SIZE_MAX; only this exercises the negative guard */
+      {0, -4096, 0, REFUSE},
+      {(size_t) -1, 1, 0, REFUSE},
+      {(size_t) -2, 0, 1, REFUSE},     /* slack alone overruns */
+      {(size_t) -1 - 8, 4, 4, REFUSE}, /* total would be the error value */
+      {(size_t) -1 - 8, 4, 8, REFUSE},
+      {0, HTS_ST_GROWSIZE_OVER32, 8192,
+       WIDTH}, /* 32-bit size_t can't hold these */
+      {10, HTS_ST_GROWSIZE_OVER32, 8192, WIDTH},
+  };
+
+  size_t k;
+  int rc = 0;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+  for (k = 0; k < sizeof(cases) / sizeof(cases[0]); k++) {
+    const size_t used = cases[k].used, slack = cases[k].slack;
+    const LLint extra = cases[k].extra;
+    const size_t got = llint_grow_size_t(used, extra, slack);
+    const hts_boolean refused = got == (size_t) -1 ? HTS_TRUE : HTS_FALSE;
+    const hts_boolean exact =
+        !refused && extra >= 0 && got - used - slack == (size_t) extra;
+    hts_boolean ok;
+
+    switch (cases[k].want) {
+    case ACCEPT:
+      ok = exact;
+      break;
+    case REFUSE:
+      ok = refused;
+      break;
+    default:
+      ok = sizeof(size_t) >= sizeof(LLint) ? exact : refused;
+      break;
+    }
+    if (!ok) {
+      fprintf(stderr,
+              "growsize: grow(" LLintP ", " LLintP ", " LLintP ") = " LLintP
+              " (want %s)\n",
+              (LLint) used, extra, (LLint) slack, (LLint) got,
+              cases[k].want == REFUSE ? "refusal" : "exact sum");
+      rc = 1;
+    }
+  }
+  printf("growsize self-test %s\n", rc == 0 ? "OK" : "FAILED");
+  return rc;
+}
+
 static int st_savename(httrackp *opt, int argc, char **argv) {
   lien_adrfilsave afs;
   cache_back cache;
@@ -2387,17 +2833,20 @@ static int st_cookies(httrackp *opt, int argc, char **argv) {
     static t_cookie ck2;
     htsblk r;
     char host[600];
+    char line[64]; /* treathead NUL-cuts the header in place: never a literal */
 
     memset(&r, 0, sizeof(r));
     memset(host, 'a', sizeof(host) - 1);
     host[sizeof(host) - 1] = '\0';
     ck2.max_len = (int) sizeof(ck2.data);
     ck2.data[0] = '\0';
-    treathead(&ck2, host, "/", &r, "Set-Cookie: SID=1; path=/");
+    strcpybuff(line, "Set-Cookie: SID=1; path=/");
+    treathead(&ck2, host, "/", &r, line);
     if (strnotempty(ck2.data)) // oversize-host cookie was not dropped
       err = 1;
     /* control: a normal host still yields a cookie through treathead */
-    treathead(&ck2, dom, "/", &r, "Set-Cookie: SID=1; path=/");
+    strcpybuff(line, "Set-Cookie: SID=1; path=/");
+    treathead(&ck2, dom, "/", &r, line);
     if (strstr(ck2.data, "SID") == NULL) // guard wrongly dropped a valid cookie
       err = 1;
   }
@@ -2658,6 +3107,32 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
   assertf(strstr(buf, "Refresh") != NULL);
   assertf(strstr(buf, "example.com") != NULL);
 
+  /* a first link whose escaped form overruns the old flat 1024-byte tempo: the
+     redirect must carry the whole URL, not a clipped prefix */
+  {
+    char BIGSTK link[HTS_URLMAXSIZE * 2];
+    char *p = link;
+
+    strcpybuff(link, "http://example.com/");
+    p += strlen(link);
+    memset(p, 'a', 1200);
+    p += 1200;
+    strcpy(p, "/end.html");
+
+    done = 0;
+    fp = fopen(path, "wb");
+    assertf(fp != NULL);
+    hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
+    assertf(fp == NULL);
+    fp = fopen(path, "rb");
+    assertf(fp != NULL);
+    n = fread(buf, 1, sizeof(buf) - 1, fp);
+    fclose(fp);
+    buf[n] = '\0';
+    /* the closing quote proves the URL was not clipped mid-way */
+    assertf(strstr(buf, "/end.html\">") != NULL);
+  }
+
   /* no single link: footer only, no refresh meta */
   done = 0;
   fp = fopen(path, "wb");
@@ -2887,7 +3362,7 @@ static int ae_write_packed(const char *path, int windowBits,
     deflateEnd(&strm);
     return 1;
   }
-  strm.next_in = (Bytef *) src;
+  strm.next_in = (const Bytef *) src;
   strm.avail_in = (uInt) len;
   do {
     size_t n;
@@ -4697,10 +5172,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   char fpath[HTS_URLMAXSIZE * 2];
   char file[HTS_URLMAXSIZE * 2];
 
-  snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
+  assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
 
   /* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
-  snprintf(file, sizeof(file), "%scookies.txt", fpath);
+  assertf(sprintfbuff(file, "%scookies.txt", fpath));
   {
     FILE *fp = FOPEN(file, "wb");
 
@@ -4710,7 +5185,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   }
 
   /* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
-  snprintf(file, sizeof(file), "%su@v.txt", fpath);
+  assertf(sprintfbuff(file, "%su@v.txt", fpath));
   {
     FILE *fp = FOPEN(file, "wb");
 
@@ -4732,7 +5207,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
 #endif
 
   (void) UNLINK(file); /* u@v.txt (already gone on Windows) */
-  snprintf(file, sizeof(file), "%scookies.txt", fpath);
+  assertf(sprintfbuff(file, "%scookies.txt", fpath));
   (void) UNLINK(file);
   dir[dirlen] = '\0';
   while (strlen(dir) > base) {
@@ -4758,6 +5233,192 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* --changes bucket accounting and JSON escaping (#714). */
+static int st_changes(httrackp *opt, int argc, char **argv) {
+  String out = STRING_EMPTY;
+  int err = 0;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+  /* A file the crawl did not rewrite is unchanged whatever the wire said. */
+  assertf(hts_changes_classify(HTS_FALSE, HTS_TRUE, HTS_FALSE, HTS_FALSE,
+                               HTS_FALSE) == HTS_CHANGE_UNCHANGED);
+  /* Rewritten with no previous copy: new, digests or not. */
+  assertf(hts_changes_classify(HTS_TRUE, HTS_FALSE, HTS_FALSE, HTS_TRUE,
+                               HTS_FALSE) == HTS_CHANGE_NEW);
+  /* Digests decide, and outrank the transfer signal both ways: a server with
+     no validators answers 200 with the same bytes, and a 304 can still sit in
+     front of a locally damaged copy. */
+  assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_FALSE, HTS_TRUE,
+                               HTS_TRUE) == HTS_CHANGE_UNCHANGED);
+  assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_TRUE, HTS_TRUE,
+                               HTS_FALSE) == HTS_CHANGE_CHANGED);
+  /* Only with no digest at all does the transfer signal get a say. */
+  assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_TRUE, HTS_FALSE,
+                               HTS_FALSE) == HTS_CHANGE_UNCHANGED);
+  assertf(hts_changes_classify(HTS_TRUE, HTS_TRUE, HTS_FALSE, HTS_FALSE,
+                               HTS_FALSE) == HTS_CHANGE_CHANGED);
+
+#define JSON_IS(SRC, WANT)                                                     \
+  do {                                                                         \
+    StringClear(out);                                                          \
+    hts_changes_json_string(&out, SRC);                                        \
+    if (strcmp(StringBuff(out), WANT) != 0) {                                  \
+      fprintf(stderr, "changes: %s -> %s, expected %s\n", #SRC,                \
+              StringBuff(out), WANT);                                          \
+      err = 1;                                                                 \
+    }                                                                          \
+  } while (0)
+
+  JSON_IS("/a/b.html", "\"/a/b.html\"");
+  JSON_IS("a\"b\\c", "\"a\\\"b\\\\c\"");
+  JSON_IS("tab\there", "\"tab\\u0009here\"");
+  /* Valid UTF-8 rides through; a lone Latin-1 byte, a truncated sequence and
+     an overlong encoding of '/' each become U+FFFD rather than invalid JSON. */
+  JSON_IS("caf\xc3\xa9", "\"caf\xc3\xa9\"");
+  JSON_IS("caf\xe9", "\"caf\\ufffd\"");
+  JSON_IS("\xc3", "\"\\ufffd\"");
+  JSON_IS("\xc0\xaf", "\"\\ufffd\\ufffd\"");
+  /* A UTF-16 surrogate half is well-formed UTF-8 by shape only. */
+  JSON_IS("\xed\xa0\x80", "\"\\ufffd\\ufffd\\ufffd\"");
+
+#undef JSON_IS
+  StringFree(out);
+  printf("changes self-test: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
+#define CHANGES_RACE_FILES 8
+#define CHANGES_RACE_ROUNDS 400
+
+static void changes_race_notify(httrackp *opt, int n) {
+  char fil[64];
+  char BIGSTK save[HTS_URLMAXSIZE * 2];
+
+  snprintf(fil, sizeof(fil), "/f%d.bin", n);
+  strlcpybuff(save, StringBuff(opt->path_html), sizeof(save));
+  strlcatbuff(save, "race.example", sizeof(save));
+  strlcatbuff(save, fil, sizeof(save));
+  hts_changes_notify(opt, "race.example", fil, save, HTS_TRUE, HTS_FALSE);
+}
+
+/* htsthread_wait() counts a thread only once it is running, so it can return
+   before any of them started; join on our own counter instead. */
+static htsmutex changes_race_lock = HTSMUTEX_INIT;
+static int changes_race_started = 0;
+static int changes_race_live = 0;
+
+static int changes_race_count(int *which) {
+  int n;
+
+  hts_mutexlock(&changes_race_lock);
+  n = *which;
+  hts_mutexrelease(&changes_race_lock);
+  return n;
+}
+
+static void changes_race_thread(void *arg) {
+  httrackp *const opt = (httrackp *) arg;
+  int i;
+
+  hts_mutexlock(&changes_race_lock);
+  changes_race_started++;
+  hts_mutexrelease(&changes_race_lock);
+  for (i = 0; i < CHANGES_RACE_ROUNDS; i++)
+    changes_race_notify(opt, i % CHANGES_RACE_FILES);
+  hts_mutexlock(&changes_race_lock);
+  changes_race_live--;
+  hts_mutexrelease(&changes_race_lock);
+}
+
+/* A transfer thread the crawl never joins (FTP) reaches hts_changes_notify()
+   while the report is being resolved and written. Run it under TSan. */
+static int st_changes_race(httrackp *opt, int argc, char **argv) {
+  String out = STRING_EMPTY;
+  char base[HTS_URLMAXSIZE];
+  int err = 0;
+  int i;
+
+  if (argc < 1) {
+    fprintf(stderr, "usage: -#test=changes-race <writable directory>\n");
+    return 1;
+  }
+  strcpybuff(base, argv[0]);
+  if (base[0] != '\0' && base[strlen(base) - 1] != '/')
+    strcatbuff(base, "/");
+  StringCopy(opt->path_html, base);
+  StringCopy(opt->path_html_utf8, base);
+  StringCopy(opt->path_log, base);
+  opt->changes = HTS_TRUE;
+  hts_changes_free_opt(opt);
+
+  /* Real files, so the reader hashes and stats them for as long as it takes. */
+  {
+    char BIGSTK dir[HTS_URLMAXSIZE * 2];
+
+    strlcpybuff(dir, base, sizeof(dir));
+    strlcatbuff(dir, "race.example", sizeof(dir));
+    for (i = 0; i < CHANGES_RACE_FILES; i++) {
+      char BIGSTK path[HTS_URLMAXSIZE * 2];
+      char name[64];
+      FILE *fp;
+      int n;
+
+      snprintf(name, sizeof(name), "/f%d.bin", i);
+      strlcpybuff(path, dir, sizeof(path));
+      strlcatbuff(path, name, sizeof(path));
+      structcheck(path);
+      fp = FOPEN(path, "wb");
+      if (fp == NULL) {
+        fprintf(stderr, "changes-race: cannot write %s\n", path);
+        return 1;
+      }
+      for (n = 0; n < 16384; n++)
+        fwrite("0123456789abcdef", 1, 16, fp);
+      fclose(fp);
+    }
+  }
+
+  /* Take both locks once here: hts_mutexlock() initializes lazily, and two
+     threads reaching a fresh one together race on the init itself. */
+  hts_mutexlock(&changes_race_lock);
+  changes_race_started = 0;
+  changes_race_live = 4;
+  hts_mutexrelease(&changes_race_lock);
+  for (i = 0; i < 4; i++) {
+    if (hts_newthread(changes_race_thread, opt) != 0) {
+      fprintf(stderr, "changes-race: cannot spawn a notifier thread\n");
+      return 1;
+    }
+  }
+  /* Report only once they are all notifying, or there is nothing to race. */
+  while (changes_race_count(&changes_race_started) < 4)
+    Sleep(10);
+  for (i = 0; i < 64; i++)
+    hts_changes_report(opt, &out);
+  hts_changes_close_opt(opt);
+  while (changes_race_count(&changes_race_live) > 0)
+    Sleep(10);
+
+  /* Sealed: a straggler must be dropped, not start a report nobody writes. */
+  changes_race_notify(opt, CHANGES_RACE_FILES + 1);
+  hts_changes_report(opt, &out);
+  if (StringLength(out) == 0) {
+    fprintf(stderr, "changes-race: the report was lost after close\n");
+    err = 1;
+  } else if (strstr(StringBuff(out), "f9.bin") != NULL) {
+    fprintf(stderr, "changes-race: a post-close notify reached the report\n");
+    err = 1;
+  }
+
+  StringFree(out);
+  hts_changes_free_opt(opt);
+  printf("changes-race self-test: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
 /* ------------------------------------------------------------ */
 /* Registry: name -> handler, with a usage hint and a one-line description. */
 /* ------------------------------------------------------------ */
@@ -4806,10 +5467,17 @@ static const struct selftest_entry {
      st_footerfmt},
     {"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
      st_unescape_bounds},
+    {"cmdline-split", "",
+     "webhttrack command-line to argv split (bounds, quoting)",
+     st_cmdlinesplit},
     {"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
-    {"strsafe", "[overflow|overflow-buff [str]]", "bounded string-op self-test",
-     st_strsafe},
+    {"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
+     "bounded string-op self-test", st_strsafe},
     {"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt},
+    {"changes", "", "--changes bucket accounting and JSON escaping (#714)",
+     st_changes},
+    {"changes-race", "<dir>", "--changes under a late transfer thread (#714)",
+     st_changes_race},
     {"pause", "", "randomized inter-file pause target self-test", st_pause},
     {"relative", "<link> <curr-file>", "relative link between two paths",
      st_relative},
@@ -4838,6 +5506,8 @@ static const struct selftest_entry {
     {"sniff", "<content-type> <hex:..|text>", "MIME magic consistency",
      st_sniff},
     {"fsize", "<dir>", "file size past the 2GB signed-32-bit wrap", st_fsize},
+    {"growsize", "", "buffer capacity for a 64-bit file size (no int wrap)",
+     st_growsize},
     {"cache", "<dir>", "cache read/write round-trip self-test", st_cache},
     {"cacheindex", "", "cache-index (.ndx) parse must stay in bounds",
      st_cacheindex},

src/htsserver.c

@@ -147,7 +147,8 @@ HTS_UNUSED static int LANG_LIST(const char *path, char *buffer, size_t size);
 // 0- Init the URL catcher with standard port
 
 // smallserver_init(&port,&return_host);
-T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort) {
+T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort,
+                           const char *bindAddr) {
   T_SOC soc;
 
   if (defaultPort <= 0) {
@@ -160,12 +161,12 @@ T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort) {
     int i = 0;
 
     do {
-      soc = smallserver_init(&try_to_listen_to[i], adr_prox);
+      soc = smallserver_init(&try_to_listen_to[i], adr_prox, bindAddr);
       *port_prox = try_to_listen_to[i];
       i++;
     } while((soc == INVALID_SOCKET) && (try_to_listen_to[i] >= 0));
   } else {
-    soc = smallserver_init(&defaultPort, adr_prox);
+    soc = smallserver_init(&defaultPort, adr_prox, bindAddr);
     *port_prox = defaultPort;
   }
   return soc;
@@ -243,9 +244,10 @@ static int my_gethostname(char *h_loc, size_t size) {
 }
 
 // smallserver_init(&port,&return_host);
-T_SOC smallserver_init(int *port, char *adr) {
+T_SOC smallserver_init(int *port, char *adr, const char *bindAddr) {
   T_SOC soc = INVALID_SOCKET;
   char h_loc[256 + 2];
+  SOCaddr server;
 
   commandRunning = commandEnd = commandReturn = commandReturnSet =
     commandEndRequested = 0;
@@ -256,25 +258,23 @@ T_SOC smallserver_init(int *port, char *adr) {
     free(commandReturnCmdl);
   commandReturnCmdl = NULL;
 
-  if (my_gethostname(h_loc, 256) == 0) {   // host name
-    SOCaddr server;
+  SOCaddr_initany(server);
+  if (bindAddr != NULL && *bindAddr != '\0') {
+    /* advertise the bound address, else the URL we print is unreachable */
+    if (strlen(bindAddr) >= sizeof(h_loc) || !gethost(bindAddr, &server)) {
+      return INVALID_SOCKET;
+    }
+    strcpybuff(h_loc, bindAddr);
+  } else if (my_gethostname(h_loc, 256) != 0) { // host name
+    return INVALID_SOCKET;
+  }
 
-    SOCaddr_initany(server);
-    if ((soc =
-         (T_SOC) socket(SOCaddr_sinfamily(server), SOCK_STREAM,
-                        0)) != INVALID_SOCKET) {
-      SOCaddr_initport(server, *port);
-      if (bind(soc, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
-        if (listen(soc, 10) >= 0) {
-          strcpy(adr, h_loc);
-        } else {
-#ifdef _WIN32
-          closesocket(soc);
-#else
-          close(soc);
-#endif
-          soc = INVALID_SOCKET;
-        }
+  if ((soc = (T_SOC) socket(SOCaddr_sinfamily(server), SOCK_STREAM, 0)) !=
+      INVALID_SOCKET) {
+    SOCaddr_initport(server, *port);
+    if (bind(soc, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
+      if (listen(soc, 10) >= 0) {
+        strcpy(adr, h_loc);
       } else {
 #ifdef _WIN32
         closesocket(soc);
@@ -283,6 +283,13 @@ T_SOC smallserver_init(int *port, char *adr) {
 #endif
         soc = INVALID_SOCKET;
       }
+    } else {
+#ifdef _WIN32
+      closesocket(soc);
+#else
+      close(soc);
+#endif
+      soc = INVALID_SOCKET;
     }
   }
   return soc;
@@ -311,6 +318,116 @@ typedef struct {
   error_redirect = "/server/error.html"; \
 } while(0)
 
+/* Longest error message shown on the error page; the rest is clipped. */
+#define ERROR_MESSAGE_MAX 1024
+
+/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
+   whose length the client picks. */
+#define SET_ERRORF(...)                                                        \
+  do {                                                                         \
+    char errbuf[ERROR_MESSAGE_MAX];                                            \
+    slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__);                    \
+    SET_ERROR(errbuf);                                                         \
+  } while (0)
+
+/* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
+   so anything near this is already invalid and is rejected unread. */
+#define SID_VALUE_MAX 64
+
+/** Does the urlencoded request body present the expected session id?
+    True only if at least one "sid" field is present and every occurrence
+    matches, so it holds whichever one a later last-write-wins parse keeps.
+    Non-destructive: it runs before the body is tokenized in place. */
+static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
+  const char *s = body;
+  hts_boolean seen = HTS_FALSE;
+
+  while (s != NULL && *s != '\0') {
+    const char *const amp = strchr(s, '&');
+    const char *const eq = strchr(s, '=');
+
+    if (eq != NULL && (amp == NULL || eq < amp) && (size_t) (eq - s) == 3 &&
+        strncmp(s, "sid", 3) == 0) {
+      const size_t len = amp != NULL ? (size_t) (amp - eq - 1) : strlen(eq + 1);
+      hts_boolean match = HTS_FALSE;
+
+      if (len < SID_VALUE_MAX) {
+        char raw[SID_VALUE_MAX];
+        String value = STRING_EMPTY;
+
+        memcpy(raw, eq + 1, len);
+        raw[len] = '\0';
+        unescapehttp(raw, &value);
+        /* StringBuff is NULL until written, so an empty value lands here. */
+        if (StringBuff(value) != NULL &&
+            strcmp(StringBuff(value), expected) == 0) {
+          match = HTS_TRUE;
+        }
+        StringFree(value);
+      }
+      if (!match) {
+        return HTS_FALSE;
+      }
+      seen = HTS_TRUE;
+    }
+    s = amp != NULL ? amp + 1 : NULL;
+  }
+  return seen;
+}
+
+/** Append src to the NUL-terminated dst of capacity size (NUL included).
+    False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
+    never aborts, because every piece appended here is client-supplied. */
+static hts_boolean path_append(char *dst, size_t size, const char *src) {
+  const size_t used = strlen(dst);
+  const size_t len = strlen(src);
+
+  /* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
+     len stays alone: "used + len < size" could wrap and pass. */
+  if (len >= size - used) {
+    return HTS_FALSE;
+  }
+  memcpy(dst + used, src, len + 1);
+  return HTS_TRUE;
+}
+
+/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
+static hts_boolean cat_html_escaped(String *dst, char c) {
+  switch (c) {
+  case '<':
+    StringCat(*dst, "&lt;");
+    break;
+  case '>':
+    StringCat(*dst, "&gt;");
+    break;
+  case '&':
+    StringCat(*dst, "&amp;");
+    break;
+  case '\'':
+    StringCat(*dst, "&#39;");
+    break;
+  default:
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Append the value of a double-quoted command-line argument: escaped for HTML,
+   which the browser undoes when it posts the command line back, and for the
+   argv splitter, which does not. */
+static void cat_cmdline_arg(String *output, const char *value) {
+  const char *a;
+
+  for (a = value; *a != '\0'; a++) {
+    if (*a == '\\' || *a == '\"') {
+      StringCat(*output, "\\");
+    }
+    if (!cat_html_escaped(output, *a)) {
+      StringMemcat(*output, a, 1);
+    }
+  }
+}
+
 int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   int timeout = 30;
   int retour = 0;
@@ -322,6 +439,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   String tmpbuff = STRING_EMPTY;
   String tmpbuff2 = STRING_EMPTY;
   String fspath = STRING_EMPTY;
+  /* Project directory this server set up; the only root /website/ serves from,
+     and deliberately not cleared between requests. */
+  String website = STRING_EMPTY;
   char catbuff[CATBUFF_SIZE];
 
   /* Load strings */
@@ -395,6 +515,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
     T_SOC soc_c;
     LLint length = 0;
     const char *error_redirect = NULL;
+    hts_boolean denied = HTS_FALSE;
 
     line[0] = '\0';
     buffer[0] = '\0';
@@ -506,6 +627,22 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
         }
       }
 
+      /* Authenticate the body before parsing it: every field it carries is
+         written straight into the global key store below, "command" included,
+         and that one reaches the engine. Checking afterwards cannot work — the
+         damage is already done, and the pre-seeded "sid" above would compare
+         equal to itself for a request that simply omits the field. */
+      if (meth && buffer[0]) {
+        intptr_t expected = 0;
+
+        if (!coucal_readptr(NewLangList, "_sid", &expected) ||
+            !body_sid_is_valid(buffer, (const char *) expected)) {
+          buffer[0] = '\0';
+          meth = 0;
+          denied = HTS_TRUE;
+        }
+      }
+
       /* check variables */
       if (meth && buffer[0]) {
         char *s = buffer;
@@ -526,20 +663,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
         }
       }
 
-      /* Error check */
-      {
-        intptr_t adr = 0;
-        intptr_t adr2 = 0;
-
-        if (coucal_readptr(NewLangList, "sid", &adr)) {
-          if (coucal_readptr(NewLangList, "_sid", &adr2)) {
-            if (strcmp((char *) adr, (char *) adr2) != 0) {
-              meth = 0;
-            }
-          }
-        }
-      }
-
       /* Check variables (internal) */
       if (meth) {
         int doLoad = 0;
@@ -614,7 +737,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           fp = fopen(StringBuff(fspath), "rb");
           if (fp) {
             /* Read file */
-            while(!feof(fp)) {
+            while (!feof(fp) && !ferror(fp)) {
               char *str = line;
               char *pos;
 
@@ -730,6 +853,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                   if (!structcheck(StringBuff(tmpbuff))) {
                     FILE *fp;
 
+                    /* Both halves of fspath come from posted fields, so a ".."
+                       in them would escape the mirror once served. */
+                    if (strstr(StringBuff(fspath), "..") == NULL) {
+                      StringCopy(website, StringBuff(fspath));
+                    }
                     StringCat(tmpbuff, "winprofile.ini");
                     fp = fopen(StringBuff(tmpbuff), "wb");
                     if (fp != NULL) {
@@ -763,28 +891,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                           commandEnd = 1;
                         }
                       } else {
-                        char tmp[1024];
-
-                        sprintf(tmp,
-                                "Unable to write %d bytes in the the init file %s",
-                                count, StringBuff(fspath));
-                        SET_ERROR(tmp);
+                        SET_ERRORF(
+                            "Unable to write %d bytes in the the init file %s",
+                            count, StringBuff(fspath));
                       }
                       fclose(fp);
                     } else {
-                      char tmp[1024];
-
-                      sprintf(tmp, "Unable to create the init file %s",
-                              StringBuff(fspath));
-                      SET_ERROR(tmp);
+                      SET_ERRORF("Unable to create the init file %s",
+                                 StringBuff(fspath));
                     }
                   } else {
-                    char tmp[1024];
-
-                    sprintf(tmp,
-                            "Unable to create the directory structure in %s",
-                            StringBuff(fspath));
-                    SET_ERROR(tmp);
+                    SET_ERRORF("Unable to create the directory structure in %s",
+                               StringBuff(fspath));
                   }
 
                 } else {
@@ -802,7 +920,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
 
       /* Response */
       if (meth) {
-        int virtualpath = 0;
+        hts_boolean virtualpath = HTS_FALSE;
         char *pos;
         char *url = strchr(line1, ' ');
 
@@ -813,11 +931,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           char *qpos;
 
           /* get the URL */
+          fsfile[0] = '\0';
           if (error_redirect == NULL) {
             if ((qpos = strchr(url, '?'))) {
               *qpos = '\0';
             }
-            fsfile[0] = '\0';
             if (strcmp(url, "/") == 0) {
               file = "/server/index.html";
               meth = 2;
@@ -830,7 +948,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           }
 
           if (strncmp(file, "/website/", 9) == 0) {
-            virtualpath = 1;
+            virtualpath = HTS_TRUE;
           }
 
           /* override */
@@ -844,20 +962,28 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             }
           }
 
-          if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
-            if (strncmp(file, "/website/", 9) != 0) {
-              sprintf(fsfile, "%shtml%s", path, file);
-            } else {
-              intptr_t adr = 0;
+          /* the override above may have swapped a mirror path for a GUI page */
+          virtualpath = strncmp(file, "/website/", 9) == 0;
 
-              if (coucal_readptr(NewLangList, "projpath", &adr)) {
-                sprintf(fsfile, "%s%s", (char *) adr, file + 9);
-              }
+          if (!virtualpath) {
+            if (!path_append(fsfile, sizeof(fsfile), path) ||
+                !path_append(fsfile, sizeof(fsfile), "html") ||
+                !path_append(fsfile, sizeof(fsfile), file)) {
+              fsfile[0] = '\0';
+            }
+          } else if (StringNotEmpty(website)) {
+            /* Never the posted "projpath": a client root reads any file. */
+            if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
+                !path_append(fsfile, sizeof(fsfile), "/") ||
+                !path_append(fsfile, sizeof(fsfile), file + 9)) {
+              fsfile[0] = '\0';
             }
           }
 
-          if (fsfile[0] && strstr(file, "..") == NULL
-              && (fp = fopen(fsfile, "rb"))) {
+          /* Regular files only: reading a directory or FIFO never ends, and
+             "path" may hold "..", so only the untrusted halves are checked. */
+          if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
+              (fp = fopen(fsfile, "rb"))) {
             char ok[] =
               "HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
               "Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
@@ -903,20 +1029,20 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                 }
               }
               StringMemcat(headers, redir, strlen(redir));
-              {
-                char tmp[256];
-
-                if (strlen(file) < sizeof(tmp) - 32) {
-                  sprintf(tmp, "Location: %s\r\n", newfile);
-                  StringMemcat(headers, tmp, strlen(tmp));
-                }
+              /* client-supplied: a CR/LF here would split the response */
+              if (newfile[strcspn(newfile, "\r\n")] == '\0') {
+                StringCat(headers, "Location: ");
+                StringCat(headers, newfile);
+                StringCat(headers, "\r\n");
               }
               coucal_write(NewLangList, "redirect", (intptr_t) NULL);
-            } else if (is_html(file)) {
+            } else if (!virtualpath && is_html(file)) {
+              /* GUI templates only: ${_sid} in a mirrored page would hand the
+                 crawled site the session id that authenticates commands */
               int outputmode = 0;
 
               StringMemcat(headers, ok, sizeof(ok) - 1);
-              while(!feof(fp)) {
+              while (!feof(fp) && !ferror(fp)) {
                 char *str = line;
                 int prevlen = (int) StringLength(output);
                 int nocr = 0;
@@ -940,6 +1066,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                     int p;
                     int format = 0;
                     int listDefault = 0;
+                    hts_boolean unquoted = HTS_FALSE;
 
                     name[0] = '\0';
                     strlncatbuff(name, str, sizeof(name_), n);
@@ -949,6 +1076,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                     } else if ((p = strfield(name, "html:"))) {
                       name += p;
                       format = 1;
+                    } else if ((p = strfield(name, "unquoted:"))) {
+                      name += p;
+                      unquoted = HTS_TRUE;
+                    } else if ((p = strfield(name, "arg:"))) {
+                      name += p;
+                      format = 5;
                     } else if ((p = strfield(name, "list:"))) {
                       name += p;
                       format = 2;
@@ -1085,8 +1218,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                        test:<if ==0>:<if ==1>:<if == 2>..
                        ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
                      */
-                    else if ((p = strfield(name, "test:"))
-                             || (p = strfield(name, "ztest:"))) {
+                    else if ((p = strfield(name, "test:")) ||
+                             (p = strfield(name, "ztest:"))) {
                       intptr_t adr = 0;
                       char *pos2;
                       int ztest = (name[0] == 'z');
@@ -1189,6 +1322,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                         }
                       }
                     }
+                    /* consumed here: it shares nothing with the list and
+                       option formats below */
+                    if (format == 5 && langstr != NULL && outputmode != -1) {
+                      cat_cmdline_arg(&output, langstr);
+                      langstr = NULL;
+                    }
                     if (langstr && outputmode != -1) {
                       switch (format) {
                       case 0:
@@ -1206,18 +1345,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                                 StringMemcat(output, &c, 1);
                               }
                               a += 2;
-                            } else if (outputmode && a[0] == '<') {
-                              StringCat(output, "&lt;");
-                            } else if (outputmode && a[0] == '>') {
-                              StringCat(output, "&gt;");
-                            } else if (outputmode && a[0] == '&') {
-                              StringCat(output, "&amp;");
-                            } else if (outputmode && a[0] == '\'') {
-                              StringCat(output, "&#39;");
+                            } else if (unquoted && a[0] == '\"') {
+                              /* the browser posts an entity back as a raw
+                                 quote, which would open a quoted run in the
+                                 argv splitter; a URI cannot hold one anyway */
+                              StringCat(output, "%22");
+                            } else if (outputmode &&
+                                       cat_html_escaped(&output, a[0])) {
+                              /* appended as an entity */
                             } else if (outputmode == 3 && a[0] == ' ') {
                               StringCat(output, "%20");
-                            } else if (outputmode >= 2
-                                       && ((unsigned char) a[0]) < 32) {
+                            } else if (outputmode >= 2 &&
+                                       ((unsigned char) a[0]) < 32) {
                               char tmp[32];
 
                               sprintf(tmp, "%%%02x", (unsigned char) a[0]);
@@ -1278,20 +1417,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                               }
                               StringClear(tmpbuff);
                               break;
-                            case '<':
-                              StringCat(tmpbuff, "&lt;");
-                              break;
-                            case '>':
-                              StringCat(tmpbuff, "&gt;");
-                              break;
-                            case '&':
-                              StringCat(tmpbuff, "&amp;");
-                              break;
-                            case '\'':
-                              StringCat(tmpbuff, "&#39;");
-                              break;
                             default:
-                              StringMemcat(tmpbuff, fstr, 1);
+                              if (!cat_html_escaped(&tmpbuff, *fstr)) {
+                                StringMemcat(tmpbuff, fstr, 1);
+                              }
                               break;
                             }
                             fstr++;
@@ -1331,7 +1460,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               }
 #endif
             } else {
-              if (is_text(file)) {
+              if (is_html(file)) {
+                StringMemcat(headers, ok, sizeof(ok) - 1);
+              } else if (is_text(file)) {
                 StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
               } else if (is_js(file)) {
                 StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
@@ -1345,14 +1476,15 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               while(!feof(fp)) {
                 int n = (int) fread(line, 1, sizeof(line) - 2, fp);
 
-                if (n > 0) {
-                  StringMemcat(output, line, n);
+                if (n <= 0) {
+                  break; /* short read: EOF or error, never a retry */
                 }
+                StringMemcat(output, line, n);
               }
             }
             fclose(fp);
-          } else if (strcmp(file, "/ping") == 0
-                     || strncmp(file, "/ping?", 6) == 0) {
+          } else if (strcmp(file, "/ping") == 0 ||
+                     strncmp(file, "/ping?", 6) == 0) {
             char error_hdr[] =
               "HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
               "Content-type: text/html\r\n";
@@ -1368,6 +1500,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             StringCat(output, error);
           }
         }
+      } else if (denied) {
+        StringCat(headers, "HTTP/1.0 403 Forbidden\r\n"
+                           "Server: httrack small server\r\n"
+                           "Content-type: text/html\r\n");
+        StringCat(output, "Missing or invalid session id.\r\n");
       } else {
 #ifdef _DEBUG
         char error_hdr[] =
@@ -1433,6 +1570,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   StringFree(tmpbuff);
   StringFree(tmpbuff2);
   StringFree(fspath);
+  StringFree(website);
 
   if (buffer)
     free(buffer);

src/htsserver.h

@@ -43,8 +43,11 @@ Please visit our Website: http://www.httrack.com
 
 // Fonctions
 void socinput(T_SOC soc, char *s, int max);
-T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort);
-T_SOC smallserver_init(int *port, char *adr);
+/* Listen on bindAddr, or every interface if NULL/empty; adr (>= 258 bytes) gets
+   the address to advertise. INVALID_SOCKET on error. */
+T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort,
+                           const char *bindAddr);
+T_SOC smallserver_init(int *port, char *adr, const char *bindAddr);
 int smallserver(T_SOC soc, char *url, char *method, char *data, char *path);
 
 #define CATCH_RESPONSE \

src/htsstats.h

@@ -0,0 +1,63 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: in-progress display rows, shared by httrack and htsserver .h */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_STATS_DEFH
+#define HTS_STATS_DEFH
+
+#include "htsglobal.h"
+
+/* One row of the "in progress" display, shared so the CLI (httrack) and the
+   web GUI (htsserver) cannot drift apart: each fills its own array. */
+
+#define NStatsBuffer 14
+
+#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
+#define HTS_DEF_FWSTRUCT_t_StatsBuffer
+typedef struct t_StatsBuffer t_StatsBuffer;
+#endif
+struct t_StatsBuffer {
+  char name[1024];
+  char file[1024];
+  char state[288];                         // a short label plus back->info[256]
+  char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
+  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
+  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
+  LLint size;
+  LLint sizetot;
+  int offset;
+  //
+  int back;
+  //
+  int actived; // pour disabled
+};
+
+#endif

src/htstools.c

@@ -308,8 +308,10 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
   // copy only the current path
   curr = _curr;
   strlcpybuff(curr, curr_fil, sizeof(_curr));
-  if ((a = strchr(curr, '?')) == NULL)  // couper au ? (params)
-    a = curr + strlen(curr) - 1;        // pas de params: aller à la fin
+  if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
+    // an empty path has no last character: curr-1 would read before the buffer
+    a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
+  }
   while((*a != '/') && (a > curr))
     a--;                        // chercher dernier / du chemin courant
   if (*a == '/')
@@ -1318,19 +1320,19 @@ HTSEXT_API hts_boolean hts_findnext(find_handle find) {
   if (find) {
 #ifdef _WIN32
     if ((FindNextFileA(find->handle, &find->hdata)))
-      return 1;
+      return HTS_TRUE;
 #else
     char catbuff[CATBUFF_SIZE];
 
     memset(&(find->filestat), 0, sizeof(find->filestat));
     if ((find->dirp = readdir(find->hdir)))
-      if (find->dirp->d_name)
-        if (!STAT
-            (concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name), &find->filestat))
-          return 1;
+      if (!STAT(
+              concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name),
+              &find->filestat))
+        return HTS_TRUE;
 #endif
   }
-  return 0;
+  return HTS_FALSE;
 }
 
 HTSEXT_API int hts_findclose(find_handle find) {

src/htswarc.c

@@ -462,22 +462,6 @@ static void warc_make_id(warc_writer *w, char out[64]) {
            b[11], b[12], b[13], b[14], b[15]);
 }
 
-static void warc_now_iso8601(char out[32]) {
-  time_t t = time(NULL);
-  struct tm tmv;
-#if defined(_WIN32)
-  struct tm *g = gmtime(&t);
-  if (g != NULL)
-    tmv = *g;
-  else
-    memset(&tmv, 0, sizeof(tmv));
-#else
-  if (gmtime_r(&t, &tmv) == NULL)
-    memset(&tmv, 0, sizeof(tmv));
-#endif
-  strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
-}
-
 /* Case-insensitive "is this the header named name?" test, tolerating optional
    whitespace before the ':' (non-compliant "Name : value" is still matched). */
 static int header_is(const char *line, size_t line_len, const char *name) {
@@ -1063,7 +1047,7 @@ static void warc_wacz_package(warc_writer *w) {
   wbuf_free(&pages);
 
   /* datapackage.json listing every stored file with its sha256 + size. */
-  warc_now_iso8601(created);
+  hts_now_iso8601(created);
   memset(&dp, 0, sizeof(dp));
   if (!err &&
       (wbuf_printf(&dp,
@@ -1166,7 +1150,7 @@ static int warc_emit(warc_writer *w, const char *type, const char *content_type,
 
   memset(&hdr, 0, sizeof(hdr));
   warc_make_id(w, id);
-  warc_now_iso8601(date);
+  hts_now_iso8601(date);
 
 #if HTS_USEOPENSSL
   /* Block digest over the whole block, in one streaming pass. */
@@ -1670,7 +1654,7 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
     char date[32];
     char *slot;
     size_t need;
-    warc_now_iso8601(date);
+    hts_now_iso8601(date);
     need = strlen(target_uri) + 1 + strlen(date) + 1;
     slot = malloct(need);
     if (slot != NULL) {

src/htsweb.c

@@ -62,6 +62,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsmd5.c"
 #include "md5.c"
 
+#include "htscmdline.h"
 #include "htsserver.h"
 #include "htsurlport.h"
 #include "htsweb.h"
@@ -88,7 +89,7 @@ Please visit our Website: http://www.httrack.com
 
 static htsmutex refreshMutex = HTSMUTEX_INIT;
 
-static int help_server(char *dest_path, int defaultPort);
+static int help_server(char *dest_path, int defaultPort, const char *bindAddr);
 extern int commandRunning;
 extern int commandEnd;
 extern int commandReturn;
@@ -153,6 +154,8 @@ int main(int argc, char *argv[]) {
   int ret = 0;
   int defaultPort = 0;
   int parentPid = 0;
+  /* loopback by default: the handler trusts its input; --bind widens it */
+  const char *bindAddr = "127.0.0.1";
 
   printf("Initializing the server..\n");
 
@@ -179,7 +182,8 @@ int main(int argc, char *argv[]) {
   if (argc < 2 || (argc % 2) != 0) {
     fprintf(stderr, "** Warning: use the webhttrack frontend if available\n");
     fprintf(stderr,
-            "usage: %s [--port <port>] [--ppid parent-pid] <path-to-html-root-dir> [key value [key value]..]\n",
+            "usage: %s [--port <port>] [--bind <address>] [--ppid parent-pid] "
+            "<path-to-html-root-dir> [key value [key value]..]\n",
             argv[0]);
     fprintf(stderr, "example: %s /usr/share/httrack/\n", argv[0]);
     return 1;
@@ -267,6 +271,14 @@ int main(int argc, char *argv[]) {
         fprintf(stderr, "couldn't set the port number to %s\n", argv[i + 1]);
         return -1;
       }
+    } else if (strcmp(argv[i], "--bind") == 0 && i + 1 < argc) {
+      /* empty would fall back to every interface, silently undoing the default
+       */
+      if (!strnotempty(argv[i + 1])) {
+        fprintf(stderr, "--bind needs an address\n");
+        return -1;
+      }
+      bindAddr = argv[i + 1];
     } else if (strcmp(argv[i], "--ppid") == 0 && i + 1 < argc) {
       if (sscanf(argv[i + 1], "%u", &parentPid) != 1) {
         fprintf(stderr, "couldn't set the parent PID to %s\n", argv[i + 1]);
@@ -293,7 +305,7 @@ int main(int argc, char *argv[]) {
   }
 
   /* launch */
-  ret = help_server(argv[1], defaultPort);
+  ret = help_server(argv[1], defaultPort, bindAddr);
 
   htsthread_wait_n(background_threads - 1);
   hts_uninit();
@@ -308,10 +320,8 @@ int main(int argc, char *argv[]) {
 static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
 static void back_launch_cmd(void *pP) {
   char *cmd = (char *) pP;
-  char **argv = (char **) malloct(1024 * sizeof(char *));
+  char **argv;
   int argc = 0;
-  int i = 0;
-  int g = 0;
 
   //
   httrackp *opt;
@@ -322,28 +332,19 @@ static void back_launch_cmd(void *pP) {
   commandReturnCmdl = strdup(cmd);
 
   /* split */
-  argv[0] = strdup("webhttrack");
-  argv[1] = cmd;
-  argc++;
-  i = 0;
-  while(cmd[i]) {
-    if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
-      cmd[i] = ' ';
-    }
-    i++;
-  }
-  i = 0;
-  while(cmd[i]) {
-    if (cmd[i] == '\"')
-      g = !g;
-    if (cmd[i] == ' ') {
-      if (!g) {
-        cmd[i] = '\0';
-        argv[argc++] = cmd + i + 1;
-      }
-    }
-    i++;
+  argv = hts_split_cmdline(cmd, &argc);
+  if (argv == NULL) {
+    if (commandReturnMsg)
+      free(commandReturnMsg);
+    commandReturnMsg = strdup("could not parse the command line");
+    commandReturn = -1;
+    commandRunning = 0;
+    commandEnd = 1;
+    free(cmd);
+    return;
   }
+  /* drop the program name the posted command line carries */
+  argv[0] = strdupt("webhttrack");
 
   /* init */
   hts_init();
@@ -372,6 +373,7 @@ static void back_launch_cmd(void *pP) {
 
   /* free */
   free(cmd);
+  freet(argv[0]);
   freet(argv);
   return;
 }
@@ -427,11 +429,11 @@ static int webhttrack_runmain(httrackp * opt, int argc, char **argv) {
   return ret;
 }
 
-static int help_server(char *dest_path, int defaultPort) {
+static int help_server(char *dest_path, int defaultPort, const char *bindAddr) {
   int returncode = 0;
   char adr_prox[HTS_URLMAXSIZE * 2];
   int port_prox;
-  T_SOC soc = smallserver_init_std(&port_prox, adr_prox, defaultPort);
+  T_SOC soc = smallserver_init_std(&port_prox, adr_prox, defaultPort, bindAddr);
 
   if (soc != INVALID_SOCKET) {
     char url[HTS_URLMAXSIZE * 2];
@@ -671,10 +673,10 @@ int __cdecl htsshow_loop(t_hts_callbackarg * carg, httrackp * opt, lien_back * b
                     char *eps = strchr(back[i].url_adr, '/');
                     int count;
 
-                    if (ep != NULL && ep < eps
-                        && (count = (int) (ep - back[i].url_adr)) < 4) {
+                    if (ep != NULL && eps != NULL && ep < eps &&
+                        (count = (int) (ep - back[i].url_adr)) < 4) {
                       proto[0] = '\0';
-                      strncat(proto, back[i].url_adr, count);
+                      strncatbuff(proto, back[i].url_adr, count);
                     }
                   }
                   snprintf(StatsBuffer[index].state, sizeof(StatsBuffer[index].state),

src/htsweb.h

@@ -35,26 +35,10 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsstats.h"
 
-#define NStatsBuffer     14
 #define MAX_LEN_INPROGRESS 40
 
-typedef struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[256];
-  char url_sav[HTS_URLMAXSIZE * 2];     // pour cancel
-  char url_adr[HTS_URLMAXSIZE * 2];
-  char url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-} t_StatsBuffer;
-
 typedef struct t_InpInfo {
   int ask_refresh;
   int refresh;

src/httrack.c

@@ -46,6 +46,7 @@ Please visit our Website: http://www.httrack.com
 #include "httrack.h"
 #include "htslib.h"
 #include "htscharset.h" // after htslib.h: winsock2.h must precede windows.h
+#include "htsbacktrace.h"
 
 /* Static definitions */
 static int fexist(const char *s);
@@ -72,10 +73,6 @@ static int linput(FILE * fp, char *s, int max);
 #include <sys/ioctl.h>
 #endif
 #include <ctype.h>
-#if (defined(__linux) && defined(HAVE_EXECINFO_H))
-#include <execinfo.h>
-#define USES_BACKTRACE
-#endif
 /* END specific definitions */
 
 static void __cdecl htsshow_init(t_hts_callbackarg * carg);
@@ -233,8 +230,7 @@ static hts_boolean vt_size_refresh(void) {
 */
 #define STYLE_STATVALUES VT_BOLD
 #define STYLE_STATTEXT   VT_UNBOLD
-#define STYLE_STATRESET  VT_UNBOLD
-#define NStatsBuffer     14
+#define STYLE_STATRESET VT_UNBOLD
 /* Rows the stats block and "Current job" take above the in-progress list. */
 #define NStatsHeaderRows 7
 
@@ -880,21 +876,6 @@ static void sig_doback(int blind) {     // mettre en backing
 #undef FD_ERR
 #define FD_ERR 2
 
-static void print_backtrace(void) {
-#ifdef USES_BACKTRACE
-  void *stack[256];
-  const int size = backtrace(stack, sizeof(stack)/sizeof(stack[0]));
-  if (size != 0) {
-    backtrace_symbols_fd(stack, size, FD_ERR);
-  }
-#else
-  const char msg[] = "No stack trace available on this OS :(\n";
-  if (write(FD_ERR, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
-    /* sorry GCC */
-  }
-#endif
-}
-
 static size_t print_num(char *buffer, int num) {
   size_t i, j;
   if (num < 0) {
@@ -928,7 +909,7 @@ static void sig_fatal(int code) {
   size += print_num(&buffer[size], code);
   buffer[size++] = '\n';
   (void) (write(FD_ERR, buffer, size) == size);
-  print_backtrace();
+  hts_print_backtrace(FD_ERR);
   (void) (write(FD_ERR, msgreport, sizeof(msgreport) - 1)
     == sizeof(msgreport) - 1);
   abort();
@@ -951,6 +932,7 @@ static void sig_leave(int code) {
 }
 
 static void signal_handlers(void) {
+  hts_backtrace_init();
 #ifdef _WIN32
   signal(SIGINT, sig_leave);    // ^C
   signal(SIGTERM, sig_finish);  // kill <process>

src/httrack.h

@@ -36,26 +36,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsglobal.h"
 #include "htscore.h"
 #include "htssafe.h"
-
-#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
-#define HTS_DEF_FWSTRUCT_t_StatsBuffer
-typedef struct t_StatsBuffer t_StatsBuffer;
-#endif
-struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[256];
-  char BIGSTK url_sav[HTS_URLMAXSIZE * 2];      // pour cancel
-  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
-  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-};
+#include "htsstats.h"
 
 #ifndef HTS_DEF_FWSTRUCT_t_InpInfo
 #define HTS_DEF_FWSTRUCT_t_InpInfo

src/httrack.vcxproj

@@ -100,6 +100,7 @@
 
   <ItemGroup>
     <ClCompile Include="httrack.c" />
+    <ClCompile Include="htsbacktrace.c" />
   </ItemGroup>
 
   <!-- Pulls in libhttrack.lib and builds it first. -->

src/libhttrack.vcxproj

@@ -59,9 +59,10 @@
   <ItemDefinitionGroup>
     <ClCompile>
       <!-- LIBHTTRACK_EXPORTS turns HTSEXT_API into __declspec(dllexport); ZLIB_DLL
-           imports zlib from its DLL. HTS_USEBROTLI/HTS_USEZSTD enable the br and
-           zstd content codings. Windows 7 floor, matching WinHTTrack. -->
-      <PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+           imports zlib from its DLL, ZLIB_CONST makes its next_in const as the
+           autotools build does. HTS_USEBROTLI/HTS_USEZSTD enable the br and zstd
+           content codings. Windows 7 floor, matching WinHTTrack. -->
+      <PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;ZLIB_CONST;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>$(MSBuildThisFileDirectory);$(MSBuildThisFileDirectory)coucal;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <WarningLevel>Level3</WarningLevel>
       <MultiProcessorCompilation>true</MultiProcessorCompilation>
@@ -121,6 +122,7 @@
     <ClCompile Include="htshelp.c" />
     <ClCompile Include="htsindex.c" />
     <ClCompile Include="htslib.c" />
+    <ClCompile Include="htscmdline.c" />
     <ClCompile Include="htsurlport.c" />
     <ClCompile Include="htsmd5.c" />
     <ClCompile Include="htsmodules.c" />
@@ -136,6 +138,7 @@
     <ClCompile Include="htswrap.c" />
     <ClCompile Include="htszlib.c" />
     <ClCompile Include="htswarc.c" />
+    <ClCompile Include="htschanges.c" />
     <ClCompile Include="md5.c" />
     <ClCompile Include="minizip\ioapi.c" />
     <ClCompile Include="minizip\iowin32.c" />

src/proxy/proxytrack.c

@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     PT_Element elt = PT_ElementNew();
 
     elt->statuscode = 405;
-    strcpy(elt->msg, "Method Not Allowed");
+    strcpybuff(elt->msg, "Method Not Allowed");
     return elt;
   }
 
@@ -771,7 +771,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
                   PT_ReadIndex(indexes, StringBuff(itemUrl) + 1, FETCH_HEADERS);
                 if (file != NULL && file->statuscode == HTTP_OK) {
                   size = file->size;
-                  if (file->lastmodified) {
+                  if (file->lastmodified[0] != '\0') {
                     timestamp = get_time_rfc822(file->lastmodified);
                   }
                   if (timestamp == (time_t) 0) {
@@ -785,7 +785,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
                     }
                     timestamp = timestampRep;
                   }
-                  if (file->contenttype) {
+                  if (file->contenttype[0] != '\0') {
                     mimeType = file->contenttype;
                   }
                 }
@@ -820,9 +820,9 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     elt->size = StringLength(response);
     elt->adr = StringAcquire(&response);
     elt->statuscode = 207;      /* Multi-Status */
-    strcpy(elt->charset, "utf-8");
-    strcpy(elt->contenttype, "text/xml");
-    strcpy(elt->msg, "Multi-Status");
+    strcpybuff(elt->charset, "utf-8");
+    strcpybuff(elt->contenttype, "text/xml");
+    strcpybuff(elt->msg, "Multi-Status");
     StringFree(response);
 
     fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
@@ -877,9 +877,9 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
     elt->size = StringLength(html);
     elt->adr = StringAcquire(&html);
     elt->statuscode = HTTP_OK;
-    strcpy(elt->charset, "iso-8859-1");
-    strcpy(elt->contenttype, "text/html");
-    strcpy(elt->msg, "OK");
+    strcpybuff(elt->charset, "iso-8859-1");
+    strcpybuff(elt->contenttype, "text/html");
+    strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;
   }

src/proxy/proxytrack.h

@@ -368,7 +368,11 @@ HTS_UNUSED static struct tm PT_GetTime(time_t t) {
   if (tm != NULL)
     return *tm;
   else {
+    /* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
+       day of "00"; the epoch is the conventional "date unknown" */
     memset(&tmbuf, 0, sizeof(tmbuf));
+    tmbuf.tm_year = 70;
+    tmbuf.tm_mday = 1;
     return tmbuf;
   }
 }

src/proxy/store.c

@@ -405,9 +405,9 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
     elt->size = StringLength(html);
     elt->adr = StringAcquire(&html);
     elt->statuscode = HTTP_OK;
-    strcpy(elt->charset, "iso-8859-1");
-    strcpy(elt->contenttype, "text/html");
-    strcpy(elt->msg, "OK");
+    strcpybuff(elt->charset, "iso-8859-1");
+    strcpybuff(elt->contenttype, "text/html");
+    strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;
   }
@@ -579,9 +579,10 @@ PT_Index PT_LoadCache(const char *filename) {
         if (chain != NULL) {
           const char *scheme = link_has_authority(chain->name) ? "" : "http://";
 
-          snprintf(index->slots.common.startUrl,
-                   sizeof(index->slots.common.startUrl), "%s%s", scheme,
-                   (const char *) chain->name);
+          /* dropped rather than truncated: empty already reads as "unset" */
+          if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
+                           (const char *) chain->name))
+            index->slots.common.startUrl[0] = '\0';
         }
       }
     }
@@ -827,6 +828,10 @@ PT_Element PT_ElementNew(void) {
   return r;
 }
 
+/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
+#define PT_Element_failf(R, ...)                                               \
+  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
+
 PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
   if (index != NULL && SAFE_INDEX(index)) {
     return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
@@ -865,12 +870,15 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
   sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
   (headersSize) += (int) strlen(headers + headersSize); \
 } while(0)
-#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
-  if (line[0] != '\0' && strfield2(line, refline)) { \
-    strcpy(refvalue, value); \
-    line[0] = '\0'; \
-	} \
-} while(0)
+/* refvalue_size is mandatory: the cache line is bounded only by the line
+   buffer, not by the destination. */
+#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
+  do {                                                                         \
+    if (line[0] != '\0' && strfield2(line, refline)) {                         \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
+      line[0] = '\0';                                                          \
+    }                                                                          \
+  } while (0)
 #define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
   if (line[0] != '\0' && strfield2(line, refline)) { \
     int intval = 0; \
@@ -972,9 +980,12 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
                     const char *scheme =
                         link_has_authority(filenameIndex) ? "" : "http://";
 
-                    firstSeen = 1;
-                    snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
-                             scheme, filenameIndex);
+                    /* dropped rather than truncated; try the next entry */
+                    if (sprintfbuff(index->startUrl, "%s%s", scheme,
+                                    filenameIndex))
+                      firstSeen = 1;
+                    else
+                      index->startUrl[0] = '\0';
                   }
                 }
               } else {
@@ -1021,7 +1032,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
   previous_save[0] = previous_save_[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  strcpy(r->location, "");
+  r->location[0] = '\0';
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -1070,16 +1081,23 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                 value++;
               ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
               ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
-              ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg);     // msg
+              ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
+                                   sizeof(r->msg));
               ZIP_READFIELD_INT(line, value, "X-Size", r->size);        // size
-              ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype);        // contenttype
-              ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset);       // contenttype
-              ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified);      // last-modified
-              ZIP_READFIELD_STRING(line, value, "Etag", r->etag);       // Etag
-              ZIP_READFIELD_STRING(line, value, "Location", r->location);       // 'location' pour moved
+              ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
+                                   sizeof(r->contenttype));
+              ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
+                                   sizeof(r->charset));
+              ZIP_READFIELD_STRING(line, value, "Last-Modified",
+                                   r->lastmodified, sizeof(r->lastmodified));
+              ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
+                                   sizeof(r->etag));
+              ZIP_READFIELD_STRING(line, value, "Location", r->location,
+                                   sizeof(location_default));
               ZIP_READFIELD_STRING(line, value, "Content-Disposition",
-                                   r->cdispo); // Content-disposition
-              ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_);      // Original save filename
+                                   r->cdispo, sizeof(r->cdispo));
+              ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
+                                   sizeof(previous_save_));
               if (line[0] != '\0') {
                 int len = r->headers ? ((int) strlen(r->headers)) : 0;
                 int nlen =
@@ -1099,7 +1117,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
             int pathLen = (int) strlen(index->path);
 
             if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) {    // old (<3.40) buggy format
-              strcpy(previous_save, previous_save_);
+              strcpybuff(previous_save, previous_save_);
             }
             // relative ? (hack)
             else if (index->safeCache || (previous_save_[0] != '/'      // /home/foo/bar.gif
@@ -1135,8 +1153,9 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                   snprintf(previous_save, sizeof(previous_save), "%s%s",
                            index->path, previous_save_ + index->fixedPath);
                 } else {
-                  snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
-                          previous_save_, (int) index->fixedPath);
+                  PT_Element_failf(
+                      r, "Bogus fixePath prefix for %s (prefixLen=%d)",
+                      previous_save_, (int) index->fixedPath);
                   r->statuscode = STATUSCODE_INVALID;
                 }
               } else {
@@ -1155,7 +1174,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
             // Peut-on stocker le fichier directement sur disque?
             if (ok) {
               if (r->msg[0] == '\0') {
-                strcpy(r->msg, "Cache Read Error : Unexpected error");
+                strcpybuff(r->msg, "Cache Read Error : Unexpected error");
               }
             } else {            // lire en mémoire
 
@@ -1173,24 +1192,27 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                           int last_errno = errno;
 
                           r->statuscode = STATUSCODE_INVALID;
-                          sprintf(r->msg, "Read error in cache disk data: %s",
-                                  strerror(last_errno));
+                          PT_Element_failf(r,
+                                           "Read error in cache disk data: %s",
+                                           strerror(last_errno));
                         }
                         r->adr[r->size] = '\0';
                       } else {
                         r->statuscode = STATUSCODE_INVALID;
-                        strcpy(r->msg,
-                               "Read error (memory exhausted) from cache");
+                        strcpybuff(r->msg,
+                                   "Read error (memory exhausted) from cache");
                       }
                       fclose(fp);
                     } else {
                       r->statuscode = STATUSCODE_INVALID;
-                      snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
-                              file_convert(catbuff, sizeof(catbuff), previous_save));
+                      PT_Element_failf(
+                          r, "Read error (can't open '%s') from cache",
+                          file_convert(catbuff, sizeof(catbuff),
+                                       previous_save));
                     }
                   } else {
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpy(r->msg, "Cached file name is invalid");
+                    strcpybuff(r->msg, "Cached file name is invalid");
                   }
                 }
               } else {
@@ -1202,12 +1224,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                       free(r->adr);
                       r->adr = NULL;
                       r->statuscode = STATUSCODE_INVALID;
-                      strcpy(r->msg, "Cache Read Error : Read Data");
+                      strcpybuff(r->msg, "Cache Read Error : Read Data");
                     } else
                       *(r->adr + r->size) = '\0';
                   } else {      // erreur
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpy(r->msg, "Cache Memory Error");
+                    strcpybuff(r->msg, "Cache Memory Error");
                   }
                 }
               }
@@ -1215,21 +1237,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
           }                     // si save==null, ne rien charger (juste en tête)
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpy(r->msg, "Cache Read Error : Read Header Data");
+          strcpybuff(r->msg, "Cache Read Error : Read Header Data");
         }
         unzCloseCurrentFile(index->zFile);
       } else {
         r->statuscode = STATUSCODE_INVALID;
-        strcpy(r->msg, "Cache Read Error : Open File");
+        strcpybuff(r->msg, "Cache Read Error : Open File");
       }
 
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpy(r->msg, "Cache Read Error : Bad Offset");
+      strcpybuff(r->msg, "Cache Read Error : Bad Offset");
     }
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpy(r->msg, "File Cache Entry Not Found");
+    strcpybuff(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -1522,7 +1544,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
             cache->version = (int) (firstline[8] - '0');        // cache 1.x
             if (cache->version <= 5) {
               a += cache_brstr(a, firstline, sizeof(firstline));
-              strcpy(cache->lastmodified, firstline);
+              strcpybuff(cache->lastmodified, firstline);
             } else {
               fclose(cache->dat);
               cache->dat = NULL;
@@ -1539,7 +1561,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
         } else {                // Vieille version du cache
           /* */
           cache->version = 0;   // cache 1.0
-          strcpy(cache->lastmodified, firstline);
+          strcpybuff(cache->lastmodified, firstline);
         }
 
         /* Create hash table for the cache (MUCH FASTER!) */
@@ -1555,7 +1577,14 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
               a++;
               /* read "host/file" */
               a += binput(a, line, HTS_URLMAXSIZE);
-              a += binput(a, line + strlen(line), HTS_URLMAXSIZE);
+              {
+                /* binput writes its NUL at s[max], so the second field must be
+                   bounded by what is left of line[], not by the same constant
+                 */
+                const size_t used = strlen(line);
+
+                a += binput(a, line + used, (int) (sizeof(line) - used - 1));
+              }
               /* read position */
               a += binput(a, linepos, 200);
               sscanf(linepos, "%d", &pos);
@@ -1570,9 +1599,11 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
                   const char *scheme =
                       link_has_authority(line) ? "" : "http://";
 
-                  firstSeen = 1;
-                  snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
-                           scheme, line);
+                  /* dropped rather than truncated; try the next entry */
+                  if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
+                    firstSeen = 1;
+                  else
+                    index->startUrl[0] = '\0';
                 }
               }
 
@@ -1650,7 +1681,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
   previous_save[0] = previous_save_[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  strcpy(r->location, "");
+  r->location[0] = '\0';
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(cache->hash, url, &hash_pos);
@@ -1757,7 +1788,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
           int pathLen = (int) strlen(index->path);
 
           if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) {      // old (<3.40) buggy format
-            strcpy(previous_save, previous_save_);
+            strcpybuff(previous_save, previous_save_);
           }
           // relative ? (hack)
           else if (index->safeCache || (previous_save_[0] != '/'        // /home/foo/bar.gif
@@ -1793,8 +1824,9 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                 snprintf(previous_save, sizeof(previous_save), "%s%s",
                          index->path, previous_save_ + index->fixedPath);
               } else {
-                snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
-                        previous_save_, (int) index->fixedPath);
+                PT_Element_failf(r,
+                                 "Bogus fixePath prefix for %s (prefixLen=%d)",
+                                 previous_save_, (int) index->fixedPath);
                 r->statuscode = STATUSCODE_INVALID;
               }
             } else {
@@ -1820,17 +1852,18 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                 if (r->adr != NULL) {
                   if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpy(r->msg, "Read error in cache disk data");
+                    strcpybuff(r->msg, "Read error in cache disk data");
                   }
                   r->adr[r->size] = '\0';
                 } else {
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpy(r->msg, "Read error (memory exhausted) from cache");
+                  strcpybuff(r->msg,
+                             "Read error (memory exhausted) from cache");
                 }
                 fclose(fp);
               } else {
                 r->statuscode = STATUSCODE_INVALID;
-                strcpy(r->msg, "Previous cache file not found (2)");
+                strcpybuff(r->msg, "Previous cache file not found (2)");
               }
             }
           } else {
@@ -1842,30 +1875,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                   free(r->adr);
                   r->adr = NULL;
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpy(r->msg, "Cache Read Error : Read Data");
+                  strcpybuff(r->msg, "Cache Read Error : Read Data");
                 } else
                   r->adr[r->size] = '\0';
               } else {          // erreur
                 r->statuscode = STATUSCODE_INVALID;
-                strcpy(r->msg, "Cache Memory Error");
+                strcpybuff(r->msg, "Cache Memory Error");
               }
             }
           }
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpy(r->msg, "Cache Read Error : Bad Data");
+          strcpybuff(r->msg, "Cache Read Error : Bad Data");
         }
       } else {                  // erreur
         r->statuscode = STATUSCODE_INVALID;
-        strcpy(r->msg, "Cache Read Error : Read Header");
+        strcpybuff(r->msg, "Cache Read Error : Read Header");
       }
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpy(r->msg, "Cache Read Error : Seek Failed");
+      strcpybuff(r->msg, "Cache Read Error : Seek Failed");
     }
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpy(r->msg, "File Cache Entry Not Found");
+    strcpybuff(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -2128,12 +2161,14 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
   return 0;
 }
 
-#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
-  if (line[0] != '\0' && strfield2(line, refline)) { \
-    strcpy(refvalue, value); \
-    line[0] = '\0'; \
-	} \
-} while(0)
+/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
+#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)   \
+  do {                                                                         \
+    if (line[0] != '\0' && strfield2(line, refline)) {                         \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
+      line[0] = '\0';                                                          \
+    }                                                                          \
+  } while (0)
 #define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
   if (line[0] != '\0' && strfield2(line, refline)) { \
     int intval = 0; \
@@ -2169,7 +2204,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
   location_default[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  strcpy(r->location, "");
+  r->location[0] = '\0';
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -2191,7 +2226,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
           }
           if ((pos = getArcField(index->line, 2)) != NULL) {
             r->msg[0] = '\0';
-            strncat(r->msg, pos, sizeof(pos) - 1);
+            strncatbuff(r->msg, pos, sizeof(r->msg) - 1);
           }
           while(linput(index->file, index->line, sizeof(index->line) - 1)
                 && index->line[0] != '\0') {
@@ -2202,11 +2237,16 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
               *value = '\0';
               for(value++; *value == ' ' || *value == '\t'; value++) ;
               HTTP_READFIELD_INT(line, value, "Content-Length", r->size);       // size
-              HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype);       // contenttype
-              HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified);     // last-modified
-              HTTP_READFIELD_STRING(line, value, "Etag", r->etag);      // Etag
-              HTTP_READFIELD_STRING(line, value, "Location", r->location);      // 'location' pour moved
-              HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo);     // Content-disposition
+              HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
+                                    sizeof(r->contenttype));
+              HTTP_READFIELD_STRING(line, value, "Last-Modified",
+                                    r->lastmodified, sizeof(r->lastmodified));
+              HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
+                                    sizeof(r->etag));
+              HTTP_READFIELD_STRING(line, value, "Location", r->location,
+                                    sizeof(location_default));
+              HTTP_READFIELD_STRING(line, value, "Content-Disposition",
+                                    r->cdispo, sizeof(r->cdispo));
               if (line[0] != '\0') {
                 int len = r->headers ? ((int) strlen(r->headers)) : 0;
                 int nlen =
@@ -2246,7 +2286,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
                 fetchSize = dataLength - metaSize;
               } else if (fetchSize > dataLength - metaSize) {
                 r->statuscode = STATUSCODE_INVALID;
-                strcpy(r->msg, "Cache Read Error : Truncated Data");
+                strcpybuff(r->msg, "Cache Read Error : Truncated Data");
               }
               r->size = 0;
               if (r->statuscode != STATUSCODE_INVALID) {
@@ -2259,12 +2299,13 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
                     int last_errno = errno;
 
                     r->statuscode = STATUSCODE_INVALID;
-                    sprintf(r->msg, "Read error in cache disk data: %s",
-                            strerror(last_errno));
+                    PT_Element_failf(r, "Read error in cache disk data: %s",
+                                     strerror(last_errno));
                   }
                 } else {
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpy(r->msg, "Read error (memory exhausted) from cache");
+                  strcpybuff(r->msg,
+                             "Read error (memory exhausted) from cache");
                 }
               }
             }
@@ -2272,21 +2313,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
 
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpy(r->msg, "Cache Read Error : Read Header Error");
+          strcpybuff(r->msg, "Cache Read Error : Read Header Error");
         }
 
       } else {
         r->statuscode = STATUSCODE_INVALID;
-        strcpy(r->msg, "Cache Read Error : Read Header Error");
+        strcpybuff(r->msg, "Cache Read Error : Read Header Error");
       }
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpy(r->msg, "Cache Read Error : Seek Error");
+      strcpybuff(r->msg, "Cache Read Error : Seek Error");
     }
 
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpy(r->msg, "File Cache Entry Not Found");
+    strcpybuff(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -2335,8 +2376,18 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
   FILE *const fp = st->fp;
   struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
+  struct tm unknown_date;
   int size_headers;
 
+  /* a cached entry with no parseable Last-Modified must not take the writer
+     down; the epoch is the conventional "date unknown" */
+  if (tm == NULL) {
+    memset(&unknown_date, 0, sizeof(unknown_date));
+    unknown_date.tm_year = 70;
+    unknown_date.tm_mday = 1;
+    tm = &unknown_date;
+  }
+
   sprintf(st->headers,
           "HTTP/1.0 %d %s"
           "\r\n"
@@ -2372,7 +2423,7 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   if (element->adr != NULL) {
     domd5mem(element->adr, element->size, st->md5, 1);
   } else {
-    strcpy(st->md5, "-");
+    strcpybuff(st->md5, "-");
   }
   fprintf(fp,
           /* nl */

src/webhttrack.vcxproj

@@ -97,6 +97,7 @@
   <ItemGroup>
     <ClCompile Include="htsserver.c" />
     <ClCompile Include="htsweb.c" />
+    <ClCompile Include="htscmdline.c" />
     <ClCompile Include="htsurlport.c" />
   </ItemGroup>
 

tests/01_engine-changes.test

@@ -0,0 +1,30 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_changes_st.XXXXXX") || exit 1
+trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM
+
+# No pipe into grep: SIGPIPE would mask a failing exit status.
+expect_ok() {
+    local label="$1" out
+    shift
+    out=$("$@" 2>&1) || {
+        echo "FAIL: ${label} exited non-zero: ${out}"
+        exit 1
+    }
+    case "$out" in
+    *"${label}: OK"*) ;;
+    *)
+        echo "FAIL: ${out}"
+        exit 1
+        ;;
+    esac
+}
+
+# --changes bucket accounting and JSON escaping (#714).
+expect_ok "changes self-test" httrack -O "${tmpdir}/o1" -#test=changes run
+# The report must hold against a transfer thread the crawl never joins.
+expect_ok "changes-race self-test" httrack -O "${tmpdir}/o2" \
+    -#test=changes-race "${tmpdir}/race"

tests/01_engine-cmdline-split.test

@@ -0,0 +1,8 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+# webhttrack posts its command line as one string: the argv split must grow past
+# 1024 arguments and keep a quote inside a value out of the option parser.
+httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

tests/01_engine-growsize.test

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Buffer capacity for a 64-bit file size ('httrack -#test=growsize'): a -%S list
+# file past 4GB must size its buffer exactly or be refused, never wrap.
+
+set -euo pipefail
+
+out=$(httrack -#test=growsize)
+echo "$out"
+test "$out" == "growsize self-test OK"
+
+tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_growsize.XXXXXX")
+trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
+
+echo '<html><body>hi</body></html>' >"$tmp/index.html"
+printf -- '-*/zzmarker*\n' >"$tmp/rules.txt"
+
+# the rules file lands in the URL/filter string, echoed back by the banner
+run=$(httrack -O "$tmp/out" --quiet -n "-%S" "$tmp/rules.txt" \
+    "file://$tmp/index.html" 2>&1) || true
+printf '%s\n' "$run" | grep -q 'zzmarker' || {
+    echo "FAIL: -%S rules file was not loaded"
+    printf '%s\n' "$run"
+    exit 1
+}

tests/01_engine-strsafe.test

@@ -47,3 +47,18 @@ case "$err" in
     exit 1
     ;;
 esac
+
+# An array source with no NUL must abort on the source bound rather than run
+# off the array: the array half of the source-capacity selection.
+err=$(httrack -#test=strsafe overflow-src "x" 2>&1) || true
+case "$err" in
+*"strsafe: NOT aborted"*)
+    echo "unterminated source array was NOT caught" >&2
+    exit 1
+    ;;
+*"size < sizeof_source"*) ;;
+*)
+    echo "expected htssafe source-bound abort, got: $err" >&2
+    exit 1
+    ;;
+esac

tests/53_local-proxytrack-arc-reason.test

@@ -0,0 +1,32 @@
+#!/bin/bash
+# An ARC entry's HTTP reason phrase must survive a proxytrack --convert
+# round-trip; it used to be clipped to sizeof(char*) - 1 bytes.
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+printf 'HTTP/1.1 404 Not Found Here At All\r\nContent-Type: text/html\r\nLast-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
+printf 'hello' >"$dir/body"
+alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+
+# ARC 1.0: filedesc record and version block, then per entry
+# <nl> <URL-record> <nl> <headers> <body>; the record's last field is that length.
+{
+    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+    printf '2 0 test\n'
+    printf '\n\n'
+    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 404 - - 0 t.arc %d\n' "$alen"
+    cat "$dir/hdr" "$dir/body"
+} >"$dir/in.arc"
+
+proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1
+
+# HTTP/1.0 is the writer's own prefix (the input says 1.1), so a verbatim copy
+# of the input cannot satisfy this match.
+grep -aqF 'HTTP/1.0 404 Not Found Here At All' "$dir/out.arc" || {
+    echo "reason phrase lost in the ARC round-trip:" >&2
+    grep -a '^HTTP/' "$dir/out.arc" >&2 || echo "(no status line at all)" >&2
+    exit 1
+}

tests/68_webhttrack-outdir-charset.test

@@ -60,13 +60,20 @@ test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
 
 # Post a "start" whose -O dir is 'café' in the form's ISO-8859-1 charset.
 "${python}" - "${url}" "${work}" <<'PY'
-import sys, urllib.parse, urllib.request
+import re, sys, urllib.parse, urllib.request
 
 url, work = sys.argv[1], sys.argv[2]
 outdir = work + "/caf\xe9"  # 'café' as the single byte the browser would send
 # Port 1 refuses at once: only the decoded -O dir matters, not the fetch.
 cmd = "httrack --quiet --robots=0 http://127.0.0.1:1/x.html -O " + outdir
-fields = [("path", work), ("projname", "proj"), ("command_do", "start"),
+# The body is refused without the session id the server renders into each form.
+# Note the wizard page is under /server/; a bare /step4.html is the doc page.
+page = urllib.request.urlopen(url + "server/step4.html", timeout=20).read()
+m = re.search(rb'name="sid" value="([0-9a-f]+)"', page)
+if not m:
+    raise SystemExit("no session id in server/step4.html")
+fields = [("sid", m.group(1).decode()),
+          ("path", work), ("projname", "proj"), ("command_do", "start"),
           ("winprofile", "x"), ("command", cmd)]
 body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="", encoding="latin-1"))
                 for k, v in fields)

tests/77_webhttrack-redirect.test

@@ -0,0 +1,187 @@
+#!/bin/bash
+#
+# htsserver's POST redirect: the Location header is built from a client-supplied
+# value, and the listen address is loopback unless --bind widens it.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# run_with_timeout/kill_tree: timeout(1) is absent on macOS
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+log=$(mktemp)
+# start() runs inside a command substitution, so its $! never reaches this
+# shell. Take the pid from the server's own announcement instead: a missed kill
+# leaves an orphan holding the CI job open long after the suite has passed.
+srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
+cleanup() {
+    stop
+    rm -f "${log}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Start htsserver, echo the announced URL. Extra args are passed through.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" "$@" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+stop() {
+    local pid
+    pid=$(srvpid)
+    test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
+}
+
+# The session id gates the request body, so a POST has to carry one. The server
+# renders it into every form, which is where a browser picks it up too.
+scrape_sid() {
+    python3 -c 'import socket, sys
+s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
+s.settimeout(20)
+s.sendall(b"GET /server/index.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n")
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" |
+        sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
+}
+
+# POST redirect=$1 to 127.0.0.1:$2 with session id $3, print the raw headers.
+post_redirect() {
+    python3 -c 'import socket, sys, urllib.parse
+body = ("sid=" + sys.argv[3] + "&redirect="
+        + urllib.parse.quote(sys.argv[1], safe=""))
+req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+       "Content-type: application/x-www-form-urlencoded\r\n"
+       "Content-length: %d\r\n\r\n%s" % (len(body), body))
+s = socket.create_connection(("127.0.0.1", int(sys.argv[2])), 10)
+s.settimeout(20)
+s.sendall(req.encode())
+out = b""
+# stop at the end of the header block: the server need not close the connection,
+# and an unbounded recv() hung the macOS runner for over an hour.
+while b"\r\n\r\n" not in out:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.split(b"\r\n\r\n")[0].decode("latin-1"))' "$1" "$2" "$3"
+}
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# "free"/"inuse"/"unusable": can $1 still be bound on 127.0.0.2? A wildcard
+# listener takes the port on every address, a loopback-only one does not, so
+# this discriminates where the announced URL cannot.
+probe_alias() {
+    python3 -c 'import socket, sys
+s = socket.socket()
+try:
+    s.bind(("127.0.0.2", int(sys.argv[1])))
+except OSError as e:
+    print("unusable" if e.errno in (99, 49, 10049) else "inuse")
+else:
+    print("free")
+finally:
+    s.close()' "$1"
+}
+
+# Non-repeating and not a round number: a truncation, a reorder or a dropped
+# interior byte all stay visible.
+long=$(python3 -c 'print("".join(chr(33 + i % 90) for i in range(4097)))')
+url=$(start)
+port=$(portof "${url}")
+sid=$(scrape_sid "${port}")
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
+resp=$(post_redirect "${long}" "${port}" "${sid}") ||
+    fail "no response to an oversized redirect value"
+stop
+loc=$(printf '%s' "${resp}" | sed -n 's/^Location: //p' | tr -d '\r')
+test "${loc}" = "${long}" ||
+    fail "oversized redirect not echoed whole (got ${#loc} bytes, want ${#long})"
+
+# CR/LF must suppress the header outright. Sanitising it instead would keep the
+# grep for an injected header quiet while still emitting a mangled Location.
+url=$(start)
+port=$(portof "${url}")
+sid=$(scrape_sid "${port}")
+resp=$(post_redirect '/foo
+X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
+stop
+# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
+# pipefail then turns the hit into a silent pass.
+grep -qi 'X-Injected' <<<"${resp}" &&
+    fail "CRLF in the redirect value reached the response"
+test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
+    fail "CRLF redirect still emitted a Location header"
+test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
+    fail "CRLF redirect did not yield exactly one status line"
+
+# Loopback unless asked otherwise. Assert the socket, not the announcement.
+url=$(start)
+port=$(portof "${url}")
+alias_state=$(probe_alias "${port}")
+stop
+if test "${alias_state}" = unusable; then
+    echo "no 127.0.0.2 alias; skipping the listen-address assertions" >&2
+else
+    test "${alias_state}" = free ||
+        fail "default listen address is not loopback-only (127.0.0.2:${port} taken)"
+    case "${url}" in
+    http://127.0.0.1:*) ;;
+    *) fail "default announcement is not loopback: ${url}" ;;
+    esac
+
+    url=$(start --bind 0.0.0.0)
+    port=$(portof "${url}")
+    alias_state=$(probe_alias "${port}")
+    stop
+    test "${alias_state}" = inuse ||
+        fail "--bind 0.0.0.0 did not take the wildcard (127.0.0.2:${port} ${alias_state})"
+fi
+
+# An empty --bind must not quietly fall back to every interface. Bounded: if it
+# were accepted the server would listen instead of exiting.
+run_with_timeout 15 htsserver "${distdir}/" --bind "" >"${log}" 2>&1 || true
+grep -q -- "--bind needs an address" "${log}" ||
+    fail "empty --bind not refused: $(cat "${log}")"
+
+echo "PASS"

tests/78_webhttrack-sid.test

@@ -0,0 +1,150 @@
+#!/bin/bash
+#
+# htsserver's session id must gate the request body: every field in it is
+# written into the global key store, and "command" from there reaches the engine.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+srv=
+log=$(mktemp)
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    rm -f "${log}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL. Runs in a command substitution, so it is a
+# subshell and cannot export the pid: the caller reads it back with srvpid.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+# The server reports its own pid; the aliveness assertions below hang off it.
+srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
+
+alive() { kill -0 "$1" 2>/dev/null; }
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
+# default index). Prints the reply.
+request() {
+    python3 -c 'import socket, sys
+port, body = int(sys.argv[1]), sys.argv[2]
+page = sys.argv[3] if len(sys.argv) > 3 else "index"
+if body:
+    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+           "Content-type: application/x-www-form-urlencoded\r\n"
+           "Content-length: %d\r\n\r\n%s" % (len(body), body))
+else:
+    req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
+s = socket.create_connection(("127.0.0.1", port), 10)
+s.settimeout(30)
+s.sendall(req.encode())
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
+}
+
+# The server hands the sid to any client in every form; scraping it is the
+# legitimate flow, and it is what makes the accept case below meaningful.
+scrape_sid() {
+    request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
+}
+
+url=$(start)
+port=$(portof "${url}")
+srv=$(srvpid)
+test -n "${srv}" || fail "htsserver did not report its pid"
+alive "${srv}" || fail "htsserver is not running"
+
+sid=$(scrape_sid "${port}")
+# Control: without a real sid every assertion below would pass vacuously.
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
+
+# Accept: the legitimate flow still works. "redirect" is the cheapest field
+# with a reply that is visible in the headers. Matched from a here-string, not a
+# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
+resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
+grep -q '^Location: /accepted' <<<"${resp}" ||
+    fail "a body carrying the correct sid was refused"
+
+# Refuse: missing and wrong. A missing one is the regression under test — the
+# expected value used to be pre-seeded into the compared key, so omitting the
+# field compared equal to itself.
+for bad in "redirect=/nosid" "sid=&redirect=/empty" \
+    "sid=00000000000000000000000000000000&redirect=/wrong"; do
+    resp=$(request "${port}" "${bad}")
+    grep -q '^Location:' <<<"${resp}" &&
+        fail "body accepted without a valid sid: ${bad}"
+    # A refusal has to be a well-formed reply, not a headerless fragment: the
+    # release build used to emit only Content-length, which reads as a protocol
+    # error to any client and hides the reason.
+    grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
+        fail "refusal was not a 403: ${bad}"
+done
+
+# Suppressing the reply is not the same as refusing the write: the body is
+# applied to one global key store that later requests render from, and the
+# command dispatcher reads it from there. Probe the store itself — step3
+# interpolates ${projname} into its title — rather than the refused reply.
+store_page() {
+    # a dead probe or a non-page reply must not read as "the store was not written"
+    page=$(request "$1" "" step3) || fail "the key store probe failed"
+    grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
+        fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
+}
+
+request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
+store_page "${port}"
+grep -q 'UNAUTHWRITE' <<<"${page}" &&
+    fail "a body without a valid sid was written to the key store"
+
+# Paired accept case: without it the assertion above passes even if the server
+# simply ignores every body, which would prove nothing.
+request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
+store_page "${port}"
+grep -q 'AUTHWRITE' <<<"${page}" ||
+    fail "a body carrying the correct sid was not written to the key store"
+
+echo "PASS"

tests/79_local-proxytrack-webdav-mime.test

@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# proxytrack's WebDAV PROPFIND fallback must behave identically whether or not
+# the cache entry supplies Content-Type and Last-Modified.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+# shellcheck source=tests/testlib.sh
+. "$top_srcdir/tests/testlib.sh"
+
+python=$(find_python) || {
+    echo "python3 missing, skipping"
+    exit 77
+}
+command -v curl >/dev/null 2>&1 || {
+    echo "curl missing, skipping"
+    exit 77
+}
+# First test to run proxytrack as a live server, and MSYS cannot reap a native
+# listener: the orphan wedged the whole Windows suite for 48 minutes (#595).
+if is_windows; then
+    echo "windows: cannot reap a backgrounded proxytrack, skipping"
+    exit 77
+fi
+
+dir=$(mktemp -d)
+ptpid=
+cleanup() {
+    stop_server "$ptpid"
+    rm -rf "$dir"
+}
+trap cleanup EXIT
+
+# Neither header is present, so file->contenttype and ->lastmodified stay "".
+printf 'HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
+printf 'hello' >"$dir/body"
+alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+{
+    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+    printf '2 0 test\n'
+    printf '\n\n'
+    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+    cat "$dir/hdr" "$dir/body"
+} >"$dir/in.arc"
+
+freeport() {
+    "$python" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'
+}
+proxyport=$(freeport)
+icpport=$(freeport)
+
+proxytrack "127.0.0.1:$proxyport" "127.0.0.1:$icpport" "$dir/in.arc" >"$dir/pt.log" 2>&1 &
+ptpid=$!
+waited=0
+until grep -qE "HTTP Proxy installed on|Unable to (initialize a temporary server|create the server)" "$dir/pt.log"; do
+    kill -0 "$ptpid" 2>/dev/null || {
+        echo "FAIL: proxytrack exited before listening"
+        cat "$dir/pt.log"
+        exit 1
+    }
+    test "$waited" -lt 50 || {
+        echo "FAIL: proxytrack never announced its listen port"
+        exit 1
+    }
+    sleep 0.1
+    waited=$((waited + 1))
+done
+grep -q "HTTP Proxy installed on" "$dir/pt.log" || {
+    echo "FAIL: proxytrack failed to bind"
+    cat "$dir/pt.log"
+    exit 1
+}
+
+# --max-time: an unbounded read here would wedge the runner rather than fail
+curl -s --max-time 30 -X PROPFIND -H "Depth: 1" \
+    "http://127.0.0.1:$proxyport/webdav/example.com/" >"$dir/resp.xml"
+
+grep -q '<getcontenttype>application/octet-stream</getcontenttype>' "$dir/resp.xml" || {
+    echo "FAIL: missing Content-Type did not fall back to application/octet-stream"
+    cat "$dir/resp.xml"
+    exit 1
+}
+grep -q '<getlastmodified>Wed, 01 Jan 2025 00:00:00 GMT</getlastmodified>' "$dir/resp.xml" || {
+    echo "FAIL: missing Last-Modified did not fall back to the index timestamp"
+    cat "$dir/resp.xml"
+    exit 1
+}
+
+echo "OK: WebDAV listing falls back correctly for an entry with no Content-Type/Last-Modified"