tests: assert the retry exhaustion, not the per-platform failure message

The message a cut connection produces depends on whether any bytes arrived, so
matching it would fail on a runner that sees none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/workflows/ci.yml

@@ -46,10 +46,32 @@ jobs:
       - name: Configure
         run: |
           set -euo pipefail
-          # Regenerate from configure.ac/Makefile.am to validate them; the
-          # committed generated files already let a plain checkout build.
+          # Regenerate: configure and the Makefile.in's are not tracked.
           autoreconf -fi
+          # Disabling zlib must fail here rather than at link with a pile of
+          # undefined minizip references (#735). Both spellings, so a rewrite
+          # cannot keep one and lose the other. Probed out-of-tree to leave
+          # nothing behind.
+          nozlib="$RUNNER_TEMP/nozlib"
+          for arg in --without-zlib --with-zlib=no; do
+            rm -rf "$nozlib" && mkdir -p "$nozlib"
+            if (cd "$nozlib" && "$GITHUB_WORKSPACE/configure" "$arg" >out.log 2>&1); then
+              echo "::error::configure $arg succeeded; it must be rejected"
+              exit 1
+            fi
+            # ... and for the stated reason, not an unrelated configure failure.
+            grep -q "zlib cannot be disabled" "$nozlib/out.log" \
+              || { cat "$nozlib/out.log"; exit 1; }
+          done
           ./configure
+          # Same dead end from the compile side. The bare compile is the
+          # control: without it a broken probe would pass vacuously.
+          hdr='#include "htsglobal.h"'
+          echo "$hdr" | $CC -I. -Isrc -fsyntax-only -xc -
+          if echo "$hdr" | $CC -DHTS_USEZLIB=0 -I. -Isrc -fsyntax-only -xc - 2>/dev/null; then
+            echo "::error::-DHTS_USEZLIB=0 compiled; htsglobal.h must reject it"
+            exit 1
+          fi
           # a missing decoder would silently drop the coding from Accept-Encoding
           grep -q "define HTS_USEBROTLI 1" config.h
           grep -q "define HTS_USEZSTD 1" config.h

.github/workflows/windows-build.yml

@@ -226,8 +226,10 @@ jobs:
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
           # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
-          # webdav-mime needs a reapable background listener, which MSYS cannot give it.
-          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
+          # webdav-mime needs a reapable background listener, which MSYS cannot give it;
+          # badmtime needs a filesystem that stores an mtime past gmtime's range;
+          # single-file ends on a GUI half needing htsserver, which this job does not build.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test 94_local-single-file.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -26,6 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
+- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
+  pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
+  and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
+  as "marker absent". Capture the reply, assert the status line it must carry
+  (an empty, truncated or redirected one is marker-free too), then match with a
+  here-string.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -48,6 +54,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
+- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
+  (`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
+  our own data, wrong for anything read back from a cache, a header or the
+  wire, where it trades a memory smash for a crash on malformed input. Clip
+  with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
+- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
+  a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
+  into the same buffer never appears on it. Before scoping a hardening pass off
+  compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
+  `\bstrcpy\s*\(`, `\bstrcat\s*\(`).
 
 ## C conventions
 - **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -84,6 +100,17 @@ Before pushing, and when reviewing others, don't skim for bugs:
   layout/ABI, cache/wire format, or a security path? A static or unit check
   isn't enough; exercise the wrong behavior at runtime. Claude Code:
   `/review-recipe`.
+- **Poison a canary, never compare it against zero.** Checking that a
+  neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
+  terminator writes — the exact bug the canary is there for. Fill it with a
+  non-zero byte, and prove it by killing both the stray-`'X'` and the
+  stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
+  lands inside the same struct.
+- **Overshoot every destination, not one.** A bounds test that oversizes a
+  single field cannot tell a per-field bound from a one-size-fits-all one, nor
+  from a fix that bounds that field and leaves its neighbours raw. Exercise
+  each destination the path touches, spanning at least two capacities, and
+  check what the code actually emits before writing the expected values.
 
 ## Commits
 - **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

configure.ac

@@ -175,7 +175,7 @@ AX_CHECK_ALIGNED_ACCESS_REQUIRED
 # check for various headers
 AC_CHECK_HEADERS([execinfo.h sys/ioctl.h])
 
-### zlib
+### zlib (mandatory)
 CHECK_ZLIB()
 
 ### brotli and zstd content codings (optional)

fuzz/Makefile.am

@@ -2,7 +2,7 @@
 if FUZZERS
 noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \
 	fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \
-	fuzz-htsparse
+	fuzz-htsparse fuzz-singlefile fuzz-sitemap
 endif
 
 AM_CPPFLAGS = \
@@ -27,6 +27,8 @@ fuzz_url_SOURCES = fuzz-url.c fuzz.h
 fuzz_header_SOURCES = fuzz-header.c fuzz.h
 fuzz_cachendx_SOURCES = fuzz-cachendx.c fuzz.h
 fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h
+fuzz_singlefile_SOURCES = fuzz-singlefile.c fuzz.h
+fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h
 
 # List corpus files explicitly: automake does not expand EXTRA_DIST globs.
 EXTRA_DIST = README.md run-fuzzers.sh \
@@ -47,4 +49,10 @@ EXTRA_DIST = README.md run-fuzzers.sh \
 	corpus/cachendx/regress-overadvance.bin \
 	corpus/cachendx/regress-truncated-entry.bin \
 	corpus/htsparse/basic.html corpus/htsparse/script-inscript.html \
-	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html
+	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html \
+	corpus/singlefile/img-src.html corpus/singlefile/link-rel.html \
+	corpus/singlefile/style-block.html corpus/singlefile/style-attr.html \
+	corpus/singlefile/srcset.html corpus/singlefile/rawtext.html \
+	corpus/singlefile/malformed.html corpus/singlefile/many-attrs.html \
+	corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
+	corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz

fuzz/corpus/singlefile/img-src.html

@@ -0,0 +1,3 @@
+<img src="a.png">
+<img src=big.png alt=over-cap>
+<img src="../escape.png"><img src="/abs.png"><img src="data:,x">

fuzz/corpus/singlefile/link-rel.html

@@ -0,0 +1,4 @@
+<link rel="stylesheet" href="s.css">
+<link rel=icon href=a.png>
+<link rel="next" href="p2.html">
+<link rel="preload" href="j.js">

fuzz/corpus/singlefile/malformed.html

@@ -0,0 +1,5 @@
+<img src="unterminated.png
+<div style="background:url(a.png">
+<style>@import url(
+<!-- unterminated comment
+<a href=

fuzz/corpus/singlefile/many-attrs.html

@@ -0,0 +1 @@
+<img src="a.png" a0="v" a1="v" a2="v" a3="v" a4="v" a5="v" a6="v" a7="v" a8="v" a9="v" a10="v" a11="v" a12="v" a13="v" a14="v" a15="v" a16="v" a17="v" a18="v" a19="v" a20="v" a21="v" a22="v" a23="v" a24="v" a25="v" a26="v" a27="v" a28="v" a29="v" a30="v" a31="v" a32="v" a33="v" a34="v" a35="v" a36="v" a37="v" a38="v" a39="v" a40="v" a41="v" a42="v" a43="v" a44="v" a45="v" a46="v" a47="v" a48="v" a49="v" a50="v" a51="v" a52="v" a53="v" a54="v" a55="v" a56="v" a57="v" a58="v" a59="v" a60="v" a61="v" a62="v" a63="v" a64="v" a65="v" a66="v" a67="v" a68="v" a69="v">

fuzz/corpus/singlefile/rawtext.html

@@ -0,0 +1,4 @@
+<script>var s="</scripting>"; if(a</b) x=1;</script>
+<script src="j.js"></script>
+<textarea></textareas></textarea>
+<title></titles></title>

fuzz/corpus/singlefile/srcset.html

@@ -0,0 +1,2 @@
+<img srcset="a.png 1x, big.png 2x, a.png 100w">
+<source srcset="a.png,, a.png 2x," src="a.png">

fuzz/corpus/singlefile/style-attr.html

@@ -0,0 +1,2 @@
+<div style="background:url(a.png);list-style:url('a.png')"></div>
+<p style='background:url("a.png")'>x</p>

fuzz/corpus/singlefile/style-block.html

@@ -0,0 +1,5 @@
+<style>@import "s.css";
+@import url(sub/b.css);
+div{background:url(a.png)}
+/* url(a.png) */ p:after{content:"url(a.png)"}
+</style>

fuzz/corpus/sitemap/sitemapindex.xml

@@ -0,0 +1 @@
+<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc></sitemap></sitemapindex>

fuzz/corpus/sitemap/truncated.xml

@@ -0,0 +1 @@
+<urlset><loc>http://h.test/x

fuzz/corpus/sitemap/urlset.xml

@@ -0,0 +1 @@
+
<?xml version="1.0"?><urlset><url><loc>http://h.test/a.html</loc></url><url><loc>https://h.test/b?x=1&amp;y=2</loc></url></urlset>

fuzz/fuzz-singlefile.c

@@ -0,0 +1,131 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* Fuzz the --single-file rewriter (htssinglefile.c): hostile HTML walked
+   through the tag, CSS url()/@import and srcset parsers, then re-serialized.
+   The resolver is aimed at a private temp tree, so the inlining half (MIME
+   guess, base64, nested stylesheet) is reached and nothing else on disk is. */
+#include "fuzz.h"
+
+#include "httrack-library.h"
+#include "htssinglefile.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+
+/* Between a.png and big.png, so one input reaches both the inline path and the
+   over-cap fallback. */
+#define FUZZ_SF_CAP 64
+
+static char sf_root[512];
+static char sf_page[600];
+
+/* The asset tree, in removal order: the subdirectory comes after its file. */
+static const char *const sf_files[] = {"a.png",     "big.png", "j.js", "s.css",
+                                       "sub/b.css", "sub",     NULL};
+
+static void sf_cleanup(void) {
+  char path[700];
+  int i;
+
+  for (i = 0; sf_files[i] != NULL; i++) {
+    snprintf(path, sizeof(path), "%s/%s", sf_root, sf_files[i]);
+    (void) remove(path);
+  }
+  (void) remove(sf_root);
+}
+
+/* A missing asset would silently reduce the target to its parser half. */
+static void sf_write(const char *name, const char *data, size_t len) {
+  char path[700];
+  FILE *fp;
+
+  snprintf(path, sizeof(path), "%s/%s", sf_root, name);
+  fp = fopen(path, "wb");
+  if (fp == NULL || fwrite(data, 1, len, fp) != len)
+    abort();
+  fclose(fp);
+}
+
+static void sf_text(const char *name, const char *data) {
+  sf_write(name, data, strlen(data));
+}
+
+static void sf_init(void) {
+  static const char png[] = "\x89PNG\r\n\x1a\n";
+  static const char big[4096] = "\x89PNG";
+  const char *tmp = getenv("TMPDIR");
+  char path[700];
+
+  hts_init();
+  snprintf(sf_root, sizeof(sf_root), "%s/httrack-fuzz-sf-XXXXXX",
+           tmp != NULL && tmp[0] != '\0' ? tmp : "/tmp");
+  if (mkdtemp(sf_root) == NULL)
+    abort();
+  atexit(sf_cleanup);
+  snprintf(sf_page, sizeof(sf_page), "%s/page.html", sf_root);
+  snprintf(path, sizeof(path), "%s/sub", sf_root);
+  if (mkdir(path, 0700) != 0)
+    abort();
+  sf_write("a.png", png, sizeof(png) - 1);
+  sf_write("big.png", big, sizeof(big));
+  sf_text("j.js", "var x=1;\n");
+  /* @import plus a url(), so an inlined stylesheet recurses and its own
+     relative reference is rebased. */
+  sf_text("s.css", "@import url(sub/b.css);\ndiv{background:url(a.png)}\n");
+  sf_text("sub/b.css", "p{background:url(../a.png)}\n");
+}
+
+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
+  static int inited = 0;
+
+  String out = STRING_EMPTY;
+  httrackp *opt;
+  /* Exact-length, unterminated: the rewriter is span-based, so ASan bounds a
+     read past html_len instead of it landing on a terminator. */
+  char *html = malloct(size != 0 ? size : 1);
+
+  if (!inited) {
+    sf_init();
+    inited = 1;
+  }
+  memcpy(html, data, size);
+
+  opt = hts_create_opt();
+  opt->log = opt->errlog = NULL;
+  opt->single_file_max_size = FUZZ_SF_CAP;
+
+  StringClear(out);
+  (void) singlefile_rewrite_html(opt, sf_root, sf_page, html, size,
+                                 SINGLEFILE_MAX_PAGE_SIZE, &out);
+
+  StringFree(out);
+  freet(html);
+  hts_free_opt(opt);
+  return 0;
+}

fuzz/fuzz-sitemap.c

@@ -0,0 +1,60 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* Fuzz the sitemap <loc> scanner (htssitemap.c): raw XML, gzip-framed bodies
+   and truncated streams all arrive here straight off the network. */
+#include "fuzz.h"
+#include "htssitemap.h"
+
+static hts_boolean sm_count(void *arg, const char *url) {
+  int *const n = (int *) arg;
+
+  (void) url;
+  (*n)++;
+  return HTS_TRUE;
+}
+
+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
+  static const int caps[] = {0, 1, 16, HTS_SITEMAP_MAX_URLS_DOC};
+  hts_boolean is_index;
+  char *body;
+  int n = 0, cap;
+
+  if (size == 0)
+    return 0;
+  cap = caps[data[0] % (sizeof(caps) / sizeof(caps[0]))];
+  data++, size--;
+  /* A heap copy of exactly `size` bytes: the scanner must never rely on a
+     terminator, and ASan turns any overread into a report. */
+  body = malloct(size != 0 ? size : 1);
+  memcpy(body, data, size);
+
+  (void) hts_sitemap_scan(body, size, cap, &is_index, sm_count, &n);
+
+  freet(body);
+  return 0;
+}

html/changes.html

@@ -0,0 +1,263 @@
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+
+<head>
+	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+	<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
+	<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local  site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
+	<title>HTTrack Website Copier - Change report format specification</title>
+
+	<style type="text/css">
+	<!--
+
+body {
+	margin: 0;  padding: 0;  margin-bottom: 15px;  margin-top: 8px;
+	background: #77b;
+}
+body, td {
+	font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
+	}
+
+#subTitle {
+	background: #000;  color: #fff;  padding: 4px;  font-weight: bold; 
+	}
+
+#siteNavigation a, #siteNavigation .current {
+	font-weight: bold;  color: #448;
+	}
+#siteNavigation a:link    { text-decoration: none; }
+#siteNavigation a:visited { text-decoration: none; }
+
+#siteNavigation .current { background-color: #ccd; }
+
+#siteNavigation a:hover   { text-decoration: none;  background-color: #fff;  color: #000; }
+#siteNavigation a:active  { text-decoration: none;  background-color: #ccc; }
+
+
+a:link    { text-decoration: underline;  color: #00f; }
+a:visited { text-decoration: underline;  color: #000; }
+a:hover   { text-decoration: underline;  color: #c00; }
+a:active  { text-decoration: underline; }
+
+#pageContent {
+	clear: both;
+	border-bottom: 6px solid #000;
+	padding: 10px;  padding-top: 20px;
+	line-height: 1.65em;
+	background-image: url(images/bg_rings.gif);
+	background-repeat: no-repeat;
+	background-position: top right;
+	}
+
+#pageContent, #siteNavigation {
+	background-color: #ccd;
+	}
+
+
+.imgLeft  { float: left;   margin-right: 10px;  margin-bottom: 10px; }
+.imgRight { float: right;  margin-left: 10px;   margin-bottom: 10px; }
+
+hr { height: 1px;  color: #000;  background-color: #000;  margin-bottom: 15px; }
+
+h1 { margin: 0;  font-weight: bold;  font-size: 2em; }
+h2 { margin: 0;  font-weight: bold;  font-size: 1.6em; }
+h3 { margin: 0;  font-weight: bold;  font-size: 1.3em; }
+h4 { margin: 0;  font-weight: bold;  font-size: 1.18em; }
+
+.blak { background-color: #000; }
+.hide { display: none; }
+.tableWidth { min-width: 400px; }
+
+.tblRegular       { border-collapse: collapse; }
+.tblRegular td    { padding: 6px;  background-image: url(fade.gif);  border: 2px solid #99c; }
+.tblHeaderColor, .tblHeaderColor td { background: #99c; }
+.tblNoBorder td   { border: 0; }
+
+
+// -->
+</style>
+
+</head>
+
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+	<tr>
+	<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
+	<tr>
+	<td id="subTitle">Open Source offline browser</td>
+	</tr>
+</table>
+<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
+<tr class="blak">
+<td>
+	<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
+	<tr>
+	<td colspan="6"> 
+		<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
+		<tr> 
+		<td id="pageContent"> 
+<!-- ==================== End prologue ==================== -->
+
+<h2 align="center"><em>Change report format specification</em></h2>
+
+<br />
+
+Run with <tt>--changes</tt> (<tt>-%d</tt>), HTTrack writes <tt>hts-changes.json</tt>
+in the project directory, next to <tt>hts-log.txt</tt>, describing what the crawl
+left new, changed, unchanged and gone compared to the previous mirror. The file is
+rewritten from scratch at the end of every run, and the log carries a one-line
+summary of the same counts.
+
+<br /><br />
+
+<h3>What "changed" means</h3>
+
+A resource is changed when its bytes differ, not when the server merely re-sent
+it. HTTrack compares the payload it just received against the copy the previous
+run left behind: for pages it parses, the previous payload comes from the cache
+(the file on disk carries the mirror footer and its crawl date, so its bytes
+differ on every run); for everything else, the mirrored file is the payload
+verbatim and is compared directly.
+
+<br /><br />
+
+Where no digest can be taken on either side, because the cache is disabled or
+the previous copy is gone, the report falls back to the transfer signal, and a
+server that answers 200 rather than 304 reads as changed. Keeping the cache on
+(the default) is what makes the report precise.
+
+<br /><br />
+
+<h3>With the cache off</h3>
+
+<tt>--cache=0</tt> costs the report more than the digest of a parsed page. The
+mirror's file index (<tt>hts-cache/new.lst</tt>) is what records which files a
+run produced, so without it there is no previous mirror to subtract from: nothing
+is reported <tt>gone</tt>, and whether the run is a first crawl cannot be decided
+at all, which <tt>first_crawl</tt> states as <tt>null</tt> rather than guess. What
+is on disk is still compared byte for byte, so the other three lists stay
+meaningful, except for the pages HTTrack parses: those have no cached payload to
+compare against and fall back to the transfer signal.
+
+<br /><br />
+
+<h3>Fields</h3>
+
+<ul>
+<li><tt>schema</tt>: format version, currently <tt>1</tt>. It is bumped only
+on an incompatible change; new fields may appear without one.</li>
+<li><tt>generator</tt>: the HTTrack build that wrote the file.</li>
+<li><tt>date</tt>: when the report was written, UTC, <tt>YYYY-MM-DDThh:mm:ssZ</tt>.</li>
+<li><tt>first_crawl</tt>: true when no index of a previous mirror
+(<tt>hts-cache/old.lst</tt>) was found, so there was nothing to compare against and
+everything is listed as new. Null when the run kept no index at all and the
+question cannot be answered (see above).</li>
+<li><tt>partial</tt>: true when the report ran out of memory and lists only
+part of the mirror.</li>
+<li><tt>purged</tt>: true when <tt>--purge-old</tt> was in effect, so the
+files under <tt>gone</tt> were also deleted from disk.</li>
+<li><tt>counts</tt>: the size of each of the four lists.</li>
+<li><tt>new</tt>, <tt>changed</tt>, <tt>unchanged</tt>, <tt>gone</tt>: the
+lists themselves. Every mirrored file appears in exactly one of them.</li>
+</ul>
+
+Each entry is an object:
+
+<ul>
+<li><tt>url</tt>: the absolute URL the file came from. Empty under
+<tt>gone</tt>: deletions are computed from the mirror's file index, which records
+paths, not URLs.</li>
+<li><tt>file</tt>: the path relative to the mirror root, with forward
+slashes. This is the entry's identity: a URL and a redirect that resolve to the
+same local file are one entry, not two.</li>
+<li><tt>size</tt>: the mirrored file's size in bytes, absent when the file
+is not on disk.</li>
+<li><tt>previous_size</tt>: under <tt>changed</tt> only, the size of the
+copy the previous run left.</li>
+</ul>
+
+<br />
+
+<h3>Encoding</h3>
+
+The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are
+not guaranteed to be valid UTF-8; any byte sequence that is not becomes
+U+FFFD (<tt>\ufffd</tt>), so the file always parses. Compare on <tt>file</tt>
+rather than on <tt>url</tt> when a mirror is known to carry legacy-charset URLs.
+
+<br /><br />
+
+<h3>Example</h3>
+
+<pre>
+{
+  "schema": 1,
+  "generator": "HTTrack Website Copier/3.49-14",
+  "date": "2026-07-26T15:29:03Z",
+  "first_crawl": false,
+  "partial": false,
+  "purged": true,
+  "counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
+  "new": [
+    { "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
+  ],
+  "changed": [
+    { "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
+  ],
+  "unchanged": [
+    { "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
+  ],
+  "gone": [
+    { "url": "", "file": "example.com/c.html" }
+  ]
+}
+</pre>
+
+<br /><br />
+
+<h3>Notes</h3>
+
+<ul>
+<li>A file listed under <tt>gone</tt> is only deleted when <tt>--purge-old</tt> is
+on. Left in place it drops out of the mirror's index, so it is reported once and
+not again.</li>
+<li>A resource whose local file name changed since the previous mirror (a new
+MIME type, say) is reported as <tt>new</tt> under its new name; the old name is
+reported as <tt>gone</tt> only if the file is still on disk. The two entries are
+not paired.</li>
+<li>A resource this run tried and failed to transfer also drops out of the
+mirror's index, but its previous copy is untouched, so it is reported
+<tt>unchanged</tt>. Under <tt>--purge-old</tt> that copy is deleted anyway, and
+the report says <tt>gone</tt> to match.</li>
+<li>A run that transfers no data at all is rolled back: HTTrack restores the
+previous cache generation and leaves the previous report in place, so a lost
+connection does not overwrite a good report with an empty one.</li>
+<li>Content diffs, and keeping the previous copy of a changed page, are out of
+scope: both change what a mirror directory contains.</li>
+</ul>
+
+<br /><br />
+
+<!-- ==================== Start epilogue ==================== -->
+		</td>
+		</tr>
+		</table>
+	</td>
+	</tr>
+	</table>
+</td>
+</tr>
+</table>
+
+<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
+	<tr>
+	<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
+	</tr>
+</table>
+
+</body>
+
+</html>
+
+

html/cmdguide.html

@@ -163,8 +163,26 @@ the index" problems disappear.</p>
   <tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
   <tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
   <tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
+  <tr><td><tt>--sitemap (-%m), --sitemap-url URL (-%mu)</tt></td><td>Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.</td></tr>
 </table>
 
+<p>Link-following only finds what something links to. Anything a site publishes
+solely in its sitemap is invisible to HTTrack unless you ask for it.
+<tt>--sitemap</tt> reads the start host's <tt>robots.txt</tt> for
+<tt>Sitemap:</tt> lines and falls back to <tt>/sitemap.xml</tt>;
+<tt>--sitemap-url</tt> names one directly. Nested <tt>sitemapindex</tt> files
+and gzipped <tt>.xml.gz</tt> sitemaps are followed. The URLs found become start
+URLs with the full depth budget, but they still go through your filters and
+scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is
+off by default because a sitemap can list thousands of pages nothing links
+to.</p>
+
+<p>One surprise worth knowing: a sitemap you name with <tt>--sitemap-url</tt>,
+and one the site itself declares in <tt>robots.txt</tt>, are fetched even when
+<tt>robots.txt</tt> disallows that path, because naming or declaring a sitemap
+is an invitation to read it. Only the guessed <tt>/sitemap.xml</tt> obeys a
+<tt>Disallow</tt>. The URLs listed inside are gated normally either way.</p>
+
 <p>The single most common surprise is "only the home page came down." That is
 usually not a scope option at all: it is an off-host redirect. A start URL of
 <tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
@@ -300,6 +318,7 @@ ones it kept so the local copy browses offline. These options tune both halves.<
   <tr><td><tt>--preserve (-%p), --disable-passwords (-%x)</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
   <tr><td><tt>--extended-parsing (-%P), --parse-java (-j)</tt></td><td>Aggressive link discovery, and how much script content is parsed for links.</td></tr>
   <tr><td><tt>--mime-html (-%M)</tt></td><td>Save the whole mirror as a single MIME-encapsulated <tt>.mht</tt> archive (<tt>index.mht</tt>).</td></tr>
+  <tr><td><tt>--single-file (-%Z), --single-file-max-size N</tt></td><td>Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded as <tt>data:</tt> URIs. Assets over the cap (10&nbsp;MB by default) keep their link, as do audio, video, and the links from one page to another. A sibling of <tt>-%M</tt>, not a replacement: see the recipe below for which to pick.</td></tr>
   <tr><td><tt>--index (-I), --build-top-index (-%i), --search-index (-%I)</tt></td><td>Build a per-mirror index, a top index across projects, and a searchable keyword index.</td></tr>
 </table>
 
@@ -482,6 +501,34 @@ add <tt>--warc-cdx</tt> for a sorted CDXJ index, or <tt>--wacz</tt> to bundle th
 archive, index and pages into one WACZ for replay tools such as
 replayweb.page.</small></p>
 
+<h4>See what a re-crawl changed</h4>
+<p><tt>httrack https://example.com/ --update --changes --path mydir</tt><br>
+<small>Writes <tt>hts-changes.json</tt> in the project folder, listing every
+mirrored file as new, changed, unchanged or gone, plus a one-line summary in the
+log. &quot;Changed&quot; means the bytes really differ: a server that answers 200
+with the same content it served last time lands in <tt>unchanged</tt>. Deletions
+are reported whether or not <tt>--purge-old</tt> is deleting them. The format is
+documented in <a href="changes.html">the change report specification</a>.</small></p>
+
+<h4>Pages you can hand to someone as one file</h4>
+<p><tt>httrack https://example.com/ --single-file --path mydir</tt><br>
+<small>Rewrites each saved page after the crawl so its stylesheets, scripts,
+images and fonts are embedded as <tt>data:</tt> URIs. The mirror stays a normal
+browsable tree, with links between pages relative and the assets still on disk,
+but any single <tt>.html</tt> file can now be mailed or archived on its own.
+Raise or lower the 10&nbsp;MB per-asset limit with
+<tt>--single-file-max-size N</tt>; anything over it, plus audio and video, keeps
+its link. One caveat if you raise it: an inlined stylesheet becomes a
+<tt>data:</tt> URL, whose path is opaque, so an asset it referenced that stayed
+over the cap no longer resolves from inside it. Raising the cap past that asset
+embeds it too and the question goes away.<br>
+Reach for this when the file has to open for someone you cannot make assumptions
+about: it is plain HTML and needs no add-on. Reach for <tt>--mime-html</tt>
+instead when a Chromium-family browser is a given and the mirror is large: MIME
+carries text parts without the base64 tax, keeps each resource's original URL,
+and stores a shared asset once rather than re-embedding it in every page that
+uses it.</small></p>
+
 <h4>HTTrack as a fetch tool</h4>
 <p><tt>httrack --get https://host/file.bin --path tmp</tt><br>
 <small><tt>--get</tt> fetches one file with cache, index, depth, cookies and robots
@@ -490,7 +537,8 @@ all disabled.</small></p>
 <p><br>For the complete, always-current option list, see
 <a href="httrack.man.html">the manual page</a>. For the filter language, see
 <a href="filters.html">filters</a>; for the cache and updates, see
-<a href="cache.html">cache</a>.</p>
+<a href="cache.html">cache</a>; for the change report, see
+<a href="changes.html">changes</a>.</p>
 
 <!-- ==================== Start epilogue ==================== -->
 		</td>

html/dev.html

@@ -129,6 +129,9 @@ The library can be used to write graphical GUIs for httrack, or to run mirrors f
 <li><a href="cache.html">Cache format</a></li><br>
 HTTrack stores original HTML data and references to downloaded files in a cache, located in the hts-cache directory.
 This page describes the HTTrack cache format.
+<li><a href="changes.html">Change report format</a></li><br>
+With --changes, HTTrack writes hts-changes.json describing what the crawl left new, changed, unchanged and gone
+compared to the previous mirror. This page describes that file.
 </ul>
 
 

html/httrack.man.html

@@ -87,12 +87,13 @@ offline browser : copy websites to a local directory</p>
 --host-control[=N]</b> ] [ <b>-%P,
 --extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
 --test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
-] [ <b>-NN, --structure[=N]</b> ] [ <b>-%N,
---delayed-type-check</b> ] [ <b>-%D,
+] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
+[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
 --cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
-] [ <b>-LN, --long-names[=N]</b> ] [ <b>-KN,
---keep-links[=N]</b> ] [ <b>-x, --replace-external</b> ] [
-<b>-%x, --disable-passwords</b> ] [ <b>-%q,
+] [ <b>-%Z, --single-file</b> ] [ <b>-LN,
+--long-names[=N]</b> ] [ <b>-KN, --keep-links[=N]</b> ] [
+<b>-x, --replace-external</b> ] [ <b>-%x,
+--disable-passwords</b> ] [ <b>-%q,
 --include-query-string</b> ] [ <b>-%g, --strip-query</b> ] [
 <b>-o, --generate-errors</b> ] [ <b>-X, --purge-old[=N]</b>
 ] [ <b>-%p, --preserve</b> ] [ <b>-%T, --utf8-conversion</b>
@@ -108,16 +109,17 @@ offline browser : copy websites to a local directory</p>
 --footer</b> ] [ <b>-%l, --language</b> ] [ <b>-%a,
 --accept</b> ] [ <b>-%X, --headers</b> ] [ <b>-C,
 --cache[=N]</b> ] [ <b>-k, --store-all-in-cache</b> ] [
-<b>-%r, --warc</b> ] [ <b>-%n, --do-not-recatch</b> ] [
-<b>-%v, --display</b> ] [ <b>-Q, --do-not-log</b> ] [ <b>-q,
---quiet</b> ] [ <b>-z, --extra-log</b> ] [ <b>-Z,
---debug-log</b> ] [ <b>-v, --verbose</b> ] [ <b>-f,
---file-log</b> ] [ <b>-f2, --single-log</b> ] [ <b>-I,
---index</b> ] [ <b>-%i, --build-top-index</b> ] [ <b>-%I,
---search-index</b> ] [ <b>-pN, --priority[=N]</b> ] [ <b>-S,
---stay-on-same-dir</b> ] [ <b>-D, --can-go-down</b> ] [
-<b>-U, --can-go-up</b> ] [ <b>-B, --can-go-up-and-down</b> ]
-[ <b>-a, --stay-on-same-address</b> ] [ <b>-d,
+<b>-%r, --warc</b> ] [ <b>-%d, --changes</b> ] [ <b>-%n,
+--do-not-recatch</b> ] [ <b>-%v, --display</b> ] [ <b>-Q,
+--do-not-log</b> ] [ <b>-q, --quiet</b> ] [ <b>-z,
+--extra-log</b> ] [ <b>-Z, --debug-log</b> ] [ <b>-v,
+--verbose</b> ] [ <b>-f, --file-log</b> ] [ <b>-f2,
+--single-log</b> ] [ <b>-I, --index</b> ] [ <b>-%i,
+--build-top-index</b> ] [ <b>-%I, --search-index</b> ] [
+<b>-pN, --priority[=N]</b> ] [ <b>-S, --stay-on-same-dir</b>
+] [ <b>-D, --can-go-down</b> ] [ <b>-U, --can-go-up</b> ] [
+<b>-B, --can-go-up-and-down</b> ] [ <b>-a,
+--stay-on-same-address</b> ] [ <b>-d,
 --stay-on-same-domain</b> ] [ <b>-l, --stay-on-same-tld</b>
 ] [ <b>-e, --go-everywhere</b> ] [ <b>-%H,
 --debug-headers</b> ] [ <b>-%!,
@@ -575,6 +577,22 @@ URL per line) (--list &lt;param&gt;)</p></td></tr>
 
 <p>&lt;file&gt; add all scan rules located in this text
 file (one scan rule per line) (--urllist &lt;param&gt;)</p></td></tr>
+<tr valign="top" align="left">
+<td width="9%"></td>
+<td width="4%">
+
+
+<p>-%m</p></td>
+<td width="5%"></td>
+<td width="82%">
+
+
+<p>seed the crawl from the site&rsquo;s sitemap (robots.txt
+Sitemap:, then /sitemap.xml); --sitemap-url URL names one
+explicitly. A sitemap you name, or one the site declares, is
+fetched even under robots.txt Disallow; only the guessed
+/sitemap.xml obeys it. The URLs found still pass every
+filter and scope rule (--sitemap)</p></td></tr>
 </table>
 
 <h3>Build options:
@@ -648,6 +666,24 @@ don&rsquo;t wait) (--cached-delayed-type-check)</p></td></tr>
 <td width="4%">
 
 
+<p>-%Z</p></td>
+<td width="5%"></td>
+<td width="82%">
+
+
+<p>after the mirror, rewrite each saved page with its
+stylesheets, scripts, images and fonts inlined as data:
+URIs, so any page opens by double-click anywhere (links
+between pages stay relative; audio and video stay links);
+--single-file-max-size N caps each asset (default 10485760
+bytes). %M is the better container where a Chromium-family
+browser is a given: one archive, no base64 tax on text, a
+shared asset stored once (--single-file)</p></td></tr>
+<tr valign="top" align="left">
+<td width="9%"></td>
+<td width="4%">
+
+
 <p>-%t</p></td>
 <td width="5%"></td>
 <td width="82%">
@@ -1143,6 +1179,19 @@ past N bytes, --warc-cdx also writes a sorted CDXJ index,
 <td width="4%">
 
 
+<p>-%d</p></td>
+<td width="5%"></td>
+<td width="82%">
+
+
+<p>write hts-changes.json listing what this crawl left new,
+changed, unchanged and gone compared to the previous mirror
+(--changes)</p> </td></tr>
+<tr valign="top" align="left">
+<td width="9%"></td>
+<td width="4%">
+
+
 <p>-%n</p></td>
 <td width="5%"></td>
 <td width="82%">

html/server/option10.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ftpprox" value="">
+
 ${LANG_PROXYTYPE}
 <select name="proxytype"
 			title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

html/server/option2.html

@@ -98,6 +98,14 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="errpage" value="">
+<input type="hidden" name="external" value="">
+<input type="hidden" name="hidepwd" value="">
+<input type="hidden" name="hidequery" value="">
+<input type="hidden" name="nopurge" value="">
+<input type="hidden" name="singlefile" value="">
+
 ${LANG_I33}
 <br>
 <select name="build"
@@ -136,6 +144,13 @@ ${listid:build:LISTDEF_3}
 <tr><td><input type="checkbox" name="nopurge"   ${checked:nopurge}  
 			title='${html:LANG_I1a}' onMouseOver="info('${html:LANG_I1a}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I57}</td></tr>
+<tr><td><input type="checkbox" name="singlefile" ${checked:singlefile}
+			title='${html:LANG_SINGLEFILETIP}' onMouseOver="info('${html:LANG_SINGLEFILETIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+> ${LANG_SINGLEFILE}</td></tr>
+<tr><td>${LANG_SINGLEFILEMAX}
+<input name="singlefilemax" value="${singlefilemax}" size="12"
+			title='${html:LANG_SINGLEFILEMAXTIP}' onMouseOver="info('${html:LANG_SINGLEFILEMAXTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+></td></tr>
 </table>
 
 <tr><td>

html/server/option3.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="windebug" value="">
+
 ${LANG_I40c}
 <br>
 

html/server/option4.html

@@ -98,6 +98,11 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ka" value="">
+<input type="hidden" name="remt" value="">
+<input type="hidden" name="rems" value="">
+
 <table border="0" width="100%" cellspacing="0">
 
 <tr><td>

html/server/option8.html

@@ -98,6 +98,18 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="cookies" value="">
+<input type="hidden" name="parsejava" value="">
+<input type="hidden" name="updhack" value="">
+<input type="hidden" name="urlhack" value="">
+<input type="hidden" name="keepwww" value="">
+<input type="hidden" name="keepslashes" value="">
+<input type="hidden" name="keepqueryorder" value="">
+<input type="hidden" name="toler" value="">
+<input type="hidden" name="http10" value="">
+<input type="hidden" name="sitemap" value="">
+
 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I58}
@@ -132,6 +144,17 @@ ${listid:robots:LISTDEF_8}
 </select>
 <br><br>
 
+<input type="checkbox" name="sitemap" ${checked:sitemap}
+			title='${html:LANG_SITEMAPTIP}' onMouseOver="info('${html:LANG_SITEMAPTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+> ${LANG_SITEMAP}
+<br><br>
+
+${LANG_SITEMAPURL}
+<input name="sitemapurl" value="${sitemapurl}" size="40"
+			title='${html:LANG_SITEMAPURLTIP}' onMouseOver="info('${html:LANG_SITEMAPURLTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+>
+<br><br>
+
 <input type="checkbox" name="updhack" ${checked:updhack}
 			title='${html:LANG_I1k}' onMouseOver="info('${html:LANG_I1k}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I62b}

html/server/option9.html

@@ -98,6 +98,14 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="warc" value="">
+<input type="hidden" name="changes" value="">
+<input type="hidden" name="norecatch" value="">
+<input type="hidden" name="logf" value="">
+<input type="hidden" name="index" value="">
+<input type="hidden" name="index2" value="">
+
 <input type="checkbox" name="cache2" ${checked:cache2}
 			title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I61}
@@ -114,6 +122,11 @@ ${LANG_WARCFILE}
 >
 <br><br>
 
+<input type="checkbox" name="changes" ${checked:changes}
+			title='${html:LANG_CHANGESTIP}' onMouseOver="info('${html:LANG_CHANGESTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+> ${LANG_CHANGES}
+<br><br>
+
 <input type="checkbox" name="norecatch" ${checked:norecatch}
 			title='${html:LANG_I5b}' onMouseOver="info('${html:LANG_I5b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I34b}

html/server/step2.html

@@ -141,8 +141,13 @@ ${do:copy:KeepSlashes:keepslashes}
 ${do:copy:KeepQueryOrder:keepqueryorder}
 ${do:copy:StripQuery:stripquery}
 ${do:copy:StoreAllInCache:cache2}
+${do:copy:Sitemap:sitemap}
+${do:copy:SitemapUrl:sitemapurl}
 ${do:copy:Warc:warc}
 ${do:copy:WarcFile:warcfile}
+${do:copy:Changes:changes}
+${do:copy:SingleFile:singlefile}
+${do:copy:SingleFileMaxSize:singlefilemax}
 ${do:copy:LogType:logtype}
 ${do:copy:UseHTTPProxyForFTP:ftpprox}
 ${do:copy:ProxyType:proxytype}

html/server/step4.html

@@ -114,7 +114,7 @@ ${do:end-if}
 
 ${/* Real commands and ini file generated below */}
 
-<!-- engine commandline -->
+<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
 ${do:output-mode:html}
 <textarea name="command" cols="50" rows="4" style="visibility:hidden">
 httrack \
@@ -175,8 +175,8 @@ ${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one *
 \
 	${unquoted:url2}
 \
-	${test:cookies:--cookies=0:}
-	${test:parsejava:--parse-java=0:}
+	${ztest:cookies:--cookies=0:}
+	${ztest:parsejava:--parse-java=0:}
 	${test:updhack:--updatehack}
 	${test:urlhack:--urlhack=0:--urlhack}
 	${test:keepwww:--keep-www-prefix}
@@ -188,8 +188,13 @@ ${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one *
 	${test:toler:--tolerant}
 	${test:http10:--http-10}
 	${test:cache2:--store-all-in-cache}
+	${test:sitemap:--sitemap}
+	${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"}
 	${test:warc:--warc}
 	${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
+	${test:changes:--changes}
+	${test:singlefile:--single-file}
+	${test:singlefilemax:--single-file-max-size=}${unquoted:singlefilemax}
 	${test:norecatch:--do-not-recatch}
 	${test:logf:--single-log}
 	${test:logtype:::--extra-log:--debug-log}
@@ -240,8 +245,13 @@ KeepSlashes=${ztest:keepslashes:0:1}
 KeepQueryOrder=${ztest:keepqueryorder:0:1}
 StripQuery=${stripquery}
 StoreAllInCache=${ztest:cache2:0:1}
+Sitemap=${ztest:sitemap:0:1}
+SitemapUrl=${sitemapurl}
 Warc=${ztest:warc:0:1}
 WarcFile=${warcfile}
+Changes=${ztest:changes:0:1}
+SingleFile=${ztest:singlefile:0:1}
+SingleFileMaxSize=${singlefilemax}
 LogType=${logtype}
 UseHTTPProxyForFTP=${ztest:ftpprox:0:1}
 ProxyType=${proxytype}

lang.def

@@ -1042,3 +1042,23 @@ LANG_WARCFILE
 WARC archive name:
 LANG_WARCFILETIP
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+LANG_CHANGES
+Report what changed since the previous mirror
+LANG_CHANGESTIP
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+LANG_SINGLEFILE
+Inline assets as data: URIs (self-contained pages)
+LANG_SINGLEFILETIP
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+LANG_SINGLEFILEMAX
+Largest inlined asset (bytes):
+LANG_SINGLEFILEMAXTIP
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+LANG_SITEMAP
+Seed the crawl from the site's sitemap
+LANG_SITEMAPTIP
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+LANG_SITEMAPURL
+Sitemap address:
+LANG_SITEMAPURLTIP
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/Bulgarian.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Èìå íà WARC àðõèâà:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåçàäúëæèòåëíî áàçîâî èìå çà WARC àðõèâà; îñòàâåòå ïðàçíî çà àâòîìàòè÷íî èìåíóâàíå â èçõîäíàòà äèðåêòîðèÿ.
+Report what changed since the previous mirror
+Îò÷åò çà ïðîìåíèòå ñïðÿìî ïðåäèøíîòî îãëåäàëî
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Çàïèñâàíå è íà hts-changes.json ñúñ ñïèñúê íà íîâèòå, ïðîìåíåíèòå, íåïðîìåíåíèòå è èç÷åçíàëèòå ôàéëîâå ñïðÿìî ïðåäèøíîòî îãëåäàëî.
+Inline assets as data: URIs (self-contained pages)
+Âãðàæäàíå íà ðåñóðñèòå êàòî data: URI (ñàìîñòîÿòåëíè ñòðàíèöè)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Ñëåä çàâúðøâàíå íà îãëåäàëîòî âñÿêà çàïàçåíà ñòðàíèöà ñå ïðåçàïèñâà ñ âãðàäåíè ñòèëîâå, ñêðèïòîâå, èçîáðàæåíèÿ è øðèôòîâå, òàêà ÷å ñòðàíèöàòà äà ìîæå äà ñå îòâîðè è ñàìîñòîÿòåëíî.
+Largest inlined asset (bytes):
+Íàé-ãîëÿì âãðàäåí ðåñóðñ (áàéòîâå):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Ðåñóðñ íàä òîçè ðàçìåð çàïàçâà îáèêíîâåíà âðúçêà; îñòàâåòå ïðàçíî çà ñòîéíîñòòà ïî ïîäðàçáèðàíå îò 10485760 áàéòà.
+Seed the crawl from the site's sitemap
+Çàïî÷âàíå íà îáõîæäàíåòî îò êàðòàòà íà ñàéòà
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ïðî÷èòàíå íà êàðòàòà íà ñàéòà (ðåäîâåòå Sitemap: â robots.txt, ñëåä òîâà /sitemap.xml) è äîáàâÿíå íà âñåêè ïîñî÷åí URL àäðåñ êàòî íà÷àëåí.
+Sitemap address:
+Àäðåñ íà êàðòàòà íà ñàéòà:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Àäðåñ íà êàðòà íà ñàéòà, êîÿòî äà áúäå ïðî÷åòåíà âìåñòî ñîíäèðàíå íà ñàéòà; îñòàâåòå ïðàçíî, çà äà ñå ïðîâåðè robots.txt, ñëåä òîâà /sitemap.xml.

lang/Castellano.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Nombre del archivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida.
+Report what changed since the previous mirror
+Informar de los cambios desde la copia anterior
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Escribir también hts-changes.json con la lista de lo que esta captura deja como nuevo, modificado, sin cambios o desaparecido respecto a la copia anterior.
+Inline assets as data: URIs (self-contained pages)
+Incrustar los recursos como URI data: (páginas autónomas)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Una vez terminada la copia, reescribir cada página guardada con sus hojas de estilo, scripts, imágenes y fuentes incrustadas, de modo que una página también pueda abrirse por sí sola.
+Largest inlined asset (bytes):
+Tamaño máximo del recurso incrustado (bytes):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Un recurso mayor que este tamaño conserva un enlace normal; déjelo vacío para el valor predeterminado de 10485760 bytes.
+Seed the crawl from the site's sitemap
+Iniciar el rastreo desde el mapa del sitio
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Leer el mapa del sitio (líneas Sitemap: de robots.txt, luego /sitemap.xml) y añadir como dirección inicial cada URL que incluya.
+Sitemap address:
+Dirección del mapa del sitio:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Dirección de un mapa del sitio que leer en lugar de sondear el sitio; déjelo vacío para sondear robots.txt y luego /sitemap.xml.

lang/Cesky.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Název archivu WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Volitelný základní název archivu WARC; ponechte prázdné pro automatické pojmenování ve výstupním adresáøi.
+Report what changed since the previous mirror
+Hlásit, co se od pøedchozího zrcadlení zmìnilo
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapsat také hts-changes.json se seznamem toho, co je oproti pøedchozímu zrcadlení nové, zmìnìné, nezmìnìné nebo chybìjící.
+Inline assets as data: URIs (self-contained pages)
+Vložit zdroje jako URI data: (samostatné stránky)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Po dokonèení zrcadlení pøepsat každou uloženou stránku s vloženými styly, skripty, obrázky a písmy, aby ji bylo možné otevøít i samostatnì.
+Largest inlined asset (bytes):
+Nejvìtší vložený zdroj (bajty):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Zdroj vìtší než tato velikost si ponechá bìžný odkaz; ponechte prázdné pro výchozí hodnotu 10485760 bajtù.
+Seed the crawl from the site's sitemap
+Zahájit procházení z mapy webu
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Naèíst mapu webu (øádky Sitemap: v souboru robots.txt, poté /sitemap.xml) a pøidat každou uvedenou adresu URL jako výchozí.
+Sitemap address:
+Adresa mapy webu:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa mapy webu, která se má naèíst místo zjiš<69>ování na webu; ponechte prázdné pro zjištìní z robots.txt a poté /sitemap.xml.

lang/Chinese-BIG5.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC 封存檔名稱：
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC 封存檔的選用基本名稱；留空則於輸出目錄中自動命名。
+Report what changed since the previous mirror
+回報自上次鏡射以來的變更
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+同時寫入 hts-changes.json，列出這次擷取相對於上次鏡射的新增、變更、未變更與消失的項目。
+Inline assets as data: URIs (self-contained pages)
+將資源內嵌為 data: URI（獨立網頁）
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+鏡射完成後，重寫每個已儲存的網頁，將樣式表、指令碼、圖片與字型內嵌其中，讓網頁也能單獨開啟。
+Largest inlined asset (bytes):
+內嵌資源大小上限（位元組）：
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+超過此大小的資源會保留一般連結；留空則使用預設的 10485760 位元組。
+Seed the crawl from the site's sitemap
+從網站的 Sitemap 開始擷取
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+讀取網站的 Sitemap（robots.txt 中的 Sitemap: 行，然後 /sitemap.xml），並將其中列出的每個網址加入為起始網址。
+Sitemap address:
+Sitemap 位址：
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+要讀取的 Sitemap 位址，用來取代自動探測；留空則先探測 robots.txt 再探測 /sitemap.xml。

lang/Chinese-Simplified.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC 归档名称：
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC 归档的可选基本名称；留空则在输出目录中自动命名。
+Report what changed since the previous mirror
+报告自上次镜像以来的变更
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+同时写入 hts-changes.json，列出本次抓取相对于上次镜像的新增、更改、未更改和消失的项目。
+Inline assets as data: URIs (self-contained pages)
+将资源内嵌为 data: URI（独立网页）
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+镜像完成后，重写每个已保存的网页，将样式表、脚本、图片和字体内嵌其中，使网页也能单独打开。
+Largest inlined asset (bytes):
+内嵌资源大小上限（字节）：
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+超过此大小的资源会保留普通链接；留空则使用默认的 10485760 字节。
+Seed the crawl from the site's sitemap
+从网站的 Sitemap 开始抓取
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+读取网站的 Sitemap（robots.txt 中的 Sitemap: 行，然后 /sitemap.xml），并将其中列出的每个网址添加为起始网址。
+Sitemap address:
+Sitemap 地址：
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+要读取的 Sitemap 地址，用来代替自动探测；留空则先探测 robots.txt 再探测 /sitemap.xml。

lang/Croatian.txt

@@ -966,3 +966,23 @@ WARC archive name:
 Naziv WARC arhive:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju.
+Report what changed since the previous mirror
+Prijavi ¹to se promijenilo od prethodnog zrcaljenja
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapi¹i i hts-changes.json s popisom onoga ¹to je u odnosu na prethodno zrcaljenje novo, promijenjeno, nepromijenjeno ili nestalo.
+Inline assets as data: URIs (self-contained pages)
+Ugradi resurse kao data: URI-je (samostalne stranice)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Nakon dovr¹etka zrcaljenja ponovno zapi¹i svaku spremljenu stranicu s ugraðenim stilskim listovima, skriptama, slikama i fontovima, tako da se stranica mo¾e otvoriti i zasebno.
+Largest inlined asset (bytes):
+Najveæi ugraðeni resurs (bajtovi):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Resurs veæi od ove velièine zadr¾ava obiènu poveznicu; ostavite prazno za zadanih 10485760 bajtova.
+Seed the crawl from the site's sitemap
+Pokreni pretra¾ivanje iz karte web-mjesta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Proèitaj kartu web-mjesta (retke Sitemap: iz robots.txt, zatim /sitemap.xml) i dodaj svaki navedeni URL kao poèetnu adresu.
+Sitemap address:
+Adresa karte web-mjesta:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa karte web-mjesta koju treba proèitati umjesto ispitivanja web-mjesta; ostavite prazno za ispitivanje robots.txt pa /sitemap.xml.

lang/Dansk.txt

@@ -1012,3 +1012,23 @@ WARC archive name:
 Navn på WARC-arkiv:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen.
+Report what changed since the previous mirror
+Rapportér hvad der er ændret siden den forrige spejling
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Skriv også hts-changes.json med en liste over, hvad denne gennemgang efterlader som nyt, ændret, uændret eller forsvundet i forhold til den forrige spejling.
+Inline assets as data: URIs (self-contained pages)
+Indlejr ressourcer som data:-URI'er (selvstændige sider)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Når spejlingen er færdig, omskrives hver gemt side med dens typografiark, scripts, billeder og skrifttyper indlejret, så en side også kan åbnes alene.
+Largest inlined asset (bytes):
+Største indlejrede ressource (byte):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+En ressource over denne størrelse beholder et almindeligt link; lad feltet stå tomt for standardværdien på 10485760 byte.
+Seed the crawl from the site's sitemap
+Start gennemgangen fra webstedets sitemap
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Læs webstedets sitemap (Sitemap:-linjer i robots.txt, derefter /sitemap.xml) og tilføj hver angivet URL som startadresse.
+Sitemap address:
+Sitemap-adresse:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adressen på et sitemap, der skal læses i stedet for at undersøge webstedet; lad feltet stå tomt for at undersøge robots.txt og derefter /sitemap.xml.

lang/Deutsch.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Name des WARC-Archivs:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen.
+Report what changed since the previous mirror
+Melden, was sich seit der vorherigen Spiegelung geändert hat
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zusätzlich hts-changes.json schreiben, das auflistet, was dieser Durchlauf gegenüber der vorherigen Spiegelung als neu, geändert, unverändert oder verschwunden hinterlässt.
+Inline assets as data: URIs (self-contained pages)
+Ressourcen als data:-URIs einbetten (eigenständige Seiten)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Nach Abschluss der Spiegelung jede gespeicherte Seite mit eingebetteten Stylesheets, Skripten, Bildern und Schriftarten neu schreiben, sodass eine Seite auch für sich allein geöffnet werden kann.
+Largest inlined asset (bytes):
+Größte eingebettete Ressource (Bytes):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Eine Ressource über dieser Größe behält einen gewöhnlichen Link; leer lassen für den Standardwert von 10485760 Bytes.
+Seed the crawl from the site's sitemap
+Erfassung mit der Sitemap der Website beginnen
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Die Sitemap der Website lesen (Sitemap:-Zeilen in robots.txt, dann /sitemap.xml) und jede dort aufgeführte URL als Startadresse hinzufügen.
+Sitemap address:
+Sitemap-Adresse:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresse einer Sitemap, die anstelle der Suche auf der Website gelesen wird; leer lassen, um robots.txt und dann /sitemap.xml zu prüfen.

lang/Eesti.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC-arhiivi nimi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC-arhiivi valikuline põhinimi; jäta tühjaks, et see väljundkataloogis automaatselt nimetada.
+Report what changed since the previous mirror
+Teata, mis on eelmisest peegeldusest saadik muutunud
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Kirjuta ka hts-changes.json, mis loetleb, mis on võrreldes eelmise peegeldusega uus, muutunud, muutumatu või kadunud.
+Inline assets as data: URIs (self-contained pages)
+Manusta ressursid data: URI-dena (iseseisvad lehed)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Pärast peegelduse lõppu kirjuta iga salvestatud leht ümber, manustades selle laaditabelid, skriptid, pildid ja fondid, nii et lehte saab avada ka eraldi.
+Largest inlined asset (bytes):
+Suurim manustatud ressurss (baiti):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Sellest suurem ressurss säilitab tavalise lingi; jäta tühjaks vaikeväärtuse 10485760 baiti jaoks.
+Seed the crawl from the site's sitemap
+Alusta kogumist saidi saidikaardist
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Loe saidi saidikaarti (robots.txt-i Sitemap:-read, seejärel /sitemap.xml) ja lisa iga seal loetletud URL alguslingina.
+Sitemap address:
+Saidikaardi aadress:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Saidikaardi aadress, mida lugeda saidi sondeerimise asemel; jäta tühjaks, et kontrollida robots.txt-i ja seejärel /sitemap.xml-i.

lang/English.txt

@@ -1012,3 +1012,23 @@ WARC archive name:
 WARC archive name:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+Report what changed since the previous mirror
+Report what changed since the previous mirror
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Inline assets as data: URIs (self-contained pages)
+Inline assets as data: URIs (self-contained pages)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Largest inlined asset (bytes):
+Largest inlined asset (bytes):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Seed the crawl from the site's sitemap
+Seed the crawl from the site's sitemap
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Sitemap address:
+Sitemap address:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/Finnish.txt

@@ -966,3 +966,23 @@ WARC archive name:
 WARC-arkiston nimi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valinnainen WARC-arkiston perusnimi; jätä tyhjäksi, jotta se nimetään automaattisesti tulostehakemistoon.
+Report what changed since the previous mirror
+Raportoi, mikä on muuttunut edellisen peilauksen jälkeen
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Kirjoita myös hts-changes.json, joka luettelee, mikä on edelliseen peilaukseen verrattuna uutta, muuttunutta, muuttumatonta tai kadonnutta.
+Inline assets as data: URIs (self-contained pages)
+Upota resurssit data:-URI-osoitteina (itsenäiset sivut)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Kun peilaus on valmis, kirjoita jokainen tallennettu sivu uudelleen niin, että sen tyylitiedostot, komentosarjat, kuvat ja fontit upotetaan, jolloin sivun voi avata myös yksinään.
+Largest inlined asset (bytes):
+Suurin upotettu resurssi (tavua):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Tätä suurempi resurssi säilyttää tavallisen linkin; jätä tyhjäksi, jolloin käytetään oletusarvoa 10485760 tavua.
+Seed the crawl from the site's sitemap
+Aloita haku sivuston sivukartasta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Lue sivuston sivukartta (robots.txt-tiedoston Sitemap:-rivit, sitten /sitemap.xml) ja lisää jokainen siinä lueteltu URL-osoite aloitusosoitteeksi.
+Sitemap address:
+Sivukartan osoite:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Luettavan sivukartan osoite sivuston luotaamisen sijaan; jätä tyhjäksi, jolloin tarkistetaan robots.txt ja sitten /sitemap.xml.

lang/Francais.txt

@@ -1012,3 +1012,23 @@ WARC archive name:
 Nom de l'archive WARC :
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.
+Report what changed since the previous mirror
+Signaler ce qui a changé depuis le miroir précédent
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Écrire aussi hts-changes.json, qui liste ce que ce crawl laisse nouveau, modifié, inchangé ou disparu par rapport au miroir précédent.
+Inline assets as data: URIs (self-contained pages)
+Intégrer les ressources en URI data: (pages autonomes)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Une fois le miroir terminé, réécrire chaque page enregistrée avec ses feuilles de style, scripts, images et polices intégrés, afin qu'une page puisse aussi être ouverte seule.
+Largest inlined asset (bytes):
+Taille maximale d'une ressource intégrée (octets) :
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Au-delà de cette taille, la ressource reste un lien ordinaire ; laissez vide pour la valeur par défaut de 10485760 octets.
+Seed the crawl from the site's sitemap
+Partir du plan de site (sitemap)
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Lire le plan de site (lignes Sitemap: de robots.txt, puis /sitemap.xml) et ajouter chaque URL listée comme adresse de départ.
+Sitemap address:
+Adresse du plan de site :
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresse d'un plan de site à lire au lieu de sonder le site ; laissez vide pour sonder robots.txt puis /sitemap.xml.

lang/Greek.txt

@@ -966,3 +966,23 @@ WARC archive name:
 Όνομα αρχείου WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Προαιρετικό βασικό όνομα για το αρχείο WARC. Αφήστε το κενό για αυτόματη ονομασία στον κατάλογο εξόδου.
+Report what changed since the previous mirror
+Αναφορά των αλλαγών από το προηγούμενο αντίγραφο
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Εγγραφή και του hts-changes.json, που παραθέτει τι είναι νέο, τι άλλαξε, τι έμεινε ίδιο και τι χάθηκε σε σχέση με το προηγούμενο αντίγραφο.
+Inline assets as data: URIs (self-contained pages)
+Ενσωμάτωση των πόρων ως URI data: (αυτόνομες σελίδες)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Μόλις ολοκληρωθεί το αντίγραφο, κάθε αποθηκευμένη σελίδα ξαναγράφεται με ενσωματωμένα τα φύλλα στυλ, τα σενάρια, τις εικόνες και τις γραμματοσειρές της, ώστε η σελίδα να μπορεί να ανοίξει και μόνη της.
+Largest inlined asset (bytes):
+Μέγιστος ενσωματωμένος πόρος (byte):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Πόρος μεγαλύτερος από αυτό το μέγεθος διατηρεί κανονικό σύνδεσμο. Αφήστε το κενό για την προεπιλογή των 10485760 byte.
+Seed the crawl from the site's sitemap
+Έναρξη της ανίχνευσης από τον χάρτη του ιστότοπου
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ανάγνωση του χάρτη του ιστότοπου (γραμμές Sitemap: στο robots.txt, έπειτα /sitemap.xml) και προσθήκη κάθε διεύθυνσης URL που περιέχει ως αρχικής διεύθυνσης.
+Sitemap address:
+Διεύθυνση χάρτη ιστότοπου:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Διεύθυνση χάρτη ιστότοπου προς ανάγνωση αντί για αναζήτηση στον ιστότοπο. Αφήστε το κενό για έλεγχο του robots.txt και έπειτα του /sitemap.xml.

lang/Italiano.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Nome dell'archivio WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome di base facoltativo per l'archivio WARC; lascia vuoto per assegnarlo automaticamente nella directory di output.
+Report what changed since the previous mirror
+Segnala che cosa è cambiato dalla copia precedente
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Scrive anche hts-changes.json, che elenca ciò che questa scansione lascia come nuovo, modificato, invariato o scomparso rispetto alla copia precedente.
+Inline assets as data: URIs (self-contained pages)
+Incorpora le risorse come URI data: (pagine autonome)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Al termine della copia, riscrive ogni pagina salvata incorporando fogli di stile, script, immagini e caratteri, in modo che una pagina possa essere aperta anche da sola.
+Largest inlined asset (bytes):
+Dimensione massima della risorsa incorporata (byte):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Una risorsa oltre questa dimensione mantiene un collegamento normale; lasciare vuoto per il valore predefinito di 10485760 byte.
+Seed the crawl from the site's sitemap
+Avvia la scansione dalla mappa del sito
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Legge la mappa del sito (righe Sitemap: in robots.txt, poi /sitemap.xml) e aggiunge come indirizzo iniziale ogni URL elencato.
+Sitemap address:
+Indirizzo della mappa del sito:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Indirizzo di una mappa del sito da leggere invece di sondare il sito; lasciare vuoto per sondare robots.txt e poi /sitemap.xml.

lang/Japanese.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC アーカイブ名:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC アーカイブの任意のベース名。空欄にすると出力ディレクトリ内で自動的に名前が付けられます。
+Report what changed since the previous mirror
+前回のミラーからの変更点を報告する
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+hts-changes.json も書き出し、前回のミラーと比べて新規、変更、変更なし、消滅となったものを一覧にします。
+Inline assets as data: URIs (self-contained pages)
+リソースを data: URI として埋め込む (単独で開けるページ)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+ミラーの完了後、保存された各ページを、スタイルシート、スクリプト、画像、フォントを埋め込んだ形で書き直します。ページ単体でも開けるようになります。
+Largest inlined asset (bytes):
+埋め込む最大サイズ (バイト):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+このサイズを超えるリソースは通常のリンクのままになります。空欄にすると既定値の 10485760 バイトになります。
+Seed the crawl from the site's sitemap
+サイトマップからミラーリングを開始する
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+サイトマップ (robots.txt の Sitemap: 行、次に /sitemap.xml) を読み込み、記載されているすべての URL を開始アドレスとして追加します。
+Sitemap address:
+サイトマップのアドレス:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+サイトを探索する代わりに読み込むサイトマップのアドレス。空欄にすると robots.txt、次に /sitemap.xml を探索します。

lang/Macedonian.txt

@@ -964,3 +964,23 @@ WARC archive name:
 ¸ÜÕ ÝÐ WARC ÐàåØÒÐâÐ:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 ¸×ÑÞàÝÞ ÞáÝÞÒÝÞ ØÜÕ ×Ð WARC ÐàåØÒÐâÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÐÒâÞÜÐâáÚÞ ØÜÕÝãÒÐúÕ ÒÞ Ø×ÛÕ×ÝØÞâ ÔØàÕÚâÞàØãÜ.
+Report what changed since the previous mirror
+¸×ÒÕáâØ èâÞ Õ ßàÞÜÕÝÕâÞ ÞÔ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+·ÐßØèØ Ø hts-changes.json áÞ áßØáÞÚ ÝÐ âÞÐ èâÞ Õ ÝÞÒÞ, ßàÞÜÕÝÕâÞ, ÝÕßàÞÜÕÝÕâÞ ØÛØ ØáçÕ×ÝÐâÞ ÒÞ ÞÔÝÞá ÝÐ ßàÕâåÞÔÝÞâÞ ÞÓÛÕÔÐÛÞ.
+Inline assets as data: URIs (self-contained pages)
+²ÓàÐÔØ ÓØ àÕáãàáØâÕ ÚÐÚÞ data: URI (áÐÜÞáâÞøÝØ áâàÐÝØæØ)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+¿Þ ×ÐÒàèãÒÐúÕ ÝÐ ÞÓÛÕÔÐÛÞâÞ, áÕÚÞøÐ ×ÐçãÒÐÝÐ áâàÐÝØæÐ áÕ ßàÕßØèãÒÐ áÞ ÒÓàÐÔÕÝØ áâØÛáÚØ ÛØáâÞÒØ, áÚàØßâØ, áÛØÚØ Ø äÞÝâÞÒØ, ×Ð ÔÐ ÜÞÖÕ áâàÐÝØæÐâÐ ÔÐ áÕ ÞâÒÞàØ Ø áÐÜÞáâÞøÝÞ.
+Largest inlined asset (bytes):
+½ÐøÓÞÛÕÜ ÒÓàÐÔÕÝ àÕáãàá (ÑÐøâØ):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+ÀÕáãàá ßÞÓÞÛÕÜ ÞÔ ÞÒÐÐ ÓÞÛÕÜØÝÐ ×ÐÔàÖãÒÐ ÞÑØçÝÐ ÒàáÚÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð áâÐÝÔÐàÔÝØâÕ 10485760 ÑÐøâØ.
+Seed the crawl from the site's sitemap
+·ÐßÞçÝØ ÓÞ ßàÕÑÐàãÒÐúÕâÞ ÞÔ ÚÐàâÐâÐ ÝÐ áÐøâÞâ
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+¿àÞçØâÐø øÐ ÚÐàâÐâÐ ÝÐ áÐøâÞâ (àÕÔÞÒØâÕ Sitemap: ÒÞ robots.txt, ßÞâÞÐ /sitemap.xml) Ø ÔÞÔÐø øÐ áÕÚÞøÐ ÝÐÒÕÔÕÝÐ URL ÐÔàÕáÐ ÚÐÚÞ ßÞçÕâÝÐ.
+Sitemap address:
+°ÔàÕáÐ ÝÐ ÚÐàâÐâÐ ÝÐ áÐøâÞâ:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+°ÔàÕáÐ ÝÐ ÚÐàâÐ ÝÐ áÐøâÞâ èâÞ âàÕÑÐ ÔÐ áÕ ßàÞçØâÐ ÝÐÜÕáâÞ ØáßØâãÒÐúÕ ÝÐ áÐøâÞâ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÔÐ áÕ ØáßØâÐ robots.txt, ßÐ /sitemap.xml.

lang/Magyar.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC archívum neve:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 A WARC archívum opcionális alapneve; hagyja üresen az automatikus elnevezéshez a kimeneti könyvtárban.
+Report what changed since the previous mirror
+Jelentés arról, mi változott az elõzõ tükrözés óta
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+A hts-changes.json fájl írása is, amely felsorolja, mi új, mi változott, mi maradt változatlan és mi tûnt el az elõzõ tükrözéshez képest.
+Inline assets as data: URIs (self-contained pages)
+Erõforrások beágyazása data: URI-ként (önálló oldalak)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+A tükrözés befejezése után minden mentett oldal újraírása a stíluslapok, parancsfájlok, képek és betûtípusok beágyazásával, így az oldal önmagában is megnyitható.
+Largest inlined asset (bytes):
+Legnagyobb beágyazott erõforrás (bájt):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Az ennél nagyobb erõforrás közönséges hivatkozás marad; hagyja üresen a 10485760 bájtos alapértelmezéshez.
+Seed the crawl from the site's sitemap
+A letöltés indítása a webhely webhelytérképérõl
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+A webhely webhelytérképének beolvasása (a robots.txt Sitemap: sorai, majd a /sitemap.xml), és a benne felsorolt összes URL felvétele kiindulási címként.
+Sitemap address:
+Webhelytérkép címe:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+A webhely vizsgálata helyett beolvasandó webhelytérkép címe; hagyja üresen a robots.txt, majd a /sitemap.xml vizsgálatához.

lang/Nederlands.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Naam van WARC-archief:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optionele basisnaam voor het WARC-archief; laat leeg om het automatisch een naam te geven in de uitvoermap.
+Report what changed since the previous mirror
+Rapporteren wat er sinds de vorige spiegeling is gewijzigd
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Ook hts-changes.json schrijven met een overzicht van wat deze doorloop nieuw, gewijzigd, ongewijzigd of verdwenen laat ten opzichte van de vorige spiegeling.
+Inline assets as data: URIs (self-contained pages)
+Bronnen insluiten als data:-URI's (op zichzelf staande pagina's)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Zodra de spiegeling klaar is, elke opgeslagen pagina herschrijven met ingesloten stijlbladen, scripts, afbeeldingen en lettertypen, zodat een pagina ook los geopend kan worden.
+Largest inlined asset (bytes):
+Grootste ingesloten bron (bytes):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Een bron boven deze grootte houdt een gewone koppeling; laat leeg voor de standaardwaarde van 10485760 bytes.
+Seed the crawl from the site's sitemap
+De crawl starten vanaf de sitemap van de site
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+De sitemap van de site lezen (Sitemap:-regels in robots.txt, daarna /sitemap.xml) en elke vermelde URL als startadres toevoegen.
+Sitemap address:
+Sitemap-adres:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adres van een sitemap die gelezen moet worden in plaats van de site te onderzoeken; laat leeg om robots.txt en daarna /sitemap.xml te controleren.

lang/Norsk.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Navn på WARC-arkiv:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valgfritt basisnavn for WARC-arkivet; la feltet stå tomt for automatisk navngivning i utdatamappen.
+Report what changed since the previous mirror
+Rapporter hva som er endret siden forrige speiling
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Skriv også hts-changes.json som lister opp hva denne gjennomgangen etterlater som nytt, endret, uendret eller forsvunnet sammenlignet med forrige speiling.
+Inline assets as data: URIs (self-contained pages)
+Bygg inn ressurser som data:-URI-er (selvstendige sider)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Når speilingen er ferdig, skrives hver lagrede side om med stilark, skript, bilder og skrifter innebygd, slik at en side også kan åpnes alene.
+Largest inlined asset (bytes):
+Største innebygde ressurs (byte):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+En ressurs over denne størrelsen beholder en vanlig lenke; la feltet stå tomt for standardverdien på 10485760 byte.
+Seed the crawl from the site's sitemap
+Start gjennomgangen fra nettstedets nettstedskart
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Les nettstedets nettstedskart (Sitemap:-linjer i robots.txt, deretter /sitemap.xml) og legg til hver oppført URL som startadresse.
+Sitemap address:
+Adresse til nettstedskart:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adressen til et nettstedskart som skal leses i stedet for å undersøke nettstedet; la feltet stå tomt for å undersøke robots.txt og deretter /sitemap.xml.

lang/Polski.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Nazwa archiwum WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Opcjonalna nazwa bazowa archiwum WARC; pozostaw puste, aby nazwaæ je automatycznie w katalogu wyj¶ciowym.
+Report what changed since the previous mirror
+Zg³o¶, co zmieni³o siê od poprzedniej kopii
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapisz równie¿ plik hts-changes.json z list± tego, co w porównaniu z poprzedni± kopi± jest nowe, zmienione, niezmienione lub usuniête.
+Inline assets as data: URIs (self-contained pages)
+Osad¼ zasoby jako identyfikatory data: URI (samodzielne strony)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Po zakoñczeniu tworzenia kopii przepisz ka¿d± zapisan± stronê z osadzonymi arkuszami stylów, skryptami, obrazami i czcionkami, aby stronê mo¿na by³o otworzyæ równie¿ samodzielnie.
+Largest inlined asset (bytes):
+Najwiêkszy osadzony zasób (bajty):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Zasób wiêkszy ni¿ ten rozmiar zachowuje zwyk³y odno¶nik; pozostaw puste, aby u¿yæ domy¶lnych 10485760 bajtów.
+Seed the crawl from the site's sitemap
+Rozpocznij pobieranie od mapy witryny
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Odczytaj mapê witryny (wiersze Sitemap: w pliku robots.txt, nastêpnie /sitemap.xml) i dodaj ka¿dy wymieniony adres URL jako adres pocz±tkowy.
+Sitemap address:
+Adres mapy witryny:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adres mapy witryny do odczytania zamiast sondowania witryny; pozostaw puste, aby sprawdziæ robots.txt, a nastêpnie /sitemap.xml.

lang/Portugues-Brasil.txt

@@ -1012,3 +1012,23 @@ WARC archive name:
 Nome do arquivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome base opcional para o arquivo WARC; deixe em branco para nomeá-lo automaticamente no diretório de saída.
+Report what changed since the previous mirror
+Relatar o que mudou desde o espelhamento anterior
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Gravar também hts-changes.json listando o que esta captura deixa como novo, alterado, inalterado ou removido em relação ao espelhamento anterior.
+Inline assets as data: URIs (self-contained pages)
+Incorporar os recursos como URIs data: (páginas autônomas)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Ao terminar o espelhamento, reescrever cada página salva com suas folhas de estilo, scripts, imagens e fontes incorporadas, para que a página também possa ser aberta sozinha.
+Largest inlined asset (bytes):
+Maior recurso incorporado (bytes):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Um recurso acima desse tamanho mantém um link comum; deixe em branco para o padrão de 10485760 bytes.
+Seed the crawl from the site's sitemap
+Iniciar a captura pelo mapa do site
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ler o mapa do site (linhas Sitemap: do robots.txt, depois /sitemap.xml) e adicionar como endereço inicial cada URL nele listada.
+Sitemap address:
+Endereço do mapa do site:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Endereço de um mapa do site a ser lido em vez de sondar o site; deixe em branco para sondar robots.txt e depois /sitemap.xml.

lang/Portugues.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Nome do arquivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome base opcional para o arquivo WARC; deixe em branco para o nomear automaticamente no diretório de saída.
+Report what changed since the previous mirror
+Comunicar o que mudou desde o espelho anterior
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Escrever também hts-changes.json, que lista o que esta recolha deixa como novo, alterado, inalterado ou desaparecido em relação ao espelho anterior.
+Inline assets as data: URIs (self-contained pages)
+Incorporar os recursos como URI data: (páginas autónomas)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Depois de terminado o espelho, reescrever cada página guardada com as suas folhas de estilo, scripts, imagens e tipos de letra incorporados, para que a página também possa ser aberta sozinha.
+Largest inlined asset (bytes):
+Maior recurso incorporado (bytes):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Um recurso acima deste tamanho mantém uma ligação normal; deixe em branco para o valor predefinido de 10485760 bytes.
+Seed the crawl from the site's sitemap
+Iniciar a recolha pelo mapa do site
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ler o mapa do site (linhas Sitemap: do robots.txt, depois /sitemap.xml) e adicionar como endereço inicial cada URL nele listado.
+Sitemap address:
+Endereço do mapa do site:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Endereço de um mapa do site a ler em vez de sondar o site; deixe em branco para sondar robots.txt e depois /sitemap.xml.

lang/Romanian.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Numele arhivei WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nume de baza optional pentru arhiva WARC; lasati gol pentru a-l denumi automat in directorul de iesire.
+Report what changed since the previous mirror
+Raporteaza ce s-a schimbat fata de copia anterioara
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Scrie si hts-changes.json, care listeaza ce este nou, modificat, nemodificat sau disparut fata de copia anterioara.
+Inline assets as data: URIs (self-contained pages)
+Încorporeaza resursele ca URI data: (pagini de sine statatoare)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Dupa terminarea copiei, fiecare pagina salvata este rescrisa cu foile de stil, scripturile, imaginile si fonturile încorporate, astfel încât pagina sa poata fi deschisa si singura.
+Largest inlined asset (bytes):
+Cea mai mare resursa încorporata (octeti):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+O resursa mai mare decât aceasta dimensiune pastreaza o legatura obisnuita; lasati gol pentru valoarea implicita de 10485760 de octeti.
+Seed the crawl from the site's sitemap
+Porneste explorarea de la harta sitului
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Citeste harta sitului (liniile Sitemap: din robots.txt, apoi /sitemap.xml) si adauga fiecare URL listat ca adresa de pornire.
+Sitemap address:
+Adresa hartii sitului:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa unei harti a sitului care sa fie citita în loc de sondarea sitului; lasati gol pentru a sonda robots.txt, apoi /sitemap.xml.

lang/Russian.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Èìÿ WARC-àðõèâà:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåîáÿçàòåëüíîå áàçîâîå èìÿ WARC-àðõèâà; îñòàâüòå ïóñòûì äëÿ àâòîìàòè÷åñêîãî èìåíîâàíèÿ â âûõîäíîì êàòàëîãå.
+Report what changed since the previous mirror
+Ñîîáùàòü, ÷òî èçìåíèëîñü ñ ïðåäûäóùåãî çåðêàëà
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Òàêæå çàïèñûâàòü hts-changes.json ñî ñïèñêîì òîãî, ÷òî ïî ñðàâíåíèþ ñ ïðåäûäóùèì çåðêàëîì ñòàëî íîâûì, èçìåí¸ííûì, íåèçìåí¸ííûì èëè èñ÷åçëî.
+Inline assets as data: URIs (self-contained pages)
+Âñòðàèâàòü ðåñóðñû êàê data: URI (ñàìîñòîÿòåëüíûå ñòðàíèöû)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Ïîñëå çàâåðøåíèÿ çåðêàëèðîâàíèÿ êàæäàÿ ñîõðàí¸ííàÿ ñòðàíèöà ïåðåçàïèñûâàåòñÿ ñî âñòðîåííûìè òàáëèöàìè ñòèëåé, ñöåíàðèÿìè, èçîáðàæåíèÿìè è øðèôòàìè, ÷òîáû ñòðàíèöó ìîæíî áûëî îòêðûòü îòäåëüíî.
+Largest inlined asset (bytes):
+Íàèáîëüøèé âñòðàèâàåìûé ðåñóðñ (áàéòû):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Ðåñóðñ áîëüøå ýòîãî ðàçìåðà ñîõðàíÿåò îáû÷íóþ ññûëêó; îñòàâüòå ïóñòûì äëÿ çíà÷åíèÿ ïî óìîë÷àíèþ 10485760 áàéò.
+Seed the crawl from the site's sitemap
+Íà÷èíàòü îáõîä ñ êàðòû ñàéòà
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ïðî÷èòàòü êàðòó ñàéòà (ñòðîêè Sitemap: â robots.txt, çàòåì /sitemap.xml) è äîáàâèòü êàæäûé óêàçàííûé â íåé URL êàê íà÷àëüíûé àäðåñ.
+Sitemap address:
+Àäðåñ êàðòû ñàéòà:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Àäðåñ êàðòû ñàéòà, êîòîðóþ íóæíî ïðî÷èòàòü âìåñòî îïðîñà ñàéòà; îñòàâüòå ïóñòûì, ÷òîáû ïðîâåðèòü robots.txt, çàòåì /sitemap.xml.

lang/Slovak.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Názov archívu WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Voliteµný základný názov archívu WARC; ponechajte prázdne pre automatické pomenovanie vo výstupnom adresári.
+Report what changed since the previous mirror
+Oznámi», èo sa od predchádzajúceho zrkadlenia zmenilo
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapísa» aj hts-changes.json so zoznamom toho, èo je oproti predchádzajúcemu zrkadleniu nové, zmenené, nezmenené alebo chýbajúce.
+Inline assets as data: URIs (self-contained pages)
+Vlo¾i» zdroje ako URI data: (samostatné stránky)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Po dokonèení zrkadlenia prepísa» ka¾dú ulo¾enú stránku s vlo¾enými ¹týlmi, skriptmi, obrázkami a písmami, aby sa stránka dala otvori» aj samostatne.
+Largest inlined asset (bytes):
+Najväè¹í vlo¾ený zdroj (bajty):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Zdroj väè¹í ne¾ táto veµkos» si ponechá be¾ný odkaz; ponechajte prázdne pre predvolených 10485760 bajtov.
+Seed the crawl from the site's sitemap
+Zaèa» prehliadanie z mapy stránok
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Naèíta» mapu stránok (riadky Sitemap: v súbore robots.txt, potom /sitemap.xml) a prida» ka¾dú uvedenú adresu URL ako poèiatoènú.
+Sitemap address:
+Adresa mapy stránok:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa mapy stránok, ktorá sa má naèíta» namiesto zis»ovania na stránke; ponechajte prázdne na zistenie z robots.txt a potom /sitemap.xml.

lang/Slovenian.txt

@@ -964,3 +964,23 @@ WARC archive name:
 Ime arhiva WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Neobvezno osnovno ime arhiva WARC; pustite prazno za samodejno poimenovanje v izhodni mapi.
+Report what changed since the previous mirror
+Porocaj, kaj se je spremenilo od prejsnjega zrcaljenja
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Zapisi tudi hts-changes.json s seznamom tega, kar je v primerjavi s prejsnjim zrcaljenjem novo, spremenjeno, nespremenjeno ali izginilo.
+Inline assets as data: URIs (self-contained pages)
+Vgradi vire kot data: URI (samostojne strani)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Ko je zrcaljenje koncano, se vsaka shranjena stran prepise z vgrajenimi slogovnimi listi, skripti, slikami in pisavami, tako da jo je mogoce odpreti tudi samostojno.
+Largest inlined asset (bytes):
+Najvecji vgrajeni vir (bajti):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Vir, vecji od te velikosti, ohrani obicajno povezavo; pustite prazno za privzetih 10485760 bajtov.
+Seed the crawl from the site's sitemap
+Zacni zajem z zemljevidom spletnega mesta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Preberi zemljevid spletnega mesta (vrstice Sitemap: v robots.txt, nato /sitemap.xml) in dodaj vsak navedeni URL kot zacetni naslov.
+Sitemap address:
+Naslov zemljevida spletnega mesta:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Naslov zemljevida spletnega mesta, ki naj se prebere namesto preverjanja mesta; pustite prazno za preverjanje robots.txt in nato /sitemap.xml.

lang/Svenska.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC-arkivets namn:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valfritt basnamn för WARC-arkivet; lämna tomt för att namnge det automatiskt i utdatakatalogen.
+Report what changed since the previous mirror
+Rapportera vad som ändrats sedan föregående spegling
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Skriv även hts-changes.json som listar vad den här genomgången lämnar som nytt, ändrat, oförändrat eller försvunnet jämfört med föregående spegling.
+Inline assets as data: URIs (self-contained pages)
+Bädda in resurser som data:-URI:er (fristående sidor)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+När speglingen är klar skrivs varje sparad sida om med sina formatmallar, skript, bilder och teckensnitt inbäddade, så att en sida också kan öppnas för sig.
+Largest inlined asset (bytes):
+Största inbäddade resurs (byte):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+En resurs över den här storleken behåller en vanlig länk; lämna tomt för standardvärdet 10485760 byte.
+Seed the crawl from the site's sitemap
+Starta insamlingen från webbplatsens webbplatskarta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Läs webbplatsens webbplatskarta (Sitemap:-rader i robots.txt, sedan /sitemap.xml) och lägg till varje angiven URL som startadress.
+Sitemap address:
+Webbplatskartans adress:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adress till en webbplatskarta som ska läsas i stället för att söka på webbplatsen; lämna tomt för att kontrollera robots.txt och sedan /sitemap.xml.

lang/Turkish.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC arþivi adý:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC arþivi için isteðe baðlý temel ad; çýktý dizininde otomatik adlandýrma için boþ býrakýn.
+Report what changed since the previous mirror
+Önceki yansýmadan bu yana deðiþenleri bildir
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Ayrýca hts-changes.json yazarak bu taramanýn önceki yansýmaya göre neyi yeni, deðiþmiþ, deðiþmemiþ veya kaybolmuþ býraktýðýný listele.
+Inline assets as data: URIs (self-contained pages)
+Kaynaklarý data: URI olarak göm (kendi kendine yeten sayfalar)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Yansýlama bittiðinde, kaydedilen her sayfa stil sayfalarý, betikleri, görselleri ve yazý tipleri gömülü olarak yeniden yazýlýr; böylece sayfa tek baþýna da açýlabilir.
+Largest inlined asset (bytes):
+En büyük gömülü kaynak (bayt):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Bu boyutun üzerindeki bir kaynak sýradan baðlantýsýný korur; 10485760 baytlýk varsayýlan için boþ býrakýn.
+Seed the crawl from the site's sitemap
+Taramayý sitenin site haritasýndan baþlat
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Sitenin site haritasýný oku (robots.txt içindeki Sitemap: satýrlarý, ardýndan /sitemap.xml) ve listelenen her URL'yi baþlangýç adresi olarak ekle.
+Sitemap address:
+Site haritasý adresi:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Siteyi yoklamak yerine okunacak site haritasýnýn adresi; robots.txt ve ardýndan /sitemap.xml yoklamasý için boþ býrakýn.

lang/Ukrainian.txt

@@ -964,3 +964,23 @@ WARC archive name:
 ²ì'ÿ WARC-àðõ³âó:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåîáîâ'ÿçêîâà áàçîâà íàçâà WARC-àðõ³âó; çàëèøòå ïîðîæí³ì äëÿ àâòîìàòè÷íîãî íàéìåíóâàííÿ ó âèõ³äíîìó êàòàëîç³.
+Report what changed since the previous mirror
+Ïîâ³äîìëÿòè, ùî çì³íèëîñÿ ç ïîïåðåäíüîãî äçåðêàëà
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+Òàêîæ çàïèñóâàòè hts-changes.json ç³ ñïèñêîì òîãî, ùî ïîð³âíÿíî ç ïîïåðåäí³ì äçåðêàëîì º íîâèì, çì³íåíèì, íåçì³íåíèì àáî çíèêëèì.
+Inline assets as data: URIs (self-contained pages)
+Âáóäîâóâàòè ðåñóðñè ÿê data: URI (ñàìîñò³éí³ ñòîð³íêè)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+ϳñëÿ çàâåðøåííÿ äçåðêàëþâàííÿ êîæíà çáåðåæåíà ñòîð³íêà ïåðåçàïèñóºòüñÿ ç âáóäîâàíèìè òàáëèöÿìè ñòèë³â, ñöåíàð³ÿìè, çîáðàæåííÿìè òà øðèôòàìè, ùîá ñòîð³íêó ìîæíà áóëî â³äêðèòè îêðåìî.
+Largest inlined asset (bytes):
+Íàéá³ëüøèé âáóäîâàíèé ðåñóðñ (áàéòè):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Ðåñóðñ, á³ëüøèé çà öåé ðîçì³ð, çáåð³ãຠçâè÷àéíå ïîñèëàííÿ; çàëèøòå ïîðîæí³ì äëÿ òèïîâîãî çíà÷åííÿ 10485760 áàéò³â.
+Seed the crawl from the site's sitemap
+Ïî÷èíàòè îáõ³ä ç êàðòè ñàéòó
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ïðî÷èòàòè êàðòó ñàéòó (ðÿäêè Sitemap: ó robots.txt, ïîò³ì /sitemap.xml) ³ äîäàòè êîæíó âêàçàíó â í³é URL-àäðåñó ÿê ïî÷àòêîâó.
+Sitemap address:
+Àäðåñà êàðòè ñàéòó:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Àäðåñà êàðòè ñàéòó, ÿêó ñë³ä ïðî÷èòàòè çàì³ñòü îïèòóâàííÿ ñàéòó; çàëèøòå ïîðîæí³ì, ùîá ïåðåâ³ðèòè robots.txt, à ïîò³ì /sitemap.xml.

lang/Uzbek.txt

@@ -964,3 +964,23 @@ WARC archive name:
 WARC arxivi nomi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC arxivi uchun ixtiyoriy asosiy nom; chiqish katalogida avtomatik nomlash uchun bo'sh qoldiring.
+Report what changed since the previous mirror
+Oldingi nusxadan beri nima o’zgarganini xabar qilish
+Also write hts-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror.
+hts-changes.json ham yoziladi: unda ushbu yig’ish oldingi nusxaga nisbatan nimani yangi, o’zgargan, o’zgarmagan yoki yo’qolgan holda qoldirgani ro’yxati bo’ladi.
+Inline assets as data: URIs (self-contained pages)
+Resurslarni data: URI sifatida joylash (mustaqil sahifalar)
+Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded, so a page can also be opened on its own.
+Nusxalash tugagach, har bir saqlangan sahifa uslublar jadvallari, skriptlar, rasmlar va shriftlar joylangan holda qayta yoziladi, shunda sahifani alohida ham ochish mumkin.
+Largest inlined asset (bytes):
+Eng katta joylangan resurs (bayt):
+An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
+Bu o’lchamdan katta resurs oddiy havolani saqlab qoladi; standart 10485760 bayt uchun bo’sh qoldiring.
+Seed the crawl from the site's sitemap
+Yig’ishni saytning sayt xaritasidan boshlash
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Saytning sayt xaritasini o’qish (robots.txt dagi Sitemap: qatorlari, so’ngra /sitemap.xml) va unda ko’rsatilgan har bir URL manzilni boshlang’ich manzil sifatida qo’shish.
+Sitemap address:
+Sayt xaritasi manzili:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Saytni tekshirish o’rniga o’qiladigan sayt xaritasi manzili; robots.txt, so’ngra /sitemap.xml ni tekshirish uchun bo’sh qoldiring.

libtest/callbacks-example-filenameiisbug.c

@@ -71,7 +71,9 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
       for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
       if (iisBogus[i][j] == '\0'
           && (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
-        strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
+        /* j bytes matched, so j fit: copying j cannot overrun whatever the
+           table holds, and the tail must survive untouched */
+        memcpy(a, iisBogusReplace[i], (size_t) j);
         break;
       }
     }

m4/check_zlib.m4

@@ -1,82 +1,33 @@
 dnl @synopsis CHECK_ZLIB()
 dnl
-dnl This macro searches for an installed zlib library. If nothing
-dnl was specified when calling configure, it searches first in /usr/local
-dnl and then in /usr. If the --with-zlib=DIR is specified, it will try
-dnl to find it in DIR/include/zlib.h and DIR/lib/libz.a. If --without-zlib
-dnl is specified, the library is not searched at all.
-dnl
-dnl If either the header file (zlib.h) or the library (libz) is not
-dnl found, the configuration exits on error, asking for a valid
-dnl zlib installation directory or --without-zlib.
-dnl
-dnl The macro defines the symbol HAVE_LIBZ if the library is found. You should
-dnl use autoheader to include a definition for this symbol in a config.h
-dnl file. Sample usage in a C/C++ source is as follows:
-dnl
-dnl   #ifdef HAVE_LIBZ
-dnl   #include <zlib.h>
-dnl   #endif /* HAVE_LIBZ */
-dnl
-dnl @version $Id$
-dnl @author Loic Dachary <loic@senga.org>
+dnl Look for zlib. It is a hard requirement, not an option: the cache and the
+dnl WARC output are zip/gzip containers, and the bundled minizip calls zlib
+dnl directly. --with-zlib=DIR points at a non-standard prefix.
 dnl
+dnl Adds -lz to LIBS and defines HAVE_LIBZ.
 
-AC_DEFUN([CHECK_ZLIB],
-#
-# Handle user hints
-#
-[AC_MSG_CHECKING(if zlib is wanted)
-AC_ARG_WITH(zlib,
-[  --with-zlib=DIR root directory path of zlib installation [defaults to
-                    /usr/local or /usr if not found in /usr/local]
-  --without-zlib to disable zlib usage completely],
-[if test "$withval" != no ; then
-  AC_MSG_RESULT(yes)
-  ZLIB_HOME="$withval"
-else
-  AC_MSG_RESULT(no)
-fi], [
-AC_MSG_RESULT(yes)
-ZLIB_HOME=/usr/local
-if test ! -f "${ZLIB_HOME}/include/zlib.h"
-then
-        ZLIB_HOME=/usr
+AC_DEFUN([CHECK_ZLIB], [
+AC_ARG_WITH([zlib],
+	[AS_HELP_STRING([--with-zlib=DIR],[root directory of the zlib installation])],
+	[zlib_want=$withval], [zlib_want=yes])
+if test "$zlib_want" = "no"; then
+	AC_MSG_ERROR([zlib cannot be disabled: the cache and the WARC output are zip/gzip containers, and the bundled minizip calls zlib directly])
 fi
-])
-
-#
-# Locate zlib, if wanted
-#
-if test -n "${ZLIB_HOME}"
-then
-        ZLIB_OLD_LDFLAGS=$LDFLAGS
-        ZLIB_OLD_CPPFLAGS=$LDFLAGS
-        LDFLAGS="$LDFLAGS -L${ZLIB_HOME}/lib"
-        CPPFLAGS="$CPPFLAGS -I${ZLIB_HOME}/include"
-        AC_LANG_SAVE
-        AC_LANG_C
-        AC_CHECK_LIB(z, inflateEnd, [zlib_cv_libz=yes], [zlib_cv_libz=no])
-        AC_CHECK_HEADER(zlib.h, [zlib_cv_zlib_h=yes], [zlib_cv_zlib_h=no])
-        AC_LANG_RESTORE
-        if test "$zlib_cv_libz" = "yes" -a "$zlib_cv_zlib_h" = "yes"
-        then
-                #
-                # If both library and header were found, use them
-                #
-                AC_CHECK_LIB(z, inflateEnd)
-                AC_MSG_CHECKING(zlib in ${ZLIB_HOME})
-                AC_MSG_RESULT(ok)
-        else
-                #
-                # If either header or library was not found, revert and bomb
-                #
-                AC_MSG_CHECKING(zlib in ${ZLIB_HOME})
-                LDFLAGS="$ZLIB_OLD_LDFLAGS"
-                CPPFLAGS="$ZLIB_OLD_CPPFLAGS"
-                AC_MSG_RESULT(failed)
-                AC_MSG_ERROR(either specify a valid zlib installation with --with-zlib=DIR or disable zlib usage with --without-zlib)
-        fi
+if test "$zlib_want" != "yes"; then
+	# An explicit prefix is authoritative: if the header is not under it,
+	# error rather than silently pick a system copy.
+	if test ! -f "$zlib_want/include/zlib.h"; then
+		AC_MSG_ERROR([zlib requested at $zlib_want but $zlib_want/include/zlib.h is missing])
+	fi
+	CPPFLAGS="$CPPFLAGS -I$zlib_want/include"
+	LDFLAGS="$LDFLAGS -L$zlib_want/lib"
+elif test -f /usr/local/include/zlib.h; then
+	# Where the BSD ports tree lands zlib, and not always searched by default.
+	CPPFLAGS="$CPPFLAGS -I/usr/local/include"
+	LDFLAGS="$LDFLAGS -L/usr/local/lib"
 fi
-
+AC_CHECK_HEADER([zlib.h], [],
+	[AC_MSG_ERROR([zlib.h not found; install the zlib development files or pass --with-zlib=DIR])])
+AC_CHECK_LIB([z], [inflateEnd], [],
+	[AC_MSG_ERROR([libz not found; install the zlib development files or pass --with-zlib=DIR])])
 ])

man/httrack.1

@@ -3,7 +3,7 @@
 .\"
 .\" This file is generated by man/makeman.sh; do not edit by hand.
 .\" SPDX-License-Identifier: GPL-3.0-or-later
-.TH httrack 1 "23 July 2026" "httrack website copier"
+.TH httrack 1 "26 July 2026" "httrack website copier"
 .SH NAME
 httrack \- offline browser : copy websites to a local directory
 .SH SYNOPSIS
@@ -36,10 +36,12 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-t, \-\-test\fR ]
 [ \fB\-%L, \-\-list\fR ]
 [ \fB\-%S, \-\-urllist\fR ]
+[ \fB\-%m, \-\-sitemap\fR ]
 [ \fB\-NN, \-\-structure[=N]\fR ]
 [ \fB\-%N, \-\-delayed\-type\-check\fR ]
 [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
 [ \fB\-%M, \-\-mime\-html\fR ]
+[ \fB\-%Z, \-\-single\-file\fR ]
 [ \fB\-LN, \-\-long\-names[=N]\fR ]
 [ \fB\-KN, \-\-keep\-links[=N]\fR ]
 [ \fB\-x, \-\-replace\-external\fR ]
@@ -75,6 +77,7 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-C, \-\-cache[=N]\fR ]
 [ \fB\-k, \-\-store\-all\-in\-cache\fR ]
 [ \fB\-%r, \-\-warc\fR ]
+[ \fB\-%d, \-\-changes\fR ]
 [ \fB\-%n, \-\-do\-not\-recatch\fR ]
 [ \fB\-%v, \-\-display\fR ]
 [ \fB\-Q, \-\-do\-not\-log\fR ]
@@ -187,6 +190,8 @@ test all URLs (even forbidden ones) (\-\-test)
 <file> add all URL located in this text file (one URL per line) (\-\-list <param>)
 .IP \-%S
 <file> add all scan rules located in this text file (one scan rule per line) (\-\-urllist <param>)
+.IP \-%m
+seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (\-\-sitemap)
 .SS Build options:
 .IP \-NN
 structure type (0 *original structure, 1+: see below) (\-\-structure[=N])
@@ -198,6 +203,8 @@ delayed type check, don't make any link test but wait for files download to star
 cached delayed type check, don't wait for remote type during updates, to speedup them (%D0 wait, * %D1 don't wait) (\-\-cached\-delayed\-type\-check)
 .IP \-%M
 generate a RFC MIME\-encapsulated full\-archive (.mht) (\-\-mime\-html)
+.IP \-%Z
+after the mirror, rewrite each saved page with its stylesheets, scripts, images and fonts inlined as data: URIs, so any page opens by double\-click anywhere (links between pages stay relative; audio and video stay links); \-\-single\-file\-max\-size N caps each asset (default 10485760 bytes). %M is the better container where a Chromium\-family browser is a given: one archive, no base64 tax on text, a shared asset stored once (\-\-single\-file)
 .IP \-%t
 keep the original file extension, don't rewrite it from the MIME type (%t0 rewrite)
 .IP \-LN
@@ -279,6 +286,8 @@ create/use a cache for updates and retries (C0 no cache,C1 cache is prioritary,*
 store all files in cache (not useful if files on disk) (\-\-store\-all\-in\-cache)
 .IP \-%r
 write an ISO\-28500 WARC/1.1 archive; \-\-warc\-file NAME sets the output name, \-\-warc\-max\-size N rotates segments past N bytes, \-\-warc\-cdx also writes a sorted CDXJ index, \-\-wacz packages it all as a WACZ file (\-\-warc)
+.IP \-%d
+write hts\-changes.json listing what this crawl left new, changed, unchanged and gone compared to the previous mirror (\-\-changes)
 .IP \-%n
 do not re\-download locally erased files (\-\-do\-not\-recatch)
 .IP \-%v

src/Makefile.am

@@ -48,7 +48,7 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
 
 lib_LTLIBRARIES = libhttrack.la
 
-htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
+htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
 	htscmdline.c htscmdline.h \
 	htsurlport.c htsurlport.h
 proxytrack_SOURCES = proxy/main.c \
@@ -66,7 +66,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htschanges.c htssinglefile.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
@@ -77,7 +77,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htschanges.h htssinglefile.h htssitemap.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \
@@ -87,7 +87,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
 libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
 libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
 
-EXTRA_DIST = httrack.h webhttrack \
+EXTRA_DIST = httrack.h htsstats.h webhttrack \
 		version.rc \
 		libhttrack.rc \
 		httrack.rc \

src/htsalias.c

@@ -114,6 +114,12 @@ const char *hts_optalias[][4] = {
    "strip [host/pattern=]key1,key2,... from URLs"},
   {"cookies-file", "-%K", "param1",
    "load extra cookies from a Netscape cookies.txt"},
+  {"changes", "-%d", "single",
+   "write hts-changes.json: what this crawl changed vs. the previous mirror"},
+  {"sitemap", "-%m", "single",
+   "seed the crawl from the start host's sitemap (robots.txt, then "
+   "/sitemap.xml)"},
+  {"sitemap-url", "-%mu", "param1", "seed the crawl from this sitemap URL"},
   {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
   {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
   {"warc-max-size", "-%rs", "param1",
@@ -123,6 +129,10 @@ const char *hts_optalias[][4] = {
   {"warc-cdxj", "-%rc", "single", ""},
   {"wacz", "-%rz", "single",
    "package the WARC archive, CDXJ index and pages as a WACZ file"},
+  {"single-file", "-%Z", "single",
+   "after the mirror, inline each page's assets as data: URIs"},
+  {"single-file-max-size", "-%Zs", "param1",
+   "per-asset cap for --single-file, in bytes (implies it; default 10485760)"},
   {"why", "-%Y", "param1",
    "explain which filter rule accepts or rejects a URL, then exit"},
   {"pause", "-%G", "param1",

src/htsback.c

@@ -38,6 +38,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsnet.h"
 #include "htscore.h"
 #include "htswarc.h"
+#include "htschanges.h"
 #include "htsthread.h"
 #include <time.h>
 /* END specific definitions */
@@ -47,11 +48,6 @@ Please visit our Website: http://www.httrack.com
 #include "htsftp.h"
 #include "htscodec.h"
 #include "htsproxy.h"
-#if HTS_USEZLIB
-#include "htszlib.h"
-#else
-#error HTS_USEZLIB not defined
-#endif
 
 #ifdef _WIN32
 #ifndef __cplusplus
@@ -728,6 +724,13 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
                 if ((size = hts_codec_unpack(codec, back[p].tmpfile,
                                              unpacked)) >= 0) {
                   back[p].r.size = back[p].r.totalsize = size;
+                  if (back[p].r.is_write) {
+                    /* Sample the previous copy now: the rename below replaces
+                       it, and file_notify() only fires once it is gone. */
+                    hts_changes_notify(
+                        opt, back[p].url_adr, back[p].url_fil, back[p].url_sav,
+                        HTS_TRUE, back[p].r.notmodified ? HTS_TRUE : HTS_FALSE);
+                  }
                   if (!back[p].r.is_write) {
                     // fichier -> mémoire ; le fichier est écrit plus tard
                     deleteaddr(&back[p].r);

src/htscache.c

@@ -142,10 +142,12 @@ struct cache_back_zip_entry {
   int compressionMethod;
 };
 
+/* A corrupt cache can carry a field wider than ours; clipping it keeps the
+   entry, where aborting would take the crawl down. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      strlcpybuff(refvalue, value, refvalue_size);                             \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -1295,12 +1297,17 @@ char *readfile2(const char *fil, LLint * size) {
 }
 
 /* Note: utf-8 */
-char *readfile_utf8(const char *fil) {
+char *readfile_utf8(const char *fil) { return readfile2_utf8(fil, NULL); }
+
+/* Note: utf-8 */
+char *readfile2_utf8(const char *fil, LLint *size) {
   char *adr = NULL;
   char catbuff[CATBUFF_SIZE];
   const LLint len = fsize_utf8(fil);
   const size_t buflen = len >= 0 ? llint_to_size_t(len) : (size_t) -1;
 
+  if (size != NULL)
+    *size = len;
   if (buflen != (size_t) -1) { // exists, and is addressable (see readfile2)
     FILE *const fp = FOPEN(fconv(catbuff, sizeof(catbuff), fil), "rb");
 

src/htscache_selftest.c

@@ -1195,6 +1195,12 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
 
 /* --- read-side corruption injection --------------------------------------- */
 
+/* 100 'A's: a placeholder header line long enough to be overwritten by a
+   forged, over-long X-StatusMessage of the same byte length. */
+#define CORRUPT_LONG_ETAG                                                      \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"                         \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+
 /* canary read back intact after each corruption; victim gets the byte surgery
  */
 #define CORRUPT_ADR "corrupt.example.com"
@@ -1219,6 +1225,24 @@ static void corrupt_build(httrackp *opt) {
   selftest_close(&cache);
 }
 
+/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
+static void corrupt_build_longetag(httrackp *opt) {
+  cache_back cache;
+
+  memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
+  memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
+  remove(reconcile_st_path(opt, "hts-cache/new.zip"));
+  remove(reconcile_st_path(opt, "hts-cache/old.zip"));
+  selftest_open_for_write(&cache, opt);
+  store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
+              "OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
+              strlen(corrupt_body_a));
+  store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
+              "OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
+              corrupt_body_b, strlen(corrupt_body_b));
+  selftest_close(&cache);
+}
+
 /* Like corrupt_build, but the victim carries a 20-char Etag whose header line
    is later overwritten with a forged oversized X-Size (same byte length). */
 static void corrupt_build_etag(httrackp *opt) {
@@ -1403,6 +1427,48 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
   return fail;
 }
 
+/* An over-long field from a foreign cache must clip, not abort: the entry
+   still reads, clipped to capacity, and the canary survives. */
+static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
+                                         size_t wantlastmod, const char *what) {
+  cache_back cache;
+  htsblk v, c;
+  char BIGSTK lv[HTS_URLMAXSIZE * 2];
+  char BIGSTK lc[HTS_URLMAXSIZE * 2];
+  int fail = 0;
+
+  selftest_open_for_read(&cache, opt);
+  lv[0] = lc[0] = '\0';
+  v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
+  if (v.statuscode != 200) {
+    fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
+            v.statuscode);
+    fail++;
+  }
+  if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
+    fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
+            (unsigned) strlen(v.msg), (unsigned) wantmsg);
+    fail++;
+  }
+  if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
+    fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
+            what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
+    fail++;
+  }
+  c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
+  if (c.statuscode != 200) {
+    fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
+            c.statuscode);
+    fail++;
+  }
+  if (v.adr != NULL)
+    freet(v.adr);
+  if (c.adr != NULL)
+    freet(c.adr);
+  selftest_close(&cache);
+  return fail;
+}
+
 /* One zip corruption case: build, patch, then check victim+canary in-session.
  */
 static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1439,6 +1505,31 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
   failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
                                     "garbled deflate stream");
 
+  /* A corrupt cache can hold a field wider than ours. Clipping keeps the
+     entry; aborting would take the crawl down. Overwrite the placeholder Etag
+     line in place, same byte length, so the zip offsets stay intact. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "X-StatusMessage: " /* 17 + 89 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures +=
+      corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
+                                    (size_t) -1, "over-long X-StatusMessage");
+
+  /* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
+     cannot satisfy both. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "Last-Modified: " /* 15 + 91 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures += corrupt_expect_victim_clipped(
+      opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
+      "over-long Last-Modified");
+
   /* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
      truncates negative in the (int) cast the malloc uses: a wraparound alloc.
      cache_add asserts size fits an int, so such a value only reaches the reader

src/htschanges.c

@@ -0,0 +1,679 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: htschanges.c subroutines:                              */
+/*       --changes: what this crawl changed vs. the previous    */
+/*       mirror (hts-changes.json)                              */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#define HTS_INTERNAL_BYTECODE
+
+#include "htschanges.h"
+
+#include "htscache.h"
+#include "htscharset.h"
+#include "htscore.h"
+#include "htslib.h"
+#include "htsmd5.h"
+#include "htssafe.h"
+#include "htsthread.h"
+#include "htstools.h"
+#include "coucal/coucal.h"
+#include "md5.h"
+
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+
+#define DIGEST_SIZE 16
+
+typedef struct hts_changes hts_changes;
+
+/* One mirrored file. The mirror-relative path is the key, not the URL: a
+   redirect and its target can share a save name, and counting that file twice
+   would put it in two buckets. */
+typedef struct {
+  char *url;               /* absolute URL, "" for an engine-generated file */
+  char *file;              /* mirror-relative path, as listed in new.lst */
+  hts_boolean rewritten;   /* the crawl wrote over the local copy */
+  hts_boolean not_updated; /* transfer signal: the server reported no change */
+  hts_boolean existed;     /* the file was part of the previous mirror */
+  hts_boolean on_disk;     /* a copy was on disk when the crawl first saw it */
+  hts_boolean listed_prev; /* the previous mirror's index listed this file */
+  hts_boolean has_prev;    /* prev_digest holds the previous payload's digest */
+  hts_boolean has_new;     /* new_digest was taken from the payload, not disk */
+  unsigned char prev_digest[DIGEST_SIZE];
+  unsigned char new_digest[DIGEST_SIZE];
+  LLint prev_size; /* -1 when unknown */
+  LLint size;      /* size after the crawl, -1 when unknown */
+  hts_change_bucket bucket;
+} changes_entry;
+
+struct hts_changes {
+  coucal index; /* mirror-relative path -> entry slot + 1 */
+  changes_entry *entries;
+  size_t count;
+  size_t capacity;
+  hts_boolean has_index; /* the run keeps a mirror index (the cache is on) */
+  hts_boolean old_index; /* the previous mirror's index was read */
+  hts_boolean overflow;  /* an allocation failed; the report is partial */
+  hts_boolean closed;    /* reported: late notifies are dropped, not recorded */
+};
+
+/* ------------------------------------------------------------ */
+/* JSON                                                          */
+/* ------------------------------------------------------------ */
+
+/* Length of the UTF-8 sequence led by c, or 0 if c cannot lead one. */
+static size_t utf8_lead_length(unsigned char c) {
+  if (c >= 0xc2 && c <= 0xdf)
+    return 2;
+  if (c >= 0xe0 && c <= 0xef)
+    return 3;
+  if (c >= 0xf0 && c <= 0xf4)
+    return 4;
+  return 0;
+}
+
+void hts_changes_json_string(String *out, const char *s) {
+  const unsigned char *p = (const unsigned char *) s;
+
+  StringAddchar(*out, '"');
+  while (*p != '\0') {
+    const unsigned char c = *p;
+
+    if (c == '"' || c == '\\') {
+      StringAddchar(*out, '\\');
+      StringAddchar(*out, (char) c);
+      p++;
+    } else if (c < 0x20 || c == 0x7f) {
+      char esc[8];
+
+      snprintf(esc, sizeof(esc), "\\u%04x", (unsigned) c);
+      StringCat(*out, esc);
+      p++;
+    } else if (c < 0x80) {
+      StringAddchar(*out, (char) c);
+      p++;
+    } else {
+      /* Non-ASCII: emit it verbatim only if it is a well-formed sequence, so
+         a legacy-charset URL cannot produce unparseable JSON. */
+      const size_t len = utf8_lead_length(c);
+
+      if (len != 0 && strnlen((const char *) p, len) == len &&
+          hts_isStringUTF8((const char *) p, len)) {
+        StringMemcat(*out, (const char *) p, len);
+        p += len;
+      } else {
+        StringCat(*out, "\\ufffd");
+        p++;
+      }
+    }
+  }
+  StringAddchar(*out, '"');
+}
+
+/* ------------------------------------------------------------ */
+/* Accumulator                                                   */
+/* ------------------------------------------------------------ */
+
+static hts_changes *changes_new(void) {
+  hts_changes *changes = calloct(1, sizeof(*changes));
+
+  if (changes == NULL)
+    return NULL;
+  changes->index = coucal_new(0);
+  if (changes->index == NULL) {
+    freet(changes);
+    return NULL;
+  }
+  return changes;
+}
+
+static void changes_free(hts_changes **pchanges) {
+  hts_changes *changes = *pchanges;
+
+  if (changes == NULL)
+    return;
+  if (changes->entries != NULL) {
+    size_t i;
+
+    for (i = 0; i < changes->count; i++) {
+      freet(changes->entries[i].url);
+      freet(changes->entries[i].file);
+    }
+    freet(changes->entries);
+  }
+  if (changes->index != NULL)
+    coucal_delete(&changes->index);
+  freet(changes);
+  *pchanges = NULL;
+}
+
+/* Slot of `file`, or -1 when it has not been recorded. */
+static intptr_t changes_find(const hts_changes *changes, const char *file) {
+  intptr_t slot = 0;
+
+  return coucal_read(changes->index, file, &slot) ? slot - 1 : -1;
+}
+
+/* Slot of `file`, appending a fresh entry when it is new; -1 if allocation
+   failed, which flags the report partial. */
+static intptr_t changes_slot(hts_changes *changes, const char *file,
+                             hts_boolean *is_new) {
+  intptr_t slot = 0;
+
+  *is_new = HTS_FALSE;
+  if (coucal_read(changes->index, file, &slot))
+    return slot - 1;
+
+  if (changes->count == changes->capacity) {
+    const size_t capacity = changes->capacity != 0 ? changes->capacity * 2 : 64;
+    changes_entry *const entries =
+        realloct(changes->entries, capacity * sizeof(*entries));
+
+    if (entries == NULL) {
+      changes->overflow = HTS_TRUE;
+      return -1;
+    }
+    changes->entries = entries;
+    changes->capacity = capacity;
+  }
+  slot = (intptr_t) changes->count;
+  memset(&changes->entries[slot], 0, sizeof(changes->entries[slot]));
+  changes->entries[slot].file = strdupt(file);
+  changes->entries[slot].prev_size = -1;
+  changes->entries[slot].size = -1;
+  if (changes->entries[slot].file == NULL) {
+    changes->overflow = HTS_TRUE;
+    return -1;
+  }
+  changes->count++;
+  coucal_write(changes->index, file, slot + 1);
+  *is_new = HTS_TRUE;
+  return slot;
+}
+
+/* MD5 of the file at `path`; not a security boundary. Never call it holding
+   changes_mutex: hashing a large file would stall every other connection.
+   HTS_FALSE if it cannot be read. */
+static hts_boolean digest_file(const char *path,
+                               unsigned char digest[DIGEST_SIZE]) {
+  const int endian = 1;
+  struct MD5Context ctx;
+  char BIGSTK buffer[32768];
+  FILE *fp = FOPEN(path, "rb");
+  size_t nread;
+
+  if (fp == NULL)
+    return HTS_FALSE;
+  MD5Init(&ctx, *((const char *) &endian));
+  while ((nread = fread(buffer, 1, sizeof(buffer), fp)) > 0)
+    MD5Update(&ctx, (const unsigned char *) buffer, (unsigned int) nread);
+  if (ferror(fp) != 0) {
+    fclose(fp);
+    return HTS_FALSE;
+  }
+  fclose(fp);
+  MD5Final(digest, &ctx);
+  return HTS_TRUE;
+}
+
+static void digest_mem(const char *buffer, size_t len,
+                       unsigned char digest[DIGEST_SIZE]) {
+  domd5mem(buffer, len, (char *) digest, 0);
+}
+
+hts_change_bucket hts_changes_classify(hts_boolean rewritten,
+                                       hts_boolean existed,
+                                       hts_boolean not_updated,
+                                       hts_boolean have_digests,
+                                       hts_boolean digests_equal) {
+  if (!rewritten)
+    return HTS_CHANGE_UNCHANGED; /* the crawl left the copy alone */
+  if (!existed)
+    return HTS_CHANGE_NEW;
+  if (have_digests)
+    return digests_equal ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED;
+  /* No digest to compare: a server with no validators answers 200 with the
+     same bytes, so this signal alone over-reports. */
+  return not_updated ? HTS_CHANGE_UNCHANGED : HTS_CHANGE_CHANGED;
+}
+
+/* ------------------------------------------------------------ */
+/* Engine hooks                                                  */
+/* ------------------------------------------------------------ */
+
+/* FTP transfers reach file_notify() from a thread the crawl never joins, so
+   every entry point below, report and teardown included, takes this lock. */
+static htsmutex changes_mutex = HTSMUTEX_INIT;
+
+/* The live accumulator, created on first use; NULL when --changes is off or
+   the report is already written. Call under the lock. */
+static hts_changes *changes_get(httrackp *opt) {
+  hts_changes *changes = (hts_changes *) opt->changes_state;
+
+  if (!opt->changes)
+    return NULL;
+  if (changes == NULL) {
+    changes = changes_new();
+    opt->changes_state = changes;
+  }
+  return changes != NULL && !changes->closed ? changes : NULL;
+}
+
+void hts_changes_notify(httrackp *opt, const char *adr, const char *fil,
+                        const char *save, hts_boolean rewritten,
+                        hts_boolean not_updated) {
+  hts_changes *changes;
+  char BIGSTK file[HTS_URLMAXSIZE * 2];
+  char BIGSTK url[HTS_URLMAXSIZE * 4 + 8]; /* holds adr + fil + a scheme */
+  unsigned char prev_digest[DIGEST_SIZE];
+  hts_boolean has_prev = HTS_FALSE;
+  hts_boolean is_new = HTS_FALSE;
+  intptr_t slot = -1;
+  LLint prev_size;
+
+  if (!opt->changes)
+    return;
+  /* Engine-generated scaffolding (the top index) carries no URL and is not a
+     mirrored resource. */
+  if (save == NULL || !strnotempty(save) || adr == NULL || fil == NULL ||
+      (!strnotempty(adr) && !strnotempty(fil)))
+    return;
+
+  hts_savename_listed(StringBuff(opt->path_html), save, file, sizeof(file));
+
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL) {
+    slot = changes_slot(changes, file, &is_new);
+    if (slot >= 0 && !is_new) {
+      /* Retry, or a second call site for the same file: the pre-run state was
+         already sampled and the copy on disk may no longer be it. */
+      changes->entries[slot].rewritten =
+          changes->entries[slot].rewritten || rewritten;
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+  if (slot < 0 || !is_new)
+    return;
+
+  url[0] = '\0';
+  if (!link_has_authority(adr))
+    strlcatbuff(url, "http://", sizeof(url));
+  strlcatbuff(url, adr, sizeof(url));
+  strlcatbuff(url, fil, sizeof(url));
+
+  /* Unlocked: hashing a large file would stall every other connection. Hash
+     it only when it is about to be overwritten, the last moment it exists. */
+  prev_size = fsize_utf8(save);
+  if (prev_size >= 0 && rewritten)
+    has_prev = digest_file(save, prev_digest);
+
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  /* The slot index is stable, entries being append-only; the array is not. */
+  if (changes != NULL && (size_t) slot < changes->count) {
+    changes_entry *const entry = &changes->entries[slot];
+
+    entry->url = strdupt(url);
+    if (entry->url == NULL)
+      changes->overflow = HTS_TRUE;
+    entry->rewritten = entry->rewritten || rewritten;
+    entry->not_updated = not_updated;
+    entry->prev_size = prev_size;
+    entry->on_disk = prev_size >= 0;
+    /* hts_changes_html() compares payloads, and its digests win. */
+    if (has_prev && !entry->has_new) {
+      entry->has_prev = HTS_TRUE;
+      memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE);
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r,
+                      const char *adr, const char *fil, const char *save) {
+  hts_changes *changes;
+  char BIGSTK file[HTS_URLMAXSIZE * 2];
+  char BIGSTK location[HTS_URLMAXSIZE * 2];
+  unsigned char new_digest[DIGEST_SIZE];
+  unsigned char prev_digest[DIGEST_SIZE];
+  hts_boolean has_prev = HTS_FALSE;
+  intptr_t slot;
+  htsblk prev;
+
+  /* Ahead of the digest and the cache read: off must cost nothing. */
+  if (!opt->changes)
+    return;
+  if (r->adr == NULL || r->size < 0 || save == NULL || !strnotempty(save))
+    return;
+
+  /* On disk this is the payload plus rewritten links and a footer dated by the
+     crawl, so it differs every run; compare payloads, the previous one being
+     the body the cache kept. */
+  digest_mem(r->adr, (size_t) r->size, new_digest);
+  prev = cache_read_ro(opt, cache, adr, fil, "", location);
+  if (HTTP_IS_OK(prev.statuscode) && prev.adr != NULL && prev.size >= 0) {
+    digest_mem(prev.adr, (size_t) prev.size, prev_digest);
+    has_prev = HTS_TRUE;
+  }
+  freet(prev.adr);
+
+  /* Only now take the lock and look the entry up: cache_read_ro() can itself
+     reach file_notify(), which would move the entries array. */
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  /* file_notify() records the entry; this call only refines it. */
+  if (changes != NULL) {
+    hts_savename_listed(StringBuff(opt->path_html), save, file, sizeof(file));
+    slot = changes_find(changes, file);
+    if (slot >= 0) {
+      changes_entry *const entry = &changes->entries[slot];
+
+      entry->has_new = HTS_TRUE;
+      memcpy(entry->new_digest, new_digest, DIGEST_SIZE);
+      entry->has_prev = has_prev;
+      if (has_prev)
+        memcpy(entry->prev_digest, prev_digest, DIGEST_SIZE);
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept) {
+  hts_changes *changes;
+  hts_boolean is_new;
+  intptr_t slot;
+
+  if (!opt->changes || file == NULL || !strnotempty(file))
+    return;
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL) {
+    slot = changes_slot(changes, file, &is_new);
+    if (slot >= 0 && is_new) {
+      changes->entries[slot].url = strdupt("");
+      changes->entries[slot].listed_prev = HTS_TRUE;
+      /* A kept file leaves rewritten clear, which resolves to unchanged. */
+      if (!kept)
+        changes->entries[slot].bucket = HTS_CHANGE_GONE;
+    }
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_previous(httrackp *opt, const char *file) {
+  hts_changes *changes;
+  intptr_t slot;
+
+  if (!opt->changes || file == NULL)
+    return;
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL) {
+    changes->has_index = HTS_TRUE;
+    changes->old_index = HTS_TRUE;
+    slot = changes_find(changes, file);
+    if (slot >= 0)
+      changes->entries[slot].listed_prev = HTS_TRUE;
+  }
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_indexed(httrackp *opt) {
+  hts_changes *changes;
+
+  if (!opt->changes)
+    return;
+  hts_mutexlock(&changes_mutex);
+  changes = changes_get(opt);
+  if (changes != NULL)
+    changes->has_index = HTS_TRUE;
+  hts_mutexrelease(&changes_mutex);
+}
+
+/* ------------------------------------------------------------ */
+/* Report                                                        */
+/* ------------------------------------------------------------ */
+
+/* Assign each entry its final bucket from the bytes now on disk. */
+static void changes_resolve(hts_changes *changes, httrackp *opt) {
+  char catbuff[CATBUFF_SIZE];
+  size_t i;
+
+  for (i = 0; i < changes->count; i++) {
+    changes_entry *const entry = &changes->entries[i];
+    const char *path;
+    unsigned char digest[DIGEST_SIZE];
+    hts_boolean have_digests = HTS_FALSE;
+    hts_boolean digests_equal = HTS_FALSE;
+
+    if (entry->bucket == HTS_CHANGE_GONE)
+      continue;
+
+    /* The previous mirror's index is the authority on what was there before:
+       a partial left by this crawl's own failed attempt is on disk but was
+       never part of the previous mirror. Without an index, fall back to what
+       the first notify saw on disk. */
+    entry->existed = changes->old_index ? entry->listed_prev : entry->on_disk;
+    path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_html),
+                   entry->file);
+    entry->size = fsize_utf8(path);
+    if (entry->rewritten && entry->existed) {
+      /* The size shortcut only holds when both digests describe the file on
+         disk; has_new means they are payload digests, and a parsed page's
+         rendered size moves with its links and footer. */
+      if (!entry->has_new && entry->size >= 0 && entry->prev_size >= 0 &&
+          entry->size != entry->prev_size) {
+        have_digests = HTS_TRUE; /* different lengths: no need to hash */
+        digests_equal = HTS_FALSE;
+      } else if (entry->has_prev &&
+                 (entry->has_new
+                      ? (memcpy(digest, entry->new_digest, DIGEST_SIZE), 1)
+                      : digest_file(path, digest))) {
+        have_digests = HTS_TRUE;
+        digests_equal = memcmp(digest, entry->prev_digest, DIGEST_SIZE) == 0
+                            ? HTS_TRUE
+                            : HTS_FALSE;
+      }
+    }
+    entry->bucket =
+        hts_changes_classify(entry->rewritten, entry->existed,
+                             entry->not_updated, have_digests, digests_equal);
+  }
+}
+
+static const char *const bucket_names[HTS_CHANGE_BUCKETS] = {
+    "new", "changed", "unchanged", "gone"};
+
+/* Serialize the report. Call under changes_mutex. */
+static void changes_serialize(httrackp *opt, String *out) {
+  hts_changes *const changes = (hts_changes *) opt->changes_state;
+  size_t counts[HTS_CHANGE_BUCKETS];
+  char date[32];
+  char scratch[64];
+  int bucket;
+  size_t i;
+
+  StringClear(*out);
+  if (changes == NULL)
+    return;
+  changes_resolve(changes, opt);
+
+  memset(counts, 0, sizeof(counts));
+  for (i = 0; i < changes->count; i++)
+    counts[changes->entries[i].bucket]++;
+
+  hts_now_iso8601(date);
+  StringCat(*out, "{\n  \"schema\": ");
+  snprintf(scratch, sizeof(scratch), "%d", HTS_CHANGES_SCHEMA);
+  StringCat(*out, scratch);
+  StringCat(*out, ",\n  \"generator\": ");
+  hts_changes_json_string(out, "HTTrack Website Copier/" HTTRACK_VERSION);
+  StringCat(*out, ",\n  \"date\": ");
+  hts_changes_json_string(out, date);
+  StringCat(*out, ",\n  \"first_crawl\": ");
+  StringCat(*out, !changes->has_index
+                      ? "null"
+                      : (changes->old_index ? "false" : "true"));
+  StringCat(*out, ",\n  \"partial\": ");
+  StringCat(*out, changes->overflow ? "true" : "false");
+  StringCat(*out, ",\n  \"purged\": ");
+  StringCat(*out, opt->delete_old ? "true" : "false");
+
+  StringCat(*out, ",\n  \"counts\": {");
+  for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) {
+    StringCat(*out, bucket != 0 ? ", " : " ");
+    hts_changes_json_string(out, bucket_names[bucket]);
+    snprintf(scratch, sizeof(scratch), ": %d", (int) counts[bucket]);
+    StringCat(*out, scratch);
+  }
+  StringCat(*out, " }");
+
+  for (bucket = 0; bucket < HTS_CHANGE_BUCKETS; bucket++) {
+    hts_boolean first = HTS_TRUE;
+
+    StringCat(*out, ",\n  ");
+    hts_changes_json_string(out, bucket_names[bucket]);
+    StringCat(*out, ": [");
+    for (i = 0; i < changes->count; i++) {
+      const changes_entry *const entry = &changes->entries[i];
+
+      if (entry->bucket != bucket)
+        continue;
+      StringCat(*out, first ? "\n    { \"url\": " : ",\n    { \"url\": ");
+      first = HTS_FALSE;
+      hts_changes_json_string(out, entry->url != NULL ? entry->url : "");
+      StringCat(*out, ", \"file\": ");
+      hts_changes_json_string(out, entry->file);
+      if (entry->size >= 0) {
+        snprintf(scratch, sizeof(scratch), ", \"size\": " LLintP,
+                 (LLint) entry->size);
+        StringCat(*out, scratch);
+      }
+      if (bucket == HTS_CHANGE_CHANGED && entry->prev_size >= 0) {
+        snprintf(scratch, sizeof(scratch), ", \"previous_size\": " LLintP,
+                 (LLint) entry->prev_size);
+        StringCat(*out, scratch);
+      }
+      StringCat(*out, " }");
+    }
+    StringCat(*out, first ? "]" : "\n  ]");
+  }
+  StringCat(*out, "\n}\n");
+}
+
+void hts_changes_report(httrackp *opt, String *out) {
+  hts_mutexlock(&changes_mutex);
+  changes_serialize(opt, out);
+  hts_mutexrelease(&changes_mutex);
+}
+
+void hts_changes_close_opt(httrackp *opt) {
+  char catbuff[CATBUFF_SIZE];
+  const char *path;
+  String report = STRING_EMPTY;
+  size_t counts[HTS_CHANGE_BUCKETS];
+  size_t total;
+  hts_boolean has_index, old_index;
+  FILE *fp;
+
+  hts_mutexlock(&changes_mutex);
+  {
+    /* changes_get(), not the raw field: a crawl that mirrored nothing still
+       owes the user a report, and a stale one from the previous run must not
+       survive on disk as if it described this one. */
+    hts_changes *const changes = changes_get(opt);
+    size_t i;
+
+    if (changes == NULL) {
+      hts_mutexrelease(&changes_mutex);
+      return;
+    }
+    changes_serialize(opt, &report);
+    memset(counts, 0, sizeof(counts));
+    for (i = 0; i < changes->count; i++)
+      counts[changes->entries[i].bucket]++;
+    total = changes->count;
+    has_index = changes->has_index;
+    old_index = changes->old_index;
+    /* Sticky: whatever the crawl's stragglers do next is not in this report. */
+    changes->closed = HTS_TRUE;
+  }
+  hts_mutexrelease(&changes_mutex);
+
+  path = fconcat(catbuff, sizeof(catbuff), StringBuff(opt->path_log),
+                 HTS_CHANGES_FILE);
+  fp = FOPEN(path, "wb");
+  if (fp != NULL) {
+    const size_t len = StringLength(report);
+
+    if (len != 0 && fwrite(StringBuff(report), 1, len, fp) != len)
+      hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
+                    "Unable to write the change report %s", path);
+    fclose(fp);
+  } else {
+    hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
+                  "Unable to create the change report %s", path);
+  }
+
+  if (!has_index) {
+    hts_log_print(opt, LOG_NOTICE,
+                  "Change report: no mirror index (the cache is off), %d files "
+                  "mirrored, deletions not detected (%s)",
+                  (int) total, HTS_CHANGES_FILE);
+  } else if (!old_index) {
+    hts_log_print(opt, LOG_NOTICE,
+                  "Change report: first crawl, %d files mirrored, nothing to "
+                  "compare against (%s)",
+                  (int) total, HTS_CHANGES_FILE);
+  } else {
+    hts_log_print(opt, LOG_NOTICE,
+                  "Change report: %d new, %d changed, %d unchanged, %d gone "
+                  "(%s)",
+                  (int) counts[HTS_CHANGE_NEW],
+                  (int) counts[HTS_CHANGE_CHANGED],
+                  (int) counts[HTS_CHANGE_UNCHANGED],
+                  (int) counts[HTS_CHANGE_GONE], HTS_CHANGES_FILE);
+  }
+
+  StringFree(report);
+}
+
+void hts_changes_free_opt(httrackp *opt) {
+  hts_changes *changes;
+
+  hts_mutexlock(&changes_mutex);
+  changes = (hts_changes *) opt->changes_state;
+  changes_free(&changes);
+  opt->changes_state = NULL;
+  hts_mutexrelease(&changes_mutex);
+}

src/htschanges.h

@@ -0,0 +1,125 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack change report (--changes). Internal, not installed.
+   Accumulates what the crawl did to each mirrored file, compares the bytes
+   against the copy the previous run left behind, and writes
+   hts-changes.json next to the log. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_CHANGES_DEFH
+#define HTS_CHANGES_DEFH
+
+#include "htsopt.h"
+#include "htsstrings.h"
+
+#ifndef HTS_DEF_FWSTRUCT_cache_back
+#define HTS_DEF_FWSTRUCT_cache_back
+typedef struct cache_back cache_back;
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Report file name, written under the project's log directory. */
+#define HTS_CHANGES_FILE "hts-changes.json"
+
+/* Schema version carried by the report; bump on an incompatible change. */
+#define HTS_CHANGES_SCHEMA 1
+
+/* Which side of the comparison a mirrored file ended up on. */
+typedef enum {
+  HTS_CHANGE_NEW = 0,   /* no local copy before this run */
+  HTS_CHANGE_CHANGED,   /* rewritten, and the bytes differ */
+  HTS_CHANGE_UNCHANGED, /* the bytes are those of the previous mirror */
+  HTS_CHANGE_GONE,      /* in the previous mirror, absent from this one */
+  HTS_CHANGE_BUCKETS
+} hts_change_bucket;
+
+/* Record what this crawl is doing to the local file `save` (absolute path;
+   adr/fil form its URL, either may be empty for an engine-generated file).
+   `rewritten` means the copy on disk is being written over, `not_updated` is
+   the 200-versus-304 signal, used only when no digest can be taken. Only the
+   first call for a file samples the previous copy, so a retried or
+   twice-notified resource is counted once. No-op unless --changes is on. */
+void hts_changes_notify(httrackp *opt, const char *adr, const char *fil,
+                        const char *save, hts_boolean rewritten,
+                        hts_boolean not_updated);
+
+/* Refine the entry for a parsed HTML file: its local copy carries rewritten
+   links and a footer dated by the crawl, so it differs every run. Compares the
+   payload `r` holds against the body the previous run left in the cache. */
+void hts_changes_html(httrackp *opt, cache_back *cache, const htsblk *r,
+                      const char *adr, const char *fil, const char *save);
+
+/* Record `file` (mirror-relative, as listed in new.lst) as listed by the
+   previous mirror's index and absent from this run's. `kept` means its local
+   copy survives the run because the crawl tried and failed to replace it: a
+   failed transfer is not a deletion, so it is reported unchanged, not gone. */
+void hts_changes_dropped(httrackp *opt, const char *file, hts_boolean kept);
+
+/* Record that the previous mirror's index listed `file`. That index, not the
+   file's presence on disk, decides what counts as already mirrored. */
+void hts_changes_previous(httrackp *opt, const char *file);
+
+/* Record that this run keeps a mirror index. Without one (the cache is off)
+   deletions and first_crawl are undecidable, and the report says so. */
+void hts_changes_indexed(httrackp *opt);
+
+/* Resolve every entry against the bytes now on disk and serialize the report
+   into `out` (replaced). Exposed for the self-tests. */
+void hts_changes_report(httrackp *opt, String *out);
+
+/* Write the report and log a one-line summary. Idempotent, and seals the
+   accumulator: a late notify is dropped rather than starting a second one. */
+void hts_changes_close_opt(httrackp *opt);
+
+/* Drop the accumulator, for a run that never reached its end and to start the
+   next one clean. Null-safe and idempotent. */
+void hts_changes_free_opt(httrackp *opt);
+
+/* Bucket for one entry, from what was observed. No filesystem access:
+   `have_digests` says both sides could be hashed, `digests_equal` compares
+   them, and `not_updated` is the fallback signal when they could not. */
+hts_change_bucket hts_changes_classify(hts_boolean rewritten,
+                                       hts_boolean existed,
+                                       hts_boolean not_updated,
+                                       hts_boolean have_digests,
+                                       hts_boolean digests_equal);
+
+/* Append `s` to `out` as a quoted JSON string. Byte sequences that are not
+   valid UTF-8 become U+FFFD, so a mirror carrying legacy-charset URLs still
+   produces parseable JSON. */
+void hts_changes_json_string(String *out, const char *s);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/htscore.c

@@ -39,7 +39,10 @@ Please visit our Website: http://www.httrack.com
 
 /* File defs */
 #include "htscore.h"
+#include "htssitemap.h"
 #include "htswarc.h"
+#include "htschanges.h"
+#include "htssinglefile.h"
 
 /* specific definitions */
 #include "htsbase.h"
@@ -488,6 +491,7 @@ void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r,
                           const char *adr, const char *fil, const char *save) {
   {
     file_notify(opt, adr, fil, save, 1, 1, r->notmodified);
+    hts_changes_html(opt, cache, r, adr, fil, save);
     *fp = filecreate(&opt->state.strc, save);
     if (*fp) {
       if (ht_len > 0 && fwrite(ht_buff, 1, ht_len, *fp) != ht_len) {
@@ -691,6 +695,9 @@ int httpmirror(char *url1, httrackp * opt) {
   // hash table
   opt->hash = &hash;
 
+  // a change report left by a previous crawl on this opt is not this one's
+  hts_changes_free_opt(opt);
+
   // initialize link heap
   hts_record_init(opt);
 
@@ -943,6 +950,22 @@ int httpmirror(char *url1, httrackp * opt) {
     heap_top()->premier = heap_top_index();        // premier lien, objet-père=objet              
     heap_top()->precedent = heap_top_index();      // lien précédent
 
+    /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are
+       injected before the crawl gets far. */
+    hts_sitemap_free(opt); /* an earlier mirror may have left a doc list */
+    if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) {
+      char BIGSTK first[HTS_URLMAXSIZE * 2];
+      const char *const eol = strchr(primary, '\n');
+      const size_t len = eol != NULL ? (size_t) (eol - primary) : 0;
+
+      first[0] = '\0';
+      if (len > 0 && len < sizeof(first)) {
+        memcpy(first, primary, len);
+        first[len] = '\0';
+      }
+      hts_sitemap_seed(opt, first);
+    }
+
     // Initialiser cache
     {
       opt->state._hts_in_html_parsing = 4;
@@ -1587,11 +1610,21 @@ int httpmirror(char *url1, httrackp * opt) {
         stre.maketrack_fp = maketrack_fp;
 
         /* Parse */
-        if (hts_mirror_check_moved(&str, &stre) != 0) {
-          XH_uninit;
-          return -1;
-        }
+        {
+          const int nlinks = opt->lien_tot;
 
+          if (hts_mirror_check_moved(&str, &stre) != 0) {
+            XH_uninit;
+            return -1;
+          }
+          /* A redirect re-queues the target as a fresh link; without carrying
+             the marking over, a moved sitemap is fetched and then ignored. */
+          if (opt->sitemap_state != NULL && opt->lien_tot > nlinks &&
+              hts_sitemap_pending(opt, urladr(), urlfil())) {
+            hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr,
+                                 heap_top()->fil);
+          }
+        }
       }
 
     }                           // if !error
@@ -1609,6 +1642,29 @@ int httpmirror(char *url1, httrackp * opt) {
       /* Load file and decode if necessary, after redirect check. */
       LOAD_IN_MEMORY_IF_NECESSARY();
 
+      /* Sitemap document: turn its <loc> URLs into top-level seeds. They go
+         through htsAddLink, so the wizard's filters and scope rules decide, and
+         this link's max depth leaves them the full budget. */
+      if (opt->sitemap_state != NULL &&
+          hts_sitemap_pending(opt, urladr(), urlfil())) {
+        htsmoduleStruct BIGSTK smstr;
+        int smptr = ptr;
+
+        memset(&smstr, 0, sizeof(smstr));
+        smstr.opt = opt;
+        smstr.sback = sback;
+        smstr.cache = &cache;
+        smstr.hashptr = hashptr;
+        smstr.numero_passe = numero_passe;
+        smstr.ptr_ = &smptr; /* scratch: the ingester retargets the wizard */
+        smstr.addLink = htsAddLink;
+        smstr.url_host = urladr();
+        smstr.url_file = urlfil();
+        smstr.mime = r.contenttype;
+        hts_sitemap_ingest(opt, &smstr, urladr(), urlfil(), r.adr,
+                           r.adr != NULL && r.size > 0 ? (size_t) r.size : 0);
+      }
+
       // ------------------------------------------------------
       // ok, fichier chargé localement
       // ------------------------------------------------------
@@ -1819,6 +1875,9 @@ int httpmirror(char *url1, httrackp * opt) {
 
         if (strnotempty(savename()) == 0) {       // pas de chemin de sauvegarde
           if (strcmp(urlfil(), "/robots.txt") == 0) {     // robots.txt
+            char BIGSTK sitemaps[8192];
+
+            sitemaps[0] = '\0';
             if (r.adr) {
               char BIGSTK infobuff[8192];
 #ifdef IGNORE_RESTRICTIVE_ROBOTS
@@ -1830,7 +1889,8 @@ int httpmirror(char *url1, httrackp * opt) {
 #endif
 
               robots_parse(&robots, urladr(), r.adr, r.size, infobuff,
-                           sizeof(infobuff), keep_root);
+                           sizeof(infobuff), keep_root, sitemaps,
+                           sizeof(sitemaps));
               if (strnotempty(infobuff)) {
                 hts_log_print(opt, LOG_INFO,
                               "Note: robots.txt forbidden links for %s are: %s",
@@ -1840,6 +1900,10 @@ int httpmirror(char *url1, httrackp * opt) {
                               urladr(), infobuff);
               }
             }
+            /* After robots_parse, so the rules this very body carries already
+               gate the sitemap fetch. Runs even on a failed probe, which is
+               what falls back to the well-known location. */
+            hts_sitemap_robots(opt, urladr(), sitemaps);
           }
         } else if (r.is_write) {        // déja sauvé sur disque
           /*
@@ -1916,10 +1980,9 @@ int httpmirror(char *url1, httrackp * opt) {
           }
 
           // ATTENTION C'EST ICI QU'ON SAUVE LE FICHIER!!
-          // An empty body must not overwrite the file when the transfer failed
-          // (statuscode <= 0, e.g. an -M hard-stop): it would truncate a good
-          // copy to 0 (#77 follow-up).
-          if (r.adr != NULL || (r.size == 0 && r.statuscode > 0)) {
+          // A failed transfer has no body: r.adr holds debris from the aborted
+          // read, which would destroy the copy being re-fetched (#748).
+          if (r.statuscode > 0 && (r.adr != NULL || r.size == 0)) {
             file_notify(opt, urladr(), urlfil(), savename(), 1, 1, r.notmodified);
             if (filesave(opt, r.adr, (int) r.size, savename(), urladr(), urlfil()) !=
                 0) {
@@ -2089,9 +2152,12 @@ int httpmirror(char *url1, httrackp * opt) {
   if (cache.lst) {
     fclose(cache.lst);
     cache.lst = opt->state.strc.lst = NULL;
-    if (opt->delete_old) {
+    /* old.lst minus new.lst is the set of files the previous mirror had and
+       this one does not. --changes reports it; only --purge-old acts on it. */
+    if (opt->delete_old || opt->changes) {
       FILE *old_lst, *new_lst;
 
+      hts_changes_indexed(opt);
       //
       opt->state._hts_in_html_parsing = 3;
       //
@@ -2117,21 +2183,37 @@ int httpmirror(char *url1, httrackp * opt) {
               int purge = 0;
 
               while(!feof(old_lst)) {
-                linput(old_lst, line, 1000);
-                if (!strstr(adr, line)) { // not found in the new list?
-                  char BIGSTK file[HTS_URLMAXSIZE * 2];
+                char BIGSTK file[HTS_URLMAXSIZE * 2];
 
-                  strcpybuff(file, StringBuff(opt->path_html));
-                  strcatbuff(file, line + 1);
-                  file[strlen(file) - 1] = '\0';
-                  if (fexist_utf8(file)) { // toujours sur disque: virer
-                    hts_log_print(opt, LOG_INFO, "Purging %s", file);
-                    UNLINK(file);
-                    purge = 1;
+                linput(old_lst, line, 1000);
+                if (!strnotempty(line))
+                  continue;
+                strcpybuff(file, StringBuff(opt->path_html));
+                strcatbuff(file, line + 1);
+                file[strlen(file) - 1] = '\0';
+                hts_changes_previous(opt, file + StringLength(opt->path_html));
+                if (!strstr(adr, line)) { // not found in the new list?
+                  if (fexist_utf8(file)) { // still on disk
+                    /* A link this crawl did try but never wrote (a transfer
+                       killed mid-flight) also drops out of new.lst. Unless it
+                       is about to be purged, its previous copy stands and the
+                       file is not gone. */
+                    const hts_boolean kept =
+                        !opt->delete_old &&
+                        hash_read(opt->hash, file, NULL,
+                                  HASH_STRUCT_FILENAME) >= 0;
+
+                    hts_changes_dropped(
+                        opt, file + StringLength(opt->path_html), kept);
+                    if (opt->delete_old) {
+                      hts_log_print(opt, LOG_INFO, "Purging %s", file);
+                      UNLINK(file);
+                      purge = 1;
+                    }
                   }
                 }
               }
-              {
+              if (opt->delete_old) { // emptied directories go with the files
                 fseek(old_lst, 0, SEEK_SET);
                 while(!feof(old_lst)) {
                   linput(old_lst, line, 1000);
@@ -2166,7 +2248,7 @@ int httpmirror(char *url1, httrackp * opt) {
                 }
               }
               //
-              if (!purge) {
+              if (opt->delete_old && !purge) {
                 hts_log_print(opt, LOG_INFO, "No files purged");
               }
             }
@@ -2182,6 +2264,10 @@ int httpmirror(char *url1, httrackp * opt) {
   }
   // fin purge!
 
+  /* --single-file: inline each page's assets now the tree is final, after the
+     purge has deleted whatever this run dropped. */
+  singlefile_process_mirror(opt);
+
   // Indexation
   if (opt->kindex)
     index_finish(StringBuff(opt->path_html), opt->kindex);
@@ -2256,6 +2342,8 @@ int httpmirror(char *url1, httrackp * opt) {
   // ending
   usercommand(opt, 0, NULL, NULL, NULL, NULL);
   warc_close_opt(opt);
+  hts_changes_close_opt(opt);
+  hts_sitemap_free(opt);
 
   // désallocation mémoire & buffers
   XH_uninit;
@@ -2881,6 +2969,17 @@ int filecreateempty(filenote_strc * strc, const char *filename) {
     return 0;
 }
 
+void hts_savename_listed(const char *root, const char *s, char *dest,
+                         size_t destsize) {
+  char catbuff[CATBUFF_SIZE];
+
+  strlcpybuff(dest, fslash(catbuff, sizeof(catbuff), s), destsize);
+  if (strnotempty(root) && strncmp(fslash(catbuff, sizeof(catbuff), root), dest,
+                                   strlen(root)) == 0) {
+    strlcpybuff(dest, s + strlen(root), destsize);
+  }
+}
+
 // noter fichier
 int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
   // gestion du fichier liste liste
@@ -2890,15 +2989,8 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
     return 0;
   } else if (strc->lst) {
     char BIGSTK savelst[HTS_URLMAXSIZE * 2];
-    char catbuff[CATBUFF_SIZE];
 
-    strcpybuff(savelst, fslash(catbuff, sizeof(catbuff), s));
-    // couper chemin?
-    if (strnotempty(strc->path)) {
-      if (strncmp(fslash(catbuff, sizeof(catbuff), strc->path), savelst, strlen(strc->path)) == 0) {     // couper
-        strcpybuff(savelst, s + strlen(strc->path));
-      }
-    }
+    hts_savename_listed(strc->path, s, savelst, sizeof(savelst));
     fprintf(strc->lst, "[%s]" LF, savelst);
     fflush(strc->lst);
   }
@@ -2908,6 +3000,9 @@ int filenote(filenote_strc * strc, const char *s, filecreate_params * params) {
 /* Note: utf-8 */
 void file_notify(httrackp * opt, const char *adr, const char *fil,
                  const char *save, int create, int modify, int not_updated) {
+  hts_changes_notify(opt, adr, fil, save,
+                     (create || modify) ? HTS_TRUE : HTS_FALSE,
+                     not_updated ? HTS_TRUE : HTS_FALSE);
   RUN_CALLBACK6(opt, filesave2, adr, fil, save, create, modify, not_updated);
 }
 
@@ -3642,6 +3737,15 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
   to->warc_max_size = from->warc_max_size;
   to->warc_cdx = from->warc_cdx;
   to->warc_wacz = from->warc_wacz;
+  to->changes = from->changes;
+
+  to->single_file = from->single_file;
+  if (from->single_file_max_size > 0)
+    to->single_file_max_size = from->single_file_max_size;
+  if (from->sitemap)
+    to->sitemap = from->sitemap;
+  if (StringNotEmpty(from->sitemap_url))
+    StringCopyS(to->sitemap_url, from->sitemap_url);
 
   if (from->pause_max_ms > 0) {
     to->pause_min_ms = from->pause_min_ms;

src/htscore.h

@@ -354,6 +354,12 @@ int filecreateempty(filenote_strc * strct, const char *filename);
 
 int filenote(filenote_strc * strct, const char *s, filecreate_params * params);
 
+/* Copy into dest (destsize bytes) the form under which the local path s is
+   listed in new.lst: forward slashes, with the mirror `root` stripped when s
+   sits under it. Also the change report's key, so the two must not drift. */
+void hts_savename_listed(const char *root, const char *s, char *dest,
+                         size_t destsize);
+
 void file_notify(httrackp * opt, const char *adr, const char *fil,
                  const char *save, int create, int modify, int wasupdated);
 
@@ -409,6 +415,8 @@ char *readfile2(const char *fil, LLint * size);
 
 char *readfile_utf8(const char *fil);
 
+char *readfile2_utf8(const char *fil, LLint *size);
+
 char *readfile_or(const char *fil, const char *defaultdata);
 
 /* Backing (download-slot) scheduler. Operate on the back[] ring (struct_back).

src/htscoremain.c

@@ -41,6 +41,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsdefines.h"
 #include "htsalias.h"
 #include "htswarc.h"
+#include "htschanges.h"
 #include "htsbauth.h"
 #include "htswrap.h"
 #include "htsmodules.h"
@@ -1747,6 +1748,13 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->cookies_file, argv[na]);
                 }
                 break;
+              case 'd': // --changes: report what this crawl changed
+                opt->changes = HTS_TRUE;
+                if (*(com + 1) == '0') {
+                  opt->changes = HTS_FALSE;
+                  com++;
+                }
+                break;
               case 'r': // warc / warc-file: write an ISO-28500 WARC archive
                 if (*(com + 1) == 'f') { // --warc-file NAME: explicit basename
                   com++;
@@ -1795,6 +1803,56 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->warc_file, WARC_AUTONAME);
                 }
                 break;
+              case 'Z': // single-file: inline each page's assets as data: URIs
+                if (*(com + 1) == 's') { // --single-file-max-size N
+                  com++;
+                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
+                    HTS_PANIC_PRINTF(
+                        "Option single-file-max-size needs a blank "
+                        "space and a size");
+                    htsmain_free();
+                    return -1;
+                  }
+                  na++;
+                  { // reject non-numeric/negative/overflow; keep the default
+                    char *end;
+                    LLint v;
+
+                    errno = 0;
+                    v = strtoll(argv[na], &end, 10);
+                    if (isdigit((unsigned char) argv[na][0]) && *end == '\0' &&
+                        errno != ERANGE && v > 0)
+                      opt->single_file_max_size = v;
+                  }
+                  opt->single_file = HTS_TRUE;
+                } else {
+                  opt->single_file = HTS_TRUE;
+                  if (*(com + 1) == '0') {
+                    opt->single_file = HTS_FALSE;
+                    com++;
+                  }
+                }
+                break;
+              case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps
+                if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap
+                  com++;
+                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
+                    HTS_PANIC_PRINTF(
+                        "Option sitemap-url needs a blank space and a URL");
+                    htsmain_free();
+                    return -1;
+                  }
+                  na++;
+                  if (strlen(argv[na]) >= HTS_URLMAXSIZE) {
+                    HTS_PANIC_PRINTF("Sitemap URL too long");
+                    htsmain_free();
+                    return -1;
+                  }
+                  StringCopy(opt->sitemap_url, argv[na]);
+                } else { // --sitemap: robots.txt probe, then /sitemap.xml
+                  opt->sitemap = HTS_TRUE;
+                }
+                break;
               case 'Y': // why: explain the filter verdict for a URL, no crawl
                 if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                   HTS_PANIC_PRINTF("Option why needs a blank space and a URL");

src/htsglobal.h

@@ -138,10 +138,11 @@ Please visit our Website: http://www.httrack.com
 #define HTS_DOSNAME 0
 #endif
 
-// utiliser zlib?
+// zlib is mandatory: the cache is a zip and minizip calls it regardless
 #ifndef HTS_USEZLIB
-// autoload
 #define HTS_USEZLIB 1
+#elif !HTS_USEZLIB
+#error HTS_USEZLIB=0 is not a supported configuration
 #endif
 
 // brotli and zstd content codings; off unless the build opted in (configure,

src/htshelp.c

@@ -433,7 +433,8 @@ void help_catchurl(const char *dest_path) {
       }
       // former URL!
       {
-        char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
+        /* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
+        char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
 
         inplace_escape_check_url(dest, sizeof(dest));
         snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);
@@ -525,6 +526,11 @@ void help(const char *app, int more) {
     (" %L <file> add all URL located in this text file (one URL per line)");
   infomsg
     (" %S <file> add all scan rules located in this text file (one scan rule per line)");
+  infomsg(" %m  seed the crawl from the site's sitemap (robots.txt Sitemap:, "
+          "then /sitemap.xml); --sitemap-url URL names one explicitly. A "
+          "sitemap you name, or one the site declares, is fetched even under "
+          "robots.txt Disallow; only the guessed /sitemap.xml obeys it. The "
+          "URLs found still pass every filter and scope rule");
   infomsg("");
   infomsg("Build options:");
   infomsg("  NN structure type (0 *original structure, 1+: see below)");
@@ -534,6 +540,13 @@ void help(const char *app, int more) {
   infomsg
     (" %D  cached delayed type check, don't wait for remote type during updates, to speedup them (%D0 wait, * %D1 don't wait)");
   infomsg(" %M  generate a RFC MIME-encapsulated full-archive (.mht)");
+  infomsg(" %Z  after the mirror, rewrite each saved page with its "
+          "stylesheets, scripts, images and fonts inlined as data: URIs, so "
+          "any page opens by double-click anywhere (links between pages stay "
+          "relative; audio and video stay links); --single-file-max-size N "
+          "caps each asset (default 10485760 bytes). %M is the better "
+          "container where a Chromium-family browser is a given: one archive, "
+          "no base64 tax on text, a shared asset stored once");
   infomsg(" %t  keep the original file extension, don't rewrite it from the "
           "MIME type (%t0 rewrite)");
   infomsg
@@ -607,6 +620,8 @@ void help(const char *app, int more) {
           "output name, --warc-max-size N rotates segments past N bytes, "
           "--warc-cdx also writes a sorted CDXJ index, --wacz packages it all "
           "as a WACZ file");
+  infomsg(" %d  write hts-changes.json listing what this crawl left new, "
+          "changed, unchanged and gone compared to the previous mirror");
   infomsg(" %n  do not re-download locally erased files");
   infomsg
     (" %v  display on screen filenames downloaded (in realtime) - * %v1 short version - %v2 full animation");

src/htslib.c

@@ -36,7 +36,10 @@ Please visit our Website: http://www.httrack.com
 // Fichier librairie .c
 
 #include "htscore.h"
+#include "htssitemap.h"
 #include "htswarc.h"
+#include "htschanges.h"
+#include "htssinglefile.h"
 
 /* specific definitions */
 #include "htsbase.h"
@@ -2537,15 +2540,6 @@ void fil_simplifie(char *f) {
   }
 }
 
-void htsblk_failf(htsblk *r, const char *fmt, ...) {
-  va_list args;
-
-  va_start(args, fmt);
-  // deliberate clip: the reason is quoted from a remote reply
-  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
-  va_end(args);
-}
-
 // fermer liaison fichier ou socket
 void deletehttp(htsblk * r) {
 #if HTS_DEBUG_CLOSESOCK
@@ -2707,6 +2701,24 @@ void time_gmt_rfc822(char *s) {
   time_rfc822(s, A);
 }
 
+void hts_now_iso8601(char out[32]) {
+  time_t t = time(NULL);
+  struct tm tmv;
+
+#if defined(_WIN32)
+  struct tm *g = gmtime(&t);
+
+  if (g != NULL)
+    tmv = *g;
+  else
+    memset(&tmv, 0, sizeof(tmv));
+#else
+  if (gmtime_r(&t, &tmv) == NULL)
+    memset(&tmv, 0, sizeof(tmv));
+#endif
+  strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
+}
+
 // heure actuelle, format rfc (taille buffer 256o)
 void time_local_rfc822(char *s) {
   time_t tt;
@@ -6019,7 +6031,12 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   StringCopy(opt->strip_query, "");
   StringCopy(opt->cookies_file, "");
   StringCopy(opt->warc_file, "");
+  StringCopy(opt->sitemap_url, "");
   opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
+  opt->changes = HTS_FALSE;
+  opt->changes_state = NULL;
+  opt->single_file = HTS_FALSE;
+  opt->single_file_max_size = SINGLEFILE_DEFAULT_MAX_SIZE;
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
   opt->pause_max_ms = 0;
@@ -6172,6 +6189,10 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->cookies_file);
     StringFree(opt->why_url);
     StringFree(opt->warc_file);
+    StringFree(opt->sitemap_url);
+    hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */
+
+    hts_changes_free_opt(opt);
 
     StringFree(opt->path_html);
     StringFree(opt->path_html_utf8);

src/htslib.h

@@ -201,7 +201,8 @@ T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
                    int waitconnect, int addr_index, int *addr_count);
 /* Clips the formatted failure reason into r->msg, which also round-trips
    through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
-void htsblk_failf(htsblk *r, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
+#define htsblk_failf(R, ...)                                                   \
+  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
 
 HTS_INLINE void deletehttp(htsblk * r);
 HTS_INLINE int deleteaddr(htsblk * r);
@@ -257,6 +258,10 @@ void sec2str(char *s, TStamp t);
 
 void time_gmt_rfc822(char *s);
 void time_local_rfc822(char *s);
+
+/* Current UTC time as "YYYY-MM-DDThh:mm:ssZ". */
+void hts_now_iso8601(char out[32]);
+
 struct tm *convert_time_rfc822(struct tm *buffer, const char *s);
 int set_filetime(const char *file, struct tm *tm_time);
 int set_filetime_rfc822(const char *file, const char *date);

src/htsopt.h

@@ -547,6 +547,23 @@ struct httrackp {
                              archive. Tail: ABI */
   hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip
                               (implies --warc + --warc-cdx). Tail: ABI */
+  hts_boolean changes;   /**< --changes: report what this crawl changed against
+                              the previous mirror. Tail: ABI */
+  void *changes_state;   /**< live change-report accumulator (hts_changes*),
+                              engine-owned. Tail: ABI */
+  hts_boolean single_file; /**< --single-file: once the mirror is done, rewrite
+                                each saved page with its assets inlined as
+                                data: URIs. Tail: ABI */
+  LLint single_file_max_size; /**< --single-file-max-size: per-asset cap in
+                                   bytes; a bigger asset stays a link.
+                                   Tail: ABI */
+  hts_boolean sitemap; /**< --sitemap: probe the start host's robots.txt for
+                            Sitemap: lines, else /sitemap.xml. Tail: ABI */
+  String sitemap_url;  /**< --sitemap-url: sitemap to ingest. Tail: ABI */
+  /* Live state, not an option: copy_htsopt must leave it alone. It sits here
+     rather than in htsoptstate because that struct is embedded by value, so
+     growing it would shift every httrackp field declared after it. */
+  void *sitemap_state; /**< hts_sitemap_state*, or NULL. Tail: ABI */
 };
 
 /* Running statistics for a mirror. */

src/htsrobots.c

@@ -147,7 +147,8 @@ static void robots_blob_add(char *blob, size_t blobsize, char marker,
 
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow) {
+                  hts_boolean keep_root_disallow, char *sitemaps,
+                  size_t sitemapsize) {
   size_t bptr = 0;
   int record = 0;
   char BIGSTK line[1024];
@@ -156,6 +157,8 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
   blob[0] = '\0';
   if (info != NULL && infosize > 0)
     info[0] = '\0';
+  if (sitemaps != NULL && sitemapsize > 0)
+    sitemaps[0] = '\0';
 #if DEBUG_ROBOTS
   printf("robots.txt dump:\n%s\n", body);
 #endif
@@ -172,7 +175,19 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
       line[llen - 1] = '\0';
       llen--;
     }
-    if (strfield(line, "user-agent:")) {
+    if (sitemaps != NULL && strfield(line, "sitemap:")) {
+      // group-independent record (RFC 9309): collected whatever the group
+      char *a = line + 8;
+
+      while (is_realspace(*a))
+        a++;
+      /* A line at the buffer limit was truncated: a half URL is not one. */
+      if (strnotempty(a) && strlen(line) < sizeof(line) - 3 &&
+          strlen(a) + 2 < sitemapsize - strlen(sitemaps)) {
+        strlcatbuff(sitemaps, a, sitemapsize);
+        strlcatbuff(sitemaps, "\n", sitemapsize);
+      }
+    } else if (strfield(line, "user-agent:")) {
       char *a = line + 11;
 
       while (is_realspace(*a))

src/htsrobots.h

@@ -56,10 +56,12 @@ int checkrobots(robots_wizard * robots, const char *adr, const char *fil);
 void checkrobots_free(robots_wizard * robots);
 int checkrobots_set(robots_wizard * robots, const char *adr, const char *data);
 /* Parse robots.txt `body` for `adr`, storing the HTTrack group's rules; `info`
-   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /". */
+   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /", and
+   `sitemaps` (optional) collects the Sitemap: URLs, one per line. */
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow);
+                  hts_boolean keep_root_disallow, char *sitemaps,
+                  size_t sitemapsize);
 #endif
 
 #endif

src/htssafe.h

@@ -460,6 +460,25 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
   return b->buf;
 }
 
+/**
+ * Copy src into dest (capacity size, NUL included), truncating to fit and
+ * always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
+ * value read back from a cache, a header or the wire, where refusing the whole
+ * record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
+ * callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
+ */
+static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
+                                                     const char *src) {
+  size_t len, copy;
+
+  assertf(dest != NULL && src != NULL && size != 0);
+  len = strlen(src);
+  copy = len < size ? len : size - 1;
+  memcpy(dest, src, copy);
+  dest[copy] = '\0';
+  return copy == len ? HTS_TRUE : HTS_FALSE;
+}
+
 /**
  * Callers that deliberately ignore truncation use this instead of
  * slprintfbuff(), so it is not HTS_CHECK_RESULT.
@@ -469,6 +488,9 @@ static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
   int ret;
 
   assertf(dest != NULL && size != 0);
+  /* a vsnprintf failing outright may write nothing at all, leaving whatever
+     the caller had on the stack for it to publish */
+  dest[0] = '\0';
   ret = vsnprintf(dest, size, fmt, args);
   /* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
   dest[size - 1] = '\0';
@@ -492,6 +514,20 @@ static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
   return ret;
 }
 
+/**
+ * slprintfbuff() for diagnostics quoting remote or client text, which are
+ * meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
+ * cast on slprintfbuff() is no substitute, GCC warns through it.
+ */
+static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
+    char *dest, size_t size, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  (void) vslprintfbuff(dest, size, fmt, args);
+  va_end(args);
+}
+
 /**
  * slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
  * On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.

src/htsserver.c

@@ -318,6 +318,18 @@ typedef struct {
   error_redirect = "/server/error.html"; \
 } while(0)
 
+/* Longest error message shown on the error page; the rest is clipped. */
+#define ERROR_MESSAGE_MAX 1024
+
+/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
+   whose length the client picks. */
+#define SET_ERRORF(...)                                                        \
+  do {                                                                         \
+    char errbuf[ERROR_MESSAGE_MAX];                                            \
+    slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__);                    \
+    SET_ERROR(errbuf);                                                         \
+  } while (0)
+
 /* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
    so anything near this is already invalid and is rejected unread. */
 #define SID_VALUE_MAX 64
@@ -725,7 +737,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           fp = fopen(StringBuff(fspath), "rb");
           if (fp) {
             /* Read file */
-            while(!feof(fp)) {
+            while (!feof(fp) && !ferror(fp)) {
               char *str = line;
               char *pos;
 
@@ -879,28 +891,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                           commandEnd = 1;
                         }
                       } else {
-                        char tmp[1024];
-
-                        sprintf(tmp,
-                                "Unable to write %d bytes in the the init file %s",
-                                count, StringBuff(fspath));
-                        SET_ERROR(tmp);
+                        SET_ERRORF(
+                            "Unable to write %d bytes in the the init file %s",
+                            count, StringBuff(fspath));
                       }
                       fclose(fp);
                     } else {
-                      char tmp[1024];
-
-                      sprintf(tmp, "Unable to create the init file %s",
-                              StringBuff(fspath));
-                      SET_ERROR(tmp);
+                      SET_ERRORF("Unable to create the init file %s",
+                                 StringBuff(fspath));
                     }
                   } else {
-                    char tmp[1024];
-
-                    sprintf(tmp,
-                            "Unable to create the directory structure in %s",
-                            StringBuff(fspath));
-                    SET_ERROR(tmp);
+                    SET_ERRORF("Unable to create the directory structure in %s",
+                               StringBuff(fspath));
                   }
 
                 } else {
@@ -978,10 +980,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             }
           }
 
-          /* path itself may hold ".." (webhttrack passes "<bin>/../share"), so
-             only the untrusted halves are checked: file here, website above. */
-          if (fsfile[0] && strstr(file, "..") == NULL
-              && (fp = fopen(fsfile, "rb"))) {
+          /* Regular files only: reading a directory or FIFO never ends, and
+             "path" may hold "..", so only the untrusted halves are checked. */
+          if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
+              (fp = fopen(fsfile, "rb"))) {
             char ok[] =
               "HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
               "Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
@@ -1040,7 +1042,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               int outputmode = 0;
 
               StringMemcat(headers, ok, sizeof(ok) - 1);
-              while(!feof(fp)) {
+              while (!feof(fp) && !ferror(fp)) {
                 char *str = line;
                 int prevlen = (int) StringLength(output);
                 int nocr = 0;
@@ -1474,14 +1476,15 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               while(!feof(fp)) {
                 int n = (int) fread(line, 1, sizeof(line) - 2, fp);
 
-                if (n > 0) {
-                  StringMemcat(output, line, n);
+                if (n <= 0) {
+                  break; /* short read: EOF or error, never a retry */
                 }
+                StringMemcat(output, line, n);
               }
             }
             fclose(fp);
-          } else if (strcmp(file, "/ping") == 0
-                     || strncmp(file, "/ping?", 6) == 0) {
+          } else if (strcmp(file, "/ping") == 0 ||
+                     strncmp(file, "/ping?", 6) == 0) {
             char error_hdr[] =
               "HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
               "Content-type: text/html\r\n";

src/htssinglefile.h

@@ -0,0 +1,84 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* --single-file: post-mirror pass embedding each page's assets as data: URIs.
+   Internal, not installed. Unlike -%M (one MHT archive for the whole mirror)
+   it rewrites the saved pages in place and keeps page-to-page links relative,
+   so the mirror stays browsable and every page also stands alone.
+   Limitation: an inlined stylesheet becomes a data: URL, whose path is opaque,
+   so an asset inside it that stayed a link (over the cap) no longer resolves.
+   Raise --single-file-max-size past it to inline it too. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_SINGLEFILE_DEFH
+#define HTS_SINGLEFILE_DEFH
+
+#include "htsopt.h"
+#include "htsstrings.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* --single-file-max-size default: a bigger asset stays an ordinary link. */
+#define SINGLEFILE_DEFAULT_MAX_SIZE (10 * 1024 * 1024)
+
+/* Never base64 more than this, whatever the cap: code64() sizes in int. */
+#define SINGLEFILE_HARD_MAX_SIZE (256 * 1024 * 1024)
+
+/* Total bytes one page may inline. Nested @import fans out multiplicatively,
+   so a few hundred bytes of hostile CSS can otherwise ask for gigabytes. */
+#define SINGLEFILE_MAX_PAGE_SIZE (64 * 1024 * 1024)
+
+/* Rewrite every HTML page the mirror produced. No-op unless opt->single_file;
+   call once the tree is final, after the update purge. */
+void singlefile_process_mirror(httrackp *opt);
+
+/* Rewrite one HTML document held in memory, appending the result to out.
+   root is the mirror directory that references may not escape; page_path is
+   the document's own path under it (both UTF-8, '/' or native separators).
+   page_budget caps the total inlined bytes, since nested @import fans out
+   multiplicatively; the mirror pass passes SINGLEFILE_MAX_PAGE_SIZE.
+   Returns HTS_TRUE if at least one reference was replaced; out may still
+   differ from the input when that is HTS_FALSE, since a style or srcset value
+   is re-serialized in place. */
+hts_boolean singlefile_rewrite_html(httrackp *opt, const char *root,
+                                    const char *page_path, const char *html,
+                                    size_t html_len, LLint page_budget,
+                                    String *out);
+
+/* singlefile_rewrite_html() over the file at page_path, rewritten in place.
+   Returns HTS_TRUE if the file changed. */
+hts_boolean singlefile_rewrite_file(httrackp *opt, const char *root,
+                                    const char *page_path);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/htssitemap.c

@@ -0,0 +1,615 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: sitemap ingestion (sitemaps.org 0.9)                   */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#define HTS_INTERNAL_BYTECODE
+
+#include "htscore.h"
+#include "htssitemap.h"
+
+#include "htsbase.h"
+#include "htscodec.h"
+#include "htsencoding.h"
+#include "htsfilters.h"
+#include "htshash.h"
+#include "htsmodules.h"
+#include "htslib.h"
+#include "htsrobots.h"
+#include "htssafe.h"
+#include "htstools.h"
+
+#include <ctype.h>
+#include <string.h>
+
+/* One queued sitemap document awaiting ingestion. */
+typedef struct sitemap_doc {
+  char adr[HTS_URLMAXSIZE];
+  char fil[HTS_URLMAXSIZE];
+  int level;
+  hts_sitemap_source src;
+  hts_boolean done;
+  struct sitemap_doc *next;
+} sitemap_doc;
+
+struct hts_sitemap_state {
+  sitemap_doc *docs;
+  int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */
+  int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */
+  hts_boolean probe_done;    /* the robots.txt probe has been answered */
+  hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */
+  /* The crawl's own start URL. Seeded URLs are judged against it, so a site
+     cannot widen a subtree crawl by putting its sitemap at the root. */
+  char anchor_adr[HTS_URLMAXSIZE];
+  char anchor_fil[HTS_URLMAXSIZE];
+};
+typedef struct hts_sitemap_state hts_sitemap_state;
+
+/* --------------------------------------------------------------------- */
+/* Document parsing (no engine state: fuzzable and self-testable)         */
+/* --------------------------------------------------------------------- */
+
+/* Accept only an absolute http(s) URL with no space or control byte. */
+static hts_boolean sitemap_url_ok(const char *url) {
+  const char *p;
+
+  if (!strfield(url, "http://") && !strfield(url, "https://"))
+    return HTS_FALSE;
+  for (p = url; *p != '\0'; p++) {
+    if ((unsigned char) *p <= ' ' || (unsigned char) *p == 0x7f)
+      return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Skip to the character after the next '>' at or after p, or NULL. */
+static const char *sitemap_tag_end(const char *p, const char *end) {
+  while (p < end && *p != '>')
+    p++;
+  return p < end ? p + 1 : NULL;
+}
+
+/* HTS_TRUE when the document's root element is `name`. Skips the XML
+   declaration, comments and processing instructions first, so a comment
+   mentioning the other root element cannot decide the document type. */
+static hts_boolean sitemap_root_is(const char *doc, size_t size,
+                                   const char *name) {
+  const size_t nlen = strlen(name);
+  size_t i = 0;
+
+  if (size >= 3 && memcmp(doc, "\xef\xbb\xbf", 3) == 0)
+    i = 3; /* UTF-8 BOM */
+  while (i < size) {
+    if (isspace((unsigned char) doc[i])) {
+      i++;
+    } else if (doc[i] != '<') {
+      return HTS_FALSE; /* character data before any element: not XML */
+    } else if (i + 4 <= size && memcmp(doc + i, "<!--", 4) == 0) {
+      const char *const e = hts_memstr(doc + i, size - i, "-->", 3);
+
+      if (e == NULL)
+        return HTS_FALSE;
+      i = (size_t) (e - doc) + 3;
+    } else if (i + 2 <= size && (doc[i + 1] == '?' || doc[i + 1] == '!')) {
+      while (i < size && doc[i] != '>')
+        i++;
+      i++;
+    } else {
+      size_t j = i + 1;
+
+      /* an optional namespace prefix: <sm:sitemapindex> is the same element */
+      while (j < size && doc[j] != ':' && doc[j] != '>' &&
+             !isspace((unsigned char) doc[j]))
+        j++;
+      if (j >= size || doc[j] != ':')
+        j = i + 1;
+      else
+        j++;
+      return j + nlen <= size && memcmp(doc + j, name, nlen) == 0 &&
+                     (j + nlen == size ||
+                      isspace((unsigned char) doc[j + nlen]) ||
+                      doc[j + nlen] == '>' || doc[j + nlen] == '/')
+                 ? HTS_TRUE
+                 : HTS_FALSE;
+    }
+  }
+  return HTS_FALSE;
+}
+
+/* Decompress a gzip-framed body into a fresh buffer. The 64 MiB cap is what
+   binds in practice; deflate tops out near 1032:1, so the tree's codec budget
+   only matters as the shared policy for a coding that could go further. */
+static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) {
+  const LLint budget = hts_codec_maxout((LLint) size);
+  size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES
+                   ? (size_t) budget
+                   : (size_t) HTS_SITEMAP_MAX_BYTES;
+  char *out;
+  size_t n;
+
+  if (cap == 0)
+    return NULL;
+  out = malloct(cap + 1);
+  if (out == NULL)
+    return NULL;
+  n = hts_codec_head(HTS_CODEC_DEFLATE, body, size, out, cap);
+  if (n == 0) {
+    freet(out);
+    return NULL;
+  }
+  out[n] = '\0';
+  *outsize = n;
+  return out;
+}
+
+int hts_sitemap_scan(const char *body, size_t size, int maxurls,
+                     hts_boolean *is_index, hts_sitemap_handler handler,
+                     void *arg) {
+  char *unpacked = NULL;
+  const char *doc;
+  const char *end;
+  const char *p;
+  int n = 0;
+
+  if (is_index != NULL)
+    *is_index = HTS_FALSE;
+  if (body == NULL || size < 2 || handler == NULL)
+    return 0;
+
+  /* Content-Encoding gzip is undone upstream; only the container is left. */
+  if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) {
+    unpacked = sitemap_gunzip(body, size, &size);
+    if (unpacked == NULL)
+      return -1;
+    doc = unpacked;
+  } else {
+    if (size > (size_t) HTS_SITEMAP_MAX_BYTES)
+      size = (size_t) HTS_SITEMAP_MAX_BYTES;
+    doc = body;
+  }
+  end = doc + size;
+
+  /* Set before the first callback: the handler reads the verdict. */
+  if (is_index != NULL)
+    *is_index = sitemap_root_is(doc, size, "sitemapindex");
+
+  for (p = doc; n < maxurls;) {
+    const char *loc = hts_memstr(p, (size_t) (end - p), "<loc", 4);
+    const char *val;
+    const char *stop;
+    size_t len;
+    char BIGSTK url[HTS_URLMAXSIZE];
+
+    if (loc == NULL)
+      break;
+    /* "<loc>" or "<loc xmlns:..>", never "<location>" */
+    if (loc + 4 >= end || (loc[4] != '>' && !isspace((unsigned char) loc[4]))) {
+      p = loc + 4;
+      continue;
+    }
+    val = sitemap_tag_end(loc + 4, end);
+    if (val == NULL)
+      break;
+    for (stop = val; stop < end && *stop != '<'; stop++)
+      ;
+    /* No closing tag: truncated document, so the value may be a partial URL. */
+    if (stop == end)
+      break;
+    p = stop;
+    while (val < stop && isspace((unsigned char) *val))
+      val++;
+    while (stop > val && isspace((unsigned char) *(stop - 1)))
+      stop--;
+    len = (size_t) (stop - val);
+    /* Overflow-safe: the untrusted length alone against the room left. */
+    if (len == 0 || len >= sizeof(url))
+      continue;
+    memcpy(url, val, len);
+    url[len] = '\0';
+    /* hts_unescapeEntities decodes in place and tolerates src == dest; a
+       reference to a control byte survives as one and sitemap_url_ok drops it.
+     */
+    if (hts_unescapeEntities(url, url, sizeof(url)) != 0 ||
+        !sitemap_url_ok(url))
+      continue;
+    n++;
+    if (!handler(arg, url))
+      break;
+  }
+
+  if (unpacked != NULL)
+    freet(unpacked);
+  return n;
+}
+
+/* --------------------------------------------------------------------- */
+/* Engine glue                                                           */
+/* --------------------------------------------------------------------- */
+
+static hts_sitemap_state *sitemap_get_state(httrackp *opt) {
+  if (opt->sitemap_state == NULL)
+    opt->sitemap_state = calloct(1, sizeof(hts_sitemap_state));
+  return (hts_sitemap_state *) opt->sitemap_state;
+}
+
+static sitemap_doc *sitemap_find(httrackp *opt, const char *adr,
+                                 const char *fil) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+  sitemap_doc *d;
+
+  if (st == NULL)
+    return NULL;
+  for (d = st->docs; d != NULL; d = d->next) {
+    if (strfield2(d->adr, adr) && strcmp(d->fil, fil) == 0)
+      return d;
+  }
+  return NULL;
+}
+
+/* Who asked for this document decides how far it is gated. The wizard proper
+   is not usable here: it wants a referring link, and its up/down travel rules
+   would judge a child sitemap against the parent sitemap's directory. */
+static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr,
+                                         const char *fil,
+                                         hts_sitemap_source src) {
+  /* adr and fil are each capped just under HTS_URLMAXSIZE, and lfull prefixes
+     a scheme and a slash on top of both: 2 * HTS_URLMAXSIZE does not fit. */
+  char BIGSTK l[HTS_URLMAXSIZE * 2 + 16], lfull[HTS_URLMAXSIZE * 2 + 16];
+  int jokdepth = 0, jok;
+
+  hts_boolean refused;
+
+  /* The user naming a sitemap is the same intent as naming a start URL, which
+     the wizard admits unconditionally. */
+  if (src == HTS_SITEMAP_SRC_USER)
+    return HTS_TRUE;
+  strcpybuff(l, jump_identification_const(adr));
+  if (*fil != '/')
+    strcatbuff(l, "/");
+  strcatbuff(l, fil);
+  strcpybuff(lfull, link_has_authority(adr) ? "" : "http://");
+  strcatbuff(lfull, adr);
+  if (*fil != '/')
+    strcatbuff(lfull, "/");
+  strcatbuff(lfull, fil);
+  jok = fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull,
+                         l, NULL, NULL, &jokdepth);
+  refused = (jok == -1) ? HTS_TRUE : HTS_FALSE;
+  if (refused) {
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s",
+                  jokdepth + 1, adr, fil);
+    return HTS_FALSE;
+  }
+  /* A Sitemap: line, or a sitemapindex entry, is the site inviting the fetch;
+     a Disallow elsewhere in the same file does not retract it. The well-known
+     location is only ever a guess, so there a Disallow wins. */
+  if (src == HTS_SITEMAP_SRC_GUESSED &&
+      hts_robots_forbids(opt, adr, fil, (jok != 0) ? HTS_TRUE : HTS_FALSE,
+                         refused)) {
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr,
+                  fil);
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Record the link with save="" so the body stays in memory: a sitemap is
+   ingested, never mirrored. */
+static hts_boolean sitemap_queue_(httrackp *opt, const char *adr,
+                                  const char *fil, int level,
+                                  hts_sitemap_source src, hts_boolean link_it) {
+  hts_sitemap_state *const st = sitemap_get_state(opt);
+  sitemap_doc *d;
+
+  if (st == NULL)
+    return HTS_FALSE;
+  if (st->ndocs >= HTS_SITEMAP_MAX_DOCS || level > HTS_SITEMAP_MAX_LEVEL) {
+    hts_log_print(opt, LOG_WARNING, "Sitemap: cap reached, skipping %s%s", adr,
+                  fil);
+    return HTS_FALSE;
+  }
+  if (strlen(adr) >= sizeof(d->adr) || strlen(fil) >= sizeof(d->fil))
+    return HTS_FALSE;
+  if (sitemap_find(opt, adr, fil) != NULL)
+    return HTS_FALSE;
+  if (!sitemap_fetch_allowed(opt, adr, fil, src))
+    return HTS_FALSE;
+  d = calloct(1, sizeof(sitemap_doc));
+  if (d == NULL)
+    return HTS_FALSE;
+  strcpybuff(d->adr, adr);
+  strcpybuff(d->fil, fil);
+  d->level = level;
+  d->src = src;
+  d->next = st->docs;
+  st->docs = d;
+  st->ndocs++;
+
+  if (!link_it)
+    return HTS_TRUE;
+  if (!hts_record_link(opt, adr, fil, "", "", "", NULL))
+    return HTS_FALSE;
+  heap_top()->testmode = 0;
+  heap_top()->link_import = 0;
+  heap_top()->depth = opt->depth + 1;
+  heap_top()->pass2 = 0;
+  heap_top()->retry = opt->retry;
+  heap_top()->premier = heap_top_index();
+  heap_top()->precedent = heap_top_index();
+  hts_log_print(opt, LOG_INFO, "Sitemap: queued %s%s", adr, fil);
+  return HTS_TRUE;
+}
+
+static hts_boolean sitemap_queue(httrackp *opt, const char *adr,
+                                 const char *fil, int level,
+                                 hts_sitemap_source src) {
+  return sitemap_queue_(opt, adr, fil, level, src, HTS_TRUE);
+}
+
+void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
+                          const char *newadr, const char *newfil) {
+  sitemap_doc *const d = sitemap_find(opt, adr, fil);
+
+  if (d == NULL || d->done)
+    return;
+  d->done = HTS_TRUE; /* the body lives at the target now */
+  /* The engine already queued the target link, so only the marking moves. */
+  (void) sitemap_queue_(opt, newadr, newfil, d->level, d->src, HTS_FALSE);
+  hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil,
+                newadr, newfil);
+}
+
+void hts_sitemap_seed(httrackp *opt, const char *starturl) {
+  char BIGSTK url[HTS_URLMAXSIZE * 2];
+  lien_adrfil af;
+
+  if (StringNotEmpty(opt->sitemap_url)) {
+    if (strlen(StringBuff(opt->sitemap_url)) >= sizeof(url)) {
+      hts_log_print(opt, LOG_ERROR, "Sitemap URL too long");
+    } else {
+      strcpybuff(url, StringBuff(opt->sitemap_url));
+      if (strstr(url, ":/") == NULL)
+        hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url);
+      else if (ident_url_absolute(url, &af) >= 0)
+        (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_USER);
+    }
+  }
+  if (starturl == NULL || starturl[0] == '\0' ||
+      strlen(starturl) >= sizeof(url))
+    return;
+  strcpybuff(url, starturl);
+  if (ident_url_absolute(url, &af) < 0)
+    return;
+  {
+    hts_sitemap_state *const st = sitemap_get_state(opt);
+
+    if (st != NULL && strlen(af.adr) < sizeof(st->anchor_adr) &&
+        strlen(af.fil) < sizeof(st->anchor_fil)) {
+      strcpybuff(st->anchor_adr, af.adr);
+      strcpybuff(st->anchor_fil, af.fil);
+    }
+  }
+  if (!opt->sitemap)
+    return;
+  /* Answered in hts_sitemap_robots, once the parsed rules are installed. */
+  if (hts_record_link(opt, af.adr, "/robots.txt", "", "", "", NULL)) {
+    heap_top()->testmode = 0;
+    heap_top()->link_import = 0;
+    heap_top()->depth = 0;
+    heap_top()->pass2 = 0;
+    heap_top()->retry = opt->retry;
+    heap_top()->premier = heap_top_index();
+    heap_top()->precedent = heap_top_index();
+    /* Claim the host so the parser does not queue robots.txt a second time. */
+    if (opt->robotsptr != NULL)
+      (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, "");
+  }
+}
+
+void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+  int queued = 0;
+
+  if (st == NULL || !opt->sitemap || st->probe_done ||
+      !strfield2(st->anchor_adr, adr))
+    return;
+  st->probe_done = HTS_TRUE;
+  if (sitemaps != NULL) {
+    const char *p = sitemaps;
+
+    while (*p != '\0') {
+      const char *const eol = strchr(p, '\n');
+      const size_t len = eol != NULL ? (size_t) (eol - p) : strlen(p);
+      char BIGSTK line[HTS_URLMAXSIZE];
+      lien_adrfil af;
+
+      if (len > 0 && len < sizeof(line)) {
+        memcpy(line, p, len);
+        line[len] = '\0';
+        /* Same host: a Sitemap: line must not aim the fetcher elsewhere. */
+        if (sitemap_url_ok(line) && ident_url_absolute(line, &af) >= 0 &&
+            strfield2(af.adr, adr) &&
+            sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_DECLARED))
+          queued++;
+      }
+      if (eol == NULL)
+        break;
+      p = eol + 1;
+    }
+  }
+  if (queued == 0 && !st->fallback_done) {
+    st->fallback_done = HTS_TRUE;
+    if (sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_SITEMAP_SRC_GUESSED))
+      queued++;
+  }
+  hts_log_print(opt, LOG_NOTICE, "Sitemap: %d sitemap(s) queued for %s", queued,
+                adr);
+}
+
+hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
+                                const char *fil) {
+  const sitemap_doc *const d = sitemap_find(opt, adr, fil);
+
+  return d != NULL && !d->done ? HTS_TRUE : HTS_FALSE;
+}
+
+/* Handler context: seeding URLs from one document. */
+typedef struct sitemap_ingest_ctx {
+  httrackp *opt;
+  htsmoduleStruct *str;
+  const char *adr; /* host of the document being ingested */
+  int level;
+  hts_boolean is_index;
+  int accepted; /* URLs seeded or documents queued, not merely parsed */
+} sitemap_ingest_ctx;
+
+/* A <loc> of a <urlset>: hand it to the wizard as a top-level seed.
+   The wizard is pointed at the crawl's own start URL, not at the sitemap: the
+   site picks where its sitemap lives, so anchoring travel there would let a
+   root sitemap widen a subtree crawl to the whole host. The URL then becomes
+   its own anchor, exactly as a command-line seed does. */
+static hts_boolean sitemap_seed_url(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+  httrackp *const opt = c->opt;
+  hts_sitemap_state *const st = sitemap_get_state(opt);
+  char BIGSTK buff[HTS_URLMAXSIZE];
+  int before;
+
+  if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) {
+    hts_log_print(opt, LOG_WARNING,
+                  "Sitemap: URL cap reached, ignoring the rest");
+    return HTS_FALSE;
+  }
+  /* strcpybuff aborts rather than truncating: never feed it unchecked input. */
+  if (strlen(url) >= sizeof(buff))
+    return HTS_TRUE;
+  st->nurls++;
+  strcpybuff(buff, url);
+  before = opt->lien_tot;
+  if (htsAddLink(c->str, buff))
+    c->accepted++;
+  if (opt->lien_tot > before)
+    heap_top()->premier = heap_top_index(); /* a seed anchors on itself */
+  return HTS_TRUE;
+}
+
+/* A <loc> of a <sitemapindex>: cross-host children are dropped, so a hostile
+   sitemap cannot aim the fetcher elsewhere. */
+static hts_boolean sitemap_seed_child(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+  char BIGSTK buff[HTS_URLMAXSIZE];
+  lien_adrfil af;
+
+  if (strlen(url) >= sizeof(buff))
+    return HTS_TRUE;
+  strcpybuff(buff, url);
+  if (ident_url_absolute(buff, &af) < 0)
+    return HTS_TRUE;
+  if (!strfield2(af.adr, c->adr)) {
+    hts_log_print(c->opt, LOG_WARNING,
+                  "Sitemap: ignoring off-host child sitemap %s%s", af.adr,
+                  af.fil);
+    return HTS_TRUE;
+  }
+  if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1,
+                    HTS_SITEMAP_SRC_DECLARED))
+    c->accepted++;
+  return HTS_TRUE;
+}
+
+/* The scan classifies the document before the first callback. */
+static hts_boolean sitemap_seed_any(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+
+  return c->is_index ? sitemap_seed_child(arg, url)
+                     : sitemap_seed_url(arg, url);
+}
+
+void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
+                        const char *fil, const char *body, size_t size) {
+  sitemap_doc *const d = sitemap_find(opt, adr, fil);
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+  sitemap_ingest_ctx ctx;
+  int n, anchor, saved_depth;
+
+  if (d == NULL || d->done)
+    return;
+  d->done = HTS_TRUE;
+  /* str->ptr_ is a scratch int owned by the caller, so nothing else moves. */
+  anchor = *str->ptr_;
+  if (st != NULL && st->anchor_adr[0] != '\0' && opt->hash != NULL) {
+    const int i = hash_read((const hash_struct *) opt->hash, st->anchor_adr,
+                            st->anchor_fil, 1);
+
+    if (i >= 0)
+      anchor = i;
+  }
+  *str->ptr_ = anchor;
+  /* Borrow the anchor's position but keep a seed's full depth budget. */
+  saved_depth = heap(anchor)->depth;
+  heap(anchor)->depth = opt->depth + 1;
+  ctx.opt = opt;
+  ctx.str = str;
+  ctx.adr = adr;
+  ctx.level = d->level;
+  ctx.is_index = HTS_FALSE;
+  ctx.accepted = 0;
+
+  n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index,
+                       sitemap_seed_any, &ctx);
+  heap(anchor)->depth = saved_depth;
+  if (n < 0) {
+    hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr,
+                  fil);
+    return;
+  }
+  if (ctx.is_index)
+    hts_log_print(opt, LOG_NOTICE,
+                  "Sitemap: %d of %d child sitemap(s) listed by %s%s",
+                  ctx.accepted, n, adr, fil);
+  else
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: %d of %d URL(s) added from %s%s",
+                  ctx.accepted, n, adr, fil);
+}
+
+void hts_sitemap_free(httrackp *opt) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+
+  if (st == NULL)
+    return;
+  while (st->docs != NULL) {
+    sitemap_doc *const next = st->docs->next;
+
+    freet(st->docs);
+    st->docs = next;
+  }
+  freet(opt->sitemap_state);
+  opt->sitemap_state = NULL;
+}

src/htssitemap.h

@@ -0,0 +1,108 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack sitemap ingestion (sitemaps.org 0.9). Internal, not installed.
+   Reads <urlset>/<sitemapindex> documents, plain or gzip-framed, and feeds
+   their <loc> URLs to the crawl as top-level seeds. The whole input is
+   attacker-controlled, so every entry point below is capped. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_SITEMAP_DEFH
+#define HTS_SITEMAP_DEFH
+
+#include "htsdefines.h"
+#include "htsopt.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Caps. sitemaps.org allows 50000 URLs and 50 MB uncompressed per document;
+   the byte cap sits above that so a conformant sitemap always fits. */
+#define HTS_SITEMAP_MAX_URLS_DOC 50000           /* <loc> per document */
+#define HTS_SITEMAP_MAX_URLS_TOTAL 200000        /* <loc> per mirror */
+#define HTS_SITEMAP_MAX_DOCS 256                 /* documents per mirror */
+#define HTS_SITEMAP_MAX_LEVEL 4                  /* sitemapindex nesting */
+#define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */
+
+/* Who asked for a sitemap document, which decides how far its fetch is gated.
+   The user naming one is the same intent as a start URL; a site declaring one
+   invites the fetch; the well-known location is only ever our guess. */
+typedef enum {
+  HTS_SITEMAP_SRC_USER,     /**< --sitemap-url */
+  HTS_SITEMAP_SRC_DECLARED, /**< a Sitemap: line or a sitemapindex entry */
+  HTS_SITEMAP_SRC_GUESSED   /**< the /sitemap.xml fallback */
+} hts_sitemap_source;
+
+/* Per-URL handler; returning HTS_FALSE stops the scan. */
+typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url);
+
+/* Scan one sitemap document, plain or gzip-framed, handing every acceptable
+   absolute http(s) <loc> URL to `handler`. Stops after `maxurls` URLs, or when
+   the handler refuses. `is_index` (optional) reports a <sitemapindex>, whose
+   URLs are child sitemaps rather than pages. Returns the number of URLs handed
+   out, or -1 when the document could not be decompressed within the caps. */
+int hts_sitemap_scan(const char *body, size_t size, int maxurls,
+                     hts_boolean *is_index, hts_sitemap_handler handler,
+                     void *arg);
+
+/* --- Engine glue (needs a live httrackp). --- */
+
+/* Queue the first sitemap document of the mirror: the explicit --sitemap-url,
+   or the start host's /robots.txt probe for --sitemap. `starturl` is the first
+   command-line seed. No-op when neither option is set. */
+void hts_sitemap_seed(httrackp *opt, const char *starturl);
+
+/* Act on the start host's robots.txt once its rules are installed: queue the
+   Sitemap: URLs it names (newline-separated, from robots_parse), or the
+   well-known /sitemap.xml when it names none. No-op unless --sitemap. */
+void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps);
+
+/* Carry the sitemap marking of (adr,fil) over to the target of a redirect the
+   engine has already queued, so a moved sitemap is still ingested. */
+void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
+                          const char *newadr, const char *newfil);
+
+/* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */
+hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
+                                const char *fil);
+
+/* Ingest a fetched sitemap document (or the robots.txt probe): seed its URLs
+   through the wizard via htsAddLink, and queue nested sitemaps. `str` supplies
+   the parser context of the document being processed. */
+void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
+                        const char *fil, const char *body, size_t size);
+
+/* Release the ingestion state held in opt (NULL-safe, idempotent). */
+void hts_sitemap_free(httrackp *opt);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/htsstats.h

@@ -0,0 +1,63 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: in-progress display rows, shared by httrack and htsserver .h */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_STATS_DEFH
+#define HTS_STATS_DEFH
+
+#include "htsglobal.h"
+
+/* One row of the "in progress" display, shared so the CLI (httrack) and the
+   web GUI (htsserver) cannot drift apart: each fills its own array. */
+
+#define NStatsBuffer 14
+
+#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
+#define HTS_DEF_FWSTRUCT_t_StatsBuffer
+typedef struct t_StatsBuffer t_StatsBuffer;
+#endif
+struct t_StatsBuffer {
+  char name[1024];
+  char file[1024];
+  char state[288];                         // a short label plus back->info[256]
+  char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
+  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
+  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
+  LLint size;
+  LLint sizetot;
+  int offset;
+  //
+  int back;
+  //
+  int actived; // pour disabled
+};
+
+#endif

src/htsthread.c

@@ -44,9 +44,18 @@ Please visit our Website: http://www.httrack.com
 #endif
 #endif
 
+/* Outstanding threads, counted at spawn rather than by the child at entry, so
+   that a caller which spawns and immediately waits still joins them (#747). */
 static int process_chain = 0;
 static htsmutex process_chain_mutex = HTSMUTEX_INIT;
 
+static void process_chain_add(int delta) {
+  hts_mutexlock(&process_chain_mutex);
+  process_chain += delta;
+  assertf(process_chain >= 0);
+  hts_mutexrelease(&process_chain_mutex);
+}
+
 HTSEXT_API void htsthread_wait(void) {
   htsthread_wait_n(0);
 }
@@ -99,20 +108,12 @@ static void *hts_entry_point(void *tharg)
   void *const arg = s_args->arg;
   void (*fun) (void *arg) = s_args->fun;
 
-  free(tharg);
-
-  hts_mutexlock(&process_chain_mutex);
-  process_chain++;
-  assertf(process_chain > 0);
-  hts_mutexrelease(&process_chain_mutex);
+  freet(tharg);
 
   /* run */
   fun(arg);
 
-  hts_mutexlock(&process_chain_mutex);
-  process_chain--;
-  assertf(process_chain >= 0);
-  hts_mutexrelease(&process_chain_mutex);
+  process_chain_add(-1);
 #ifdef _WIN32
   return 0;
 #else
@@ -122,18 +123,20 @@ static void *hts_entry_point(void *tharg)
 
 /* create a thread */
 HTSEXT_API int hts_newthread(void (*fun) (void *arg), void *arg) {
-  hts_thread_s *s_args = malloc(sizeof(hts_thread_s));
+  hts_thread_s *s_args = malloct(sizeof(hts_thread_s));
 
   assertf(s_args != NULL);
   s_args->arg = arg;
   s_args->fun = fun;
+  process_chain_add(1);
 #ifdef _WIN32
   {
     unsigned int idt;
     HANDLE handle =
       (HANDLE) _beginthreadex(NULL, 0, hts_entry_point, s_args, 0, &idt);
     if (handle == 0) {
-      free(s_args);
+      process_chain_add(-1);
+      freet(s_args);
       return -1;
     } else {
       /* detach the thread from the main process so that is can be independent */
@@ -151,7 +154,8 @@ HTSEXT_API int hts_newthread(void (*fun) (void *arg), void *arg) {
         || pthread_attr_setstacksize(&attr, stackSize) != 0
         || (retcode =
             pthread_create(&handle, &attr, hts_entry_point, s_args)) != 0) {
-      free(s_args);
+      process_chain_add(-1);
+      freet(s_args);
       return -1;
     } else {
       /* detach the thread from the main process so that is can be independent */

src/htstools.c

@@ -276,6 +276,21 @@ int ident_url_relatif(const char *lien, const char *origin_adr,
   return ok;
 }
 
+/* Bounded substring search: bodies and archive records carry NUL bytes, so
+   strstr() would stop at the first one. */
+const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
+                       size_t nlen) {
+  size_t i;
+
+  if (nlen == 0 || haylen < nlen)
+    return NULL;
+  for (i = 0; i + nlen <= haylen; i++) {
+    if (hay[i] == *needle && memcmp(hay + i, needle, nlen) == 0)
+      return hay + i;
+  }
+  return NULL;
+}
+
 // créer dans s, à partir du chemin courant curr_fil, le lien vers link (absolu)
 // un ident_url_relatif a déja été fait avant, pour que link ne soit pas un chemin relatif
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {

src/htstools.h

@@ -61,6 +61,11 @@ typedef struct lien_adrfilsave lien_adrfilsave;
 int ident_url_relatif(const char *lien, const char *origin_adr,
                       const char *origin_fil,
                       lien_adrfil* const adrfil);
+/* Bounded substring search over data that may hold NUL bytes; NULL if absent.
+ */
+const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
+                       size_t nlen);
+
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr);
 int link_has_authority(const char *lien);
 int link_has_authorization(const char *lien);

src/htswarc.c

@@ -462,22 +462,6 @@ static void warc_make_id(warc_writer *w, char out[64]) {
            b[11], b[12], b[13], b[14], b[15]);
 }
 
-static void warc_now_iso8601(char out[32]) {
-  time_t t = time(NULL);
-  struct tm tmv;
-#if defined(_WIN32)
-  struct tm *g = gmtime(&t);
-  if (g != NULL)
-    tmv = *g;
-  else
-    memset(&tmv, 0, sizeof(tmv));
-#else
-  if (gmtime_r(&t, &tmv) == NULL)
-    memset(&tmv, 0, sizeof(tmv));
-#endif
-  strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
-}
-
 /* Case-insensitive "is this the header named name?" test, tolerating optional
    whitespace before the ':' (non-compliant "Name : value" is still matched). */
 static int header_is(const char *line, size_t line_len, const char *name) {
@@ -1063,7 +1047,7 @@ static void warc_wacz_package(warc_writer *w) {
   wbuf_free(&pages);
 
   /* datapackage.json listing every stored file with its sha256 + size. */
-  warc_now_iso8601(created);
+  hts_now_iso8601(created);
   memset(&dp, 0, sizeof(dp));
   if (!err &&
       (wbuf_printf(&dp,
@@ -1166,7 +1150,7 @@ static int warc_emit(warc_writer *w, const char *type, const char *content_type,
 
   memset(&hdr, 0, sizeof(hdr));
   warc_make_id(w, id);
-  warc_now_iso8601(date);
+  hts_now_iso8601(date);
 
 #if HTS_USEOPENSSL
   /* Block digest over the whole block, in one streaming pass. */
@@ -1670,7 +1654,7 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
     char date[32];
     char *slot;
     size_t need;
-    warc_now_iso8601(date);
+    hts_now_iso8601(date);
     need = strlen(target_uri) + 1 + strlen(date) + 1;
     slot = malloct(need);
     if (slot != NULL) {

src/htsweb.h

@@ -35,26 +35,10 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsstats.h"
 
-#define NStatsBuffer     14
 #define MAX_LEN_INPROGRESS 40
 
-typedef struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[288];                      // a short label plus back->info[256]
-  char url_sav[HTS_URLMAXSIZE * 2];     // pour cancel
-  char url_adr[HTS_URLMAXSIZE * 2];
-  char url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-} t_StatsBuffer;
-
 typedef struct t_InpInfo {
   int ask_refresh;
   int refresh;

src/htswizard.c

@@ -151,6 +151,23 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag,
   return HTS_FALSE;
 }
 
+/* The engine's robots.txt verdict for (adr,fil). Under HTS_ROBOTS_SOMETIMES an
+   explicit filter acceptance overrides the ban, which is why the filter outcome
+   is an input; the sitemap fetcher asks the same question outside the wizard.
+ */
+hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
+                               hts_boolean filters_decided,
+                               hts_boolean filters_refused) {
+  if (!opt->robots || opt->robotsptr == NULL)
+    return HTS_FALSE;
+  if (checkrobots((robots_wizard *) opt->robotsptr, adr, fil) != -1)
+    return HTS_FALSE;
+  if (filters_decided && !filters_refused &&
+      opt->robots == HTS_ROBOTS_SOMETIMES)
+    return HTS_FALSE;
+  return HTS_TRUE;
+}
+
 static int hts_acceptlink_(httrackp * opt, int ptr,
                            const char *adr, const char *fil, const char *tag,
                            const char *attribute, int *set_prio_to,
@@ -576,30 +593,26 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
       }
     }
     // vérifier robots.txt
-    if (opt->robots) {
-      int r = checkrobots(_ROBOTS, adr, fil);
-
-      if (r == -1) {            // interdiction
+    if (opt->robots && checkrobots(_ROBOTS, adr, fil) == -1) {
 #if DEBUG_ROBOTS
-        printf("robots.txt forbidden: %s%s\n", adr, fil);
+      printf("robots.txt forbidden: %s%s\n", adr, fil);
 #endif
-        // question résolue, par les filtres, et mode robot non strict
-        if ((!question) && (filters_answer) &&
-            (opt->robots == HTS_ROBOTS_SOMETIMES) && (forbidden_url != 1)) {
-          r = 0;                // annuler interdiction des robots
-          if (!forbidden_url) {
-            hts_log_print(opt, LOG_DEBUG,
-                          "Warning link followed against robots.txt: link %s at %s%s",
-                          l, adr, fil);
-          }
-        }
-        if (r == -1) {          // interdire
-          forbidden_url = 1;
-          question = 0;
-          hts_log_print(opt, LOG_DEBUG,
-                        "(robots.txt) forbidden link: link %s at %s%s", l, adr,
-                        fil);
+      if (!hts_robots_forbids(opt, adr, fil,
+                              (!question && filters_answer) ? HTS_TRUE
+                                                            : HTS_FALSE,
+                              (forbidden_url == 1) ? HTS_TRUE : HTS_FALSE)) {
+        if (!forbidden_url) {
+          hts_log_print(
+              opt, LOG_DEBUG,
+              "Warning link followed against robots.txt: link %s at %s%s", l,
+              adr, fil);
         }
+      } else {
+        forbidden_url = 1;
+        question = 0;
+        hts_log_print(opt, LOG_DEBUG,
+                      "(robots.txt) forbidden link: link %s at %s%s", l, adr,
+                      fil);
       }
     }
 

src/htswizard.h

@@ -49,6 +49,13 @@ typedef struct httrackp httrackp;
 typedef struct lien_url lien_url;
 #endif
 
+/* The engine's robots.txt verdict for (adr,fil): HTS_TRUE when the fetch is
+   forbidden. `filters_decided`/`filters_refused` carry the filter outcome,
+   which overrides a ban under -s1 (HTS_ROBOTS_SOMETIMES). */
+hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
+                               hts_boolean filters_decided,
+                               hts_boolean filters_refused);
+
 int hts_acceptlink(httrackp * opt, int ptr,
                    const char *adr, const char *fil,
                    const char *tag, const char *attribute,

src/httrack.c

@@ -230,8 +230,7 @@ static hts_boolean vt_size_refresh(void) {
 */
 #define STYLE_STATVALUES VT_BOLD
 #define STYLE_STATTEXT   VT_UNBOLD
-#define STYLE_STATRESET  VT_UNBOLD
-#define NStatsBuffer     14
+#define STYLE_STATRESET VT_UNBOLD
 /* Rows the stats block and "Current job" take above the in-progress list. */
 #define NStatsHeaderRows 7
 
@@ -307,8 +306,8 @@ int main(int argc, char **argv) {
     fprintf(stderr, "* %s\n", hts_errmsg(opt));
   }
   global_opt = NULL;
+  htsthread_wait(); /* pending threads still read opt */
   hts_free_opt(opt);
-  htsthread_wait();             /* wait for pending threads */
   hts_uninit();
 
 #ifdef _WIN32

src/httrack.h

@@ -36,26 +36,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsglobal.h"
 #include "htscore.h"
 #include "htssafe.h"
-
-#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
-#define HTS_DEF_FWSTRUCT_t_StatsBuffer
-typedef struct t_StatsBuffer t_StatsBuffer;
-#endif
-struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[288]; // a short label plus back->info[256]
-  char BIGSTK url_sav[HTS_URLMAXSIZE * 2];      // pour cancel
-  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
-  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-};
+#include "htsstats.h"
 
 #ifndef HTS_DEF_FWSTRUCT_t_InpInfo
 #define HTS_DEF_FWSTRUCT_t_InpInfo

src/libhttrack.vcxproj

@@ -131,6 +131,7 @@
     <ClCompile Include="htsproxy.c" />
     <ClCompile Include="htsrobots.c" />
     <ClCompile Include="htsselftest.c" />
+    <ClCompile Include="htssinglefile.c" />
     <ClCompile Include="htssniff.c" />
     <ClCompile Include="htsthread.c" />
     <ClCompile Include="htstools.c" />
@@ -138,6 +139,8 @@
     <ClCompile Include="htswrap.c" />
     <ClCompile Include="htszlib.c" />
     <ClCompile Include="htswarc.c" />
+    <ClCompile Include="htschanges.c" />
+    <ClCompile Include="htssitemap.c" />
     <ClCompile Include="md5.c" />
     <ClCompile Include="minizip\ioapi.c" />
     <ClCompile Include="minizip\iowin32.c" />