sitemap: keep the ingestion state out of htsoptstate

htsoptstate is embedded by value as httrackp.state, so a field at its tail
shifts every httrackp member declared after it: an offsetof probe put
warc_file at 141752 on master and 141760 on the branch. Move the pointer to
httrackp's own tail, where every existing offset holds and copy_htsopt still
ignores it.

Also renumber the crawl test to 89, master having taken 87 and 90, and give
the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear
requires, plus its row in that test's table.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/workflows/windows-build.yml

@@ -225,8 +225,9 @@ jobs:
 
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
-          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581.
-          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test"
+          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
+          # webdav-mime needs a reapable background listener, which MSYS cannot give it.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -26,6 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
+- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
+  pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
+  and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
+  as "marker absent". Capture the reply, assert the status line it must carry
+  (an empty, truncated or redirected one is marker-free too), then match with a
+  here-string.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -48,6 +54,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
+- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
+  (`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
+  our own data, wrong for anything read back from a cache, a header or the
+  wire, where it trades a memory smash for a crash on malformed input. Clip
+  with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
+- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
+  a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
+  into the same buffer never appears on it. Before scoping a hardening pass off
+  compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
+  `\bstrcpy\s*\(`, `\bstrcat\s*\(`).
 
 ## C conventions
 - **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -84,6 +100,17 @@ Before pushing, and when reviewing others, don't skim for bugs:
   layout/ABI, cache/wire format, or a security path? A static or unit check
   isn't enough; exercise the wrong behavior at runtime. Claude Code:
   `/review-recipe`.
+- **Poison a canary, never compare it against zero.** Checking that a
+  neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
+  terminator writes — the exact bug the canary is there for. Fill it with a
+  non-zero byte, and prove it by killing both the stray-`'X'` and the
+  stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
+  lands inside the same struct.
+- **Overshoot every destination, not one.** A bounds test that oversizes a
+  single field cannot tell a per-field bound from a one-size-fits-all one, nor
+  from a fix that bounds that field and leaves its neighbours raw. Exercise
+  each destination the path touches, spanning at least two capacities, and
+  check what the code actually emits before writing the expected values.
 
 ## Commits
 - **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

configure.ac

@@ -66,10 +66,11 @@ AC_SUBST(LT_CV_OBJDIR,$lt_cv_objdir)
 AC_SUBST(VERSION_INFO)
 
 ### Default CFLAGS
+# No -Wdeclaration-after-statement: nothing sets -std=, so this builds as gnu17.
 DEFAULT_CFLAGS="-Wall -Wformat -Wformat-security \
 -Wmultichar -Wwrite-strings -Wcast-qual -Wcast-align \
 -Wstrict-prototypes -Wmissing-prototypes \
--Wmissing-declarations -Wdeclaration-after-statement \
+-Wmissing-declarations \
 -Wpointer-arith -Wsequence-point -Wnested-externs \
 -D_REENTRANT"
 AC_SUBST(DEFAULT_CFLAGS)
@@ -77,27 +78,30 @@ DEFAULT_LDFLAGS=""
 AC_SUBST(DEFAULT_LDFLAGS)
 
 ### Additional flags (if supported)
-AX_CHECK_COMPILE_FLAG([-Wparentheses], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wparentheses"])
-AX_CHECK_COMPILE_FLAG([-Winit-self], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Winit-self"])
-AX_CHECK_COMPILE_FLAG([-Wunused-but-set-parameter], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wunused-but-set-parameter"])
-AX_CHECK_COMPILE_FLAG([-Waddress], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Waddress"])
-AX_CHECK_COMPILE_FLAG([-Wuninitialized], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wuninitialized"])
-AX_CHECK_COMPILE_FLAG([-Wformat=2], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat=2"])
-AX_CHECK_COMPILE_FLAG([-Wformat-nonliteral], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat-nonliteral"])
-AX_CHECK_COMPILE_FLAG([-Wmissing-parameter-type], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wmissing-parameter-type"])
-AX_CHECK_COMPILE_FLAG([-Wold-style-definition], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wold-style-definition"])
-AX_CHECK_COMPILE_FLAG([-Wignored-qualifiers], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wignored-qualifiers"])
+# -Werror on probes: exit-status-only checks let clang's warn-on-unknown-flag through.
+AX_CHECK_COMPILE_FLAG([-Wparentheses], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wparentheses"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Winit-self], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Winit-self"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wunused-but-set-parameter], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wunused-but-set-parameter"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Waddress], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Waddress"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wuninitialized], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wuninitialized"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wformat=2], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat=2"], [], [-Werror])
+# -Wformat-nonliteral needs -Wformat in the probe or gcc rejects it as ignored.
+AX_CHECK_COMPILE_FLAG([-Wformat-nonliteral], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat-nonliteral"], [], [-Werror -Wformat])
+AX_CHECK_COMPILE_FLAG([-Wmissing-parameter-type], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wmissing-parameter-type"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wold-style-definition], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wold-style-definition"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wignored-qualifiers], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wignored-qualifiers"], [], [-Werror])
 # Make htssafe.h's pointer-dest 'warning' attribute a hard error in our build
 # (migration is at zero; a new char* dest is a regression). gcc/clang each take
 # only their own spelling; downstream keeps the plain warning, not a build break.
-AX_CHECK_COMPILE_FLAG([-Werror=attribute-warning], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=attribute-warning"])
-AX_CHECK_COMPILE_FLAG([-Werror=user-defined-warnings], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=user-defined-warnings"])
-AX_CHECK_COMPILE_FLAG([-fstrict-aliasing -Wstrict-aliasing], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstrict-aliasing -Wstrict-aliasing"])
+AX_CHECK_COMPILE_FLAG([-Werror=attribute-warning], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=attribute-warning"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Werror=user-defined-warnings], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=user-defined-warnings"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fstrict-aliasing -Wstrict-aliasing], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstrict-aliasing -Wstrict-aliasing"], [], [-Werror])
 AX_CHECK_COMPILE_FLAG([-fstack-protector-strong], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector-strong"],
-	[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"])])
-AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"])
-AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"])
-AX_CHECK_LINK_FLAG([-Wl,--discard-all], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--discard-all"])
+	[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"], [], [-Werror])], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"], [], [-Werror])
+# No --discard-all: it drops the local symbols naming every static function, so
+# a trace misattributes them to the nearest surviving global. Costs 0.6% size.
 AX_CHECK_LINK_FLAG([-Wl,--no-undefined], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--no-undefined"])
 AX_CHECK_LINK_FLAG([-Wl,-z,relro,-z,now], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,relro,-z,now"])
 AX_CHECK_LINK_FLAG([-Wl,-z,noexecstack], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,noexecstack"])
@@ -119,13 +123,13 @@ AC_SUBST([LIBC_FORCE_LINK])
 ### PIE
 CFLAGS_PIE=""
 LDFLAGS_PIE=""
-AX_CHECK_COMPILE_FLAG([-fpie -pie], [CFLAGS_PIE="-fpie -pie"])
+AX_CHECK_COMPILE_FLAG([-fpie], [CFLAGS_PIE="-fpie"], [], [-Werror])
 AX_CHECK_LINK_FLAG([-pie], [LDFLAGS_PIE="-pie"])
 AC_SUBST([CFLAGS_PIE])
 AC_SUBST([LDFLAGS_PIE])
 
-## Export all symbols for backtraces
-AX_CHECK_COMPILE_FLAG([-rdynamic], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -rdynamic"])
+# Ties a crash trace from a stripped build back to its separate debug symbols.
+AX_CHECK_LINK_FLAG([-Wl,--build-id], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--build-id"])
 
 ### Check for -fvisibility=hidden support
 gl_VISIBILITY

fuzz/Makefile.am

@@ -2,7 +2,7 @@
 if FUZZERS
 noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \
 	fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \
-	fuzz-htsparse
+	fuzz-htsparse fuzz-sitemap
 endif
 
 AM_CPPFLAGS = \
@@ -27,6 +27,7 @@ fuzz_url_SOURCES = fuzz-url.c fuzz.h
 fuzz_header_SOURCES = fuzz-header.c fuzz.h
 fuzz_cachendx_SOURCES = fuzz-cachendx.c fuzz.h
 fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h
+fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h
 
 # List corpus files explicitly: automake does not expand EXTRA_DIST globs.
 EXTRA_DIST = README.md run-fuzzers.sh \
@@ -47,4 +48,6 @@ EXTRA_DIST = README.md run-fuzzers.sh \
 	corpus/cachendx/regress-overadvance.bin \
 	corpus/cachendx/regress-truncated-entry.bin \
 	corpus/htsparse/basic.html corpus/htsparse/script-inscript.html \
-	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html
+	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html \
+	corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
+	corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz

fuzz/corpus/sitemap/sitemapindex.xml

@@ -0,0 +1 @@
+<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc></sitemap></sitemapindex>

fuzz/corpus/sitemap/truncated.xml

@@ -0,0 +1 @@
+<urlset><loc>http://h.test/x

fuzz/corpus/sitemap/urlset.xml

@@ -0,0 +1 @@
+
<?xml version="1.0"?><urlset><url><loc>http://h.test/a.html</loc></url><url><loc>https://h.test/b?x=1&amp;y=2</loc></url></urlset>

fuzz/fuzz-sitemap.c

@@ -0,0 +1,60 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* Fuzz the sitemap <loc> scanner (htssitemap.c): raw XML, gzip-framed bodies
+   and truncated streams all arrive here straight off the network. */
+#include "fuzz.h"
+#include "htssitemap.h"
+
+static hts_boolean sm_count(void *arg, const char *url) {
+  int *const n = (int *) arg;
+
+  (void) url;
+  (*n)++;
+  return HTS_TRUE;
+}
+
+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
+  static const int caps[] = {0, 1, 16, HTS_SITEMAP_MAX_URLS_DOC};
+  hts_boolean is_index;
+  char *body;
+  int n = 0, cap;
+
+  if (size == 0)
+    return 0;
+  cap = caps[data[0] % (sizeof(caps) / sizeof(caps[0]))];
+  data++, size--;
+  /* A heap copy of exactly `size` bytes: the scanner must never rely on a
+     terminator, and ASan turns any overread into a report. */
+  body = malloct(size != 0 ? size : 1);
+  memcpy(body, data, size);
+
+  (void) hts_sitemap_scan(body, size, cap, &is_index, sm_count, &n);
+
+  freet(body);
+  return 0;
+}

html/cmdguide.html

@@ -163,8 +163,26 @@ the index" problems disappear.</p>
   <tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
   <tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
   <tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
+  <tr><td><tt>--sitemap (-%m), --sitemap-url URL (-%mu)</tt></td><td>Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.</td></tr>
 </table>
 
+<p>Link-following only finds what something links to. Anything a site publishes
+solely in its sitemap is invisible to HTTrack unless you ask for it.
+<tt>--sitemap</tt> reads the start host's <tt>robots.txt</tt> for
+<tt>Sitemap:</tt> lines and falls back to <tt>/sitemap.xml</tt>;
+<tt>--sitemap-url</tt> names one directly. Nested <tt>sitemapindex</tt> files
+and gzipped <tt>.xml.gz</tt> sitemaps are followed. The URLs found become start
+URLs with the full depth budget, but they still go through your filters and
+scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is
+off by default because a sitemap can list thousands of pages nothing links
+to.</p>
+
+<p>One surprise worth knowing: a sitemap you name with <tt>--sitemap-url</tt>,
+and one the site itself declares in <tt>robots.txt</tt>, are fetched even when
+<tt>robots.txt</tt> disallows that path, because naming or declaring a sitemap
+is an invitation to read it. Only the guessed <tt>/sitemap.xml</tt> obeys a
+<tt>Disallow</tt>. The URLs listed inside are gated normally either way.</p>
+
 <p>The single most common surprise is "only the home page came down." That is
 usually not a scope option at all: it is an off-host redirect. A start URL of
 <tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>

html/httrack.man.html

@@ -87,8 +87,8 @@ offline browser : copy websites to a local directory</p>
 --host-control[=N]</b> ] [ <b>-%P,
 --extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
 --test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
-] [ <b>-NN, --structure[=N]</b> ] [ <b>-%N,
---delayed-type-check</b> ] [ <b>-%D,
+] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
+[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
 --cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
 ] [ <b>-LN, --long-names[=N]</b> ] [ <b>-KN,
 --keep-links[=N]</b> ] [ <b>-x, --replace-external</b> ] [
@@ -575,6 +575,22 @@ URL per line) (--list &lt;param&gt;)</p></td></tr>
 
 <p>&lt;file&gt; add all scan rules located in this text
 file (one scan rule per line) (--urllist &lt;param&gt;)</p></td></tr>
+<tr valign="top" align="left">
+<td width="9%"></td>
+<td width="4%">
+
+
+<p>-%m</p></td>
+<td width="5%"></td>
+<td width="82%">
+
+
+<p>seed the crawl from the site&rsquo;s sitemap (robots.txt
+Sitemap:, then /sitemap.xml); --sitemap-url URL names one
+explicitly. A sitemap you name, or one the site declares, is
+fetched even under robots.txt Disallow; only the guessed
+/sitemap.xml obeys it. The URLs found still pass every
+filter and scope rule (--sitemap)</p></td></tr>
 </table>
 
 <h3>Build options:

html/server/finished.html

@@ -97,9 +97,8 @@ ${do:end-if}
 </pre>
 
 ${LANG_G8} : 
-${do:output-mode:html-urlescaped}
-<a href="file://${path}/${projname}/" target="_new">
-${do:output-mode:}
+${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
+<a href="/website/index.html" target="_new">
 ${path}/${projname}
 </a></li>
 <ul>

html/server/option10.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ftpprox" value="">
+
 ${LANG_PROXYTYPE}
 <select name="proxytype"
 			title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

html/server/option2.html

@@ -98,6 +98,13 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="errpage" value="">
+<input type="hidden" name="external" value="">
+<input type="hidden" name="hidepwd" value="">
+<input type="hidden" name="hidequery" value="">
+<input type="hidden" name="nopurge" value="">
+
 ${LANG_I33}
 <br>
 <select name="build"

html/server/option2b.html

@@ -103,7 +103,7 @@ ${LANG_Q3}
 <tr><td>
 	<table width="100%">
 	<tr><td align="left">
-	<input type="submit" value="${LANG_OK]" 
+	<input type="submit" value="${LANG_OK}"
 ${do:output-mode:html-urlescaped}
 		onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
 ${do:output-mode:}

html/server/option3.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="windebug" value="">
+
 ${LANG_I40c}
 <br>
 

html/server/option4.html

@@ -98,6 +98,11 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ka" value="">
+<input type="hidden" name="remt" value="">
+<input type="hidden" name="rems" value="">
+
 <table border="0" width="100%" cellspacing="0">
 
 <tr><td>

html/server/option8.html

@@ -98,6 +98,18 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="cookies" value="">
+<input type="hidden" name="parsejava" value="">
+<input type="hidden" name="updhack" value="">
+<input type="hidden" name="urlhack" value="">
+<input type="hidden" name="keepwww" value="">
+<input type="hidden" name="keepslashes" value="">
+<input type="hidden" name="keepqueryorder" value="">
+<input type="hidden" name="toler" value="">
+<input type="hidden" name="http10" value="">
+<input type="hidden" name="sitemap" value="">
+
 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I58}
@@ -132,6 +144,17 @@ ${listid:robots:LISTDEF_8}
 </select>
 <br><br>
 
+<input type="checkbox" name="sitemap" ${checked:sitemap}
+			title='${html:LANG_SITEMAPTIP}' onMouseOver="info('${html:LANG_SITEMAPTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+> ${LANG_SITEMAP}
+<br><br>
+
+${LANG_SITEMAPURL}
+<input name="sitemapurl" value="${sitemapurl}" size="40"
+			title='${html:LANG_SITEMAPURLTIP}' onMouseOver="info('${html:LANG_SITEMAPURLTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+>
+<br><br>
+
 <input type="checkbox" name="updhack" ${checked:updhack}
 			title='${html:LANG_I1k}' onMouseOver="info('${html:LANG_I1k}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I62b}

html/server/option9.html

@@ -98,6 +98,13 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="warc" value="">
+<input type="hidden" name="norecatch" value="">
+<input type="hidden" name="logf" value="">
+<input type="hidden" name="index" value="">
+<input type="hidden" name="index2" value="">
+
 <input type="checkbox" name="cache2" ${checked:cache2}
 			title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I61}

html/server/step2.html

@@ -141,6 +141,8 @@ ${do:copy:KeepSlashes:keepslashes}
 ${do:copy:KeepQueryOrder:keepqueryorder}
 ${do:copy:StripQuery:stripquery}
 ${do:copy:StoreAllInCache:cache2}
+${do:copy:Sitemap:sitemap}
+${do:copy:SitemapUrl:sitemapurl}
 ${do:copy:Warc:warc}
 ${do:copy:WarcFile:warcfile}
 ${do:copy:LogType:logtype}

html/server/step4.html

@@ -114,16 +114,16 @@ ${do:end-if}
 
 ${/* Real commands and ini file generated below */}
 
-<!-- engine commandline -->
+<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
 ${do:output-mode:html}
 <textarea name="command" cols="50" rows="4" style="visibility:hidden">
 httrack \
 	--quiet \
 	--build-top-index \
 	${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
-	${urls}
-	${test:filelist:-%L "}${filelist}${test:filelist:"}
-	--path "${html:path}/${html:projname}" 
+	${unquoted:urls}
+	${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
+	--path "${arg:path}/${arg:projname}" 
 \
 	${test:parseall:--near}
 	${test:link:--test}
@@ -131,7 +131,7 @@ httrack \
 	${test:htmlfirst::--priority=7}
 \
 ${do:if-not-empty:BuildString}
-	--structure "${BuildString}"
+	--structure "${arg:BuildString}"
 ${do:end-if}
 	${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
 \
@@ -150,51 +150,54 @@ ${do:end-if}
 	${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
 	${test:windebug:--debug-headers}
 \
-	${test:connexion:--sockets=}${connexion}
+	${test:connexion:--sockets=}${unquoted:connexion}
 	${test:ka:--keep-alive}
-	${test:timeout:--timeout=}${timeout}
+	${test:timeout:--timeout=}${unquoted:timeout}
 	${test:remt:--host-control=1}
-	${test:retry:--retries=}${retry}
-	${test:rate:--min-rate=}${rate}
+	${test:retry:--retries=}${unquoted:retry}
+	${test:rate:--min-rate=}${unquoted:rate}
 	${test:rems:--host-control=2}
 \
-	${test:depth:--depth=}${depth}
-	${test:depth2:--ext-depth=}${depth2}
-	${test:maxhtml:--max-files=,}${maxhtml}
-	${test:othermax:--max-files=}${othermax}
-	${test:sizemax:--max-files=}${sizemax}
-	${test:pausebytes:--max-pause=}${pausebytes}
-	${test:maxtime:--max-time=}${maxtime}
-	${test:maxrate:--max-rate=}${maxrate}
-	${test:maxconn:--connection-per-second=}${maxconn}
-	${test:maxlinks:--advanced-maxlinks=}${maxlinks}
+	${test:depth:--depth=}${unquoted:depth}
+	${test:depth2:--ext-depth=}${unquoted:depth2}
+${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
+	${test:othermax:--max-files=}${unquoted:othermax}
+	${test:maxhtml:--max-files=,}${unquoted:maxhtml}
+	${test:sizemax:--max-size=}${unquoted:sizemax}
+	${test:pausebytes:--max-pause=}${unquoted:pausebytes}
+	${test:maxtime:--max-time=}${unquoted:maxtime}
+	${test:maxrate:--max-rate=}${unquoted:maxrate}
+	${test:maxconn:--connection-per-second=}${unquoted:maxconn}
+	${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
 \
-	--user-agent "${html:user}"
-	--footer "${html:footer}"
+	--user-agent "${arg:user}"
+	--footer "${arg:footer}"
 \
-	${url2}
+	${unquoted:url2}
 \
-	${test:cookies:--cookies=0:}
-	${test:parsejava:--parse-java=0:}
+	${ztest:cookies:--cookies=0:}
+	${ztest:parsejava:--parse-java=0:}
 	${test:updhack:--updatehack}
 	${test:urlhack:--urlhack=0:--urlhack}
 	${test:keepwww:--keep-www-prefix}
 	${test:keepslashes:--keep-double-slashes}
 	${test:keepqueryorder:--keep-query-order}
-	${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
-	${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
-	${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
+	${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
+	${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
+	${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
 	${test:toler:--tolerant}
 	${test:http10:--http-10}
 	${test:cache2:--store-all-in-cache}
+	${test:sitemap:--sitemap}
+	${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"}
 	${test:warc:--warc}
-	${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
+	${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
 	${test:norecatch:--do-not-recatch}
 	${test:logf:--single-log}
 	${test:logtype:::--extra-log:--debug-log}
 	${test:index:--index=0:}
 	${test:index2:--search-index=0:--search-index}
-	${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
+	${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
 	${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
 </textarea>
 
@@ -213,7 +216,7 @@ ParseAll=${ztest:parseall:0:1}
 HTMLFirst=${ztest:htmlfirst:0:1}
 Cache=${ztest:cache:0:1}
 NoRecatch=${ztest:norecatch:0:1}
-Dos=${dos
+Dos=${dos}
 Index=${ztest:index:0:1}
 WordIndex=${ztest:index2:0:1}
 Log=${ztest:logf:0:1:2}
@@ -239,6 +242,8 @@ KeepSlashes=${ztest:keepslashes:0:1}
 KeepQueryOrder=${ztest:keepqueryorder:0:1}
 StripQuery=${stripquery}
 StoreAllInCache=${ztest:cache2:0:1}
+Sitemap=${ztest:sitemap:0:1}
+SitemapUrl=${sitemapurl}
 Warc=${ztest:warc:0:1}
 WarcFile=${warcfile}
 LogType=${logtype}

lang.def

@@ -1042,3 +1042,11 @@ LANG_WARCFILE
 WARC archive name:
 LANG_WARCFILETIP
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+LANG_SITEMAP
+Seed the crawl from the site's sitemap
+LANG_SITEMAPTIP
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+LANG_SITEMAPURL
+Sitemap address:
+LANG_SITEMAPURLTIP
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/English.txt

@@ -1012,3 +1012,11 @@ WARC archive name:
 WARC archive name:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+Seed the crawl from the site's sitemap
+Seed the crawl from the site's sitemap
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Sitemap address:
+Sitemap address:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/Francais.txt

@@ -1012,3 +1012,11 @@ WARC archive name:
 Nom de l'archive WARC :
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.
+Seed the crawl from the site's sitemap
+Partir du plan de site (sitemap)
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Lire le plan de site (lignes Sitemap: de robots.txt, puis /sitemap.xml) et ajouter chaque URL listée comme adresse de départ.
+Sitemap address:
+Adresse du plan de site :
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresse d'un plan de site à lire au lieu de sonder le site ; laissez vide pour sonder robots.txt puis /sitemap.xml.

libtest/callbacks-example-filenameiisbug.c

@@ -71,7 +71,9 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
       for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
       if (iisBogus[i][j] == '\0'
           && (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
-        strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
+        /* j bytes matched, so j fit: copying j cannot overrun whatever the
+           table holds, and the tail must survive untouched */
+        memcpy(a, iisBogusReplace[i], (size_t) j);
         break;
       }
     }

man/httrack.1

@@ -3,7 +3,7 @@
 .\"
 .\" This file is generated by man/makeman.sh; do not edit by hand.
 .\" SPDX-License-Identifier: GPL-3.0-or-later
-.TH httrack 1 "23 July 2026" "httrack website copier"
+.TH httrack 1 "26 July 2026" "httrack website copier"
 .SH NAME
 httrack \- offline browser : copy websites to a local directory
 .SH SYNOPSIS
@@ -36,6 +36,7 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-t, \-\-test\fR ]
 [ \fB\-%L, \-\-list\fR ]
 [ \fB\-%S, \-\-urllist\fR ]
+[ \fB\-%m, \-\-sitemap\fR ]
 [ \fB\-NN, \-\-structure[=N]\fR ]
 [ \fB\-%N, \-\-delayed\-type\-check\fR ]
 [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
@@ -187,6 +188,8 @@ test all URLs (even forbidden ones) (\-\-test)
 <file> add all URL located in this text file (one URL per line) (\-\-list <param>)
 .IP \-%S
 <file> add all scan rules located in this text file (one scan rule per line) (\-\-urllist <param>)
+.IP \-%m
+seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (\-\-sitemap)
 .SS Build options:
 .IP \-NN
 structure type (0 *original structure, 1+: see below) (\-\-structure[=N])

src/Makefile.am

@@ -32,7 +32,9 @@ AM_LDFLAGS = \
 
 bin_PROGRAMS = proxytrack httrack htsserver
 
-httrack_LDADD = $(THREADS_LIBS) libhttrack.la
+httrack_SOURCES = httrack.c htsbacktrace.c htsbacktrace.h
+# $(DL_LIBS): dladdr() in the crash handler, still in libdl on pre-2.34 glibc.
+httrack_LDADD = $(THREADS_LIBS) $(DL_LIBS) libhttrack.la
 htsserver_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS) libhttrack.la
 proxytrack_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS)
 
@@ -46,7 +48,8 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
 
 lib_LTLIBRARIES = libhttrack.la
 
-htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
+htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
+	htscmdline.c htscmdline.h \
 	htsurlport.c htsurlport.h
 proxytrack_SOURCES = proxy/main.c \
 	proxy/proxytrack.c proxy/store.c \
@@ -60,21 +63,21 @@ whttrackrun_SCRIPTS = webhttrack
 libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscache_selftest.c htsdns_selftest.c htsselftest.c \
 	htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
-	htshelp.c htslib.c htsurlport.c htscoremain.c \
+	htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
 	hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
 	htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h  \
-	htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
+	htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
 	htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htssitemap.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \
@@ -84,7 +87,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
 libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
 libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
 
-EXTRA_DIST = httrack.h webhttrack \
+EXTRA_DIST = httrack.h htsstats.h webhttrack \
 		version.rc \
 		libhttrack.rc \
 		httrack.rc \

src/htsalias.c

@@ -114,6 +114,10 @@ const char *hts_optalias[][4] = {
    "strip [host/pattern=]key1,key2,... from URLs"},
   {"cookies-file", "-%K", "param1",
    "load extra cookies from a Netscape cookies.txt"},
+  {"sitemap", "-%m", "single",
+   "seed the crawl from the start host's sitemap (robots.txt, then "
+   "/sitemap.xml)"},
+  {"sitemap-url", "-%mu", "param1", "seed the crawl from this sitemap URL"},
   {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
   {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
   {"warc-max-size", "-%rs", "param1",

src/htsback.c

@@ -125,6 +125,22 @@ void back_free(struct_back ** sback) {
    above a normal handshake. The last candidate still gets the full timeout. */
 #define HTS_CONNECT_FALLBACK_TIMEOUT 10
 
+void back_read_ftp_result(FILE *fp, htsblk *r) {
+  size_t j = 0;
+
+  if (fscanf(fp, "%d ", &r->statuscode) != 1)
+    r->statuscode = STATUSCODE_INVALID;
+  // an external helper writes this file: stop at capacity, not at EOF
+  while (j + 1 < sizeof(r->msg)) {
+    const int c = fgetc(fp);
+
+    if (c == EOF)
+      break;
+    r->msg[j++] = (char) c;
+  }
+  r->msg[j] = '\0';
+}
+
 int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
                               int timeout) {
   int deadline;
@@ -541,9 +557,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
                                const char *ext) {
   // do not use tempnam() but a regular filename
   back->tmpfile_buffer[0] = '\0';
-  if (back->url_sav != NULL && back->url_sav[0] != '\0') {
-    snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
-             back->url_sav, ext);
+  if (back->url_sav[0] != '\0') {
+    /* same capacity as url_sav, so truncation drops the extension and aliases
+       the temp name onto the live file that back_finalize_backup() UNLINKs */
+    if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
+      hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
+                    back->url_sav);
+      back->tmpfile_buffer[0] = '\0';
+      return -1;
+    }
     back->tmpfile = back->tmpfile_buffer;
     if (structcheck(back->tmpfile) != 0) {
       hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
@@ -551,8 +573,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
       return -1;
     }
   } else {
-    snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
-             StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
+    /* truncation here would collide distinct tmpnameid's onto one name */
+    if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
+                     StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
+                     ext)) {
+      hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
+                    StringBuff(opt->path_html_utf8));
+      back->tmpfile_buffer[0] = '\0';
+      return -1;
+    }
     back->tmpfile = back->tmpfile_buffer;
   }
   /* OK */
@@ -887,8 +916,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
               HTS_STAT.stat_bytes += back[p].r.size;
               HTS_STAT.stat_files++;
               hts_log_print(opt, LOG_TRACE, "added file %s%s => %s",
-                            back[p].url_adr, back[p].url_fil,
-                            back[p].url_sav != NULL ? back[p].url_sav : "");
+                            back[p].url_adr, back[p].url_fil, back[p].url_sav);
             }
             if ((!back[p].r.notmodified) && (opt->is_update)) {
               HTS_STAT.stat_updated_files++;    // page modifiée
@@ -2302,26 +2330,24 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
               && slot_can_be_finalized(opt, &back[i]);
             int may_serialize = slot_can_be_cached_on_disk(&back[i]);
 
-            hts_log_print(opt, LOG_DEBUG,
-                          "back[%03d]: may_clean=%d, may_finalize_disk=%d, may_serialize=%d:"
-                          LF "\t"
-                          "finalized(%d), status(%d), locked(%d), delayed(%d), test(%d), "
-                          LF "\t"
-                          "statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), "
-                          LF "\t" "contenttype(%s), url(%s%s), save(%s)", i,
-                          may_clean, may_finalize, may_serialize,
-                          back[i].finalized, back[i].status, back[i].locked,
-                          IS_DELAYED_EXT(back[i].url_sav), back[i].testmode,
-                          back[i].r.statuscode, (int) back[i].r.size,
-                          back[i].r.is_write, may_be_hypertext_mime(opt,
-                                                                    back[i].r.
-                                                                    contenttype,
-                                                                    back[i].
-                                                                    url_fil),
-                          /* */
-                          back[i].r.contenttype, back[i].url_adr,
-                          back[i].url_fil,
-                          back[i].url_sav ? back[i].url_sav : "<null>");
+            hts_log_print(
+                opt, LOG_DEBUG,
+                "back[%03d]: may_clean=%d, may_finalize_disk=%d, "
+                "may_serialize=%d:" LF "\t"
+                "finalized(%d), status(%d), locked(%d), delayed(%d), "
+                "test(%d), " LF "\t"
+                "statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), " LF
+                "\t"
+                "contenttype(%s), url(%s%s), save(%s)",
+                i, may_clean, may_finalize, may_serialize, back[i].finalized,
+                back[i].status, back[i].locked, IS_DELAYED_EXT(back[i].url_sav),
+                back[i].testmode, back[i].r.statuscode, (int) back[i].r.size,
+                back[i].r.is_write,
+                may_be_hypertext_mime(opt, back[i].r.contenttype,
+                                      back[i].url_fil),
+                /* */
+                back[i].r.contenttype, back[i].url_adr, back[i].url_fil,
+                back[i].url_sav);
           }
         }
       }
@@ -2857,7 +2883,8 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               // new session
               back[i].r.ssl_con = SSL_new(openssl_ctx);
               if (back[i].r.ssl_con) {
-                const char* hostname = jump_protocol_const(back[i].url_adr);
+                /* non-const twin: the OpenSSL macro casts the qualifier away */
+                char *hostname = jump_protocol(back[i].url_adr);
                 // some servers expect the hostname on the clienthello (SNI TLS extension)
                 SSL_set_tlsext_host_name(back[i].r.ssl_con, hostname);
                 SSL_clear(back[i].r.ssl_con);
@@ -2949,7 +2976,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               back[i].r.msg[0] = '\0';
               strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
               if (!strnotempty(back[i].r.msg)) {
-                sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
+                htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
               }
               deletehttp(&back[i].r);
               back[i].r.soc = INVALID_SOCKET;
@@ -3022,16 +3049,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
             FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
                   "rb");
           if (fp) {
-            int j = 0;
-
-            fscanf(fp, "%d ", &(back[i].r.statuscode));
-            while(!feof(fp)) {
-              int c = fgetc(fp);
-
-              if (c != EOF)
-                back[i].r.msg[j++] = c;
-            }
-            back[i].r.msg[j++] = '\0';
+            back_read_ftp_result(fp, &back[i].r);
             fclose(fp);
             UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
             strcpybuff(fconcat
@@ -3332,10 +3350,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                   deleteaddr(&back[i].r);
                   if (back[i].r.size < back[i].r.totalsize)
                     back[i].r.statuscode = STATUSCODE_CONNERROR;        // recatch
-                  sprintf(back[i].r.msg,
-                          "Incorrect length (" LLintP " Bytes, " LLintP
-                          " expected)", (LLint) back[i].r.size,
-                          (LLint) back[i].r.totalsize);
+                  htsblk_failf(&back[i].r,
+                               "Incorrect length (" LLintP " Bytes, " LLintP
+                               " expected)",
+                               (LLint) back[i].r.size,
+                               (LLint) back[i].r.totalsize);
                 } else {
                   // Un warning suffira..
                   hts_log_print(opt, LOG_WARNING,

src/htsback.h

@@ -74,6 +74,10 @@ void back_free(struct_back ** sback);
 // backing
 #define BACK_ADD_TEST "(dummy)"
 #define BACK_ADD_TEST2 "(dummy2)"
+/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
+   clipping the message to r->msg. */
+void back_read_ftp_result(FILE *fp, htsblk *r);
+
 int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
                const char *sav);
 int back_available(const struct_back * sback);

src/htsbacktrace.c

@@ -0,0 +1,238 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: crash backtrace printer                                 */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+/* Before every header: glibc gates dladdr() on it. */
+#if defined(__linux) && !defined(_GNU_SOURCE)
+#define _GNU_SOURCE
+#endif
+
+#include "htsbacktrace.h"
+
+#include "htsglobal.h"
+
+#include <stdlib.h>
+#include <string.h>
+#ifdef _WIN32
+#include <io.h> /* write */
+#endif
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+#if (defined(__linux) && defined(HAVE_EXECINFO_H))
+#include <dlfcn.h>
+#include <errno.h>
+#include <execinfo.h>
+#include <signal.h>
+#include <time.h>
+#include <sys/wait.h>
+#define USES_BACKTRACE
+#endif
+
+#ifdef USES_BACKTRACE
+#define BT_MAX_FRAMES 64     /* frames we try to name */
+#define BT_MAX_MODULES 8     /* distinct modules, one child each */
+#define BT_HEX_SIZE 19       /* "0x" + 16 nibbles + NUL */
+#define BT_PATH_SIZE 1024    /* module path; longer is skipped */
+#define BT_WAIT_TICKS 300    /* 10ms ticks, shared: cap a slow child */
+#define BT_NO_SYMBOLIZER 127 /* child exit: execvp() found none */
+
+static hts_boolean symbolize_crash = HTS_TRUE;
+
+/* "0x"-prefixed hex: the handler must stay stdio-free. */
+static void print_hex(char *buffer, uintptr_t value) {
+  static const char digits[] = "0123456789abcdef";
+  size_t i = 2, a, b;
+
+  buffer[0] = '0';
+  buffer[1] = 'x';
+  do {
+    buffer[i++] = digits[value & 0xf];
+    value >>= 4;
+  } while (value != 0);
+  buffer[i] = '\0';
+  for (a = 2, b = i - 1; a < b; a++, b--) {
+    const char c = buffer[a];
+
+    buffer[a] = buffer[b];
+    buffer[b] = c;
+  }
+}
+
+/* HTS_FALSE if src does not fit: a truncated module path would point the
+   symbolizer at the wrong file. */
+static hts_boolean copy_bounded(char *dest, size_t size, const char *src) {
+  size_t i;
+
+  for (i = 0; i < size - 1 && src[i] != '\0'; i++) {
+    dest[i] = src[i];
+  }
+  dest[i] = '\0';
+  return src[i] == '\0' ? HTS_TRUE : HTS_FALSE;
+}
+
+/* Run the symbolizer on argv, output on fd, within *budget ticks. HTS_FALSE
+   only if none could be run at all; otherwise silent, the raw trace stands. */
+static hts_boolean spawn_symbolizer(char **argv, int fd, int *budget) {
+  const pid_t pid = fork();
+  int status = 0;
+
+  if (pid == -1)
+    return HTS_FALSE;
+  if (pid == 0) {
+    static char llvm_prog[] = "llvm-symbolizer";
+    static char llvm_opts[] = "-p";
+
+    dup2(fd, 1); /* both symbolizers write on stdout */
+    execvp(argv[0], argv);
+    argv[0] = llvm_prog; /* an LLVM-only install ships no addr2line */
+    argv[1] = llvm_opts;
+    execvp(argv[0], argv);
+    _exit(BT_NO_SYMBOLIZER);
+  }
+  for (; *budget > 0; (*budget)--) {
+    const struct timespec tick = {0, 10 * 1000 * 1000};
+    const pid_t reaped = waitpid(pid, &status, WNOHANG);
+
+    if (reaped == pid)
+      return WIFEXITED(status) && WEXITSTATUS(status) == BT_NO_SYMBOLIZER
+                 ? HTS_FALSE
+                 : HTS_TRUE;
+    if (reaped == -1 && errno != EINTR)
+      return HTS_TRUE;
+    nanosleep(&tick, NULL);
+  }
+  kill(pid, SIGKILL);
+  waitpid(pid, NULL, 0);
+  return HTS_TRUE;
+}
+
+/* Name the frames backtrace_symbols_fd() leaves as module+offset:
+   -fvisibility=hidden keeps them out of .dynsym, but DWARF has them. dladdr()
+   is not formally async-signal-safe; accepted, this path is already fatal. */
+static void symbolize_backtrace(void *const *stack, int size, int fd) {
+  static char prog[] = "addr2line";
+  static char opts[] = "-Cfipa";
+  static char dashe[] = "-e";
+  char hex[BT_MAX_FRAMES][BT_HEX_SIZE];
+  const void *base[BT_MAX_FRAMES];
+  const char *name[BT_MAX_FRAMES];
+  hts_boolean grouped[BT_MAX_FRAMES];
+  char module[BT_PATH_SIZE];
+  char *argv[4 + BT_MAX_FRAMES + 1];
+  int budget = BT_WAIT_TICKS;
+  int i, spawned;
+
+  if (size > BT_MAX_FRAMES)
+    size = BT_MAX_FRAMES;
+
+  for (i = 0; i < size; i++) {
+    Dl_info info;
+
+    grouped[i] = HTS_TRUE; /* skipped unless dladdr() places the frame */
+    if (dladdr(stack[i], &info) == 0 || info.dli_fname == NULL ||
+        info.dli_fname[0] == '\0')
+      continue;
+    base[i] = info.dli_fbase;
+    name[i] = info.dli_fname;
+    print_hex(hex[i], (uintptr_t) ((const char *) stack[i] -
+                                   (const char *) info.dli_fbase));
+    grouped[i] = HTS_FALSE;
+  }
+
+  /* One child per module: addr2line takes a single -e. Each frame is claimed
+     once, so argc cannot exceed argv[]. */
+  for (spawned = 0; spawned < BT_MAX_MODULES; spawned++) {
+    int first, j, argc = 0;
+
+    for (first = 0; first < size && grouped[first]; first++)
+      ;
+    if (first >= size)
+      break;
+    argv[argc++] = prog;
+    argv[argc++] = opts;
+    argv[argc++] = dashe;
+    argv[argc++] = module;
+    for (j = first; j < size; j++) {
+      if (grouped[j] || base[j] != base[first])
+        continue;
+      grouped[j] = HTS_TRUE;
+      argv[argc++] = hex[j];
+    }
+    argv[argc] = NULL;
+    /* access(): skip pseudo-modules like linux-vdso, which have no file and
+       would draw nothing but an addr2line complaint. */
+    if (copy_bounded(module, sizeof(module), name[first]) &&
+        access(module, R_OK) == 0) {
+      const size_t len = strlen(module);
+
+      /* addr2line -a prints offsets only: say which module they are in. */
+      (void) (write(fd, module, len) == (ssize_t) len);
+      (void) (write(fd, ":\n", 2) == 2);
+      if (!spawn_symbolizer(argv, fd, &budget))
+        break; /* no symbolizer: stop at one header */
+    }
+  }
+}
+#endif
+
+void hts_backtrace_init(void) {
+#ifdef USES_BACKTRACE
+  symbolize_crash =
+      getenv("HTTRACK_NO_SYMBOLIZE") == NULL ? HTS_TRUE : HTS_FALSE;
+#endif
+}
+
+void hts_print_backtrace(int fd) {
+#ifdef USES_BACKTRACE
+  void *stack[256];
+  const int size = backtrace(stack, sizeof(stack) / sizeof(stack[0]));
+
+  /* A fault inside the handler lands back here: symbolizing twice interleaves
+     two traces on fd and spends a second budget. */
+  static volatile sig_atomic_t entered = 0;
+
+  if (size != 0) {
+    backtrace_symbols_fd(stack, size, fd);
+    if (symbolize_crash && entered == 0) {
+      entered = 1;
+      symbolize_backtrace(stack, size, fd);
+      entered = 0;
+    }
+  }
+#else
+  const char msg[] = "No stack trace available on this OS :(\n";
+
+  if (write(fd, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
+    /* sorry GCC */
+  }
+#endif
+}

src/htsbacktrace.h

@@ -0,0 +1,45 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: crash backtrace printer                                 */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#ifndef HTSBACKTRACE_DEFH
+#define HTSBACKTRACE_DEFH
+
+/* Sample HTTRACK_NO_SYMBOLIZE before any crash: getenv() is not signal-safe.
+   Call once, from the process that installs the fatal-signal handlers. */
+void hts_backtrace_init(void);
+
+/* Write the calling thread's stack to fd, callable from a fatal signal handler:
+   raw frames first, then whatever an external symbolizer can name. Allocates
+   nothing; prints a one-line notice where the OS has no backtrace(). */
+void hts_print_backtrace(int fd);
+
+#endif

src/htscache.c

@@ -142,10 +142,13 @@ struct cache_back_zip_entry {
   int compressionMethod;
 };
 
+/* Clip: strlcpybuff aborts on overflow, which would take the crawl down on a
+   corrupt cache carrying a field wider than ours. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      strlcpybuff(refvalue, value, refvalue_size);                             \
+      (refvalue)[0] = '\0';                                                    \
+      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -629,8 +632,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
                 // File exists on disk with declared cache name (this is expected!)
                 if (fexist_utf8(fconv(catbuff, sizeof(catbuff), previous_save))) {       // un fichier existe déja
                   // Expected size ?
-                  const size_t fsize =
-                    fsize_utf8(fconv(catbuff, sizeof(catbuff), previous_save));
+                  const LLint fsize = fsize_utf8(
+                      fconv(catbuff, sizeof(catbuff), previous_save));
                   if (fsize == r.size) {
                     // Target name is the previous name, and the file looks good: nothing to do!
                     if (strcmp(previous_save, target_save) == 0) {
@@ -666,7 +669,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
                 // Suppose a broken mirror, with a file being renamed: OK
                 else if (fexist_utf8(fconv(catbuff, sizeof(catbuff), target_save))) {
                   // Expected size ?
-                  const size_t fsize = fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
+                  const LLint fsize =
+                      fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
 
                   if (fsize == r.size) {
                     // So far so good
@@ -1440,7 +1444,7 @@ int cache_brstr(char *adr, char *s, size_t s_size) {
 
 /* binput bounded to a NUL-terminated buffer: refuse to start a read at or
    past `end`, so a prior over-advance can't walk a cache-index parse OOB. */
-int cache_binput(char *adr, const char *end, char *s, int max) {
+int cache_binput(const char *adr, const char *end, char *s, int max) {
   if (adr >= end) {
     s[0] = '\0';
     return 0;

src/htscache.h

@@ -93,7 +93,7 @@ void cache_rstr(FILE *fp, char *s, size_t s_size);
 char *cache_rstr_addr(FILE * fp);
 int cache_brstr(char *adr, char *s, size_t s_size);
 /* binput over a NUL-terminated buffer, bounded: no read starts at/past end. */
-int cache_binput(char *adr, const char *end, char *s, int max);
+int cache_binput(const char *adr, const char *end, char *s, int max);
 int cache_brint(char *adr, int *i);
 void cache_rint(FILE * fp, int *i);
 void cache_rLLint(FILE * fp, LLint * i);

src/htscache_selftest.c

@@ -1195,6 +1195,12 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
 
 /* --- read-side corruption injection --------------------------------------- */
 
+/* 100 'A's: a placeholder header line long enough to be overwritten by a
+   forged, over-long X-StatusMessage of the same byte length. */
+#define CORRUPT_LONG_ETAG                                                      \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"                         \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+
 /* canary read back intact after each corruption; victim gets the byte surgery
  */
 #define CORRUPT_ADR "corrupt.example.com"
@@ -1219,6 +1225,24 @@ static void corrupt_build(httrackp *opt) {
   selftest_close(&cache);
 }
 
+/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
+static void corrupt_build_longetag(httrackp *opt) {
+  cache_back cache;
+
+  memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
+  memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
+  remove(reconcile_st_path(opt, "hts-cache/new.zip"));
+  remove(reconcile_st_path(opt, "hts-cache/old.zip"));
+  selftest_open_for_write(&cache, opt);
+  store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
+              "OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
+              strlen(corrupt_body_a));
+  store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
+              "OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
+              corrupt_body_b, strlen(corrupt_body_b));
+  selftest_close(&cache);
+}
+
 /* Like corrupt_build, but the victim carries a 20-char Etag whose header line
    is later overwritten with a forged oversized X-Size (same byte length). */
 static void corrupt_build_etag(httrackp *opt) {
@@ -1403,6 +1427,48 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
   return fail;
 }
 
+/* An over-long field from a foreign cache must clip, not abort: the entry
+   still reads, clipped to capacity, and the canary survives. */
+static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
+                                         size_t wantlastmod, const char *what) {
+  cache_back cache;
+  htsblk v, c;
+  char BIGSTK lv[HTS_URLMAXSIZE * 2];
+  char BIGSTK lc[HTS_URLMAXSIZE * 2];
+  int fail = 0;
+
+  selftest_open_for_read(&cache, opt);
+  lv[0] = lc[0] = '\0';
+  v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
+  if (v.statuscode != 200) {
+    fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
+            v.statuscode);
+    fail++;
+  }
+  if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
+    fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
+            (unsigned) strlen(v.msg), (unsigned) wantmsg);
+    fail++;
+  }
+  if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
+    fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
+            what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
+    fail++;
+  }
+  c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
+  if (c.statuscode != 200) {
+    fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
+            c.statuscode);
+    fail++;
+  }
+  if (v.adr != NULL)
+    freet(v.adr);
+  if (c.adr != NULL)
+    freet(c.adr);
+  selftest_close(&cache);
+  return fail;
+}
+
 /* One zip corruption case: build, patch, then check victim+canary in-session.
  */
 static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1439,6 +1505,31 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
   failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
                                     "garbled deflate stream");
 
+  /* A corrupt cache can hold a field wider than ours. Clipping keeps the
+     entry; aborting would take the crawl down. Overwrite the placeholder Etag
+     line in place, same byte length, so the zip offsets stay intact. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "X-StatusMessage: " /* 17 + 89 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures +=
+      corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
+                                    (size_t) -1, "over-long X-StatusMessage");
+
+  /* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
+     cannot satisfy both. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "Last-Modified: " /* 15 + 91 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures += corrupt_expect_victim_clipped(
+      opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
+      "over-long Last-Modified");
+
   /* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
      truncates negative in the (int) cast the malloc uses: a wraparound alloc.
      cache_add asserts size fits an int, so such a value only reaches the reader

src/htscmdline.c

@@ -0,0 +1,95 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: command line splitter, shared by the engine and         */
+/*       htsserver                                               */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#include "htscmdline.h"
+
+#include "htssafe.h"
+
+#include <limits.h>
+#include <stdint.h>
+
+char **hts_split_cmdline(char *cmd, int *nargs) {
+  size_t nsep = 0;
+  size_t capacity;
+  size_t r;
+  size_t w;
+  int argc = 0;
+  hts_boolean quoted = HTS_FALSE;
+  char **argv;
+
+  *nargs = 0;
+
+  /* fold the other separators, so counting them sizes the vector exactly */
+  for (r = 0; cmd[r] != '\0'; r++) {
+    if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
+      cmd[r] = ' ';
+    }
+    if (cmd[r] == ' ') {
+      nsep++;
+    }
+  }
+
+  /* at most one argument per separator, plus the leading one and the NULL */
+  if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
+    return NULL;
+  }
+  capacity = nsep + 2;
+  argv = (char **) malloct(capacity * sizeof(char *));
+  if (argv == NULL) {
+    return NULL;
+  }
+
+  argv[argc++] = cmd;
+  for (r = 0, w = 0; cmd[r] != '\0';) {
+    if (quoted && cmd[r] == '\\' &&
+        (cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
+      r++;
+      cmd[w++] = cmd[r++];
+    } else if (cmd[r] == '\"') {
+      quoted = !quoted;
+      cmd[w++] = cmd[r++];
+    } else if (cmd[r] == ' ' && !quoted) {
+      cmd[w++] = '\0';
+      assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
+      argv[argc++] = cmd + w;
+      r++;
+    } else {
+      cmd[w++] = cmd[r++];
+    }
+  }
+  cmd[w] = '\0';
+  argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
+
+  *nargs = argc;
+  return argv;
+}

src/htscmdline.h

@@ -0,0 +1,45 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: command line splitter, shared by the engine and         */
+/*       htsserver                                               */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#ifndef HTSCMDLINE_DEFH
+#define HTSCMDLINE_DEFH
+
+#include "htsglobal.h"
+
+/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
+   argv[0] being the program name and quotes left for the engine to strip.
+   Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
+   entries), or NULL when it cannot be sized or allocated. */
+char **hts_split_cmdline(char *cmd, int *nargs);
+
+#endif

src/htscore.c

@@ -39,6 +39,7 @@ Please visit our Website: http://www.httrack.com
 
 /* File defs */
 #include "htscore.h"
+#include "htssitemap.h"
 #include "htswarc.h"
 
 /* specific definitions */
@@ -447,13 +448,22 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
                           const char *fil) {
   if (!*makeindex_done) {
     if (*makeindex_fp) {
-      char BIGSTK tempo[1024];
+      /* sized off link_escaped below: at the old flat 1024 a long first link
+         produced a redirect to a clipped URL */
+      char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
       if (makeindex_links == 1) {
         char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
         escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
-        snprintf(tempo, sizeof(tempo),
-                 "<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
-                 link_escaped);
+        /* no redirect beats one pointing at a clipped URL */
+        if (!sprintfbuff(
+                tempo,
+                "<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
+                link_escaped)) {
+          hts_log_print(opt, LOG_WARNING,
+                        "index redirect omitted: first link too long (%s)",
+                        makeindex_firstlink);
+          tempo[0] = '\0';
+        }
       } else
         tempo[0] = '\0';
       hts_template_format(*makeindex_fp, template_footer,
@@ -696,17 +706,19 @@ int httpmirror(char *url1, httrackp * opt) {
   // copier adresse(s) dans liste des adresses
   {
     char *a = url1;
-    int primary_len = 8192;
-
-    if (StringNotEmpty(opt->filelist)) {
-      primary_len += max(0, fsize_utf8(StringBuff(opt->filelist)) * 2);
-    }
-    primary_len += (int) strlen(url1) * 2;
+    const LLint list_sz = StringNotEmpty(opt->filelist)
+                              ? fsize_utf8(StringBuff(opt->filelist))
+                              : 0;
+    /* two bytes reserved per list byte; -1 makes an undoublable size refused */
+    const LLint list_room =
+        list_sz > 0 ? (list_sz <= INT64_MAX / 2 ? list_sz * 2 : -1) : 0;
+    const size_t primary_len =
+        llint_grow_size_t(8192 + strlen(url1) * 2, list_room, 0);
 
     // création de la première page, qui contient les liens de base à scanner
     // c'est plus propre et plus logique que d'entrer à la main les liens dans la pile
     // on bénéficie ainsi des vérifications et des tests du robot pour les liens "primaires"
-    primary = (char *) malloct(primary_len);
+    primary = primary_len != (size_t) -1 ? (char *) malloct(primary_len) : NULL;
     if (!primary) {
       printf("PANIC! : Not enough memory [%d]\n", __LINE__);
       XH_extuninit;
@@ -887,7 +899,7 @@ int httpmirror(char *url1, httrackp * opt) {
       }
 
       if (filelist_buff != NULL) {
-        int filelist_ptr = 0;
+        size_t filelist_ptr = 0;
         int n = 0;
         char BIGSTK line[HTS_URLMAXSIZE * 2];
 
@@ -932,6 +944,22 @@ int httpmirror(char *url1, httrackp * opt) {
     heap_top()->premier = heap_top_index();        // premier lien, objet-père=objet              
     heap_top()->precedent = heap_top_index();      // lien précédent
 
+    /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are
+       injected before the crawl gets far. */
+    hts_sitemap_free(opt); /* an earlier mirror may have left a doc list */
+    if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) {
+      char BIGSTK first[HTS_URLMAXSIZE * 2];
+      const char *const eol = strchr(primary, '\n');
+      const size_t len = eol != NULL ? (size_t) (eol - primary) : 0;
+
+      first[0] = '\0';
+      if (len > 0 && len < sizeof(first)) {
+        memcpy(first, primary, len);
+        first[len] = '\0';
+      }
+      hts_sitemap_seed(opt, first);
+    }
+
     // Initialiser cache
     {
       opt->state._hts_in_html_parsing = 4;
@@ -1576,11 +1604,21 @@ int httpmirror(char *url1, httrackp * opt) {
         stre.maketrack_fp = maketrack_fp;
 
         /* Parse */
-        if (hts_mirror_check_moved(&str, &stre) != 0) {
-          XH_uninit;
-          return -1;
-        }
+        {
+          const int nlinks = opt->lien_tot;
 
+          if (hts_mirror_check_moved(&str, &stre) != 0) {
+            XH_uninit;
+            return -1;
+          }
+          /* A redirect re-queues the target as a fresh link; without carrying
+             the marking over, a moved sitemap is fetched and then ignored. */
+          if (opt->sitemap_state != NULL && opt->lien_tot > nlinks &&
+              hts_sitemap_pending(opt, urladr(), urlfil())) {
+            hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr,
+                                 heap_top()->fil);
+          }
+        }
       }
 
     }                           // if !error
@@ -1598,6 +1636,29 @@ int httpmirror(char *url1, httrackp * opt) {
       /* Load file and decode if necessary, after redirect check. */
       LOAD_IN_MEMORY_IF_NECESSARY();
 
+      /* Sitemap document: turn its <loc> URLs into top-level seeds. They go
+         through htsAddLink, so the wizard's filters and scope rules decide, and
+         this link's max depth leaves them the full budget. */
+      if (opt->sitemap_state != NULL &&
+          hts_sitemap_pending(opt, urladr(), urlfil())) {
+        htsmoduleStruct BIGSTK smstr;
+        int smptr = ptr;
+
+        memset(&smstr, 0, sizeof(smstr));
+        smstr.opt = opt;
+        smstr.sback = sback;
+        smstr.cache = &cache;
+        smstr.hashptr = hashptr;
+        smstr.numero_passe = numero_passe;
+        smstr.ptr_ = &smptr; /* scratch: the ingester retargets the wizard */
+        smstr.addLink = htsAddLink;
+        smstr.url_host = urladr();
+        smstr.url_file = urlfil();
+        smstr.mime = r.contenttype;
+        hts_sitemap_ingest(opt, &smstr, urladr(), urlfil(), r.adr,
+                           r.adr != NULL && r.size > 0 ? (size_t) r.size : 0);
+      }
+
       // ------------------------------------------------------
       // ok, fichier chargé localement
       // ------------------------------------------------------
@@ -1808,6 +1869,9 @@ int httpmirror(char *url1, httrackp * opt) {
 
         if (strnotempty(savename()) == 0) {       // pas de chemin de sauvegarde
           if (strcmp(urlfil(), "/robots.txt") == 0) {     // robots.txt
+            char BIGSTK sitemaps[8192];
+
+            sitemaps[0] = '\0';
             if (r.adr) {
               char BIGSTK infobuff[8192];
 #ifdef IGNORE_RESTRICTIVE_ROBOTS
@@ -1819,7 +1883,8 @@ int httpmirror(char *url1, httrackp * opt) {
 #endif
 
               robots_parse(&robots, urladr(), r.adr, r.size, infobuff,
-                           sizeof(infobuff), keep_root);
+                           sizeof(infobuff), keep_root, sitemaps,
+                           sizeof(sitemaps));
               if (strnotempty(infobuff)) {
                 hts_log_print(opt, LOG_INFO,
                               "Note: robots.txt forbidden links for %s are: %s",
@@ -1829,6 +1894,10 @@ int httpmirror(char *url1, httrackp * opt) {
                               urladr(), infobuff);
               }
             }
+            /* After robots_parse, so the rules this very body carries already
+               gate the sitemap fetch. Runs even on a failed probe, which is
+               what falls back to the well-known location. */
+            hts_sitemap_robots(opt, urladr(), sitemaps);
           }
         } else if (r.is_write) {        // déja sauvé sur disque
           /*
@@ -2245,6 +2314,7 @@ int httpmirror(char *url1, httrackp * opt) {
   // ending
   usercommand(opt, 0, NULL, NULL, NULL, NULL);
   warc_close_opt(opt);
+  hts_sitemap_free(opt);
 
   // désallocation mémoire & buffers
   XH_uninit;
@@ -3632,6 +3702,11 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
   to->warc_cdx = from->warc_cdx;
   to->warc_wacz = from->warc_wacz;
 
+  if (from->sitemap)
+    to->sitemap = from->sitemap;
+  if (StringNotEmpty(from->sitemap_url))
+    StringCopyS(to->sitemap_url, from->sitemap_url);
+
   if (from->pause_max_ms > 0) {
     to->pause_min_ms = from->pause_min_ms;
     to->pause_max_ms = from->pause_max_ms;

src/htscoremain.c

@@ -145,7 +145,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
   int argv_url = -1;            // ==0 : utiliser cache et doit.log
   char *argv_firsturl = NULL;   // utilisé pour nommage par défaut
   char *url = NULL;             // URLS séparées par un espace
-  int url_sz = 65535;
+  size_t url_sz = 65535;
 
   // the parametres
   int httrack_logmode = 3;      // ONE log file
@@ -224,21 +224,22 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
   /* create x_argvblk buffer for transformed command line */
   {
-    int current_size = 0;
-    int size;
+    size_t current_size = 0;
+    const LLint size = fsize("config");
+    size_t blk_size;
     int na;
 
     for(na = 0; na < argc; na++)
-      current_size += (int) (strlen(argv[na]) + 1);
-    if ((size = fsize("config")) > 0)
-      current_size += size;
-    x_argvblk = (char *) malloct(current_size + 32768);
+      current_size += strlen(argv[na]) + 1;
+    /* a huge file named "config" must saturate, not wrap, the capacity */
+    blk_size = llint_grow_size_t(current_size, size > 0 ? size : 0, 32768);
+    x_argvblk = blk_size != (size_t) -1 ? (char *) malloct(blk_size) : NULL;
     if (x_argvblk == NULL) {
       HTS_PANIC_PRINTF("Error, not enough memory");
       htsmain_free();
       return -1;
     }
-    x_argvblk_size = (size_t) (current_size + 32768);
+    x_argvblk_size = blk_size;
     x_argvblk[0] = '\0';
     x_ptr = 0;
 
@@ -1456,20 +1457,29 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                     FILE *fp = FOPEN(argv[na], "rb");
 
                     if (fp != NULL) {
-                      int cl = (int) strlen(url);
+                      size_t cl = strlen(url);
+                      const size_t fzs = llint_to_size_t(fz);
+                      const size_t capa = llint_grow_size_t(cl, fz, 8192);
 
-                      ensureUrlCapacity(url, url_sz, cl + fz + 8192);
+                      if (capa == (size_t) -1) {
+                        fclose(fp);
+                        HTS_PANIC_PRINTF("File url list too large");
+                        htsmain_free();
+                        return -1;
+                      }
+                      ensureUrlCapacity(url, url_sz, capa);
                       if (cl > 0) {     /* don't stick! (3.43) */
                         url[cl] = ' ';
                         cl++;
                       }
-                      if (fread(url + cl, 1, fz, fp) != fz) {
+                      if (fread(url + cl, 1, fzs, fp) != fzs) {
+                        fclose(fp);
                         HTS_PANIC_PRINTF("File url list could not be read");
                         htsmain_free();
                         return -1;
                       }
                       fclose(fp);
-                      *(url + cl + fz) = '\0';
+                      *(url + cl + fzs) = '\0';
                     }
                   }
                 }
@@ -1785,6 +1795,26 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->warc_file, WARC_AUTONAME);
                 }
                 break;
+              case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps
+                if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap
+                  com++;
+                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
+                    HTS_PANIC_PRINTF(
+                        "Option sitemap-url needs a blank space and a URL");
+                    htsmain_free();
+                    return -1;
+                  }
+                  na++;
+                  if (strlen(argv[na]) >= HTS_URLMAXSIZE) {
+                    HTS_PANIC_PRINTF("Sitemap URL too long");
+                    htsmain_free();
+                    return -1;
+                  }
+                  StringCopy(opt->sitemap_url, argv[na]);
+                } else { // --sitemap: robots.txt probe, then /sitemap.xml
+                  opt->sitemap = HTS_TRUE;
+                }
+                break;
               case 'Y': // why: explain the filter verdict for a URL, no crawl
                 if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                   HTS_PANIC_PRINTF("Option why needs a blank space and a URL");
@@ -2296,8 +2326,8 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
       } else {                  // URL/filters
         char catbuff[CATBUFF_SIZE];
-        const int urlSize = (int) strlen(argv[na]);
-        const int capa = (int) (strlen(url) + urlSize + 32);
+        const size_t urlSize = strlen(argv[na]);
+        const size_t capa = strlen(url) + urlSize + 32;
 
         assertf(urlSize < HTS_URLMAXSIZE);
         if (urlSize < HTS_URLMAXSIZE) {

src/htsftp.c

@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
       // folding a nonsense port into 1..65535 fetches one the link never named;
       // an empty "host:" just means the default (#614)
       if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
-        snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
+        htsblk_failf(&back->r, "Invalid port: %s", a + 1);
         back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
         _HALT_FTP return 0;
       }
@@ -262,8 +262,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
     // récupérer adresse résolue
     strcpybuff(back->info, "host name");
     if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
-      snprintf(back->r.msg, sizeof(back->r.msg),
-               "Unable to get server's address: %s", error);
+      htsblk_failf(&back->r, "Unable to get server's address: %s", error);
       back->r.statuscode = STATUSCODE_NON_FATAL;
       _HALT_FTP return 0;
     }
@@ -332,18 +331,15 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
             }
 
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }
         } else {
-          snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
-                   linejmp(line));
+          htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
           back->r.statuscode = STATUSCODE_INVALID;
         }
       } else {
-        snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
-                 linejmp(line));
+        htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
         back->r.statuscode = STATUSCODE_INVALID;
       }
 
@@ -410,8 +406,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
             }
             // -- fin analyse de l'adresse IP et du port --
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         } else {
@@ -442,13 +437,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 }
               }
             } else {
-              snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
-                       linejmp(line));
+              htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
               back->r.statuscode = STATUSCODE_INVALID;
             }
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         }
@@ -554,8 +547,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                   deletesoc(soc_dat);
                   soc_dat = INVALID_SOCKET;
                   //
-                  snprintf(back->r.msg, sizeof(back->r.msg),
-                           "RETR command error: %s", linejmp(line));
+                  htsblk_failf(&back->r, "RETR command error: %s",
+                               linejmp(line));
                   back->r.statuscode = STATUSCODE_INVALID;
                 }               // sinon on est prêts
               } else {
@@ -573,13 +566,12 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
               back->r.statuscode = STATUSCODE_INVALID;
             }                   // sinon on est prêts
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg),
-                     "Unable to resolve IP %s: %s", adr_ip, error);
+            htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
+                         error);
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         } else {
-          snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
-                   linejmp(line));
+          htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
           back->r.statuscode = STATUSCODE_INVALID;
         }                       // sinon on est prêts
 #else
@@ -603,13 +595,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 back->r.statuscode = STATUSCODE_INVALID;
               }
             } else {
-              snprintf(back->r.msg, sizeof(back->r.msg),
-                       "RETR command error: %s", linejmp(line));
+              htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
               back->r.statuscode = STATUSCODE_INVALID;
             }
           } else {
-            snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
-                     linejmp(line));
+            htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }
 #ifdef _WIN32
@@ -652,8 +642,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 len = 0;        // fin
                 break;
               case 0:
-                snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
-                         timeout);
+                htsblk_failf(&back->r, "Time out (%d)", timeout);
                 back->r.statuscode = STATUSCODE_INVALID;
                 len = 0;        // fin
                 break;
@@ -716,8 +705,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 strcpybuff(back->r.msg, "OK");
                 back->r.statuscode = HTTP_OK;
               } else {
-                snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
-                         linejmp(line));
+                htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
                 back->r.statuscode = STATUSCODE_INVALID;
               }
             } else {

src/htsglobal.h

@@ -72,7 +72,8 @@ Please visit our Website: http://www.httrack.com
    HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
    static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
    compiler type-checks the format string at argument index fmt against the
-   varargs starting at arg. */
+   varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
+   error signal, so dropping it is a bug; a (void) cast does not silence it. */
 #ifndef HTS_UNUSED
 #ifdef __GNUC__
 #define HTS_UNUSED __attribute__((unused))
@@ -80,10 +81,13 @@ Please visit our Website: http://www.httrack.com
 #define HTS_STATIC static __attribute__((unused))
 
 #define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
+
+#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
 #else
 #define HTS_UNUSED
 #define HTS_STATIC static
 #define HTS_PRINTF_FUN(fmt, arg)
+#define HTS_CHECK_RESULT
 #endif
 #endif
 
@@ -336,16 +340,20 @@ typedef int INTsys;
 #endif
 
 /* Socket-handle type. An unsigned integer wide enough for a Windows SOCKET;
-   a plain int file descriptor on POSIX. */
+   a plain int file descriptor on POSIX. T_SOCP is its printf conversion,
+   '%' included: unsigned __int64 on Win64 must not be printed with "%d". */
 #ifdef _WIN32
 #if defined(_WIN64)
 
 typedef unsigned __int64 T_SOC;
+#define T_SOCP "%" PRIu64
 #else
 typedef unsigned __int32 T_SOC;
+#define T_SOCP "%" PRIu32
 #endif
 #else
 typedef int T_SOC;
+#define T_SOCP "%d"
 #endif
 
 /* Buffer size for a printed network address (IPv4 or IPv6, NUL included). */

src/htshelp.c

@@ -124,7 +124,9 @@ typedef struct help_wizard_buffers {
   char stropt[2048];        // options
   char stropt2[2048];       // options longues
   char strwild[2048];       // wildcards
-  char cmd[4096];
+  /* holds all four of the above plus separators: at 4096 a long answer set
+     clipped the filters off the command line */
+  char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
   char str[256];
   char *argv[256];
 } help_wizard_buffers;
@@ -156,9 +158,7 @@ void help_wizard(httrackp * opt) {
   char *a;
 
   //
-  if (urls == NULL || mainpath == NULL || projname == NULL || stropt == NULL
-      || stropt2 == NULL || strwild == NULL || cmd == NULL || str == NULL
-      || argv == NULL) {
+  if (buffers == NULL) {
     fprintf(stderr, "* memory exhausted in %s, line %d\n", __FILE__, __LINE__);
     return;
   }
@@ -251,6 +251,7 @@ void help_wizard(httrackp * opt) {
       strcatbuff(stropt2, "--update ");
       break;
     case 0:
+      freet(buffers);
       return;
       break;
     }
@@ -309,14 +310,23 @@ void help_wizard(httrackp * opt) {
       printf("\n");
       if (strlen(stropt) == 1)
         stropt[0] = '\0';       // aucune
-      snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
+      /* the tail is the filter list, and cmd is split into the argv handed to
+         hts_main() below: a clipped line would silently widen the crawl */
+      if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
+        printf("* command line too long (%d bytes max)\n",
+               (int) sizeof(cmd) - 1);
+        freet(buffers);
+        return;
+      }
       printf("---> Wizard command line: httrack %s\n\n", cmd);
       printf("Ready to launch the mirror? (Y/n) :");
       fflush(stdout);
       linput(stdin, str, 250);
       if (strnotempty(str)) {
-        if (!((str[0] == 'y') || (str[0] == 'Y')))
+        if (!((str[0] == 'y') || (str[0] == 'Y'))) {
+          freet(buffers);
           return;
+        }
       }
       printf("\n");
 
@@ -340,7 +350,7 @@ void help_wizard(httrackp * opt) {
   }
 
   /* Free buffers */
-  free(buffers);
+  freet(buffers);
 #undef urls
 #undef mainpath
 #undef projname
@@ -423,7 +433,8 @@ void help_catchurl(const char *dest_path) {
       }
       // former URL!
       {
-        char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
+        /* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
+        char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
 
         inplace_escape_check_url(dest, sizeof(dest));
         snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);
@@ -515,6 +526,11 @@ void help(const char *app, int more) {
     (" %L <file> add all URL located in this text file (one URL per line)");
   infomsg
     (" %S <file> add all scan rules located in this text file (one scan rule per line)");
+  infomsg(" %m  seed the crawl from the site's sitemap (robots.txt Sitemap:, "
+          "then /sitemap.xml); --sitemap-url URL names one explicitly. A "
+          "sitemap you name, or one the site declares, is fetched even under "
+          "robots.txt Disallow; only the guessed /sitemap.xml obeys it. The "
+          "URLs found still pass every filter and scope rule");
   infomsg("");
   infomsg("Build options:");
   infomsg("  NN structure type (0 *original structure, 1+: see below)");

src/htslib.c

@@ -36,6 +36,7 @@ Please visit our Website: http://www.httrack.com
 // Fichier librairie .c
 
 #include "htscore.h"
+#include "htssitemap.h"
 #include "htswarc.h"
 
 /* specific definitions */
@@ -704,18 +705,16 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
   /* Check for errors */
   if (soc == INVALID_SOCKET) {
     if (retour) {
-      if (retour->msg) {
-        if (!strnotempty(retour->msg)) {
+      if (!strnotempty(retour->msg)) {
 #ifdef _WIN32
-          int last_errno = WSAGetLastError();
+        int last_errno = WSAGetLastError();
 
-          sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
+        htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
 #else
-          int last_errno = errno;
+        int last_errno = errno;
 
-          sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
+        htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
 #endif
-        }
       }
     }
   }
@@ -2207,7 +2206,7 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #if DEBUG
       printf("erreur gethostbyname\n");
 #endif
-      if (retour && retour->msg) {
+      if (retour != NULL) {
 #ifdef _WIN32
         snprintf(retour->msg, sizeof(retour->msg),
                  "Unable to get server's address: %s", error);
@@ -2238,17 +2237,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
     DEBUG_W("socket()=%d\n" _(int) soc);
 #endif
     if (soc == INVALID_SOCKET) {
-      if (retour && retour->msg) {
+      if (retour != NULL) {
 #ifdef _WIN32
         int last_errno = WSAGetLastError();
 
-        sprintf(retour->msg, "Unable to create a socket: %s",
-                strerror(last_errno));
+        htsblk_failf(retour, "Unable to create a socket: %s",
+                     strerror(last_errno));
 #else
         int last_errno = errno;
 
-        sprintf(retour->msg, "Unable to create a socket: %s",
-                strerror(last_errno));
+        htsblk_failf(retour, "Unable to create a socket: %s",
+                     strerror(last_errno));
 #endif
       }
       return INVALID_SOCKET;    // erreur création socket impossible
@@ -2262,17 +2261,8 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
                              &bind_addr, &error) == NULL
           || bind(soc, &SOCaddr_sockaddr(bind_addr), 
                   SOCaddr_size(bind_addr)) != 0) {
-        if (retour && retour->msg) {
-#ifdef _WIN32
-          snprintf(retour->msg, sizeof(retour->msg),
-                   "Unable to bind the specificied server address: %s",
-                   error);
-#else
-          snprintf(retour->msg, sizeof(retour->msg),
-                   "Unable to bind the specificied server address: %s",
-                   error);
-#endif
-        }
+        snprintf(retour->msg, sizeof(retour->msg),
+                 "Unable to bind the specificied server address: %s", error);
         deletesoc(soc);
         return INVALID_SOCKET;
       }
@@ -2319,17 +2309,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #if HDEBUG
         printf("unable to connect!\n");
 #endif
-        if (retour != NULL && retour->msg) {
+        if (retour != NULL) {
 #ifdef _WIN32
           const int last_errno = WSAGetLastError();
 
-          sprintf(retour->msg, "Unable to connect to the server: %s",
-                  strerror(last_errno));
+          htsblk_failf(retour, "Unable to connect to the server: %s",
+                       strerror(last_errno));
 #else
           const int last_errno = errno;
 
-          sprintf(retour->msg, "Unable to connect to the server: %s",
-                  strerror(last_errno));
+          htsblk_failf(retour, "Unable to connect to the server: %s",
+                       strerror(last_errno));
 #endif
         }
         /* Close the socket and notify the error!!! */
@@ -2548,6 +2538,15 @@ void fil_simplifie(char *f) {
   }
 }
 
+void htsblk_failf(htsblk *r, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  // deliberate clip: the reason is quoted from a remote reply
+  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
+  va_end(args);
+}
+
 // fermer liaison fichier ou socket
 void deletehttp(htsblk * r) {
 #if HTS_DEBUG_CLOSESOCK
@@ -2598,13 +2597,15 @@ void deletesoc(T_SOC soc) {
     if (closesocket(soc) != 0) {
       int err = WSAGetLastError();
 
-      fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
+      fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
+              strerror(err));
     }
 #else
     if (close(soc) != 0) {
       const int err = errno;
 
-      fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
+      fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
+              strerror(err));
     }
 #endif
 #if HTS_WIDE_DEBUG
@@ -3017,7 +3018,7 @@ int finput(T_SOC fd, char *s, int max) {
 }
 
 // Like linput, but in memory (optimized)
-int binput(char *buff, char *s, int max) {
+int binput(const char *buff, char *s, int max) {
   int count = 0;
   int destCount = 0;
 
@@ -6019,6 +6020,7 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   StringCopy(opt->strip_query, "");
   StringCopy(opt->cookies_file, "");
   StringCopy(opt->warc_file, "");
+  StringCopy(opt->sitemap_url, "");
   opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
@@ -6172,6 +6174,8 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->cookies_file);
     StringFree(opt->why_url);
     StringFree(opt->warc_file);
+    StringFree(opt->sitemap_url);
+    hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */
 
     StringFree(opt->path_html);
     StringFree(opt->path_html_utf8);

src/htslib.h

@@ -199,6 +199,10 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
    etc.). */
 T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
                    int waitconnect, int addr_index, int *addr_count);
+/* Clips the formatted failure reason into r->msg, which also round-trips
+   through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
+void htsblk_failf(htsblk *r, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
+
 HTS_INLINE void deletehttp(htsblk * r);
 HTS_INLINE int deleteaddr(htsblk * r);
 HTS_INLINE void deletesoc(T_SOC soc);
@@ -262,7 +266,7 @@ HTS_INLINE void time_rfc822_local(char *s, struct tm *A);
 
 HTS_INLINE int sendc(htsblk * r, const char *s);
 int finput(T_SOC fd, char *s, int max);
-int binput(char *buff, char *s, int max);
+int binput(const char *buff, char *s, int max);
 int linput(FILE * fp, char *s, int max);
 int linputsoc(T_SOC soc, char *s, int max);
 int linputsoc_t(T_SOC soc, char *s, int max, int timeout);
@@ -607,6 +611,21 @@ static HTS_UNUSED size_t llint_to_size_t(LLint o) {
   }
 }
 
+/* Capacity for @p used bytes plus @p extra more plus @p slack spare;
+   (size_t) -1 if the total exceeds (size_t) -2 or @p extra is negative
+   (llint_to_size_t() would map that to a huge valid-looking size). */
+static HTS_UNUSED size_t llint_grow_size_t(size_t used, LLint extra,
+                                           size_t slack) {
+  const size_t max = (size_t) -2; /* (size_t) -1 is the error value */
+  const size_t e = extra >= 0 ? llint_to_size_t(extra) : (size_t) -1;
+
+  if (e == (size_t) -1 || used > max || slack > max - used ||
+      e > max - used - slack) {
+    return (size_t) -1;
+  }
+  return used + e + slack;
+}
+
 /* dirent() compatibility */
 #ifdef _WIN32
 /* Holds a UTF-8 d_name: MAX_PATH (260) UTF-16 units expand to <=3 bytes each.

src/htsopt.h

@@ -547,6 +547,13 @@ struct httrackp {
                              archive. Tail: ABI */
   hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip
                               (implies --warc + --warc-cdx). Tail: ABI */
+  hts_boolean sitemap;   /**< --sitemap: probe the start host's robots.txt for
+                              Sitemap: lines, else /sitemap.xml. Tail: ABI */
+  String sitemap_url;    /**< --sitemap-url: sitemap to ingest. Tail: ABI */
+  /* Live state, not an option: copy_htsopt must leave it alone. It sits here
+     rather than in htsoptstate because that struct is embedded by value, so
+     growing it would shift every httrackp field declared after it. */
+  void *sitemap_state; /**< hts_sitemap_state*, or NULL. Tail: ABI */
 };
 
 /* Running statistics for a mirror. */

src/htsparse.c

@@ -4583,7 +4583,7 @@ int hts_wait_delayed(htsmoduleStruct * str, lien_adrfilsave *afs,
             /* seen as in error */
             in_error = back[b].r.statuscode;
             in_error_msg[0] = 0;
-            strncat(in_error_msg, back[b].r.msg, sizeof(in_error_msg) - 1);
+            strncatbuff(in_error_msg, back[b].r.msg, sizeof(in_error_msg) - 1);
             in_error_size = back[b].r.totalsize;
             /* don't break, even with "don't take error pages" switch, because we need to process the slot anyway (and cache the error) */
           }

src/htsproxy.c

@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
   if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
     code = 0;
   if (code < 200 || code >= 300) {
-    snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
-             strnotempty(line) ? line : "(no status)");
+    htsblk_failf(retour, "proxy CONNECT refused: %s",
+                 strnotempty(line) ? line : "(no status)");
     return 0;
   }
 

src/htsrobots.c

@@ -147,7 +147,8 @@ static void robots_blob_add(char *blob, size_t blobsize, char marker,
 
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow) {
+                  hts_boolean keep_root_disallow, char *sitemaps,
+                  size_t sitemapsize) {
   size_t bptr = 0;
   int record = 0;
   char BIGSTK line[1024];
@@ -156,6 +157,8 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
   blob[0] = '\0';
   if (info != NULL && infosize > 0)
     info[0] = '\0';
+  if (sitemaps != NULL && sitemapsize > 0)
+    sitemaps[0] = '\0';
 #if DEBUG_ROBOTS
   printf("robots.txt dump:\n%s\n", body);
 #endif
@@ -172,7 +175,19 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
       line[llen - 1] = '\0';
       llen--;
     }
-    if (strfield(line, "user-agent:")) {
+    if (sitemaps != NULL && strfield(line, "sitemap:")) {
+      // group-independent record (RFC 9309): collected whatever the group
+      char *a = line + 8;
+
+      while (is_realspace(*a))
+        a++;
+      /* A line at the buffer limit was truncated: a half URL is not one. */
+      if (strnotempty(a) && strlen(line) < sizeof(line) - 3 &&
+          strlen(a) + 2 < sitemapsize - strlen(sitemaps)) {
+        strlcatbuff(sitemaps, a, sitemapsize);
+        strlcatbuff(sitemaps, "\n", sitemapsize);
+      }
+    } else if (strfield(line, "user-agent:")) {
       char *a = line + 11;
 
       while (is_realspace(*a))

src/htsrobots.h

@@ -56,10 +56,12 @@ int checkrobots(robots_wizard * robots, const char *adr, const char *fil);
 void checkrobots_free(robots_wizard * robots);
 int checkrobots_set(robots_wizard * robots, const char *adr, const char *data);
 /* Parse robots.txt `body` for `adr`, storing the HTTrack group's rules; `info`
-   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /". */
+   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /", and
+   `sitemaps` (optional) collects the Sitemap: URLs, one per line. */
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow);
+                  hts_boolean keep_root_disallow, char *sitemaps,
+                  size_t sitemapsize);
 #endif
 
 #endif

src/htssafe.h

@@ -33,6 +33,7 @@ Please visit our Website: http://www.httrack.com
 #ifndef HTSSAFE_DEFH
 #define HTSSAFE_DEFH
 
+#include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -116,6 +117,15 @@ static HTS_UNUSED void abortf_(const char *exp, const char *file, int line) {
 #endif
 #define HTS_IS_NOT_CHAR_BUFFER(VAR) (!HTS_IS_CHAR_BUFFER(VAR))
 
+/* Source capacity for the buff() family, (size_t)-1 when unknown; sizeof of the
+   TYPE keeps a decayed operand ("buf + 1") off -Wsizeof-array-decay. */
+#if (defined(__GNUC__) && !defined(__cplusplus))
+#define HTS_SIZEOF_SRC_(B)                                                     \
+  (HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(__typeof__(B)))
+#else
+#define HTS_SIZEOF_SRC_(B) (HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B))
+#endif
+
 /* Compile-time checks. */
 static HTS_UNUSED void htssafe_compile_time_check_(void) {
   char array[32];
@@ -205,19 +215,17 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #if (defined(__GNUC__) && !defined(__cplusplus))
 
 #define strncatbuff(A, B, N)                                                   \
-  __builtin_choose_expr(                                                       \
-      HTS_IS_CHAR_BUFFER(A),                                                   \
-      strncat_safe_(A, sizeof(A), B,                                           \
-                    HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N,    \
-                    "overflow while appending '" #B "' to '" #A "'", __FILE__, \
-                    __LINE__),                                                 \
-      strncatbuff_ptr_((A), (B), (N)))
+  __builtin_choose_expr(HTS_IS_CHAR_BUFFER(A),                                 \
+                        strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), N,  \
+                                      "overflow while appending '" #B          \
+                                      "' to '" #A "'",                         \
+                                      __FILE__, __LINE__),                     \
+                        strncatbuff_ptr_((A), (B), (N)))
 #else
 #define strncatbuff(A, B, N)                                                   \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strncat(A, B, N)                                                      \
-       : strncat_safe_(A, sizeof(A), B,                                        \
-                       HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N, \
+       : strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), N,                 \
                        "overflow while appending '" #B "' to '" #A "'",        \
                        __FILE__, __LINE__))
 #endif
@@ -232,9 +240,7 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #define strcatbuff(A, B)                                                       \
   __builtin_choose_expr(                                                       \
       HTS_IS_CHAR_BUFFER(A),                                                   \
-      strncat_safe_(A, sizeof(A), B,                                           \
-                    HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),       \
-                    (size_t) -1,                                               \
+      strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), (size_t) -1,          \
                     "overflow while appending '" #B "' to '" #A "'", __FILE__, \
                     __LINE__),                                                 \
       strcatbuff_ptr_((A), (B)))
@@ -242,9 +248,7 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #define strcatbuff(A, B)                                                       \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strcat(A, B)                                                          \
-       : strncat_safe_(A, sizeof(A), B,                                        \
-                       HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),    \
-                       (size_t) -1,                                            \
+       : strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), (size_t) -1,       \
                        "overflow while appending '" #B "' to '" #A "'",        \
                        __FILE__, __LINE__))
 #endif
@@ -257,19 +261,17 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #if (defined(__GNUC__) && !defined(__cplusplus))
 
 #define strcpybuff(A, B)                                                       \
-  __builtin_choose_expr(                                                       \
-      HTS_IS_CHAR_BUFFER(A),                                                   \
-      strcpy_safe_(A, sizeof(A), B,                                            \
-                   HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),        \
-                   "overflow while copying '" #B "' to '" #A "'", __FILE__,    \
-                   __LINE__),                                                  \
-      strcpybuff_ptr_((A), (B)))
+  __builtin_choose_expr(HTS_IS_CHAR_BUFFER(A),                                 \
+                        strcpy_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B),      \
+                                     "overflow while copying '" #B "' to '" #A \
+                                     "'",                                      \
+                                     __FILE__, __LINE__),                      \
+                        strcpybuff_ptr_((A), (B)))
 #else
 #define strcpybuff(A, B)                                                       \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strcpy(A, B)                                                          \
-       : strcpy_safe_(A, sizeof(A), B,                                         \
-                      HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),     \
+       : strcpy_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B),                     \
                       "overflow while copying '" #B "' to '" #A "'", __FILE__, \
                       __LINE__))
 #endif
@@ -286,24 +288,24 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
  * Append characters of "B" to "A", "A" having a maximum capacity of "S".
  */
 #define strlcatbuff(A, B, S)                                                   \
-  strncat_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),  \
-                (size_t) -1, "overflow while appending '" #B "' to '" #A "'",  \
-                __FILE__, __LINE__)
+  strncat_safe_(A, S, B, HTS_SIZEOF_SRC_(B), (size_t) -1,                      \
+                "overflow while appending '" #B "' to '" #A "'", __FILE__,     \
+                __LINE__)
 
 /**
  * Append at most "N" characters of "B" to "A", "A" having a maximum capacity
  * of "S".
  */
 #define strlncatbuff(A, B, S, N)                                               \
-  strncat_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),  \
-                N, "overflow while appending '" #B "' to '" #A "'", __FILE__,  \
+  strncat_safe_(A, S, B, HTS_SIZEOF_SRC_(B), N,                                \
+                "overflow while appending '" #B "' to '" #A "'", __FILE__,     \
                 __LINE__)
 
 /**
  * Copy characters of "B" to "A", "A" having a maximum capacity of "S".
  */
 #define strlcpybuff(A, B, S)                                                   \
-  strcpy_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),   \
+  strcpy_safe_(A, S, B, HTS_SIZEOF_SRC_(B),                                    \
                "overflow while copying '" #B "' to '" #A "'", __FILE__,        \
                __LINE__)
 
@@ -422,7 +424,9 @@ static HTS_INLINE HTS_UNUSED htsbuff htsbuff_ptr_(char *buf, size_t cap) {
  */
 static HTS_INLINE HTS_UNUSED void htsbuff_catn(htsbuff *b, const char *s,
                                                size_t n) {
-  const size_t add = strnlen(s, n);
+  /* the (size_t)-1 "no limit" sentinel would reach strnlen as a bound past
+     PTRDIFF_MAX */
+  const size_t add = n != (size_t) -1 ? strnlen(s, n) : strlen(s);
   /* Overflow-safe: keep the (potentially huge) 'add' alone on one side. The
      maintained invariant len < cap makes 'cap - len' >= 1 (no underflow), so
      'add < cap - len' cannot wrap the way 'len + add < cap' could. */
@@ -456,6 +460,49 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
   return b->buf;
 }
 
+/**
+ * Callers that deliberately ignore truncation use this instead of
+ * slprintfbuff(), so it is not HTS_CHECK_RESULT.
+ */
+static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
+    vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
+  int ret;
+
+  assertf(dest != NULL && size != 0);
+  ret = vsnprintf(dest, size, fmt, args);
+  /* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
+  dest[size - 1] = '\0';
+  return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
+}
+
+/**
+ * Formatted print into dest (capacity size, NUL included), truncating to fit
+ * and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
+ * result is the only truncation signal, so it must be acted on. Unlike
+ * strcpybuff() it never aborts, so it suits text built from remote input.
+ */
+static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
+    slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
+  va_list args;
+  hts_boolean ret;
+
+  va_start(args, fmt);
+  ret = vslprintfbuff(dest, size, fmt, args);
+  va_end(args);
+  return ret;
+}
+
+/**
+ * slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
+ * On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
+ */
+#if (defined(__GNUC__) && !defined(__cplusplus))
+#define sprintfbuff(ARR, ...)                                                  \
+  slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
+#else
+#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
+#endif
+
 /* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
    added bounds checking. freet() also NULLs the freed pointer and tolerates
    NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

src/htsselftest.c

@@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsback.h"
 #include "htsdefines.h"
 #include "htslib.h"
 #include "htsalias.h"
@@ -51,12 +52,14 @@ Please visit our Website: http://www.httrack.com
 #include "htscache_selftest.h"
 #include "htsdns_selftest.h"
 #include "htscharset.h"
+#include "htscmdline.h"
 #include "htsencoding.h"
 #include "htsftp.h"
 #include "htsmd5.h"
 #include "htssniff.h"
 #include "htscodec.h"
 #include "htsproxy.h"
+#include "htssitemap.h"
 #include "htswarc.h"
 #if HTS_USEZLIB
 #include "htszlib.h"
@@ -455,6 +458,12 @@ static void basic_selftests(void) {
     // link one level up -> a "../" prefix
     assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
     assertf(strcmp(s, "../a.html") == 0);
+    // an empty current path: the trim used to walk off the front of it, which
+    // "?x" reaches too because the query pre-pass hands on the part before it
+    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
+    assertf(strcmp(s, "dir/page.html") == 0);
+    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
+    assertf(strcmp(s, "dir/page.html") == 0);
   }
 }
 
@@ -481,6 +490,28 @@ static int string_safety_selftests(void) {
   if (strcmp(buf, "abcd") != 0)
     return 1;
 
+  /* A decayed source has no known capacity, so the whole tail must land; a
+     sizeof(char*) capacity would abort here instead. */
+  {
+    char src[32] = "0123456789abcdefghij";
+    char dst[32];
+
+    strcpybuff(dst, src + 1);
+    if (strcmp(dst, "123456789abcdefghij") != 0)
+      return 1;
+  }
+
+  /* Truncating append: stops at N without aborting, what the status-message
+     call sites rely on. */
+  {
+    char dst[10]; /* never sizeof(char*), or MSVC reads it as a pointer */
+
+    dst[0] = '\0';
+    strncatbuff(dst, "abcdefghijkl", sizeof(dst) - 1);
+    if (strcmp(dst, "abcdefghi") != 0)
+      return 1;
+  }
+
   /* strlcpybuff: explicit-capacity copy into a pointer destination, the form
      the migration moves toward */
   {
@@ -546,6 +577,154 @@ static int string_safety_selftests(void) {
       return 1;
   }
 
+  /* sprintfbuff: truncate-and-report. Must never abort (its callers format
+     remote banners) nor write past the array, which the canary catches. */
+  {
+    struct {
+      char dst[8];
+      char canary[8];
+    } s;
+
+    const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
+
+    /* repoison before every call, or an implementation that measures first and
+       writes nothing "passes" the truncating cases on the previous content */
+#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
+
+    memset(&s, '#', sizeof(s));
+    if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
+      return 1;
+
+    /* exact fit: 7 characters plus the NUL */
+    POISON_DST();
+    if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
+      return 1;
+
+    /* one over, then far over: truncated to the prefix, terminated, reported */
+    POISON_DST();
+    if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
+      return 1;
+    POISON_DST();
+    if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
+      return 1;
+
+    /* explicit-capacity form, down to the degenerate size 1 */
+    {
+      char *const p = s.dst;
+
+      POISON_DST();
+      if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
+        return 1;
+      POISON_DST();
+      if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
+        return 1;
+    }
+#undef POISON_DST
+
+    if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
+      return 1;
+  }
+
+  /* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
+     touches nothing else in the block */
+  {
+    htsblk r;
+    char expect[sizeof(r.msg)];
+    char big[4 * sizeof(r.msg)];
+
+    /* contenttype abuts msg, so a one-past-the-end store lands in it rather
+       than in padding. Poison it: a stray NUL is invisible against zeroes,
+       and a stray NUL is exactly what an off-by-one terminator writes. */
+#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
+
+    memset(&r, 0, sizeof(r));
+    memset(r.contenttype, '#', sizeof(r.contenttype));
+    r.statuscode = 1234;
+
+    memset(r.msg, '#', sizeof(r.msg));
+    htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
+    if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
+      return 1;
+
+    /* exact fit: capacity - 1 characters plus the NUL */
+    memset(expect, 'y', sizeof(expect) - 1);
+    expect[sizeof(expect) - 1] = '\0';
+    memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
+    memset(r.msg, '#', sizeof(r.msg));
+    htsblk_failf(&r, "%s", expect);
+    if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
+      return 1;
+
+    /* far over: the expected bytes differ from the cases above, so writing
+       nothing cannot pass on the leftovers */
+    memset(big, 'z', sizeof(big) - 1);
+    big[sizeof(big) - 1] = '\0';
+    memset(expect, 'z', sizeof(expect) - 1);
+    expect[sizeof(expect) - 1] = '\0';
+    memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
+
+    memset(r.msg, '#', sizeof(r.msg));
+    htsblk_failf(&r, "Bad user name: %s", big);
+    if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
+      return 1;
+#undef NEIGHBOURS_INTACT
+  }
+
+  /* back_read_ftp_result: the helper's result file is external input, so an
+     over-long message must stop at msg[]'s capacity */
+  {
+    htsblk r;
+    size_t k;
+
+    /* poisoned so a short message cannot pass on leftovers, and so a stray
+       NUL past msg[] is visible in the neighbour */
+#define FTP_RESULT_CASE(BODY)                                                  \
+  do {                                                                         \
+    FILE *fp_ = tmpfile();                                                     \
+                                                                               \
+    if (fp_ == NULL)                                                           \
+      return 1;                                                                \
+    BODY;                                                                      \
+    rewind(fp_);                                                               \
+    memset(&r, 0, sizeof(r));                                                  \
+    memset(r.msg, '#', sizeof(r.msg));                                         \
+    memset(r.contenttype, '#', sizeof(r.contenttype));                         \
+    back_read_ftp_result(fp_, &r);                                             \
+    fclose(fp_);                                                               \
+    if (r.contenttype[0] != '#')                                               \
+      return 1;                                                                \
+  } while (0)
+
+    /* over capacity: clipped to 79 payload bytes plus the NUL */
+    FTP_RESULT_CASE({
+      fprintf(fp_, "226 ");
+      for (k = 0; k < 4 * sizeof(r.msg); k++)
+        fputc('q', fp_);
+    });
+    if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
+      return 1;
+    for (k = 0; k < sizeof(r.msg) - 1; k++) {
+      if (r.msg[k] != 'q')
+        return 1;
+    }
+
+    /* well under capacity: nothing padded, nothing eaten off the end */
+    FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
+    if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
+      return 1;
+
+    /* a byte over 0x7f must not read as EOF and cut the message short */
+    FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
+    if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
+      return 1;
+
+    /* unparseable status: the message still loads, the code reports failure */
+    FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
+    if (r.statuscode != STATUSCODE_INVALID)
+      return 1;
+#undef FTP_RESULT_CASE
+  }
+
   /* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
      n_eval == 2 on a double-eval macro. */
   {
@@ -1104,6 +1283,128 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+// hts_split_cmdline(): the vector must grow with the argument count, and a
+// quote inside a value must not end the argument and hand -V to the parser.
+static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
+  char line[512];
+  char **args;
+  int nargs = 0;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+  // control: every separator splits, and argv[0] is the program name
+  strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(args != NULL && nargs == 6);
+  assertf(args[nargs] == NULL); // callers may walk to the terminator
+  assertf(strcmp(args[0], "httrack") == 0);
+  assertf(strcmp(args[1], "http://x/") == 0);
+  assertf(strcmp(args[2], "--quiet") == 0);
+  assertf(strcmp(args[3], "-c8") == 0);
+  assertf(strcmp(args[4], "-O") == 0);
+  assertf(strcmp(args[5], "out") == 0);
+  freet(args);
+
+  // the template pads with whitespace: empty arguments are kept (the engine
+  // skips them), so the count is one per separator
+  strcpybuff(line, "httrack  --quiet");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3 && args[1][0] == '\0');
+  assertf(strcmp(args[2], "--quiet") == 0);
+  freet(args);
+
+  // a quoted run keeps both its spaces and its quotes: the engine unquotes
+  strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 4);
+  assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
+  assertf(strcmp(args[3], "-c8") == 0);
+  freet(args);
+
+  // an escaped quote is a literal quote, not the end of the argument: the
+  // engine strips only the outer pair
+  strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 4);
+  assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
+  assertf(strcmp(args[3], "-c8") == 0);
+  freet(args);
+
+  // \\ is a literal backslash, so a Windows path survives
+  strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3);
+  assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
+  freet(args);
+
+  // outside a quoted run a backslash is literal: the url and wildcard-filter
+  // fields, which the wizard cannot quote, read as before
+  strcpybuff(line, "httrack -*\\** +*.png");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3);
+  assertf(strcmp(args[1], "-*\\**") == 0);
+  assertf(strcmp(args[2], "+*.png") == 0);
+  freet(args);
+
+  // a quoted run leaves slots unused, so the terminator has to be written and
+  // not inherited: size the vector from a full line first, so freeing it hands
+  // the same chunk back with stale pointers in those slots
+  strcpybuff(line, "httrack a b c d e");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 6);
+  freet(args);
+  strcpybuff(line, "httrack \"a b c d e\"");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 2);
+  assertf(args[nargs] == NULL);
+  freet(args);
+
+  // an unterminated quote protects the rest of the line, as one argument
+  strcpybuff(line, "httrack --footer \"unbalanced -V x");
+  args = hts_split_cmdline(line, &nargs);
+  assertf(nargs == 3);
+  assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
+  freet(args);
+
+  // past the 1024 entries the vector used to hold: distinct arguments, so a
+  // write beyond the allocation cannot read back as the expected parse
+  {
+    const int n = 2000;
+    const size_t size = 16 * (size_t) n + 16;
+    char *big = malloct(size);
+    size_t pos = 0;
+    int i;
+
+    assertf(big != NULL);
+    pos = (size_t) snprintf(big, size, "httrack");
+    assertf(pos < size);
+    for (i = 0; i < n; i++) {
+      // snprintf returns what it wanted to write, so accumulating it blind
+      // would let the next size argument wrap
+      const int len = snprintf(big + pos, size - pos, " a%d", i);
+
+      assertf(len > 0 && (size_t) len < size - pos);
+      pos += (size_t) len;
+    }
+    args = hts_split_cmdline(big, &nargs);
+    assertf(args != NULL && nargs == n + 1);
+    assertf(args[nargs] == NULL);
+    for (i = 0; i < n; i++) {
+      char expect[16];
+
+      snprintf(expect, sizeof(expect), "a%d", i);
+      assertf(strcmp(args[i + 1], expect) == 0);
+    }
+    freet(args);
+    freet(big);
+  }
+
+  printf("cmdline-split self-test OK\n");
+  return 0;
+}
+
 static int st_hashtable(httrackp *opt, int argc, char **argv) {
   char *snum;
   unsigned long count = 0;
@@ -1225,7 +1526,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
       size_t i;
       for (i = bench[loop].offset; i < (size_t) count;
            i += bench[loop].modulus) {
-        int result;
+        int result = 0; /* no final else: an unknown type reports failure */
         FMT();
         if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
           size_t k;
@@ -1299,6 +1600,14 @@ static int st_strsafe(httrackp *opt, int argc, char **argv) {
       htsbuff b = htsbuff_array(small);
 
       htsbuff_cat(&b, src);
+    } else if (strcmp(argv[0], "overflow-src") == 0) {
+      /* Array source with no NUL: its capacity still comes from sizeof(), so
+         the bounded strlen aborts rather than running off the array. */
+      char nonul[6]; /* never sizeof(char*), per the note above */
+      char big[64];
+
+      memset(nonul, src[0], sizeof(nonul));
+      strcpybuff(big, nonul);
     } else {
       strcpybuff(small, src);
     }
@@ -1372,6 +1681,22 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
   if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0)
     err = 1;
 
+  /* sitemap pair: the flag latches on, the URL takes the String deep copy */
+  from->sitemap = HTS_TRUE;
+  StringCopy(from->sitemap_url, "http://h.test/sitemap.xml");
+  to->sitemap = HTS_FALSE;
+  StringCopy(to->sitemap_url, "");
+  copy_htsopt(from, to);
+  if (!to->sitemap ||
+      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
+    err = 1;
+  from->sitemap = HTS_FALSE;
+  StringCopy(from->sitemap_url, "");
+  copy_htsopt(from, to);
+  if (!to->sitemap ||
+      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
+    err = 1;
+
   /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
   from->pause_min_ms = 5000;
   from->pause_max_ms = 10000;
@@ -1993,6 +2318,77 @@ static int st_fsize(httrackp *opt, int argc, char **argv) {
   return rc;
 }
 
+/* 4GB+100KB wraps to ~108KB through an int, and needs 33 unsigned bits. A
+   macro, not a static const: MSVC's C mode (/TC) rejects a const object
+   used inside another object's static initializer below (C2099). */
+#define HTS_ST_GROWSIZE_OVER32 (4LL * 1024 * 1024 * 1024 + 100 * 1024)
+
+/* llint_grow_size_t() sizes the buffer holding a whole -%S list file: the
+   result must be the exact 64-bit sum or a clean refusal, never a short one. */
+static int st_growsize(httrackp *opt, int argc, char **argv) {
+  enum { REFUSE, ACCEPT, WIDTH };
+
+  static const struct {
+    size_t used;
+    LLint extra;
+    size_t slack;
+    int want;
+  } cases[] = {
+      {0, 0, 0, ACCEPT},
+      {10, 100, 8192, ACCEPT},
+      {(size_t) -2 - 8, 4, 4, ACCEPT}, /* exact fit, no room to spare */
+      {(size_t) -2, 0, 0, ACCEPT},     /* largest representable capacity */
+      {0, -1, 0, REFUSE},              /* fsize() failure */
+      /* -1 already maps to SIZE_MAX; only this exercises the negative guard */
+      {0, -4096, 0, REFUSE},
+      {(size_t) -1, 1, 0, REFUSE},
+      {(size_t) -2, 0, 1, REFUSE},     /* slack alone overruns */
+      {(size_t) -1 - 8, 4, 4, REFUSE}, /* total would be the error value */
+      {(size_t) -1 - 8, 4, 8, REFUSE},
+      {0, HTS_ST_GROWSIZE_OVER32, 8192,
+       WIDTH}, /* 32-bit size_t can't hold these */
+      {10, HTS_ST_GROWSIZE_OVER32, 8192, WIDTH},
+  };
+
+  size_t k;
+  int rc = 0;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+  for (k = 0; k < sizeof(cases) / sizeof(cases[0]); k++) {
+    const size_t used = cases[k].used, slack = cases[k].slack;
+    const LLint extra = cases[k].extra;
+    const size_t got = llint_grow_size_t(used, extra, slack);
+    const hts_boolean refused = got == (size_t) -1 ? HTS_TRUE : HTS_FALSE;
+    const hts_boolean exact =
+        !refused && extra >= 0 && got - used - slack == (size_t) extra;
+    hts_boolean ok;
+
+    switch (cases[k].want) {
+    case ACCEPT:
+      ok = exact;
+      break;
+    case REFUSE:
+      ok = refused;
+      break;
+    default:
+      ok = sizeof(size_t) >= sizeof(LLint) ? exact : refused;
+      break;
+    }
+    if (!ok) {
+      fprintf(stderr,
+              "growsize: grow(" LLintP ", " LLintP ", " LLintP ") = " LLintP
+              " (want %s)\n",
+              (LLint) used, extra, (LLint) slack, (LLint) got,
+              cases[k].want == REFUSE ? "refusal" : "exact sum");
+      rc = 1;
+    }
+  }
+  printf("growsize self-test %s\n", rc == 0 ? "OK" : "FAILED");
+  return rc;
+}
+
 static int st_savename(httrackp *opt, int argc, char **argv) {
   lien_adrfilsave afs;
   cache_back cache;
@@ -2387,17 +2783,20 @@ static int st_cookies(httrackp *opt, int argc, char **argv) {
     static t_cookie ck2;
     htsblk r;
     char host[600];
+    char line[64]; /* treathead NUL-cuts the header in place: never a literal */
 
     memset(&r, 0, sizeof(r));
     memset(host, 'a', sizeof(host) - 1);
     host[sizeof(host) - 1] = '\0';
     ck2.max_len = (int) sizeof(ck2.data);
     ck2.data[0] = '\0';
-    treathead(&ck2, host, "/", &r, "Set-Cookie: SID=1; path=/");
+    strcpybuff(line, "Set-Cookie: SID=1; path=/");
+    treathead(&ck2, host, "/", &r, line);
     if (strnotempty(ck2.data)) // oversize-host cookie was not dropped
       err = 1;
     /* control: a normal host still yields a cookie through treathead */
-    treathead(&ck2, dom, "/", &r, "Set-Cookie: SID=1; path=/");
+    strcpybuff(line, "Set-Cookie: SID=1; path=/");
+    treathead(&ck2, dom, "/", &r, line);
     if (strstr(ck2.data, "SID") == NULL) // guard wrongly dropped a valid cookie
       err = 1;
   }
@@ -2658,6 +3057,32 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
   assertf(strstr(buf, "Refresh") != NULL);
   assertf(strstr(buf, "example.com") != NULL);
 
+  /* a first link whose escaped form overruns the old flat 1024-byte tempo: the
+     redirect must carry the whole URL, not a clipped prefix */
+  {
+    char BIGSTK link[HTS_URLMAXSIZE * 2];
+    char *p = link;
+
+    strcpybuff(link, "http://example.com/");
+    p += strlen(link);
+    memset(p, 'a', 1200);
+    p += 1200;
+    strcpy(p, "/end.html");
+
+    done = 0;
+    fp = fopen(path, "wb");
+    assertf(fp != NULL);
+    hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
+    assertf(fp == NULL);
+    fp = fopen(path, "rb");
+    assertf(fp != NULL);
+    n = fread(buf, 1, sizeof(buf) - 1, fp);
+    fclose(fp);
+    buf[n] = '\0';
+    /* the closing quote proves the URL was not clipped mid-way */
+    assertf(strstr(buf, "/end.html\">") != NULL);
+  }
+
   /* no single link: footer only, no refresh meta */
   done = 0;
   fp = fopen(path, "wb");
@@ -2887,7 +3312,7 @@ static int ae_write_packed(const char *path, int windowBits,
     deflateEnd(&strm);
     return 1;
   }
-  strm.next_in = (Bytef *) src;
+  strm.next_in = (const Bytef *) src;
   strm.avail_in = (uInt) len;
   do {
     size_t n;
@@ -3196,7 +3621,7 @@ static int rb_decide(robots_wizard *r, const char *txt, const char *path) {
   char host[64];
 
   snprintf(host, sizeof(host), "h%d.example", n++);
-  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE);
+  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE, NULL, 0);
   return checkrobots(r, host, path);
 }
 
@@ -3273,6 +3698,267 @@ static int st_robots(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* Collect the URLs a sitemap scan hands out. */
+typedef struct sm_collect {
+  int n;
+  char url[8][HTS_URLMAXSIZE];
+} sm_collect;
+
+static hts_boolean sm_take(void *arg, const char *url) {
+  sm_collect *const c = (sm_collect *) arg;
+
+  if (c->n < (int) (sizeof(c->url) / sizeof(c->url[0])))
+    strcpybuff(c->url[c->n], url);
+  c->n++;
+  return HTS_TRUE;
+}
+
+/* Scan `doc` off a heap buffer with no NUL terminator, so a read past the
+   declared size is an ASan error rather than a silent pass. */
+static int sm_scan(const char *doc, int maxurls, hts_boolean *is_index,
+                   sm_collect *out) {
+  const size_t len = strlen(doc);
+  char *raw = malloct(len);
+  int n;
+
+  memset(out, 0, sizeof(*out));
+  assertf(raw != NULL);
+  memcpy(raw, doc, len);
+  n = hts_sitemap_scan(raw, len, maxurls, is_index, sm_take, out);
+  freet(raw);
+  return n;
+}
+
+static int st_sitemap(httrackp *opt, int argc, char **argv) {
+  sm_collect c;
+  hts_boolean idx;
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+  /* A urlset yields its <loc> URLs, in order, unescaped. */
+  assertf(sm_scan("<?xml version=\"1.0\"?><urlset>"
+                  "<url><loc>http://h.test/a.html</loc></url>"
+                  "<url><loc>  https://h.test/b?x=1&amp;y=2\n  </loc></url>"
+                  "</urlset>",
+                  100, &idx, &c) == 2);
+  assertf(!idx);
+  assertf(strcmp(c.url[0], "http://h.test/a.html") == 0);
+  assertf(strcmp(c.url[1], "https://h.test/b?x=1&y=2") == 0);
+
+  /* A sitemapindex is flagged: its URLs are child sitemaps, not pages. */
+  assertf(sm_scan("<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc>"
+                  "</sitemap></sitemapindex>",
+                  100, &idx, &c) == 1);
+  assertf(idx);
+
+  /* Root element decides even when the other name appears later as text. */
+  assertf(sm_scan("<urlset><url><loc>http://h.test/a</loc></url>"
+                  "<!-- sitemapindex --></urlset>",
+                  100, &idx, &c) == 1);
+  assertf(!idx);
+
+  /* Numeric character references, decimal and hex, decode to ASCII. */
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#63;b&#x3D;c</loc></urlset>",
+                  100, &idx, &c) == 1);
+  assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0);
+
+  /* A reference decoding to a control byte is dropped: the shared decoder
+     writes the real character and the URL check refuses it. A reference the
+     decoder cannot represent (&#0;) stays verbatim, like an unknown entity. */
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#10;b</loc></urlset>", 100,
+                  &idx, &c) == 0);
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#9;b</loc></urlset>", 100, &idx,
+                  &c) == 0);
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#0;b</loc></urlset>", 100, &idx,
+                  &c) == 1);
+  assertf(strcmp(c.url[0], "http://h.test/a&#0;b") == 0);
+
+  /* A comment naming the other root element must not flip the verdict. */
+  assertf(sm_scan("<!-- <sitemapindex> --><urlset><url>"
+                  "<loc>http://h.test/p</loc></url></urlset>",
+                  100, &idx, &c) == 1);
+  assertf(!idx);
+  assertf(sm_scan("<?xml version=\"1.0\"?><!-- <urlset> -->"
+                  "<sitemapindex><loc>http://h.test/s</loc></sitemapindex>",
+                  100, &idx, &c) == 1);
+  assertf(idx);
+
+  /* <location> is not <loc>. */
+  assertf(sm_scan("<urlset><location>http://h.test/a</location></urlset>", 100,
+                  &idx, &c) == 0);
+
+  /* Rejected: relative, non-http scheme, embedded space, empty. */
+  assertf(sm_scan("<urlset><loc>/a.html</loc><loc>ftp://h.test/a</loc>"
+                  "<loc>javascript:alert(1)</loc>"
+                  "<loc>http://h.test/a b</loc><loc></loc></urlset>",
+                  100, &idx, &c) == 0);
+
+  /* The URL length bound: one under fits, exactly at it is dropped rather than
+     truncated into a different URL. */
+  {
+    char BIGSTK doc[HTS_URLMAXSIZE * 2];
+    char BIGSTK url[HTS_URLMAXSIZE + 1];
+    size_t i;
+
+    strcpybuff(url, "http://h.test/");
+    for (i = strlen(url); i < HTS_URLMAXSIZE - 1; i++)
+      url[i] = 'a';
+    url[i] = '\0';
+    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
+    assertf(sm_scan(doc, 100, &idx, &c) == 1);
+
+    url[i] = 'a';
+    url[i + 1] = '\0';
+    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
+    assertf(sm_scan(doc, 100, &idx, &c) == 0);
+  }
+
+  /* The URL cap stops the scan. */
+  assertf(sm_scan("<urlset><loc>http://h.test/1</loc><loc>http://h.test/2</loc>"
+                  "<loc>http://h.test/3</loc></urlset>",
+                  2, &idx, &c) == 2);
+
+  /* The per-document cap at the value the engine actually uses. */
+  {
+    const int many = HTS_SITEMAP_MAX_URLS_DOC + 10;
+    const size_t cap = (size_t) many * 40 + 32;
+    char *big = malloct(cap);
+    size_t off;
+    int i;
+
+    assertf(big != NULL);
+    off = (size_t) snprintf(big, cap, "<urlset>");
+    assertf(off < cap);
+    for (i = 0; i < many; i++) {
+      const int len =
+          snprintf(big + off, cap - off, "<loc>http://h.test/%d</loc>", i);
+
+      assertf(len > 0 && (size_t) len < cap - off);
+      off += (size_t) len;
+    }
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(big, off, HTS_SITEMAP_MAX_URLS_DOC, &idx, sm_take,
+                             &c) == HTS_SITEMAP_MAX_URLS_DOC);
+    /* The handler count, not just the return: a call site hardcoding a smaller
+       cap would still return its own argument. */
+    assertf(c.n == HTS_SITEMAP_MAX_URLS_DOC);
+    freet(big);
+  }
+
+#if HTS_USEZLIB
+  /* A highly compressible document decodes without running away: the ratio
+     budget cannot bind (deflate tops out near 1032:1), so this pins the
+     decompression path itself rather than the 64 MiB ceiling. */
+  {
+    const char *const one = "<url><loc>http://h.test/bomb</loc></url>";
+    const size_t reps = 40000;
+    size_t xlen = 8 + reps * strlen(one) + 10, i;
+    char *x = malloct(xlen + 1);
+    uLongf zlen;
+    char *z;
+    z_stream zs;
+
+    assertf(x != NULL);
+    {
+      size_t w = (size_t) snprintf(x, xlen, "<urlset>");
+      int len;
+
+      assertf(w < xlen);
+      for (i = 0; i < reps; i++) {
+        len = snprintf(x + w, xlen - w, "%s", one);
+        assertf(len > 0 && (size_t) len < xlen - w);
+        w += (size_t) len;
+      }
+      len = snprintf(x + w, xlen - w, "</urlset>");
+      assertf(len > 0 && (size_t) len < xlen - w);
+      w += (size_t) len;
+      xlen = w;
+    }
+    zlen = compressBound((uLong) xlen) + 32;
+    z = malloct((size_t) zlen);
+    assertf(z != NULL);
+    memset(&zs, 0, sizeof(zs));
+    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
+                         Z_DEFAULT_STRATEGY) == Z_OK);
+    zs.next_in = (const Bytef *) x;
+    zs.avail_in = (uInt) xlen;
+    zs.next_out = (Bytef *) z;
+    zs.avail_out = (uInt) zlen;
+    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
+    zlen = (uLongf) zs.total_out;
+    deflateEnd(&zs);
+    /* well over the 4096:1 budget's 1 MiB floor, and far under the 64 MiB cap
+     */
+    assertf(xlen > 1024 * 1024 && (size_t) zlen < xlen / 100);
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(z, (size_t) zlen, 10, &idx, sm_take, &c) == 10);
+    assertf(strcmp(c.url[0], "http://h.test/bomb") == 0);
+    freet(z);
+    freet(x);
+  }
+#endif
+
+  /* An unterminated <loc> at end of buffer must not read past it. */
+  assertf(sm_scan("<urlset><loc>http://h.test/a", 100, &idx, &c) == 0);
+  assertf(sm_scan("<urlset><lo", 100, &idx, &c) == 0);
+
+#if HTS_USEZLIB
+  /* A gzip-framed document is decompressed before scanning. */
+  {
+    const char *const xml =
+        "<urlset><url><loc>http://h.test/gz.html</loc></url></urlset>";
+    uLongf zlen = compressBound((uLong) strlen(xml)) + 32;
+    char *z = malloct((size_t) zlen);
+    z_stream zs;
+
+    assertf(z != NULL);
+    memset(&zs, 0, sizeof(zs));
+    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
+                         Z_DEFAULT_STRATEGY) == Z_OK);
+    zs.next_in = (const Bytef *) xml;
+    zs.avail_in = (uInt) strlen(xml);
+    zs.next_out = (Bytef *) z;
+    zs.avail_out = (uInt) zlen;
+    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
+    zlen = (uLongf) zs.total_out;
+    deflateEnd(&zs);
+
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(z, (size_t) zlen, 100, &idx, sm_take, &c) == 1);
+    assertf(strcmp(c.url[0], "http://h.test/gz.html") == 0);
+
+    /* Truncated gzip: refused, not scanned as plain text. */
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(z, 4, 100, &idx, sm_take, &c) == -1);
+    freet(z);
+  }
+#endif
+
+  /* robots.txt: only Sitemap: records, comments stripped, case-insensitive,
+     and group-independent (no User-agent line needed). */
+  /* robots_parse collects Sitemap: whatever the user-agent group, strips the
+     comment and keeps the rules working alongside it. */
+  {
+    const char *const txt = "User-agent: *\nDisallow: /x\n"
+                            "SITEMAP:  http://h.test/s1.xml  # first\n"
+                            "Sitemapper: http://h.test/no.xml\n"
+                            "Sitemap:\thttps://h.test/s2.xml\n";
+    char BIGSTK maps[1024];
+    robots_wizard rb;
+
+    memset(&rb, 0, sizeof(rb));
+    robots_parse(&rb, "h.test", txt, strlen(txt), NULL, 0, HTS_TRUE, maps,
+                 sizeof(maps));
+    assertf(strcmp(maps, "http://h.test/s1.xml\nhttps://h.test/s2.xml\n") == 0);
+    assertf(checkrobots(&rb, "h.test", "/x") == -1);
+    checkrobots_free(&rb);
+  }
+
+  printf("sitemap self-test OK\n");
+  return 0;
+}
+
 /* Connected stream pair over loopback; Windows has no socketpair(). */
 static int st_socketpair(T_SOC sv[2]) {
   struct sockaddr_in sa;
@@ -3370,21 +4056,6 @@ static int st_ftpuser(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
-/* Bounded substring search (records carry NUL bytes; strstr won't do). */
-static const char *warc_memstr(const char *hay, const char *needle,
-                               size_t haylen, size_t nlen) {
-  if (nlen == 0 || haylen < nlen)
-    return NULL;
-  {
-    size_t i;
-    for (i = 0; i + nlen <= haylen; i++) {
-      if (memcmp(hay + i, needle, nlen) == 0)
-        return hay + i;
-    }
-  }
-  return NULL;
-}
-
 /* Slurp a whole file into a malloc'd buffer; sets *len. NULL on error. */
 static unsigned char *warc_slurp(const char *path, size_t *len) {
   FILE *f = FOPEN(path, "rb");
@@ -3462,6 +4133,13 @@ static unsigned char *warc_next_member(const unsigned char **in,
    Content-Length == block length, the \r\n\r\n trailer intact, the response
    body round-trips, and the hop-by-hop Transfer-Encoding is dropped (a real
    Content-Encoding is kept verbatim; see warc-verbatim). */
+/* Argument order kept for the existing call sites; the search itself is the
+   shared hts_memstr. */
+static const char *warc_memstr(const char *hay, const char *needle,
+                               size_t haylen, size_t nlen) {
+  return hts_memstr(hay, haylen, needle, nlen);
+}
+
 static int st_warc(httrackp *opt, int argc, char **argv) {
   char path[HTS_URLMAXSIZE];
   warc_writer *w;
@@ -4697,10 +5375,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   char fpath[HTS_URLMAXSIZE * 2];
   char file[HTS_URLMAXSIZE * 2];
 
-  snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
+  assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
 
   /* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
-  snprintf(file, sizeof(file), "%scookies.txt", fpath);
+  assertf(sprintfbuff(file, "%scookies.txt", fpath));
   {
     FILE *fp = FOPEN(file, "wb");
 
@@ -4710,7 +5388,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   }
 
   /* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
-  snprintf(file, sizeof(file), "%su@v.txt", fpath);
+  assertf(sprintfbuff(file, "%su@v.txt", fpath));
   {
     FILE *fp = FOPEN(file, "wb");
 
@@ -4732,7 +5410,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
 #endif
 
   (void) UNLINK(file); /* u@v.txt (already gone on Windows) */
-  snprintf(file, sizeof(file), "%scookies.txt", fpath);
+  assertf(sprintfbuff(file, "%scookies.txt", fpath));
   (void) UNLINK(file);
   dir[dirlen] = '\0';
   while (strlen(dir) > base) {
@@ -4806,9 +5484,12 @@ static const struct selftest_entry {
      st_footerfmt},
     {"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
      st_unescape_bounds},
+    {"cmdline-split", "",
+     "webhttrack command-line to argv split (bounds, quoting)",
+     st_cmdlinesplit},
     {"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
-    {"strsafe", "[overflow|overflow-buff [str]]", "bounded string-op self-test",
-     st_strsafe},
+    {"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
+     "bounded string-op self-test", st_strsafe},
     {"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt},
     {"pause", "", "randomized inter-file pause target self-test", st_pause},
     {"relative", "<link> <curr-file>", "relative link between two paths",
@@ -4838,6 +5519,8 @@ static const struct selftest_entry {
     {"sniff", "<content-type> <hex:..|text>", "MIME magic consistency",
      st_sniff},
     {"fsize", "<dir>", "file size past the 2GB signed-32-bit wrap", st_fsize},
+    {"growsize", "", "buffer capacity for a 64-bit file size (no int wrap)",
+     st_growsize},
     {"cache", "<dir>", "cache read/write round-trip self-test", st_cache},
     {"cacheindex", "", "cache-index (.ndx) parse must stay in bounds",
      st_cacheindex},
@@ -4876,6 +5559,8 @@ static const struct selftest_entry {
      st_contentcodings},
     {"robots", "", "robots.txt RFC 9309 Allow/Disallow precedence self-test",
      st_robots},
+    {"sitemap", "",
+     "sitemap <loc> extraction, caps and robots.txt Sitemap:", st_sitemap},
     {"ftp-line", "", "get_ftp_line bounds a hostile FTP reply line",
      st_ftpline},
     {"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser},

src/htsserver.c

@@ -147,7 +147,8 @@ HTS_UNUSED static int LANG_LIST(const char *path, char *buffer, size_t size);
 // 0- Init the URL catcher with standard port
 
 // smallserver_init(&port,&return_host);
-T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort) {
+T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort,
+                           const char *bindAddr) {
   T_SOC soc;
 
   if (defaultPort <= 0) {
@@ -160,12 +161,12 @@ T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort) {
     int i = 0;
 
     do {
-      soc = smallserver_init(&try_to_listen_to[i], adr_prox);
+      soc = smallserver_init(&try_to_listen_to[i], adr_prox, bindAddr);
       *port_prox = try_to_listen_to[i];
       i++;
     } while((soc == INVALID_SOCKET) && (try_to_listen_to[i] >= 0));
   } else {
-    soc = smallserver_init(&defaultPort, adr_prox);
+    soc = smallserver_init(&defaultPort, adr_prox, bindAddr);
     *port_prox = defaultPort;
   }
   return soc;
@@ -243,9 +244,10 @@ static int my_gethostname(char *h_loc, size_t size) {
 }
 
 // smallserver_init(&port,&return_host);
-T_SOC smallserver_init(int *port, char *adr) {
+T_SOC smallserver_init(int *port, char *adr, const char *bindAddr) {
   T_SOC soc = INVALID_SOCKET;
   char h_loc[256 + 2];
+  SOCaddr server;
 
   commandRunning = commandEnd = commandReturn = commandReturnSet =
     commandEndRequested = 0;
@@ -256,25 +258,23 @@ T_SOC smallserver_init(int *port, char *adr) {
     free(commandReturnCmdl);
   commandReturnCmdl = NULL;
 
-  if (my_gethostname(h_loc, 256) == 0) {   // host name
-    SOCaddr server;
+  SOCaddr_initany(server);
+  if (bindAddr != NULL && *bindAddr != '\0') {
+    /* advertise the bound address, else the URL we print is unreachable */
+    if (strlen(bindAddr) >= sizeof(h_loc) || !gethost(bindAddr, &server)) {
+      return INVALID_SOCKET;
+    }
+    strcpybuff(h_loc, bindAddr);
+  } else if (my_gethostname(h_loc, 256) != 0) { // host name
+    return INVALID_SOCKET;
+  }
 
-    SOCaddr_initany(server);
-    if ((soc =
-         (T_SOC) socket(SOCaddr_sinfamily(server), SOCK_STREAM,
-                        0)) != INVALID_SOCKET) {
-      SOCaddr_initport(server, *port);
-      if (bind(soc, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
-        if (listen(soc, 10) >= 0) {
-          strcpy(adr, h_loc);
-        } else {
-#ifdef _WIN32
-          closesocket(soc);
-#else
-          close(soc);
-#endif
-          soc = INVALID_SOCKET;
-        }
+  if ((soc = (T_SOC) socket(SOCaddr_sinfamily(server), SOCK_STREAM, 0)) !=
+      INVALID_SOCKET) {
+    SOCaddr_initport(server, *port);
+    if (bind(soc, &SOCaddr_sockaddr(server), SOCaddr_size(server)) == 0) {
+      if (listen(soc, 10) >= 0) {
+        strcpy(adr, h_loc);
       } else {
 #ifdef _WIN32
         closesocket(soc);
@@ -283,6 +283,13 @@ T_SOC smallserver_init(int *port, char *adr) {
 #endif
         soc = INVALID_SOCKET;
       }
+    } else {
+#ifdef _WIN32
+      closesocket(soc);
+#else
+      close(soc);
+#endif
+      soc = INVALID_SOCKET;
     }
   }
   return soc;
@@ -311,6 +318,104 @@ typedef struct {
   error_redirect = "/server/error.html"; \
 } while(0)
 
+/* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
+   so anything near this is already invalid and is rejected unread. */
+#define SID_VALUE_MAX 64
+
+/** Does the urlencoded request body present the expected session id?
+    True only if at least one "sid" field is present and every occurrence
+    matches, so it holds whichever one a later last-write-wins parse keeps.
+    Non-destructive: it runs before the body is tokenized in place. */
+static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
+  const char *s = body;
+  hts_boolean seen = HTS_FALSE;
+
+  while (s != NULL && *s != '\0') {
+    const char *const amp = strchr(s, '&');
+    const char *const eq = strchr(s, '=');
+
+    if (eq != NULL && (amp == NULL || eq < amp) && (size_t) (eq - s) == 3 &&
+        strncmp(s, "sid", 3) == 0) {
+      const size_t len = amp != NULL ? (size_t) (amp - eq - 1) : strlen(eq + 1);
+      hts_boolean match = HTS_FALSE;
+
+      if (len < SID_VALUE_MAX) {
+        char raw[SID_VALUE_MAX];
+        String value = STRING_EMPTY;
+
+        memcpy(raw, eq + 1, len);
+        raw[len] = '\0';
+        unescapehttp(raw, &value);
+        /* StringBuff is NULL until written, so an empty value lands here. */
+        if (StringBuff(value) != NULL &&
+            strcmp(StringBuff(value), expected) == 0) {
+          match = HTS_TRUE;
+        }
+        StringFree(value);
+      }
+      if (!match) {
+        return HTS_FALSE;
+      }
+      seen = HTS_TRUE;
+    }
+    s = amp != NULL ? amp + 1 : NULL;
+  }
+  return seen;
+}
+
+/** Append src to the NUL-terminated dst of capacity size (NUL included).
+    False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
+    never aborts, because every piece appended here is client-supplied. */
+static hts_boolean path_append(char *dst, size_t size, const char *src) {
+  const size_t used = strlen(dst);
+  const size_t len = strlen(src);
+
+  /* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
+     len stays alone: "used + len < size" could wrap and pass. */
+  if (len >= size - used) {
+    return HTS_FALSE;
+  }
+  memcpy(dst + used, src, len + 1);
+  return HTS_TRUE;
+}
+
+/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
+static hts_boolean cat_html_escaped(String *dst, char c) {
+  switch (c) {
+  case '<':
+    StringCat(*dst, "&lt;");
+    break;
+  case '>':
+    StringCat(*dst, "&gt;");
+    break;
+  case '&':
+    StringCat(*dst, "&amp;");
+    break;
+  case '\'':
+    StringCat(*dst, "&#39;");
+    break;
+  default:
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Append the value of a double-quoted command-line argument: escaped for HTML,
+   which the browser undoes when it posts the command line back, and for the
+   argv splitter, which does not. */
+static void cat_cmdline_arg(String *output, const char *value) {
+  const char *a;
+
+  for (a = value; *a != '\0'; a++) {
+    if (*a == '\\' || *a == '\"') {
+      StringCat(*output, "\\");
+    }
+    if (!cat_html_escaped(output, *a)) {
+      StringMemcat(*output, a, 1);
+    }
+  }
+}
+
 int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   int timeout = 30;
   int retour = 0;
@@ -322,6 +427,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   String tmpbuff = STRING_EMPTY;
   String tmpbuff2 = STRING_EMPTY;
   String fspath = STRING_EMPTY;
+  /* Project directory this server set up; the only root /website/ serves from,
+     and deliberately not cleared between requests. */
+  String website = STRING_EMPTY;
   char catbuff[CATBUFF_SIZE];
 
   /* Load strings */
@@ -395,6 +503,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
     T_SOC soc_c;
     LLint length = 0;
     const char *error_redirect = NULL;
+    hts_boolean denied = HTS_FALSE;
 
     line[0] = '\0';
     buffer[0] = '\0';
@@ -506,6 +615,22 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
         }
       }
 
+      /* Authenticate the body before parsing it: every field it carries is
+         written straight into the global key store below, "command" included,
+         and that one reaches the engine. Checking afterwards cannot work — the
+         damage is already done, and the pre-seeded "sid" above would compare
+         equal to itself for a request that simply omits the field. */
+      if (meth && buffer[0]) {
+        intptr_t expected = 0;
+
+        if (!coucal_readptr(NewLangList, "_sid", &expected) ||
+            !body_sid_is_valid(buffer, (const char *) expected)) {
+          buffer[0] = '\0';
+          meth = 0;
+          denied = HTS_TRUE;
+        }
+      }
+
       /* check variables */
       if (meth && buffer[0]) {
         char *s = buffer;
@@ -526,20 +651,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
         }
       }
 
-      /* Error check */
-      {
-        intptr_t adr = 0;
-        intptr_t adr2 = 0;
-
-        if (coucal_readptr(NewLangList, "sid", &adr)) {
-          if (coucal_readptr(NewLangList, "_sid", &adr2)) {
-            if (strcmp((char *) adr, (char *) adr2) != 0) {
-              meth = 0;
-            }
-          }
-        }
-      }
-
       /* Check variables (internal) */
       if (meth) {
         int doLoad = 0;
@@ -730,6 +841,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                   if (!structcheck(StringBuff(tmpbuff))) {
                     FILE *fp;
 
+                    /* Both halves of fspath come from posted fields, so a ".."
+                       in them would escape the mirror once served. */
+                    if (strstr(StringBuff(fspath), "..") == NULL) {
+                      StringCopy(website, StringBuff(fspath));
+                    }
                     StringCat(tmpbuff, "winprofile.ini");
                     fp = fopen(StringBuff(tmpbuff), "wb");
                     if (fp != NULL) {
@@ -802,7 +918,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
 
       /* Response */
       if (meth) {
-        int virtualpath = 0;
+        hts_boolean virtualpath = HTS_FALSE;
         char *pos;
         char *url = strchr(line1, ' ');
 
@@ -813,11 +929,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           char *qpos;
 
           /* get the URL */
+          fsfile[0] = '\0';
           if (error_redirect == NULL) {
             if ((qpos = strchr(url, '?'))) {
               *qpos = '\0';
             }
-            fsfile[0] = '\0';
             if (strcmp(url, "/") == 0) {
               file = "/server/index.html";
               meth = 2;
@@ -830,7 +946,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           }
 
           if (strncmp(file, "/website/", 9) == 0) {
-            virtualpath = 1;
+            virtualpath = HTS_TRUE;
           }
 
           /* override */
@@ -844,18 +960,26 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             }
           }
 
-          if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
-            if (strncmp(file, "/website/", 9) != 0) {
-              sprintf(fsfile, "%shtml%s", path, file);
-            } else {
-              intptr_t adr = 0;
+          /* the override above may have swapped a mirror path for a GUI page */
+          virtualpath = strncmp(file, "/website/", 9) == 0;
 
-              if (coucal_readptr(NewLangList, "projpath", &adr)) {
-                sprintf(fsfile, "%s%s", (char *) adr, file + 9);
-              }
+          if (!virtualpath) {
+            if (!path_append(fsfile, sizeof(fsfile), path) ||
+                !path_append(fsfile, sizeof(fsfile), "html") ||
+                !path_append(fsfile, sizeof(fsfile), file)) {
+              fsfile[0] = '\0';
+            }
+          } else if (StringNotEmpty(website)) {
+            /* Never the posted "projpath": a client root reads any file. */
+            if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
+                !path_append(fsfile, sizeof(fsfile), "/") ||
+                !path_append(fsfile, sizeof(fsfile), file + 9)) {
+              fsfile[0] = '\0';
             }
           }
 
+          /* path itself may hold ".." (webhttrack passes "<bin>/../share"), so
+             only the untrusted halves are checked: file here, website above. */
           if (fsfile[0] && strstr(file, "..") == NULL
               && (fp = fopen(fsfile, "rb"))) {
             char ok[] =
@@ -903,16 +1027,16 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                 }
               }
               StringMemcat(headers, redir, strlen(redir));
-              {
-                char tmp[256];
-
-                if (strlen(file) < sizeof(tmp) - 32) {
-                  sprintf(tmp, "Location: %s\r\n", newfile);
-                  StringMemcat(headers, tmp, strlen(tmp));
-                }
+              /* client-supplied: a CR/LF here would split the response */
+              if (newfile[strcspn(newfile, "\r\n")] == '\0') {
+                StringCat(headers, "Location: ");
+                StringCat(headers, newfile);
+                StringCat(headers, "\r\n");
               }
               coucal_write(NewLangList, "redirect", (intptr_t) NULL);
-            } else if (is_html(file)) {
+            } else if (!virtualpath && is_html(file)) {
+              /* GUI templates only: ${_sid} in a mirrored page would hand the
+                 crawled site the session id that authenticates commands */
               int outputmode = 0;
 
               StringMemcat(headers, ok, sizeof(ok) - 1);
@@ -940,6 +1064,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                     int p;
                     int format = 0;
                     int listDefault = 0;
+                    hts_boolean unquoted = HTS_FALSE;
 
                     name[0] = '\0';
                     strlncatbuff(name, str, sizeof(name_), n);
@@ -949,6 +1074,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                     } else if ((p = strfield(name, "html:"))) {
                       name += p;
                       format = 1;
+                    } else if ((p = strfield(name, "unquoted:"))) {
+                      name += p;
+                      unquoted = HTS_TRUE;
+                    } else if ((p = strfield(name, "arg:"))) {
+                      name += p;
+                      format = 5;
                     } else if ((p = strfield(name, "list:"))) {
                       name += p;
                       format = 2;
@@ -1085,8 +1216,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                        test:<if ==0>:<if ==1>:<if == 2>..
                        ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
                      */
-                    else if ((p = strfield(name, "test:"))
-                             || (p = strfield(name, "ztest:"))) {
+                    else if ((p = strfield(name, "test:")) ||
+                             (p = strfield(name, "ztest:"))) {
                       intptr_t adr = 0;
                       char *pos2;
                       int ztest = (name[0] == 'z');
@@ -1189,6 +1320,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                         }
                       }
                     }
+                    /* consumed here: it shares nothing with the list and
+                       option formats below */
+                    if (format == 5 && langstr != NULL && outputmode != -1) {
+                      cat_cmdline_arg(&output, langstr);
+                      langstr = NULL;
+                    }
                     if (langstr && outputmode != -1) {
                       switch (format) {
                       case 0:
@@ -1206,18 +1343,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                                 StringMemcat(output, &c, 1);
                               }
                               a += 2;
-                            } else if (outputmode && a[0] == '<') {
-                              StringCat(output, "&lt;");
-                            } else if (outputmode && a[0] == '>') {
-                              StringCat(output, "&gt;");
-                            } else if (outputmode && a[0] == '&') {
-                              StringCat(output, "&amp;");
-                            } else if (outputmode && a[0] == '\'') {
-                              StringCat(output, "&#39;");
+                            } else if (unquoted && a[0] == '\"') {
+                              /* the browser posts an entity back as a raw
+                                 quote, which would open a quoted run in the
+                                 argv splitter; a URI cannot hold one anyway */
+                              StringCat(output, "%22");
+                            } else if (outputmode &&
+                                       cat_html_escaped(&output, a[0])) {
+                              /* appended as an entity */
                             } else if (outputmode == 3 && a[0] == ' ') {
                               StringCat(output, "%20");
-                            } else if (outputmode >= 2
-                                       && ((unsigned char) a[0]) < 32) {
+                            } else if (outputmode >= 2 &&
+                                       ((unsigned char) a[0]) < 32) {
                               char tmp[32];
 
                               sprintf(tmp, "%%%02x", (unsigned char) a[0]);
@@ -1278,20 +1415,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                               }
                               StringClear(tmpbuff);
                               break;
-                            case '<':
-                              StringCat(tmpbuff, "&lt;");
-                              break;
-                            case '>':
-                              StringCat(tmpbuff, "&gt;");
-                              break;
-                            case '&':
-                              StringCat(tmpbuff, "&amp;");
-                              break;
-                            case '\'':
-                              StringCat(tmpbuff, "&#39;");
-                              break;
                             default:
-                              StringMemcat(tmpbuff, fstr, 1);
+                              if (!cat_html_escaped(&tmpbuff, *fstr)) {
+                                StringMemcat(tmpbuff, fstr, 1);
+                              }
                               break;
                             }
                             fstr++;
@@ -1331,7 +1458,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               }
 #endif
             } else {
-              if (is_text(file)) {
+              if (is_html(file)) {
+                StringMemcat(headers, ok, sizeof(ok) - 1);
+              } else if (is_text(file)) {
                 StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
               } else if (is_js(file)) {
                 StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
@@ -1368,6 +1497,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             StringCat(output, error);
           }
         }
+      } else if (denied) {
+        StringCat(headers, "HTTP/1.0 403 Forbidden\r\n"
+                           "Server: httrack small server\r\n"
+                           "Content-type: text/html\r\n");
+        StringCat(output, "Missing or invalid session id.\r\n");
       } else {
 #ifdef _DEBUG
         char error_hdr[] =
@@ -1433,6 +1567,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   StringFree(tmpbuff);
   StringFree(tmpbuff2);
   StringFree(fspath);
+  StringFree(website);
 
   if (buffer)
     free(buffer);

src/htsserver.h

@@ -43,8 +43,11 @@ Please visit our Website: http://www.httrack.com
 
 // Fonctions
 void socinput(T_SOC soc, char *s, int max);
-T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort);
-T_SOC smallserver_init(int *port, char *adr);
+/* Listen on bindAddr, or every interface if NULL/empty; adr (>= 258 bytes) gets
+   the address to advertise. INVALID_SOCKET on error. */
+T_SOC smallserver_init_std(int *port_prox, char *adr_prox, int defaultPort,
+                           const char *bindAddr);
+T_SOC smallserver_init(int *port, char *adr, const char *bindAddr);
 int smallserver(T_SOC soc, char *url, char *method, char *data, char *path);
 
 #define CATCH_RESPONSE \

src/htssitemap.c

@@ -0,0 +1,615 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: sitemap ingestion (sitemaps.org 0.9)                   */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#define HTS_INTERNAL_BYTECODE
+
+#include "htscore.h"
+#include "htssitemap.h"
+
+#include "htsbase.h"
+#include "htscodec.h"
+#include "htsencoding.h"
+#include "htsfilters.h"
+#include "htshash.h"
+#include "htsmodules.h"
+#include "htslib.h"
+#include "htsrobots.h"
+#include "htssafe.h"
+#include "htstools.h"
+
+#include <ctype.h>
+#include <string.h>
+
+/* One queued sitemap document awaiting ingestion. */
+typedef struct sitemap_doc {
+  char adr[HTS_URLMAXSIZE];
+  char fil[HTS_URLMAXSIZE];
+  int level;
+  hts_sitemap_source src;
+  hts_boolean done;
+  struct sitemap_doc *next;
+} sitemap_doc;
+
+struct hts_sitemap_state {
+  sitemap_doc *docs;
+  int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */
+  int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */
+  hts_boolean probe_done;    /* the robots.txt probe has been answered */
+  hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */
+  /* The crawl's own start URL. Seeded URLs are judged against it, so a site
+     cannot widen a subtree crawl by putting its sitemap at the root. */
+  char anchor_adr[HTS_URLMAXSIZE];
+  char anchor_fil[HTS_URLMAXSIZE];
+};
+typedef struct hts_sitemap_state hts_sitemap_state;
+
+/* --------------------------------------------------------------------- */
+/* Document parsing (no engine state: fuzzable and self-testable)         */
+/* --------------------------------------------------------------------- */
+
+/* Accept only an absolute http(s) URL with no space or control byte. */
+static hts_boolean sitemap_url_ok(const char *url) {
+  const char *p;
+
+  if (!strfield(url, "http://") && !strfield(url, "https://"))
+    return HTS_FALSE;
+  for (p = url; *p != '\0'; p++) {
+    if ((unsigned char) *p <= ' ' || (unsigned char) *p == 0x7f)
+      return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Skip to the character after the next '>' at or after p, or NULL. */
+static const char *sitemap_tag_end(const char *p, const char *end) {
+  while (p < end && *p != '>')
+    p++;
+  return p < end ? p + 1 : NULL;
+}
+
+/* HTS_TRUE when the document's root element is `name`. Skips the XML
+   declaration, comments and processing instructions first, so a comment
+   mentioning the other root element cannot decide the document type. */
+static hts_boolean sitemap_root_is(const char *doc, size_t size,
+                                   const char *name) {
+  const size_t nlen = strlen(name);
+  size_t i = 0;
+
+  if (size >= 3 && memcmp(doc, "\xef\xbb\xbf", 3) == 0)
+    i = 3; /* UTF-8 BOM */
+  while (i < size) {
+    if (isspace((unsigned char) doc[i])) {
+      i++;
+    } else if (doc[i] != '<') {
+      return HTS_FALSE; /* character data before any element: not XML */
+    } else if (i + 4 <= size && memcmp(doc + i, "<!--", 4) == 0) {
+      const char *const e = hts_memstr(doc + i, size - i, "-->", 3);
+
+      if (e == NULL)
+        return HTS_FALSE;
+      i = (size_t) (e - doc) + 3;
+    } else if (i + 2 <= size && (doc[i + 1] == '?' || doc[i + 1] == '!')) {
+      while (i < size && doc[i] != '>')
+        i++;
+      i++;
+    } else {
+      size_t j = i + 1;
+
+      /* an optional namespace prefix: <sm:sitemapindex> is the same element */
+      while (j < size && doc[j] != ':' && doc[j] != '>' &&
+             !isspace((unsigned char) doc[j]))
+        j++;
+      if (j >= size || doc[j] != ':')
+        j = i + 1;
+      else
+        j++;
+      return j + nlen <= size && memcmp(doc + j, name, nlen) == 0 &&
+                     (j + nlen == size ||
+                      isspace((unsigned char) doc[j + nlen]) ||
+                      doc[j + nlen] == '>' || doc[j + nlen] == '/')
+                 ? HTS_TRUE
+                 : HTS_FALSE;
+    }
+  }
+  return HTS_FALSE;
+}
+
+/* Decompress a gzip-framed body into a fresh buffer. The 64 MiB cap is what
+   binds in practice; deflate tops out near 1032:1, so the tree's codec budget
+   only matters as the shared policy for a coding that could go further. */
+static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) {
+  const LLint budget = hts_codec_maxout((LLint) size);
+  size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES
+                   ? (size_t) budget
+                   : (size_t) HTS_SITEMAP_MAX_BYTES;
+  char *out;
+  size_t n;
+
+  if (cap == 0)
+    return NULL;
+  out = malloct(cap + 1);
+  if (out == NULL)
+    return NULL;
+  n = hts_codec_head(HTS_CODEC_DEFLATE, body, size, out, cap);
+  if (n == 0) {
+    freet(out);
+    return NULL;
+  }
+  out[n] = '\0';
+  *outsize = n;
+  return out;
+}
+
+int hts_sitemap_scan(const char *body, size_t size, int maxurls,
+                     hts_boolean *is_index, hts_sitemap_handler handler,
+                     void *arg) {
+  char *unpacked = NULL;
+  const char *doc;
+  const char *end;
+  const char *p;
+  int n = 0;
+
+  if (is_index != NULL)
+    *is_index = HTS_FALSE;
+  if (body == NULL || size < 2 || handler == NULL)
+    return 0;
+
+  /* Content-Encoding gzip is undone upstream; only the container is left. */
+  if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) {
+    unpacked = sitemap_gunzip(body, size, &size);
+    if (unpacked == NULL)
+      return -1;
+    doc = unpacked;
+  } else {
+    if (size > (size_t) HTS_SITEMAP_MAX_BYTES)
+      size = (size_t) HTS_SITEMAP_MAX_BYTES;
+    doc = body;
+  }
+  end = doc + size;
+
+  /* Set before the first callback: the handler reads the verdict. */
+  if (is_index != NULL)
+    *is_index = sitemap_root_is(doc, size, "sitemapindex");
+
+  for (p = doc; n < maxurls;) {
+    const char *loc = hts_memstr(p, (size_t) (end - p), "<loc", 4);
+    const char *val;
+    const char *stop;
+    size_t len;
+    char BIGSTK url[HTS_URLMAXSIZE];
+
+    if (loc == NULL)
+      break;
+    /* "<loc>" or "<loc xmlns:..>", never "<location>" */
+    if (loc + 4 >= end || (loc[4] != '>' && !isspace((unsigned char) loc[4]))) {
+      p = loc + 4;
+      continue;
+    }
+    val = sitemap_tag_end(loc + 4, end);
+    if (val == NULL)
+      break;
+    for (stop = val; stop < end && *stop != '<'; stop++)
+      ;
+    /* No closing tag: truncated document, so the value may be a partial URL. */
+    if (stop == end)
+      break;
+    p = stop;
+    while (val < stop && isspace((unsigned char) *val))
+      val++;
+    while (stop > val && isspace((unsigned char) *(stop - 1)))
+      stop--;
+    len = (size_t) (stop - val);
+    /* Overflow-safe: the untrusted length alone against the room left. */
+    if (len == 0 || len >= sizeof(url))
+      continue;
+    memcpy(url, val, len);
+    url[len] = '\0';
+    /* hts_unescapeEntities decodes in place and tolerates src == dest; a
+       reference to a control byte survives as one and sitemap_url_ok drops it.
+     */
+    if (hts_unescapeEntities(url, url, sizeof(url)) != 0 ||
+        !sitemap_url_ok(url))
+      continue;
+    n++;
+    if (!handler(arg, url))
+      break;
+  }
+
+  if (unpacked != NULL)
+    freet(unpacked);
+  return n;
+}
+
+/* --------------------------------------------------------------------- */
+/* Engine glue                                                           */
+/* --------------------------------------------------------------------- */
+
+static hts_sitemap_state *sitemap_get_state(httrackp *opt) {
+  if (opt->sitemap_state == NULL)
+    opt->sitemap_state = calloct(1, sizeof(hts_sitemap_state));
+  return (hts_sitemap_state *) opt->sitemap_state;
+}
+
+static sitemap_doc *sitemap_find(httrackp *opt, const char *adr,
+                                 const char *fil) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+  sitemap_doc *d;
+
+  if (st == NULL)
+    return NULL;
+  for (d = st->docs; d != NULL; d = d->next) {
+    if (strfield2(d->adr, adr) && strcmp(d->fil, fil) == 0)
+      return d;
+  }
+  return NULL;
+}
+
+/* Who asked for this document decides how far it is gated. The wizard proper
+   is not usable here: it wants a referring link, and its up/down travel rules
+   would judge a child sitemap against the parent sitemap's directory. */
+static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr,
+                                         const char *fil,
+                                         hts_sitemap_source src) {
+  /* adr and fil are each capped just under HTS_URLMAXSIZE, and lfull prefixes
+     a scheme and a slash on top of both: 2 * HTS_URLMAXSIZE does not fit. */
+  char BIGSTK l[HTS_URLMAXSIZE * 2 + 16], lfull[HTS_URLMAXSIZE * 2 + 16];
+  int jokdepth = 0, jok;
+
+  hts_boolean refused;
+
+  /* The user naming a sitemap is the same intent as naming a start URL, which
+     the wizard admits unconditionally. */
+  if (src == HTS_SITEMAP_SRC_USER)
+    return HTS_TRUE;
+  strcpybuff(l, jump_identification_const(adr));
+  if (*fil != '/')
+    strcatbuff(l, "/");
+  strcatbuff(l, fil);
+  strcpybuff(lfull, link_has_authority(adr) ? "" : "http://");
+  strcatbuff(lfull, adr);
+  if (*fil != '/')
+    strcatbuff(lfull, "/");
+  strcatbuff(lfull, fil);
+  jok = fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull,
+                         l, NULL, NULL, &jokdepth);
+  refused = (jok == -1) ? HTS_TRUE : HTS_FALSE;
+  if (refused) {
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s",
+                  jokdepth + 1, adr, fil);
+    return HTS_FALSE;
+  }
+  /* A Sitemap: line, or a sitemapindex entry, is the site inviting the fetch;
+     a Disallow elsewhere in the same file does not retract it. The well-known
+     location is only ever a guess, so there a Disallow wins. */
+  if (src == HTS_SITEMAP_SRC_GUESSED &&
+      hts_robots_forbids(opt, adr, fil, (jok != 0) ? HTS_TRUE : HTS_FALSE,
+                         refused)) {
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr,
+                  fil);
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Record the link with save="" so the body stays in memory: a sitemap is
+   ingested, never mirrored. */
+static hts_boolean sitemap_queue_(httrackp *opt, const char *adr,
+                                  const char *fil, int level,
+                                  hts_sitemap_source src, hts_boolean link_it) {
+  hts_sitemap_state *const st = sitemap_get_state(opt);
+  sitemap_doc *d;
+
+  if (st == NULL)
+    return HTS_FALSE;
+  if (st->ndocs >= HTS_SITEMAP_MAX_DOCS || level > HTS_SITEMAP_MAX_LEVEL) {
+    hts_log_print(opt, LOG_WARNING, "Sitemap: cap reached, skipping %s%s", adr,
+                  fil);
+    return HTS_FALSE;
+  }
+  if (strlen(adr) >= sizeof(d->adr) || strlen(fil) >= sizeof(d->fil))
+    return HTS_FALSE;
+  if (sitemap_find(opt, adr, fil) != NULL)
+    return HTS_FALSE;
+  if (!sitemap_fetch_allowed(opt, adr, fil, src))
+    return HTS_FALSE;
+  d = calloct(1, sizeof(sitemap_doc));
+  if (d == NULL)
+    return HTS_FALSE;
+  strcpybuff(d->adr, adr);
+  strcpybuff(d->fil, fil);
+  d->level = level;
+  d->src = src;
+  d->next = st->docs;
+  st->docs = d;
+  st->ndocs++;
+
+  if (!link_it)
+    return HTS_TRUE;
+  if (!hts_record_link(opt, adr, fil, "", "", "", NULL))
+    return HTS_FALSE;
+  heap_top()->testmode = 0;
+  heap_top()->link_import = 0;
+  heap_top()->depth = opt->depth + 1;
+  heap_top()->pass2 = 0;
+  heap_top()->retry = opt->retry;
+  heap_top()->premier = heap_top_index();
+  heap_top()->precedent = heap_top_index();
+  hts_log_print(opt, LOG_INFO, "Sitemap: queued %s%s", adr, fil);
+  return HTS_TRUE;
+}
+
+static hts_boolean sitemap_queue(httrackp *opt, const char *adr,
+                                 const char *fil, int level,
+                                 hts_sitemap_source src) {
+  return sitemap_queue_(opt, adr, fil, level, src, HTS_TRUE);
+}
+
+void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
+                          const char *newadr, const char *newfil) {
+  sitemap_doc *const d = sitemap_find(opt, adr, fil);
+
+  if (d == NULL || d->done)
+    return;
+  d->done = HTS_TRUE; /* the body lives at the target now */
+  /* The engine already queued the target link, so only the marking moves. */
+  (void) sitemap_queue_(opt, newadr, newfil, d->level, d->src, HTS_FALSE);
+  hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil,
+                newadr, newfil);
+}
+
+void hts_sitemap_seed(httrackp *opt, const char *starturl) {
+  char BIGSTK url[HTS_URLMAXSIZE * 2];
+  lien_adrfil af;
+
+  if (StringNotEmpty(opt->sitemap_url)) {
+    if (strlen(StringBuff(opt->sitemap_url)) >= sizeof(url)) {
+      hts_log_print(opt, LOG_ERROR, "Sitemap URL too long");
+    } else {
+      strcpybuff(url, StringBuff(opt->sitemap_url));
+      if (strstr(url, ":/") == NULL)
+        hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url);
+      else if (ident_url_absolute(url, &af) >= 0)
+        (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_USER);
+    }
+  }
+  if (starturl == NULL || starturl[0] == '\0' ||
+      strlen(starturl) >= sizeof(url))
+    return;
+  strcpybuff(url, starturl);
+  if (ident_url_absolute(url, &af) < 0)
+    return;
+  {
+    hts_sitemap_state *const st = sitemap_get_state(opt);
+
+    if (st != NULL && strlen(af.adr) < sizeof(st->anchor_adr) &&
+        strlen(af.fil) < sizeof(st->anchor_fil)) {
+      strcpybuff(st->anchor_adr, af.adr);
+      strcpybuff(st->anchor_fil, af.fil);
+    }
+  }
+  if (!opt->sitemap)
+    return;
+  /* Answered in hts_sitemap_robots, once the parsed rules are installed. */
+  if (hts_record_link(opt, af.adr, "/robots.txt", "", "", "", NULL)) {
+    heap_top()->testmode = 0;
+    heap_top()->link_import = 0;
+    heap_top()->depth = 0;
+    heap_top()->pass2 = 0;
+    heap_top()->retry = opt->retry;
+    heap_top()->premier = heap_top_index();
+    heap_top()->precedent = heap_top_index();
+    /* Claim the host so the parser does not queue robots.txt a second time. */
+    if (opt->robotsptr != NULL)
+      (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, "");
+  }
+}
+
+void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+  int queued = 0;
+
+  if (st == NULL || !opt->sitemap || st->probe_done ||
+      !strfield2(st->anchor_adr, adr))
+    return;
+  st->probe_done = HTS_TRUE;
+  if (sitemaps != NULL) {
+    const char *p = sitemaps;
+
+    while (*p != '\0') {
+      const char *const eol = strchr(p, '\n');
+      const size_t len = eol != NULL ? (size_t) (eol - p) : strlen(p);
+      char BIGSTK line[HTS_URLMAXSIZE];
+      lien_adrfil af;
+
+      if (len > 0 && len < sizeof(line)) {
+        memcpy(line, p, len);
+        line[len] = '\0';
+        /* Same host: a Sitemap: line must not aim the fetcher elsewhere. */
+        if (sitemap_url_ok(line) && ident_url_absolute(line, &af) >= 0 &&
+            strfield2(af.adr, adr) &&
+            sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_DECLARED))
+          queued++;
+      }
+      if (eol == NULL)
+        break;
+      p = eol + 1;
+    }
+  }
+  if (queued == 0 && !st->fallback_done) {
+    st->fallback_done = HTS_TRUE;
+    if (sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_SITEMAP_SRC_GUESSED))
+      queued++;
+  }
+  hts_log_print(opt, LOG_NOTICE, "Sitemap: %d sitemap(s) queued for %s", queued,
+                adr);
+}
+
+hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
+                                const char *fil) {
+  const sitemap_doc *const d = sitemap_find(opt, adr, fil);
+
+  return d != NULL && !d->done ? HTS_TRUE : HTS_FALSE;
+}
+
+/* Handler context: seeding URLs from one document. */
+typedef struct sitemap_ingest_ctx {
+  httrackp *opt;
+  htsmoduleStruct *str;
+  const char *adr; /* host of the document being ingested */
+  int level;
+  hts_boolean is_index;
+  int accepted; /* URLs seeded or documents queued, not merely parsed */
+} sitemap_ingest_ctx;
+
+/* A <loc> of a <urlset>: hand it to the wizard as a top-level seed.
+   The wizard is pointed at the crawl's own start URL, not at the sitemap: the
+   site picks where its sitemap lives, so anchoring travel there would let a
+   root sitemap widen a subtree crawl to the whole host. The URL then becomes
+   its own anchor, exactly as a command-line seed does. */
+static hts_boolean sitemap_seed_url(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+  httrackp *const opt = c->opt;
+  hts_sitemap_state *const st = sitemap_get_state(opt);
+  char BIGSTK buff[HTS_URLMAXSIZE];
+  int before;
+
+  if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) {
+    hts_log_print(opt, LOG_WARNING,
+                  "Sitemap: URL cap reached, ignoring the rest");
+    return HTS_FALSE;
+  }
+  /* strcpybuff aborts rather than truncating: never feed it unchecked input. */
+  if (strlen(url) >= sizeof(buff))
+    return HTS_TRUE;
+  st->nurls++;
+  strcpybuff(buff, url);
+  before = opt->lien_tot;
+  if (htsAddLink(c->str, buff))
+    c->accepted++;
+  if (opt->lien_tot > before)
+    heap_top()->premier = heap_top_index(); /* a seed anchors on itself */
+  return HTS_TRUE;
+}
+
+/* A <loc> of a <sitemapindex>: cross-host children are dropped, so a hostile
+   sitemap cannot aim the fetcher elsewhere. */
+static hts_boolean sitemap_seed_child(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+  char BIGSTK buff[HTS_URLMAXSIZE];
+  lien_adrfil af;
+
+  if (strlen(url) >= sizeof(buff))
+    return HTS_TRUE;
+  strcpybuff(buff, url);
+  if (ident_url_absolute(buff, &af) < 0)
+    return HTS_TRUE;
+  if (!strfield2(af.adr, c->adr)) {
+    hts_log_print(c->opt, LOG_WARNING,
+                  "Sitemap: ignoring off-host child sitemap %s%s", af.adr,
+                  af.fil);
+    return HTS_TRUE;
+  }
+  if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1,
+                    HTS_SITEMAP_SRC_DECLARED))
+    c->accepted++;
+  return HTS_TRUE;
+}
+
+/* The scan classifies the document before the first callback. */
+static hts_boolean sitemap_seed_any(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+
+  return c->is_index ? sitemap_seed_child(arg, url)
+                     : sitemap_seed_url(arg, url);
+}
+
+void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
+                        const char *fil, const char *body, size_t size) {
+  sitemap_doc *const d = sitemap_find(opt, adr, fil);
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+  sitemap_ingest_ctx ctx;
+  int n, anchor, saved_depth;
+
+  if (d == NULL || d->done)
+    return;
+  d->done = HTS_TRUE;
+  /* str->ptr_ is a scratch int owned by the caller, so nothing else moves. */
+  anchor = *str->ptr_;
+  if (st != NULL && st->anchor_adr[0] != '\0' && opt->hash != NULL) {
+    const int i = hash_read((const hash_struct *) opt->hash, st->anchor_adr,
+                            st->anchor_fil, 1);
+
+    if (i >= 0)
+      anchor = i;
+  }
+  *str->ptr_ = anchor;
+  /* Borrow the anchor's position but keep a seed's full depth budget. */
+  saved_depth = heap(anchor)->depth;
+  heap(anchor)->depth = opt->depth + 1;
+  ctx.opt = opt;
+  ctx.str = str;
+  ctx.adr = adr;
+  ctx.level = d->level;
+  ctx.is_index = HTS_FALSE;
+  ctx.accepted = 0;
+
+  n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index,
+                       sitemap_seed_any, &ctx);
+  heap(anchor)->depth = saved_depth;
+  if (n < 0) {
+    hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr,
+                  fil);
+    return;
+  }
+  if (ctx.is_index)
+    hts_log_print(opt, LOG_NOTICE,
+                  "Sitemap: %d of %d child sitemap(s) listed by %s%s",
+                  ctx.accepted, n, adr, fil);
+  else
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: %d of %d URL(s) added from %s%s",
+                  ctx.accepted, n, adr, fil);
+}
+
+void hts_sitemap_free(httrackp *opt) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
+
+  if (st == NULL)
+    return;
+  while (st->docs != NULL) {
+    sitemap_doc *const next = st->docs->next;
+
+    freet(st->docs);
+    st->docs = next;
+  }
+  freet(opt->sitemap_state);
+  opt->sitemap_state = NULL;
+}

src/htssitemap.h

@@ -0,0 +1,108 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack sitemap ingestion (sitemaps.org 0.9). Internal, not installed.
+   Reads <urlset>/<sitemapindex> documents, plain or gzip-framed, and feeds
+   their <loc> URLs to the crawl as top-level seeds. The whole input is
+   attacker-controlled, so every entry point below is capped. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_SITEMAP_DEFH
+#define HTS_SITEMAP_DEFH
+
+#include "htsdefines.h"
+#include "htsopt.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Caps. sitemaps.org allows 50000 URLs and 50 MB uncompressed per document;
+   the byte cap sits above that so a conformant sitemap always fits. */
+#define HTS_SITEMAP_MAX_URLS_DOC 50000           /* <loc> per document */
+#define HTS_SITEMAP_MAX_URLS_TOTAL 200000        /* <loc> per mirror */
+#define HTS_SITEMAP_MAX_DOCS 256                 /* documents per mirror */
+#define HTS_SITEMAP_MAX_LEVEL 4                  /* sitemapindex nesting */
+#define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */
+
+/* Who asked for a sitemap document, which decides how far its fetch is gated.
+   The user naming one is the same intent as a start URL; a site declaring one
+   invites the fetch; the well-known location is only ever our guess. */
+typedef enum {
+  HTS_SITEMAP_SRC_USER,     /**< --sitemap-url */
+  HTS_SITEMAP_SRC_DECLARED, /**< a Sitemap: line or a sitemapindex entry */
+  HTS_SITEMAP_SRC_GUESSED   /**< the /sitemap.xml fallback */
+} hts_sitemap_source;
+
+/* Per-URL handler; returning HTS_FALSE stops the scan. */
+typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url);
+
+/* Scan one sitemap document, plain or gzip-framed, handing every acceptable
+   absolute http(s) <loc> URL to `handler`. Stops after `maxurls` URLs, or when
+   the handler refuses. `is_index` (optional) reports a <sitemapindex>, whose
+   URLs are child sitemaps rather than pages. Returns the number of URLs handed
+   out, or -1 when the document could not be decompressed within the caps. */
+int hts_sitemap_scan(const char *body, size_t size, int maxurls,
+                     hts_boolean *is_index, hts_sitemap_handler handler,
+                     void *arg);
+
+/* --- Engine glue (needs a live httrackp). --- */
+
+/* Queue the first sitemap document of the mirror: the explicit --sitemap-url,
+   or the start host's /robots.txt probe for --sitemap. `starturl` is the first
+   command-line seed. No-op when neither option is set. */
+void hts_sitemap_seed(httrackp *opt, const char *starturl);
+
+/* Act on the start host's robots.txt once its rules are installed: queue the
+   Sitemap: URLs it names (newline-separated, from robots_parse), or the
+   well-known /sitemap.xml when it names none. No-op unless --sitemap. */
+void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps);
+
+/* Carry the sitemap marking of (adr,fil) over to the target of a redirect the
+   engine has already queued, so a moved sitemap is still ingested. */
+void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
+                          const char *newadr, const char *newfil);
+
+/* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */
+hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
+                                const char *fil);
+
+/* Ingest a fetched sitemap document (or the robots.txt probe): seed its URLs
+   through the wizard via htsAddLink, and queue nested sitemaps. `str` supplies
+   the parser context of the document being processed. */
+void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
+                        const char *fil, const char *body, size_t size);
+
+/* Release the ingestion state held in opt (NULL-safe, idempotent). */
+void hts_sitemap_free(httrackp *opt);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/htsstats.h

@@ -0,0 +1,63 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: in-progress display rows, shared by httrack and htsserver .h */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_STATS_DEFH
+#define HTS_STATS_DEFH
+
+#include "htsglobal.h"
+
+/* One row of the "in progress" display, shared so the CLI (httrack) and the
+   web GUI (htsserver) cannot drift apart: each fills its own array. */
+
+#define NStatsBuffer 14
+
+#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
+#define HTS_DEF_FWSTRUCT_t_StatsBuffer
+typedef struct t_StatsBuffer t_StatsBuffer;
+#endif
+struct t_StatsBuffer {
+  char name[1024];
+  char file[1024];
+  char state[288];                         // a short label plus back->info[256]
+  char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
+  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
+  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
+  LLint size;
+  LLint sizetot;
+  int offset;
+  //
+  int back;
+  //
+  int actived; // pour disabled
+};
+
+#endif

src/htstools.c

@@ -276,6 +276,21 @@ int ident_url_relatif(const char *lien, const char *origin_adr,
   return ok;
 }
 
+/* Bounded substring search: bodies and archive records carry NUL bytes, so
+   strstr() would stop at the first one. */
+const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
+                       size_t nlen) {
+  size_t i;
+
+  if (nlen == 0 || haylen < nlen)
+    return NULL;
+  for (i = 0; i + nlen <= haylen; i++) {
+    if (hay[i] == *needle && memcmp(hay + i, needle, nlen) == 0)
+      return hay + i;
+  }
+  return NULL;
+}
+
 // créer dans s, à partir du chemin courant curr_fil, le lien vers link (absolu)
 // un ident_url_relatif a déja été fait avant, pour que link ne soit pas un chemin relatif
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
@@ -308,8 +323,10 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
   // copy only the current path
   curr = _curr;
   strlcpybuff(curr, curr_fil, sizeof(_curr));
-  if ((a = strchr(curr, '?')) == NULL)  // couper au ? (params)
-    a = curr + strlen(curr) - 1;        // pas de params: aller à la fin
+  if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
+    // an empty path has no last character: curr-1 would read before the buffer
+    a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
+  }
   while((*a != '/') && (a > curr))
     a--;                        // chercher dernier / du chemin courant
   if (*a == '/')
@@ -1318,19 +1335,19 @@ HTSEXT_API hts_boolean hts_findnext(find_handle find) {
   if (find) {
 #ifdef _WIN32
     if ((FindNextFileA(find->handle, &find->hdata)))
-      return 1;
+      return HTS_TRUE;
 #else
     char catbuff[CATBUFF_SIZE];
 
     memset(&(find->filestat), 0, sizeof(find->filestat));
     if ((find->dirp = readdir(find->hdir)))
-      if (find->dirp->d_name)
-        if (!STAT
-            (concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name), &find->filestat))
-          return 1;
+      if (!STAT(
+              concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name),
+              &find->filestat))
+        return HTS_TRUE;
 #endif
   }
-  return 0;
+  return HTS_FALSE;
 }
 
 HTSEXT_API int hts_findclose(find_handle find) {

src/htstools.h

@@ -61,6 +61,11 @@ typedef struct lien_adrfilsave lien_adrfilsave;
 int ident_url_relatif(const char *lien, const char *origin_adr,
                       const char *origin_fil,
                       lien_adrfil* const adrfil);
+/* Bounded substring search over data that may hold NUL bytes; NULL if absent.
+ */
+const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
+                       size_t nlen);
+
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr);
 int link_has_authority(const char *lien);
 int link_has_authorization(const char *lien);

src/htsweb.c

@@ -62,6 +62,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsmd5.c"
 #include "md5.c"
 
+#include "htscmdline.h"
 #include "htsserver.h"
 #include "htsurlport.h"
 #include "htsweb.h"
@@ -88,7 +89,7 @@ Please visit our Website: http://www.httrack.com
 
 static htsmutex refreshMutex = HTSMUTEX_INIT;
 
-static int help_server(char *dest_path, int defaultPort);
+static int help_server(char *dest_path, int defaultPort, const char *bindAddr);
 extern int commandRunning;
 extern int commandEnd;
 extern int commandReturn;
@@ -153,6 +154,8 @@ int main(int argc, char *argv[]) {
   int ret = 0;
   int defaultPort = 0;
   int parentPid = 0;
+  /* loopback by default: the handler trusts its input; --bind widens it */
+  const char *bindAddr = "127.0.0.1";
 
   printf("Initializing the server..\n");
 
@@ -179,7 +182,8 @@ int main(int argc, char *argv[]) {
   if (argc < 2 || (argc % 2) != 0) {
     fprintf(stderr, "** Warning: use the webhttrack frontend if available\n");
     fprintf(stderr,
-            "usage: %s [--port <port>] [--ppid parent-pid] <path-to-html-root-dir> [key value [key value]..]\n",
+            "usage: %s [--port <port>] [--bind <address>] [--ppid parent-pid] "
+            "<path-to-html-root-dir> [key value [key value]..]\n",
             argv[0]);
     fprintf(stderr, "example: %s /usr/share/httrack/\n", argv[0]);
     return 1;
@@ -267,6 +271,14 @@ int main(int argc, char *argv[]) {
         fprintf(stderr, "couldn't set the port number to %s\n", argv[i + 1]);
         return -1;
       }
+    } else if (strcmp(argv[i], "--bind") == 0 && i + 1 < argc) {
+      /* empty would fall back to every interface, silently undoing the default
+       */
+      if (!strnotempty(argv[i + 1])) {
+        fprintf(stderr, "--bind needs an address\n");
+        return -1;
+      }
+      bindAddr = argv[i + 1];
     } else if (strcmp(argv[i], "--ppid") == 0 && i + 1 < argc) {
       if (sscanf(argv[i + 1], "%u", &parentPid) != 1) {
         fprintf(stderr, "couldn't set the parent PID to %s\n", argv[i + 1]);
@@ -293,7 +305,7 @@ int main(int argc, char *argv[]) {
   }
 
   /* launch */
-  ret = help_server(argv[1], defaultPort);
+  ret = help_server(argv[1], defaultPort, bindAddr);
 
   htsthread_wait_n(background_threads - 1);
   hts_uninit();
@@ -308,10 +320,8 @@ int main(int argc, char *argv[]) {
 static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
 static void back_launch_cmd(void *pP) {
   char *cmd = (char *) pP;
-  char **argv = (char **) malloct(1024 * sizeof(char *));
+  char **argv;
   int argc = 0;
-  int i = 0;
-  int g = 0;
 
   //
   httrackp *opt;
@@ -322,28 +332,19 @@ static void back_launch_cmd(void *pP) {
   commandReturnCmdl = strdup(cmd);
 
   /* split */
-  argv[0] = strdup("webhttrack");
-  argv[1] = cmd;
-  argc++;
-  i = 0;
-  while(cmd[i]) {
-    if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
-      cmd[i] = ' ';
-    }
-    i++;
-  }
-  i = 0;
-  while(cmd[i]) {
-    if (cmd[i] == '\"')
-      g = !g;
-    if (cmd[i] == ' ') {
-      if (!g) {
-        cmd[i] = '\0';
-        argv[argc++] = cmd + i + 1;
-      }
-    }
-    i++;
+  argv = hts_split_cmdline(cmd, &argc);
+  if (argv == NULL) {
+    if (commandReturnMsg)
+      free(commandReturnMsg);
+    commandReturnMsg = strdup("could not parse the command line");
+    commandReturn = -1;
+    commandRunning = 0;
+    commandEnd = 1;
+    free(cmd);
+    return;
   }
+  /* drop the program name the posted command line carries */
+  argv[0] = strdupt("webhttrack");
 
   /* init */
   hts_init();
@@ -372,6 +373,7 @@ static void back_launch_cmd(void *pP) {
 
   /* free */
   free(cmd);
+  freet(argv[0]);
   freet(argv);
   return;
 }
@@ -427,11 +429,11 @@ static int webhttrack_runmain(httrackp * opt, int argc, char **argv) {
   return ret;
 }
 
-static int help_server(char *dest_path, int defaultPort) {
+static int help_server(char *dest_path, int defaultPort, const char *bindAddr) {
   int returncode = 0;
   char adr_prox[HTS_URLMAXSIZE * 2];
   int port_prox;
-  T_SOC soc = smallserver_init_std(&port_prox, adr_prox, defaultPort);
+  T_SOC soc = smallserver_init_std(&port_prox, adr_prox, defaultPort, bindAddr);
 
   if (soc != INVALID_SOCKET) {
     char url[HTS_URLMAXSIZE * 2];
@@ -671,10 +673,10 @@ int __cdecl htsshow_loop(t_hts_callbackarg * carg, httrackp * opt, lien_back * b
                     char *eps = strchr(back[i].url_adr, '/');
                     int count;
 
-                    if (ep != NULL && ep < eps
-                        && (count = (int) (ep - back[i].url_adr)) < 4) {
+                    if (ep != NULL && eps != NULL && ep < eps &&
+                        (count = (int) (ep - back[i].url_adr)) < 4) {
                       proto[0] = '\0';
-                      strncat(proto, back[i].url_adr, count);
+                      strncatbuff(proto, back[i].url_adr, count);
                     }
                   }
                   snprintf(StatsBuffer[index].state, sizeof(StatsBuffer[index].state),

src/htsweb.h

@@ -35,26 +35,10 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsstats.h"
 
-#define NStatsBuffer     14
 #define MAX_LEN_INPROGRESS 40
 
-typedef struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[256];
-  char url_sav[HTS_URLMAXSIZE * 2];     // pour cancel
-  char url_adr[HTS_URLMAXSIZE * 2];
-  char url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-} t_StatsBuffer;
-
 typedef struct t_InpInfo {
   int ask_refresh;
   int refresh;

src/htswizard.c

@@ -151,6 +151,23 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag,
   return HTS_FALSE;
 }
 
+/* The engine's robots.txt verdict for (adr,fil). Under HTS_ROBOTS_SOMETIMES an
+   explicit filter acceptance overrides the ban, which is why the filter outcome
+   is an input; the sitemap fetcher asks the same question outside the wizard.
+ */
+hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
+                               hts_boolean filters_decided,
+                               hts_boolean filters_refused) {
+  if (!opt->robots || opt->robotsptr == NULL)
+    return HTS_FALSE;
+  if (checkrobots((robots_wizard *) opt->robotsptr, adr, fil) != -1)
+    return HTS_FALSE;
+  if (filters_decided && !filters_refused &&
+      opt->robots == HTS_ROBOTS_SOMETIMES)
+    return HTS_FALSE;
+  return HTS_TRUE;
+}
+
 static int hts_acceptlink_(httrackp * opt, int ptr,
                            const char *adr, const char *fil, const char *tag,
                            const char *attribute, int *set_prio_to,
@@ -576,30 +593,26 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
       }
     }
     // vérifier robots.txt
-    if (opt->robots) {
-      int r = checkrobots(_ROBOTS, adr, fil);
-
-      if (r == -1) {            // interdiction
+    if (opt->robots && checkrobots(_ROBOTS, adr, fil) == -1) {
 #if DEBUG_ROBOTS
-        printf("robots.txt forbidden: %s%s\n", adr, fil);
+      printf("robots.txt forbidden: %s%s\n", adr, fil);
 #endif
-        // question résolue, par les filtres, et mode robot non strict
-        if ((!question) && (filters_answer) &&
-            (opt->robots == HTS_ROBOTS_SOMETIMES) && (forbidden_url != 1)) {
-          r = 0;                // annuler interdiction des robots
-          if (!forbidden_url) {
-            hts_log_print(opt, LOG_DEBUG,
-                          "Warning link followed against robots.txt: link %s at %s%s",
-                          l, adr, fil);
-          }
-        }
-        if (r == -1) {          // interdire
-          forbidden_url = 1;
-          question = 0;
-          hts_log_print(opt, LOG_DEBUG,
-                        "(robots.txt) forbidden link: link %s at %s%s", l, adr,
-                        fil);
+      if (!hts_robots_forbids(opt, adr, fil,
+                              (!question && filters_answer) ? HTS_TRUE
+                                                            : HTS_FALSE,
+                              (forbidden_url == 1) ? HTS_TRUE : HTS_FALSE)) {
+        if (!forbidden_url) {
+          hts_log_print(
+              opt, LOG_DEBUG,
+              "Warning link followed against robots.txt: link %s at %s%s", l,
+              adr, fil);
         }
+      } else {
+        forbidden_url = 1;
+        question = 0;
+        hts_log_print(opt, LOG_DEBUG,
+                      "(robots.txt) forbidden link: link %s at %s%s", l, adr,
+                      fil);
       }
     }
 

src/htswizard.h

@@ -49,6 +49,13 @@ typedef struct httrackp httrackp;
 typedef struct lien_url lien_url;
 #endif
 
+/* The engine's robots.txt verdict for (adr,fil): HTS_TRUE when the fetch is
+   forbidden. `filters_decided`/`filters_refused` carry the filter outcome,
+   which overrides a ban under -s1 (HTS_ROBOTS_SOMETIMES). */
+hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
+                               hts_boolean filters_decided,
+                               hts_boolean filters_refused);
+
 int hts_acceptlink(httrackp * opt, int ptr,
                    const char *adr, const char *fil,
                    const char *tag, const char *attribute,

src/httrack.c

@@ -46,6 +46,7 @@ Please visit our Website: http://www.httrack.com
 #include "httrack.h"
 #include "htslib.h"
 #include "htscharset.h" // after htslib.h: winsock2.h must precede windows.h
+#include "htsbacktrace.h"
 
 /* Static definitions */
 static int fexist(const char *s);
@@ -72,10 +73,6 @@ static int linput(FILE * fp, char *s, int max);
 #include <sys/ioctl.h>
 #endif
 #include <ctype.h>
-#if (defined(__linux) && defined(HAVE_EXECINFO_H))
-#include <execinfo.h>
-#define USES_BACKTRACE
-#endif
 /* END specific definitions */
 
 static void __cdecl htsshow_init(t_hts_callbackarg * carg);
@@ -233,8 +230,7 @@ static hts_boolean vt_size_refresh(void) {
 */
 #define STYLE_STATVALUES VT_BOLD
 #define STYLE_STATTEXT   VT_UNBOLD
-#define STYLE_STATRESET  VT_UNBOLD
-#define NStatsBuffer     14
+#define STYLE_STATRESET VT_UNBOLD
 /* Rows the stats block and "Current job" take above the in-progress list. */
 #define NStatsHeaderRows 7
 
@@ -880,21 +876,6 @@ static void sig_doback(int blind) {     // mettre en backing
 #undef FD_ERR
 #define FD_ERR 2
 
-static void print_backtrace(void) {
-#ifdef USES_BACKTRACE
-  void *stack[256];
-  const int size = backtrace(stack, sizeof(stack)/sizeof(stack[0]));
-  if (size != 0) {
-    backtrace_symbols_fd(stack, size, FD_ERR);
-  }
-#else
-  const char msg[] = "No stack trace available on this OS :(\n";
-  if (write(FD_ERR, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
-    /* sorry GCC */
-  }
-#endif
-}
-
 static size_t print_num(char *buffer, int num) {
   size_t i, j;
   if (num < 0) {
@@ -928,7 +909,7 @@ static void sig_fatal(int code) {
   size += print_num(&buffer[size], code);
   buffer[size++] = '\n';
   (void) (write(FD_ERR, buffer, size) == size);
-  print_backtrace();
+  hts_print_backtrace(FD_ERR);
   (void) (write(FD_ERR, msgreport, sizeof(msgreport) - 1)
     == sizeof(msgreport) - 1);
   abort();
@@ -951,6 +932,7 @@ static void sig_leave(int code) {
 }
 
 static void signal_handlers(void) {
+  hts_backtrace_init();
 #ifdef _WIN32
   signal(SIGINT, sig_leave);    // ^C
   signal(SIGTERM, sig_finish);  // kill <process>

src/httrack.h

@@ -36,26 +36,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsglobal.h"
 #include "htscore.h"
 #include "htssafe.h"
-
-#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
-#define HTS_DEF_FWSTRUCT_t_StatsBuffer
-typedef struct t_StatsBuffer t_StatsBuffer;
-#endif
-struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[256];
-  char BIGSTK url_sav[HTS_URLMAXSIZE * 2];      // pour cancel
-  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
-  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-};
+#include "htsstats.h"
 
 #ifndef HTS_DEF_FWSTRUCT_t_InpInfo
 #define HTS_DEF_FWSTRUCT_t_InpInfo

src/httrack.vcxproj

@@ -100,6 +100,7 @@
 
   <ItemGroup>
     <ClCompile Include="httrack.c" />
+    <ClCompile Include="htsbacktrace.c" />
   </ItemGroup>
 
   <!-- Pulls in libhttrack.lib and builds it first. -->

src/libhttrack.vcxproj

@@ -59,9 +59,10 @@
   <ItemDefinitionGroup>
     <ClCompile>
       <!-- LIBHTTRACK_EXPORTS turns HTSEXT_API into __declspec(dllexport); ZLIB_DLL
-           imports zlib from its DLL. HTS_USEBROTLI/HTS_USEZSTD enable the br and
-           zstd content codings. Windows 7 floor, matching WinHTTrack. -->
-      <PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+           imports zlib from its DLL, ZLIB_CONST makes its next_in const as the
+           autotools build does. HTS_USEBROTLI/HTS_USEZSTD enable the br and zstd
+           content codings. Windows 7 floor, matching WinHTTrack. -->
+      <PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;ZLIB_CONST;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>$(MSBuildThisFileDirectory);$(MSBuildThisFileDirectory)coucal;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <WarningLevel>Level3</WarningLevel>
       <MultiProcessorCompilation>true</MultiProcessorCompilation>
@@ -121,6 +122,7 @@
     <ClCompile Include="htshelp.c" />
     <ClCompile Include="htsindex.c" />
     <ClCompile Include="htslib.c" />
+    <ClCompile Include="htscmdline.c" />
     <ClCompile Include="htsurlport.c" />
     <ClCompile Include="htsmd5.c" />
     <ClCompile Include="htsmodules.c" />
@@ -136,6 +138,7 @@
     <ClCompile Include="htswrap.c" />
     <ClCompile Include="htszlib.c" />
     <ClCompile Include="htswarc.c" />
+    <ClCompile Include="htssitemap.c" />
     <ClCompile Include="md5.c" />
     <ClCompile Include="minizip\ioapi.c" />
     <ClCompile Include="minizip\iowin32.c" />

src/proxy/proxytrack.c

@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     PT_Element elt = PT_ElementNew();
 
     elt->statuscode = 405;
-    strcpy(elt->msg, "Method Not Allowed");
+    strcpybuff(elt->msg, "Method Not Allowed");
     return elt;
   }
 
@@ -771,7 +771,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
                   PT_ReadIndex(indexes, StringBuff(itemUrl) + 1, FETCH_HEADERS);
                 if (file != NULL && file->statuscode == HTTP_OK) {
                   size = file->size;
-                  if (file->lastmodified) {
+                  if (file->lastmodified[0] != '\0') {
                     timestamp = get_time_rfc822(file->lastmodified);
                   }
                   if (timestamp == (time_t) 0) {
@@ -785,7 +785,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
                     }
                     timestamp = timestampRep;
                   }
-                  if (file->contenttype) {
+                  if (file->contenttype[0] != '\0') {
                     mimeType = file->contenttype;
                   }
                 }
@@ -822,7 +822,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     elt->statuscode = 207;      /* Multi-Status */
     strcpy(elt->charset, "utf-8");
     strcpy(elt->contenttype, "text/xml");
-    strcpy(elt->msg, "Multi-Status");
+    strcpybuff(elt->msg, "Multi-Status");
     StringFree(response);
 
     fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
@@ -879,7 +879,7 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
     elt->statuscode = HTTP_OK;
     strcpy(elt->charset, "iso-8859-1");
     strcpy(elt->contenttype, "text/html");
-    strcpy(elt->msg, "OK");
+    strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;
   }

src/proxy/store.c

@@ -407,7 +407,7 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
     elt->statuscode = HTTP_OK;
     strcpy(elt->charset, "iso-8859-1");
     strcpy(elt->contenttype, "text/html");
-    strcpy(elt->msg, "OK");
+    strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;
   }
@@ -579,9 +579,10 @@ PT_Index PT_LoadCache(const char *filename) {
         if (chain != NULL) {
           const char *scheme = link_has_authority(chain->name) ? "" : "http://";
 
-          snprintf(index->slots.common.startUrl,
-                   sizeof(index->slots.common.startUrl), "%s%s", scheme,
-                   (const char *) chain->name);
+          /* dropped rather than truncated: empty already reads as "unset" */
+          if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
+                           (const char *) chain->name))
+            index->slots.common.startUrl[0] = '\0';
         }
       }
     }
@@ -827,6 +828,18 @@ PT_Element PT_ElementNew(void) {
   return r;
 }
 
+/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
+static void PT_Element_failf(PT_Element r, const char *fmt, ...)
+    HTS_PRINTF_FUN(2, 3);
+
+static void PT_Element_failf(PT_Element r, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
+  va_end(args);
+}
+
 PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
   if (index != NULL && SAFE_INDEX(index)) {
     return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
@@ -865,12 +878,17 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
   sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
   (headersSize) += (int) strlen(headers + headersSize); \
 } while(0)
-#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
-  if (line[0] != '\0' && strfield2(line, refline)) { \
-    strcpy(refvalue, value); \
-    line[0] = '\0'; \
-	} \
-} while(0)
+/* refvalue_size is mandatory: the cache line is bounded only by the line
+   buffer, not by the destination. Clip rather than reject, since an
+   engine-written field can be wider than ours. */
+#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
+  do {                                                                         \
+    if (line[0] != '\0' && strfield2(line, refline)) {                         \
+      (refvalue)[0] = '\0';                                                    \
+      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
+      line[0] = '\0';                                                          \
+    }                                                                          \
+  } while (0)
 #define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
   if (line[0] != '\0' && strfield2(line, refline)) { \
     int intval = 0; \
@@ -972,9 +990,12 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
                     const char *scheme =
                         link_has_authority(filenameIndex) ? "" : "http://";
 
-                    firstSeen = 1;
-                    snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
-                             scheme, filenameIndex);
+                    /* dropped rather than truncated; try the next entry */
+                    if (sprintfbuff(index->startUrl, "%s%s", scheme,
+                                    filenameIndex))
+                      firstSeen = 1;
+                    else
+                      index->startUrl[0] = '\0';
                   }
                 }
               } else {
@@ -1070,16 +1091,23 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                 value++;
               ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
               ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
-              ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg);     // msg
+              ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
+                                   sizeof(r->msg));
               ZIP_READFIELD_INT(line, value, "X-Size", r->size);        // size
-              ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype);        // contenttype
-              ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset);       // contenttype
-              ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified);      // last-modified
-              ZIP_READFIELD_STRING(line, value, "Etag", r->etag);       // Etag
-              ZIP_READFIELD_STRING(line, value, "Location", r->location);       // 'location' pour moved
+              ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
+                                   sizeof(r->contenttype));
+              ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
+                                   sizeof(r->charset));
+              ZIP_READFIELD_STRING(line, value, "Last-Modified",
+                                   r->lastmodified, sizeof(r->lastmodified));
+              ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
+                                   sizeof(r->etag));
+              ZIP_READFIELD_STRING(line, value, "Location", r->location,
+                                   sizeof(location_default));
               ZIP_READFIELD_STRING(line, value, "Content-Disposition",
-                                   r->cdispo); // Content-disposition
-              ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_);      // Original save filename
+                                   r->cdispo, sizeof(r->cdispo));
+              ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
+                                   sizeof(previous_save_));
               if (line[0] != '\0') {
                 int len = r->headers ? ((int) strlen(r->headers)) : 0;
                 int nlen =
@@ -1135,8 +1163,9 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                   snprintf(previous_save, sizeof(previous_save), "%s%s",
                            index->path, previous_save_ + index->fixedPath);
                 } else {
-                  snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
-                          previous_save_, (int) index->fixedPath);
+                  PT_Element_failf(
+                      r, "Bogus fixePath prefix for %s (prefixLen=%d)",
+                      previous_save_, (int) index->fixedPath);
                   r->statuscode = STATUSCODE_INVALID;
                 }
               } else {
@@ -1155,7 +1184,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
             // Peut-on stocker le fichier directement sur disque?
             if (ok) {
               if (r->msg[0] == '\0') {
-                strcpy(r->msg, "Cache Read Error : Unexpected error");
+                strcpybuff(r->msg, "Cache Read Error : Unexpected error");
               }
             } else {            // lire en mémoire
 
@@ -1173,24 +1202,27 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                           int last_errno = errno;
 
                           r->statuscode = STATUSCODE_INVALID;
-                          sprintf(r->msg, "Read error in cache disk data: %s",
-                                  strerror(last_errno));
+                          PT_Element_failf(r,
+                                           "Read error in cache disk data: %s",
+                                           strerror(last_errno));
                         }
                         r->adr[r->size] = '\0';
                       } else {
                         r->statuscode = STATUSCODE_INVALID;
-                        strcpy(r->msg,
-                               "Read error (memory exhausted) from cache");
+                        strcpybuff(r->msg,
+                                   "Read error (memory exhausted) from cache");
                       }
                       fclose(fp);
                     } else {
                       r->statuscode = STATUSCODE_INVALID;
-                      snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
-                              file_convert(catbuff, sizeof(catbuff), previous_save));
+                      PT_Element_failf(
+                          r, "Read error (can't open '%s') from cache",
+                          file_convert(catbuff, sizeof(catbuff),
+                                       previous_save));
                     }
                   } else {
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpy(r->msg, "Cached file name is invalid");
+                    strcpybuff(r->msg, "Cached file name is invalid");
                   }
                 }
               } else {
@@ -1202,12 +1234,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                       free(r->adr);
                       r->adr = NULL;
                       r->statuscode = STATUSCODE_INVALID;
-                      strcpy(r->msg, "Cache Read Error : Read Data");
+                      strcpybuff(r->msg, "Cache Read Error : Read Data");
                     } else
                       *(r->adr + r->size) = '\0';
                   } else {      // erreur
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpy(r->msg, "Cache Memory Error");
+                    strcpybuff(r->msg, "Cache Memory Error");
                   }
                 }
               }
@@ -1215,21 +1247,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
           }                     // si save==null, ne rien charger (juste en tête)
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpy(r->msg, "Cache Read Error : Read Header Data");
+          strcpybuff(r->msg, "Cache Read Error : Read Header Data");
         }
         unzCloseCurrentFile(index->zFile);
       } else {
         r->statuscode = STATUSCODE_INVALID;
-        strcpy(r->msg, "Cache Read Error : Open File");
+        strcpybuff(r->msg, "Cache Read Error : Open File");
       }
 
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpy(r->msg, "Cache Read Error : Bad Offset");
+      strcpybuff(r->msg, "Cache Read Error : Bad Offset");
     }
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpy(r->msg, "File Cache Entry Not Found");
+    strcpybuff(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -1570,9 +1602,11 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
                   const char *scheme =
                       link_has_authority(line) ? "" : "http://";
 
-                  firstSeen = 1;
-                  snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
-                           scheme, line);
+                  /* dropped rather than truncated; try the next entry */
+                  if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
+                    firstSeen = 1;
+                  else
+                    index->startUrl[0] = '\0';
                 }
               }
 
@@ -1793,8 +1827,9 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                 snprintf(previous_save, sizeof(previous_save), "%s%s",
                          index->path, previous_save_ + index->fixedPath);
               } else {
-                snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
-                        previous_save_, (int) index->fixedPath);
+                PT_Element_failf(r,
+                                 "Bogus fixePath prefix for %s (prefixLen=%d)",
+                                 previous_save_, (int) index->fixedPath);
                 r->statuscode = STATUSCODE_INVALID;
               }
             } else {
@@ -1820,17 +1855,18 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                 if (r->adr != NULL) {
                   if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpy(r->msg, "Read error in cache disk data");
+                    strcpybuff(r->msg, "Read error in cache disk data");
                   }
                   r->adr[r->size] = '\0';
                 } else {
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpy(r->msg, "Read error (memory exhausted) from cache");
+                  strcpybuff(r->msg,
+                             "Read error (memory exhausted) from cache");
                 }
                 fclose(fp);
               } else {
                 r->statuscode = STATUSCODE_INVALID;
-                strcpy(r->msg, "Previous cache file not found (2)");
+                strcpybuff(r->msg, "Previous cache file not found (2)");
               }
             }
           } else {
@@ -1842,30 +1878,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                   free(r->adr);
                   r->adr = NULL;
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpy(r->msg, "Cache Read Error : Read Data");
+                  strcpybuff(r->msg, "Cache Read Error : Read Data");
                 } else
                   r->adr[r->size] = '\0';
               } else {          // erreur
                 r->statuscode = STATUSCODE_INVALID;
-                strcpy(r->msg, "Cache Memory Error");
+                strcpybuff(r->msg, "Cache Memory Error");
               }
             }
           }
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpy(r->msg, "Cache Read Error : Bad Data");
+          strcpybuff(r->msg, "Cache Read Error : Bad Data");
         }
       } else {                  // erreur
         r->statuscode = STATUSCODE_INVALID;
-        strcpy(r->msg, "Cache Read Error : Read Header");
+        strcpybuff(r->msg, "Cache Read Error : Read Header");
       }
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpy(r->msg, "Cache Read Error : Seek Failed");
+      strcpybuff(r->msg, "Cache Read Error : Seek Failed");
     }
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpy(r->msg, "File Cache Entry Not Found");
+    strcpybuff(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -2128,12 +2164,15 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
   return 0;
 }
 
-#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
-  if (line[0] != '\0' && strfield2(line, refline)) { \
-    strcpy(refvalue, value); \
-    line[0] = '\0'; \
-	} \
-} while(0)
+/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
+#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)   \
+  do {                                                                         \
+    if (line[0] != '\0' && strfield2(line, refline)) {                         \
+      (refvalue)[0] = '\0';                                                    \
+      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
+      line[0] = '\0';                                                          \
+    }                                                                          \
+  } while (0)
 #define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
   if (line[0] != '\0' && strfield2(line, refline)) { \
     int intval = 0; \
@@ -2191,7 +2230,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
           }
           if ((pos = getArcField(index->line, 2)) != NULL) {
             r->msg[0] = '\0';
-            strncat(r->msg, pos, sizeof(pos) - 1);
+            strncatbuff(r->msg, pos, sizeof(r->msg) - 1);
           }
           while(linput(index->file, index->line, sizeof(index->line) - 1)
                 && index->line[0] != '\0') {
@@ -2202,11 +2241,16 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
               *value = '\0';
               for(value++; *value == ' ' || *value == '\t'; value++) ;
               HTTP_READFIELD_INT(line, value, "Content-Length", r->size);       // size
-              HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype);       // contenttype
-              HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified);     // last-modified
-              HTTP_READFIELD_STRING(line, value, "Etag", r->etag);      // Etag
-              HTTP_READFIELD_STRING(line, value, "Location", r->location);      // 'location' pour moved
-              HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo);     // Content-disposition
+              HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
+                                    sizeof(r->contenttype));
+              HTTP_READFIELD_STRING(line, value, "Last-Modified",
+                                    r->lastmodified, sizeof(r->lastmodified));
+              HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
+                                    sizeof(r->etag));
+              HTTP_READFIELD_STRING(line, value, "Location", r->location,
+                                    sizeof(location_default));
+              HTTP_READFIELD_STRING(line, value, "Content-Disposition",
+                                    r->cdispo, sizeof(r->cdispo));
               if (line[0] != '\0') {
                 int len = r->headers ? ((int) strlen(r->headers)) : 0;
                 int nlen =
@@ -2246,7 +2290,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
                 fetchSize = dataLength - metaSize;
               } else if (fetchSize > dataLength - metaSize) {
                 r->statuscode = STATUSCODE_INVALID;
-                strcpy(r->msg, "Cache Read Error : Truncated Data");
+                strcpybuff(r->msg, "Cache Read Error : Truncated Data");
               }
               r->size = 0;
               if (r->statuscode != STATUSCODE_INVALID) {
@@ -2259,12 +2303,13 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
                     int last_errno = errno;
 
                     r->statuscode = STATUSCODE_INVALID;
-                    sprintf(r->msg, "Read error in cache disk data: %s",
-                            strerror(last_errno));
+                    PT_Element_failf(r, "Read error in cache disk data: %s",
+                                     strerror(last_errno));
                   }
                 } else {
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpy(r->msg, "Read error (memory exhausted) from cache");
+                  strcpybuff(r->msg,
+                             "Read error (memory exhausted) from cache");
                 }
               }
             }
@@ -2272,21 +2317,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
 
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpy(r->msg, "Cache Read Error : Read Header Error");
+          strcpybuff(r->msg, "Cache Read Error : Read Header Error");
         }
 
       } else {
         r->statuscode = STATUSCODE_INVALID;
-        strcpy(r->msg, "Cache Read Error : Read Header Error");
+        strcpybuff(r->msg, "Cache Read Error : Read Header Error");
       }
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpy(r->msg, "Cache Read Error : Seek Error");
+      strcpybuff(r->msg, "Cache Read Error : Seek Error");
     }
 
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpy(r->msg, "File Cache Entry Not Found");
+    strcpybuff(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -2335,8 +2380,18 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
   FILE *const fp = st->fp;
   struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
+  struct tm unknown_date;
   int size_headers;
 
+  /* a cached entry with no parseable Last-Modified must not take the writer
+     down; the epoch is the conventional "date unknown" */
+  if (tm == NULL) {
+    memset(&unknown_date, 0, sizeof(unknown_date));
+    unknown_date.tm_year = 70;
+    unknown_date.tm_mday = 1;
+    tm = &unknown_date;
+  }
+
   sprintf(st->headers,
           "HTTP/1.0 %d %s"
           "\r\n"

src/webhttrack.vcxproj

@@ -97,6 +97,7 @@
   <ItemGroup>
     <ClCompile Include="htsserver.c" />
     <ClCompile Include="htsweb.c" />
+    <ClCompile Include="htscmdline.c" />
     <ClCompile Include="htsurlport.c" />
   </ItemGroup>
 

tests/01_engine-cmdline-split.test

@@ -0,0 +1,8 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+# webhttrack posts its command line as one string: the argv split must grow past
+# 1024 arguments and keep a quote inside a value out of the option parser.
+httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

tests/01_engine-growsize.test

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Buffer capacity for a 64-bit file size ('httrack -#test=growsize'): a -%S list
+# file past 4GB must size its buffer exactly or be refused, never wrap.
+
+set -euo pipefail
+
+out=$(httrack -#test=growsize)
+echo "$out"
+test "$out" == "growsize self-test OK"
+
+tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_growsize.XXXXXX")
+trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
+
+echo '<html><body>hi</body></html>' >"$tmp/index.html"
+printf -- '-*/zzmarker*\n' >"$tmp/rules.txt"
+
+# the rules file lands in the URL/filter string, echoed back by the banner
+run=$(httrack -O "$tmp/out" --quiet -n "-%S" "$tmp/rules.txt" \
+    "file://$tmp/index.html" 2>&1) || true
+printf '%s\n' "$run" | grep -q 'zzmarker' || {
+    echo "FAIL: -%S rules file was not loaded"
+    printf '%s\n' "$run"
+    exit 1
+}

tests/01_engine-strsafe.test

@@ -47,3 +47,18 @@ case "$err" in
     exit 1
     ;;
 esac
+
+# An array source with no NUL must abort on the source bound rather than run
+# off the array: the array half of the source-capacity selection.
+err=$(httrack -#test=strsafe overflow-src "x" 2>&1) || true
+case "$err" in
+*"strsafe: NOT aborted"*)
+    echo "unterminated source array was NOT caught" >&2
+    exit 1
+    ;;
+*"size < sizeof_source"*) ;;
+*)
+    echo "expected htssafe source-bound abort, got: $err" >&2
+    exit 1
+    ;;
+esac

tests/01_zlib-sitemap.test

@@ -0,0 +1,10 @@
+#!/bin/bash
+#
+# Sitemap parser self-test: <loc> extraction, entity decoding, URL and length
+# rejections, the URL cap, gzip framing and robots.txt Sitemap: records.
+
+set -euo pipefail
+
+out=$(httrack -O /dev/null '-#test=sitemap')
+echo "$out"
+test "$out" = "sitemap self-test OK"

tests/53_local-proxytrack-arc-reason.test

@@ -0,0 +1,32 @@
+#!/bin/bash
+# An ARC entry's HTTP reason phrase must survive a proxytrack --convert
+# round-trip; it used to be clipped to sizeof(char*) - 1 bytes.
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+printf 'HTTP/1.1 404 Not Found Here At All\r\nContent-Type: text/html\r\nLast-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
+printf 'hello' >"$dir/body"
+alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+
+# ARC 1.0: filedesc record and version block, then per entry
+# <nl> <URL-record> <nl> <headers> <body>; the record's last field is that length.
+{
+    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+    printf '2 0 test\n'
+    printf '\n\n'
+    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 404 - - 0 t.arc %d\n' "$alen"
+    cat "$dir/hdr" "$dir/body"
+} >"$dir/in.arc"
+
+proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1
+
+# HTTP/1.0 is the writer's own prefix (the input says 1.1), so a verbatim copy
+# of the input cannot satisfy this match.
+grep -aqF 'HTTP/1.0 404 Not Found Here At All' "$dir/out.arc" || {
+    echo "reason phrase lost in the ARC round-trip:" >&2
+    grep -a '^HTTP/' "$dir/out.arc" >&2 || echo "(no status line at all)" >&2
+    exit 1
+}

tests/68_webhttrack-outdir-charset.test

@@ -60,13 +60,20 @@ test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
 
 # Post a "start" whose -O dir is 'café' in the form's ISO-8859-1 charset.
 "${python}" - "${url}" "${work}" <<'PY'
-import sys, urllib.parse, urllib.request
+import re, sys, urllib.parse, urllib.request
 
 url, work = sys.argv[1], sys.argv[2]
 outdir = work + "/caf\xe9"  # 'café' as the single byte the browser would send
 # Port 1 refuses at once: only the decoded -O dir matters, not the fetch.
 cmd = "httrack --quiet --robots=0 http://127.0.0.1:1/x.html -O " + outdir
-fields = [("path", work), ("projname", "proj"), ("command_do", "start"),
+# The body is refused without the session id the server renders into each form.
+# Note the wizard page is under /server/; a bare /step4.html is the doc page.
+page = urllib.request.urlopen(url + "server/step4.html", timeout=20).read()
+m = re.search(rb'name="sid" value="([0-9a-f]+)"', page)
+if not m:
+    raise SystemExit("no session id in server/step4.html")
+fields = [("sid", m.group(1).decode()),
+          ("path", work), ("projname", "proj"), ("command_do", "start"),
           ("winprofile", "x"), ("command", cmd)]
 body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="", encoding="latin-1"))
                 for k, v in fields)

tests/77_webhttrack-redirect.test

@@ -0,0 +1,187 @@
+#!/bin/bash
+#
+# htsserver's POST redirect: the Location header is built from a client-supplied
+# value, and the listen address is loopback unless --bind widens it.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# run_with_timeout/kill_tree: timeout(1) is absent on macOS
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+log=$(mktemp)
+# start() runs inside a command substitution, so its $! never reaches this
+# shell. Take the pid from the server's own announcement instead: a missed kill
+# leaves an orphan holding the CI job open long after the suite has passed.
+srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
+cleanup() {
+    stop
+    rm -f "${log}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Start htsserver, echo the announced URL. Extra args are passed through.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" "$@" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+stop() {
+    local pid
+    pid=$(srvpid)
+    test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
+}
+
+# The session id gates the request body, so a POST has to carry one. The server
+# renders it into every form, which is where a browser picks it up too.
+scrape_sid() {
+    python3 -c 'import socket, sys
+s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
+s.settimeout(20)
+s.sendall(b"GET /server/index.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n")
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" |
+        sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
+}
+
+# POST redirect=$1 to 127.0.0.1:$2 with session id $3, print the raw headers.
+post_redirect() {
+    python3 -c 'import socket, sys, urllib.parse
+body = ("sid=" + sys.argv[3] + "&redirect="
+        + urllib.parse.quote(sys.argv[1], safe=""))
+req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+       "Content-type: application/x-www-form-urlencoded\r\n"
+       "Content-length: %d\r\n\r\n%s" % (len(body), body))
+s = socket.create_connection(("127.0.0.1", int(sys.argv[2])), 10)
+s.settimeout(20)
+s.sendall(req.encode())
+out = b""
+# stop at the end of the header block: the server need not close the connection,
+# and an unbounded recv() hung the macOS runner for over an hour.
+while b"\r\n\r\n" not in out:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.split(b"\r\n\r\n")[0].decode("latin-1"))' "$1" "$2" "$3"
+}
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# "free"/"inuse"/"unusable": can $1 still be bound on 127.0.0.2? A wildcard
+# listener takes the port on every address, a loopback-only one does not, so
+# this discriminates where the announced URL cannot.
+probe_alias() {
+    python3 -c 'import socket, sys
+s = socket.socket()
+try:
+    s.bind(("127.0.0.2", int(sys.argv[1])))
+except OSError as e:
+    print("unusable" if e.errno in (99, 49, 10049) else "inuse")
+else:
+    print("free")
+finally:
+    s.close()' "$1"
+}
+
+# Non-repeating and not a round number: a truncation, a reorder or a dropped
+# interior byte all stay visible.
+long=$(python3 -c 'print("".join(chr(33 + i % 90) for i in range(4097)))')
+url=$(start)
+port=$(portof "${url}")
+sid=$(scrape_sid "${port}")
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
+resp=$(post_redirect "${long}" "${port}" "${sid}") ||
+    fail "no response to an oversized redirect value"
+stop
+loc=$(printf '%s' "${resp}" | sed -n 's/^Location: //p' | tr -d '\r')
+test "${loc}" = "${long}" ||
+    fail "oversized redirect not echoed whole (got ${#loc} bytes, want ${#long})"
+
+# CR/LF must suppress the header outright. Sanitising it instead would keep the
+# grep for an injected header quiet while still emitting a mangled Location.
+url=$(start)
+port=$(portof "${url}")
+sid=$(scrape_sid "${port}")
+resp=$(post_redirect '/foo
+X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
+stop
+# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
+# pipefail then turns the hit into a silent pass.
+grep -qi 'X-Injected' <<<"${resp}" &&
+    fail "CRLF in the redirect value reached the response"
+test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
+    fail "CRLF redirect still emitted a Location header"
+test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
+    fail "CRLF redirect did not yield exactly one status line"
+
+# Loopback unless asked otherwise. Assert the socket, not the announcement.
+url=$(start)
+port=$(portof "${url}")
+alias_state=$(probe_alias "${port}")
+stop
+if test "${alias_state}" = unusable; then
+    echo "no 127.0.0.2 alias; skipping the listen-address assertions" >&2
+else
+    test "${alias_state}" = free ||
+        fail "default listen address is not loopback-only (127.0.0.2:${port} taken)"
+    case "${url}" in
+    http://127.0.0.1:*) ;;
+    *) fail "default announcement is not loopback: ${url}" ;;
+    esac
+
+    url=$(start --bind 0.0.0.0)
+    port=$(portof "${url}")
+    alias_state=$(probe_alias "${port}")
+    stop
+    test "${alias_state}" = inuse ||
+        fail "--bind 0.0.0.0 did not take the wildcard (127.0.0.2:${port} ${alias_state})"
+fi
+
+# An empty --bind must not quietly fall back to every interface. Bounded: if it
+# were accepted the server would listen instead of exiting.
+run_with_timeout 15 htsserver "${distdir}/" --bind "" >"${log}" 2>&1 || true
+grep -q -- "--bind needs an address" "${log}" ||
+    fail "empty --bind not refused: $(cat "${log}")"
+
+echo "PASS"

tests/78_webhttrack-sid.test

@@ -0,0 +1,150 @@
+#!/bin/bash
+#
+# htsserver's session id must gate the request body: every field in it is
+# written into the global key store, and "command" from there reaches the engine.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+srv=
+log=$(mktemp)
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    rm -f "${log}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL. Runs in a command substitution, so it is a
+# subshell and cannot export the pid: the caller reads it back with srvpid.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+# The server reports its own pid; the aliveness assertions below hang off it.
+srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
+
+alive() { kill -0 "$1" 2>/dev/null; }
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
+# default index). Prints the reply.
+request() {
+    python3 -c 'import socket, sys
+port, body = int(sys.argv[1]), sys.argv[2]
+page = sys.argv[3] if len(sys.argv) > 3 else "index"
+if body:
+    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+           "Content-type: application/x-www-form-urlencoded\r\n"
+           "Content-length: %d\r\n\r\n%s" % (len(body), body))
+else:
+    req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
+s = socket.create_connection(("127.0.0.1", port), 10)
+s.settimeout(30)
+s.sendall(req.encode())
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
+}
+
+# The server hands the sid to any client in every form; scraping it is the
+# legitimate flow, and it is what makes the accept case below meaningful.
+scrape_sid() {
+    request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
+}
+
+url=$(start)
+port=$(portof "${url}")
+srv=$(srvpid)
+test -n "${srv}" || fail "htsserver did not report its pid"
+alive "${srv}" || fail "htsserver is not running"
+
+sid=$(scrape_sid "${port}")
+# Control: without a real sid every assertion below would pass vacuously.
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
+
+# Accept: the legitimate flow still works. "redirect" is the cheapest field
+# with a reply that is visible in the headers. Matched from a here-string, not a
+# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
+resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
+grep -q '^Location: /accepted' <<<"${resp}" ||
+    fail "a body carrying the correct sid was refused"
+
+# Refuse: missing and wrong. A missing one is the regression under test — the
+# expected value used to be pre-seeded into the compared key, so omitting the
+# field compared equal to itself.
+for bad in "redirect=/nosid" "sid=&redirect=/empty" \
+    "sid=00000000000000000000000000000000&redirect=/wrong"; do
+    resp=$(request "${port}" "${bad}")
+    grep -q '^Location:' <<<"${resp}" &&
+        fail "body accepted without a valid sid: ${bad}"
+    # A refusal has to be a well-formed reply, not a headerless fragment: the
+    # release build used to emit only Content-length, which reads as a protocol
+    # error to any client and hides the reason.
+    grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
+        fail "refusal was not a 403: ${bad}"
+done
+
+# Suppressing the reply is not the same as refusing the write: the body is
+# applied to one global key store that later requests render from, and the
+# command dispatcher reads it from there. Probe the store itself — step3
+# interpolates ${projname} into its title — rather than the refused reply.
+store_page() {
+    # a dead probe or a non-page reply must not read as "the store was not written"
+    page=$(request "$1" "" step3) || fail "the key store probe failed"
+    grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
+        fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
+}
+
+request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
+store_page "${port}"
+grep -q 'UNAUTHWRITE' <<<"${page}" &&
+    fail "a body without a valid sid was written to the key store"
+
+# Paired accept case: without it the assertion above passes even if the server
+# simply ignores every body, which would prove nothing.
+request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
+store_page "${port}"
+grep -q 'AUTHWRITE' <<<"${page}" ||
+    fail "a body carrying the correct sid was not written to the key store"
+
+echo "PASS"

tests/79_local-proxytrack-webdav-mime.test

@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# proxytrack's WebDAV PROPFIND fallback must behave identically whether or not
+# the cache entry supplies Content-Type and Last-Modified.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+# shellcheck source=tests/testlib.sh
+. "$top_srcdir/tests/testlib.sh"
+
+python=$(find_python) || {
+    echo "python3 missing, skipping"
+    exit 77
+}
+command -v curl >/dev/null 2>&1 || {
+    echo "curl missing, skipping"
+    exit 77
+}
+# First test to run proxytrack as a live server, and MSYS cannot reap a native
+# listener: the orphan wedged the whole Windows suite for 48 minutes (#595).
+if is_windows; then
+    echo "windows: cannot reap a backgrounded proxytrack, skipping"
+    exit 77
+fi
+
+dir=$(mktemp -d)
+ptpid=
+cleanup() {
+    stop_server "$ptpid"
+    rm -rf "$dir"
+}
+trap cleanup EXIT
+
+# Neither header is present, so file->contenttype and ->lastmodified stay "".
+printf 'HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
+printf 'hello' >"$dir/body"
+alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+{
+    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+    printf '2 0 test\n'
+    printf '\n\n'
+    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+    cat "$dir/hdr" "$dir/body"
+} >"$dir/in.arc"
+
+freeport() {
+    "$python" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'
+}
+proxyport=$(freeport)
+icpport=$(freeport)
+
+proxytrack "127.0.0.1:$proxyport" "127.0.0.1:$icpport" "$dir/in.arc" >"$dir/pt.log" 2>&1 &
+ptpid=$!
+waited=0
+until grep -qE "HTTP Proxy installed on|Unable to (initialize a temporary server|create the server)" "$dir/pt.log"; do
+    kill -0 "$ptpid" 2>/dev/null || {
+        echo "FAIL: proxytrack exited before listening"
+        cat "$dir/pt.log"
+        exit 1
+    }
+    test "$waited" -lt 50 || {
+        echo "FAIL: proxytrack never announced its listen port"
+        exit 1
+    }
+    sleep 0.1
+    waited=$((waited + 1))
+done
+grep -q "HTTP Proxy installed on" "$dir/pt.log" || {
+    echo "FAIL: proxytrack failed to bind"
+    cat "$dir/pt.log"
+    exit 1
+}
+
+# --max-time: an unbounded read here would wedge the runner rather than fail
+curl -s --max-time 30 -X PROPFIND -H "Depth: 1" \
+    "http://127.0.0.1:$proxyport/webdav/example.com/" >"$dir/resp.xml"
+
+grep -q '<getcontenttype>application/octet-stream</getcontenttype>' "$dir/resp.xml" || {
+    echo "FAIL: missing Content-Type did not fall back to application/octet-stream"
+    cat "$dir/resp.xml"
+    exit 1
+}
+grep -q '<getlastmodified>Wed, 01 Jan 2025 00:00:00 GMT</getlastmodified>' "$dir/resp.xml" || {
+    echo "FAIL: missing Last-Modified did not fall back to the index timestamp"
+    cat "$dir/resp.xml"
+    exit 1
+}
+
+echo "OK: WebDAV listing falls back correctly for an entry with no Content-Type/Last-Modified"

tests/80_engine-crash-symbolize.test

@@ -0,0 +1,97 @@
+#!/bin/bash
+# A fatal signal prints the raw backtrace, then names the frames
+# backtrace_symbols_fd() leaves as module+offset (-fvisibility=hidden hides them).
+set -eu
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+if is_windows; then
+    echo "no backtrace() on Windows; skipping" >&2
+    exit 77
+fi
+command -v httrack >/dev/null || {
+    echo "could not find httrack" >&2
+    exit 1
+}
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_symbolize.XXXXXX") || exit 1
+trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM
+out="${tmpdir}/trace"
+raw="${tmpdir}/trace-optout"
+overflow="this string is far too long for the buffer"
+# A frame glibc could not name. Stop at the offset: it prints the trailing
+# "[0xADDR]" with or without a leading space depending on its version.
+rawframe='(+0x[0-9a-f][0-9a-f]*)'
+
+# The strsafe selftest aborts on purpose, which lands in sig_fatal.
+rc=0
+run_with_timeout 60 httrack -#test=strsafe overflow "$overflow" >"$out" 2>&1 || rc=$?
+test "$rc" -ne 124 || {
+    echo "the crash handler did not finish within the deadline" >&2
+    exit 1
+}
+
+grep -q '^Caught signal ' "$out" || {
+    echo "no 'Caught signal' line:" >&2
+    cat "$out" >&2
+    exit 1
+}
+if grep -q 'No stack trace available on this OS' "$out"; then
+    echo "no backtrace() on this platform; skipping" >&2
+    exit 77
+fi
+# Unconditional and first: symbolizing must never cost the raw trace.
+grep -q "$rawframe" "$out" || {
+    echo "raw module+offset frames are gone:" >&2
+    cat "$out" >&2
+    exit 1
+}
+
+command -v addr2line >/dev/null || {
+    echo "addr2line not installed; skipping" >&2
+    exit 77
+}
+
+# Control: resolve the same frames ourselves, so a stripped build cannot turn
+# the assertion below into a vacuous pass.
+oracle="${tmpdir}/oracle"
+sed -n 's/^\([^(]*\)(+\(0x[0-9a-f]*\)).*$/\1 \2/p' "$out" |
+    while read -r mod off; do
+        test -f "$mod" || continue
+        addr2line -Cfip -e "$mod" "$off" 2>/dev/null || true
+    done >"$oracle"
+grep -qE 'st_strsafe|strcpy_safe_' "$oracle" || {
+    echo "addr2line names no hidden symbol in this build; skipping" >&2
+    exit 77
+}
+
+# The payload. Dropping the raw frames first is what makes it specific: both
+# names are static, so .dynsym could never have carried them. Not anchored on
+# the "0xOFF:" line, the inline chain puts the name on either half.
+grep -v "$rawframe" "$out" | grep -qE 'st_strsafe|strcpy_safe_' || {
+    echo "the handler did not name the hidden frames:" >&2
+    cat "$out" >&2
+    exit 1
+}
+
+# Opt-out: raw trace kept, nothing spawned.
+export HTTRACK_NO_SYMBOLIZE=1
+rc=0
+run_with_timeout 60 httrack -#test=strsafe overflow "$overflow" >"$raw" 2>&1 || rc=$?
+unset HTTRACK_NO_SYMBOLIZE
+test "$rc" -ne 124 || {
+    echo "the opt-out run did not finish within the deadline" >&2
+    exit 1
+}
+grep -q "$rawframe" "$raw" || {
+    echo "the opt-out lost the raw trace:" >&2
+    cat "$raw" >&2
+    exit 1
+}
+! grep -q '^0x[0-9a-f]*: ' "$raw" || {
+    echo "the opt-out still symbolized:" >&2
+    cat "$raw" >&2
+    exit 1
+}

tests/81_webhttrack-maxsize.test

@@ -0,0 +1,142 @@
+#!/bin/bash
+#
+# The wizard's size fields must render as distinct caps: sizemax is -M, othermax
+# then maxhtml share -m.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_maxsize.XXXXXX") || fail "no tmpdir"
+srvlog=$(mktemp)
+srv=
+cleanup() {
+    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
+    srv=
+    rm -rf "${work}" "${srvlog}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
+# ~/.httrack.ini out of it.
+sport=$("${python}" -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()')
+(
+    trap '' TERM TTOU
+    export HOME="${work}"
+    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
+) &
+srv=$!
+for _ in $(seq 1 40); do
+    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
+    kill -0 "${srv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
+
+# Audit what step4.html renders back; without command_do nothing is launched.
+"${python}" - "${url}" <<'PY' || fail "wizard rendered the wrong options (see above)"
+import re, sys, urllib.parse, urllib.request
+
+url = sys.argv[1]
+html, other, site = "111111", "222222", "333333"
+
+
+def post(fields):
+    # The body is refused without the session id the server puts in each form.
+    form = urllib.request.urlopen(url + "server/index.html", timeout=20).read()
+    m = re.search(rb'name="sid" value="([0-9a-f]+)"', form)
+    if not m:
+        raise SystemExit("no session id in server/index.html")
+    fields = [("sid", m.group(1).decode())] + fields
+    body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in fields)
+    req = urllib.request.Request(url + "server/step4.html",
+                                 data=body.encode("latin-1"), method="POST")
+    return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
+
+
+def textarea(page, name):
+    m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
+    if m is None:
+        sys.exit("no %s textarea in the rendered step4.html" % name)
+    return m.group(1)
+
+
+def want(ok, msg, ctx):
+    if not ok:
+        sys.exit("%s\nrendered:%s" % (msg, ctx))
+
+
+page = post([("maxhtml", html), ("othermax", other), ("sizemax", site),
+             ("dos", "2")])
+cmd = textarea(page, "command")
+want("--max-size=" + site in cmd, "site cap not emitted as --max-size", cmd)
+want("--max-files=" + site not in cmd, "site cap still an --max-files", cmd)
+# Positive controls: both per-file caps still ride -m, one option each.
+want("--max-files=" + other in cmd, "non-HTML cap not emitted as --max-files",
+     cmd)
+want("--max-files=," + html in cmd, "HTML cap not emitted as --max-files=,",
+     cmd)
+want(cmd.count("--max-files=") == 2, "unexpected --max-files count", cmd)
+want(cmd.index("--max-files=" + other) < cmd.index("--max-files=," + html),
+     "the bare --max-files must precede the --max-files=, form", cmd)
+
+# winprofile.ini feeds the same three caps to the Windows GUI, one key each.
+ini = textarea(page, "winprofile").replace("\r\n", "\n")  # the ini is CRLF
+want("\nDos=2" in ini, "Dos not written from the dos field", ini)
+want("\nMaxHtml=" + html + "\n" in ini, "MaxHtml not written from maxhtml", ini)
+want("\nMaxOther=" + other + "\n" in ini, "MaxOther not written from othermax",
+     ini)
+want("\nMaxAll=" + site + "\n" in ini, "MaxAll not written from sizemax", ini)
+
+# A bare --max-files= (an unguarded empty field) parses as -m with no digits,
+# silently clearing the html cap.
+page = post([("maxhtml", ""), ("othermax", ""), ("sizemax", ""), ("dos", "2")])
+cmd = textarea(page, "command")
+want("--max-rate=" in cmd, "empty size fields rendered no command line", cmd)
+want("--max-files" not in cmd and "--max-size" not in cmd,
+     "empty size fields still rendered a size option", cmd)
+
+# A mistyped ${LANG_OK] key renders the OK button's label empty.
+opts = urllib.request.urlopen(url + "server/option2b.html",
+                              timeout=20).read().decode("latin-1")
+want("LANG_OK" not in opts, "unexpanded LANG_OK in option2b.html", "")
+want('<input type="submit" value="OK"' in opts, "no OK label in option2b.html",
+     "")
+PY
+
+cleanup
+
+# A bare -m<n> resets the html limit, so only the bare-then-comma order keeps
+# both caps live. basic.html is 487 bytes, well past the 10-byte html cap.
+crawl() {
+    bash "${distdir}/tests/local-crawl.sh" "$@"
+}
+crawl --errors 1 --log-found 'File too big' \
+    httrack 'BASEURL/simple/basic.html' '--max-files=,10'
+crawl --errors 1 --log-found 'File too big' \
+    httrack 'BASEURL/simple/basic.html' '--max-files=500000' '--max-files=,10'
+crawl --errors 0 --found 'simple/basic.html' --log-not-found 'File too big' \
+    httrack 'BASEURL/simple/basic.html' '--max-files=,10' '--max-files=500000'
+
+echo "PASS"

tests/82_webhttrack-browse-links.test

@@ -0,0 +1,135 @@
+#!/bin/bash
+#
+# A browser will not follow an http: page to a file: URL, so the GUI's "browse
+# the mirror" link has to go through the server's own /website/ route.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+# Control: on a path that does not resolve, grep "passes" without reading anything.
+test -f "${distdir}/html/server/finished.html" || fail "no GUI pages under ${distdir}"
+bad=$(grep -l 'file://' "${distdir}"/html/server/finished.html || true)
+test -z "${bad}" || fail "file: link left in: ${bad}"
+
+work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_browse.XXXXXX") || fail "no tmpdir"
+srvlog=$(mktemp)
+srv=
+srvpid=
+cleanup() {
+    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
+    test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
+    rm -rf "${work}" "${srvlog}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+# Stand in for a finished mirror: the crawl itself is not under test.
+proj="${work}/websites/proj"
+mkdir -p "${proj}"
+printf 'MIRROR-PROBE-OK\n' >"${proj}/probe.txt"
+printf '<html><body>MIRROR-INDEX-OK</body></html>\n' >"${proj}/index.html"
+
+# An isolated HOME keeps a stray ~/.httrack.ini out of the server's settings.
+sport=$("${python}" -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()')
+(
+    trap '' TERM TTOU
+    export HOME="${work}"
+    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
+) &
+srv=$!
+for _ in $(seq 1 40); do
+    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
+    kill -0 "${srv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
+srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
+
+# htsserver resolves the posted paths itself, so hand it native ones.
+"${python}" - "${url}" "$(nativepath "${work}/websites")" <<'PY' || fail "browse-link checks failed"
+import re, sys, urllib.error, urllib.parse, urllib.request
+
+url, base = sys.argv[1].rstrip("/"), sys.argv[2]
+rc = 0
+
+
+def check(ok, what):
+    global rc
+    print(("ok: " if ok else "FAIL: ") + what)
+    if not ok:
+        rc = 1
+
+
+def get(path):
+    # The UI is served ISO-8859-1, so stay in bytes.
+    return urllib.request.urlopen(url + path, timeout=20).read()
+
+
+# No crawl has run, so no commandRunning/commandEnd override swaps the page out.
+finished = get("/server/finished.html").decode("latin-1")
+# Control: an empty or unrendered page would pass "no file:" on its own.
+check("HTTrack Website Copier" in finished, "finished.html rendered")
+check("file://" not in finished, "finished.html has no file: link")
+
+# Control: /website/ 404s until a project is set, so the fetches below are real.
+try:
+    get("/website/probe.txt")
+    check(False, "/website/ served with no project set")
+except urllib.error.HTTPError as e:
+    check(e.code == 404, "/website/ is bound to a project (got %d)" % e.code)
+
+# Point the server at the mirror the way step4.html does; a body without the
+# session id is refused outright. Only a profile save records the served root,
+# so post one; command_do=save runs no crawl.
+sid = re.search(r'name="sid" value="([0-9a-f]+)"', finished)
+if sid is None:
+    print("FAIL: no sid in the rendered form")
+    sys.exit(1)
+fields = [("sid", sid.group(1)), ("path", base), ("projname", "proj"),
+          ("command", "httrack"), ("command_do", "save"), ("winprofile", "x")]
+body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="")) for k, v in fields)
+urllib.request.urlopen(urllib.request.Request(
+    url + "/server/step4.html", data=body.encode("latin-1"), method="POST"),
+    timeout=20).read()
+
+check(get("/website/probe.txt") == b"MIRROR-PROBE-OK\n", "mirror file over http")
+check("MIRROR-INDEX-OK" in get("/website/index.html").decode("latin-1"),
+      "mirror index over http")
+
+body = get("/server/finished.html").decode("latin-1")
+# Match the anchor by its mirror-path label: the list below it links /website/
+# too, so a bare "is the route mentioned" check passes on the unfixed page.
+check(re.search(r'href="/website/index\.html"[^>]*>\s*' +
+                re.escape(base + "/proj"), body) is not None,
+      "the mirror-path link points at the served mirror")
+check("file://" not in body, "finished.html has no file: link")
+
+sys.exit(rc)
+PY
+
+# A leaked htsserver wedges the parallel harness behind a green log.
+cleanup
+! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
+
+echo "PASS"

tests/83_webhttrack-argescape.test

@@ -0,0 +1,148 @@
+#!/bin/bash
+#
+# The wizard renders its httrack command line as one string, which the engine
+# splits back into argv: a double quote in a field must reach the split escaped,
+# or the rest of the value is parsed as fresh options (-V runs a shell command).
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+srv=
+log=$(mktemp)
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    rm -f "${log}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL. Runs in a command substitution, so it is a
+# subshell and cannot export the pid: the caller reads it back with srvpid.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
+# default index). Prints the reply.
+request() {
+    python3 -c 'import socket, sys
+port, body = int(sys.argv[1]), sys.argv[2]
+page = sys.argv[3] if len(sys.argv) > 3 else "index"
+if body:
+    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+           "Content-type: application/x-www-form-urlencoded\r\n"
+           "Content-length: %d\r\n\r\n%s" % (len(body), body))
+else:
+    req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
+s = socket.create_connection(("127.0.0.1", port), 10)
+s.settimeout(30)
+s.sendall(req.encode())
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
+}
+
+scrape_sid() {
+    request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
+}
+
+# urlencode the key=value pairs given as arguments
+formencode() {
+    python3 -c 'import sys, urllib.parse
+print(urllib.parse.urlencode([tuple(a.split("=", 1)) for a in sys.argv[1:]]))' "$@"
+}
+
+url=$(start)
+port=$(portof "${url}")
+srv=$(srvpid)
+test -n "${srv}" || fail "htsserver did not report its pid"
+
+sid=$(scrape_sid "${port}")
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
+
+# Fill the wizard fields the command line quotes, then read back the generated
+# command line: the user-agent carries a break-out attempt, the footer a
+# backslash (which must survive the escape round trip), the project name a plain
+# value.
+body=$(formencode "sid=${sid}" 'user=Moz" -V "touch /tmp/pwn' 'footer=a\b"c' \
+    "path=/tmp/p" "projname=plain proj" 'urls=http://x/a"b' 'url2=+*.png"')
+request "${port}" "${body}" >/dev/null
+cmdline=$(request "${port}" "" step4 |
+    sed -n '/<textarea name="command"/,/<\/textarea>/p')
+
+# Control: without the fields in the page every assertion below is vacuous.
+grep -q -- '--user-agent' <<<"${cmdline}" ||
+    fail "no --user-agent in the generated command line (probe blind)"
+
+# A plain value is passed through untouched: the escaping must not mangle the
+# ordinary case.
+grep -qF -- '--path "/tmp/p/plain proj"' <<<"${cmdline}" ||
+    fail "a plain quoted value was not passed through: ${cmdline}"
+
+# The quote is escaped, so the split keeps it inside the value...
+grep -qF -- '--user-agent "Moz\" -V \"touch /tmp/pwn"' <<<"${cmdline}" ||
+    fail "the quote in the user-agent was not escaped: ${cmdline}"
+
+# ...and the raw form, which would end the argument and hand -V to the option
+# parser, is gone.
+grep -qF -- '--user-agent "Moz" -V "touch' <<<"${cmdline}" &&
+    fail "the user-agent still closes its argument early: ${cmdline}"
+
+# A backslash is escaped too, or the split would eat it along with the quote
+# that follows.
+grep -qF -- '--footer "a\\b\"c"' <<<"${cmdline}" ||
+    fail "the backslash in the footer was not escaped: ${cmdline}"
+
+# The url and filter fields sit outside quotes, where a backslash cannot escape
+# anything: one quote there flips the parity of every quote after it, so the
+# escaping above would protect nothing. They must not emit a raw quote at all.
+grep -qF -- 'http://x/a%22b' <<<"${cmdline}" ||
+    fail "the quote in the url field was not neutralised: ${cmdline}"
+grep -qF -- '+*.png%22' <<<"${cmdline}" ||
+    fail "the quote in the filter field was not neutralised: ${cmdline}"
+grep -qF -- 'http://x/a"b' <<<"${cmdline}" &&
+    fail "the url field still emits a raw quote: ${cmdline}"
+
+echo "PASS"

tests/84_webhttrack-mirror-verbatim.test

@@ -0,0 +1,167 @@
+#!/bin/bash
+#
+# htsserver serves the crawled mirror under /website/ alongside its own GUI:
+# mirrored pages must skip the ${...} expander, or ${_sid} leaks the session id.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+log=$(mktemp)
+work=$(mktemp -d)
+csrv=
+# start() runs in a command substitution, so its $! never reaches this shell. A
+# missed kill leaves an orphan holding the CI job open long after a green suite.
+srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
+cleanup() {
+    local pid
+    pid=$(srvpid)
+    test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
+    test -z "${csrv}" || kill -9 "${csrv}" 2>/dev/null || true
+    wait "${csrv}" 2>/dev/null || true # absorb bash's async "Killed" notice
+    rm -rf "${log}" "${work}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# GET $2 from 127.0.0.1:$1, headers into $3 and the body, byte for byte, into $4.
+fetch() {
+    python3 -c 'import socket, sys
+s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
+s.settimeout(20)
+s.sendall(("GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % sys.argv[2]).encode())
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+head, _, body = out.partition(b"\r\n\r\n")
+open(sys.argv[3], "wb").write(head)
+open(sys.argv[4], "wb").write(body)' "$1" "$2" "$3" "$4"
+}
+
+# POST the remaining args as urlencoded key=value fields to 127.0.0.1:$1.
+post() {
+    local port=$1
+    shift
+    python3 -c 'import socket, sys, urllib.parse
+body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe=""))
+                for k, v in (a.split("=", 1) for a in sys.argv[2:])).encode()
+s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
+s.settimeout(30)
+s.sendall(b"POST /step4.html HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+          b"Content-type: application/x-www-form-urlencoded\r\n"
+          b"Content-length: %d\r\n\r\n" % len(body) + body)
+while s.recv(65536):
+    pass
+s.close()' "${port}" "$@" >/dev/null
+}
+
+# LF-only, and a line ending in a backslash: the expander rewrites both, so a
+# mangled reply fails the byte comparison even where no directive is present.
+mirror="${work}/proj/hostile.html"
+hdr="${work}/hdr"
+body="${work}/body"
+
+# The server merges $HOME/.httrack.ini into the same store on the first request;
+# point it somewhere empty so a developer's own file cannot shadow the fields.
+export HOME="${work}"
+
+url=$(start)
+port=$(portof "${url}")
+
+# Positive control: the GUI's own templates must still expand. This is also
+# where the real token comes from, so its absence below can be asserted.
+fetch "${port}" /server/index.html "${hdr}" "${body}"
+sid=$(sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' "${body}" | head -1)
+test "${#sid}" -eq 32 || fail "GUI page did not expand \${sid} (got '${sid}')"
+
+# /website/ serves only the root the server itself recorded, so save a profile
+# (no command_do=start, so nothing crawls) to create it, then plant the file.
+post "${port}" "sid=${sid}" command=httrack command_do=save winprofile=x \
+    "path=${work}" projname=proj
+test -f "${work}/proj/hts-cache/winprofile.ini" ||
+    fail "profile save did not create the project: $(cat "${log}")"
+mkdir -p "$(dirname "${mirror}")"
+# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
+printf 'Hostile mirrored page.\nsid=${_sid} copy=${sid}\ntrailing backslash: \\\n' \
+    >"${mirror}"
+
+fetch "${port}" /website/hostile.html "${hdr}" "${body}"
+grep -q '^HTTP/1\.0 200 ' "${hdr}" || fail "mirrored page not served: $(head -1 "${hdr}")"
+grep -qF "${sid}" "${body}" &&
+    fail "the session id was expanded into mirrored content"
+# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
+grep -qF '${_sid}' "${body}" || fail "\${_sid} did not survive verbatim"
+# shellcheck disable=SC2016
+grep -qF '${sid}' "${body}" || fail "\${sid} did not survive verbatim"
+cmp -s "${mirror}" "${body}" || fail "mirrored file not served byte for byte"
+# The mirror stays browsable: verbatim must not mean served as a download.
+grep -qi '^Content-type: text/html' "${hdr}" ||
+    fail "mirrored page lost its text/html type: $(cat "${hdr}")"
+
+# The other direction: while a crawl runs, every .html request is overridden to
+# the GUI's own refresh page, so a /website/ URL stops naming mirrored content.
+# /trickle/ dribbles for a minute, which holds the crawl open for the probe.
+clog="${work}/content.log"
+python3 "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
+csrv=$!
+for _ in $(seq 1 40); do
+    cport=$(sed -n 's/^PORT //p' "${clog}") && test -n "${cport}" && break
+    kill -0 "${csrv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -n "${cport:-}" || fail "content server did not come up: $(cat "${clog}")"
+
+post "${port}" "sid=${sid}" "path=${work}" projname=crawl winprofile=x \
+    command_do=start \
+    "command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/trickle/ -O ${work}/crawl"
+
+fetch "${port}" /website/hostile.html "${hdr}" "${body}"
+grep -q '^HTTP/1\.0 200 ' "${hdr}" ||
+    fail "running crawl: /website/ was not overridden to the GUI page: $(head -1 "${hdr}")"
+grep -qF "'crawl' - HTTrack Website Copier" "${body}" ||
+    fail "the overridden GUI page was not the expanded refresh page"
+
+echo "PASS"

tests/85_webhttrack-projpath.test

@@ -0,0 +1,183 @@
+#!/bin/bash
+#
+# /website/ is served from the project directory htsserver set up itself, never
+# from a root the request body names, and composing that path must stay bounded.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+srv=
+log=$(mktemp)
+base=$(mktemp -d)
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    rm -f "${log}"
+    rm -rf "${base}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL. Runs in a command substitution, so it is a
+# subshell and cannot export the pid: the caller reads it back with srvpid.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+# The server reports its own pid; the aliveness assertion below hangs off it.
+srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
+
+alive() { kill -0 "$1" 2>/dev/null; }
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
+# $2 is empty. Prints the reply.
+request() {
+    python3 -c 'import socket, sys
+port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
+if path:
+    req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
+else:
+    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+           "Content-type: application/x-www-form-urlencoded\r\n"
+           "Content-length: %d\r\n\r\n%s" % (len(body), body))
+s = socket.create_connection(("127.0.0.1", port), 10)
+s.settimeout(30)
+s.sendall(req.encode())
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
+}
+
+get() { request "$1" "$2" ""; }
+post() { request "$1" "" "$2"; }
+
+# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
+# reads as marker-absent, and so do a truncated body and a 302 to the file.
+reply=
+fetch() {
+    reply=$(get "$1" "$2") || fail "the request for $2 failed"
+    grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
+        fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
+}
+
+url=$(start)
+port=$(portof "${url}")
+srv=$(srvpid)
+test -n "${srv}" || fail "htsserver did not report its pid"
+
+# Every request body is gated by the session id (78_webhttrack-sid.test).
+sid=$(get "${port}" /server/index.html |
+    sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
+
+# A file the mirror must never expose, next to the project that may.
+echo "SECRETMARKER" >"${base}/secret.txt"
+mkdir -p "${base}/proj"
+echo "LOGMARKER" >"${base}/proj/hts-log.txt"
+
+# error_redirect is the branch that skipped the fsfile clearing, so fail the save
+# with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
+# any successful save: commandEnd then swaps the error page for the finished one.
+get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
+saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
+grep -q '^Location: /server/error.html' <<<"${saved}" ||
+    fail "the refused save did not redirect to the error page"
+
+# No project yet, so no root to serve from.
+post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
+fetch "${port}" /website/secret.txt 404
+grep -q SECRETMARKER <<<"${reply}" &&
+    fail "a posted projpath served a file outside any project"
+
+# Positive control: step4's "save settings" flow registers the project without
+# crawling, and browsing its mirror keeps working. Without it the assertions
+# around it would pass on a server that never serves /website/ at all.
+body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
+body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
+post "${port}" "${body}" >/dev/null
+test -f "${base}/proj/hts-cache/winprofile.ini" ||
+    fail "the project was not registered: $(cat "${log}")"
+fetch "${port}" /website/hts-log.txt 200
+grep -q LOGMARKER <<<"${reply}" ||
+    fail "the registered project's mirror is not served"
+
+# Same request with the root repointed: the project is legitimate, projpath is
+# not what decides where the bytes come from.
+post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
+fetch "${port}" /website/secret.txt 404
+grep -q SECRETMARKER <<<"${reply}" &&
+    fail "a posted projpath repointed the served root"
+post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
+fetch "${port}" /website/passwd 404
+grep -q '^root:' <<<"${reply}" &&
+    fail "a posted projpath read an arbitrary system file"
+
+# A ".." anywhere in the recorded root would escape the mirror on every later
+# request, so the save must be refused and the previous root kept.
+post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
+fetch "${port}" /website/secret.txt 404
+grep -q SECRETMARKER <<<"${reply}" &&
+    fail "a '..' in the saved project path escaped the mirror root"
+fetch "${port}" /website/hts-log.txt 200
+grep -q LOGMARKER <<<"${reply}" ||
+    fail "rejecting the '..' root also lost the previous one"
+
+# The root is now the server's own, but it is still built from two posted
+# fields: 800-odd bytes of them used to be sprintf'd into a 1024-byte buffer.
+seg=$(printf '%0200d' 0)
+longpath="${base}/${seg}/${seg}/${seg}/${seg}"
+fspath="${longpath}/proj"
+# structcheck() refuses a root over HTS_URLMAXSIZE, so the URL carries the rest.
+test "$((${#fspath} + 11))" -le 1024 ||
+    fail "the long project path (${#fspath}) would not pass structcheck"
+longurl=$(printf '%0800d' 0)
+test "$((${#fspath} + 1 + ${#longurl}))" -gt 1024 ||
+    fail "the composed path (${#fspath} + ${#longurl}) would not overflow"
+post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${longpath}&projname=proj" >/dev/null
+test -f "${fspath}/hts-cache/winprofile.ini" ||
+    fail "the long-path project was not registered: $(cat "${log}")"
+get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
+alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
+# Not just alive: still answering.
+fetch "${port}" /server/index.html 200
+
+echo "PASS"

tests/86_local-proxytrack-cache-longfields.test

@@ -0,0 +1,111 @@
+#!/bin/bash
+# A cached header longer than proxytrack's field must be clipped to that field,
+# not overflow into the next one (contenttype[64] abuts the location pointer).
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+pad() { printf "%${1}s" '' | tr ' ' "$2"; }
+
+# Longest surviving run of char $2 in file $1, or 0.
+runlen() {
+    grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
+}
+
+# --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
+# Each header overshoots its own destination, so a per-field clip is the only
+# way to get every expected length right at once.
+{
+    printf 'HTTP/1.1 200 OK\r\n'
+    printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)"
+    printf 'Etag: %s\r\n' "$(pad 400 E)"
+    printf 'Content-Disposition: %s\r\n' "$(pad 400 D)"
+    printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
+    printf 'Content-Length: 5\r\n\r\n'
+} >"$dir/hdr"
+printf 'hello' >"$dir/body"
+alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+{
+    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+    printf '2 0 test\n'
+    printf '\n\n'
+    printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+    cat "$dir/hdr" "$dir/body"
+} >"$dir/in.arc"
+
+proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
+    echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2
+    exit 1
+}
+test -s "$dir/out.arc" || {
+    echo "FAIL: ARC entry dropped instead of clipped" >&2
+    exit 1
+}
+# Only contenttype survives into the ARC output; the over-long Etag and
+# Content-Disposition above are still load-bearing, since bounding just one
+# field leaves the others smashing the element (the run above would crash).
+got=$(runlen "$dir/out.arc" A)
+test "${got:-0}" = 54 || {
+    echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2
+    exit 1
+}
+
+# --- ZIP reader (ZIP_READFIELD_STRING) --------------------------------------
+python=$(find_python) || {
+    echo "python3 missing; ARC half only" >&2
+    exit 0
+}
+
+"$python" - "$dir/in.zip" <<'EOF'
+import sys, zipfile
+
+body = b"hello world"
+meta = (
+    "X-In-Cache: 1\r\n"
+    "X-StatusCode: 200\r\n"
+    "X-StatusMessage: ok%s\r\n"
+    "X-Size: %d\r\n"
+    "Content-Type: text/html%s\r\n"
+    "X-Charset: %s\r\n"
+    "Etag: %s\r\n"
+    "Content-Disposition: %s\r\n"
+    # a parseable date is required: the ARC writer dereferences it unchecked
+    "Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
+    "X-Save: out/page.html\r\n"
+    % ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400)
+).encode()
+
+zi = zipfile.ZipInfo("example.com/page.html")
+zi.compress_type = zipfile.ZIP_STORED
+zi.extra = meta
+with zipfile.ZipFile(sys.argv[1], "w") as z:
+    z.writestr(zi, body)
+EOF
+
+proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || {
+    echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2
+    exit 1
+}
+test -s "$dir/out2.arc" || {
+    echo "FAIL: zip entry dropped instead of clipped" >&2
+    exit 1
+}
+# three destinations, two capacities: msg[1024] keeps all 400 M's where a
+# one-size-fits-all clip would cut it to 63 like the others
+while read -r ch want; do
+    got=$(runlen "$dir/out2.arc" "$ch")
+    test "${got:-0}" = "$want" || {
+        echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2
+        exit 1
+    }
+done <<'EOF'
+A 54
+C 63
+M 400
+EOF

tests/87_local-proxytrack-nodate.test

@@ -0,0 +1,51 @@
+#!/bin/bash
+# A cached entry whose Last-Modified is missing or unparseable must still
+# convert: the ARC writer used to dereference the parsed date unchecked.
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+# $1 Last-Modified value (empty = omit the header), $2 expected archive date,
+# $3 label. The date is asserted exactly: a guard that fires unconditionally
+# would clobber the valid case, and one that skips the year/day would emit a
+# month and day of 00.
+run_case() {
+    lastmod=$1
+    wantdate=$2
+    what=$3
+
+    {
+        printf 'HTTP/1.1 200 OK\r\n'
+        printf 'Content-Type: text/html\r\n'
+        test -z "$lastmod" || printf 'Last-Modified: %s\r\n' "$lastmod"
+        printf 'Content-Length: 5\r\n\r\n'
+    } >"$dir/hdr"
+    printf 'hello' >"$dir/body"
+    alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+    {
+        printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+        printf '2 0 test\n'
+        printf '\n\n'
+        printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+        cat "$dir/hdr" "$dir/body"
+    } >"$dir/in.arc"
+
+    proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
+        echo "FAIL: proxytrack crashed on $what" >&2
+        exit 1
+    }
+    grep -aq "^http://example.com/p.html 0.0.0.0 ${wantdate} " "$dir/out.arc" || {
+        echo "FAIL: $what: expected archive date ${wantdate}, got:" >&2
+        grep -a '^http://example.com/' "$dir/out.arc" >&2 || echo "(entry dropped)" >&2
+        exit 1
+    }
+}
+
+# the epoch stands in for "date unknown"
+run_case '' 19700101000000 'a missing Last-Modified'
+run_case 'not a date at all' 19700101000000 'an unparseable Last-Modified'
+run_case '0' 19700101000000 'a bare 0 Last-Modified'
+# a parseable date must still come through untouched
+run_case 'Sun, 06 Nov 1994 08:49:37 GMT' 19941106084937 'a valid Last-Modified'

tests/89_local-sitemap.test

@@ -0,0 +1,134 @@
+#!/bin/bash
+#
+# --sitemap seeds the crawl from robots.txt -> sitemapindex -> gzipped urlset.
+# start.html links to nothing, so orphan*.html can only arrive through the
+# sitemap; deep1.html proves the seeds keep a full depth budget under -r2, and
+# the off-host page and child sitemap must both be refused.
+
+set -eu
+
+: "${top_srcdir:=..}"
+
+crawl() { bash "$top_srcdir/tests/local-crawl.sh" "$@"; }
+
+# robots.txt Sitemap: -> index -> .xml.gz, seeds behaving like -r2 seeds. The
+# log assertions pin which route was taken: the two documents are served from
+# both /sitemapdir/index.xml and the well-known /sitemap.xml, so "some sitemap
+# was read" would pass either way. --not-found pins ingestion-only: the sitemap
+# documents feed the crawl but never land in the mirror.
+# --rerun also walks the update path over a mirror holding sitemap seeds.
+crawl --errors 0 --rerun \
+    --found 'sitemapdir/orphan1.html' \
+    --found 'sitemapdir/orphan2.html' \
+    --found 'sitemapdir/deep1.html' \
+    --not-found 'sitemapdir/index.xml' \
+    --not-found 'sitemapdir/pages.xml.gz' \
+    --log-found '2 of 3 URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
+    --log-found '1 of 2 child sitemap\(s\) listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \
+    --log-found 'ignoring off-host child sitemap' \
+    --log-not-found '/sitemap\.xml' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2
+
+# Negative control: without the option nothing but the start page is reached.
+crawl --errors 0 \
+    --found 'sitemapdir/start.html' \
+    --not-found 'sitemapdir/orphan1.html' \
+    --not-found 'sitemapdir/orphan2.html' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2
+
+# Sitemap URLs are not a filter bypass: a -*orphan2* rule still rejects one.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --not-found 'sitemapdir/orphan2.html' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 '-*orphan2*'
+
+# A robots.txt naming no sitemap falls back to the well-known /sitemap.xml.
+# The test server drops its Sitemap: record for this User-Agent.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemap\.xml' \
+    --log-not-found 'sitemapdir/index\.xml' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -F 'nositemap-agent'
+
+# --sitemap-url alone reads the named document and probes nothing.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --found 'sitemapdir/orphan2.html' \
+    --log-not-found 'sitemapdir/index\.xml' \
+    --log-not-found '/sitemap\.xml' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/pages.xml.gz'
+
+# The two options are additive: an unfetchable --sitemap-url is parsed as an
+# empty document and does not stop --sitemap from seeding the crawl.
+crawl --errors-content 1 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-found 'Sitemap: 0 of 0 URL\(s\) added' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/missing.xml'
+
+# The sitemapindex nesting cap stops the chain before its deepest urlset. The
+# positive control is the page one level inside the cap: without it, a cap
+# mutated to 0 (nothing followed) would pass this just as well.
+crawl --errors 0 \
+    --found 'sitemapdir/cap3.html' \
+    --not-found 'sitemapdir/cap4.html' \
+    --log-found 'Sitemap: cap reached' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/chain0.xml'
+
+# A site cannot widen a subtree crawl by putting its sitemap at the root: the
+# page below the start directory is seeded, the one above it is refused.
+crawl --errors 0 \
+    --found 'deep/dir/below.html' \
+    --not-found 'elsewhere/updir.html' \
+    httrack 'BASEURL/deep/dir/start.html' --sitemap -r3 -F 'scopesitemap-agent'
+
+# How far a sitemap fetch is gated depends on who asked for it, so the three
+# cases below have to differ: one "robots is honoured" assertion would hide a
+# regression in either direction. The harness disables robots, hence -s2 here.
+
+# We guessed /sitemap.xml, so a Disallow on it wins.
+crawl --errors 0 \
+    --not-found 'sitemapdir/orphan1.html' \
+    --log-found 'Sitemap: robots.txt forbids' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
+    -F 'denysitemap-agent'
+
+# The site declared it through a Sitemap: line, which the same Disallow does
+# not retract.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-not-found 'Sitemap: robots.txt forbids' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
+    -F 'denydeclared-agent'
+
+# The user named it, which is the same intent as a start URL: not refused
+# either, even though robots.txt forbids that exact path.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-not-found 'Sitemap: robots.txt forbids' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 -s2 -F 'denysitemap-agent' \
+    --sitemap-url 'BASEURL/sitemap.xml'
+
+# A child sitemap is a fetch like any other: a filter rejecting it stops the
+# whole subtree, so the page only it lists is never reached.
+crawl --errors 0 \
+    --not-found 'sitemapdir/gated.html' \
+    --log-found 'Sitemap: filter rule #[0-9]+ refuses' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 '-*filtered.xml*' \
+    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
+
+# Control: without that filter the same chain does reach the page.
+crawl --errors 0 \
+    --found 'sitemapdir/gated.html' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
+
+# A redirected sitemap is still ingested: the marking follows the 301.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-found 'moved\.xml redirects to .*pages\.xml\.gz' \
+    --log-found 'URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/moved.xml'

tests/90_webhttrack-checkbox-clear.test

@@ -0,0 +1,196 @@
+#!/bin/bash
+#
+# An unchecked box posts nothing and the stored "1" survives: hence a hidden
+# companion per box, plus a disabling flag for the default-on ones.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
+srvlog=$(mktemp)
+srv=
+srvpid=
+cleanup() {
+    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
+    test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
+    rm -rf "${work}" "${srvlog}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
+sport=$("${python}" -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()')
+(
+    trap '' TERM TTOU
+    export HOME="${work}"
+    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
+) &
+srv=$!
+for _ in $(seq 1 40); do
+    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
+    kill -0 "${srv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
+srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
+
+"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
+import glob, os, re, sys, urllib.parse, urllib.request
+
+url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
+rc = 0
+
+
+def check(ok, what):
+    global rc
+    print(("ok: " if ok else "FAIL: ") + what)
+    if not ok:
+        rc = 1
+
+
+# cache/cache2 are left to the separate rework of the dead "Cache" seed.
+skip = {"cache", "cache2"}
+page_of = {}
+boxes = 0
+for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
+    page = open(path, "rb").read().decode("latin-1")
+    cleared = dict((m.group(1), m.start()) for m in
+                   re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
+    for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
+        boxes += 1
+        if m.group(1) in skip:
+            continue
+        page_of[m.group(1)] = os.path.basename(path)
+        at = cleared.get(m.group(1))
+        check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
+              % (os.path.basename(path), m.group(1)))
+# Control: a regex that stopped matching would leave nothing to assert on.
+check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
+
+sid = None
+
+
+def get(path):
+    # The UI is served ISO-8859-1, so decode, do not assume UTF-8.
+    return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
+
+
+def post(fields):
+    global sid
+    if sid is None:
+        m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
+        if m is None:
+            sys.exit("no session id in server/index.html")
+        sid = m.group(1)
+    body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
+                    [("sid", sid)] + fields)
+    req = urllib.request.Request(url + "/server/step4.html",
+                                 data=body.encode("latin-1"), method="POST")
+    return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
+
+
+def textarea(page, name):
+    m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
+    if m is None:
+        sys.exit("no %s textarea in the rendered step4.html" % name)
+    return m.group(1)
+
+
+def checked(page, name):
+    return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
+
+
+# What each state puts on the command line (None: nothing) and its profile key.
+# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
+# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
+BOXES = [
+    # field           profile key           set                          cleared
+    ("parseall", "ParseAll", "--near", None),
+    ("link", "Near", "--test", None),
+    ("testall", "Test", "--extended-parsing", None),
+    # htmlfirst emits --priority=7, which the scan-priority list also emits.
+    ("htmlfirst", "HTMLFirst", None, None),
+    ("errpage", "NoErrorPages", "--generate-errors=0", None),
+    ("external", "NoExternalPages", "--replace-external", None),
+    ("hidepwd", "NoPwdInPages", "--disable-passwords", None),
+    ("hidequery", "NoQueryStrings", "--include-query-string=0", None),
+    ("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
+    ("windebug", None, "--debug-headers", None),
+    ("ka", "KeepAlive", "--keep-alive", None),
+    ("remt", "RemoveTimeout", "--host-control=1", None),
+    ("rems", "RemoveRateout", "--host-control=2", None),
+    ("cookies", "Cookies", None, "--cookies=0"),
+    ("parsejava", "ParseJava", None, "--parse-java=0"),
+    ("updhack", "UpdateHack", "--updatehack", None),
+    ("urlhack", "URLHack", "--urlhack", None),
+    ("keepwww", "KeepWww", "--keep-www-prefix", None),
+    ("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
+    ("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
+    ("toler", "TolerantRequests", "--tolerant", None),
+    ("http10", "HTTP10", "--http-10", None),
+    ("sitemap", "Sitemap", "--sitemap", None),
+    ("warc", "Warc", "--warc", None),
+    ("norecatch", "NoRecatch", "--do-not-recatch", None),
+    ("logf", "Log", "--single-log", None),
+    ("index", "Index", None, None),
+    ("index2", "WordIndex", "--search-index", None),
+    ("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
+]
+
+for field, key, when_set, when_clear in BOXES:
+    for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
+                                          ("", when_clear, when_set, "0")):
+        rendered = post([(field, value)])
+        cmd = textarea(rendered, "command").split()
+        label = "%s %s" % (field, "set" if value else "cleared")
+        if want:
+            check(want in cmd, "%s emits %s" % (label, want))
+        if unwanted:
+            check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
+        if key:
+            # The Windows GUI reads the same option out of the profile.
+            check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
+                  "%s writes %s=%s" % (label, key, stored))
+        check(checked(page_of[field], field) == bool(value),
+              "%s draws %s box" % (label, "a ticked" if value else "an empty"))
+missing = sorted(set(page_of) - set(b[0] for b in BOXES))
+check(not missing, "every checkbox is exercised (missing %s)" % missing)
+
+# A browser posts the companion and the ticked box under one name, so the fix
+# rests on the body loop keeping the last value; both orders pin that down.
+cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
+check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
+check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
+cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
+check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
+check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
+
+sys.exit(rc)
+PY
+
+# A leaked htsserver wedges the parallel harness behind a green log.
+cleanup
+! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
+
+echo "PASS"

tests/Makefile.am

@@ -32,6 +32,7 @@ TESTS = \
 	00_runnable.test \
 	01_engine-charset.test \
 	01_engine-cmdline.test \
+	01_engine-cmdline-split.test \
 	01_engine-cookies.test \
 	01_engine-copyopt.test \
 	01_engine-crange.test \
@@ -64,6 +65,7 @@ TESTS = \
 	01_engine-reconcile.test \
 	01_engine-expandhome.test \
 	01_engine-fsize.test \
+	01_engine-growsize.test \
 	01_engine-redirect.test \
 	01_engine-longpath-io.test \
 	01_engine-mirror-io.test \
@@ -88,6 +90,7 @@ TESTS = \
 	01_engine-xfread.test \
 	01_zlib-acceptencoding.test \
 	01_zlib-warc.test \
+	01_zlib-sitemap.test \
 	01_zlib-warc-cdx.test \
 	01_zlib-warc-wacz.test \
 	01_zlib-contentcodings.test \
@@ -147,6 +150,7 @@ TESTS = \
 	50_local-contentcodings.test \
 	51_local-update-codec.test \
 	52_local-socks5.test \
+	53_local-proxytrack-arc-reason.test \
 	53_local-proxytrack-cache-corrupt.test \
 	54_local-update-truncate-purge.test \
 	55_local-chunked.test \
@@ -170,6 +174,19 @@ TESTS = \
 	74_local-warc-wacz.test \
 	74_local-warc-verbatim.test \
 	75_engine-longpath-posix.test \
-	76_cli-resize.test
+	76_cli-resize.test \
+	77_webhttrack-redirect.test \
+	78_webhttrack-sid.test \
+	79_local-proxytrack-webdav-mime.test \
+	80_engine-crash-symbolize.test \
+	81_webhttrack-maxsize.test \
+	82_webhttrack-browse-links.test \
+	83_webhttrack-argescape.test \
+	84_webhttrack-mirror-verbatim.test \
+	85_webhttrack-projpath.test \
+	86_local-proxytrack-cache-longfields.test \
+	87_local-proxytrack-nodate.test \
+	89_local-sitemap.test \
+	90_webhttrack-checkbox-clear.test
 
 CLEANFILES = check-network_sh.cache

tests/local-server.py

@@ -18,6 +18,7 @@ import base64
 import gzip
 import hashlib
 import os
+import re
 import sys
 import time
 from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
@@ -542,8 +543,36 @@ class Handler(SimpleHTTPRequestHandler):
             return self.fail_cookie(name)
         self.send_html("\tThis is the secret.")
 
+    # A User-Agent carrying NO_SITEMAP_UA gets a robots.txt with no Sitemap:
+    # record, so a test can drive the /sitemap.xml fallback instead.
+    NO_SITEMAP_UA = "nositemap"
+    # ... and one that additionally Disallows the well-known location, so the
+    # fallback has to be refused by the rules this very body carries.
+    DENY_SITEMAP_UA = "denysitemap"
+    # ... the same Disallow, but with the sitemap also declared: the
+    # declaration is the site inviting the fetch and must win.
+    DENY_DECLARED_UA = "denydeclared"
+    # ... and one that points the sitemap at the site root, to check a subtree
+    # crawl is not widened by where the site chooses to put its sitemap.
+    SCOPE_SITEMAP_UA = "scopesitemap"
+
     def route_robots(self):
-        body = b"User-agent: *\nDisallow:\n"
+        # The Sitemap: record is group-independent; only --sitemap acts on it.
+        ua = self.headers.get("User-Agent") or ""
+        host = self.headers.get("Host")
+        body = "User-agent: *\nDisallow:\n"
+        if self.DENY_DECLARED_UA in ua:
+            body = (
+                "User-agent: *\nDisallow: /sitemap.xml\n"
+                f"Sitemap: http://{host}/sitemap.xml\n"
+            )
+        elif self.DENY_SITEMAP_UA in ua:
+            body = "User-agent: *\nDisallow: /sitemap.xml\n"
+        elif self.SCOPE_SITEMAP_UA in ua:
+            body += f"Sitemap: http://{host}/scopesitemap.xml\n"
+        elif self.NO_SITEMAP_UA not in ua:
+            body += f"Sitemap: http://{host}/sitemapdir/index.xml\n"
+        body = body.encode()
         self.send_response(200)
         self.send_header("Content-Type", "text/plain")
         self.send_header("Content-Length", str(len(body)))
@@ -551,6 +580,132 @@ class Handler(SimpleHTTPRequestHandler):
         if self.command != "HEAD":
             self.wfile.write(body)
 
+    # --- sitemap ingestion (issue #712) ------------------------------------
+    # start.html links to nothing, so orphan*.html are reachable only through
+    # the sitemap. deep1.html proves the seeds keep a full depth budget; the
+    # off-host page <loc> must be dropped by the travel scope, and the off-host
+    # child sitemap by the ingester's same-host rule. The index is served both
+    # from /sitemapdir/ (named by robots.txt) and from the well-known
+    # /sitemap.xml (the fallback).
+
+    def route_sitemap_index(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?>\n'
+            '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
+            f"<sitemap><loc>http://{host}/sitemapdir/pages.xml.gz</loc></sitemap>"
+            "<sitemap><loc>http://sitemap-offhost.invalid/s.xml</loc></sitemap>"
+            "</sitemapindex>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_pages(self):
+        host = self.headers.get("Host")
+        xml = (
+            '<?xml version="1.0" encoding="UTF-8"?>\n'
+            '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
+            f"<url><loc>http://{host}/sitemapdir/orphan1.html</loc></url>"
+            "<url><loc>http://sitemap-offhost.invalid/x.html</loc></url>"
+            f"<url><loc>http://{host}/sitemapdir/orphan2.html</loc></url>"
+            "</urlset>\n"
+        ).encode()
+        self.send_raw(gzip.compress(xml), "application/x-gzip")
+
+    def route_sitemap_start(self):
+        self.send_html("\tNothing links to the sitemap pages.")
+
+    def route_sitemap_orphan1(self):
+        self.send_html('\t<a href="deep1.html">deeper</a>')
+
+    def route_sitemap_orphan2(self):
+        self.send_html("\tSecond orphan.")
+
+    def route_sitemap_deep1(self):
+        self.send_html("\tOne level below an orphan.")
+
+    # chainN is a sitemapindex at nesting level N, listing chain(N+1) and a
+    # urlset capN.xml whose single page is capN.html. Levels up to
+    # HTS_SITEMAP_MAX_LEVEL are followed, so capN.html appears for N below the
+    # cap and stops appearing at it: the pair pins the boundary, which a cap
+    # mutated either way would break.
+    def route_sitemap_chain(self):
+        host = self.headers.get("Host")
+        level = int(self.path.rsplit("/", 1)[-1][len("chain") : -len(".xml")])
+        self.send_raw(
+            (
+                '<?xml version="1.0" encoding="UTF-8"?>\n<sitemapindex>'
+                f"<sitemap><loc>http://{host}/sitemapdir/chain{level + 1}.xml"
+                "</loc></sitemap>"
+                f"<sitemap><loc>http://{host}/sitemapdir/cap{level}.xml"
+                "</loc></sitemap></sitemapindex>\n"
+            ).encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_capset(self):
+        host = self.headers.get("Host")
+        level = self.path.rsplit("/", 1)[-1][len("cap") : -len(".xml")]
+        self.send_raw(
+            (
+                '<?xml version="1.0" encoding="UTF-8"?>\n<urlset>'
+                f"<url><loc>http://{host}/sitemapdir/cap{level}.html</loc></url>"
+                "</urlset>\n"
+            ).encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_cappage(self):
+        self.send_html("\tReached through a nested sitemapindex.")
+
+    def route_sitemap_gatedindex(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?><sitemapindex>'
+            f"<sitemap><loc>http://{host}/sitemapdir/filtered.xml</loc></sitemap>"
+            "</sitemapindex>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_filtered(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
+            f"<url><loc>http://{host}/sitemapdir/gated.html</loc></url>"
+            "</urlset>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_gated(self):
+        self.send_html("\tListed only by the filtered child sitemap.")
+
+    # A moved sitemap: the marking has to follow the redirect.
+    # A root sitemap naming a page below the crawl's start directory and one
+    # above it. Only the first may be seeded when the crawl started at /deep/dir/.
+    def route_sitemap_scope(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
+            f"<url><loc>http://{host}/deep/dir/below.html</loc></url>"
+            f"<url><loc>http://{host}/elsewhere/updir.html</loc></url>"
+            "</urlset>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_deepstart(self):
+        self.send_html("\tA start page in a subdirectory, linking nothing.")
+
+    def route_sitemap_below(self):
+        self.send_html("\tBelow the start directory.")
+
+    def route_sitemap_updir(self):
+        self.send_html("\tAbove the start directory.")
+
+    def route_sitemap_moved(self):
+        self.send_response(301)
+        self.send_header("Location", "/sitemapdir/pages.xml.gz")
+        self.send_header("Content-Length", "0")
+        self.end_headers()
+
     # --- type/extension matrix (issue #267 family) -------------------------
 
     def send_raw(self, body, content_type, extra_headers=()):
@@ -1578,6 +1733,21 @@ class Handler(SimpleHTTPRequestHandler):
         "/gated/index.php": route_gated_index,
         "/gated/secret.php": route_gated_secret,
         "/robots.txt": route_robots,
+        "/sitemapdir/index.xml": route_sitemap_index,
+        "/sitemap.xml": route_sitemap_index,
+        "/sitemapdir/pages.xml.gz": route_sitemap_pages,
+        "/sitemapdir/start.html": route_sitemap_start,
+        "/sitemapdir/orphan1.html": route_sitemap_orphan1,
+        "/sitemapdir/orphan2.html": route_sitemap_orphan2,
+        "/sitemapdir/deep1.html": route_sitemap_deep1,
+        "/sitemapdir/gatedindex.xml": route_sitemap_gatedindex,
+        "/sitemapdir/filtered.xml": route_sitemap_filtered,
+        "/sitemapdir/gated.html": route_sitemap_gated,
+        "/sitemapdir/moved.xml": route_sitemap_moved,
+        "/scopesitemap.xml": route_sitemap_scope,
+        "/deep/dir/start.html": route_sitemap_deepstart,
+        "/deep/dir/below.html": route_sitemap_below,
+        "/elsewhere/updir.html": route_sitemap_updir,
         "/warcgz/index.html": route_warcgz_index,
         "/warcgz/page.html": route_warcgz_page,
         "/warcgz/data.bin": route_warcgz_data,
@@ -1902,6 +2072,13 @@ class Handler(SimpleHTTPRequestHandler):
             return True
         # Match percent-encoded paths (accented #157 route) by their decoded form.
         handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path))
+        if handler is None:
+            if re.fullmatch(r"/sitemapdir/chain\d+\.xml", path):
+                handler = type(self).route_sitemap_chain
+            elif re.fullmatch(r"/sitemapdir/cap\d+\.xml", path):
+                handler = type(self).route_sitemap_capset
+            elif re.fullmatch(r"/sitemapdir/cap\d+\.html", path):
+                handler = type(self).route_sitemap_cappage
         if handler is not None:
             handler(self)
             return True

tests/webhttrack-smoke.sh

@@ -46,12 +46,14 @@ cat >"$stubdir/x-www-browser" <<EOF
 echo "stub browser invoked with: \$1" >&2
 # Also fetch an option page and require a rendered title='' tooltip: proves the
 # option template expands and the \${html:} filter escapes into the attribute.
-# option9 additionally proves the WARC control renders with its expanded label.
+# option9/option8 additionally prove the WARC and sitemap controls render.
 opturl="\${1%/}/server/option2.html"
 warcurl="\${1%/}/server/option9.html"
+smurl="\${1%/}/server/option8.html"
 if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html &&
     opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" &&
-    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then
+    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC &&
+    sm="\$(curl -fsSL --max-time 20 "\$smurl")" && printf '%s' "\$sm" | grep -qaF 'name="sitemapurl"' && printf '%s' "\$sm" | grep -qaF 'name="sitemap"'; then
     echo PASS >"$marker"
 else
     echo "FAIL: unexpected response from \$1" >"$marker"