sitemap: translate the new GUI strings into the remaining 28 locales

The feature PR added the four LANG_SITEMAP* entries to lang.def with English
and Francais only; every other locale fell back to English in the WebHTTrack
form. Each file is written in its own declared charset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/workflows/windows-build.yml

@@ -226,9 +226,8 @@ jobs:
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
           # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
-          # webdav-mime needs a reapable background listener, which MSYS cannot give it;
-          # badmtime needs a filesystem that stores an mtime past gmtime's range.
-          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test"
+          # webdav-mime needs a reapable background listener, which MSYS cannot give it.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -26,12 +26,6 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
-- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
-  pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
-  and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
-  as "marker absent". Capture the reply, assert the status line it must carry
-  (an empty, truncated or redirected one is marker-free too), then match with a
-  here-string.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -54,16 +48,6 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
-- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
-  (`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
-  our own data, wrong for anything read back from a cache, a header or the
-  wire, where it trades a memory smash for a crash on malformed input. Clip
-  with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
-- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
-  a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
-  into the same buffer never appears on it. Before scoping a hardening pass off
-  compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
-  `\bstrcpy\s*\(`, `\bstrcat\s*\(`).
 
 ## C conventions
 - **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -100,17 +84,6 @@ Before pushing, and when reviewing others, don't skim for bugs:
   layout/ABI, cache/wire format, or a security path? A static or unit check
   isn't enough; exercise the wrong behavior at runtime. Claude Code:
   `/review-recipe`.
-- **Poison a canary, never compare it against zero.** Checking that a
-  neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
-  terminator writes — the exact bug the canary is there for. Fill it with a
-  non-zero byte, and prove it by killing both the stray-`'X'` and the
-  stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
-  lands inside the same struct.
-- **Overshoot every destination, not one.** A bounds test that oversizes a
-  single field cannot tell a per-field bound from a one-size-fits-all one, nor
-  from a fix that bounds that field and leaves its neighbours raw. Exercise
-  each destination the path touches, spanning at least two capacities, and
-  check what the code actually emits before writing the expected values.
 
 ## Commits
 - **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

fuzz/Makefile.am

@@ -2,7 +2,7 @@
 if FUZZERS
 noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \
 	fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \
-	fuzz-htsparse
+	fuzz-htsparse fuzz-sitemap
 endif
 
 AM_CPPFLAGS = \
@@ -27,6 +27,7 @@ fuzz_url_SOURCES = fuzz-url.c fuzz.h
 fuzz_header_SOURCES = fuzz-header.c fuzz.h
 fuzz_cachendx_SOURCES = fuzz-cachendx.c fuzz.h
 fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h
+fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h
 
 # List corpus files explicitly: automake does not expand EXTRA_DIST globs.
 EXTRA_DIST = README.md run-fuzzers.sh \
@@ -47,4 +48,6 @@ EXTRA_DIST = README.md run-fuzzers.sh \
 	corpus/cachendx/regress-overadvance.bin \
 	corpus/cachendx/regress-truncated-entry.bin \
 	corpus/htsparse/basic.html corpus/htsparse/script-inscript.html \
-	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html
+	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html \
+	corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
+	corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz

fuzz/corpus/sitemap/sitemapindex.xml

@@ -0,0 +1 @@
+<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc></sitemap></sitemapindex>

fuzz/corpus/sitemap/truncated.xml

@@ -0,0 +1 @@
+<urlset><loc>http://h.test/x

fuzz/corpus/sitemap/urlset.xml

@@ -0,0 +1 @@
+
<?xml version="1.0"?><urlset><url><loc>http://h.test/a.html</loc></url><url><loc>https://h.test/b?x=1&amp;y=2</loc></url></urlset>

fuzz/fuzz-sitemap.c

@@ -0,0 +1,60 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* Fuzz the sitemap <loc> scanner (htssitemap.c): raw XML, gzip-framed bodies
+   and truncated streams all arrive here straight off the network. */
+#include "fuzz.h"
+#include "htssitemap.h"
+
+static hts_boolean sm_count(void *arg, const char *url) {
+  int *const n = (int *) arg;
+
+  (void) url;
+  (*n)++;
+  return HTS_TRUE;
+}
+
+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
+  static const int caps[] = {0, 1, 16, HTS_SITEMAP_MAX_URLS_DOC};
+  hts_boolean is_index;
+  char *body;
+  int n = 0, cap;
+
+  if (size == 0)
+    return 0;
+  cap = caps[data[0] % (sizeof(caps) / sizeof(caps[0]))];
+  data++, size--;
+  /* A heap copy of exactly `size` bytes: the scanner must never rely on a
+     terminator, and ASan turns any overread into a report. */
+  body = malloct(size != 0 ? size : 1);
+  memcpy(body, data, size);
+
+  (void) hts_sitemap_scan(body, size, cap, &is_index, sm_count, &n);
+
+  freet(body);
+  return 0;
+}

html/cmdguide.html

@@ -163,8 +163,26 @@ the index" problems disappear.</p>
   <tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
   <tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
   <tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
+  <tr><td><tt>--sitemap (-%m), --sitemap-url URL (-%mu)</tt></td><td>Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.</td></tr>
 </table>
 
+<p>Link-following only finds what something links to. Anything a site publishes
+solely in its sitemap is invisible to HTTrack unless you ask for it.
+<tt>--sitemap</tt> reads the start host's <tt>robots.txt</tt> for
+<tt>Sitemap:</tt> lines and falls back to <tt>/sitemap.xml</tt>;
+<tt>--sitemap-url</tt> names one directly. Nested <tt>sitemapindex</tt> files
+and gzipped <tt>.xml.gz</tt> sitemaps are followed. The URLs found become start
+URLs with the full depth budget, but they still go through your filters and
+scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is
+off by default because a sitemap can list thousands of pages nothing links
+to.</p>
+
+<p>One surprise worth knowing: a sitemap you name with <tt>--sitemap-url</tt>,
+and one the site itself declares in <tt>robots.txt</tt>, are fetched even when
+<tt>robots.txt</tt> disallows that path, because naming or declaring a sitemap
+is an invitation to read it. Only the guessed <tt>/sitemap.xml</tt> obeys a
+<tt>Disallow</tt>. The URLs listed inside are gated normally either way.</p>
+
 <p>The single most common surprise is "only the home page came down." That is
 usually not a scope option at all: it is an off-host redirect. A start URL of
 <tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>

html/httrack.man.html

@@ -87,8 +87,8 @@ offline browser : copy websites to a local directory</p>
 --host-control[=N]</b> ] [ <b>-%P,
 --extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
 --test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
-] [ <b>-NN, --structure[=N]</b> ] [ <b>-%N,
---delayed-type-check</b> ] [ <b>-%D,
+] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
+[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
 --cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
 ] [ <b>-LN, --long-names[=N]</b> ] [ <b>-KN,
 --keep-links[=N]</b> ] [ <b>-x, --replace-external</b> ] [
@@ -575,6 +575,22 @@ URL per line) (--list &lt;param&gt;)</p></td></tr>
 
 <p>&lt;file&gt; add all scan rules located in this text
 file (one scan rule per line) (--urllist &lt;param&gt;)</p></td></tr>
+<tr valign="top" align="left">
+<td width="9%"></td>
+<td width="4%">
+
+
+<p>-%m</p></td>
+<td width="5%"></td>
+<td width="82%">
+
+
+<p>seed the crawl from the site&rsquo;s sitemap (robots.txt
+Sitemap:, then /sitemap.xml); --sitemap-url URL names one
+explicitly. A sitemap you name, or one the site declares, is
+fetched even under robots.txt Disallow; only the guessed
+/sitemap.xml obeys it. The URLs found still pass every
+filter and scope rule (--sitemap)</p></td></tr>
 </table>
 
 <h3>Build options:

html/server/option10.html

@@ -98,9 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="ftpprox" value="">
-
 ${LANG_PROXYTYPE}
 <select name="proxytype"
 			title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

html/server/option2.html

@@ -98,13 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="errpage" value="">
-<input type="hidden" name="external" value="">
-<input type="hidden" name="hidepwd" value="">
-<input type="hidden" name="hidequery" value="">
-<input type="hidden" name="nopurge" value="">
-
 ${LANG_I33}
 <br>
 <select name="build"

html/server/option3.html

@@ -98,9 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="windebug" value="">
-
 ${LANG_I40c}
 <br>
 

html/server/option4.html

@@ -98,11 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="ka" value="">
-<input type="hidden" name="remt" value="">
-<input type="hidden" name="rems" value="">
-
 <table border="0" width="100%" cellspacing="0">
 
 <tr><td>

html/server/option8.html

@@ -98,17 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="cookies" value="">
-<input type="hidden" name="parsejava" value="">
-<input type="hidden" name="updhack" value="">
-<input type="hidden" name="urlhack" value="">
-<input type="hidden" name="keepwww" value="">
-<input type="hidden" name="keepslashes" value="">
-<input type="hidden" name="keepqueryorder" value="">
-<input type="hidden" name="toler" value="">
-<input type="hidden" name="http10" value="">
-
 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I58}
@@ -143,6 +132,17 @@ ${listid:robots:LISTDEF_8}
 </select>
 <br><br>
 
+<input type="checkbox" name="sitemap" ${checked:sitemap}
+			title='${html:LANG_SITEMAPTIP}' onMouseOver="info('${html:LANG_SITEMAPTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+> ${LANG_SITEMAP}
+<br><br>
+
+${LANG_SITEMAPURL}
+<input name="sitemapurl" value="${sitemapurl}" size="40"
+			title='${html:LANG_SITEMAPURLTIP}' onMouseOver="info('${html:LANG_SITEMAPURLTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
+>
+<br><br>
+
 <input type="checkbox" name="updhack" ${checked:updhack}
 			title='${html:LANG_I1k}' onMouseOver="info('${html:LANG_I1k}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I62b}

html/server/option9.html

@@ -98,13 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="warc" value="">
-<input type="hidden" name="norecatch" value="">
-<input type="hidden" name="logf" value="">
-<input type="hidden" name="index" value="">
-<input type="hidden" name="index2" value="">
-
 <input type="checkbox" name="cache2" ${checked:cache2}
 			title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I61}

html/server/step2.html

@@ -141,6 +141,8 @@ ${do:copy:KeepSlashes:keepslashes}
 ${do:copy:KeepQueryOrder:keepqueryorder}
 ${do:copy:StripQuery:stripquery}
 ${do:copy:StoreAllInCache:cache2}
+${do:copy:Sitemap:sitemap}
+${do:copy:SitemapUrl:sitemapurl}
 ${do:copy:Warc:warc}
 ${do:copy:WarcFile:warcfile}
 ${do:copy:LogType:logtype}

html/server/step4.html

@@ -114,7 +114,7 @@ ${do:end-if}
 
 ${/* Real commands and ini file generated below */}
 
-<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
+<!-- engine commandline -->
 ${do:output-mode:html}
 <textarea name="command" cols="50" rows="4" style="visibility:hidden">
 httrack \
@@ -175,8 +175,8 @@ ${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one *
 \
 	${unquoted:url2}
 \
-	${ztest:cookies:--cookies=0:}
-	${ztest:parsejava:--parse-java=0:}
+	${test:cookies:--cookies=0:}
+	${test:parsejava:--parse-java=0:}
 	${test:updhack:--updatehack}
 	${test:urlhack:--urlhack=0:--urlhack}
 	${test:keepwww:--keep-www-prefix}
@@ -188,6 +188,8 @@ ${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one *
 	${test:toler:--tolerant}
 	${test:http10:--http-10}
 	${test:cache2:--store-all-in-cache}
+	${test:sitemap:--sitemap}
+	${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"}
 	${test:warc:--warc}
 	${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
 	${test:norecatch:--do-not-recatch}
@@ -240,6 +242,8 @@ KeepSlashes=${ztest:keepslashes:0:1}
 KeepQueryOrder=${ztest:keepqueryorder:0:1}
 StripQuery=${stripquery}
 StoreAllInCache=${ztest:cache2:0:1}
+Sitemap=${ztest:sitemap:0:1}
+SitemapUrl=${sitemapurl}
 Warc=${ztest:warc:0:1}
 WarcFile=${warcfile}
 LogType=${logtype}

lang.def

@@ -1042,3 +1042,11 @@ LANG_WARCFILE
 WARC archive name:
 LANG_WARCFILETIP
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+LANG_SITEMAP
+Seed the crawl from the site's sitemap
+LANG_SITEMAPTIP
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+LANG_SITEMAPURL
+Sitemap address:
+LANG_SITEMAPURLTIP
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/Bulgarian.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Èìå íà WARC àðõèâà:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåçàäúëæèòåëíî áàçîâî èìå çà WARC àðõèâà; îñòàâåòå ïðàçíî çà àâòîìàòè÷íî èìåíóâàíå â èçõîäíàòà äèðåêòîðèÿ.
+Seed the crawl from the site's sitemap
+Çàïî÷âàíå íà îáõîæäàíåòî îò êàðòàòà íà ñàéòà
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ïðî÷èòàíå íà êàðòàòà íà ñàéòà (ðåäîâåòå Sitemap: â robots.txt, ñëåä òîâà /sitemap.xml) è äîáàâÿíå íà âñåêè ïîñî÷åí URL àäðåñ êàòî íà÷àëåí.
+Sitemap address:
+Àäðåñ íà êàðòàòà íà ñàéòà:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Àäðåñ íà êàðòà íà ñàéòà, êîÿòî äà áúäå ïðî÷åòåíà âìåñòî ñîíäèðàíå íà ñàéòà; îñòàâåòå ïðàçíî, çà äà ñå ïðîâåðè robots.txt, ñëåä òîâà /sitemap.xml.

lang/Castellano.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Nombre del archivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida.
+Seed the crawl from the site's sitemap
+Iniciar el rastreo desde el mapa del sitio
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Leer el mapa del sitio (líneas Sitemap: de robots.txt, luego /sitemap.xml) y añadir como dirección inicial cada URL que incluya.
+Sitemap address:
+Dirección del mapa del sitio:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Dirección de un mapa del sitio que leer en lugar de sondear el sitio; déjelo vacío para sondear robots.txt y luego /sitemap.xml.

lang/Cesky.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Název archivu WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Volitelný základní název archivu WARC; ponechte prázdné pro automatické pojmenování ve výstupním adresáøi.
+Seed the crawl from the site's sitemap
+Zahájit procházení z mapy webu
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Naèíst mapu webu (øádky Sitemap: v souboru robots.txt, poté /sitemap.xml) a pøidat každou uvedenou adresu URL jako výchozí.
+Sitemap address:
+Adresa mapy webu:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa mapy webu, která se má naèíst místo zjiš<69>ování na webu; ponechte prázdné pro zjištìní z robots.txt a poté /sitemap.xml.

lang/Chinese-BIG5.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC 封存檔名稱：
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC 封存檔的選用基本名稱；留空則於輸出目錄中自動命名。
+Seed the crawl from the site's sitemap
+從網站的 Sitemap 開始擷取
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+讀取網站的 Sitemap（robots.txt 中的 Sitemap: 行，然後 /sitemap.xml），並將其中列出的每個網址加入為起始網址。
+Sitemap address:
+Sitemap 位址：
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+要讀取的 Sitemap 位址，用來取代自動探測；留空則先探測 robots.txt 再探測 /sitemap.xml。

lang/Chinese-Simplified.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC 归档名称：
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC 归档的可选基本名称；留空则在输出目录中自动命名。
+Seed the crawl from the site's sitemap
+从网站的 Sitemap 开始抓取
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+读取网站的 Sitemap（robots.txt 中的 Sitemap: 行，然后 /sitemap.xml），并将其中列出的每个网址添加为起始网址。
+Sitemap address:
+Sitemap 地址：
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+要读取的 Sitemap 地址，用来代替自动探测；留空则先探测 robots.txt 再探测 /sitemap.xml。

lang/Croatian.txt

@@ -966,3 +966,11 @@ WARC archive name:
 Naziv WARC arhive:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju.
+Seed the crawl from the site's sitemap
+Pokreni pretra¾ivanje iz karte web-mjesta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Proèitaj kartu web-mjesta (retke Sitemap: iz robots.txt, zatim /sitemap.xml) i dodaj svaki navedeni URL kao poèetnu adresu.
+Sitemap address:
+Adresa karte web-mjesta:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa karte web-mjesta koju treba proèitati umjesto ispitivanja web-mjesta; ostavite prazno za ispitivanje robots.txt pa /sitemap.xml.

lang/Dansk.txt

@@ -1012,3 +1012,11 @@ WARC archive name:
 Navn på WARC-arkiv:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen.
+Seed the crawl from the site's sitemap
+Start gennemgangen fra webstedets sitemap
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Læs webstedets sitemap (Sitemap:-linjer i robots.txt, derefter /sitemap.xml) og tilføj hver angivet URL som startadresse.
+Sitemap address:
+Sitemap-adresse:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adressen på et sitemap, der skal læses i stedet for at undersøge webstedet; lad feltet stå tomt for at undersøge robots.txt og derefter /sitemap.xml.

lang/Deutsch.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Name des WARC-Archivs:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen.
+Seed the crawl from the site's sitemap
+Erfassung mit der Sitemap der Website beginnen
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Die Sitemap der Website lesen (Sitemap:-Zeilen in robots.txt, dann /sitemap.xml) und jede dort aufgeführte URL als Startadresse hinzufügen.
+Sitemap address:
+Sitemap-Adresse:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresse einer Sitemap, die anstelle der Suche auf der Website gelesen wird; leer lassen, um robots.txt und dann /sitemap.xml zu prüfen.

lang/Eesti.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC-arhiivi nimi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC-arhiivi valikuline põhinimi; jäta tühjaks, et see väljundkataloogis automaatselt nimetada.
+Seed the crawl from the site's sitemap
+Alusta kogumist saidi saidikaardist
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Loe saidi saidikaarti (robots.txt-i Sitemap:-read, seejärel /sitemap.xml) ja lisa iga seal loetletud URL alguslingina.
+Sitemap address:
+Saidikaardi aadress:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Saidikaardi aadress, mida lugeda saidi sondeerimise asemel; jäta tühjaks, et kontrollida robots.txt-i ja seejärel /sitemap.xml-i.

lang/English.txt

@@ -1012,3 +1012,11 @@ WARC archive name:
 WARC archive name:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
+Seed the crawl from the site's sitemap
+Seed the crawl from the site's sitemap
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Sitemap address:
+Sitemap address:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/Finnish.txt

@@ -966,3 +966,11 @@ WARC archive name:
 WARC-arkiston nimi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valinnainen WARC-arkiston perusnimi; jätä tyhjäksi, jotta se nimetään automaattisesti tulostehakemistoon.
+Seed the crawl from the site's sitemap
+Aloita haku sivuston sivukartasta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Lue sivuston sivukartta (robots.txt-tiedoston Sitemap:-rivit, sitten /sitemap.xml) ja lisää jokainen siinä lueteltu URL-osoite aloitusosoitteeksi.
+Sitemap address:
+Sivukartan osoite:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Luettavan sivukartan osoite sivuston luotaamisen sijaan; jätä tyhjäksi, jolloin tarkistetaan robots.txt ja sitten /sitemap.xml.

lang/Francais.txt

@@ -1012,3 +1012,11 @@ WARC archive name:
 Nom de l'archive WARC :
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.
+Seed the crawl from the site's sitemap
+Partir du plan de site (sitemap)
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Lire le plan de site (lignes Sitemap: de robots.txt, puis /sitemap.xml) et ajouter chaque URL listée comme adresse de départ.
+Sitemap address:
+Adresse du plan de site :
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresse d'un plan de site à lire au lieu de sonder le site ; laissez vide pour sonder robots.txt puis /sitemap.xml.

lang/Greek.txt

@@ -966,3 +966,11 @@ WARC archive name:
 Όνομα αρχείου WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Προαιρετικό βασικό όνομα για το αρχείο WARC. Αφήστε το κενό για αυτόματη ονομασία στον κατάλογο εξόδου.
+Seed the crawl from the site's sitemap
+Έναρξη της ανίχνευσης από τον χάρτη του ιστότοπου
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ανάγνωση του χάρτη του ιστότοπου (γραμμές Sitemap: στο robots.txt, έπειτα /sitemap.xml) και προσθήκη κάθε διεύθυνσης URL που περιέχει ως αρχικής διεύθυνσης.
+Sitemap address:
+Διεύθυνση χάρτη ιστότοπου:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Διεύθυνση χάρτη ιστότοπου προς ανάγνωση αντί για αναζήτηση στον ιστότοπο. Αφήστε το κενό για έλεγχο του robots.txt και έπειτα του /sitemap.xml.

lang/Italiano.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Nome dell'archivio WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome di base facoltativo per l'archivio WARC; lascia vuoto per assegnarlo automaticamente nella directory di output.
+Seed the crawl from the site's sitemap
+Avvia la scansione dalla mappa del sito
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Legge la mappa del sito (righe Sitemap: in robots.txt, poi /sitemap.xml) e aggiunge come indirizzo iniziale ogni URL elencato.
+Sitemap address:
+Indirizzo della mappa del sito:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Indirizzo di una mappa del sito da leggere invece di sondare il sito; lasciare vuoto per sondare robots.txt e poi /sitemap.xml.

lang/Japanese.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC アーカイブ名:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC アーカイブの任意のベース名。空欄にすると出力ディレクトリ内で自動的に名前が付けられます。
+Seed the crawl from the site's sitemap
+サイトマップからミラーリングを開始する
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+サイトマップ (robots.txt の Sitemap: 行、次に /sitemap.xml) を読み込み、記載されているすべての URL を開始アドレスとして追加します。
+Sitemap address:
+サイトマップのアドレス:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+サイトを探索する代わりに読み込むサイトマップのアドレス。空欄にすると robots.txt、次に /sitemap.xml を探索します。

lang/Macedonian.txt

@@ -964,3 +964,11 @@ WARC archive name:
 ¸ÜÕ ÝÐ WARC ÐàåØÒÐâÐ:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 ¸×ÑÞàÝÞ ÞáÝÞÒÝÞ ØÜÕ ×Ð WARC ÐàåØÒÐâÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÐÒâÞÜÐâáÚÞ ØÜÕÝãÒÐúÕ ÒÞ Ø×ÛÕ×ÝØÞâ ÔØàÕÚâÞàØãÜ.
+Seed the crawl from the site's sitemap
+·ÐßÞçÝØ ÓÞ ßàÕÑÐàãÒÐúÕâÞ ÞÔ ÚÐàâÐâÐ ÝÐ áÐøâÞâ
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+¿àÞçØâÐø øÐ ÚÐàâÐâÐ ÝÐ áÐøâÞâ (àÕÔÞÒØâÕ Sitemap: ÒÞ robots.txt, ßÞâÞÐ /sitemap.xml) Ø ÔÞÔÐø øÐ áÕÚÞøÐ ÝÐÒÕÔÕÝÐ URL ÐÔàÕáÐ ÚÐÚÞ ßÞçÕâÝÐ.
+Sitemap address:
+°ÔàÕáÐ ÝÐ ÚÐàâÐâÐ ÝÐ áÐøâÞâ:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+°ÔàÕáÐ ÝÐ ÚÐàâÐ ÝÐ áÐøâÞâ èâÞ âàÕÑÐ ÔÐ áÕ ßàÞçØâÐ ÝÐÜÕáâÞ ØáßØâãÒÐúÕ ÝÐ áÐøâÞâ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÔÐ áÕ ØáßØâÐ robots.txt, ßÐ /sitemap.xml.

lang/Magyar.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC archívum neve:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 A WARC archívum opcionális alapneve; hagyja üresen az automatikus elnevezéshez a kimeneti könyvtárban.
+Seed the crawl from the site's sitemap
+A letöltés indítása a webhely webhelytérképérõl
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+A webhely webhelytérképének beolvasása (a robots.txt Sitemap: sorai, majd a /sitemap.xml), és a benne felsorolt összes URL felvétele kiindulási címként.
+Sitemap address:
+Webhelytérkép címe:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+A webhely vizsgálata helyett beolvasandó webhelytérkép címe; hagyja üresen a robots.txt, majd a /sitemap.xml vizsgálatához.

lang/Nederlands.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Naam van WARC-archief:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optionele basisnaam voor het WARC-archief; laat leeg om het automatisch een naam te geven in de uitvoermap.
+Seed the crawl from the site's sitemap
+De crawl starten vanaf de sitemap van de site
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+De sitemap van de site lezen (Sitemap:-regels in robots.txt, daarna /sitemap.xml) en elke vermelde URL als startadres toevoegen.
+Sitemap address:
+Sitemap-adres:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adres van een sitemap die gelezen moet worden in plaats van de site te onderzoeken; laat leeg om robots.txt en daarna /sitemap.xml te controleren.

lang/Norsk.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Navn på WARC-arkiv:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valgfritt basisnavn for WARC-arkivet; la feltet stå tomt for automatisk navngivning i utdatamappen.
+Seed the crawl from the site's sitemap
+Start gjennomgangen fra nettstedets nettstedskart
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Les nettstedets nettstedskart (Sitemap:-linjer i robots.txt, deretter /sitemap.xml) og legg til hver oppført URL som startadresse.
+Sitemap address:
+Adresse til nettstedskart:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adressen til et nettstedskart som skal leses i stedet for å undersøke nettstedet; la feltet stå tomt for å undersøke robots.txt og deretter /sitemap.xml.

lang/Polski.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Nazwa archiwum WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Opcjonalna nazwa bazowa archiwum WARC; pozostaw puste, aby nazwaæ je automatycznie w katalogu wyj¶ciowym.
+Seed the crawl from the site's sitemap
+Rozpocznij pobieranie od mapy witryny
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Odczytaj mapê witryny (wiersze Sitemap: w pliku robots.txt, nastêpnie /sitemap.xml) i dodaj ka¿dy wymieniony adres URL jako adres pocz±tkowy.
+Sitemap address:
+Adres mapy witryny:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adres mapy witryny do odczytania zamiast sondowania witryny; pozostaw puste, aby sprawdziæ robots.txt, a nastêpnie /sitemap.xml.

lang/Portugues-Brasil.txt

@@ -1012,3 +1012,11 @@ WARC archive name:
 Nome do arquivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome base opcional para o arquivo WARC; deixe em branco para nomeá-lo automaticamente no diretório de saída.
+Seed the crawl from the site's sitemap
+Iniciar a captura pelo mapa do site
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ler o mapa do site (linhas Sitemap: do robots.txt, depois /sitemap.xml) e adicionar como endereço inicial cada URL nele listada.
+Sitemap address:
+Endereço do mapa do site:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Endereço de um mapa do site a ser lido em vez de sondar o site; deixe em branco para sondar robots.txt e depois /sitemap.xml.

lang/Portugues.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Nome do arquivo WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nome base opcional para o arquivo WARC; deixe em branco para o nomear automaticamente no diretório de saída.
+Seed the crawl from the site's sitemap
+Iniciar a recolha pelo mapa do site
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ler o mapa do site (linhas Sitemap: do robots.txt, depois /sitemap.xml) e adicionar como endereço inicial cada URL nele listado.
+Sitemap address:
+Endereço do mapa do site:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Endereço de um mapa do site a ler em vez de sondar o site; deixe em branco para sondar robots.txt e depois /sitemap.xml.

lang/Romanian.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Numele arhivei WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nume de baza optional pentru arhiva WARC; lasati gol pentru a-l denumi automat in directorul de iesire.
+Seed the crawl from the site's sitemap
+Porneste explorarea de la harta sitului
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Citeste harta sitului (liniile Sitemap: din robots.txt, apoi /sitemap.xml) si adauga fiecare URL listat ca adresa de pornire.
+Sitemap address:
+Adresa hartii sitului:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa unei harti a sitului care sa fie citita în loc de sondarea sitului; lasati gol pentru a sonda robots.txt, apoi /sitemap.xml.

lang/Russian.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Èìÿ WARC-àðõèâà:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåîáÿçàòåëüíîå áàçîâîå èìÿ WARC-àðõèâà; îñòàâüòå ïóñòûì äëÿ àâòîìàòè÷åñêîãî èìåíîâàíèÿ â âûõîäíîì êàòàëîãå.
+Seed the crawl from the site's sitemap
+Íà÷èíàòü îáõîä ñ êàðòû ñàéòà
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ïðî÷èòàòü êàðòó ñàéòà (ñòðîêè Sitemap: â robots.txt, çàòåì /sitemap.xml) è äîáàâèòü êàæäûé óêàçàííûé â íåé URL êàê íà÷àëüíûé àäðåñ.
+Sitemap address:
+Àäðåñ êàðòû ñàéòà:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Àäðåñ êàðòû ñàéòà, êîòîðóþ íóæíî ïðî÷èòàòü âìåñòî îïðîñà ñàéòà; îñòàâüòå ïóñòûì, ÷òîáû ïðîâåðèòü robots.txt, çàòåì /sitemap.xml.

lang/Slovak.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Názov archívu WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Voliteµný základný názov archívu WARC; ponechajte prázdne pre automatické pomenovanie vo výstupnom adresári.
+Seed the crawl from the site's sitemap
+Zaèa» prehliadanie z mapy stránok
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Naèíta» mapu stránok (riadky Sitemap: v súbore robots.txt, potom /sitemap.xml) a prida» ka¾dú uvedenú adresu URL ako poèiatoènú.
+Sitemap address:
+Adresa mapy stránok:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adresa mapy stránok, ktorá sa má naèíta» namiesto zis»ovania na stránke; ponechajte prázdne na zistenie z robots.txt a potom /sitemap.xml.

lang/Slovenian.txt

@@ -964,3 +964,11 @@ WARC archive name:
 Ime arhiva WARC:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Neobvezno osnovno ime arhiva WARC; pustite prazno za samodejno poimenovanje v izhodni mapi.
+Seed the crawl from the site's sitemap
+Zacni zajem z zemljevidom spletnega mesta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Preberi zemljevid spletnega mesta (vrstice Sitemap: v robots.txt, nato /sitemap.xml) in dodaj vsak navedeni URL kot zacetni naslov.
+Sitemap address:
+Naslov zemljevida spletnega mesta:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Naslov zemljevida spletnega mesta, ki naj se prebere namesto preverjanja mesta; pustite prazno za preverjanje robots.txt in nato /sitemap.xml.

lang/Svenska.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC-arkivets namn:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Valfritt basnamn för WARC-arkivet; lämna tomt för att namnge det automatiskt i utdatakatalogen.
+Seed the crawl from the site's sitemap
+Starta insamlingen från webbplatsens webbplatskarta
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Läs webbplatsens webbplatskarta (Sitemap:-rader i robots.txt, sedan /sitemap.xml) och lägg till varje angiven URL som startadress.
+Sitemap address:
+Webbplatskartans adress:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Adress till en webbplatskarta som ska läsas i stället för att söka på webbplatsen; lämna tomt för att kontrollera robots.txt och sedan /sitemap.xml.

lang/Turkish.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC arþivi adý:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC arþivi için isteðe baðlý temel ad; çýktý dizininde otomatik adlandýrma için boþ býrakýn.
+Seed the crawl from the site's sitemap
+Taramayý sitenin site haritasýndan baþlat
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Sitenin site haritasýný oku (robots.txt içindeki Sitemap: satýrlarý, ardýndan /sitemap.xml) ve listelenen her URL'yi baþlangýç adresi olarak ekle.
+Sitemap address:
+Site haritasý adresi:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Siteyi yoklamak yerine okunacak site haritasýnýn adresi; robots.txt ve ardýndan /sitemap.xml yoklamasý için boþ býrakýn.

lang/Ukrainian.txt

@@ -964,3 +964,11 @@ WARC archive name:
 ²ì'ÿ WARC-àðõ³âó:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Íåîáîâ'ÿçêîâà áàçîâà íàçâà WARC-àðõ³âó; çàëèøòå ïîðîæí³ì äëÿ àâòîìàòè÷íîãî íàéìåíóâàííÿ ó âèõ³äíîìó êàòàëîç³.
+Seed the crawl from the site's sitemap
+Ïî÷èíàòè îáõ³ä ç êàðòè ñàéòó
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Ïðî÷èòàòè êàðòó ñàéòó (ðÿäêè Sitemap: ó robots.txt, ïîò³ì /sitemap.xml) ³ äîäàòè êîæíó âêàçàíó â í³é URL-àäðåñó ÿê ïî÷àòêîâó.
+Sitemap address:
+Àäðåñà êàðòè ñàéòó:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Àäðåñà êàðòè ñàéòó, ÿêó ñë³ä ïðî÷èòàòè çàì³ñòü îïèòóâàííÿ ñàéòó; çàëèøòå ïîðîæí³ì, ùîá ïåðåâ³ðèòè robots.txt, à ïîò³ì /sitemap.xml.

lang/Uzbek.txt

@@ -964,3 +964,11 @@ WARC archive name:
 WARC arxivi nomi:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 WARC arxivi uchun ixtiyoriy asosiy nom; chiqish katalogida avtomatik nomlash uchun bo'sh qoldiring.
+Seed the crawl from the site's sitemap
+Yig’ishni saytning sayt xaritasidan boshlash
+Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
+Saytning sayt xaritasini o’qish (robots.txt dagi Sitemap: qatorlari, so’ngra /sitemap.xml) va unda ko’rsatilgan har bir URL manzilni boshlang’ich manzil sifatida qo’shish.
+Sitemap address:
+Sayt xaritasi manzili:
+Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
+Saytni tekshirish o’rniga o’qiladigan sayt xaritasi manzili; robots.txt, so’ngra /sitemap.xml ni tekshirish uchun bo’sh qoldiring.

libtest/callbacks-example-filenameiisbug.c

@@ -71,9 +71,7 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
       for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
       if (iisBogus[i][j] == '\0'
           && (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
-        /* j bytes matched, so j fit: copying j cannot overrun whatever the
-           table holds, and the tail must survive untouched */
-        memcpy(a, iisBogusReplace[i], (size_t) j);
+        strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
         break;
       }
     }

man/httrack.1

@@ -3,7 +3,7 @@
 .\"
 .\" This file is generated by man/makeman.sh; do not edit by hand.
 .\" SPDX-License-Identifier: GPL-3.0-or-later
-.TH httrack 1 "23 July 2026" "httrack website copier"
+.TH httrack 1 "26 July 2026" "httrack website copier"
 .SH NAME
 httrack \- offline browser : copy websites to a local directory
 .SH SYNOPSIS
@@ -36,6 +36,7 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-t, \-\-test\fR ]
 [ \fB\-%L, \-\-list\fR ]
 [ \fB\-%S, \-\-urllist\fR ]
+[ \fB\-%m, \-\-sitemap\fR ]
 [ \fB\-NN, \-\-structure[=N]\fR ]
 [ \fB\-%N, \-\-delayed\-type\-check\fR ]
 [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
@@ -187,6 +188,8 @@ test all URLs (even forbidden ones) (\-\-test)
 <file> add all URL located in this text file (one URL per line) (\-\-list <param>)
 .IP \-%S
 <file> add all scan rules located in this text file (one scan rule per line) (\-\-urllist <param>)
+.IP \-%m
+seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (\-\-sitemap)
 .SS Build options:
 .IP \-NN
 structure type (0 *original structure, 1+: see below) (\-\-structure[=N])

src/Makefile.am

@@ -66,7 +66,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
@@ -77,7 +77,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htssitemap.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \

src/htsalias.c

@@ -114,6 +114,10 @@ const char *hts_optalias[][4] = {
    "strip [host/pattern=]key1,key2,... from URLs"},
   {"cookies-file", "-%K", "param1",
    "load extra cookies from a Netscape cookies.txt"},
+  {"sitemap", "-%m", "single",
+   "seed the crawl from the start host's sitemap (robots.txt, then "
+   "/sitemap.xml)"},
+  {"sitemap-url", "-%mu", "param1", "seed the crawl from this sitemap URL"},
   {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
   {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
   {"warc-max-size", "-%rs", "param1",

src/htscache.c

@@ -142,12 +142,10 @@ struct cache_back_zip_entry {
   int compressionMethod;
 };
 
-/* A corrupt cache can carry a field wider than ours; clipping it keeps the
-   entry, where aborting would take the crawl down. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (void) strclipbuff(refvalue, refvalue_size, value);                      \
+      strlcpybuff(refvalue, value, refvalue_size);                             \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)

src/htscache_selftest.c

@@ -1195,12 +1195,6 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
 
 /* --- read-side corruption injection --------------------------------------- */
 
-/* 100 'A's: a placeholder header line long enough to be overwritten by a
-   forged, over-long X-StatusMessage of the same byte length. */
-#define CORRUPT_LONG_ETAG                                                      \
-  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"                         \
-  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-
 /* canary read back intact after each corruption; victim gets the byte surgery
  */
 #define CORRUPT_ADR "corrupt.example.com"
@@ -1225,24 +1219,6 @@ static void corrupt_build(httrackp *opt) {
   selftest_close(&cache);
 }
 
-/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
-static void corrupt_build_longetag(httrackp *opt) {
-  cache_back cache;
-
-  memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
-  memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
-  remove(reconcile_st_path(opt, "hts-cache/new.zip"));
-  remove(reconcile_st_path(opt, "hts-cache/old.zip"));
-  selftest_open_for_write(&cache, opt);
-  store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
-              "OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
-              strlen(corrupt_body_a));
-  store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
-              "OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
-              corrupt_body_b, strlen(corrupt_body_b));
-  selftest_close(&cache);
-}
-
 /* Like corrupt_build, but the victim carries a 20-char Etag whose header line
    is later overwritten with a forged oversized X-Size (same byte length). */
 static void corrupt_build_etag(httrackp *opt) {
@@ -1427,48 +1403,6 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
   return fail;
 }
 
-/* An over-long field from a foreign cache must clip, not abort: the entry
-   still reads, clipped to capacity, and the canary survives. */
-static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
-                                         size_t wantlastmod, const char *what) {
-  cache_back cache;
-  htsblk v, c;
-  char BIGSTK lv[HTS_URLMAXSIZE * 2];
-  char BIGSTK lc[HTS_URLMAXSIZE * 2];
-  int fail = 0;
-
-  selftest_open_for_read(&cache, opt);
-  lv[0] = lc[0] = '\0';
-  v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
-  if (v.statuscode != 200) {
-    fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
-            v.statuscode);
-    fail++;
-  }
-  if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
-    fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
-            (unsigned) strlen(v.msg), (unsigned) wantmsg);
-    fail++;
-  }
-  if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
-    fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
-            what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
-    fail++;
-  }
-  c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
-  if (c.statuscode != 200) {
-    fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
-            c.statuscode);
-    fail++;
-  }
-  if (v.adr != NULL)
-    freet(v.adr);
-  if (c.adr != NULL)
-    freet(c.adr);
-  selftest_close(&cache);
-  return fail;
-}
-
 /* One zip corruption case: build, patch, then check victim+canary in-session.
  */
 static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1505,31 +1439,6 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
   failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
                                     "garbled deflate stream");
 
-  /* A corrupt cache can hold a field wider than ours. Clipping keeps the
-     entry; aborting would take the crawl down. Overwrite the placeholder Etag
-     line in place, same byte length, so the zip offsets stay intact. */
-  corrupt_build_longetag(opt);
-  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
-                "X-StatusMessage: " /* 17 + 89 = 106 */
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
-                1, 1);
-  failures +=
-      corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
-                                    (size_t) -1, "over-long X-StatusMessage");
-
-  /* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
-     cannot satisfy both. */
-  corrupt_build_longetag(opt);
-  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
-                "Last-Modified: " /* 15 + 91 = 106 */
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
-                1, 1);
-  failures += corrupt_expect_victim_clipped(
-      opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
-      "over-long Last-Modified");
-
   /* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
      truncates negative in the (int) cast the malloc uses: a wraparound alloc.
      cache_add asserts size fits an int, so such a value only reaches the reader

src/htscore.c

@@ -39,6 +39,7 @@ Please visit our Website: http://www.httrack.com
 
 /* File defs */
 #include "htscore.h"
+#include "htssitemap.h"
 #include "htswarc.h"
 
 /* specific definitions */
@@ -943,6 +944,22 @@ int httpmirror(char *url1, httrackp * opt) {
     heap_top()->premier = heap_top_index();        // premier lien, objet-père=objet              
     heap_top()->precedent = heap_top_index();      // lien précédent
 
+    /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are
+       injected before the crawl gets far. */
+    hts_sitemap_free(opt); /* an earlier mirror may have left a doc list */
+    if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) {
+      char BIGSTK first[HTS_URLMAXSIZE * 2];
+      const char *const eol = strchr(primary, '\n');
+      const size_t len = eol != NULL ? (size_t) (eol - primary) : 0;
+
+      first[0] = '\0';
+      if (len > 0 && len < sizeof(first)) {
+        memcpy(first, primary, len);
+        first[len] = '\0';
+      }
+      hts_sitemap_seed(opt, first);
+    }
+
     // Initialiser cache
     {
       opt->state._hts_in_html_parsing = 4;
@@ -1587,11 +1604,21 @@ int httpmirror(char *url1, httrackp * opt) {
         stre.maketrack_fp = maketrack_fp;
 
         /* Parse */
-        if (hts_mirror_check_moved(&str, &stre) != 0) {
-          XH_uninit;
-          return -1;
-        }
+        {
+          const int nlinks = opt->lien_tot;
 
+          if (hts_mirror_check_moved(&str, &stre) != 0) {
+            XH_uninit;
+            return -1;
+          }
+          /* A redirect re-queues the target as a fresh link; without carrying
+             the marking over, a moved sitemap is fetched and then ignored. */
+          if (opt->state.sitemap != NULL && opt->lien_tot > nlinks &&
+              hts_sitemap_pending(opt, urladr(), urlfil())) {
+            hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr,
+                                 heap_top()->fil);
+          }
+        }
       }
 
     }                           // if !error
@@ -1609,6 +1636,29 @@ int httpmirror(char *url1, httrackp * opt) {
       /* Load file and decode if necessary, after redirect check. */
       LOAD_IN_MEMORY_IF_NECESSARY();
 
+      /* Sitemap document: turn its <loc> URLs into top-level seeds. They go
+         through htsAddLink, so the wizard's filters and scope rules decide, and
+         this link's max depth leaves them the full budget. */
+      if (opt->state.sitemap != NULL &&
+          hts_sitemap_pending(opt, urladr(), urlfil())) {
+        htsmoduleStruct BIGSTK smstr;
+        int smptr = ptr;
+
+        memset(&smstr, 0, sizeof(smstr));
+        smstr.opt = opt;
+        smstr.sback = sback;
+        smstr.cache = &cache;
+        smstr.hashptr = hashptr;
+        smstr.numero_passe = numero_passe;
+        smstr.ptr_ = &smptr; /* scratch: the ingester retargets the wizard */
+        smstr.addLink = htsAddLink;
+        smstr.url_host = urladr();
+        smstr.url_file = urlfil();
+        smstr.mime = r.contenttype;
+        hts_sitemap_ingest(opt, &smstr, urladr(), urlfil(), r.adr,
+                           r.adr != NULL && r.size > 0 ? (size_t) r.size : 0);
+      }
+
       // ------------------------------------------------------
       // ok, fichier chargé localement
       // ------------------------------------------------------
@@ -1819,6 +1869,9 @@ int httpmirror(char *url1, httrackp * opt) {
 
         if (strnotempty(savename()) == 0) {       // pas de chemin de sauvegarde
           if (strcmp(urlfil(), "/robots.txt") == 0) {     // robots.txt
+            char BIGSTK sitemaps[8192];
+
+            sitemaps[0] = '\0';
             if (r.adr) {
               char BIGSTK infobuff[8192];
 #ifdef IGNORE_RESTRICTIVE_ROBOTS
@@ -1830,7 +1883,8 @@ int httpmirror(char *url1, httrackp * opt) {
 #endif
 
               robots_parse(&robots, urladr(), r.adr, r.size, infobuff,
-                           sizeof(infobuff), keep_root);
+                           sizeof(infobuff), keep_root, sitemaps,
+                           sizeof(sitemaps));
               if (strnotempty(infobuff)) {
                 hts_log_print(opt, LOG_INFO,
                               "Note: robots.txt forbidden links for %s are: %s",
@@ -1840,6 +1894,10 @@ int httpmirror(char *url1, httrackp * opt) {
                               urladr(), infobuff);
               }
             }
+            /* After robots_parse, so the rules this very body carries already
+               gate the sitemap fetch. Runs even on a failed probe, which is
+               what falls back to the well-known location. */
+            hts_sitemap_robots(opt, urladr(), sitemaps);
           }
         } else if (r.is_write) {        // déja sauvé sur disque
           /*
@@ -2256,6 +2314,7 @@ int httpmirror(char *url1, httrackp * opt) {
   // ending
   usercommand(opt, 0, NULL, NULL, NULL, NULL);
   warc_close_opt(opt);
+  hts_sitemap_free(opt);
 
   // désallocation mémoire & buffers
   XH_uninit;
@@ -3643,6 +3702,11 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
   to->warc_cdx = from->warc_cdx;
   to->warc_wacz = from->warc_wacz;
 
+  if (from->sitemap)
+    to->sitemap = from->sitemap;
+  if (StringNotEmpty(from->sitemap_url))
+    StringCopyS(to->sitemap_url, from->sitemap_url);
+
   if (from->pause_max_ms > 0) {
     to->pause_min_ms = from->pause_min_ms;
     to->pause_max_ms = from->pause_max_ms;

src/htscoremain.c

@@ -1795,6 +1795,26 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->warc_file, WARC_AUTONAME);
                 }
                 break;
+              case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps
+                if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap
+                  com++;
+                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
+                    HTS_PANIC_PRINTF(
+                        "Option sitemap-url needs a blank space and a URL");
+                    htsmain_free();
+                    return -1;
+                  }
+                  na++;
+                  if (strlen(argv[na]) >= HTS_URLMAXSIZE) {
+                    HTS_PANIC_PRINTF("Sitemap URL too long");
+                    htsmain_free();
+                    return -1;
+                  }
+                  StringCopy(opt->sitemap_url, argv[na]);
+                } else { // --sitemap: robots.txt probe, then /sitemap.xml
+                  opt->sitemap = HTS_TRUE;
+                }
+                break;
               case 'Y': // why: explain the filter verdict for a URL, no crawl
                 if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                   HTS_PANIC_PRINTF("Option why needs a blank space and a URL");

src/htshelp.c

@@ -433,8 +433,7 @@ void help_catchurl(const char *dest_path) {
       }
       // former URL!
       {
-        /* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
-        char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
+        char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
 
         inplace_escape_check_url(dest, sizeof(dest));
         snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);
@@ -526,6 +525,11 @@ void help(const char *app, int more) {
     (" %L <file> add all URL located in this text file (one URL per line)");
   infomsg
     (" %S <file> add all scan rules located in this text file (one scan rule per line)");
+  infomsg(" %m  seed the crawl from the site's sitemap (robots.txt Sitemap:, "
+          "then /sitemap.xml); --sitemap-url URL names one explicitly. A "
+          "sitemap you name, or one the site declares, is fetched even under "
+          "robots.txt Disallow; only the guessed /sitemap.xml obeys it. The "
+          "URLs found still pass every filter and scope rule");
   infomsg("");
   infomsg("Build options:");
   infomsg("  NN structure type (0 *original structure, 1+: see below)");

src/htslib.c

@@ -36,6 +36,7 @@ Please visit our Website: http://www.httrack.com
 // Fichier librairie .c
 
 #include "htscore.h"
+#include "htssitemap.h"
 #include "htswarc.h"
 
 /* specific definitions */
@@ -2537,6 +2538,15 @@ void fil_simplifie(char *f) {
   }
 }
 
+void htsblk_failf(htsblk *r, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  // deliberate clip: the reason is quoted from a remote reply
+  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
+  va_end(args);
+}
+
 // fermer liaison fichier ou socket
 void deletehttp(htsblk * r) {
 #if HTS_DEBUG_CLOSESOCK
@@ -6010,6 +6020,7 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   StringCopy(opt->strip_query, "");
   StringCopy(opt->cookies_file, "");
   StringCopy(opt->warc_file, "");
+  StringCopy(opt->sitemap_url, "");
   opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
@@ -6163,6 +6174,8 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->cookies_file);
     StringFree(opt->why_url);
     StringFree(opt->warc_file);
+    StringFree(opt->sitemap_url);
+    hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */
 
     StringFree(opt->path_html);
     StringFree(opt->path_html_utf8);

src/htslib.h

@@ -201,8 +201,7 @@ T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
                    int waitconnect, int addr_index, int *addr_count);
 /* Clips the formatted failure reason into r->msg, which also round-trips
    through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
-#define htsblk_failf(R, ...)                                                   \
-  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
+void htsblk_failf(htsblk *r, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
 
 HTS_INLINE void deletehttp(htsblk * r);
 HTS_INLINE int deleteaddr(htsblk * r);

src/htsopt.h

@@ -257,6 +257,7 @@ struct htsoptstate {
   unsigned int tmpnameid; /**< counter for temporary file names */
   int is_ended;           /**< mirror has finished */
   void *warc;             /**< open WARC writer (warc_writer*), or NULL */
+  void *sitemap;          /**< sitemap ingestion state, or NULL */
 };
 
 /* Library handles */
@@ -547,6 +548,9 @@ struct httrackp {
                              archive. Tail: ABI */
   hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip
                               (implies --warc + --warc-cdx). Tail: ABI */
+  hts_boolean sitemap;   /**< --sitemap: probe the start host's robots.txt for
+                              Sitemap: lines, else /sitemap.xml. Tail: ABI */
+  String sitemap_url;    /**< --sitemap-url: sitemap to ingest. Tail: ABI */
 };
 
 /* Running statistics for a mirror. */

src/htsrobots.c

@@ -147,7 +147,8 @@ static void robots_blob_add(char *blob, size_t blobsize, char marker,
 
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow) {
+                  hts_boolean keep_root_disallow, char *sitemaps,
+                  size_t sitemapsize) {
   size_t bptr = 0;
   int record = 0;
   char BIGSTK line[1024];
@@ -156,6 +157,8 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
   blob[0] = '\0';
   if (info != NULL && infosize > 0)
     info[0] = '\0';
+  if (sitemaps != NULL && sitemapsize > 0)
+    sitemaps[0] = '\0';
 #if DEBUG_ROBOTS
   printf("robots.txt dump:\n%s\n", body);
 #endif
@@ -172,7 +175,19 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
       line[llen - 1] = '\0';
       llen--;
     }
-    if (strfield(line, "user-agent:")) {
+    if (sitemaps != NULL && strfield(line, "sitemap:")) {
+      // group-independent record (RFC 9309): collected whatever the group
+      char *a = line + 8;
+
+      while (is_realspace(*a))
+        a++;
+      /* A line at the buffer limit was truncated: a half URL is not one. */
+      if (strnotempty(a) && strlen(line) < sizeof(line) - 3 &&
+          strlen(a) + 2 < sitemapsize - strlen(sitemaps)) {
+        strlcatbuff(sitemaps, a, sitemapsize);
+        strlcatbuff(sitemaps, "\n", sitemapsize);
+      }
+    } else if (strfield(line, "user-agent:")) {
       char *a = line + 11;
 
       while (is_realspace(*a))

src/htsrobots.h

@@ -56,10 +56,12 @@ int checkrobots(robots_wizard * robots, const char *adr, const char *fil);
 void checkrobots_free(robots_wizard * robots);
 int checkrobots_set(robots_wizard * robots, const char *adr, const char *data);
 /* Parse robots.txt `body` for `adr`, storing the HTTrack group's rules; `info`
-   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /". */
+   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /", and
+   `sitemaps` (optional) collects the Sitemap: URLs, one per line. */
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow);
+                  hts_boolean keep_root_disallow, char *sitemaps,
+                  size_t sitemapsize);
 #endif
 
 #endif

src/htssafe.h

@@ -460,25 +460,6 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
   return b->buf;
 }
 
-/**
- * Copy src into dest (capacity size, NUL included), truncating to fit and
- * always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
- * value read back from a cache, a header or the wire, where refusing the whole
- * record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
- * callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
- */
-static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
-                                                     const char *src) {
-  size_t len, copy;
-
-  assertf(dest != NULL && src != NULL && size != 0);
-  len = strlen(src);
-  copy = len < size ? len : size - 1;
-  memcpy(dest, src, copy);
-  dest[copy] = '\0';
-  return copy == len ? HTS_TRUE : HTS_FALSE;
-}
-
 /**
  * Callers that deliberately ignore truncation use this instead of
  * slprintfbuff(), so it is not HTS_CHECK_RESULT.
@@ -488,9 +469,6 @@ static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
   int ret;
 
   assertf(dest != NULL && size != 0);
-  /* a vsnprintf failing outright may write nothing at all, leaving whatever
-     the caller had on the stack for it to publish */
-  dest[0] = '\0';
   ret = vsnprintf(dest, size, fmt, args);
   /* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
   dest[size - 1] = '\0';
@@ -514,20 +492,6 @@ static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
   return ret;
 }
 
-/**
- * slprintfbuff() for diagnostics quoting remote or client text, which are
- * meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
- * cast on slprintfbuff() is no substitute, GCC warns through it.
- */
-static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
-    char *dest, size_t size, const char *fmt, ...) {
-  va_list args;
-
-  va_start(args, fmt);
-  (void) vslprintfbuff(dest, size, fmt, args);
-  va_end(args);
-}
-
 /**
  * slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
  * On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.

src/htsselftest.c

@@ -59,6 +59,7 @@ Please visit our Website: http://www.httrack.com
 #include "htssniff.h"
 #include "htscodec.h"
 #include "htsproxy.h"
+#include "htssitemap.h"
 #include "htswarc.h"
 #if HTS_USEZLIB
 #include "htszlib.h"
@@ -457,12 +458,9 @@ static void basic_selftests(void) {
     // link one level up -> a "../" prefix
     assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
     assertf(strcmp(s, "../a.html") == 0);
-    // an empty current path: the trim used to walk off the front of it, which
-    // "?x" reaches too because the query pre-pass hands on the part before it
+    // empty current path: the trim used to walk back off the front of it
     assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
     assertf(strcmp(s, "dir/page.html") == 0);
-    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
-    assertf(strcmp(s, "dir/page.html") == 0);
   }
 }
 
@@ -624,66 +622,6 @@ static int string_safety_selftests(void) {
       return 1;
   }
 
-  /* strclipbuff: truncate-and-report, never abort. Same canary shape; the
-     destination is poisoned before every call so a case cannot pass on the
-     previous one's bytes. */
-  {
-    struct {
-      char dst[8];
-      char canary[8];
-    } s;
-
-    memset(&s, '#', sizeof(s));
-#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
-
-    /* well under capacity: no padding, nothing eaten off the end */
-    POISON_DST();
-    if (!strclipbuff(s.dst, sizeof(s.dst), "abc") || strcmp(s.dst, "abc") != 0)
-      return 1;
-
-    /* exact fit: capacity - 1 characters plus the NUL */
-    POISON_DST();
-    if (!strclipbuff(s.dst, sizeof(s.dst), "1234567") ||
-        strcmp(s.dst, "1234567") != 0)
-      return 1;
-
-    /* one over, then far over: clipped, terminated, reported. The expected
-       bytes differ from the case above, so a write-nothing implementation
-       cannot pass on the leftovers. */
-    POISON_DST();
-    if (strclipbuff(s.dst, sizeof(s.dst), "abcdefgh") ||
-        strcmp(s.dst, "abcdefg") != 0)
-      return 1;
-    POISON_DST();
-    if (strclipbuff(s.dst, sizeof(s.dst), "0123456789abcdef") ||
-        strcmp(s.dst, "0123456") != 0)
-      return 1;
-
-    /* degenerate capacity 1: only the NUL fits. Capacity 2 pins the boundary
-       between that and the sizes above: one character plus the NUL. */
-    POISON_DST();
-    if (strclipbuff(s.dst, 1, "x") || s.dst[0] != '\0')
-      return 1;
-    POISON_DST();
-    if (strclipbuff(s.dst, 2, "yz") || strcmp(s.dst, "y") != 0)
-      return 1;
-
-    /* the empty string fits any non-zero capacity */
-    POISON_DST();
-    if (!strclipbuff(s.dst, sizeof(s.dst), "") || s.dst[0] != '\0')
-      return 1;
-
-    /* a byte over 0x7f must not end the copy early */
-    POISON_DST();
-    if (!strclipbuff(s.dst, sizeof(s.dst), "\xff\xfe") ||
-        strcmp(s.dst, "\xff\xfe") != 0)
-      return 1;
-#undef POISON_DST
-
-    if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
-      return 1;
-  }
-
   /* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
      touches nothing else in the block */
   {
@@ -1585,7 +1523,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
       size_t i;
       for (i = bench[loop].offset; i < (size_t) count;
            i += bench[loop].modulus) {
-        int result = 0; /* no final else: an unknown type reports failure */
+        int result;
         FMT();
         if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
           size_t k;
@@ -1740,6 +1678,22 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
   if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0)
     err = 1;
 
+  /* sitemap pair: the flag latches on, the URL takes the String deep copy */
+  from->sitemap = HTS_TRUE;
+  StringCopy(from->sitemap_url, "http://h.test/sitemap.xml");
+  to->sitemap = HTS_FALSE;
+  StringCopy(to->sitemap_url, "");
+  copy_htsopt(from, to);
+  if (!to->sitemap ||
+      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
+    err = 1;
+  from->sitemap = HTS_FALSE;
+  StringCopy(from->sitemap_url, "");
+  copy_htsopt(from, to);
+  if (!to->sitemap ||
+      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
+    err = 1;
+
   /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
   from->pause_min_ms = 5000;
   from->pause_max_ms = 10000;
@@ -3664,7 +3618,7 @@ static int rb_decide(robots_wizard *r, const char *txt, const char *path) {
   char host[64];
 
   snprintf(host, sizeof(host), "h%d.example", n++);
-  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE);
+  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE, NULL, 0);
   return checkrobots(r, host, path);
 }
 
@@ -3741,6 +3695,267 @@ static int st_robots(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* Collect the URLs a sitemap scan hands out. */
+typedef struct sm_collect {
+  int n;
+  char url[8][HTS_URLMAXSIZE];
+} sm_collect;
+
+static hts_boolean sm_take(void *arg, const char *url) {
+  sm_collect *const c = (sm_collect *) arg;
+
+  if (c->n < (int) (sizeof(c->url) / sizeof(c->url[0])))
+    strcpybuff(c->url[c->n], url);
+  c->n++;
+  return HTS_TRUE;
+}
+
+/* Scan `doc` off a heap buffer with no NUL terminator, so a read past the
+   declared size is an ASan error rather than a silent pass. */
+static int sm_scan(const char *doc, int maxurls, hts_boolean *is_index,
+                   sm_collect *out) {
+  const size_t len = strlen(doc);
+  char *raw = malloct(len);
+  int n;
+
+  memset(out, 0, sizeof(*out));
+  assertf(raw != NULL);
+  memcpy(raw, doc, len);
+  n = hts_sitemap_scan(raw, len, maxurls, is_index, sm_take, out);
+  freet(raw);
+  return n;
+}
+
+static int st_sitemap(httrackp *opt, int argc, char **argv) {
+  sm_collect c;
+  hts_boolean idx;
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+  /* A urlset yields its <loc> URLs, in order, unescaped. */
+  assertf(sm_scan("<?xml version=\"1.0\"?><urlset>"
+                  "<url><loc>http://h.test/a.html</loc></url>"
+                  "<url><loc>  https://h.test/b?x=1&amp;y=2\n  </loc></url>"
+                  "</urlset>",
+                  100, &idx, &c) == 2);
+  assertf(!idx);
+  assertf(strcmp(c.url[0], "http://h.test/a.html") == 0);
+  assertf(strcmp(c.url[1], "https://h.test/b?x=1&y=2") == 0);
+
+  /* A sitemapindex is flagged: its URLs are child sitemaps, not pages. */
+  assertf(sm_scan("<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc>"
+                  "</sitemap></sitemapindex>",
+                  100, &idx, &c) == 1);
+  assertf(idx);
+
+  /* Root element decides even when the other name appears later as text. */
+  assertf(sm_scan("<urlset><url><loc>http://h.test/a</loc></url>"
+                  "<!-- sitemapindex --></urlset>",
+                  100, &idx, &c) == 1);
+  assertf(!idx);
+
+  /* Numeric character references, decimal and hex, decode to ASCII. */
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#63;b&#x3D;c</loc></urlset>",
+                  100, &idx, &c) == 1);
+  assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0);
+
+  /* A reference decoding to a control byte is dropped: the shared decoder
+     writes the real character and the URL check refuses it. A reference the
+     decoder cannot represent (&#0;) stays verbatim, like an unknown entity. */
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#10;b</loc></urlset>", 100,
+                  &idx, &c) == 0);
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#9;b</loc></urlset>", 100, &idx,
+                  &c) == 0);
+  assertf(sm_scan("<urlset><loc>http://h.test/a&#0;b</loc></urlset>", 100, &idx,
+                  &c) == 1);
+  assertf(strcmp(c.url[0], "http://h.test/a&#0;b") == 0);
+
+  /* A comment naming the other root element must not flip the verdict. */
+  assertf(sm_scan("<!-- <sitemapindex> --><urlset><url>"
+                  "<loc>http://h.test/p</loc></url></urlset>",
+                  100, &idx, &c) == 1);
+  assertf(!idx);
+  assertf(sm_scan("<?xml version=\"1.0\"?><!-- <urlset> -->"
+                  "<sitemapindex><loc>http://h.test/s</loc></sitemapindex>",
+                  100, &idx, &c) == 1);
+  assertf(idx);
+
+  /* <location> is not <loc>. */
+  assertf(sm_scan("<urlset><location>http://h.test/a</location></urlset>", 100,
+                  &idx, &c) == 0);
+
+  /* Rejected: relative, non-http scheme, embedded space, empty. */
+  assertf(sm_scan("<urlset><loc>/a.html</loc><loc>ftp://h.test/a</loc>"
+                  "<loc>javascript:alert(1)</loc>"
+                  "<loc>http://h.test/a b</loc><loc></loc></urlset>",
+                  100, &idx, &c) == 0);
+
+  /* The URL length bound: one under fits, exactly at it is dropped rather than
+     truncated into a different URL. */
+  {
+    char BIGSTK doc[HTS_URLMAXSIZE * 2];
+    char BIGSTK url[HTS_URLMAXSIZE + 1];
+    size_t i;
+
+    strcpybuff(url, "http://h.test/");
+    for (i = strlen(url); i < HTS_URLMAXSIZE - 1; i++)
+      url[i] = 'a';
+    url[i] = '\0';
+    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
+    assertf(sm_scan(doc, 100, &idx, &c) == 1);
+
+    url[i] = 'a';
+    url[i + 1] = '\0';
+    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
+    assertf(sm_scan(doc, 100, &idx, &c) == 0);
+  }
+
+  /* The URL cap stops the scan. */
+  assertf(sm_scan("<urlset><loc>http://h.test/1</loc><loc>http://h.test/2</loc>"
+                  "<loc>http://h.test/3</loc></urlset>",
+                  2, &idx, &c) == 2);
+
+  /* The per-document cap at the value the engine actually uses. */
+  {
+    const int many = HTS_SITEMAP_MAX_URLS_DOC + 10;
+    const size_t cap = (size_t) many * 40 + 32;
+    char *big = malloct(cap);
+    size_t off;
+    int i;
+
+    assertf(big != NULL);
+    off = (size_t) snprintf(big, cap, "<urlset>");
+    assertf(off < cap);
+    for (i = 0; i < many; i++) {
+      const int len =
+          snprintf(big + off, cap - off, "<loc>http://h.test/%d</loc>", i);
+
+      assertf(len > 0 && (size_t) len < cap - off);
+      off += (size_t) len;
+    }
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(big, off, HTS_SITEMAP_MAX_URLS_DOC, &idx, sm_take,
+                             &c) == HTS_SITEMAP_MAX_URLS_DOC);
+    /* The handler count, not just the return: a call site hardcoding a smaller
+       cap would still return its own argument. */
+    assertf(c.n == HTS_SITEMAP_MAX_URLS_DOC);
+    freet(big);
+  }
+
+#if HTS_USEZLIB
+  /* A highly compressible document decodes without running away: the ratio
+     budget cannot bind (deflate tops out near 1032:1), so this pins the
+     decompression path itself rather than the 64 MiB ceiling. */
+  {
+    const char *const one = "<url><loc>http://h.test/bomb</loc></url>";
+    const size_t reps = 40000;
+    size_t xlen = 8 + reps * strlen(one) + 10, i;
+    char *x = malloct(xlen + 1);
+    uLongf zlen;
+    char *z;
+    z_stream zs;
+
+    assertf(x != NULL);
+    {
+      size_t w = (size_t) snprintf(x, xlen, "<urlset>");
+      int len;
+
+      assertf(w < xlen);
+      for (i = 0; i < reps; i++) {
+        len = snprintf(x + w, xlen - w, "%s", one);
+        assertf(len > 0 && (size_t) len < xlen - w);
+        w += (size_t) len;
+      }
+      len = snprintf(x + w, xlen - w, "</urlset>");
+      assertf(len > 0 && (size_t) len < xlen - w);
+      w += (size_t) len;
+      xlen = w;
+    }
+    zlen = compressBound((uLong) xlen) + 32;
+    z = malloct((size_t) zlen);
+    assertf(z != NULL);
+    memset(&zs, 0, sizeof(zs));
+    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
+                         Z_DEFAULT_STRATEGY) == Z_OK);
+    zs.next_in = (const Bytef *) x;
+    zs.avail_in = (uInt) xlen;
+    zs.next_out = (Bytef *) z;
+    zs.avail_out = (uInt) zlen;
+    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
+    zlen = (uLongf) zs.total_out;
+    deflateEnd(&zs);
+    /* well over the 4096:1 budget's 1 MiB floor, and far under the 64 MiB cap
+     */
+    assertf(xlen > 1024 * 1024 && (size_t) zlen < xlen / 100);
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(z, (size_t) zlen, 10, &idx, sm_take, &c) == 10);
+    assertf(strcmp(c.url[0], "http://h.test/bomb") == 0);
+    freet(z);
+    freet(x);
+  }
+#endif
+
+  /* An unterminated <loc> at end of buffer must not read past it. */
+  assertf(sm_scan("<urlset><loc>http://h.test/a", 100, &idx, &c) == 0);
+  assertf(sm_scan("<urlset><lo", 100, &idx, &c) == 0);
+
+#if HTS_USEZLIB
+  /* A gzip-framed document is decompressed before scanning. */
+  {
+    const char *const xml =
+        "<urlset><url><loc>http://h.test/gz.html</loc></url></urlset>";
+    uLongf zlen = compressBound((uLong) strlen(xml)) + 32;
+    char *z = malloct((size_t) zlen);
+    z_stream zs;
+
+    assertf(z != NULL);
+    memset(&zs, 0, sizeof(zs));
+    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
+                         Z_DEFAULT_STRATEGY) == Z_OK);
+    zs.next_in = (const Bytef *) xml;
+    zs.avail_in = (uInt) strlen(xml);
+    zs.next_out = (Bytef *) z;
+    zs.avail_out = (uInt) zlen;
+    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
+    zlen = (uLongf) zs.total_out;
+    deflateEnd(&zs);
+
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(z, (size_t) zlen, 100, &idx, sm_take, &c) == 1);
+    assertf(strcmp(c.url[0], "http://h.test/gz.html") == 0);
+
+    /* Truncated gzip: refused, not scanned as plain text. */
+    memset(&c, 0, sizeof(c));
+    assertf(hts_sitemap_scan(z, 4, 100, &idx, sm_take, &c) == -1);
+    freet(z);
+  }
+#endif
+
+  /* robots.txt: only Sitemap: records, comments stripped, case-insensitive,
+     and group-independent (no User-agent line needed). */
+  /* robots_parse collects Sitemap: whatever the user-agent group, strips the
+     comment and keeps the rules working alongside it. */
+  {
+    const char *const txt = "User-agent: *\nDisallow: /x\n"
+                            "SITEMAP:  http://h.test/s1.xml  # first\n"
+                            "Sitemapper: http://h.test/no.xml\n"
+                            "Sitemap:\thttps://h.test/s2.xml\n";
+    char BIGSTK maps[1024];
+    robots_wizard rb;
+
+    memset(&rb, 0, sizeof(rb));
+    robots_parse(&rb, "h.test", txt, strlen(txt), NULL, 0, HTS_TRUE, maps,
+                 sizeof(maps));
+    assertf(strcmp(maps, "http://h.test/s1.xml\nhttps://h.test/s2.xml\n") == 0);
+    assertf(checkrobots(&rb, "h.test", "/x") == -1);
+    checkrobots_free(&rb);
+  }
+
+  printf("sitemap self-test OK\n");
+  return 0;
+}
+
 /* Connected stream pair over loopback; Windows has no socketpair(). */
 static int st_socketpair(T_SOC sv[2]) {
   struct sockaddr_in sa;
@@ -3838,21 +4053,6 @@ static int st_ftpuser(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
-/* Bounded substring search (records carry NUL bytes; strstr won't do). */
-static const char *warc_memstr(const char *hay, const char *needle,
-                               size_t haylen, size_t nlen) {
-  if (nlen == 0 || haylen < nlen)
-    return NULL;
-  {
-    size_t i;
-    for (i = 0; i + nlen <= haylen; i++) {
-      if (memcmp(hay + i, needle, nlen) == 0)
-        return hay + i;
-    }
-  }
-  return NULL;
-}
-
 /* Slurp a whole file into a malloc'd buffer; sets *len. NULL on error. */
 static unsigned char *warc_slurp(const char *path, size_t *len) {
   FILE *f = FOPEN(path, "rb");
@@ -3930,6 +4130,13 @@ static unsigned char *warc_next_member(const unsigned char **in,
    Content-Length == block length, the \r\n\r\n trailer intact, the response
    body round-trips, and the hop-by-hop Transfer-Encoding is dropped (a real
    Content-Encoding is kept verbatim; see warc-verbatim). */
+/* Argument order kept for the existing call sites; the search itself is the
+   shared hts_memstr. */
+static const char *warc_memstr(const char *hay, const char *needle,
+                               size_t haylen, size_t nlen) {
+  return hts_memstr(hay, haylen, needle, nlen);
+}
+
 static int st_warc(httrackp *opt, int argc, char **argv) {
   char path[HTS_URLMAXSIZE];
   warc_writer *w;
@@ -5349,6 +5556,8 @@ static const struct selftest_entry {
      st_contentcodings},
     {"robots", "", "robots.txt RFC 9309 Allow/Disallow precedence self-test",
      st_robots},
+    {"sitemap", "",
+     "sitemap <loc> extraction, caps and robots.txt Sitemap:", st_sitemap},
     {"ftp-line", "", "get_ftp_line bounds a hostile FTP reply line",
      st_ftpline},
     {"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser},

src/htsserver.c

@@ -318,18 +318,6 @@ typedef struct {
   error_redirect = "/server/error.html"; \
 } while(0)
 
-/* Longest error message shown on the error page; the rest is clipped. */
-#define ERROR_MESSAGE_MAX 1024
-
-/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
-   whose length the client picks. */
-#define SET_ERRORF(...)                                                        \
-  do {                                                                         \
-    char errbuf[ERROR_MESSAGE_MAX];                                            \
-    slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__);                    \
-    SET_ERROR(errbuf);                                                         \
-  } while (0)
-
 /* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
    so anything near this is already invalid and is rejected unread. */
 #define SID_VALUE_MAX 64
@@ -737,7 +725,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           fp = fopen(StringBuff(fspath), "rb");
           if (fp) {
             /* Read file */
-            while (!feof(fp) && !ferror(fp)) {
+            while(!feof(fp)) {
               char *str = line;
               char *pos;
 
@@ -891,18 +879,28 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                           commandEnd = 1;
                         }
                       } else {
-                        SET_ERRORF(
-                            "Unable to write %d bytes in the the init file %s",
-                            count, StringBuff(fspath));
+                        char tmp[1024];
+
+                        sprintf(tmp,
+                                "Unable to write %d bytes in the the init file %s",
+                                count, StringBuff(fspath));
+                        SET_ERROR(tmp);
                       }
                       fclose(fp);
                     } else {
-                      SET_ERRORF("Unable to create the init file %s",
-                                 StringBuff(fspath));
+                      char tmp[1024];
+
+                      sprintf(tmp, "Unable to create the init file %s",
+                              StringBuff(fspath));
+                      SET_ERROR(tmp);
                     }
                   } else {
-                    SET_ERRORF("Unable to create the directory structure in %s",
-                               StringBuff(fspath));
+                    char tmp[1024];
+
+                    sprintf(tmp,
+                            "Unable to create the directory structure in %s",
+                            StringBuff(fspath));
+                    SET_ERROR(tmp);
                   }
 
                 } else {
@@ -980,10 +978,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             }
           }
 
-          /* Regular files only: reading a directory or FIFO never ends, and
-             "path" may hold "..", so only the untrusted halves are checked. */
-          if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
-              (fp = fopen(fsfile, "rb"))) {
+          /* path itself may hold ".." (webhttrack passes "<bin>/../share"), so
+             only the untrusted halves are checked: file here, website above. */
+          if (fsfile[0] && strstr(file, "..") == NULL
+              && (fp = fopen(fsfile, "rb"))) {
             char ok[] =
               "HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
               "Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
@@ -1042,7 +1040,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               int outputmode = 0;
 
               StringMemcat(headers, ok, sizeof(ok) - 1);
-              while (!feof(fp) && !ferror(fp)) {
+              while(!feof(fp)) {
                 char *str = line;
                 int prevlen = (int) StringLength(output);
                 int nocr = 0;
@@ -1476,15 +1474,14 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               while(!feof(fp)) {
                 int n = (int) fread(line, 1, sizeof(line) - 2, fp);
 
-                if (n <= 0) {
-                  break; /* short read: EOF or error, never a retry */
+                if (n > 0) {
+                  StringMemcat(output, line, n);
                 }
-                StringMemcat(output, line, n);
               }
             }
             fclose(fp);
-          } else if (strcmp(file, "/ping") == 0 ||
-                     strncmp(file, "/ping?", 6) == 0) {
+          } else if (strcmp(file, "/ping") == 0
+                     || strncmp(file, "/ping?", 6) == 0) {
             char error_hdr[] =
               "HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
               "Content-type: text/html\r\n";

src/htssitemap.c

@@ -0,0 +1,615 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: sitemap ingestion (sitemaps.org 0.9)                   */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#define HTS_INTERNAL_BYTECODE
+
+#include "htscore.h"
+#include "htssitemap.h"
+
+#include "htsbase.h"
+#include "htscodec.h"
+#include "htsencoding.h"
+#include "htsfilters.h"
+#include "htshash.h"
+#include "htsmodules.h"
+#include "htslib.h"
+#include "htsrobots.h"
+#include "htssafe.h"
+#include "htstools.h"
+
+#include <ctype.h>
+#include <string.h>
+
+/* One queued sitemap document awaiting ingestion. */
+typedef struct sitemap_doc {
+  char adr[HTS_URLMAXSIZE];
+  char fil[HTS_URLMAXSIZE];
+  int level;
+  hts_sitemap_source src;
+  hts_boolean done;
+  struct sitemap_doc *next;
+} sitemap_doc;
+
+struct hts_sitemap_state {
+  sitemap_doc *docs;
+  int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */
+  int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */
+  hts_boolean probe_done;    /* the robots.txt probe has been answered */
+  hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */
+  /* The crawl's own start URL. Seeded URLs are judged against it, so a site
+     cannot widen a subtree crawl by putting its sitemap at the root. */
+  char anchor_adr[HTS_URLMAXSIZE];
+  char anchor_fil[HTS_URLMAXSIZE];
+};
+typedef struct hts_sitemap_state hts_sitemap_state;
+
+/* --------------------------------------------------------------------- */
+/* Document parsing (no engine state: fuzzable and self-testable)         */
+/* --------------------------------------------------------------------- */
+
+/* Accept only an absolute http(s) URL with no space or control byte. */
+static hts_boolean sitemap_url_ok(const char *url) {
+  const char *p;
+
+  if (!strfield(url, "http://") && !strfield(url, "https://"))
+    return HTS_FALSE;
+  for (p = url; *p != '\0'; p++) {
+    if ((unsigned char) *p <= ' ' || (unsigned char) *p == 0x7f)
+      return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Skip to the character after the next '>' at or after p, or NULL. */
+static const char *sitemap_tag_end(const char *p, const char *end) {
+  while (p < end && *p != '>')
+    p++;
+  return p < end ? p + 1 : NULL;
+}
+
+/* HTS_TRUE when the document's root element is `name`. Skips the XML
+   declaration, comments and processing instructions first, so a comment
+   mentioning the other root element cannot decide the document type. */
+static hts_boolean sitemap_root_is(const char *doc, size_t size,
+                                   const char *name) {
+  const size_t nlen = strlen(name);
+  size_t i = 0;
+
+  if (size >= 3 && memcmp(doc, "\xef\xbb\xbf", 3) == 0)
+    i = 3; /* UTF-8 BOM */
+  while (i < size) {
+    if (isspace((unsigned char) doc[i])) {
+      i++;
+    } else if (doc[i] != '<') {
+      return HTS_FALSE; /* character data before any element: not XML */
+    } else if (i + 4 <= size && memcmp(doc + i, "<!--", 4) == 0) {
+      const char *const e = hts_memstr(doc + i, size - i, "-->", 3);
+
+      if (e == NULL)
+        return HTS_FALSE;
+      i = (size_t) (e - doc) + 3;
+    } else if (i + 2 <= size && (doc[i + 1] == '?' || doc[i + 1] == '!')) {
+      while (i < size && doc[i] != '>')
+        i++;
+      i++;
+    } else {
+      size_t j = i + 1;
+
+      /* an optional namespace prefix: <sm:sitemapindex> is the same element */
+      while (j < size && doc[j] != ':' && doc[j] != '>' &&
+             !isspace((unsigned char) doc[j]))
+        j++;
+      if (j >= size || doc[j] != ':')
+        j = i + 1;
+      else
+        j++;
+      return j + nlen <= size && memcmp(doc + j, name, nlen) == 0 &&
+                     (j + nlen == size ||
+                      isspace((unsigned char) doc[j + nlen]) ||
+                      doc[j + nlen] == '>' || doc[j + nlen] == '/')
+                 ? HTS_TRUE
+                 : HTS_FALSE;
+    }
+  }
+  return HTS_FALSE;
+}
+
+/* Decompress a gzip-framed body into a fresh buffer. The 64 MiB cap is what
+   binds in practice; deflate tops out near 1032:1, so the tree's codec budget
+   only matters as the shared policy for a coding that could go further. */
+static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) {
+  const LLint budget = hts_codec_maxout((LLint) size);
+  size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES
+                   ? (size_t) budget
+                   : (size_t) HTS_SITEMAP_MAX_BYTES;
+  char *out;
+  size_t n;
+
+  if (cap == 0)
+    return NULL;
+  out = malloct(cap + 1);
+  if (out == NULL)
+    return NULL;
+  n = hts_codec_head(HTS_CODEC_DEFLATE, body, size, out, cap);
+  if (n == 0) {
+    freet(out);
+    return NULL;
+  }
+  out[n] = '\0';
+  *outsize = n;
+  return out;
+}
+
+int hts_sitemap_scan(const char *body, size_t size, int maxurls,
+                     hts_boolean *is_index, hts_sitemap_handler handler,
+                     void *arg) {
+  char *unpacked = NULL;
+  const char *doc;
+  const char *end;
+  const char *p;
+  int n = 0;
+
+  if (is_index != NULL)
+    *is_index = HTS_FALSE;
+  if (body == NULL || size < 2 || handler == NULL)
+    return 0;
+
+  /* Content-Encoding gzip is undone upstream; only the container is left. */
+  if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) {
+    unpacked = sitemap_gunzip(body, size, &size);
+    if (unpacked == NULL)
+      return -1;
+    doc = unpacked;
+  } else {
+    if (size > (size_t) HTS_SITEMAP_MAX_BYTES)
+      size = (size_t) HTS_SITEMAP_MAX_BYTES;
+    doc = body;
+  }
+  end = doc + size;
+
+  /* Set before the first callback: the handler reads the verdict. */
+  if (is_index != NULL)
+    *is_index = sitemap_root_is(doc, size, "sitemapindex");
+
+  for (p = doc; n < maxurls;) {
+    const char *loc = hts_memstr(p, (size_t) (end - p), "<loc", 4);
+    const char *val;
+    const char *stop;
+    size_t len;
+    char BIGSTK url[HTS_URLMAXSIZE];
+
+    if (loc == NULL)
+      break;
+    /* "<loc>" or "<loc xmlns:..>", never "<location>" */
+    if (loc + 4 >= end || (loc[4] != '>' && !isspace((unsigned char) loc[4]))) {
+      p = loc + 4;
+      continue;
+    }
+    val = sitemap_tag_end(loc + 4, end);
+    if (val == NULL)
+      break;
+    for (stop = val; stop < end && *stop != '<'; stop++)
+      ;
+    /* No closing tag: truncated document, so the value may be a partial URL. */
+    if (stop == end)
+      break;
+    p = stop;
+    while (val < stop && isspace((unsigned char) *val))
+      val++;
+    while (stop > val && isspace((unsigned char) *(stop - 1)))
+      stop--;
+    len = (size_t) (stop - val);
+    /* Overflow-safe: the untrusted length alone against the room left. */
+    if (len == 0 || len >= sizeof(url))
+      continue;
+    memcpy(url, val, len);
+    url[len] = '\0';
+    /* hts_unescapeEntities decodes in place and tolerates src == dest; a
+       reference to a control byte survives as one and sitemap_url_ok drops it.
+     */
+    if (hts_unescapeEntities(url, url, sizeof(url)) != 0 ||
+        !sitemap_url_ok(url))
+      continue;
+    n++;
+    if (!handler(arg, url))
+      break;
+  }
+
+  if (unpacked != NULL)
+    freet(unpacked);
+  return n;
+}
+
+/* --------------------------------------------------------------------- */
+/* Engine glue                                                           */
+/* --------------------------------------------------------------------- */
+
+static hts_sitemap_state *sitemap_state(httrackp *opt) {
+  if (opt->state.sitemap == NULL)
+    opt->state.sitemap = calloct(1, sizeof(hts_sitemap_state));
+  return (hts_sitemap_state *) opt->state.sitemap;
+}
+
+static sitemap_doc *sitemap_find(httrackp *opt, const char *adr,
+                                 const char *fil) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap;
+  sitemap_doc *d;
+
+  if (st == NULL)
+    return NULL;
+  for (d = st->docs; d != NULL; d = d->next) {
+    if (strfield2(d->adr, adr) && strcmp(d->fil, fil) == 0)
+      return d;
+  }
+  return NULL;
+}
+
+/* Who asked for this document decides how far it is gated. The wizard proper
+   is not usable here: it wants a referring link, and its up/down travel rules
+   would judge a child sitemap against the parent sitemap's directory. */
+static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr,
+                                         const char *fil,
+                                         hts_sitemap_source src) {
+  /* adr and fil are each capped just under HTS_URLMAXSIZE, and lfull prefixes
+     a scheme and a slash on top of both: 2 * HTS_URLMAXSIZE does not fit. */
+  char BIGSTK l[HTS_URLMAXSIZE * 2 + 16], lfull[HTS_URLMAXSIZE * 2 + 16];
+  int jokdepth = 0, jok;
+
+  hts_boolean refused;
+
+  /* The user naming a sitemap is the same intent as naming a start URL, which
+     the wizard admits unconditionally. */
+  if (src == HTS_SITEMAP_SRC_USER)
+    return HTS_TRUE;
+  strcpybuff(l, jump_identification_const(adr));
+  if (*fil != '/')
+    strcatbuff(l, "/");
+  strcatbuff(l, fil);
+  strcpybuff(lfull, link_has_authority(adr) ? "" : "http://");
+  strcatbuff(lfull, adr);
+  if (*fil != '/')
+    strcatbuff(lfull, "/");
+  strcatbuff(lfull, fil);
+  jok = fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull,
+                         l, NULL, NULL, &jokdepth);
+  refused = (jok == -1) ? HTS_TRUE : HTS_FALSE;
+  if (refused) {
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s",
+                  jokdepth + 1, adr, fil);
+    return HTS_FALSE;
+  }
+  /* A Sitemap: line, or a sitemapindex entry, is the site inviting the fetch;
+     a Disallow elsewhere in the same file does not retract it. The well-known
+     location is only ever a guess, so there a Disallow wins. */
+  if (src == HTS_SITEMAP_SRC_GUESSED &&
+      hts_robots_forbids(opt, adr, fil, (jok != 0) ? HTS_TRUE : HTS_FALSE,
+                         refused)) {
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr,
+                  fil);
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Record the link with save="" so the body stays in memory: a sitemap is
+   ingested, never mirrored. */
+static hts_boolean sitemap_queue_(httrackp *opt, const char *adr,
+                                  const char *fil, int level,
+                                  hts_sitemap_source src, hts_boolean link_it) {
+  hts_sitemap_state *const st = sitemap_state(opt);
+  sitemap_doc *d;
+
+  if (st == NULL)
+    return HTS_FALSE;
+  if (st->ndocs >= HTS_SITEMAP_MAX_DOCS || level > HTS_SITEMAP_MAX_LEVEL) {
+    hts_log_print(opt, LOG_WARNING, "Sitemap: cap reached, skipping %s%s", adr,
+                  fil);
+    return HTS_FALSE;
+  }
+  if (strlen(adr) >= sizeof(d->adr) || strlen(fil) >= sizeof(d->fil))
+    return HTS_FALSE;
+  if (sitemap_find(opt, adr, fil) != NULL)
+    return HTS_FALSE;
+  if (!sitemap_fetch_allowed(opt, adr, fil, src))
+    return HTS_FALSE;
+  d = calloct(1, sizeof(sitemap_doc));
+  if (d == NULL)
+    return HTS_FALSE;
+  strcpybuff(d->adr, adr);
+  strcpybuff(d->fil, fil);
+  d->level = level;
+  d->src = src;
+  d->next = st->docs;
+  st->docs = d;
+  st->ndocs++;
+
+  if (!link_it)
+    return HTS_TRUE;
+  if (!hts_record_link(opt, adr, fil, "", "", "", NULL))
+    return HTS_FALSE;
+  heap_top()->testmode = 0;
+  heap_top()->link_import = 0;
+  heap_top()->depth = opt->depth + 1;
+  heap_top()->pass2 = 0;
+  heap_top()->retry = opt->retry;
+  heap_top()->premier = heap_top_index();
+  heap_top()->precedent = heap_top_index();
+  hts_log_print(opt, LOG_INFO, "Sitemap: queued %s%s", adr, fil);
+  return HTS_TRUE;
+}
+
+static hts_boolean sitemap_queue(httrackp *opt, const char *adr,
+                                 const char *fil, int level,
+                                 hts_sitemap_source src) {
+  return sitemap_queue_(opt, adr, fil, level, src, HTS_TRUE);
+}
+
+void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
+                          const char *newadr, const char *newfil) {
+  sitemap_doc *const d = sitemap_find(opt, adr, fil);
+
+  if (d == NULL || d->done)
+    return;
+  d->done = HTS_TRUE; /* the body lives at the target now */
+  /* The engine already queued the target link, so only the marking moves. */
+  (void) sitemap_queue_(opt, newadr, newfil, d->level, d->src, HTS_FALSE);
+  hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil,
+                newadr, newfil);
+}
+
+void hts_sitemap_seed(httrackp *opt, const char *starturl) {
+  char BIGSTK url[HTS_URLMAXSIZE * 2];
+  lien_adrfil af;
+
+  if (StringNotEmpty(opt->sitemap_url)) {
+    if (strlen(StringBuff(opt->sitemap_url)) >= sizeof(url)) {
+      hts_log_print(opt, LOG_ERROR, "Sitemap URL too long");
+    } else {
+      strcpybuff(url, StringBuff(opt->sitemap_url));
+      if (strstr(url, ":/") == NULL)
+        hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url);
+      else if (ident_url_absolute(url, &af) >= 0)
+        (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_USER);
+    }
+  }
+  if (starturl == NULL || starturl[0] == '\0' ||
+      strlen(starturl) >= sizeof(url))
+    return;
+  strcpybuff(url, starturl);
+  if (ident_url_absolute(url, &af) < 0)
+    return;
+  {
+    hts_sitemap_state *const st = sitemap_state(opt);
+
+    if (st != NULL && strlen(af.adr) < sizeof(st->anchor_adr) &&
+        strlen(af.fil) < sizeof(st->anchor_fil)) {
+      strcpybuff(st->anchor_adr, af.adr);
+      strcpybuff(st->anchor_fil, af.fil);
+    }
+  }
+  if (!opt->sitemap)
+    return;
+  /* Answered in hts_sitemap_robots, once the parsed rules are installed. */
+  if (hts_record_link(opt, af.adr, "/robots.txt", "", "", "", NULL)) {
+    heap_top()->testmode = 0;
+    heap_top()->link_import = 0;
+    heap_top()->depth = 0;
+    heap_top()->pass2 = 0;
+    heap_top()->retry = opt->retry;
+    heap_top()->premier = heap_top_index();
+    heap_top()->precedent = heap_top_index();
+    /* Claim the host so the parser does not queue robots.txt a second time. */
+    if (opt->robotsptr != NULL)
+      (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, "");
+  }
+}
+
+void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap;
+  int queued = 0;
+
+  if (st == NULL || !opt->sitemap || st->probe_done ||
+      !strfield2(st->anchor_adr, adr))
+    return;
+  st->probe_done = HTS_TRUE;
+  if (sitemaps != NULL) {
+    const char *p = sitemaps;
+
+    while (*p != '\0') {
+      const char *const eol = strchr(p, '\n');
+      const size_t len = eol != NULL ? (size_t) (eol - p) : strlen(p);
+      char BIGSTK line[HTS_URLMAXSIZE];
+      lien_adrfil af;
+
+      if (len > 0 && len < sizeof(line)) {
+        memcpy(line, p, len);
+        line[len] = '\0';
+        /* Same host: a Sitemap: line must not aim the fetcher elsewhere. */
+        if (sitemap_url_ok(line) && ident_url_absolute(line, &af) >= 0 &&
+            strfield2(af.adr, adr) &&
+            sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_DECLARED))
+          queued++;
+      }
+      if (eol == NULL)
+        break;
+      p = eol + 1;
+    }
+  }
+  if (queued == 0 && !st->fallback_done) {
+    st->fallback_done = HTS_TRUE;
+    if (sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_SITEMAP_SRC_GUESSED))
+      queued++;
+  }
+  hts_log_print(opt, LOG_NOTICE, "Sitemap: %d sitemap(s) queued for %s", queued,
+                adr);
+}
+
+hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
+                                const char *fil) {
+  const sitemap_doc *const d = sitemap_find(opt, adr, fil);
+
+  return d != NULL && !d->done ? HTS_TRUE : HTS_FALSE;
+}
+
+/* Handler context: seeding URLs from one document. */
+typedef struct sitemap_ingest_ctx {
+  httrackp *opt;
+  htsmoduleStruct *str;
+  const char *adr; /* host of the document being ingested */
+  int level;
+  hts_boolean is_index;
+  int accepted; /* URLs seeded or documents queued, not merely parsed */
+} sitemap_ingest_ctx;
+
+/* A <loc> of a <urlset>: hand it to the wizard as a top-level seed.
+   The wizard is pointed at the crawl's own start URL, not at the sitemap: the
+   site picks where its sitemap lives, so anchoring travel there would let a
+   root sitemap widen a subtree crawl to the whole host. The URL then becomes
+   its own anchor, exactly as a command-line seed does. */
+static hts_boolean sitemap_seed_url(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+  httrackp *const opt = c->opt;
+  hts_sitemap_state *const st = sitemap_state(opt);
+  char BIGSTK buff[HTS_URLMAXSIZE];
+  int before;
+
+  if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) {
+    hts_log_print(opt, LOG_WARNING,
+                  "Sitemap: URL cap reached, ignoring the rest");
+    return HTS_FALSE;
+  }
+  /* strcpybuff aborts rather than truncating: never feed it unchecked input. */
+  if (strlen(url) >= sizeof(buff))
+    return HTS_TRUE;
+  st->nurls++;
+  strcpybuff(buff, url);
+  before = opt->lien_tot;
+  if (htsAddLink(c->str, buff))
+    c->accepted++;
+  if (opt->lien_tot > before)
+    heap_top()->premier = heap_top_index(); /* a seed anchors on itself */
+  return HTS_TRUE;
+}
+
+/* A <loc> of a <sitemapindex>: cross-host children are dropped, so a hostile
+   sitemap cannot aim the fetcher elsewhere. */
+static hts_boolean sitemap_seed_child(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+  char BIGSTK buff[HTS_URLMAXSIZE];
+  lien_adrfil af;
+
+  if (strlen(url) >= sizeof(buff))
+    return HTS_TRUE;
+  strcpybuff(buff, url);
+  if (ident_url_absolute(buff, &af) < 0)
+    return HTS_TRUE;
+  if (!strfield2(af.adr, c->adr)) {
+    hts_log_print(c->opt, LOG_WARNING,
+                  "Sitemap: ignoring off-host child sitemap %s%s", af.adr,
+                  af.fil);
+    return HTS_TRUE;
+  }
+  if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1,
+                    HTS_SITEMAP_SRC_DECLARED))
+    c->accepted++;
+  return HTS_TRUE;
+}
+
+/* The scan classifies the document before the first callback. */
+static hts_boolean sitemap_seed_any(void *arg, const char *url) {
+  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
+
+  return c->is_index ? sitemap_seed_child(arg, url)
+                     : sitemap_seed_url(arg, url);
+}
+
+void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
+                        const char *fil, const char *body, size_t size) {
+  sitemap_doc *const d = sitemap_find(opt, adr, fil);
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap;
+  sitemap_ingest_ctx ctx;
+  int n, anchor, saved_depth;
+
+  if (d == NULL || d->done)
+    return;
+  d->done = HTS_TRUE;
+  /* str->ptr_ is a scratch int owned by the caller, so nothing else moves. */
+  anchor = *str->ptr_;
+  if (st != NULL && st->anchor_adr[0] != '\0' && opt->hash != NULL) {
+    const int i = hash_read((const hash_struct *) opt->hash, st->anchor_adr,
+                            st->anchor_fil, 1);
+
+    if (i >= 0)
+      anchor = i;
+  }
+  *str->ptr_ = anchor;
+  /* Borrow the anchor's position but keep a seed's full depth budget. */
+  saved_depth = heap(anchor)->depth;
+  heap(anchor)->depth = opt->depth + 1;
+  ctx.opt = opt;
+  ctx.str = str;
+  ctx.adr = adr;
+  ctx.level = d->level;
+  ctx.is_index = HTS_FALSE;
+  ctx.accepted = 0;
+
+  n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index,
+                       sitemap_seed_any, &ctx);
+  heap(anchor)->depth = saved_depth;
+  if (n < 0) {
+    hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr,
+                  fil);
+    return;
+  }
+  if (ctx.is_index)
+    hts_log_print(opt, LOG_NOTICE,
+                  "Sitemap: %d of %d child sitemap(s) listed by %s%s",
+                  ctx.accepted, n, adr, fil);
+  else
+    hts_log_print(opt, LOG_NOTICE, "Sitemap: %d of %d URL(s) added from %s%s",
+                  ctx.accepted, n, adr, fil);
+}
+
+void hts_sitemap_free(httrackp *opt) {
+  hts_sitemap_state *const st = (hts_sitemap_state *) opt->state.sitemap;
+
+  if (st == NULL)
+    return;
+  while (st->docs != NULL) {
+    sitemap_doc *const next = st->docs->next;
+
+    freet(st->docs);
+    st->docs = next;
+  }
+  freet(opt->state.sitemap);
+  opt->state.sitemap = NULL;
+}

src/htssitemap.h

@@ -0,0 +1,108 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 2026 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* HTTrack sitemap ingestion (sitemaps.org 0.9). Internal, not installed.
+   Reads <urlset>/<sitemapindex> documents, plain or gzip-framed, and feeds
+   their <loc> URLs to the crawl as top-level seeds. The whole input is
+   attacker-controlled, so every entry point below is capped. */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_SITEMAP_DEFH
+#define HTS_SITEMAP_DEFH
+
+#include "htsdefines.h"
+#include "htsopt.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Caps. sitemaps.org allows 50000 URLs and 50 MB uncompressed per document;
+   the byte cap sits above that so a conformant sitemap always fits. */
+#define HTS_SITEMAP_MAX_URLS_DOC 50000           /* <loc> per document */
+#define HTS_SITEMAP_MAX_URLS_TOTAL 200000        /* <loc> per mirror */
+#define HTS_SITEMAP_MAX_DOCS 256                 /* documents per mirror */
+#define HTS_SITEMAP_MAX_LEVEL 4                  /* sitemapindex nesting */
+#define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */
+
+/* Who asked for a sitemap document, which decides how far its fetch is gated.
+   The user naming one is the same intent as a start URL; a site declaring one
+   invites the fetch; the well-known location is only ever our guess. */
+typedef enum {
+  HTS_SITEMAP_SRC_USER,     /**< --sitemap-url */
+  HTS_SITEMAP_SRC_DECLARED, /**< a Sitemap: line or a sitemapindex entry */
+  HTS_SITEMAP_SRC_GUESSED   /**< the /sitemap.xml fallback */
+} hts_sitemap_source;
+
+/* Per-URL handler; returning HTS_FALSE stops the scan. */
+typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url);
+
+/* Scan one sitemap document, plain or gzip-framed, handing every acceptable
+   absolute http(s) <loc> URL to `handler`. Stops after `maxurls` URLs, or when
+   the handler refuses. `is_index` (optional) reports a <sitemapindex>, whose
+   URLs are child sitemaps rather than pages. Returns the number of URLs handed
+   out, or -1 when the document could not be decompressed within the caps. */
+int hts_sitemap_scan(const char *body, size_t size, int maxurls,
+                     hts_boolean *is_index, hts_sitemap_handler handler,
+                     void *arg);
+
+/* --- Engine glue (needs a live httrackp). --- */
+
+/* Queue the first sitemap document of the mirror: the explicit --sitemap-url,
+   or the start host's /robots.txt probe for --sitemap. `starturl` is the first
+   command-line seed. No-op when neither option is set. */
+void hts_sitemap_seed(httrackp *opt, const char *starturl);
+
+/* Act on the start host's robots.txt once its rules are installed: queue the
+   Sitemap: URLs it names (newline-separated, from robots_parse), or the
+   well-known /sitemap.xml when it names none. No-op unless --sitemap. */
+void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps);
+
+/* Carry the sitemap marking of (adr,fil) over to the target of a redirect the
+   engine has already queued, so a moved sitemap is still ingested. */
+void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
+                          const char *newadr, const char *newfil);
+
+/* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */
+hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
+                                const char *fil);
+
+/* Ingest a fetched sitemap document (or the robots.txt probe): seed its URLs
+   through the wizard via htsAddLink, and queue nested sitemaps. `str` supplies
+   the parser context of the document being processed. */
+void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
+                        const char *fil, const char *body, size_t size);
+
+/* Release the ingestion state held in opt (NULL-safe, idempotent). */
+void hts_sitemap_free(httrackp *opt);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

src/htstools.c

@@ -276,6 +276,21 @@ int ident_url_relatif(const char *lien, const char *origin_adr,
   return ok;
 }
 
+/* Bounded substring search: bodies and archive records carry NUL bytes, so
+   strstr() would stop at the first one. */
+const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
+                       size_t nlen) {
+  size_t i;
+
+  if (nlen == 0 || haylen < nlen)
+    return NULL;
+  for (i = 0; i + nlen <= haylen; i++) {
+    if (hay[i] == *needle && memcmp(hay + i, needle, nlen) == 0)
+      return hay + i;
+  }
+  return NULL;
+}
+
 // créer dans s, à partir du chemin courant curr_fil, le lien vers link (absolu)
 // un ident_url_relatif a déja été fait avant, pour que link ne soit pas un chemin relatif
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {

src/htstools.h

@@ -61,6 +61,11 @@ typedef struct lien_adrfilsave lien_adrfilsave;
 int ident_url_relatif(const char *lien, const char *origin_adr,
                       const char *origin_fil,
                       lien_adrfil* const adrfil);
+/* Bounded substring search over data that may hold NUL bytes; NULL if absent.
+ */
+const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
+                       size_t nlen);
+
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr);
 int link_has_authority(const char *lien);
 int link_has_authorization(const char *lien);

src/htswizard.c

@@ -151,6 +151,23 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag,
   return HTS_FALSE;
 }
 
+/* The engine's robots.txt verdict for (adr,fil). Under HTS_ROBOTS_SOMETIMES an
+   explicit filter acceptance overrides the ban, which is why the filter outcome
+   is an input; the sitemap fetcher asks the same question outside the wizard.
+ */
+hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
+                               hts_boolean filters_decided,
+                               hts_boolean filters_refused) {
+  if (!opt->robots || opt->robotsptr == NULL)
+    return HTS_FALSE;
+  if (checkrobots((robots_wizard *) opt->robotsptr, adr, fil) != -1)
+    return HTS_FALSE;
+  if (filters_decided && !filters_refused &&
+      opt->robots == HTS_ROBOTS_SOMETIMES)
+    return HTS_FALSE;
+  return HTS_TRUE;
+}
+
 static int hts_acceptlink_(httrackp * opt, int ptr,
                            const char *adr, const char *fil, const char *tag,
                            const char *attribute, int *set_prio_to,
@@ -576,30 +593,26 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
       }
     }
     // vérifier robots.txt
-    if (opt->robots) {
-      int r = checkrobots(_ROBOTS, adr, fil);
-
-      if (r == -1) {            // interdiction
+    if (opt->robots && checkrobots(_ROBOTS, adr, fil) == -1) {
 #if DEBUG_ROBOTS
-        printf("robots.txt forbidden: %s%s\n", adr, fil);
+      printf("robots.txt forbidden: %s%s\n", adr, fil);
 #endif
-        // question résolue, par les filtres, et mode robot non strict
-        if ((!question) && (filters_answer) &&
-            (opt->robots == HTS_ROBOTS_SOMETIMES) && (forbidden_url != 1)) {
-          r = 0;                // annuler interdiction des robots
-          if (!forbidden_url) {
-            hts_log_print(opt, LOG_DEBUG,
-                          "Warning link followed against robots.txt: link %s at %s%s",
-                          l, adr, fil);
-          }
-        }
-        if (r == -1) {          // interdire
-          forbidden_url = 1;
-          question = 0;
-          hts_log_print(opt, LOG_DEBUG,
-                        "(robots.txt) forbidden link: link %s at %s%s", l, adr,
-                        fil);
+      if (!hts_robots_forbids(opt, adr, fil,
+                              (!question && filters_answer) ? HTS_TRUE
+                                                            : HTS_FALSE,
+                              (forbidden_url == 1) ? HTS_TRUE : HTS_FALSE)) {
+        if (!forbidden_url) {
+          hts_log_print(
+              opt, LOG_DEBUG,
+              "Warning link followed against robots.txt: link %s at %s%s", l,
+              adr, fil);
         }
+      } else {
+        forbidden_url = 1;
+        question = 0;
+        hts_log_print(opt, LOG_DEBUG,
+                      "(robots.txt) forbidden link: link %s at %s%s", l, adr,
+                      fil);
       }
     }
 

src/htswizard.h

@@ -49,6 +49,13 @@ typedef struct httrackp httrackp;
 typedef struct lien_url lien_url;
 #endif
 
+/* The engine's robots.txt verdict for (adr,fil): HTS_TRUE when the fetch is
+   forbidden. `filters_decided`/`filters_refused` carry the filter outcome,
+   which overrides a ban under -s1 (HTS_ROBOTS_SOMETIMES). */
+hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
+                               hts_boolean filters_decided,
+                               hts_boolean filters_refused);
+
 int hts_acceptlink(httrackp * opt, int ptr,
                    const char *adr, const char *fil,
                    const char *tag, const char *attribute,

src/libhttrack.vcxproj

@@ -138,6 +138,7 @@
     <ClCompile Include="htswrap.c" />
     <ClCompile Include="htszlib.c" />
     <ClCompile Include="htswarc.c" />
+    <ClCompile Include="htssitemap.c" />
     <ClCompile Include="md5.c" />
     <ClCompile Include="minizip\ioapi.c" />
     <ClCompile Include="minizip\iowin32.c" />

src/proxy/proxytrack.c

@@ -820,8 +820,8 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     elt->size = StringLength(response);
     elt->adr = StringAcquire(&response);
     elt->statuscode = 207;      /* Multi-Status */
-    strcpybuff(elt->charset, "utf-8");
-    strcpybuff(elt->contenttype, "text/xml");
+    strcpy(elt->charset, "utf-8");
+    strcpy(elt->contenttype, "text/xml");
     strcpybuff(elt->msg, "Multi-Status");
     StringFree(response);
 
@@ -877,8 +877,8 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
     elt->size = StringLength(html);
     elt->adr = StringAcquire(&html);
     elt->statuscode = HTTP_OK;
-    strcpybuff(elt->charset, "iso-8859-1");
-    strcpybuff(elt->contenttype, "text/html");
+    strcpy(elt->charset, "iso-8859-1");
+    strcpy(elt->contenttype, "text/html");
     strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;

src/proxy/proxytrack.h

@@ -368,11 +368,7 @@ HTS_UNUSED static struct tm PT_GetTime(time_t t) {
   if (tm != NULL)
     return *tm;
   else {
-    /* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
-       day of "00"; the epoch is the conventional "date unknown" */
     memset(&tmbuf, 0, sizeof(tmbuf));
-    tmbuf.tm_year = 70;
-    tmbuf.tm_mday = 1;
     return tmbuf;
   }
 }

src/proxy/store.c

@@ -405,8 +405,8 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
     elt->size = StringLength(html);
     elt->adr = StringAcquire(&html);
     elt->statuscode = HTTP_OK;
-    strcpybuff(elt->charset, "iso-8859-1");
-    strcpybuff(elt->contenttype, "text/html");
+    strcpy(elt->charset, "iso-8859-1");
+    strcpy(elt->contenttype, "text/html");
     strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;
@@ -829,8 +829,16 @@ PT_Element PT_ElementNew(void) {
 }
 
 /* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
-#define PT_Element_failf(R, ...)                                               \
-  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
+static void PT_Element_failf(PT_Element r, const char *fmt, ...)
+    HTS_PRINTF_FUN(2, 3);
+
+static void PT_Element_failf(PT_Element r, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
+  va_end(args);
+}
 
 PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
   if (index != NULL && SAFE_INDEX(index)) {
@@ -871,11 +879,13 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
   (headersSize) += (int) strlen(headers + headersSize); \
 } while(0)
 /* refvalue_size is mandatory: the cache line is bounded only by the line
-   buffer, not by the destination. */
+   buffer, not by the destination. Clip rather than reject, since an
+   engine-written field can be wider than ours. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (void) strclipbuff(refvalue, refvalue_size, value);                      \
+      (refvalue)[0] = '\0';                                                    \
+      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -1032,7 +1042,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
   previous_save[0] = previous_save_[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  r->location[0] = '\0';
+  strcpy(r->location, "");
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -1117,7 +1127,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
             int pathLen = (int) strlen(index->path);
 
             if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) {    // old (<3.40) buggy format
-              strcpybuff(previous_save, previous_save_);
+              strcpy(previous_save, previous_save_);
             }
             // relative ? (hack)
             else if (index->safeCache || (previous_save_[0] != '/'      // /home/foo/bar.gif
@@ -1544,7 +1554,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
             cache->version = (int) (firstline[8] - '0');        // cache 1.x
             if (cache->version <= 5) {
               a += cache_brstr(a, firstline, sizeof(firstline));
-              strcpybuff(cache->lastmodified, firstline);
+              strcpy(cache->lastmodified, firstline);
             } else {
               fclose(cache->dat);
               cache->dat = NULL;
@@ -1561,7 +1571,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
         } else {                // Vieille version du cache
           /* */
           cache->version = 0;   // cache 1.0
-          strcpybuff(cache->lastmodified, firstline);
+          strcpy(cache->lastmodified, firstline);
         }
 
         /* Create hash table for the cache (MUCH FASTER!) */
@@ -1577,14 +1587,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
               a++;
               /* read "host/file" */
               a += binput(a, line, HTS_URLMAXSIZE);
-              {
-                /* binput writes its NUL at s[max], so the second field must be
-                   bounded by what is left of line[], not by the same constant
-                 */
-                const size_t used = strlen(line);
-
-                a += binput(a, line + used, (int) (sizeof(line) - used - 1));
-              }
+              a += binput(a, line + strlen(line), HTS_URLMAXSIZE);
               /* read position */
               a += binput(a, linepos, 200);
               sscanf(linepos, "%d", &pos);
@@ -1681,7 +1684,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
   previous_save[0] = previous_save_[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  r->location[0] = '\0';
+  strcpy(r->location, "");
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(cache->hash, url, &hash_pos);
@@ -1788,7 +1791,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
           int pathLen = (int) strlen(index->path);
 
           if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) {      // old (<3.40) buggy format
-            strcpybuff(previous_save, previous_save_);
+            strcpy(previous_save, previous_save_);
           }
           // relative ? (hack)
           else if (index->safeCache || (previous_save_[0] != '/'        // /home/foo/bar.gif
@@ -2165,7 +2168,8 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
 #define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)   \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (void) strclipbuff(refvalue, refvalue_size, value);                      \
+      (refvalue)[0] = '\0';                                                    \
+      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -2204,7 +2208,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
   location_default[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  r->location[0] = '\0';
+  strcpy(r->location, "");
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -2423,7 +2427,7 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   if (element->adr != NULL) {
     domd5mem(element->adr, element->size, st->md5, 1);
   } else {
-    strcpybuff(st->md5, "-");
+    strcpy(st->md5, "-");
   }
   fprintf(fp,
           /* nl */

tests/01_zlib-sitemap.test

@@ -0,0 +1,10 @@
+#!/bin/bash
+#
+# Sitemap parser self-test: <loc> extraction, entity decoding, URL and length
+# rejections, the URL cap, gzip framing and robots.txt Sitemap: records.
+
+set -euo pipefail
+
+out=$(httrack -O /dev/null '-#test=sitemap')
+echo "$out"
+test "$out" = "sitemap self-test OK"

tests/77_webhttrack-redirect.test

@@ -146,13 +146,11 @@ sid=$(scrape_sid "${port}")
 resp=$(post_redirect '/foo
 X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
 stop
-# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
-# pipefail then turns the hit into a silent pass.
-grep -qi 'X-Injected' <<<"${resp}" &&
+printf '%s' "${resp}" | grep -qi 'X-Injected' &&
     fail "CRLF in the redirect value reached the response"
-test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
+test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
     fail "CRLF redirect still emitted a Location header"
-test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
+test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
     fail "CRLF redirect did not yield exactly one status line"
 
 # Loopback unless asked otherwise. Assert the socket, not the announcement.

tests/78_webhttrack-sid.test

@@ -103,10 +103,9 @@ sid=$(scrape_sid "${port}")
 test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
 
 # Accept: the legitimate flow still works. "redirect" is the cheapest field
-# with a reply that is visible in the headers. Matched from a here-string, not a
-# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
+# with a reply that is visible in the headers.
 resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
-grep -q '^Location: /accepted' <<<"${resp}" ||
+printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
     fail "a body carrying the correct sid was refused"
 
 # Refuse: missing and wrong. A missing one is the regression under test — the
@@ -115,12 +114,12 @@ grep -q '^Location: /accepted' <<<"${resp}" ||
 for bad in "redirect=/nosid" "sid=&redirect=/empty" \
     "sid=00000000000000000000000000000000&redirect=/wrong"; do
     resp=$(request "${port}" "${bad}")
-    grep -q '^Location:' <<<"${resp}" &&
+    printf '%s' "${resp}" | grep -q '^Location:' &&
         fail "body accepted without a valid sid: ${bad}"
     # A refusal has to be a well-formed reply, not a headerless fragment: the
     # release build used to emit only Content-length, which reads as a protocol
     # error to any client and hides the reason.
-    grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
+    printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
         fail "refusal was not a 403: ${bad}"
 done
 
@@ -128,23 +127,16 @@ done
 # applied to one global key store that later requests render from, and the
 # command dispatcher reads it from there. Probe the store itself — step3
 # interpolates ${projname} into its title — rather than the refused reply.
-store_page() {
-    # a dead probe or a non-page reply must not read as "the store was not written"
-    page=$(request "$1" "" step3) || fail "the key store probe failed"
-    grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
-        fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
-}
+title() { request "$1" "" step3; }
 
 request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
-store_page "${port}"
-grep -q 'UNAUTHWRITE' <<<"${page}" &&
+title "${port}" | grep -q 'UNAUTHWRITE' &&
     fail "a body without a valid sid was written to the key store"
 
 # Paired accept case: without it the assertion above passes even if the server
 # simply ignores every body, which would prove nothing.
 request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
-store_page "${port}"
-grep -q 'AUTHWRITE' <<<"${page}" ||
+title "${port}" | grep -q 'AUTHWRITE' ||
     fail "a body carrying the correct sid was not written to the key store"
 
 echo "PASS"

tests/85_webhttrack-projpath.test

@@ -90,15 +90,6 @@ sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
 get() { request "$1" "$2" ""; }
 post() { request "$1" "" "$2"; }
 
-# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
-# reads as marker-absent, and so do a truncated body and a 302 to the file.
-reply=
-fetch() {
-    reply=$(get "$1" "$2") || fail "the request for $2 failed"
-    grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
-        fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
-}
-
 url=$(start)
 port=$(portof "${url}")
 srv=$(srvpid)
@@ -118,14 +109,13 @@ echo "LOGMARKER" >"${base}/proj/hts-log.txt"
 # with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
 # any successful save: commandEnd then swaps the error page for the finished one.
 get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
-saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
-grep -q '^Location: /server/error.html' <<<"${saved}" ||
+post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p" |
+    grep -q '^Location: /server/error.html' ||
     fail "the refused save did not redirect to the error page"
 
 # No project yet, so no root to serve from.
 post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
-fetch "${port}" /website/secret.txt 404
-grep -q SECRETMARKER <<<"${reply}" &&
+get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
     fail "a posted projpath served a file outside any project"
 
 # Positive control: step4's "save settings" flow registers the project without
@@ -136,29 +126,24 @@ body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
 post "${port}" "${body}" >/dev/null
 test -f "${base}/proj/hts-cache/winprofile.ini" ||
     fail "the project was not registered: $(cat "${log}")"
-fetch "${port}" /website/hts-log.txt 200
-grep -q LOGMARKER <<<"${reply}" ||
+get "${port}" /website/hts-log.txt | grep -q LOGMARKER ||
     fail "the registered project's mirror is not served"
 
 # Same request with the root repointed: the project is legitimate, projpath is
 # not what decides where the bytes come from.
 post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
-fetch "${port}" /website/secret.txt 404
-grep -q SECRETMARKER <<<"${reply}" &&
+get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
     fail "a posted projpath repointed the served root"
 post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
-fetch "${port}" /website/passwd 404
-grep -q '^root:' <<<"${reply}" &&
+get "${port}" /website/passwd | grep -q '^root:' &&
     fail "a posted projpath read an arbitrary system file"
 
 # A ".." anywhere in the recorded root would escape the mirror on every later
 # request, so the save must be refused and the previous root kept.
 post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
-fetch "${port}" /website/secret.txt 404
-grep -q SECRETMARKER <<<"${reply}" &&
+get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
     fail "a '..' in the saved project path escaped the mirror root"
-fetch "${port}" /website/hts-log.txt 200
-grep -q LOGMARKER <<<"${reply}" ||
+get "${port}" /website/hts-log.txt | grep -q LOGMARKER ||
     fail "rejecting the '..' root also lost the previous one"
 
 # The root is now the server's own, but it is still built from two posted
@@ -177,7 +162,7 @@ test -f "${fspath}/hts-cache/winprofile.ini" ||
     fail "the long-path project was not registered: $(cat "${log}")"
 get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
 alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
-# Not just alive: still answering.
-fetch "${port}" /server/index.html 200
+get "${port}" /server/index.html | grep -q '200 OK' ||
+    fail "the server stopped answering after the over-long project path"
 
 echo "PASS"

tests/88_local-proxytrack-badmtime.test

@@ -1,69 +0,0 @@
-#!/bin/bash
-# A cache file whose mtime is beyond what gmtime can represent must not put a
-# day of "00" in the ARC filedesc date: PT_GetTime fell back to an all-zero tm.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-python=$(find_python) || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-dir=$(mktemp -d)
-trap 'rm -rf "$dir"' EXIT
-
-rc=0
-"$python" - "$dir/in.zip" <<'EOF' || rc=$?
-import os, sys, zipfile
-
-body = b"hello world"
-meta = (
-    "X-In-Cache: 1\r\n"
-    "X-StatusCode: 200\r\n"
-    "X-StatusMessage: OK\r\n"
-    "X-Size: %d\r\n"
-    "Content-Type: text/html\r\n"
-    "Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
-    "X-Save: out/page.html\r\n" % len(body)
-).encode()
-
-zi = zipfile.ZipInfo("example.com/page.html")
-zi.compress_type = zipfile.ZIP_STORED
-zi.extra = meta
-with zipfile.ZipFile(sys.argv[1], "w") as z:
-    z.writestr(zi, body)
-
-# past gmtime's range; the index timestamp is this file's mtime
-huge = 4611686018427387903
-try:
-    os.utime(sys.argv[1], (huge, huge))
-except OSError:
-    sys.exit(77)
-if int(os.stat(sys.argv[1]).st_mtime) < huge:
-    sys.exit(77)  # filesystem clamped it, nothing to test
-EOF
-test "$rc" -eq 0 || {
-    test "$rc" -eq 77 || {
-        echo "FAIL: could not build the cache (python exit $rc)" >&2
-        exit 1
-    }
-    echo "filesystem will not hold an out-of-range mtime; skipping" >&2
-    exit 77
-}
-
-proxytrack --convert "$dir/out.arc" "$dir/in.zip" >/dev/null 2>&1 || {
-    echo "FAIL: proxytrack failed on a cache with an out-of-range mtime" >&2
-    exit 1
-}
-
-# field 3 of the filedesc line is YYYYMMDDhhmmss; the epoch stands in for
-# "unknown", and 14 digits with a real day is the point (00 is what broke)
-date=$(head -n1 "$dir/out.arc" | awk '{ print $3 }')
-test "$date" = 19700101000000 || {
-    echo "FAIL: filedesc date is '$date', expected 19700101000000" >&2
-    exit 1
-}

tests/88_local-sitemap.test

@@ -0,0 +1,134 @@
+#!/bin/bash
+#
+# --sitemap seeds the crawl from robots.txt -> sitemapindex -> gzipped urlset.
+# start.html links to nothing, so orphan*.html can only arrive through the
+# sitemap; deep1.html proves the seeds keep a full depth budget under -r2, and
+# the off-host page and child sitemap must both be refused.
+
+set -eu
+
+: "${top_srcdir:=..}"
+
+crawl() { bash "$top_srcdir/tests/local-crawl.sh" "$@"; }
+
+# robots.txt Sitemap: -> index -> .xml.gz, seeds behaving like -r2 seeds. The
+# log assertions pin which route was taken: the two documents are served from
+# both /sitemapdir/index.xml and the well-known /sitemap.xml, so "some sitemap
+# was read" would pass either way. --not-found pins ingestion-only: the sitemap
+# documents feed the crawl but never land in the mirror.
+# --rerun also walks the update path over a mirror holding sitemap seeds.
+crawl --errors 0 --rerun \
+    --found 'sitemapdir/orphan1.html' \
+    --found 'sitemapdir/orphan2.html' \
+    --found 'sitemapdir/deep1.html' \
+    --not-found 'sitemapdir/index.xml' \
+    --not-found 'sitemapdir/pages.xml.gz' \
+    --log-found '2 of 3 URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
+    --log-found '1 of 2 child sitemap\(s\) listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \
+    --log-found 'ignoring off-host child sitemap' \
+    --log-not-found '/sitemap\.xml' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2
+
+# Negative control: without the option nothing but the start page is reached.
+crawl --errors 0 \
+    --found 'sitemapdir/start.html' \
+    --not-found 'sitemapdir/orphan1.html' \
+    --not-found 'sitemapdir/orphan2.html' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2
+
+# Sitemap URLs are not a filter bypass: a -*orphan2* rule still rejects one.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --not-found 'sitemapdir/orphan2.html' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 '-*orphan2*'
+
+# A robots.txt naming no sitemap falls back to the well-known /sitemap.xml.
+# The test server drops its Sitemap: record for this User-Agent.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemap\.xml' \
+    --log-not-found 'sitemapdir/index\.xml' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -F 'nositemap-agent'
+
+# --sitemap-url alone reads the named document and probes nothing.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --found 'sitemapdir/orphan2.html' \
+    --log-not-found 'sitemapdir/index\.xml' \
+    --log-not-found '/sitemap\.xml' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/pages.xml.gz'
+
+# The two options are additive: an unfetchable --sitemap-url is parsed as an
+# empty document and does not stop --sitemap from seeding the crawl.
+crawl --errors-content 1 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-found 'Sitemap: 0 of 0 URL\(s\) added' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/missing.xml'
+
+# The sitemapindex nesting cap stops the chain before its deepest urlset. The
+# positive control is the page one level inside the cap: without it, a cap
+# mutated to 0 (nothing followed) would pass this just as well.
+crawl --errors 0 \
+    --found 'sitemapdir/cap3.html' \
+    --not-found 'sitemapdir/cap4.html' \
+    --log-found 'Sitemap: cap reached' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/chain0.xml'
+
+# A site cannot widen a subtree crawl by putting its sitemap at the root: the
+# page below the start directory is seeded, the one above it is refused.
+crawl --errors 0 \
+    --found 'deep/dir/below.html' \
+    --not-found 'elsewhere/updir.html' \
+    httrack 'BASEURL/deep/dir/start.html' --sitemap -r3 -F 'scopesitemap-agent'
+
+# How far a sitemap fetch is gated depends on who asked for it, so the three
+# cases below have to differ: one "robots is honoured" assertion would hide a
+# regression in either direction. The harness disables robots, hence -s2 here.
+
+# We guessed /sitemap.xml, so a Disallow on it wins.
+crawl --errors 0 \
+    --not-found 'sitemapdir/orphan1.html' \
+    --log-found 'Sitemap: robots.txt forbids' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
+    -F 'denysitemap-agent'
+
+# The site declared it through a Sitemap: line, which the same Disallow does
+# not retract.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-not-found 'Sitemap: robots.txt forbids' \
+    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
+    -F 'denydeclared-agent'
+
+# The user named it, which is the same intent as a start URL: not refused
+# either, even though robots.txt forbids that exact path.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-not-found 'Sitemap: robots.txt forbids' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 -s2 -F 'denysitemap-agent' \
+    --sitemap-url 'BASEURL/sitemap.xml'
+
+# A child sitemap is a fetch like any other: a filter rejecting it stops the
+# whole subtree, so the page only it lists is never reached.
+crawl --errors 0 \
+    --not-found 'sitemapdir/gated.html' \
+    --log-found 'Sitemap: filter rule #[0-9]+ refuses' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 '-*filtered.xml*' \
+    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
+
+# Control: without that filter the same chain does reach the page.
+crawl --errors 0 \
+    --found 'sitemapdir/gated.html' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
+
+# A redirected sitemap is still ingested: the marking follows the 301.
+crawl --errors 0 \
+    --found 'sitemapdir/orphan1.html' \
+    --log-found 'moved\.xml redirects to .*pages\.xml\.gz' \
+    --log-found 'URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
+    httrack 'BASEURL/sitemapdir/start.html' -r2 \
+    --sitemap-url 'BASEURL/sitemapdir/moved.xml'

tests/89_webhttrack-error-overflow.test

@@ -1,174 +0,0 @@
-#!/bin/bash
-#
-# The "save settings" failure messages quote a project path the request body
-# supplies, so composing them must clip instead of running off a fixed buffer.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-command -v python3 >/dev/null || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-# ERROR_MESSAGE_MAX - 1 in htsserver.c.
-msgmax=1023
-
-srv=
-log=$(mktemp)
-# Physical path: macOS resolves TMPDIR through /var -> private/var, and the
-# paths built below sit within a few bytes of its 1024-byte PATH_MAX.
-base=$(cd "$(mktemp -d)" && pwd -P)
-cleanup() {
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    rm -f "${log}"
-    rm -rf "${base}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-freeport() {
-    python3 -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()'
-}
-
-# Echo the announced URL. Runs in a command substitution, so it is a
-# subshell and cannot export the pid: the caller reads it back with srvpid.
-start() {
-    local port url
-    port=$(freeport)
-    : >"${log}"
-    (
-        trap '' TERM TTOU XFSZ
-        # Caps regular-file writes, which is how the failing-write branch below
-        # is reached without /dev/full; the announce rides a pipe, exempt.
-        ulimit -f 8
-        exec htsserver "${distdir}/" --port "${port}" 2>&1
-    ) | cat >"${log}" &
-    for _ in $(seq 1 40); do
-        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
-        sleep 0.25
-    done
-    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
-    echo "${url}"
-}
-
-srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
-
-alive() { kill -0 "$1" 2>/dev/null; }
-
-portof() { echo "${1##*:}" | tr -d /; }
-
-# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
-# $2 is empty. Prints the reply.
-request() {
-    python3 -c 'import socket, sys
-port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
-if path:
-    req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
-else:
-    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
-           "Content-type: application/x-www-form-urlencoded\r\n"
-           "Content-length: %d\r\n\r\n%s" % (len(body), body))
-s = socket.create_connection(("127.0.0.1", port), 10)
-s.settimeout(30)
-s.sendall(req.encode())
-out = b""
-while True:
-    b = s.recv(65536)
-    if not b:
-        break
-    out += b
-s.close()
-sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
-}
-
-get() { request "$1" "$2" ""; }
-post() { request "$1" "" "$2"; }
-
-# Echo a path under the root $1 exactly $2 bytes long, in NAME_MAX-sized parts.
-padpath() {
-    local root=$1 want=$2 p=$1 last
-    while test $((${#p} + 203)) -lt "${want}"; do
-        p="${p}/$(printf '%0200d' 0)"
-    done
-    last=$((want - ${#p} - 1))
-    test "${last}" -ge 1 || fail "cannot build a ${want}-byte path under ${root}"
-    printf '%s/%0*d' "${p}" "${last}" 0
-}
-
-# Fail unless the error page renders the message opening with $1 about project
-# path $2, clipped to msgmax. error.html puts ${error} alone on its own line.
-check_clipped() {
-    local prefix=$1 full=$((${#1} + ${#2})) page line shown
-    test "${full}" -gt "${msgmax}" ||
-        fail "a ${full}-byte message fits in ${msgmax}: it cannot show a clip"
-    page=$(get "${port}" /server/error.html | tr -d '\r')
-    line=$(printf '%s\n' "${page}" | grep "^${prefix}") || {
-        shown=$(printf '%s\n' "${page}" | grep '^Unable to ' | cut -c1-80 || true)
-        fail "the error page reports '${shown:-nothing}', want: ${prefix}"
-    }
-    test "${#line}" -eq "${msgmax}" ||
-        fail "the message rendered ${#line} bytes, want ${full} clipped to ${msgmax}"
-}
-
-url=$(start)
-port=$(portof "${url}")
-srv=$(srvpid)
-test -n "${srv}" || fail "htsserver did not report its pid"
-
-# Every request body is gated by the session id (78_webhttrack-sid.test).
-sid=$(get "${port}" /server/index.html |
-    sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
-test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
-
-save() {
-    post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=$2&path=$1&projname=$3"
-}
-
-# A project path too long to be created at all.
-path=$(padpath "${base}/nodir" 1098)
-save "${path}" x p >/dev/null
-alive "${srv}" ||
-    fail "an over-long project path crashed the server: $(cat "${log}")"
-check_clipped "Unable to create the directory structure in " "${path}/p"
-
-# A creatable project path whose hts-cache is not a directory, so the init file
-# under it cannot be opened.
-path=$(padpath "${base}/isdir" 993)
-mkdir -p "${path}/p"
-ln -s /dev/null "${path}/p/hts-cache"
-save "${path}" x p >/dev/null
-alive "${srv}" || fail "an unwritable init file crashed the server: $(cat "${log}")"
-check_clipped "Unable to create the init file " "${path}/p"
-
-# The init file opens, then the write fails: the profile is past both stdio's
-# buffer, so fwrite() reaches the descriptor, and the server's file-size limit.
-path=$(padpath "${base}/full" 993)
-mkdir -p "${path}/p/hts-cache"
-profile=$(printf '%01024d' 0)
-profile=${profile}${profile}${profile}${profile}
-profile=${profile}${profile}${profile}${profile}
-test "${#profile}" -eq 16384 || fail "built a ${#profile}-byte profile, want 16384"
-save "${path}" "${profile}" p >/dev/null
-alive "${srv}" || fail "a short init file write crashed the server: $(cat "${log}")"
-written=$(wc -c <"${path}/p/hts-cache/winprofile.ini" | tr -d ' ')
-test "${written}" -lt "${#profile}" ||
-    fail "the file-size limit did not stop the write (${written} bytes landed)"
-check_clipped "Unable to write ${#profile} bytes in the the init file " "${path}/p"
-
-get "${port}" /server/index.html | grep -q '200 OK' ||
-    fail "the server stopped answering after the refused saves"
-
-echo "PASS"

tests/90_webhttrack-checkbox-clear.test

@@ -1,195 +0,0 @@
-#!/bin/bash
-#
-# An unchecked box posts nothing and the stored "1" survives: hence a hidden
-# companion per box, plus a disabling flag for the default-on ones.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-python=$(find_python) || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
-srvlog=$(mktemp)
-srv=
-srvpid=
-cleanup() {
-    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
-    test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
-    rm -rf "${work}" "${srvlog}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
-sport=$("${python}" -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()')
-(
-    trap '' TERM TTOU
-    export HOME="${work}"
-    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
-) &
-srv=$!
-for _ in $(seq 1 40); do
-    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
-    kill -0 "${srv}" 2>/dev/null || break
-    sleep 0.25
-done
-test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
-srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
-
-"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
-import glob, os, re, sys, urllib.parse, urllib.request
-
-url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
-rc = 0
-
-
-def check(ok, what):
-    global rc
-    print(("ok: " if ok else "FAIL: ") + what)
-    if not ok:
-        rc = 1
-
-
-# cache/cache2 are left to the separate rework of the dead "Cache" seed.
-skip = {"cache", "cache2"}
-page_of = {}
-boxes = 0
-for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
-    page = open(path, "rb").read().decode("latin-1")
-    cleared = dict((m.group(1), m.start()) for m in
-                   re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
-    for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
-        boxes += 1
-        if m.group(1) in skip:
-            continue
-        page_of[m.group(1)] = os.path.basename(path)
-        at = cleared.get(m.group(1))
-        check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
-              % (os.path.basename(path), m.group(1)))
-# Control: a regex that stopped matching would leave nothing to assert on.
-check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
-
-sid = None
-
-
-def get(path):
-    # The UI is served ISO-8859-1, so decode, do not assume UTF-8.
-    return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
-
-
-def post(fields):
-    global sid
-    if sid is None:
-        m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
-        if m is None:
-            sys.exit("no session id in server/index.html")
-        sid = m.group(1)
-    body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
-                    [("sid", sid)] + fields)
-    req = urllib.request.Request(url + "/server/step4.html",
-                                 data=body.encode("latin-1"), method="POST")
-    return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
-
-
-def textarea(page, name):
-    m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
-    if m is None:
-        sys.exit("no %s textarea in the rendered step4.html" % name)
-    return m.group(1)
-
-
-def checked(page, name):
-    return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
-
-
-# What each state puts on the command line (None: nothing) and its profile key.
-# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
-# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
-BOXES = [
-    # field           profile key           set                          cleared
-    ("parseall", "ParseAll", "--near", None),
-    ("link", "Near", "--test", None),
-    ("testall", "Test", "--extended-parsing", None),
-    # htmlfirst emits --priority=7, which the scan-priority list also emits.
-    ("htmlfirst", "HTMLFirst", None, None),
-    ("errpage", "NoErrorPages", "--generate-errors=0", None),
-    ("external", "NoExternalPages", "--replace-external", None),
-    ("hidepwd", "NoPwdInPages", "--disable-passwords", None),
-    ("hidequery", "NoQueryStrings", "--include-query-string=0", None),
-    ("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
-    ("windebug", None, "--debug-headers", None),
-    ("ka", "KeepAlive", "--keep-alive", None),
-    ("remt", "RemoveTimeout", "--host-control=1", None),
-    ("rems", "RemoveRateout", "--host-control=2", None),
-    ("cookies", "Cookies", None, "--cookies=0"),
-    ("parsejava", "ParseJava", None, "--parse-java=0"),
-    ("updhack", "UpdateHack", "--updatehack", None),
-    ("urlhack", "URLHack", "--urlhack", None),
-    ("keepwww", "KeepWww", "--keep-www-prefix", None),
-    ("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
-    ("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
-    ("toler", "TolerantRequests", "--tolerant", None),
-    ("http10", "HTTP10", "--http-10", None),
-    ("warc", "Warc", "--warc", None),
-    ("norecatch", "NoRecatch", "--do-not-recatch", None),
-    ("logf", "Log", "--single-log", None),
-    ("index", "Index", None, None),
-    ("index2", "WordIndex", "--search-index", None),
-    ("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
-]
-
-for field, key, when_set, when_clear in BOXES:
-    for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
-                                          ("", when_clear, when_set, "0")):
-        rendered = post([(field, value)])
-        cmd = textarea(rendered, "command").split()
-        label = "%s %s" % (field, "set" if value else "cleared")
-        if want:
-            check(want in cmd, "%s emits %s" % (label, want))
-        if unwanted:
-            check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
-        if key:
-            # The Windows GUI reads the same option out of the profile.
-            check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
-                  "%s writes %s=%s" % (label, key, stored))
-        check(checked(page_of[field], field) == bool(value),
-              "%s draws %s box" % (label, "a ticked" if value else "an empty"))
-missing = sorted(set(page_of) - set(b[0] for b in BOXES))
-check(not missing, "every checkbox is exercised (missing %s)" % missing)
-
-# A browser posts the companion and the ticked box under one name, so the fix
-# rests on the body loop keeping the last value; both orders pin that down.
-cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
-check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
-check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
-cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
-check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
-check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
-
-sys.exit(rc)
-PY
-
-# A leaked htsserver wedges the parallel harness behind a green log.
-cleanup
-! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
-
-echo "PASS"

tests/91_webhttrack-directory.test

@@ -1,181 +0,0 @@
-#!/bin/bash
-#
-# fopen() succeeds on a directory on POSIX and reading one never reaches EOF; a
-# FIFO blocks in fopen() instead. Either used to wedge the single-threaded server.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-command -v python3 >/dev/null || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-srv=
-log=$(mktemp)
-base=$(mktemp -d)
-cleanup() {
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    rm -f "${log}"
-    rm -rf "${base}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-# First line only. A "| head -1" would close the pipe early and, under pipefail,
-# SIGPIPE the producer into a spurious failure.
-firstline() { echo "${1%%$'\n'*}"; }
-
-freeport() {
-    python3 -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()'
-}
-
-# Echo the announced URL. Runs in a command substitution, so it is a
-# subshell and cannot export the pid: the caller reads it back with srvpid.
-start() {
-    local port url
-    port=$(freeport)
-    : >"${log}"
-    (
-        trap '' TERM TTOU
-        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
-    ) &
-    for _ in $(seq 1 40); do
-        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
-        sleep 0.25
-    done
-    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
-    echo "${url}"
-}
-
-# The server reports its own pid; the aliveness assertion below hangs off it.
-srvpid() { firstline "$(sed -n 's/^PID=//p' "${log}")"; }
-
-alive() { kill -0 "$1" 2>/dev/null; }
-
-portof() { echo "${1##*:}" | tr -d /; }
-
-# GET the path $2 from 127.0.0.1:$1, or POST the body $3 to / when $2 is empty.
-# The socket timeout is what makes an unfixed tree fail rather than wedge CI.
-request() {
-    python3 -c 'import socket, sys
-port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
-if path:
-    req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
-else:
-    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
-           "Content-type: application/x-www-form-urlencoded\r\n"
-           "Content-length: %d\r\n\r\n%s" % (len(body), body))
-s = socket.create_connection(("127.0.0.1", port), 10)
-s.settimeout(15)
-s.sendall(req.encode())
-out = b""
-try:
-    while True:
-        b = s.recv(65536)
-        if not b:
-            break
-        out += b
-except socket.timeout:
-    sys.stderr.write("timed out after %d bytes\n" % len(out))
-    sys.exit(9)
-s.close()
-sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
-}
-
-# GET $2, or fail with $3 when the reply never comes.
-get() {
-    local out
-    out=$(request "$1" "$2" "") || fail "$2 did not answer: $3"
-    echo "${out}"
-}
-
-post() { request "$1" "" "$2"; }
-
-url=$(start)
-port=$(portof "${url}")
-srv=$(srvpid)
-test -n "${srv}" || fail "htsserver did not report its pid"
-
-# Needs no session id and no project: html/server is a directory of the install.
-reply=$(get "${port}" /server/ "an unauthenticated request wedged the server")
-case "${reply}" in
-"HTTP/1.0 404 "*) ;;
-*) fail "a GUI directory did not answer 404: $(firstline "${reply}")" ;;
-esac
-
-# Every request body is gated by the session id.
-reply=$(get "${port}" /server/index.html "the GUI is not served")
-sid=$(firstline "$(echo "${reply}" |
-    sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p')")
-test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
-
-mkdir -p "${base}/proj/sub"
-echo "PROBEMARKER" >"${base}/proj/probe.txt"
-
-# mkfifo and ln -s may not work on a Windows checkout, so both stay optional.
-refuse=(/website/ /website/sub/)
-if mkfifo "${base}/proj/fifo.txt" 2>/dev/null; then
-    refuse+=(/website/fifo.txt)
-fi
-ln -s probe.txt "${base}/proj/link.txt" 2>/dev/null || true
-
-# step4's "save settings" flow registers the project without crawling, and that
-# is what arms the /website/ root.
-body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
-body="${body}&path=${base}&projname=proj"
-post "${port}" "${body}" >/dev/null || fail "the save request did not answer"
-test -f "${base}/proj/hts-cache/winprofile.ini" ||
-    fail "the project was not registered: $(cat "${log}")"
-
-# Positive control: every assertion below would pass on a server serving nothing.
-served() {
-    case "$(get "$1" "$2" "$3")" in
-    *PROBEMARKER*) ;;
-    *) fail "$3" ;;
-    esac
-}
-served "${port}" /website/probe.txt "the registered project's mirror is not served"
-if test -L "${base}/proj/link.txt"; then
-    # The guard stats rather than lstats, so a symlinked mirror file still serves.
-    served "${port}" /website/link.txt "a symlink to a mirror file is not served"
-fi
-
-# /website/ is where the GUI's own "browse mirrored site" link points.
-for path in "${refuse[@]}"; do
-    reply=$(get "${port}" "${path}" "the server wedged on ${path}")
-    case "${reply}" in
-    "HTTP/1.0 404 "*) ;;
-    *) fail "${path} did not answer 404: $(firstline "${reply}")" ;;
-    esac
-done
-
-# The accept loop is single-threaded: one spin denies every later request.
-alive "${srv}" || fail "the server died: $(cat "${log}")"
-served "${port}" /website/probe.txt "the server stopped answering after a directory request"
-
-# This fopen() is reached before any guard, so its read loop must stop on ferror().
-mkdir -p "${base}/proj2/hts-cache/winprofile.ini"
-reply=$(post "${port}" "sid=${sid}&path=${base}&loadprojname=proj2") ||
-    fail "a directory winprofile.ini wedged the server"
-case "${reply}" in
-"HTTP/1.0 3"*) ;;
-*) fail "loading a project did not redirect: $(firstline "${reply}")" ;;
-esac
-
-alive "${srv}" || fail "the server died: $(cat "${log}")"
-served "${port}" /website/probe.txt "the server stopped answering after a project load"
-
-echo "PASS"

tests/92_local-proxytrack-ndx-fields.test

@@ -1,37 +0,0 @@
-#!/bin/bash
-# The two halves of an .ndx entry's URL share one buffer, so the second must be
-# bounded by what the first left. The sanitizer CI build turns a regression
-# here into a hard stack-buffer-overflow.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-python=$(find_python) || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-dir=$(mktemp -d)
-trap 'rm -rf "$dir"' EXIT
-
-# each field fills binput's own bound, so together they run one past line[]
-"$python" - "$dir/foo.ndx" <<'EOF'
-import sys
-
-with open(sys.argv[1], "w") as f:
-    f.write("CACHE-1.4\n")
-    f.write("Wed, 01 Jan 2025 00:00:00 GMT\n")
-    f.write("h" * 1024 + "\n" + "f" * 1024 + "\n" + "0\n")
-    # a trailing entry keeps the walk inside the buffer, so this exercises the
-    # field bound alone
-    f.write("tail/entry\nx\n1\n")
-EOF
-: >"$dir/foo.dat" # the reader only proceeds when the sibling .dat opens
-
-proxytrack --convert "$dir/out.arc" "$dir/foo.ndx" >/dev/null 2>&1 || {
-    echo "FAIL: proxytrack crashed on an .ndx with two full-length URL fields" >&2
-    exit 1
-}

tests/Makefile.am

@@ -90,6 +90,7 @@ TESTS = \
 	01_engine-xfread.test \
 	01_zlib-acceptencoding.test \
 	01_zlib-warc.test \
+	01_zlib-sitemap.test \
 	01_zlib-warc-cdx.test \
 	01_zlib-warc-wacz.test \
 	01_zlib-contentcodings.test \
@@ -185,10 +186,6 @@ TESTS = \
 	85_webhttrack-projpath.test \
 	86_local-proxytrack-cache-longfields.test \
 	87_local-proxytrack-nodate.test \
-	88_local-proxytrack-badmtime.test \
-	89_webhttrack-error-overflow.test \
-	90_webhttrack-checkbox-clear.test \
-	91_webhttrack-directory.test \
-	92_local-proxytrack-ndx-fields.test
+	88_local-sitemap.test
 
 CLEANFILES = check-network_sh.cache

tests/local-server.py

@@ -18,6 +18,7 @@ import base64
 import gzip
 import hashlib
 import os
+import re
 import sys
 import time
 from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
@@ -542,8 +543,36 @@ class Handler(SimpleHTTPRequestHandler):
             return self.fail_cookie(name)
         self.send_html("\tThis is the secret.")
 
+    # A User-Agent carrying NO_SITEMAP_UA gets a robots.txt with no Sitemap:
+    # record, so a test can drive the /sitemap.xml fallback instead.
+    NO_SITEMAP_UA = "nositemap"
+    # ... and one that additionally Disallows the well-known location, so the
+    # fallback has to be refused by the rules this very body carries.
+    DENY_SITEMAP_UA = "denysitemap"
+    # ... the same Disallow, but with the sitemap also declared: the
+    # declaration is the site inviting the fetch and must win.
+    DENY_DECLARED_UA = "denydeclared"
+    # ... and one that points the sitemap at the site root, to check a subtree
+    # crawl is not widened by where the site chooses to put its sitemap.
+    SCOPE_SITEMAP_UA = "scopesitemap"
+
     def route_robots(self):
-        body = b"User-agent: *\nDisallow:\n"
+        # The Sitemap: record is group-independent; only --sitemap acts on it.
+        ua = self.headers.get("User-Agent") or ""
+        host = self.headers.get("Host")
+        body = "User-agent: *\nDisallow:\n"
+        if self.DENY_DECLARED_UA in ua:
+            body = (
+                "User-agent: *\nDisallow: /sitemap.xml\n"
+                f"Sitemap: http://{host}/sitemap.xml\n"
+            )
+        elif self.DENY_SITEMAP_UA in ua:
+            body = "User-agent: *\nDisallow: /sitemap.xml\n"
+        elif self.SCOPE_SITEMAP_UA in ua:
+            body += f"Sitemap: http://{host}/scopesitemap.xml\n"
+        elif self.NO_SITEMAP_UA not in ua:
+            body += f"Sitemap: http://{host}/sitemapdir/index.xml\n"
+        body = body.encode()
         self.send_response(200)
         self.send_header("Content-Type", "text/plain")
         self.send_header("Content-Length", str(len(body)))
@@ -551,6 +580,132 @@ class Handler(SimpleHTTPRequestHandler):
         if self.command != "HEAD":
             self.wfile.write(body)
 
+    # --- sitemap ingestion (issue #712) ------------------------------------
+    # start.html links to nothing, so orphan*.html are reachable only through
+    # the sitemap. deep1.html proves the seeds keep a full depth budget; the
+    # off-host page <loc> must be dropped by the travel scope, and the off-host
+    # child sitemap by the ingester's same-host rule. The index is served both
+    # from /sitemapdir/ (named by robots.txt) and from the well-known
+    # /sitemap.xml (the fallback).
+
+    def route_sitemap_index(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?>\n'
+            '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
+            f"<sitemap><loc>http://{host}/sitemapdir/pages.xml.gz</loc></sitemap>"
+            "<sitemap><loc>http://sitemap-offhost.invalid/s.xml</loc></sitemap>"
+            "</sitemapindex>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_pages(self):
+        host = self.headers.get("Host")
+        xml = (
+            '<?xml version="1.0" encoding="UTF-8"?>\n'
+            '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
+            f"<url><loc>http://{host}/sitemapdir/orphan1.html</loc></url>"
+            "<url><loc>http://sitemap-offhost.invalid/x.html</loc></url>"
+            f"<url><loc>http://{host}/sitemapdir/orphan2.html</loc></url>"
+            "</urlset>\n"
+        ).encode()
+        self.send_raw(gzip.compress(xml), "application/x-gzip")
+
+    def route_sitemap_start(self):
+        self.send_html("\tNothing links to the sitemap pages.")
+
+    def route_sitemap_orphan1(self):
+        self.send_html('\t<a href="deep1.html">deeper</a>')
+
+    def route_sitemap_orphan2(self):
+        self.send_html("\tSecond orphan.")
+
+    def route_sitemap_deep1(self):
+        self.send_html("\tOne level below an orphan.")
+
+    # chainN is a sitemapindex at nesting level N, listing chain(N+1) and a
+    # urlset capN.xml whose single page is capN.html. Levels up to
+    # HTS_SITEMAP_MAX_LEVEL are followed, so capN.html appears for N below the
+    # cap and stops appearing at it: the pair pins the boundary, which a cap
+    # mutated either way would break.
+    def route_sitemap_chain(self):
+        host = self.headers.get("Host")
+        level = int(self.path.rsplit("/", 1)[-1][len("chain") : -len(".xml")])
+        self.send_raw(
+            (
+                '<?xml version="1.0" encoding="UTF-8"?>\n<sitemapindex>'
+                f"<sitemap><loc>http://{host}/sitemapdir/chain{level + 1}.xml"
+                "</loc></sitemap>"
+                f"<sitemap><loc>http://{host}/sitemapdir/cap{level}.xml"
+                "</loc></sitemap></sitemapindex>\n"
+            ).encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_capset(self):
+        host = self.headers.get("Host")
+        level = self.path.rsplit("/", 1)[-1][len("cap") : -len(".xml")]
+        self.send_raw(
+            (
+                '<?xml version="1.0" encoding="UTF-8"?>\n<urlset>'
+                f"<url><loc>http://{host}/sitemapdir/cap{level}.html</loc></url>"
+                "</urlset>\n"
+            ).encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_cappage(self):
+        self.send_html("\tReached through a nested sitemapindex.")
+
+    def route_sitemap_gatedindex(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?><sitemapindex>'
+            f"<sitemap><loc>http://{host}/sitemapdir/filtered.xml</loc></sitemap>"
+            "</sitemapindex>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_filtered(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
+            f"<url><loc>http://{host}/sitemapdir/gated.html</loc></url>"
+            "</urlset>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_gated(self):
+        self.send_html("\tListed only by the filtered child sitemap.")
+
+    # A moved sitemap: the marking has to follow the redirect.
+    # A root sitemap naming a page below the crawl's start directory and one
+    # above it. Only the first may be seeded when the crawl started at /deep/dir/.
+    def route_sitemap_scope(self):
+        host = self.headers.get("Host")
+        self.send_raw(
+            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
+            f"<url><loc>http://{host}/deep/dir/below.html</loc></url>"
+            f"<url><loc>http://{host}/elsewhere/updir.html</loc></url>"
+            "</urlset>\n".encode(),
+            "application/xml",
+        )
+
+    def route_sitemap_deepstart(self):
+        self.send_html("\tA start page in a subdirectory, linking nothing.")
+
+    def route_sitemap_below(self):
+        self.send_html("\tBelow the start directory.")
+
+    def route_sitemap_updir(self):
+        self.send_html("\tAbove the start directory.")
+
+    def route_sitemap_moved(self):
+        self.send_response(301)
+        self.send_header("Location", "/sitemapdir/pages.xml.gz")
+        self.send_header("Content-Length", "0")
+        self.end_headers()
+
     # --- type/extension matrix (issue #267 family) -------------------------
 
     def send_raw(self, body, content_type, extra_headers=()):
@@ -1578,6 +1733,21 @@ class Handler(SimpleHTTPRequestHandler):
         "/gated/index.php": route_gated_index,
         "/gated/secret.php": route_gated_secret,
         "/robots.txt": route_robots,
+        "/sitemapdir/index.xml": route_sitemap_index,
+        "/sitemap.xml": route_sitemap_index,
+        "/sitemapdir/pages.xml.gz": route_sitemap_pages,
+        "/sitemapdir/start.html": route_sitemap_start,
+        "/sitemapdir/orphan1.html": route_sitemap_orphan1,
+        "/sitemapdir/orphan2.html": route_sitemap_orphan2,
+        "/sitemapdir/deep1.html": route_sitemap_deep1,
+        "/sitemapdir/gatedindex.xml": route_sitemap_gatedindex,
+        "/sitemapdir/filtered.xml": route_sitemap_filtered,
+        "/sitemapdir/gated.html": route_sitemap_gated,
+        "/sitemapdir/moved.xml": route_sitemap_moved,
+        "/scopesitemap.xml": route_sitemap_scope,
+        "/deep/dir/start.html": route_sitemap_deepstart,
+        "/deep/dir/below.html": route_sitemap_below,
+        "/elsewhere/updir.html": route_sitemap_updir,
         "/warcgz/index.html": route_warcgz_index,
         "/warcgz/page.html": route_warcgz_page,
         "/warcgz/data.bin": route_warcgz_data,
@@ -1902,6 +2072,13 @@ class Handler(SimpleHTTPRequestHandler):
             return True
         # Match percent-encoded paths (accented #157 route) by their decoded form.
         handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path))
+        if handler is None:
+            if re.fullmatch(r"/sitemapdir/chain\d+\.xml", path):
+                handler = type(self).route_sitemap_chain
+            elif re.fullmatch(r"/sitemapdir/cap\d+\.xml", path):
+                handler = type(self).route_sitemap_capset
+            elif re.fullmatch(r"/sitemapdir/cap\d+\.html", path):
+                handler = type(self).route_sitemap_cappage
         if handler is not None:
             handler(self)
             return True

tests/webhttrack-smoke.sh

@@ -46,12 +46,14 @@ cat >"$stubdir/x-www-browser" <<EOF
 echo "stub browser invoked with: \$1" >&2
 # Also fetch an option page and require a rendered title='' tooltip: proves the
 # option template expands and the \${html:} filter escapes into the attribute.
-# option9 additionally proves the WARC control renders with its expanded label.
+# option9/option8 additionally prove the WARC and sitemap controls render.
 opturl="\${1%/}/server/option2.html"
 warcurl="\${1%/}/server/option9.html"
+smurl="\${1%/}/server/option8.html"
 if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html &&
     opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" &&
-    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then
+    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC &&
+    sm="\$(curl -fsSL --max-time 20 "\$smurl")" && printf '%s' "\$sm" | grep -qaF 'name="sitemapurl"' && printf '%s' "\$sm" | grep -qaF 'name="sitemap"'; then
     echo PASS >"$marker"
 else
     echo "FAIL: unexpected response from \$1" >"$marker"