prototype: format the touched lines

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
prototype: drop the fragment and query htsparse writes after the mark
2026-07-27 19:12:54 +03:00 · 2026-07-27 07:47:44 +02:00 · 2026-07-27 07:47:11 +02:00 · 2026-07-27 07:39:07 +02:00 · 2026-07-27 07:35:37 +02:00
78 changed files with 346 additions and 2694 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -46,32 +46,10 @@ jobs:
      - name: Configure
        run: |
          set -euo pipefail
-          # Regenerate: configure and the Makefile.in's are not tracked.
+          # Regenerate from configure.ac/Makefile.am to validate them; the
+          # committed generated files already let a plain checkout build.
          autoreconf -fi
-          # Disabling zlib must fail here rather than at link with a pile of
-          # undefined minizip references (#735). Both spellings, so a rewrite
-          # cannot keep one and lose the other. Probed out-of-tree to leave
-          # nothing behind.
-          nozlib="$RUNNER_TEMP/nozlib"
-          for arg in --without-zlib --with-zlib=no; do
-            rm -rf "$nozlib" && mkdir -p "$nozlib"
-            if (cd "$nozlib" && "$GITHUB_WORKSPACE/configure" "$arg" >out.log 2>&1); then
-              echo "::error::configure $arg succeeded; it must be rejected"
-              exit 1
-            fi
-            # ... and for the stated reason, not an unrelated configure failure.
-            grep -q "zlib cannot be disabled" "$nozlib/out.log" \
-              || { cat "$nozlib/out.log"; exit 1; }
-          done
          ./configure
-          # Same dead end from the compile side. The bare compile is the
-          # control: without it a broken probe would pass vacuously.
-          hdr='#include "htsglobal.h"'
-          echo "$hdr" | $CC -I. -Isrc -fsyntax-only -xc -
-          if echo "$hdr" | $CC -DHTS_USEZLIB=0 -I. -Isrc -fsyntax-only -xc - 2>/dev/null; then
-            echo "::error::-DHTS_USEZLIB=0 compiled; htsglobal.h must reject it"
-            exit 1
-          fi
          # a missing decoder would silently drop the coding from Accept-Encoding
          grep -q "define HTS_USEBROTLI 1" config.h
          grep -q "define HTS_USEZSTD 1" config.h
--- a/configure.ac
+++ b/configure.ac
@@ -175,7 +175,7 @@ AX_CHECK_ALIGNED_ACCESS_REQUIRED
 # check for various headers
 AC_CHECK_HEADERS([execinfo.h sys/ioctl.h])

-### zlib (mandatory)
+### zlib
 CHECK_ZLIB()

 ### brotli and zstd content codings (optional)
--- a/fuzz/Makefile.am
+++ b/fuzz/Makefile.am
@@ -2,7 +2,7 @@
 if FUZZERS
 noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \
 	fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \
-	fuzz-htsparse fuzz-singlefile fuzz-sitemap
+	fuzz-htsparse fuzz-singlefile
 endif

 AM_CPPFLAGS = \
@@ -28,7 +28,6 @@ fuzz_header_SOURCES = fuzz-header.c fuzz.h
 fuzz_cachendx_SOURCES = fuzz-cachendx.c fuzz.h
 fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h
 fuzz_singlefile_SOURCES = fuzz-singlefile.c fuzz.h
-fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h

 # List corpus files explicitly: automake does not expand EXTRA_DIST globs.
 EXTRA_DIST = README.md run-fuzzers.sh \
@@ -53,6 +52,4 @@ EXTRA_DIST = README.md run-fuzzers.sh \
 	corpus/singlefile/img-src.html corpus/singlefile/link-rel.html \
 	corpus/singlefile/style-block.html corpus/singlefile/style-attr.html \
 	corpus/singlefile/srcset.html corpus/singlefile/rawtext.html \
-	corpus/singlefile/malformed.html corpus/singlefile/many-attrs.html \
-	corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
-	corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz
+	corpus/singlefile/malformed.html corpus/singlefile/many-attrs.html
--- a/fuzz/corpus/sitemap/sitemapindex.xml
+++ b/fuzz/corpus/sitemap/sitemapindex.xml
@@ -1 +0,0 @@
-<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc></sitemap></sitemapindex>
--- a/fuzz/corpus/sitemap/truncated.xml
+++ b/fuzz/corpus/sitemap/truncated.xml
@@ -1 +0,0 @@
-<urlset><loc>http://h.test/x
--- a/fuzz/corpus/sitemap/urlset.xml
+++ b/fuzz/corpus/sitemap/urlset.xml
@@ -1 +0,0 @@
-<?xml version="1.0"?><urlset><url><loc>http://h.test/a.html</loc></url><url><loc>https://h.test/b?x=1&amp;y=2</loc></url></urlset>
--- a/fuzz/corpus/sitemap/urlset.xml.gz
+++ b/fuzz/corpus/sitemap/urlset.xml.gz
--- a/fuzz/fuzz-sitemap.c
+++ b/fuzz/fuzz-sitemap.c
@@ -1,60 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 2026 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* Fuzz the sitemap <loc> scanner (htssitemap.c): raw XML, gzip-framed bodies
-   and truncated streams all arrive here straight off the network. */
-#include "fuzz.h"
-#include "htssitemap.h"
-
-static hts_boolean sm_count(void *arg, const char *url) {
-  int *const n = (int *) arg;
-
-  (void) url;
-  (*n)++;
-  return HTS_TRUE;
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
-  static const int caps[] = {0, 1, 16, HTS_SITEMAP_MAX_URLS_DOC};
-  hts_boolean is_index;
-  char *body;
-  int n = 0, cap;
-
-  if (size == 0)
-    return 0;
-  cap = caps[data[0] % (sizeof(caps) / sizeof(caps[0]))];
-  data++, size--;
-  /* A heap copy of exactly `size` bytes: the scanner must never rely on a
-     terminator, and ASan turns any overread into a report. */
-  body = malloct(size != 0 ? size : 1);
-  memcpy(body, data, size);
-
-  (void) hts_sitemap_scan(body, size, cap, &is_index, sm_count, &n);
-
-  freet(body);
-  return 0;
-}
--- a/html/cmdguide.html
+++ b/html/cmdguide.html
@@ -163,26 +163,8 @@ the index" problems disappear.</p>
  <tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
  <tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
  <tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
-  <tr><td><tt>--sitemap (-%m), --sitemap-url URL (-%mu)</tt></td><td>Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.</td></tr>
 </table>

-<p>Link-following only finds what something links to. Anything a site publishes
-solely in its sitemap is invisible to HTTrack unless you ask for it.
-<tt>--sitemap</tt> reads the start host's <tt>robots.txt</tt> for
-<tt>Sitemap:</tt> lines and falls back to <tt>/sitemap.xml</tt>;
-<tt>--sitemap-url</tt> names one directly. Nested <tt>sitemapindex</tt> files
-and gzipped <tt>.xml.gz</tt> sitemaps are followed. The URLs found become start
-URLs with the full depth budget, but they still go through your filters and
-scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is
-off by default because a sitemap can list thousands of pages nothing links
-to.</p>
-
-<p>One surprise worth knowing: a sitemap you name with <tt>--sitemap-url</tt>,
-and one the site itself declares in <tt>robots.txt</tt>, are fetched even when
-<tt>robots.txt</tt> disallows that path, because naming or declaring a sitemap
-is an invitation to read it. Only the guessed <tt>/sitemap.xml</tt> obeys a
-<tt>Disallow</tt>. The URLs listed inside are gated normally either way.</p>
-
 <p>The single most common surprise is "only the home page came down." That is
 usually not a scope option at all: it is an off-host redirect. A start URL of
 <tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
--- a/html/httrack.man.html
+++ b/html/httrack.man.html
@@ -87,8 +87,8 @@ offline browser : copy websites to a local directory</p>
 --host-control[=N]</b> ] [ <b>-%P,
 --extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
 --test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
-] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
-[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
+] [ <b>-NN, --structure[=N]</b> ] [ <b>-%N,
+--delayed-type-check</b> ] [ <b>-%D,
 --cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
 ] [ <b>-%Z, --single-file</b> ] [ <b>-LN,
 --long-names[=N]</b> ] [ <b>-KN, --keep-links[=N]</b> ] [
@@ -577,22 +577,6 @@ URL per line) (--list &lt;param&gt;)</p></td></tr>

 <p>&lt;file&gt; add all scan rules located in this text
 file (one scan rule per line) (--urllist &lt;param&gt;)</p></td></tr>
-<tr valign="top" align="left">
-<td width="9%"></td>
-<td width="4%">
-
-
-<p>-%m</p></td>
-<td width="5%"></td>
-<td width="82%">
-
-
-<p>seed the crawl from the site&rsquo;s sitemap (robots.txt
-Sitemap:, then /sitemap.xml); --sitemap-url URL names one
-explicitly. A sitemap you name, or one the site declares, is
-fetched even under robots.txt Disallow; only the guessed
-/sitemap.xml obeys it. The URLs found still pass every
-filter and scope rule (--sitemap)</p></td></tr>
 </table>

 <h3>Build options:
--- a/html/server/option8.html
+++ b/html/server/option8.html
@@ -108,7 +108,6 @@ ${do:end-if}
 <input type="hidden" name="keepqueryorder" value="">
 <input type="hidden" name="toler" value="">
 <input type="hidden" name="http10" value="">
-<input type="hidden" name="sitemap" value="">

 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
@@ -144,17 +143,6 @@ ${listid:robots:LISTDEF_8}
 </select>
 <br><br>

-<input type="checkbox" name="sitemap" ${checked:sitemap}
-			title='${html:LANG_SITEMAPTIP}' onMouseOver="info('${html:LANG_SITEMAPTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
-> ${LANG_SITEMAP}
-<br><br>
-
-${LANG_SITEMAPURL}
-<input name="sitemapurl" value="${sitemapurl}" size="40"
-			title='${html:LANG_SITEMAPURLTIP}' onMouseOver="info('${html:LANG_SITEMAPURLTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
->
-<br><br>
-
 <input type="checkbox" name="updhack" ${checked:updhack}
 			title='${html:LANG_I1k}' onMouseOver="info('${html:LANG_I1k}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I62b}
--- a/html/server/step2.html
+++ b/html/server/step2.html
@@ -141,8 +141,6 @@ ${do:copy:KeepSlashes:keepslashes}
 ${do:copy:KeepQueryOrder:keepqueryorder}
 ${do:copy:StripQuery:stripquery}
 ${do:copy:StoreAllInCache:cache2}
-${do:copy:Sitemap:sitemap}
-${do:copy:SitemapUrl:sitemapurl}
 ${do:copy:Warc:warc}
 ${do:copy:WarcFile:warcfile}
 ${do:copy:Changes:changes}
--- a/html/server/step4.html
+++ b/html/server/step4.html
@@ -188,8 +188,6 @@ ${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one *
 	${test:toler:--tolerant}
 	${test:http10:--http-10}
 	${test:cache2:--store-all-in-cache}
-	${test:sitemap:--sitemap}
-	${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"}
 	${test:warc:--warc}
 	${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
 	${test:changes:--changes}
@@ -245,8 +243,6 @@ KeepSlashes=${ztest:keepslashes:0:1}
 KeepQueryOrder=${ztest:keepqueryorder:0:1}
 StripQuery=${stripquery}
 StoreAllInCache=${ztest:cache2:0:1}
-Sitemap=${ztest:sitemap:0:1}
-SitemapUrl=${sitemapurl}
 Warc=${ztest:warc:0:1}
 WarcFile=${warcfile}
 Changes=${ztest:changes:0:1}
--- a/lang.def
+++ b/lang.def
@@ -1054,11 +1054,3 @@ LANG_SINGLEFILEMAX
 Largest inlined asset (bytes):
 LANG_SINGLEFILEMAXTIP
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
-LANG_SITEMAP
-Seed the crawl from the site's sitemap
-LANG_SITEMAPTIP
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-LANG_SITEMAPURL
-Sitemap address:
-LANG_SITEMAPURLTIP
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
--- a/lang/Bulgarian.txt
+++ b/lang/Bulgarian.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Íàé-ãîëÿì âãðàäåí ðåñóðñ (áàéòîâå):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Ðåñóðñ íàä òîçè ðàçìåð çàïàçâà îáèêíîâåíà âðúçêà; îñòàâåòå ïðàçíî çà ñòîéíîñòòà ïî ïîäðàçáèðàíå îò 10485760 áàéòà.
-Seed the crawl from the site's sitemap
-Çàïî÷âàíå íà îáõîæäàíåòî îò êàðòàòà íà ñàéòà
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Ïðî÷èòàíå íà êàðòàòà íà ñàéòà (ðåäîâåòå Sitemap: â robots.txt, ñëåä òîâà /sitemap.xml) è äîáàâÿíå íà âñåêè ïîñî÷åí URL àäðåñ êàòî íà÷àëåí.
-Sitemap address:
-Àäðåñ íà êàðòàòà íà ñàéòà:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Àäðåñ íà êàðòà íà ñàéòà, êîÿòî äà áúäå ïðî÷åòåíà âìåñòî ñîíäèðàíå íà ñàéòà; îñòàâåòå ïðàçíî, çà äà ñå ïðîâåðè robots.txt, ñëåä òîâà /sitemap.xml.
--- a/lang/Castellano.txt
+++ b/lang/Castellano.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Tamaño máximo del recurso incrustado (bytes):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Un recurso mayor que este tamaño conserva un enlace normal; déjelo vacío para el valor predeterminado de 10485760 bytes.
-Seed the crawl from the site's sitemap
-Iniciar el rastreo desde el mapa del sitio
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Leer el mapa del sitio (líneas Sitemap: de robots.txt, luego /sitemap.xml) y añadir como dirección inicial cada URL que incluya.
-Sitemap address:
-Dirección del mapa del sitio:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Dirección de un mapa del sitio que leer en lugar de sondear el sitio; déjelo vacío para sondear robots.txt y luego /sitemap.xml.
--- a/lang/Cesky.txt
+++ b/lang/Cesky.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Nejvìtší vložený zdroj (bajty):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Zdroj vìtší než tato velikost si ponechá bìžný odkaz; ponechte prázdné pro výchozí hodnotu 10485760 bajtù.
-Seed the crawl from the site's sitemap
-Zahájit procházení z mapy webu
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Naèíst mapu webu (øádky Sitemap: v souboru robots.txt, poté /sitemap.xml) a pøidat každou uvedenou adresu URL jako výchozí.
-Sitemap address:
-Adresa mapy webu:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresa mapy webu, která se má naèíst místo zjiš<69>ování na webu; ponechte prázdné pro zjištìní z robots.txt a poté /sitemap.xml.
--- a/lang/Chinese-BIG5.txt
+++ b/lang/Chinese-BIG5.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 內嵌資源大小上限（位元組）：
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 超過此大小的資源會保留一般連結；留空則使用預設的 10485760 位元組。
-Seed the crawl from the site's sitemap
-從網站的 Sitemap 開始擷取
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-讀取網站的 Sitemap（robots.txt 中的 Sitemap: 行，然後 /sitemap.xml），並將其中列出的每個網址加入為起始網址。
-Sitemap address:
-Sitemap 位址：
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-要讀取的 Sitemap 位址，用來取代自動探測；留空則先探測 robots.txt 再探測 /sitemap.xml。
--- a/lang/Chinese-Simplified.txt
+++ b/lang/Chinese-Simplified.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 内嵌资源大小上限（字节）：
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 超过此大小的资源会保留普通链接；留空则使用默认的 10485760 字节。
-Seed the crawl from the site's sitemap
-从网站的 Sitemap 开始抓取
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-读取网站的 Sitemap（robots.txt 中的 Sitemap: 行，然后 /sitemap.xml），并将其中列出的每个网址添加为起始网址。
-Sitemap address:
-Sitemap 地址：
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-要读取的 Sitemap 地址，用来代替自动探测；留空则先探测 robots.txt 再探测 /sitemap.xml。
--- a/lang/Croatian.txt
+++ b/lang/Croatian.txt
@@ -978,11 +978,3 @@ Largest inlined asset (bytes):
 Najveæi ugraðeni resurs (bajtovi):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Resurs veæi od ove velièine zadr¾ava obiènu poveznicu; ostavite prazno za zadanih 10485760 bajtova.
-Seed the crawl from the site's sitemap
-Pokreni pretra¾ivanje iz karte web-mjesta
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Proèitaj kartu web-mjesta (retke Sitemap: iz robots.txt, zatim /sitemap.xml) i dodaj svaki navedeni URL kao poèetnu adresu.
-Sitemap address:
-Adresa karte web-mjesta:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresa karte web-mjesta koju treba proèitati umjesto ispitivanja web-mjesta; ostavite prazno za ispitivanje robots.txt pa /sitemap.xml.
--- a/lang/Dansk.txt
+++ b/lang/Dansk.txt
@@ -1024,11 +1024,3 @@ Largest inlined asset (bytes):
 Største indlejrede ressource (byte):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 En ressource over denne størrelse beholder et almindeligt link; lad feltet stå tomt for standardværdien på 10485760 byte.
-Seed the crawl from the site's sitemap
-Start gennemgangen fra webstedets sitemap
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Læs webstedets sitemap (Sitemap:-linjer i robots.txt, derefter /sitemap.xml) og tilføj hver angivet URL som startadresse.
-Sitemap address:
-Sitemap-adresse:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adressen på et sitemap, der skal læses i stedet for at undersøge webstedet; lad feltet stå tomt for at undersøge robots.txt og derefter /sitemap.xml.
--- a/lang/Deutsch.txt
+++ b/lang/Deutsch.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Größte eingebettete Ressource (Bytes):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Eine Ressource über dieser Größe behält einen gewöhnlichen Link; leer lassen für den Standardwert von 10485760 Bytes.
-Seed the crawl from the site's sitemap
-Erfassung mit der Sitemap der Website beginnen
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Die Sitemap der Website lesen (Sitemap:-Zeilen in robots.txt, dann /sitemap.xml) und jede dort aufgeführte URL als Startadresse hinzufügen.
-Sitemap address:
-Sitemap-Adresse:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresse einer Sitemap, die anstelle der Suche auf der Website gelesen wird; leer lassen, um robots.txt und dann /sitemap.xml zu prüfen.
--- a/lang/Eesti.txt
+++ b/lang/Eesti.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Suurim manustatud ressurss (baiti):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Sellest suurem ressurss säilitab tavalise lingi; jäta tühjaks vaikeväärtuse 10485760 baiti jaoks.
-Seed the crawl from the site's sitemap
-Alusta kogumist saidi saidikaardist
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Loe saidi saidikaarti (robots.txt-i Sitemap:-read, seejärel /sitemap.xml) ja lisa iga seal loetletud URL alguslingina.
-Sitemap address:
-Saidikaardi aadress:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Saidikaardi aadress, mida lugeda saidi sondeerimise asemel; jäta tühjaks, et kontrollida robots.txt-i ja seejärel /sitemap.xml-i.
--- a/lang/English.txt
+++ b/lang/English.txt
@@ -1024,11 +1024,3 @@ Largest inlined asset (bytes):
 Largest inlined asset (bytes):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
-Seed the crawl from the site's sitemap
-Seed the crawl from the site's sitemap
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Sitemap address:
-Sitemap address:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
--- a/lang/Finnish.txt
+++ b/lang/Finnish.txt
@@ -978,11 +978,3 @@ Largest inlined asset (bytes):
 Suurin upotettu resurssi (tavua):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Tätä suurempi resurssi säilyttää tavallisen linkin; jätä tyhjäksi, jolloin käytetään oletusarvoa 10485760 tavua.
-Seed the crawl from the site's sitemap
-Aloita haku sivuston sivukartasta
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Lue sivuston sivukartta (robots.txt-tiedoston Sitemap:-rivit, sitten /sitemap.xml) ja lisää jokainen siinä lueteltu URL-osoite aloitusosoitteeksi.
-Sitemap address:
-Sivukartan osoite:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Luettavan sivukartan osoite sivuston luotaamisen sijaan; jätä tyhjäksi, jolloin tarkistetaan robots.txt ja sitten /sitemap.xml.
--- a/lang/Francais.txt
+++ b/lang/Francais.txt
@@ -1024,11 +1024,3 @@ Largest inlined asset (bytes):
 Taille maximale d'une ressource intégrée (octets) :
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Au-delà de cette taille, la ressource reste un lien ordinaire ; laissez vide pour la valeur par défaut de 10485760 octets.
-Seed the crawl from the site's sitemap
-Partir du plan de site (sitemap)
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Lire le plan de site (lignes Sitemap: de robots.txt, puis /sitemap.xml) et ajouter chaque URL listée comme adresse de départ.
-Sitemap address:
-Adresse du plan de site :
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresse d'un plan de site à lire au lieu de sonder le site ; laissez vide pour sonder robots.txt puis /sitemap.xml.
--- a/lang/Greek.txt
+++ b/lang/Greek.txt
@@ -978,11 +978,3 @@ Largest inlined asset (bytes):
 Μέγιστος ενσωματωμένος πόρος (byte):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Πόρος μεγαλύτερος από αυτό το μέγεθος διατηρεί κανονικό σύνδεσμο. Αφήστε το κενό για την προεπιλογή των 10485760 byte.
-Seed the crawl from the site's sitemap
-Έναρξη της ανίχνευσης από τον χάρτη του ιστότοπου
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Ανάγνωση του χάρτη του ιστότοπου (γραμμές Sitemap: στο robots.txt, έπειτα /sitemap.xml) και προσθήκη κάθε διεύθυνσης URL που περιέχει ως αρχικής διεύθυνσης.
-Sitemap address:
-Διεύθυνση χάρτη ιστότοπου:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Διεύθυνση χάρτη ιστότοπου προς ανάγνωση αντί για αναζήτηση στον ιστότοπο. Αφήστε το κενό για έλεγχο του robots.txt και έπειτα του /sitemap.xml.
--- a/lang/Italiano.txt
+++ b/lang/Italiano.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Dimensione massima della risorsa incorporata (byte):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Una risorsa oltre questa dimensione mantiene un collegamento normale; lasciare vuoto per il valore predefinito di 10485760 byte.
-Seed the crawl from the site's sitemap
-Avvia la scansione dalla mappa del sito
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Legge la mappa del sito (righe Sitemap: in robots.txt, poi /sitemap.xml) e aggiunge come indirizzo iniziale ogni URL elencato.
-Sitemap address:
-Indirizzo della mappa del sito:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Indirizzo di una mappa del sito da leggere invece di sondare il sito; lasciare vuoto per sondare robots.txt e poi /sitemap.xml.
--- a/lang/Japanese.txt
+++ b/lang/Japanese.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 埋め込む最大サイズ (バイト):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 このサイズを超えるリソースは通常のリンクのままになります。空欄にすると既定値の 10485760 バイトになります。
-Seed the crawl from the site's sitemap
-サイトマップからミラーリングを開始する
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-サイトマップ (robots.txt の Sitemap: 行、次に /sitemap.xml) を読み込み、記載されているすべての URL を開始アドレスとして追加します。
-Sitemap address:
-サイトマップのアドレス:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-サイトを探索する代わりに読み込むサイトマップのアドレス。空欄にすると robots.txt、次に /sitemap.xml を探索します。
--- a/lang/Macedonian.txt
+++ b/lang/Macedonian.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 ½ÐøÓÞÛÕÜ ÒÓàÐÔÕÝ àÕáãàá (ÑÐøâØ):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 ÀÕáãàá ßÞÓÞÛÕÜ ÞÔ ÞÒÐÐ ÓÞÛÕÜØÝÐ ×ÐÔàÖãÒÐ ÞÑØçÝÐ ÒàáÚÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð áâÐÝÔÐàÔÝØâÕ 10485760 ÑÐøâØ.
-Seed the crawl from the site's sitemap
-·ÐßÞçÝØ ÓÞ ßàÕÑÐàãÒÐúÕâÞ ÞÔ ÚÐàâÐâÐ ÝÐ áÐøâÞâ
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-¿àÞçØâÐø øÐ ÚÐàâÐâÐ ÝÐ áÐøâÞâ (àÕÔÞÒØâÕ Sitemap: ÒÞ robots.txt, ßÞâÞÐ /sitemap.xml) Ø ÔÞÔÐø øÐ áÕÚÞøÐ ÝÐÒÕÔÕÝÐ URL ÐÔàÕáÐ ÚÐÚÞ ßÞçÕâÝÐ.
-Sitemap address:
-°ÔàÕáÐ ÝÐ ÚÐàâÐâÐ ÝÐ áÐøâÞâ:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-°ÔàÕáÐ ÝÐ ÚÐàâÐ ÝÐ áÐøâÞâ èâÞ âàÕÑÐ ÔÐ áÕ ßàÞçØâÐ ÝÐÜÕáâÞ ØáßØâãÒÐúÕ ÝÐ áÐøâÞâ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÔÐ áÕ ØáßØâÐ robots.txt, ßÐ /sitemap.xml.
--- a/lang/Magyar.txt
+++ b/lang/Magyar.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Legnagyobb beágyazott erõforrás (bájt):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Az ennél nagyobb erõforrás közönséges hivatkozás marad; hagyja üresen a 10485760 bájtos alapértelmezéshez.
-Seed the crawl from the site's sitemap
-A letöltés indítása a webhely webhelytérképérõl
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-A webhely webhelytérképének beolvasása (a robots.txt Sitemap: sorai, majd a /sitemap.xml), és a benne felsorolt összes URL felvétele kiindulási címként.
-Sitemap address:
-Webhelytérkép címe:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-A webhely vizsgálata helyett beolvasandó webhelytérkép címe; hagyja üresen a robots.txt, majd a /sitemap.xml vizsgálatához.
--- a/lang/Nederlands.txt
+++ b/lang/Nederlands.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Grootste ingesloten bron (bytes):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Een bron boven deze grootte houdt een gewone koppeling; laat leeg voor de standaardwaarde van 10485760 bytes.
-Seed the crawl from the site's sitemap
-De crawl starten vanaf de sitemap van de site
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-De sitemap van de site lezen (Sitemap:-regels in robots.txt, daarna /sitemap.xml) en elke vermelde URL als startadres toevoegen.
-Sitemap address:
-Sitemap-adres:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adres van een sitemap die gelezen moet worden in plaats van de site te onderzoeken; laat leeg om robots.txt en daarna /sitemap.xml te controleren.
--- a/lang/Norsk.txt
+++ b/lang/Norsk.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Største innebygde ressurs (byte):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 En ressurs over denne størrelsen beholder en vanlig lenke; la feltet stå tomt for standardverdien på 10485760 byte.
-Seed the crawl from the site's sitemap
-Start gjennomgangen fra nettstedets nettstedskart
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Les nettstedets nettstedskart (Sitemap:-linjer i robots.txt, deretter /sitemap.xml) og legg til hver oppført URL som startadresse.
-Sitemap address:
-Adresse til nettstedskart:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adressen til et nettstedskart som skal leses i stedet for å undersøke nettstedet; la feltet stå tomt for å undersøke robots.txt og deretter /sitemap.xml.
--- a/lang/Polski.txt
+++ b/lang/Polski.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Najwiêkszy osadzony zasób (bajty):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Zasób wiêkszy ni¿ ten rozmiar zachowuje zwyk³y odno¶nik; pozostaw puste, aby u¿yæ domy¶lnych 10485760 bajtów.
-Seed the crawl from the site's sitemap
-Rozpocznij pobieranie od mapy witryny
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Odczytaj mapê witryny (wiersze Sitemap: w pliku robots.txt, nastêpnie /sitemap.xml) i dodaj ka¿dy wymieniony adres URL jako adres pocz±tkowy.
-Sitemap address:
-Adres mapy witryny:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adres mapy witryny do odczytania zamiast sondowania witryny; pozostaw puste, aby sprawdziæ robots.txt, a nastêpnie /sitemap.xml.
--- a/lang/Portugues-Brasil.txt
+++ b/lang/Portugues-Brasil.txt
@@ -1024,11 +1024,3 @@ Largest inlined asset (bytes):
 Maior recurso incorporado (bytes):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Um recurso acima desse tamanho mantém um link comum; deixe em branco para o padrão de 10485760 bytes.
-Seed the crawl from the site's sitemap
-Iniciar a captura pelo mapa do site
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Ler o mapa do site (linhas Sitemap: do robots.txt, depois /sitemap.xml) e adicionar como endereço inicial cada URL nele listada.
-Sitemap address:
-Endereço do mapa do site:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Endereço de um mapa do site a ser lido em vez de sondar o site; deixe em branco para sondar robots.txt e depois /sitemap.xml.
--- a/lang/Portugues.txt
+++ b/lang/Portugues.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Maior recurso incorporado (bytes):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Um recurso acima deste tamanho mantém uma ligação normal; deixe em branco para o valor predefinido de 10485760 bytes.
-Seed the crawl from the site's sitemap
-Iniciar a recolha pelo mapa do site
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Ler o mapa do site (linhas Sitemap: do robots.txt, depois /sitemap.xml) e adicionar como endereço inicial cada URL nele listado.
-Sitemap address:
-Endereço do mapa do site:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Endereço de um mapa do site a ler em vez de sondar o site; deixe em branco para sondar robots.txt e depois /sitemap.xml.
--- a/lang/Romanian.txt
+++ b/lang/Romanian.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Cea mai mare resursa încorporata (octeti):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 O resursa mai mare decât aceasta dimensiune pastreaza o legatura obisnuita; lasati gol pentru valoarea implicita de 10485760 de octeti.
-Seed the crawl from the site's sitemap
-Porneste explorarea de la harta sitului
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Citeste harta sitului (liniile Sitemap: din robots.txt, apoi /sitemap.xml) si adauga fiecare URL listat ca adresa de pornire.
-Sitemap address:
-Adresa hartii sitului:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresa unei harti a sitului care sa fie citita în loc de sondarea sitului; lasati gol pentru a sonda robots.txt, apoi /sitemap.xml.
--- a/lang/Russian.txt
+++ b/lang/Russian.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Íàèáîëüøèé âñòðàèâàåìûé ðåñóðñ (áàéòû):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Ðåñóðñ áîëüøå ýòîãî ðàçìåðà ñîõðàíÿåò îáû÷íóþ ññûëêó; îñòàâüòå ïóñòûì äëÿ çíà÷åíèÿ ïî óìîë÷àíèþ 10485760 áàéò.
-Seed the crawl from the site's sitemap
-Íà÷èíàòü îáõîä ñ êàðòû ñàéòà
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Ïðî÷èòàòü êàðòó ñàéòà (ñòðîêè Sitemap: â robots.txt, çàòåì /sitemap.xml) è äîáàâèòü êàæäûé óêàçàííûé â íåé URL êàê íà÷àëüíûé àäðåñ.
-Sitemap address:
-Àäðåñ êàðòû ñàéòà:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Àäðåñ êàðòû ñàéòà, êîòîðóþ íóæíî ïðî÷èòàòü âìåñòî îïðîñà ñàéòà; îñòàâüòå ïóñòûì, ÷òîáû ïðîâåðèòü robots.txt, çàòåì /sitemap.xml.
--- a/lang/Slovak.txt
+++ b/lang/Slovak.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Najväè¹í vlo¾ený zdroj (bajty):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Zdroj väè¹í ne¾ táto veµkos» si ponechá be¾ný odkaz; ponechajte prázdne pre predvolených 10485760 bajtov.
-Seed the crawl from the site's sitemap
-Zaèa» prehliadanie z mapy stránok
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Naèíta» mapu stránok (riadky Sitemap: v súbore robots.txt, potom /sitemap.xml) a prida» ka¾dú uvedenú adresu URL ako poèiatoènú.
-Sitemap address:
-Adresa mapy stránok:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresa mapy stránok, ktorá sa má naèíta» namiesto zis»ovania na stránke; ponechajte prázdne na zistenie z robots.txt a potom /sitemap.xml.
--- a/lang/Slovenian.txt
+++ b/lang/Slovenian.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Najvecji vgrajeni vir (bajti):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Vir, vecji od te velikosti, ohrani obicajno povezavo; pustite prazno za privzetih 10485760 bajtov.
-Seed the crawl from the site's sitemap
-Zacni zajem z zemljevidom spletnega mesta
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Preberi zemljevid spletnega mesta (vrstice Sitemap: v robots.txt, nato /sitemap.xml) in dodaj vsak navedeni URL kot zacetni naslov.
-Sitemap address:
-Naslov zemljevida spletnega mesta:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Naslov zemljevida spletnega mesta, ki naj se prebere namesto preverjanja mesta; pustite prazno za preverjanje robots.txt in nato /sitemap.xml.
--- a/lang/Svenska.txt
+++ b/lang/Svenska.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Största inbäddade resurs (byte):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 En resurs över den här storleken behåller en vanlig länk; lämna tomt för standardvärdet 10485760 byte.
-Seed the crawl from the site's sitemap
-Starta insamlingen från webbplatsens webbplatskarta
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Läs webbplatsens webbplatskarta (Sitemap:-rader i robots.txt, sedan /sitemap.xml) och lägg till varje angiven URL som startadress.
-Sitemap address:
-Webbplatskartans adress:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adress till en webbplatskarta som ska läsas i stället för att söka på webbplatsen; lämna tomt för att kontrollera robots.txt och sedan /sitemap.xml.
--- a/lang/Turkish.txt
+++ b/lang/Turkish.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 En büyük gömülü kaynak (bayt):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Bu boyutun üzerindeki bir kaynak sýradan baðlantýsýný korur; 10485760 baytlýk varsayýlan için boþ býrakýn.
-Seed the crawl from the site's sitemap
-Taramayý sitenin site haritasýndan baþlat
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Sitenin site haritasýný oku (robots.txt içindeki Sitemap: satýrlarý, ardýndan /sitemap.xml) ve listelenen her URL'yi baþlangýç adresi olarak ekle.
-Sitemap address:
-Site haritasý adresi:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Siteyi yoklamak yerine okunacak site haritasýnýn adresi; robots.txt ve ardýndan /sitemap.xml yoklamasý için boþ býrakýn.
--- a/lang/Ukrainian.txt
+++ b/lang/Ukrainian.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Íàéá³ëüøèé âáóäîâàíèé ðåñóðñ (áàéòè):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Ðåñóðñ, á³ëüøèé çà öåé ðîçì³ð, çáåð³ãàº çâè÷àéíå ïîñèëàííÿ; çàëèøòå ïîðîæí³ì äëÿ òèïîâîãî çíà÷åííÿ 10485760 áàéò³â.
-Seed the crawl from the site's sitemap
-Ïî÷èíàòè îáõ³ä ç êàðòè ñàéòó
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Ïðî÷èòàòè êàðòó ñàéòó (ðÿäêè Sitemap: ó robots.txt, ïîò³ì /sitemap.xml) ³ äîäàòè êîæíó âêàçàíó â í³é URL-àäðåñó ÿê ïî÷àòêîâó.
-Sitemap address:
-Àäðåñà êàðòè ñàéòó:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Àäðåñà êàðòè ñàéòó, ÿêó ñë³ä ïðî÷èòàòè çàì³ñòü îïèòóâàííÿ ñàéòó; çàëèøòå ïîðîæí³ì, ùîá ïåðåâ³ðèòè robots.txt, à ïîò³ì /sitemap.xml.
--- a/lang/Uzbek.txt
+++ b/lang/Uzbek.txt
@@ -976,11 +976,3 @@ Largest inlined asset (bytes):
 Eng katta joylangan resurs (bayt):
 An asset above this size keeps an ordinary link; leave blank for the 10485760-byte default.
 Bu o’lchamdan katta resurs oddiy havolani saqlab qoladi; standart 10485760 bayt uchun bo’sh qoldiring.
-Seed the crawl from the site's sitemap
-Yig’ishni saytning sayt xaritasidan boshlash
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Saytning sayt xaritasini o’qish (robots.txt dagi Sitemap: qatorlari, so’ngra /sitemap.xml) va unda ko’rsatilgan har bir URL manzilni boshlang’ich manzil sifatida qo’shish.
-Sitemap address:
-Sayt xaritasi manzili:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Saytni tekshirish o’rniga o’qiladigan sayt xaritasi manzili; robots.txt, so’ngra /sitemap.xml ni tekshirish uchun bo’sh qoldiring.
--- a/m4/check_zlib.m4
+++ b/m4/check_zlib.m4
@@ -1,33 +1,82 @@
 dnl @synopsis CHECK_ZLIB()
 dnl
-dnl Look for zlib. It is a hard requirement, not an option: the cache and the
-dnl WARC output are zip/gzip containers, and the bundled minizip calls zlib
-dnl directly. --with-zlib=DIR points at a non-standard prefix.
+dnl This macro searches for an installed zlib library. If nothing
+dnl was specified when calling configure, it searches first in /usr/local
+dnl and then in /usr. If the --with-zlib=DIR is specified, it will try
+dnl to find it in DIR/include/zlib.h and DIR/lib/libz.a. If --without-zlib
+dnl is specified, the library is not searched at all.
+dnl
+dnl If either the header file (zlib.h) or the library (libz) is not
+dnl found, the configuration exits on error, asking for a valid
+dnl zlib installation directory or --without-zlib.
+dnl
+dnl The macro defines the symbol HAVE_LIBZ if the library is found. You should
+dnl use autoheader to include a definition for this symbol in a config.h
+dnl file. Sample usage in a C/C++ source is as follows:
+dnl
+dnl   #ifdef HAVE_LIBZ
+dnl   #include <zlib.h>
+dnl   #endif /* HAVE_LIBZ */
+dnl
+dnl @version $Id$
+dnl @author Loic Dachary <loic@senga.org>
 dnl
-dnl Adds -lz to LIBS and defines HAVE_LIBZ.

-AC_DEFUN([CHECK_ZLIB], [
-AC_ARG_WITH([zlib],
-	[AS_HELP_STRING([--with-zlib=DIR],[root directory of the zlib installation])],
-	[zlib_want=$withval], [zlib_want=yes])
-if test "$zlib_want" = "no"; then
-	AC_MSG_ERROR([zlib cannot be disabled: the cache and the WARC output are zip/gzip containers, and the bundled minizip calls zlib directly])
+AC_DEFUN([CHECK_ZLIB],
+#
+# Handle user hints
+#
+[AC_MSG_CHECKING(if zlib is wanted)
+AC_ARG_WITH(zlib,
+[  --with-zlib=DIR root directory path of zlib installation [defaults to
+                    /usr/local or /usr if not found in /usr/local]
+  --without-zlib to disable zlib usage completely],
+[if test "$withval" != no ; then
+  AC_MSG_RESULT(yes)
+  ZLIB_HOME="$withval"
+else
+  AC_MSG_RESULT(no)
+fi], [
+AC_MSG_RESULT(yes)
+ZLIB_HOME=/usr/local
+if test ! -f "${ZLIB_HOME}/include/zlib.h"
+then
+        ZLIB_HOME=/usr
 fi
-if test "$zlib_want" != "yes"; then
-	# An explicit prefix is authoritative: if the header is not under it,
-	# error rather than silently pick a system copy.
-	if test ! -f "$zlib_want/include/zlib.h"; then
-		AC_MSG_ERROR([zlib requested at $zlib_want but $zlib_want/include/zlib.h is missing])
-	fi
-	CPPFLAGS="$CPPFLAGS -I$zlib_want/include"
-	LDFLAGS="$LDFLAGS -L$zlib_want/lib"
-elif test -f /usr/local/include/zlib.h; then
-	# Where the BSD ports tree lands zlib, and not always searched by default.
-	CPPFLAGS="$CPPFLAGS -I/usr/local/include"
-	LDFLAGS="$LDFLAGS -L/usr/local/lib"
-fi
-AC_CHECK_HEADER([zlib.h], [],
-	[AC_MSG_ERROR([zlib.h not found; install the zlib development files or pass --with-zlib=DIR])])
-AC_CHECK_LIB([z], [inflateEnd], [],
-	[AC_MSG_ERROR([libz not found; install the zlib development files or pass --with-zlib=DIR])])
+])
+
+#
+# Locate zlib, if wanted
+#
+if test -n "${ZLIB_HOME}"
+then
+        ZLIB_OLD_LDFLAGS=$LDFLAGS
+        ZLIB_OLD_CPPFLAGS=$LDFLAGS
+        LDFLAGS="$LDFLAGS -L${ZLIB_HOME}/lib"
+        CPPFLAGS="$CPPFLAGS -I${ZLIB_HOME}/include"
+        AC_LANG_SAVE
+        AC_LANG_C
+        AC_CHECK_LIB(z, inflateEnd, [zlib_cv_libz=yes], [zlib_cv_libz=no])
+        AC_CHECK_HEADER(zlib.h, [zlib_cv_zlib_h=yes], [zlib_cv_zlib_h=no])
+        AC_LANG_RESTORE
+        if test "$zlib_cv_libz" = "yes" -a "$zlib_cv_zlib_h" = "yes"
+        then
+                #
+                # If both library and header were found, use them
+                #
+                AC_CHECK_LIB(z, inflateEnd)
+                AC_MSG_CHECKING(zlib in ${ZLIB_HOME})
+                AC_MSG_RESULT(ok)
+        else
+                #
+                # If either header or library was not found, revert and bomb
+                #
+                AC_MSG_CHECKING(zlib in ${ZLIB_HOME})
+                LDFLAGS="$ZLIB_OLD_LDFLAGS"
+                CPPFLAGS="$ZLIB_OLD_CPPFLAGS"
+                AC_MSG_RESULT(failed)
+                AC_MSG_ERROR(either specify a valid zlib installation with --with-zlib=DIR or disable zlib usage with --without-zlib)
+        fi
+fi
+
 ])
--- a/man/httrack.1
+++ b/man/httrack.1
@@ -36,7 +36,6 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-t, \-\-test\fR ]
 [ \fB\-%L, \-\-list\fR ]
 [ \fB\-%S, \-\-urllist\fR ]
-[ \fB\-%m, \-\-sitemap\fR ]
 [ \fB\-NN, \-\-structure[=N]\fR ]
 [ \fB\-%N, \-\-delayed\-type\-check\fR ]
 [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
@@ -190,8 +189,6 @@ test all URLs (even forbidden ones) (\-\-test)
 <file> add all URL located in this text file (one URL per line) (\-\-list <param>)
 .IP \-%S
 <file> add all scan rules located in this text file (one scan rule per line) (\-\-urllist <param>)
-.IP \-%m
-seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (\-\-sitemap)
 .SS Build options:
 .IP \-NN
 structure type (0 *original structure, 1+: see below) (\-\-structure[=N])
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -66,7 +66,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htswarc.c htschanges.c htssinglefile.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htschanges.c htssinglefile.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
@@ -77,7 +77,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htschanges.h htssinglefile.h htssitemap.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htschanges.h htssinglefile.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \
--- a/src/htsalias.c
+++ b/src/htsalias.c
@@ -116,10 +116,6 @@ const char *hts_optalias[][4] = {
   "load extra cookies from a Netscape cookies.txt"},
  {"changes", "-%d", "single",
   "write hts-changes.json: what this crawl changed vs. the previous mirror"},
-  {"sitemap", "-%m", "single",
-   "seed the crawl from the start host's sitemap (robots.txt, then "
-   "/sitemap.xml)"},
-  {"sitemap-url", "-%mu", "param1", "seed the crawl from this sitemap URL"},
  {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
  {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
  {"warc-max-size", "-%rs", "param1",
--- a/src/htsback.c
+++ b/src/htsback.c
@@ -48,6 +48,11 @@ Please visit our Website: http://www.httrack.com
 #include "htsftp.h"
 #include "htscodec.h"
 #include "htsproxy.h"
+#if HTS_USEZLIB
+#include "htszlib.h"
+#else
+#error HTS_USEZLIB not defined
+#endif

 #ifdef _WIN32
 #ifndef __cplusplus
--- a/src/htscore.c
+++ b/src/htscore.c
@@ -39,7 +39,6 @@ Please visit our Website: http://www.httrack.com

 /* File defs */
 #include "htscore.h"
-#include "htssitemap.h"
 #include "htswarc.h"
 #include "htschanges.h"
 #include "htssinglefile.h"
@@ -950,22 +949,6 @@ int httpmirror(char *url1, httrackp * opt) {
    heap_top()->premier = heap_top_index();        // premier lien, objet-père=objet              
    heap_top()->precedent = heap_top_index();      // lien précédent

-    /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are
-       injected before the crawl gets far. */
-    hts_sitemap_free(opt); /* an earlier mirror may have left a doc list */
-    if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) {
-      char BIGSTK first[HTS_URLMAXSIZE * 2];
-      const char *const eol = strchr(primary, '\n');
-      const size_t len = eol != NULL ? (size_t) (eol - primary) : 0;
-
-      first[0] = '\0';
-      if (len > 0 && len < sizeof(first)) {
-        memcpy(first, primary, len);
-        first[len] = '\0';
-      }
-      hts_sitemap_seed(opt, first);
-    }
-
    // Initialiser cache
    {
      opt->state._hts_in_html_parsing = 4;
@@ -1610,21 +1593,11 @@ int httpmirror(char *url1, httrackp * opt) {
        stre.maketrack_fp = maketrack_fp;

        /* Parse */
-        {
-          const int nlinks = opt->lien_tot;
-
-          if (hts_mirror_check_moved(&str, &stre) != 0) {
-            XH_uninit;
-            return -1;
-          }
-          /* A redirect re-queues the target as a fresh link; without carrying
-             the marking over, a moved sitemap is fetched and then ignored. */
-          if (opt->sitemap_state != NULL && opt->lien_tot > nlinks &&
-              hts_sitemap_pending(opt, urladr(), urlfil())) {
-            hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr,
-                                 heap_top()->fil);
-          }
+        if (hts_mirror_check_moved(&str, &stre) != 0) {
+          XH_uninit;
+          return -1;
        }
+
      }

    }                           // if !error
@@ -1642,29 +1615,6 @@ int httpmirror(char *url1, httrackp * opt) {
      /* Load file and decode if necessary, after redirect check. */
      LOAD_IN_MEMORY_IF_NECESSARY();

-      /* Sitemap document: turn its <loc> URLs into top-level seeds. They go
-         through htsAddLink, so the wizard's filters and scope rules decide, and
-         this link's max depth leaves them the full budget. */
-      if (opt->sitemap_state != NULL &&
-          hts_sitemap_pending(opt, urladr(), urlfil())) {
-        htsmoduleStruct BIGSTK smstr;
-        int smptr = ptr;
-
-        memset(&smstr, 0, sizeof(smstr));
-        smstr.opt = opt;
-        smstr.sback = sback;
-        smstr.cache = &cache;
-        smstr.hashptr = hashptr;
-        smstr.numero_passe = numero_passe;
-        smstr.ptr_ = &smptr; /* scratch: the ingester retargets the wizard */
-        smstr.addLink = htsAddLink;
-        smstr.url_host = urladr();
-        smstr.url_file = urlfil();
-        smstr.mime = r.contenttype;
-        hts_sitemap_ingest(opt, &smstr, urladr(), urlfil(), r.adr,
-                           r.adr != NULL && r.size > 0 ? (size_t) r.size : 0);
-      }
-
      // ------------------------------------------------------
      // ok, fichier chargé localement
      // ------------------------------------------------------
@@ -1875,9 +1825,6 @@ int httpmirror(char *url1, httrackp * opt) {

        if (strnotempty(savename()) == 0) {       // pas de chemin de sauvegarde
          if (strcmp(urlfil(), "/robots.txt") == 0) {     // robots.txt
-            char BIGSTK sitemaps[8192];
-
-            sitemaps[0] = '\0';
            if (r.adr) {
              char BIGSTK infobuff[8192];
 #ifdef IGNORE_RESTRICTIVE_ROBOTS
@@ -1889,8 +1836,7 @@ int httpmirror(char *url1, httrackp * opt) {
 #endif

              robots_parse(&robots, urladr(), r.adr, r.size, infobuff,
-                           sizeof(infobuff), keep_root, sitemaps,
-                           sizeof(sitemaps));
+                           sizeof(infobuff), keep_root);
              if (strnotempty(infobuff)) {
                hts_log_print(opt, LOG_INFO,
                              "Note: robots.txt forbidden links for %s are: %s",
@@ -1900,10 +1846,6 @@ int httpmirror(char *url1, httrackp * opt) {
                              urladr(), infobuff);
              }
            }
-            /* After robots_parse, so the rules this very body carries already
-               gate the sitemap fetch. Runs even on a failed probe, which is
-               what falls back to the well-known location. */
-            hts_sitemap_robots(opt, urladr(), sitemaps);
          }
        } else if (r.is_write) {        // déja sauvé sur disque
          /*
@@ -2344,7 +2286,6 @@ int httpmirror(char *url1, httrackp * opt) {
  usercommand(opt, 0, NULL, NULL, NULL, NULL);
  warc_close_opt(opt);
  hts_changes_close_opt(opt);
-  hts_sitemap_free(opt);

  // désallocation mémoire & buffers
  XH_uninit;
@@ -3743,10 +3684,6 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
  to->single_file = from->single_file;
  if (from->single_file_max_size > 0)
    to->single_file_max_size = from->single_file_max_size;
-  if (from->sitemap)
-    to->sitemap = from->sitemap;
-  if (StringNotEmpty(from->sitemap_url))
-    StringCopyS(to->sitemap_url, from->sitemap_url);

  if (from->pause_max_ms > 0) {
    to->pause_min_ms = from->pause_min_ms;
--- a/src/htscoremain.c
+++ b/src/htscoremain.c
@@ -1833,26 +1833,6 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                  }
                }
                break;
-              case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps
-                if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap
-                  com++;
-                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
-                    HTS_PANIC_PRINTF(
-                        "Option sitemap-url needs a blank space and a URL");
-                    htsmain_free();
-                    return -1;
-                  }
-                  na++;
-                  if (strlen(argv[na]) >= HTS_URLMAXSIZE) {
-                    HTS_PANIC_PRINTF("Sitemap URL too long");
-                    htsmain_free();
-                    return -1;
-                  }
-                  StringCopy(opt->sitemap_url, argv[na]);
-                } else { // --sitemap: robots.txt probe, then /sitemap.xml
-                  opt->sitemap = HTS_TRUE;
-                }
-                break;
              case 'Y': // why: explain the filter verdict for a URL, no crawl
                if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                  HTS_PANIC_PRINTF("Option why needs a blank space and a URL");
--- a/src/htsglobal.h
+++ b/src/htsglobal.h
@@ -138,11 +138,10 @@ Please visit our Website: http://www.httrack.com
 #define HTS_DOSNAME 0
 #endif

-// zlib is mandatory: the cache is a zip and minizip calls it regardless
+// utiliser zlib?
 #ifndef HTS_USEZLIB
+// autoload
 #define HTS_USEZLIB 1
-#elif !HTS_USEZLIB
-#error HTS_USEZLIB=0 is not a supported configuration
 #endif

 // brotli and zstd content codings; off unless the build opted in (configure,
--- a/src/htshelp.c
+++ b/src/htshelp.c
@@ -526,11 +526,6 @@ void help(const char *app, int more) {
    (" %L <file> add all URL located in this text file (one URL per line)");
  infomsg
    (" %S <file> add all scan rules located in this text file (one scan rule per line)");
-  infomsg(" %m  seed the crawl from the site's sitemap (robots.txt Sitemap:, "
-          "then /sitemap.xml); --sitemap-url URL names one explicitly. A "
-          "sitemap you name, or one the site declares, is fetched even under "
-          "robots.txt Disallow; only the guessed /sitemap.xml obeys it. The "
-          "URLs found still pass every filter and scope rule");
  infomsg("");
  infomsg("Build options:");
  infomsg("  NN structure type (0 *original structure, 1+: see below)");
--- a/src/htslib.c
+++ b/src/htslib.c
@@ -36,7 +36,6 @@ Please visit our Website: http://www.httrack.com
 // Fichier librairie .c

 #include "htscore.h"
-#include "htssitemap.h"
 #include "htswarc.h"
 #include "htschanges.h"
 #include "htssinglefile.h"
@@ -6031,7 +6030,6 @@ HTSEXT_API httrackp *hts_create_opt(void) {
  StringCopy(opt->strip_query, "");
  StringCopy(opt->cookies_file, "");
  StringCopy(opt->warc_file, "");
-  StringCopy(opt->sitemap_url, "");
  opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
  opt->changes = HTS_FALSE;
  opt->changes_state = NULL;
@@ -6189,8 +6187,6 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
    StringFree(opt->cookies_file);
    StringFree(opt->why_url);
    StringFree(opt->warc_file);
-    StringFree(opt->sitemap_url);
-    hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */

    hts_changes_free_opt(opt);

--- a/src/htsopt.h
+++ b/src/htsopt.h
@@ -557,13 +557,6 @@ struct httrackp {
  LLint single_file_max_size; /**< --single-file-max-size: per-asset cap in
                                   bytes; a bigger asset stays a link.
                                   Tail: ABI */
-  hts_boolean sitemap; /**< --sitemap: probe the start host's robots.txt for
-                            Sitemap: lines, else /sitemap.xml. Tail: ABI */
-  String sitemap_url;  /**< --sitemap-url: sitemap to ingest. Tail: ABI */
-  /* Live state, not an option: copy_htsopt must leave it alone. It sits here
-     rather than in htsoptstate because that struct is embedded by value, so
-     growing it would shift every httrackp field declared after it. */
-  void *sitemap_state; /**< hts_sitemap_state*, or NULL. Tail: ABI */
 };

 /* Running statistics for a mirror. */
--- a/src/htsparse.c
+++ b/src/htsparse.c
@@ -59,6 +59,7 @@ Please visit our Website: http://www.httrack.com

 // parser
 #include "htsparse.h"
+#include "htssinglefile.h"
 #include "htsback.h"

 // arrays
@@ -3043,6 +3044,17 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {

                        if ((opt->getmode & HTS_GETMODE_HTML) && (ptr > 0)) {

+                          /* --single-file: tag the reference for the
+                             end-of-mirror pass. The tag is a fragment, so the
+                             mirror stays valid if that pass never runs. */
+                          if (opt->single_file && !in_media && p_type == 0 &&
+                              !p_searchMETAURL &&
+                              singlefile_may_inline(
+                                  intag_start_valid ? intag_start : NULL,
+                                  tag_attr_start)) {
+                            strcatbuff(tempo, SINGLEFILE_MARK);
+                          }
+
                          // écrire le lien modifié, relatif
                          // Note: escape all chars, even >127 (no UTF)
                          HT_ADD_HTMLESCAPED_FULL(tempo);
--- a/src/htsrobots.c
+++ b/src/htsrobots.c
@@ -147,8 +147,7 @@ static void robots_blob_add(char *blob, size_t blobsize, char marker,

 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                  size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow, char *sitemaps,
-                  size_t sitemapsize) {
+                  hts_boolean keep_root_disallow) {
  size_t bptr = 0;
  int record = 0;
  char BIGSTK line[1024];
@@ -157,8 +156,6 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
  blob[0] = '\0';
  if (info != NULL && infosize > 0)
    info[0] = '\0';
-  if (sitemaps != NULL && sitemapsize > 0)
-    sitemaps[0] = '\0';
 #if DEBUG_ROBOTS
  printf("robots.txt dump:\n%s\n", body);
 #endif
@@ -175,19 +172,7 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
      line[llen - 1] = '\0';
      llen--;
    }
-    if (sitemaps != NULL && strfield(line, "sitemap:")) {
-      // group-independent record (RFC 9309): collected whatever the group
-      char *a = line + 8;
-
-      while (is_realspace(*a))
-        a++;
-      /* A line at the buffer limit was truncated: a half URL is not one. */
-      if (strnotempty(a) && strlen(line) < sizeof(line) - 3 &&
-          strlen(a) + 2 < sitemapsize - strlen(sitemaps)) {
-        strlcatbuff(sitemaps, a, sitemapsize);
-        strlcatbuff(sitemaps, "\n", sitemapsize);
-      }
-    } else if (strfield(line, "user-agent:")) {
+    if (strfield(line, "user-agent:")) {
      char *a = line + 11;

      while (is_realspace(*a))
--- a/src/htsrobots.h
+++ b/src/htsrobots.h
@@ -56,12 +56,10 @@ int checkrobots(robots_wizard * robots, const char *adr, const char *fil);
 void checkrobots_free(robots_wizard * robots);
 int checkrobots_set(robots_wizard * robots, const char *adr, const char *data);
 /* Parse robots.txt `body` for `adr`, storing the HTTrack group's rules; `info`
-   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /", and
-   `sitemaps` (optional) collects the Sitemap: URLs, one per line. */
+   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /". */
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                  size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow, char *sitemaps,
-                  size_t sitemapsize);
+                  hts_boolean keep_root_disallow);
 #endif

 #endif
--- a/src/htsselftest.c
+++ b/src/htsselftest.c
@@ -59,7 +59,6 @@ Please visit our Website: http://www.httrack.com
 #include "htssniff.h"
 #include "htscodec.h"
 #include "htsproxy.h"
-#include "htssitemap.h"
 #include "htswarc.h"
 #include "htschanges.h"
 #include "htssinglefile.h"
@@ -1763,22 +1762,6 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
  if (to->single_file_max_size != 4096)
    err = 1;

-  /* sitemap pair: the flag latches on, the URL takes the String deep copy */
-  from->sitemap = HTS_TRUE;
-  StringCopy(from->sitemap_url, "http://h.test/sitemap.xml");
-  to->sitemap = HTS_FALSE;
-  StringCopy(to->sitemap_url, "");
-  copy_htsopt(from, to);
-  if (!to->sitemap ||
-      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
-    err = 1;
-  from->sitemap = HTS_FALSE;
-  StringCopy(from->sitemap_url, "");
-  copy_htsopt(from, to);
-  if (!to->sitemap ||
-      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
-    err = 1;
-
  /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
  from->pause_min_ms = 5000;
  from->pause_max_ms = 10000;
@@ -3703,7 +3686,7 @@ static int rb_decide(robots_wizard *r, const char *txt, const char *path) {
  char host[64];

  snprintf(host, sizeof(host), "h%d.example", n++);
-  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE, NULL, 0);
+  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE);
  return checkrobots(r, host, path);
 }

@@ -3780,267 +3763,6 @@ static int st_robots(httrackp *opt, int argc, char **argv) {
  return 0;
 }

-/* Collect the URLs a sitemap scan hands out. */
-typedef struct sm_collect {
-  int n;
-  char url[8][HTS_URLMAXSIZE];
-} sm_collect;
-
-static hts_boolean sm_take(void *arg, const char *url) {
-  sm_collect *const c = (sm_collect *) arg;
-
-  if (c->n < (int) (sizeof(c->url) / sizeof(c->url[0])))
-    strcpybuff(c->url[c->n], url);
-  c->n++;
-  return HTS_TRUE;
-}
-
-/* Scan `doc` off a heap buffer with no NUL terminator, so a read past the
-   declared size is an ASan error rather than a silent pass. */
-static int sm_scan(const char *doc, int maxurls, hts_boolean *is_index,
-                   sm_collect *out) {
-  const size_t len = strlen(doc);
-  char *raw = malloct(len);
-  int n;
-
-  memset(out, 0, sizeof(*out));
-  assertf(raw != NULL);
-  memcpy(raw, doc, len);
-  n = hts_sitemap_scan(raw, len, maxurls, is_index, sm_take, out);
-  freet(raw);
-  return n;
-}
-
-static int st_sitemap(httrackp *opt, int argc, char **argv) {
-  sm_collect c;
-  hts_boolean idx;
-  (void) opt;
-  (void) argc;
-  (void) argv;
-
-  /* A urlset yields its <loc> URLs, in order, unescaped. */
-  assertf(sm_scan("<?xml version=\"1.0\"?><urlset>"
-                  "<url><loc>http://h.test/a.html</loc></url>"
-                  "<url><loc>  https://h.test/b?x=1&amp;y=2\n  </loc></url>"
-                  "</urlset>",
-                  100, &idx, &c) == 2);
-  assertf(!idx);
-  assertf(strcmp(c.url[0], "http://h.test/a.html") == 0);
-  assertf(strcmp(c.url[1], "https://h.test/b?x=1&y=2") == 0);
-
-  /* A sitemapindex is flagged: its URLs are child sitemaps, not pages. */
-  assertf(sm_scan("<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc>"
-                  "</sitemap></sitemapindex>",
-                  100, &idx, &c) == 1);
-  assertf(idx);
-
-  /* Root element decides even when the other name appears later as text. */
-  assertf(sm_scan("<urlset><url><loc>http://h.test/a</loc></url>"
-                  "<!-- sitemapindex --></urlset>",
-                  100, &idx, &c) == 1);
-  assertf(!idx);
-
-  /* Numeric character references, decimal and hex, decode to ASCII. */
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#63;b&#x3D;c</loc></urlset>",
-                  100, &idx, &c) == 1);
-  assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0);
-
-  /* A reference decoding to a control byte is dropped: the shared decoder
-     writes the real character and the URL check refuses it. A reference the
-     decoder cannot represent (&#0;) stays verbatim, like an unknown entity. */
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#10;b</loc></urlset>", 100,
-                  &idx, &c) == 0);
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#9;b</loc></urlset>", 100, &idx,
-                  &c) == 0);
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#0;b</loc></urlset>", 100, &idx,
-                  &c) == 1);
-  assertf(strcmp(c.url[0], "http://h.test/a&#0;b") == 0);
-
-  /* A comment naming the other root element must not flip the verdict. */
-  assertf(sm_scan("<!-- <sitemapindex> --><urlset><url>"
-                  "<loc>http://h.test/p</loc></url></urlset>",
-                  100, &idx, &c) == 1);
-  assertf(!idx);
-  assertf(sm_scan("<?xml version=\"1.0\"?><!-- <urlset> -->"
-                  "<sitemapindex><loc>http://h.test/s</loc></sitemapindex>",
-                  100, &idx, &c) == 1);
-  assertf(idx);
-
-  /* <location> is not <loc>. */
-  assertf(sm_scan("<urlset><location>http://h.test/a</location></urlset>", 100,
-                  &idx, &c) == 0);
-
-  /* Rejected: relative, non-http scheme, embedded space, empty. */
-  assertf(sm_scan("<urlset><loc>/a.html</loc><loc>ftp://h.test/a</loc>"
-                  "<loc>javascript:alert(1)</loc>"
-                  "<loc>http://h.test/a b</loc><loc></loc></urlset>",
-                  100, &idx, &c) == 0);
-
-  /* The URL length bound: one under fits, exactly at it is dropped rather than
-     truncated into a different URL. */
-  {
-    char BIGSTK doc[HTS_URLMAXSIZE * 2];
-    char BIGSTK url[HTS_URLMAXSIZE + 1];
-    size_t i;
-
-    strcpybuff(url, "http://h.test/");
-    for (i = strlen(url); i < HTS_URLMAXSIZE - 1; i++)
-      url[i] = 'a';
-    url[i] = '\0';
-    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
-    assertf(sm_scan(doc, 100, &idx, &c) == 1);
-
-    url[i] = 'a';
-    url[i + 1] = '\0';
-    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
-    assertf(sm_scan(doc, 100, &idx, &c) == 0);
-  }
-
-  /* The URL cap stops the scan. */
-  assertf(sm_scan("<urlset><loc>http://h.test/1</loc><loc>http://h.test/2</loc>"
-                  "<loc>http://h.test/3</loc></urlset>",
-                  2, &idx, &c) == 2);
-
-  /* The per-document cap at the value the engine actually uses. */
-  {
-    const int many = HTS_SITEMAP_MAX_URLS_DOC + 10;
-    const size_t cap = (size_t) many * 40 + 32;
-    char *big = malloct(cap);
-    size_t off;
-    int i;
-
-    assertf(big != NULL);
-    off = (size_t) snprintf(big, cap, "<urlset>");
-    assertf(off < cap);
-    for (i = 0; i < many; i++) {
-      const int len =
-          snprintf(big + off, cap - off, "<loc>http://h.test/%d</loc>", i);
-
-      assertf(len > 0 && (size_t) len < cap - off);
-      off += (size_t) len;
-    }
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(big, off, HTS_SITEMAP_MAX_URLS_DOC, &idx, sm_take,
-                             &c) == HTS_SITEMAP_MAX_URLS_DOC);
-    /* The handler count, not just the return: a call site hardcoding a smaller
-       cap would still return its own argument. */
-    assertf(c.n == HTS_SITEMAP_MAX_URLS_DOC);
-    freet(big);
-  }
-
-#if HTS_USEZLIB
-  /* A highly compressible document decodes without running away: the ratio
-     budget cannot bind (deflate tops out near 1032:1), so this pins the
-     decompression path itself rather than the 64 MiB ceiling. */
-  {
-    const char *const one = "<url><loc>http://h.test/bomb</loc></url>";
-    const size_t reps = 40000;
-    size_t xlen = 8 + reps * strlen(one) + 10, i;
-    char *x = malloct(xlen + 1);
-    uLongf zlen;
-    char *z;
-    z_stream zs;
-
-    assertf(x != NULL);
-    {
-      size_t w = (size_t) snprintf(x, xlen, "<urlset>");
-      int len;
-
-      assertf(w < xlen);
-      for (i = 0; i < reps; i++) {
-        len = snprintf(x + w, xlen - w, "%s", one);
-        assertf(len > 0 && (size_t) len < xlen - w);
-        w += (size_t) len;
-      }
-      len = snprintf(x + w, xlen - w, "</urlset>");
-      assertf(len > 0 && (size_t) len < xlen - w);
-      w += (size_t) len;
-      xlen = w;
-    }
-    zlen = compressBound((uLong) xlen) + 32;
-    z = malloct((size_t) zlen);
-    assertf(z != NULL);
-    memset(&zs, 0, sizeof(zs));
-    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
-                         Z_DEFAULT_STRATEGY) == Z_OK);
-    zs.next_in = (const Bytef *) x;
-    zs.avail_in = (uInt) xlen;
-    zs.next_out = (Bytef *) z;
-    zs.avail_out = (uInt) zlen;
-    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
-    zlen = (uLongf) zs.total_out;
-    deflateEnd(&zs);
-    /* well over the 4096:1 budget's 1 MiB floor, and far under the 64 MiB cap
-     */
-    assertf(xlen > 1024 * 1024 && (size_t) zlen < xlen / 100);
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(z, (size_t) zlen, 10, &idx, sm_take, &c) == 10);
-    assertf(strcmp(c.url[0], "http://h.test/bomb") == 0);
-    freet(z);
-    freet(x);
-  }
-#endif
-
-  /* An unterminated <loc> at end of buffer must not read past it. */
-  assertf(sm_scan("<urlset><loc>http://h.test/a", 100, &idx, &c) == 0);
-  assertf(sm_scan("<urlset><lo", 100, &idx, &c) == 0);
-
-#if HTS_USEZLIB
-  /* A gzip-framed document is decompressed before scanning. */
-  {
-    const char *const xml =
-        "<urlset><url><loc>http://h.test/gz.html</loc></url></urlset>";
-    uLongf zlen = compressBound((uLong) strlen(xml)) + 32;
-    char *z = malloct((size_t) zlen);
-    z_stream zs;
-
-    assertf(z != NULL);
-    memset(&zs, 0, sizeof(zs));
-    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
-                         Z_DEFAULT_STRATEGY) == Z_OK);
-    zs.next_in = (const Bytef *) xml;
-    zs.avail_in = (uInt) strlen(xml);
-    zs.next_out = (Bytef *) z;
-    zs.avail_out = (uInt) zlen;
-    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
-    zlen = (uLongf) zs.total_out;
-    deflateEnd(&zs);
-
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(z, (size_t) zlen, 100, &idx, sm_take, &c) == 1);
-    assertf(strcmp(c.url[0], "http://h.test/gz.html") == 0);
-
-    /* Truncated gzip: refused, not scanned as plain text. */
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(z, 4, 100, &idx, sm_take, &c) == -1);
-    freet(z);
-  }
-#endif
-
-  /* robots.txt: only Sitemap: records, comments stripped, case-insensitive,
-     and group-independent (no User-agent line needed). */
-  /* robots_parse collects Sitemap: whatever the user-agent group, strips the
-     comment and keeps the rules working alongside it. */
-  {
-    const char *const txt = "User-agent: *\nDisallow: /x\n"
-                            "SITEMAP:  http://h.test/s1.xml  # first\n"
-                            "Sitemapper: http://h.test/no.xml\n"
-                            "Sitemap:\thttps://h.test/s2.xml\n";
-    char BIGSTK maps[1024];
-    robots_wizard rb;
-
-    memset(&rb, 0, sizeof(rb));
-    robots_parse(&rb, "h.test", txt, strlen(txt), NULL, 0, HTS_TRUE, maps,
-                 sizeof(maps));
-    assertf(strcmp(maps, "http://h.test/s1.xml\nhttps://h.test/s2.xml\n") == 0);
-    assertf(checkrobots(&rb, "h.test", "/x") == -1);
-    checkrobots_free(&rb);
-  }
-
-  printf("sitemap self-test OK\n");
-  return 0;
-}
-
 /* Connected stream pair over loopback; Windows has no socketpair(). */
 static int st_socketpair(T_SOC sv[2]) {
  struct sockaddr_in sa;
@@ -4138,6 +3860,21 @@ static int st_ftpuser(httrackp *opt, int argc, char **argv) {
  return 0;
 }

+/* Bounded substring search (records carry NUL bytes; strstr won't do). */
+static const char *warc_memstr(const char *hay, const char *needle,
+                               size_t haylen, size_t nlen) {
+  if (nlen == 0 || haylen < nlen)
+    return NULL;
+  {
+    size_t i;
+    for (i = 0; i + nlen <= haylen; i++) {
+      if (memcmp(hay + i, needle, nlen) == 0)
+        return hay + i;
+    }
+  }
+  return NULL;
+}
+
 /* Slurp a whole file into a malloc'd buffer; sets *len. NULL on error. */
 static unsigned char *warc_slurp(const char *path, size_t *len) {
  FILE *f = FOPEN(path, "rb");
@@ -4215,13 +3952,6 @@ static unsigned char *warc_next_member(const unsigned char **in,
   Content-Length == block length, the \r\n\r\n trailer intact, the response
   body round-trips, and the hop-by-hop Transfer-Encoding is dropped (a real
   Content-Encoding is kept verbatim; see warc-verbatim). */
-/* Argument order kept for the existing call sites; the search itself is the
-   shared hts_memstr. */
-static const char *warc_memstr(const char *hay, const char *needle,
-                               size_t haylen, size_t nlen) {
-  return hts_memstr(hay, haylen, needle, nlen);
-}
-
 static int st_warc(httrackp *opt, int argc, char **argv) {
  char path[HTS_URLMAXSIZE];
  warc_writer *w;
@@ -5315,45 +5045,48 @@ static const char sf_png[] = "\x89PNG\r\n\x1a\n\x00\x01\x02\xff";

 static const char sf_page[] =
    "<html><head>\n"
-    "<link rel=\"stylesheet\" href=\"css/main.css\">\n"
-    "<link rel=\"canonical\" href=\"other.html\">\n"
+    "<link rel=\"stylesheet\" href=\"css/main.css#!htsinline\">\n"
+    "<link rel=\"canonical\" href=\"other.html#!htsinline\">\n"
    "<title>t</title>\n"
-    "<style>body { background: url(\"img/a%20b.png\"); }</style>\n"
+    "<style>body { background: url(\"img/a%20b.png#!htsinline\"); }</style>\n"
    "</head><body>\n"
-    "<img src=\"img/a%20b.png\" srcset=\"img/a%20b.png 1x, img/big.png 2x\">\n"
-    "<link rel=\"icon\" href=\"icon.png\">\n"
-    "<link rel=\"preload\" as=\"font\" href=\"font/f.woff2\">\n"
+    "<img src=\"img/a%20b.png#!htsinline\" srcset=\"img/a%20b.png#!htsinline "
+    "1x, img/big.png#!htsinline 2x\">\n"
+    "<link rel=\"icon\" href=\"icon.png#!htsinline\">\n"
+    "<link rel=\"preload\" as=\"font\" href=\"font/f.woff2#!htsinline\">\n"
    "<img src=\"data:image/gif;base64,QUJD\">\n"
    /* Each has a real file where its guard's removal would land it; without
       that they stay links either way, the target merely being absent. */
    "<img src=\"http://example.com/x.png\">\n"
    "<img src=\"//example.com/x.png\">\n"
-    "<input type=\"image\" src=\"img/in.png\">\n"
+    "<input type=\"image\" src=\"img/in.png#!htsinline\">\n"
    /* Lazy loading: src is the placeholder, the real image rides data-src. */
-    "<img src=\"img/ph.png\" data-src=\"img/lz.png\" "
-    "data-srcset=\"img/lz2.png 2x\" lowsrc=\"img/low.png\">\n"
-    "<object data=\"img/ob.png\"></object>\n"
-    "<embed src=\"img/em.png\">\n"
-    "<img data-src=\"other.html\">\n"
+    "<img src=\"img/ph.png#!htsinline\" data-src=\"img/lz.png#!htsinline\" "
+    "data-srcset=\"img/lz2.png#!htsinline 2x\" "
+    "lowsrc=\"img/low.png#!htsinline\">\n"
+    "<object data=\"img/ob.png#!htsinline\"></object>\n"
+    "<embed src=\"img/em.png#!htsinline\">\n"
+    "<img data-src=\"other.html#!htsinline\">\n"
    /* What a first pass emits: re-resolving it is what a second pass must not
       do, and the fallback type would inline whatever the walk found. */
    "<link rel=\"stylesheet\" href=\"data:text/css;base64,QUJD\">\n"
-    "<video poster=\"img/po.png\" controls>"
-    "<source src=\"v.mp4\" type=\"video/mp4\"></video>\n"
-    "<svg><image href=\"img/sv.png\"/></svg>\n"
-    "<table background=\"img/bg.png\"><tr><td>x</td></tr></table>\n"
+    "<video poster=\"img/po.png#!htsinline\" controls>"
+    "<source src=\"v.mp4#!htsinline\" type=\"video/mp4\"></video>\n"
+    "<svg><image href=\"img/sv.png#!htsinline\"/></svg>\n"
+    "<table background=\"img/bg.png#!htsinline\"><tr><td>x</td></tr></table>\n"
    /* The second is what bites: drop the clamp and its leading ".." lands it
       back on <root>/img/a b.png. The first can only 404 either way. */
-    "<img src=\"../escape.png\">\n"
-    "<img src=\"../img/a%20b.png\">\n"
+    "<img src=\"../escape.png#!htsinline\">\n"
+    "<img src=\"../img/a%20b.png#!htsinline\">\n"
    "<a href=\"img/a%20b.png\">link</a>\n"
-    "<script src=\"js/app.js\"></script>\n"
+    "<script src=\"js/app.js#!htsinline\"></script>\n"
    "<script>var s = \"</scripting>\"; var t = \"<img src='img/a%20b.png'>\";"
    "</script>\n"
-    "<img src=\"missing.png\" >\n"
-    "<!--><img src=\"img/a%20b.png\">\n"
-    "<div style=\"background:url(img/a%20b.png)\"></div>\n"
-    "<div style='content:\"x\"; background:url(img/a%20b.png)'></div>\n"
+    "<img src=\"missing.png#!htsinline\" >\n"
+    "<!--><img src=\"img/a%20b.png#!htsinline\">\n"
+    "<div style=\"background:url(img/a%20b.png#!htsinline)\"></div>\n"
+    "<div style='content:\"x\"; "
+    "background:url(img/a%20b.png#!htsinline)'></div>\n"
    "</body></html>\n";

 /* Lay a small mirror down under root. */
@@ -5362,18 +5095,20 @@ static void sf_fixture(const char *root) {
     inlined stylesheet could not embed has to come out relative to the page,
     not to the stylesheet, or it dangles. */
  static const char css[] =
-      "@import \"sub/nested.css\";\n"
-      "@import url(\"sub/two.css\");\n"
+      "@import \"sub/nested.css#!htsinline\";\n"
+      "@import url(\"sub/two.css#!htsinline\");\n"
      "@import \"a\\\"url(../img/a b.png)b.css\";\n"
-      "@font-face { font-family: f; src: url(../font/f.woff2); }\n"
-      "body { background: url(../img/a b.png); }\n"
-      "div { background: url(../img/big.png); }\n"
+      "@font-face { font-family: f; src: url(../font/f.woff2#!htsinline); }\n"
+      "body { background: url(../img/a%20b.png#!htsinline); }\n"
+      "div { background: url(../img/big.png#!htsinline); }\n"
      "/* url(../img/never.png) */\n";
-  static const char nested[] = "div { background: url(../../img/a b.png); }\n";
-  static const char two[] = "p { background: url(../../img/a b.png); }\n";
-  static const char deep[] =
-      "<html><head><link rel=\"stylesheet\" href=\"../../css/main.css\">\n"
-      "</head><body>d</body></html>\n";
+  static const char nested[] =
+      "div { background: url(../../img/a%20b.png#!htsinline); }\n";
+  static const char two[] =
+      "p { background: url(../../img/a%20b.png#!htsinline); }\n";
+  static const char deep[] = "<html><head><link rel=\"stylesheet\" "
+                             "href=\"../../css/main.css#!htsinline\">\n"
+                             "</head><body>d</body></html>\n";
  static const char js[] = "var app = 1;\n";
  char big[4096];

@@ -5515,13 +5250,10 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
  sf_check(strstr(out, "var t = \"<img src='img/a%20b.png'>\";") != NULL,
           "script body rewritten past a </scripting> lookalike");

-  /* Nothing an attribute value cannot hold: url() stays unquoted, and a quote
-     that was already in the CSS is escaped. */
-  sf_check(strstr(out, "url(\"data:") == NULL,
-           "a quoted url() would end a style attribute");
-  sf_check(strstr(out, "style=\"content:&quot;x&quot;; background:url(data:") !=
-               NULL,
-           "quote inside a rewritten style attribute not escaped");
+  /* Only the marked reference is touched: the value keeps its own quoting, so
+     nothing can be emitted that the attribute could not already hold. */
+  sf_check(strstr(out, "style='content:\"x\"; background:url(data:") != NULL,
+           "style attribute re-quoted instead of substituted in place");

  sf_check(strstr(out, "img/big.png 2x") != NULL, "over-cap asset inlined");
  sf_check(strstr(out, " 1x") != NULL, "srcset descriptor lost");
@@ -5541,7 +5273,7 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
             "url() inside a CSS comment was rewritten");
    sf_check(strstr(css, "url(data:font/woff2;base64,") != NULL,
             "@font-face src not inlined");
-    sf_check(strstr(css, "@import url(data:text/css;base64,") != NULL,
+    sf_check(strstr(css, "@import url(\"data:text/css;base64,") != NULL,
             "@import url() form not inlined");
    sf_check(strstr(css, "url(../img/a b.png)b.css") != NULL,
             "url() inside a string with an escaped quote was rewritten");
@@ -5630,10 +5362,14 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
    (void) singlefile_rewrite_html(opt, root, page, sf_page,
                                   sizeof(sf_page) - 1,
                                   SINGLEFILE_MAX_PAGE_SIZE, &verbatim);
-    sf_check(StringLength(verbatim) == sizeof(sf_page) - 1 &&
-                 memcmp(StringBuff(verbatim), sf_page, sizeof(sf_page) - 1) ==
-                     0,
+    /* Mark-transparent, not byte-transparent: a reference that cannot be
+       inlined loses its mark and keeps everything else. */
+    sf_check(StringLength(verbatim) == sizeof(sf_page) - 1 -
+                                           sf_count(sf_page, SINGLEFILE_MARK) *
+                                               (int) strlen(SINGLEFILE_MARK),
             "a page with nothing to inline was re-serialized differently");
+    sf_check(strstr(StringBuff(verbatim), SINGLEFILE_MARK) == NULL,
+             "an un-inlinable reference kept its mark");
    StringFree(verbatim);
  }
  (void) outlen;
@@ -5642,11 +5378,13 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
     run is the control: it proves the fan-out is real, so the small one was
     cut short by the budget and not by the fixture. */
  {
-    static const char bomb_css[] = "@import \"b.css\";@import \"b.css\";"
-                                   "@import \"b.css\";@import \"b.css\";\n";
-    static const char bomb_html[] = "<html><head>"
-                                    "<link rel=\"stylesheet\" href=\"b.css\">"
-                                    "</head></html>\n";
+    static const char bomb_css[] =
+        "@import \"b.css#!htsinline\";@import \"b.css#!htsinline\";"
+        "@import \"b.css#!htsinline\";@import \"b.css#!htsinline\";\n";
+    static const char bomb_html[] =
+        "<html><head>"
+        "<link rel=\"stylesheet\" href=\"b.css#!htsinline\">"
+        "</head></html>\n";
    const size_t css_len = sizeof(bomb_css) - 1;
    String small = STRING_EMPTY, large = STRING_EMPTY;

@@ -6345,8 +6083,6 @@ static const struct selftest_entry {
     st_contentcodings},
    {"robots", "", "robots.txt RFC 9309 Allow/Disallow precedence self-test",
     st_robots},
-    {"sitemap", "",
-     "sitemap <loc> extraction, caps and robots.txt Sitemap:", st_sitemap},
    {"ftp-line", "", "get_ftp_line bounds a hostile FTP reply line",
     st_ftpline},
    {"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser},
--- a/src/htssinglefile.c
+++ b/src/htssinglefile.c
@@ -27,9 +27,11 @@ Please visit our Website: http://www.httrack.com

 /* ------------------------------------------------------------ */
 /* --single-file asset inliner. See htssinglefile.h.
-   Runs on the finished tree, so resolving a reference is path arithmetic
-   clamped to the mirror root; anything else (absolute, scheme-bearing, already
-   data:) is left alone, which is what makes a second --update run a no-op. */
+   htsparse marked every inlinable reference while saving, so this is a
+   substitution over those marks: no HTML and no CSS is parsed here. Resolving
+   a mark is path arithmetic clamped to the mirror root; anything the mark
+   cannot be resolved to keeps its link, which is what makes a second --update
+   run a no-op. */
 /* ------------------------------------------------------------ */

 #define HTS_INTERNAL_BYTECODE
@@ -50,14 +52,9 @@ Please visit our Website: http://www.httrack.com
 #define SF_C_FONT 2
 #define SF_C_CSS 4
 #define SF_C_JS 8
-#define SF_C_ANY (SF_C_IMAGE | SF_C_FONT | SF_C_CSS | SF_C_JS)

-/* Shape of an attribute value. */
-#define SF_V_URL 0    /* one URL */
-#define SF_V_SRCSET 1 /* HTML srcset candidate list */
-#define SF_V_CSS 2    /* CSS declarations (style="...") */
-
-/* @import chains deeper than this keep their links. */
+/* An asset that carries marks of its own (a stylesheet importing another) is
+   expanded before encoding; deeper than this it keeps its links. */
 #define SF_MAX_CSS_DEPTH 4

 /* Bounds on hostile input: a longer reference, or one resolving to more
@@ -65,9 +62,6 @@ Please visit our Website: http://www.httrack.com
 #define SF_MAX_REF 4096
 #define SF_MAX_COMPONENTS 128

-/* Attributes collected from one start tag before it is re-emitted. */
-#define SF_MAX_ATTRS 64
-
 /* Over-cap assets reported per pass; beyond that the log would carry one line
   per referencing page. */
 #define SF_MAX_WARN 32
@@ -83,7 +77,41 @@ typedef struct sf_ctx {
 } sf_ctx;

 /* ------------------------------------------------------------ */
-/* Spans                                                         */
+/* Marks                                                         */
+/* ------------------------------------------------------------ */
+
+/* (tag, attribute) pairs htsparse detects that name a page or a media stream
+   rather than an asset; a NULL tag matches any. Everything else it detects is
+   offered to the pass, which decides from the referenced file's own MIME
+   type, so a new row in hts_detect[] is covered without a change here. */
+static const struct sf_deny_rule {
+  const char *tag;
+  const char *attr;
+} sf_deny_rules[] = {
+    {"a", "href"},      {"area", "href"},   {"iframe", "src"},
+    {"frame", "src"},   {"applet", "code"}, {NULL, "longdesc"},
+    {NULL, "usemap"},   {NULL, "archive"},  {NULL, "profile"},
+    {NULL, "codebase"},
+};
+
+hts_boolean singlefile_may_inline(const char *tag_start, const char *attr) {
+  size_t i;
+
+  if (attr == NULL)
+    return HTS_FALSE;
+  for (i = 0; i < sizeof(sf_deny_rules) / sizeof(sf_deny_rules[0]); i++) {
+    const struct sf_deny_rule *const r = &sf_deny_rules[i];
+
+    if (r->tag != NULL && (tag_start == NULL || !check_tag(tag_start, r->tag)))
+      continue;
+    if (rech_tageq(attr, r->attr))
+      return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* ------------------------------------------------------------ */
+/* Paths                                                         */
 /* ------------------------------------------------------------ */

 static int sf_is_space(int c) {
@@ -92,45 +120,6 @@ static int sf_is_space(int c) {

 static int sf_is_sep(int c) { return c == '/' || c == '\\'; }

-/* Case-insensitive equality between the span [p,p+n) and a lowercase literal.
- */
-static hts_boolean sf_span_eq(const char *p, size_t n, const char *lit) {
-  size_t i;
-
-  for (i = 0; i < n; i++) {
-    if (lit[i] == '\0' || tolower((unsigned char) p[i]) != lit[i])
-      return HTS_FALSE;
-  }
-  return lit[n] == '\0' ? HTS_TRUE : HTS_FALSE;
-}
-
-/* Case-insensitive "does the span start with the lowercase literal". */
-static hts_boolean sf_span_starts(const char *p, size_t n, const char *lit) {
-  size_t i;
-
-  for (i = 0; lit[i] != '\0'; i++) {
-    if (i >= n || tolower((unsigned char) p[i]) != lit[i])
-      return HTS_FALSE;
-  }
-  return HTS_TRUE;
-}
-
-/* Case-insensitive search for a lowercase literal inside a span. */
-static hts_boolean sf_span_has(const char *p, size_t n, const char *lit) {
-  const size_t l = strlen(lit);
-  size_t i;
-
-  for (i = 0; l <= n && i + l <= n; i++) {
-    if (sf_span_starts(p + i, n - i, lit))
-      return HTS_TRUE;
-  }
-  return HTS_FALSE;
-}
-
-/* ------------------------------------------------------------ */
-/* Paths                                                         */
-/* ------------------------------------------------------------ */
-
 /* Copy path into out with '/' separators and no trailing one. */
 static void sf_normalize_path(const char *path, String *out) {
  size_t i;
@@ -382,24 +371,25 @@ static void sf_warn_oversize(sf_ctx *ctx, const char *path, LLint size,
                path, size, cap);
 }

-static void sf_rewrite_css(sf_ctx *ctx, const char *base_dir, int depth,
-                           const char *css, size_t len, String *out);
+/* ------------------------------------------------------------ */
+/* Substitution                                                  */
+/* ------------------------------------------------------------ */
+
+static void sf_expand(sf_ctx *ctx, const char *base_dir, int depth,
+                      const char *body, size_t len, String *out);

 /* Replace the reference [ref,ref+reflen), resolved against base_dir, with its
-   data: URI appended to out. classes gates the acceptable MIME classes;
-   fallback_mime types an asset whose class cannot be guessed (NULL: give up).
-   rebase_dir re-expresses an un-inlinable asset relative to that directory.
-   A data: URL's path is opaque, so nothing relative inside an inlined
-   stylesheet resolves anyway; this only aims it at a lenient resolver's base.
-   Returns HTS_TRUE if out received a replacement. */
+   data: URI appended to out. rebase_dir re-expresses an un-inlinable asset
+   relative to that directory: a data: URL's path is opaque, so nothing
+   relative inside an inlined stylesheet resolves anyway; this only aims it at
+   a lenient resolver's base. Returns HTS_TRUE if out received a data: URI. */
 static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
-                             size_t reflen, int classes,
-                             const char *fallback_mime, const char *rebase_dir,
-                             int depth, String *out) {
+                             size_t reflen, const char *rebase_dir, int depth,
+                             String *out) {
  String path = STRING_EMPTY;
  char mime[HTS_MIMETYPE_SIZE];
-  char *body;
-  size_t body_len = 0;
+  char *file;
+  size_t file_len = 0;
  LLint size, cap;
  int cls;
  hts_boolean done = HTS_FALSE;
@@ -409,13 +399,9 @@ static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
    return HTS_FALSE;
  }
  cls = sf_mime_class(ctx->opt, StringBuff(path), mime, sizeof(mime));
-  if (cls == 0 && fallback_mime != NULL) {
-    cls = classes;
-    strlcpybuff(mime, fallback_mime, sizeof(mime));
-  }
  cap = ctx->opt->single_file_max_size;
  size = fsize_utf8(StringBuff(path));
-  if ((cls & classes) == 0 || size < 0)
+  if (cls == 0 || size < 0)
    goto fallback;
  if (size > cap || size > SINGLEFILE_HARD_MAX_SIZE) {
    sf_warn_oversize(ctx, StringBuff(path), size, cap);
@@ -425,11 +411,11 @@ static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
    sf_warn_oversize(ctx, StringBuff(path), size, ctx->budget);
    goto fallback;
  }
-  body = sf_readfile(StringBuff(path), &body_len);
-  if (body == NULL)
+  file = sf_readfile(StringBuff(path), &file_len);
+  if (file == NULL)
    goto fallback;
-  /* Charged before the nested rewrite: an @import chain otherwise spends what
-     its ancestors already claimed, and the budget ends up negative. */
+  /* Charged before the nested expansion: an @import chain otherwise spends
+     what its ancestors already claimed, and the budget ends up negative. */
  ctx->budget -= size;
  /* Encode into a scratch String: a failed encode must leave out untouched,
     not a truncated "data:...;base64," with no payload. */
@@ -437,20 +423,20 @@ static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
    String payload = STRING_EMPTY;

    StringClear(payload);
-    if ((cls & SF_C_CSS) != 0 && depth < SF_MAX_CSS_DEPTH) {
+    if ((cls & (SF_C_CSS | SF_C_JS)) != 0 && depth < SF_MAX_CSS_DEPTH) {
      String nested = STRING_EMPTY;
      String nested_dir = STRING_EMPTY;

      sf_dirname(StringBuff(path), &nested_dir);
      StringClear(nested);
-      sf_rewrite_css(ctx, StringBuff(nested_dir), depth + 1, body, body_len,
-                     &nested);
+      sf_expand(ctx, StringBuff(nested_dir), depth + 1, file, file_len,
+                &nested);
      done = sf_append_base64(&payload, StringBuffRW(nested),
                              StringLength(nested));
      StringFree(nested);
      StringFree(nested_dir);
    } else {
-      done = sf_append_base64(&payload, body, body_len);
+      done = sf_append_base64(&payload, file, file_len);
    }
    if (done) {
      StringCat(*out, "data:");
@@ -460,7 +446,7 @@ static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
    }
    StringFree(payload);
  }
-  freet(body);
+  freet(file);
  if (done) {
    ctx->inlined++;
    StringFree(path);
@@ -484,547 +470,43 @@ fallback:
  return done;
 }

-/* ------------------------------------------------------------ */
-/* CSS                                                           */
-/* ------------------------------------------------------------ */
-
-/* Rewrite the CSS in [css,css+len) into out, resolving its references against
-   base_dir. depth counts the @import nesting. */
-static void sf_rewrite_css(sf_ctx *ctx, const char *base_dir, int depth,
-                           const char *css, size_t len, String *out) {
-  size_t i = 0;
-  int importing = 0; /* set for exactly one iteration: see the @import branch */
-
-  while (i < len) {
-    const int is_import = importing;
-
-    importing = 0;
-    /* A comment or a string is copied verbatim: a url( inside either is not a
-       reference. */
-    if (css[i] == '/' && i + 1 < len && css[i + 1] == '*') {
-      const size_t start = i;
-
-      i += 2;
-      while (i + 1 < len && !(css[i] == '*' && css[i + 1] == '/'))
-        i++;
-      i = i + 1 < len ? i + 2 : len;
-      StringMemcat(*out, css + start, i - start);
-      continue;
-    }
-    if (css[i] == '@' && sf_span_starts(css + i, len - i, "@import")) {
-      size_t j = i + 7;
-
-      while (j < len && sf_is_space((unsigned char) css[j]))
-        j++;
-      /* url(...) here names a stylesheet, so hand it to the url( branch with
-         the import's class rather than the image/font one. */
-      if (j < len && sf_span_starts(css + j, len - j, "url(")) {
-        StringMemcat(*out, css + i, j - i);
-        i = j;
-        importing = 1;
-        continue;
-      }
-      if (j < len && (css[j] == '"' || css[j] == '\'')) {
-        const char quote = css[j];
-        const size_t vstart = j + 1;
-        size_t vend = vstart;
-
-        while (vend < len && css[vend] != quote && css[vend] != '\n') {
-          if (css[vend] == '\\' && vend + 1 < len)
-            vend++; /* an escaped quote does not end the string */
-          vend++;
-        }
-        if (vend < len && css[vend] == quote) {
-          String repl = STRING_EMPTY;
-
-          StringClear(repl);
-          StringMemcat(*out, css + i, j - i);
-          if (sf_inline(ctx, base_dir, css + vstart, vend - vstart, SF_C_CSS,
-                        "text/css", NULL, depth, &repl)) {
-            StringCat(*out, "\"");
-            StringMemcat(*out, StringBuff(repl), StringLength(repl));
-            StringCat(*out, "\"");
-          } else {
-            StringMemcat(*out, css + j, vend + 1 - j);
-          }
-          StringFree(repl);
-          i = vend + 1;
-          continue;
-        }
-      }
-      StringMemcat(*out, css + i, 7);
-      i += 7;
-      continue;
-    }
-    if ((css[i] == 'u' || css[i] == 'U') &&
-        sf_span_starts(css + i, len - i, "url(") &&
-        (i == 0 || (!isalnum((unsigned char) css[i - 1]) && css[i - 1] != '-' &&
-                    css[i - 1] != '_'))) {
-      const size_t start = i;
-      size_t j = i + 4, vstart, vend;
-      char quote = '\0';
-
-      while (j < len && sf_is_space((unsigned char) css[j]))
-        j++;
-      if (j < len && (css[j] == '"' || css[j] == '\'')) {
-        quote = css[j];
-        j++;
-      }
-      vstart = j;
-      while (j < len && (quote != '\0' ? css[j] != quote : css[j] != ')') &&
-             css[j] != '\n') {
-        if (quote != '\0' && css[j] == '\\' && j + 1 < len)
-          j++; /* an escaped quote does not end the string */
-        j++;
-      }
-      vend = j;
-      if (quote != '\0' && j < len && css[j] == quote) {
-        j++;
-        while (j < len && sf_is_space((unsigned char) css[j]))
-          j++;
-      }
-      if (j < len && css[j] == ')') {
-        String repl = STRING_EMPTY;
-
-        j++;
-        StringClear(repl);
-        /* Emit unquoted: the CSS could be a style="..." attribute value, where
-           a quote would end the attribute. A data: payload and an escaped path
-           both stay inside the unquoted url-token alphabet. */
-        if (sf_inline(ctx, base_dir, css + vstart, vend - vstart,
-                      is_import ? SF_C_CSS : SF_C_IMAGE | SF_C_FONT,
-                      is_import ? "text/css" : NULL,
-                      !is_import && depth > 0 ? ctx->page_dir : NULL, depth,
-                      &repl)) {
-          StringCat(*out, "url(");
-          StringMemcat(*out, StringBuff(repl), StringLength(repl));
-          StringCat(*out, ")");
-        } else {
-          StringMemcat(*out, css + start, j - start);
-        }
-        StringFree(repl);
-        i = j;
-        continue;
-      }
-      StringMemcat(*out, css + start, 4); /* unterminated url(: leave it */
-      i = start + 4;
-      continue;
-    }
-    if (css[i] == '"' || css[i] == '\'') {
-      const char quote = css[i];
-      const size_t start = i;
-
-      i++;
-      while (i < len && css[i] != quote && css[i] != '\n') {
-        if (css[i] == '\\' && i + 1 < len)
-          i++;
-        i++;
-      }
-      if (i < len && css[i] == quote)
-        i++;
-      StringMemcat(*out, css + start, i - start);
-      continue;
-    }
-    StringAddchar(*out, css[i]);
-    i++;
-  }
+/* The mark ends a reference; these end the token the mark was appended to. */
+static int sf_is_ref_delim(int c) {
+  return sf_is_space(c) || c == '"' || c == '\'' || c == '(' || c == ')' ||
+         c == '=' || c == ',' || c == '<' || c == '>' || c == ';';
 }

-/* ------------------------------------------------------------ */
-/* HTML                                                          */
-/* ------------------------------------------------------------ */
+/* Copy [body,body+len) to out, replacing each marked reference by its data:
+   URI, or by the bare reference when it cannot be inlined. */
+static void sf_expand(sf_ctx *ctx, const char *base_dir, int depth,
+                      const char *body, size_t len, String *out) {
+  const size_t marklen = strlen(SINGLEFILE_MARK);
+  size_t i = 0, flushed = 0;

-/* srcset is "url [descriptor]" candidates separated by commas. Per the HTML
-   candidate-parsing rules the URL token runs to the next whitespace and may
-   carry trailing commas, so an emitted "data:...;base64,AAAA 2x" round-trips.
- */
-static void sf_rewrite_srcset(sf_ctx *ctx, const char *v, size_t len,
-                              String *out) {
-  size_t i = 0;
-  int first = 1;
+  while (i + marklen <= len) {
+    size_t start, tail;

-  while (i < len) {
-    size_t ustart, uend, dstart, dend;
-
-    while (i < len && (sf_is_space((unsigned char) v[i]) || v[i] == ','))
-      i++;
-    if (i >= len)
-      break;
-    ustart = i;
-    while (i < len && !sf_is_space((unsigned char) v[i]))
-      i++;
-    uend = i;
-    while (uend > ustart && v[uend - 1] == ',')
-      uend--;
-    dstart = dend = i;
-    if (uend == i) { /* no trailing comma, so a descriptor may follow */
-      while (i < len && v[i] != ',')
-        i++;
-      dstart = uend;
-      dend = i;
-      while (dend > dstart && sf_is_space((unsigned char) v[dend - 1]))
-        dend--;
-      while (dstart < dend && sf_is_space((unsigned char) v[dstart]))
-        dstart++;
-    }
-    if (!first)
-      StringCat(*out, ", ");
-    first = 0;
-    if (!sf_inline(ctx, ctx->page_dir, v + ustart, uend - ustart, SF_C_IMAGE,
-                   NULL, NULL, 0, out))
-      StringMemcat(*out, v + ustart, uend - ustart);
-    if (dend > dstart) {
-      StringAddchar(*out, ' ');
-      StringMemcat(*out, v + dstart, dend - dstart);
-    }
-  }
-}
-
-/* (tag, attribute) pairs naming an inlinable asset; a NULL tag matches any.
-   Deliberately absent though hts_detect[] downloads them: a@href, iframe@src
-   and longdesc name pages, dynsrc and non-image source@src name media,
-   track@src has no inlinable MIME class, usemap is a same-document fragment,
-   archive lists applet jars. link@href is decided from its rel
-   (sf_link_classes); base@href needs no rule, htsparse drops it when saving. */
-static const struct sf_attr_rule {
-  const char *tag;
-  const char *attr;
-  int classes;
-  int shape;
-} sf_attr_rules[] = {
-    {"img", "src", SF_C_IMAGE, SF_V_URL},
-    {"img", "srcset", SF_C_IMAGE, SF_V_SRCSET},
-    {"source", "src", SF_C_IMAGE, SF_V_URL},
-    {"source", "srcset", SF_C_IMAGE, SF_V_SRCSET},
-    {"input", "src", SF_C_IMAGE, SF_V_URL},
-    {"video", "poster", SF_C_IMAGE, SF_V_URL},
-    {"image", "href", SF_C_IMAGE, SF_V_URL},
-    {"image", "xlink:href", SF_C_IMAGE, SF_V_URL},
-    {"object", "data", SF_C_IMAGE, SF_V_URL},
-    {"embed", "src", SF_C_IMAGE, SF_V_URL},
-    {"script", "src", SF_C_JS, SF_V_URL},
-    /* Lazy loading: src is a placeholder and the real image rides one of
-       these. The image class keeps a lazy <iframe> or <script> a link. */
-    {NULL, "data-src", SF_C_IMAGE, SF_V_URL},
-    {NULL, "data-srcset", SF_C_IMAGE, SF_V_SRCSET},
-    {NULL, "lowsrc", SF_C_IMAGE, SF_V_URL},
-    {NULL, "background", SF_C_IMAGE, SF_V_URL},
-    {NULL, "style", 0, SF_V_CSS},
-};
-
-typedef struct sf_attr {
-  const char *pre; /* whitespace before the attribute */
-  size_t pre_len;
-  const char *raw; /* name plus value, with the original quoting */
-  size_t raw_len;
-  const char *name;
-  size_t name_len;
-  const char *value;
-  size_t value_len;
-  hts_boolean has_value;
-} sf_attr;
-
-/* Accepted classes and fallback type for <link href>, read from its rel: a
-   navigational rel (canonical, alternate, next) yields 0 and keeps the link. */
-static int sf_link_classes(const sf_attr *attrs, int nattrs,
-                           const char **fallback) {
-  int i;
-
-  *fallback = NULL;
-  for (i = 0; i < nattrs; i++) {
-    if (!sf_span_eq(attrs[i].name, attrs[i].name_len, "rel"))
-      continue;
-    if (sf_span_has(attrs[i].value, attrs[i].value_len, "stylesheet")) {
-      *fallback = "text/css";
-      return SF_C_CSS;
-    }
-    if (sf_span_has(attrs[i].value, attrs[i].value_len, "icon"))
-      return SF_C_IMAGE;
-    if (sf_span_has(attrs[i].value, attrs[i].value_len, "preload") ||
-        sf_span_has(attrs[i].value, attrs[i].value_len, "prefetch"))
-      return SF_C_ANY;
-    return 0;
-  }
-  return 0;
-}
-
-static const struct sf_attr_rule *sf_find_rule(const char *tag, size_t tag_len,
-                                               const char *attr,
-                                               size_t attr_len) {
-  size_t i;
-
-  for (i = 0; i < sizeof(sf_attr_rules) / sizeof(sf_attr_rules[0]); i++) {
-    const struct sf_attr_rule *const r = &sf_attr_rules[i];
-
-    if (r->tag != NULL && !sf_span_eq(tag, tag_len, r->tag))
-      continue;
-    if (sf_span_eq(attr, attr_len, r->attr))
-      return r;
-  }
-  return NULL;
-}
-
-/* Emit one attribute, substituting its value when a rule applies. */
-static void sf_emit_attr(sf_ctx *ctx, const char *tag, size_t tag_len,
-                         const sf_attr *a, int link_classes,
-                         const char *link_fallback, String *out) {
-  const struct sf_attr_rule *rule = NULL;
-  const char *fallback = NULL;
-  String repl = STRING_EMPTY;
-  hts_boolean done = HTS_FALSE;
-  int classes;
-
-  StringMemcat(*out, a->pre, a->pre_len);
-  if (!a->has_value) {
-    StringMemcat(*out, a->raw, a->raw_len);
-    return;
-  }
-  if (sf_span_eq(tag, tag_len, "link") &&
-      sf_span_eq(a->name, a->name_len, "href")) {
-    classes = link_classes;
-    fallback = link_fallback;
-  } else {
-    rule = sf_find_rule(tag, tag_len, a->name, a->name_len);
-    classes = rule != NULL ? rule->classes : 0;
-    if (rule != NULL && rule->classes == SF_C_JS)
-      fallback = "text/javascript";
-  }
-  if (rule == NULL && classes == 0) {
-    StringMemcat(*out, a->raw, a->raw_len);
-    return;
-  }
-  StringClear(repl);
-  if (rule != NULL && (rule->shape == SF_V_SRCSET || rule->shape == SF_V_CSS)) {
-    /* These always produce a value; substitute only when it really differs, so
-       an untouched attribute keeps its original spelling and quoting. */
-    if (rule->shape == SF_V_SRCSET)
-      sf_rewrite_srcset(ctx, a->value, a->value_len, &repl);
-    else
-      sf_rewrite_css(ctx, ctx->page_dir, 0, a->value, a->value_len, &repl);
-    done = StringLength(repl) != a->value_len ||
-                   memcmp(StringBuff(repl), a->value, a->value_len) != 0
-               ? HTS_TRUE
-               : HTS_FALSE;
-  } else {
-    done = sf_inline(ctx, ctx->page_dir, a->value, a->value_len, classes,
-                     fallback, NULL, 0, &repl);
-  }
-  if (done) {
-    size_t n;
-
-    /* Always re-quote with '"', so a value carrying a literal quote (a CSS
-       declaration that was single-quoted in the source) must be escaped. */
-    StringMemcat(*out, a->name, a->name_len);
-    StringCat(*out, "=\"");
-    for (n = 0; n < StringLength(repl); n++) {
-      const char c = StringSub(repl, n);
-
-      if (c == '"')
-        StringCat(*out, "&quot;");
-      else
-        StringAddchar(*out, c);
-    }
-    StringCat(*out, "\"");
-  } else {
-    StringMemcat(*out, a->raw, a->raw_len);
-  }
-  StringFree(repl);
-}
-
-/* Elements whose content is raw text, never markup. */
-static const char *const sf_rawtext_tags[] = {"script", "style", "textarea",
-                                              "title"};
-
-/* The raw-text tag matching [tag,tag+len), or NULL. */
-static const char *sf_rawtext_tag(const char *tag, size_t len) {
-  size_t i;
-
-  for (i = 0; i < sizeof(sf_rawtext_tags) / sizeof(sf_rawtext_tags[0]); i++) {
-    if (sf_span_eq(tag, len, sf_rawtext_tags[i]))
-      return sf_rawtext_tags[i];
-  }
-  return NULL;
-}
-
-static void sf_rewrite_html_(sf_ctx *ctx, const char *p, size_t len,
-                             String *out) {
-  size_t i = 0;
-
-  while (i < len) {
-    sf_attr attrs[SF_MAX_ATTRS];
-    const char *tag, *rawtext, *link_fallback = NULL;
-    size_t tag_len, tag_start;
-    int nattrs = 0, k, link_classes = 0;
-    hts_boolean selfclose = HTS_FALSE, overflow = HTS_FALSE;
-
-    if (p[i] != '<') {
-      StringAddchar(*out, p[i]);
+    if (memcmp(body + i, SINGLEFILE_MARK, marklen) != 0) {
      i++;
      continue;
    }
-    if (i + 4 <= len && memcmp(p + i, "<!--", 4) == 0) {
-      const size_t start = i;
-
-      i += 4;
-      /* "<!-->" and "<!--->" are empty comments, not unterminated ones. */
-      if (i < len && p[i] == '>')
-        i++;
-      else if (i + 2 <= len && memcmp(p + i, "->", 2) == 0)
-        i += 2;
-      else {
-        while (i + 3 <= len && memcmp(p + i, "-->", 3) != 0)
-          i++;
-        i = i + 3 <= len ? i + 3 : len;
-      }
-      StringMemcat(*out, p + start, i - start);
-      continue;
-    }
-    if (i + 1 >= len || !isalpha((unsigned char) p[i + 1])) {
-      const size_t start = i; /* doctype, processing instruction, end tag */
-
-      i++;
-      while (i < len && p[i] != '>') {
-        /* A '>' inside a quoted value does not close the tag. */
-        if (p[i] == '"' || p[i] == '\'') {
-          const char q = p[i++];
-
-          while (i < len && p[i] != q)
-            i++;
-        }
-        if (i < len)
-          i++;
-      }
-      if (i < len)
-        i++;
-      StringMemcat(*out, p + start, i - start);
-      continue;
-    }
-
-    /* Start tag. Collect every attribute before deciding: <link href> depends
-       on the rel, which may come after it. */
-    tag_start = i;
-    i++;
-    tag = p + i;
-    while (i < len && !sf_is_space((unsigned char) p[i]) && p[i] != '>' &&
-           p[i] != '/')
-      i++;
-    tag_len = (size_t) (p + i - tag);
-    while (i < len) {
-      const char *const pre = p + i;
-      size_t pre_len;
-      sf_attr *a;
-
-      while (i < len && sf_is_space((unsigned char) p[i]))
-        i++;
-      pre_len = (size_t) (p + i - pre);
-      if (i >= len || p[i] == '>' ||
-          (p[i] == '/' && i + 1 < len && p[i + 1] == '>')) {
-        i = (size_t) (pre - p); /* the caller copies this whitespace */
-        break;
-      }
-      if (nattrs == SF_MAX_ATTRS) {
-        overflow = HTS_TRUE; /* keep parsing, but stop recording */
-        nattrs--;
-      }
-      a = &attrs[nattrs++];
-      a->pre = pre;
-      a->pre_len = pre_len;
-      a->name = p + i;
-      a->raw = p + i;
-      a->has_value = HTS_FALSE;
-      a->value = NULL;
-      a->value_len = 0;
-      while (i < len && !sf_is_space((unsigned char) p[i]) && p[i] != '=' &&
-             p[i] != '>' && !(p[i] == '/' && i + 1 < len && p[i + 1] == '>'))
-        i++;
-      a->name_len = (size_t) (p + i - a->name);
-      if (a->name_len == 0) { /* a stray '=' or '/': copy it and move on */
-        i++;
-        a->name_len = 1;
-        a->raw_len = 1;
-        continue;
-      }
-      {
-        const size_t save = i;
-
-        while (i < len && sf_is_space((unsigned char) p[i]))
-          i++;
-        if (i >= len || p[i] != '=') {
-          i = save;
-          a->raw_len = a->name_len;
-          continue;
-        }
-      }
-      i++;
-      while (i < len && sf_is_space((unsigned char) p[i]))
-        i++;
-      if (i < len && (p[i] == '"' || p[i] == '\'')) {
-        const char quote = p[i];
-
-        i++;
-        a->value = p + i;
-        while (i < len && p[i] != quote)
-          i++;
-        a->value_len = (size_t) (p + i - a->value);
-        if (i < len)
-          i++;
-      } else {
-        a->value = p + i;
-        while (i < len && !sf_is_space((unsigned char) p[i]) && p[i] != '>')
-          i++;
-        a->value_len = (size_t) (p + i - a->value);
-      }
-      a->has_value = HTS_TRUE;
-      a->raw_len = (size_t) (p + i - a->raw);
-    }
-    if (overflow) { /* pathological tag: copy the whole of it untouched */
-      StringMemcat(*out, p + tag_start, i - tag_start);
-    } else {
-      if (sf_span_eq(tag, tag_len, "link"))
-        link_classes = sf_link_classes(attrs, nattrs, &link_fallback);
-      StringAddchar(*out, '<');
-      StringMemcat(*out, tag, tag_len);
-      for (k = 0; k < nattrs; k++)
-        sf_emit_attr(ctx, tag, tag_len, &attrs[k], link_classes, link_fallback,
-                     out);
-    }
-    while (i < len && sf_is_space((unsigned char) p[i])) {
-      StringAddchar(*out, p[i]);
-      i++;
-    }
-    if (i < len && p[i] == '/') {
-      selfclose = HTS_TRUE;
-      StringAddchar(*out, '/');
-      i++;
-    }
-    if (i < len && p[i] == '>') {
-      StringAddchar(*out, '>');
-      i++;
-    }
-
-    rawtext = selfclose ? NULL : sf_rawtext_tag(tag, tag_len);
-    if (rawtext != NULL) {
-      const size_t body = i;
-      size_t end = i;
-
-      while (end < len) {
-        if (p[end] == '<' && end + 2 < len && p[end + 1] == '/' &&
-            sf_span_starts(p + end + 2, len - end - 2, rawtext)) {
-          const size_t after = end + 2 + strlen(rawtext);
-
-          if (after >= len || sf_is_space((unsigned char) p[after]) ||
-              p[after] == '>' || p[after] == '/')
-            break;
-        }
-        end++;
-      }
-      if (sf_span_eq(tag, tag_len, "style"))
-        sf_rewrite_css(ctx, ctx->page_dir, 0, p + body, end - body, out);
-      else
-        StringMemcat(*out, p + body, end - body);
-      i = end;
+    for (start = i; start > flushed && !sf_is_ref_delim(body[start - 1]);
+         start--)
+      ;
+    /* htsparse writes the fragment and the kept query string after the mark;
+       a data: URI has no use for either. */
+    for (tail = i + marklen; tail < len && !sf_is_ref_delim(body[tail]); tail++)
+      ;
+    StringMemcat(*out, body + flushed, start - flushed);
+    if (!sf_inline(ctx, base_dir, body + start, i - start,
+                   depth > 0 ? ctx->page_dir : NULL, depth, out)) {
+      StringMemcat(*out, body + start, i - start);
+      StringMemcat(*out, body + i + marklen, tail - i - marklen);
    }
+    i = tail;
+    flushed = i;
  }
+  StringMemcat(*out, body + flushed, len - flushed);
 }

 /* ------------------------------------------------------------ */
@@ -1051,15 +533,7 @@ hts_boolean singlefile_rewrite_html(httrackp *opt, const char *root,
  ctx.warn_budget = &budget;
  ctx.budget = page_budget;
  ctx.inlined = 0;
-  /* A UTF-16/32 page is not ASCII-delimited, so byte scanning would corrupt
-     it; every ASCII-compatible charset is safe. */
-  if (html_len >= 2 &&
-      (((unsigned char) html[0] == 0xff && (unsigned char) html[1] == 0xfe) ||
-       ((unsigned char) html[0] == 0xfe && (unsigned char) html[1] == 0xff))) {
-    StringMemcat(*out, html, html_len);
-  } else {
-    sf_rewrite_html_(&ctx, html, html_len, out);
-  }
+  sf_expand(&ctx, StringBuff(dir), 0, html, html_len, out);
  StringFree(nroot);
  StringFree(npage);
  StringFree(dir);
--- a/src/htssinglefile.h
+++ b/src/htssinglefile.h
@@ -55,18 +55,27 @@ extern "C" {
   so a few hundred bytes of hostile CSS can otherwise ask for gigabytes. */
 #define SINGLEFILE_MAX_PAGE_SIZE (64 * 1024 * 1024)

+/* Fragment htsparse appends to a saved reference the pass may inline. A
+   fragment and not a scheme, so the mirror still resolves if the pass never
+   runs, and so a mirrored .css or .js keeps its marks across an --update. */
+#define SINGLEFILE_MARK "#!htsinline"
+
+/* HTS_TRUE if a reference htsparse detected in this context may become a
+   data: URI. tag_start points at the '<' of the enclosing start tag, or NULL
+   when there is none (inside a stylesheet or a script); attr at the attribute
+   name. Everything htsparse detects is inlinable unless it names a page. */
+hts_boolean singlefile_may_inline(const char *tag_start, const char *attr);
+
 /* Rewrite every HTML page the mirror produced. No-op unless opt->single_file;
   call once the tree is final, after the update purge. */
 void singlefile_process_mirror(httrackp *opt);

-/* Rewrite one HTML document held in memory, appending the result to out.
-   root is the mirror directory that references may not escape; page_path is
-   the document's own path under it (both UTF-8, '/' or native separators).
-   page_budget caps the total inlined bytes, since nested @import fans out
+/* Expand the marks in the document held in memory, appending the result to
+   out. root is the mirror directory that references may not escape; page_path
+   is the document's own path under it (both UTF-8, '/' or native separators).
+   page_budget caps the total inlined bytes, since a nested @import fans out
   multiplicatively; the mirror pass passes SINGLEFILE_MAX_PAGE_SIZE.
-   Returns HTS_TRUE if at least one reference was replaced; out may still
-   differ from the input when that is HTS_FALSE, since a style or srcset value
-   is re-serialized in place. */
+   Returns HTS_TRUE if at least one reference was replaced. */
 hts_boolean singlefile_rewrite_html(httrackp *opt, const char *root,
                                    const char *page_path, const char *html,
                                    size_t html_len, LLint page_budget,
--- a/src/htssitemap.c
+++ b/src/htssitemap.c
@@ -1,615 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 2026 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: sitemap ingestion (sitemaps.org 0.9)                   */
-/* Author: Xavier Roche                                         */
-/* ------------------------------------------------------------ */
-
-#define HTS_INTERNAL_BYTECODE
-
-#include "htscore.h"
-#include "htssitemap.h"
-
-#include "htsbase.h"
-#include "htscodec.h"
-#include "htsencoding.h"
-#include "htsfilters.h"
-#include "htshash.h"
-#include "htsmodules.h"
-#include "htslib.h"
-#include "htsrobots.h"
-#include "htssafe.h"
-#include "htstools.h"
-
-#include <ctype.h>
-#include <string.h>
-
-/* One queued sitemap document awaiting ingestion. */
-typedef struct sitemap_doc {
-  char adr[HTS_URLMAXSIZE];
-  char fil[HTS_URLMAXSIZE];
-  int level;
-  hts_sitemap_source src;
-  hts_boolean done;
-  struct sitemap_doc *next;
-} sitemap_doc;
-
-struct hts_sitemap_state {
-  sitemap_doc *docs;
-  int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */
-  int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */
-  hts_boolean probe_done;    /* the robots.txt probe has been answered */
-  hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */
-  /* The crawl's own start URL. Seeded URLs are judged against it, so a site
-     cannot widen a subtree crawl by putting its sitemap at the root. */
-  char anchor_adr[HTS_URLMAXSIZE];
-  char anchor_fil[HTS_URLMAXSIZE];
-};
-typedef struct hts_sitemap_state hts_sitemap_state;
-
-/* --------------------------------------------------------------------- */
-/* Document parsing (no engine state: fuzzable and self-testable)         */
-/* --------------------------------------------------------------------- */
-
-/* Accept only an absolute http(s) URL with no space or control byte. */
-static hts_boolean sitemap_url_ok(const char *url) {
-  const char *p;
-
-  if (!strfield(url, "http://") && !strfield(url, "https://"))
-    return HTS_FALSE;
-  for (p = url; *p != '\0'; p++) {
-    if ((unsigned char) *p <= ' ' || (unsigned char) *p == 0x7f)
-      return HTS_FALSE;
-  }
-  return HTS_TRUE;
-}
-
-/* Skip to the character after the next '>' at or after p, or NULL. */
-static const char *sitemap_tag_end(const char *p, const char *end) {
-  while (p < end && *p != '>')
-    p++;
-  return p < end ? p + 1 : NULL;
-}
-
-/* HTS_TRUE when the document's root element is `name`. Skips the XML
-   declaration, comments and processing instructions first, so a comment
-   mentioning the other root element cannot decide the document type. */
-static hts_boolean sitemap_root_is(const char *doc, size_t size,
-                                   const char *name) {
-  const size_t nlen = strlen(name);
-  size_t i = 0;
-
-  if (size >= 3 && memcmp(doc, "\xef\xbb\xbf", 3) == 0)
-    i = 3; /* UTF-8 BOM */
-  while (i < size) {
-    if (isspace((unsigned char) doc[i])) {
-      i++;
-    } else if (doc[i] != '<') {
-      return HTS_FALSE; /* character data before any element: not XML */
-    } else if (i + 4 <= size && memcmp(doc + i, "<!--", 4) == 0) {
-      const char *const e = hts_memstr(doc + i, size - i, "-->", 3);
-
-      if (e == NULL)
-        return HTS_FALSE;
-      i = (size_t) (e - doc) + 3;
-    } else if (i + 2 <= size && (doc[i + 1] == '?' || doc[i + 1] == '!')) {
-      while (i < size && doc[i] != '>')
-        i++;
-      i++;
-    } else {
-      size_t j = i + 1;
-
-      /* an optional namespace prefix: <sm:sitemapindex> is the same element */
-      while (j < size && doc[j] != ':' && doc[j] != '>' &&
-             !isspace((unsigned char) doc[j]))
-        j++;
-      if (j >= size || doc[j] != ':')
-        j = i + 1;
-      else
-        j++;
-      return j + nlen <= size && memcmp(doc + j, name, nlen) == 0 &&
-                     (j + nlen == size ||
-                      isspace((unsigned char) doc[j + nlen]) ||
-                      doc[j + nlen] == '>' || doc[j + nlen] == '/')
-                 ? HTS_TRUE
-                 : HTS_FALSE;
-    }
-  }
-  return HTS_FALSE;
-}
-
-/* Decompress a gzip-framed body into a fresh buffer. The 64 MiB cap is what
-   binds in practice; deflate tops out near 1032:1, so the tree's codec budget
-   only matters as the shared policy for a coding that could go further. */
-static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) {
-  const LLint budget = hts_codec_maxout((LLint) size);
-  size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES
-                   ? (size_t) budget
-                   : (size_t) HTS_SITEMAP_MAX_BYTES;
-  char *out;
-  size_t n;
-
-  if (cap == 0)
-    return NULL;
-  out = malloct(cap + 1);
-  if (out == NULL)
-    return NULL;
-  n = hts_codec_head(HTS_CODEC_DEFLATE, body, size, out, cap);
-  if (n == 0) {
-    freet(out);
-    return NULL;
-  }
-  out[n] = '\0';
-  *outsize = n;
-  return out;
-}
-
-int hts_sitemap_scan(const char *body, size_t size, int maxurls,
-                     hts_boolean *is_index, hts_sitemap_handler handler,
-                     void *arg) {
-  char *unpacked = NULL;
-  const char *doc;
-  const char *end;
-  const char *p;
-  int n = 0;
-
-  if (is_index != NULL)
-    *is_index = HTS_FALSE;
-  if (body == NULL || size < 2 || handler == NULL)
-    return 0;
-
-  /* Content-Encoding gzip is undone upstream; only the container is left. */
-  if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) {
-    unpacked = sitemap_gunzip(body, size, &size);
-    if (unpacked == NULL)
-      return -1;
-    doc = unpacked;
-  } else {
-    if (size > (size_t) HTS_SITEMAP_MAX_BYTES)
-      size = (size_t) HTS_SITEMAP_MAX_BYTES;
-    doc = body;
-  }
-  end = doc + size;
-
-  /* Set before the first callback: the handler reads the verdict. */
-  if (is_index != NULL)
-    *is_index = sitemap_root_is(doc, size, "sitemapindex");
-
-  for (p = doc; n < maxurls;) {
-    const char *loc = hts_memstr(p, (size_t) (end - p), "<loc", 4);
-    const char *val;
-    const char *stop;
-    size_t len;
-    char BIGSTK url[HTS_URLMAXSIZE];
-
-    if (loc == NULL)
-      break;
-    /* "<loc>" or "<loc xmlns:..>", never "<location>" */
-    if (loc + 4 >= end || (loc[4] != '>' && !isspace((unsigned char) loc[4]))) {
-      p = loc + 4;
-      continue;
-    }
-    val = sitemap_tag_end(loc + 4, end);
-    if (val == NULL)
-      break;
-    for (stop = val; stop < end && *stop != '<'; stop++)
-      ;
-    /* No closing tag: truncated document, so the value may be a partial URL. */
-    if (stop == end)
-      break;
-    p = stop;
-    while (val < stop && isspace((unsigned char) *val))
-      val++;
-    while (stop > val && isspace((unsigned char) *(stop - 1)))
-      stop--;
-    len = (size_t) (stop - val);
-    /* Overflow-safe: the untrusted length alone against the room left. */
-    if (len == 0 || len >= sizeof(url))
-      continue;
-    memcpy(url, val, len);
-    url[len] = '\0';
-    /* hts_unescapeEntities decodes in place and tolerates src == dest; a
-       reference to a control byte survives as one and sitemap_url_ok drops it.
-     */
-    if (hts_unescapeEntities(url, url, sizeof(url)) != 0 ||
-        !sitemap_url_ok(url))
-      continue;
-    n++;
-    if (!handler(arg, url))
-      break;
-  }
-
-  if (unpacked != NULL)
-    freet(unpacked);
-  return n;
-}
-
-/* --------------------------------------------------------------------- */
-/* Engine glue                                                           */
-/* --------------------------------------------------------------------- */
-
-static hts_sitemap_state *sitemap_get_state(httrackp *opt) {
-  if (opt->sitemap_state == NULL)
-    opt->sitemap_state = calloct(1, sizeof(hts_sitemap_state));
-  return (hts_sitemap_state *) opt->sitemap_state;
-}
-
-static sitemap_doc *sitemap_find(httrackp *opt, const char *adr,
-                                 const char *fil) {
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-  sitemap_doc *d;
-
-  if (st == NULL)
-    return NULL;
-  for (d = st->docs; d != NULL; d = d->next) {
-    if (strfield2(d->adr, adr) && strcmp(d->fil, fil) == 0)
-      return d;
-  }
-  return NULL;
-}
-
-/* Who asked for this document decides how far it is gated. The wizard proper
-   is not usable here: it wants a referring link, and its up/down travel rules
-   would judge a child sitemap against the parent sitemap's directory. */
-static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr,
-                                         const char *fil,
-                                         hts_sitemap_source src) {
-  /* adr and fil are each capped just under HTS_URLMAXSIZE, and lfull prefixes
-     a scheme and a slash on top of both: 2 * HTS_URLMAXSIZE does not fit. */
-  char BIGSTK l[HTS_URLMAXSIZE * 2 + 16], lfull[HTS_URLMAXSIZE * 2 + 16];
-  int jokdepth = 0, jok;
-
-  hts_boolean refused;
-
-  /* The user naming a sitemap is the same intent as naming a start URL, which
-     the wizard admits unconditionally. */
-  if (src == HTS_SITEMAP_SRC_USER)
-    return HTS_TRUE;
-  strcpybuff(l, jump_identification_const(adr));
-  if (*fil != '/')
-    strcatbuff(l, "/");
-  strcatbuff(l, fil);
-  strcpybuff(lfull, link_has_authority(adr) ? "" : "http://");
-  strcatbuff(lfull, adr);
-  if (*fil != '/')
-    strcatbuff(lfull, "/");
-  strcatbuff(lfull, fil);
-  jok = fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull,
-                         l, NULL, NULL, &jokdepth);
-  refused = (jok == -1) ? HTS_TRUE : HTS_FALSE;
-  if (refused) {
-    hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s",
-                  jokdepth + 1, adr, fil);
-    return HTS_FALSE;
-  }
-  /* A Sitemap: line, or a sitemapindex entry, is the site inviting the fetch;
-     a Disallow elsewhere in the same file does not retract it. The well-known
-     location is only ever a guess, so there a Disallow wins. */
-  if (src == HTS_SITEMAP_SRC_GUESSED &&
-      hts_robots_forbids(opt, adr, fil, (jok != 0) ? HTS_TRUE : HTS_FALSE,
-                         refused)) {
-    hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr,
-                  fil);
-    return HTS_FALSE;
-  }
-  return HTS_TRUE;
-}
-
-/* Record the link with save="" so the body stays in memory: a sitemap is
-   ingested, never mirrored. */
-static hts_boolean sitemap_queue_(httrackp *opt, const char *adr,
-                                  const char *fil, int level,
-                                  hts_sitemap_source src, hts_boolean link_it) {
-  hts_sitemap_state *const st = sitemap_get_state(opt);
-  sitemap_doc *d;
-
-  if (st == NULL)
-    return HTS_FALSE;
-  if (st->ndocs >= HTS_SITEMAP_MAX_DOCS || level > HTS_SITEMAP_MAX_LEVEL) {
-    hts_log_print(opt, LOG_WARNING, "Sitemap: cap reached, skipping %s%s", adr,
-                  fil);
-    return HTS_FALSE;
-  }
-  if (strlen(adr) >= sizeof(d->adr) || strlen(fil) >= sizeof(d->fil))
-    return HTS_FALSE;
-  if (sitemap_find(opt, adr, fil) != NULL)
-    return HTS_FALSE;
-  if (!sitemap_fetch_allowed(opt, adr, fil, src))
-    return HTS_FALSE;
-  d = calloct(1, sizeof(sitemap_doc));
-  if (d == NULL)
-    return HTS_FALSE;
-  strcpybuff(d->adr, adr);
-  strcpybuff(d->fil, fil);
-  d->level = level;
-  d->src = src;
-  d->next = st->docs;
-  st->docs = d;
-  st->ndocs++;
-
-  if (!link_it)
-    return HTS_TRUE;
-  if (!hts_record_link(opt, adr, fil, "", "", "", NULL))
-    return HTS_FALSE;
-  heap_top()->testmode = 0;
-  heap_top()->link_import = 0;
-  heap_top()->depth = opt->depth + 1;
-  heap_top()->pass2 = 0;
-  heap_top()->retry = opt->retry;
-  heap_top()->premier = heap_top_index();
-  heap_top()->precedent = heap_top_index();
-  hts_log_print(opt, LOG_INFO, "Sitemap: queued %s%s", adr, fil);
-  return HTS_TRUE;
-}
-
-static hts_boolean sitemap_queue(httrackp *opt, const char *adr,
-                                 const char *fil, int level,
-                                 hts_sitemap_source src) {
-  return sitemap_queue_(opt, adr, fil, level, src, HTS_TRUE);
-}
-
-void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
-                          const char *newadr, const char *newfil) {
-  sitemap_doc *const d = sitemap_find(opt, adr, fil);
-
-  if (d == NULL || d->done)
-    return;
-  d->done = HTS_TRUE; /* the body lives at the target now */
-  /* The engine already queued the target link, so only the marking moves. */
-  (void) sitemap_queue_(opt, newadr, newfil, d->level, d->src, HTS_FALSE);
-  hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil,
-                newadr, newfil);
-}
-
-void hts_sitemap_seed(httrackp *opt, const char *starturl) {
-  char BIGSTK url[HTS_URLMAXSIZE * 2];
-  lien_adrfil af;
-
-  if (StringNotEmpty(opt->sitemap_url)) {
-    if (strlen(StringBuff(opt->sitemap_url)) >= sizeof(url)) {
-      hts_log_print(opt, LOG_ERROR, "Sitemap URL too long");
-    } else {
-      strcpybuff(url, StringBuff(opt->sitemap_url));
-      if (strstr(url, ":/") == NULL)
-        hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url);
-      else if (ident_url_absolute(url, &af) >= 0)
-        (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_USER);
-    }
-  }
-  if (starturl == NULL || starturl[0] == '\0' ||
-      strlen(starturl) >= sizeof(url))
-    return;
-  strcpybuff(url, starturl);
-  if (ident_url_absolute(url, &af) < 0)
-    return;
-  {
-    hts_sitemap_state *const st = sitemap_get_state(opt);
-
-    if (st != NULL && strlen(af.adr) < sizeof(st->anchor_adr) &&
-        strlen(af.fil) < sizeof(st->anchor_fil)) {
-      strcpybuff(st->anchor_adr, af.adr);
-      strcpybuff(st->anchor_fil, af.fil);
-    }
-  }
-  if (!opt->sitemap)
-    return;
-  /* Answered in hts_sitemap_robots, once the parsed rules are installed. */
-  if (hts_record_link(opt, af.adr, "/robots.txt", "", "", "", NULL)) {
-    heap_top()->testmode = 0;
-    heap_top()->link_import = 0;
-    heap_top()->depth = 0;
-    heap_top()->pass2 = 0;
-    heap_top()->retry = opt->retry;
-    heap_top()->premier = heap_top_index();
-    heap_top()->precedent = heap_top_index();
-    /* Claim the host so the parser does not queue robots.txt a second time. */
-    if (opt->robotsptr != NULL)
-      (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, "");
-  }
-}
-
-void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps) {
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-  int queued = 0;
-
-  if (st == NULL || !opt->sitemap || st->probe_done ||
-      !strfield2(st->anchor_adr, adr))
-    return;
-  st->probe_done = HTS_TRUE;
-  if (sitemaps != NULL) {
-    const char *p = sitemaps;
-
-    while (*p != '\0') {
-      const char *const eol = strchr(p, '\n');
-      const size_t len = eol != NULL ? (size_t) (eol - p) : strlen(p);
-      char BIGSTK line[HTS_URLMAXSIZE];
-      lien_adrfil af;
-
-      if (len > 0 && len < sizeof(line)) {
-        memcpy(line, p, len);
-        line[len] = '\0';
-        /* Same host: a Sitemap: line must not aim the fetcher elsewhere. */
-        if (sitemap_url_ok(line) && ident_url_absolute(line, &af) >= 0 &&
-            strfield2(af.adr, adr) &&
-            sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_DECLARED))
-          queued++;
-      }
-      if (eol == NULL)
-        break;
-      p = eol + 1;
-    }
-  }
-  if (queued == 0 && !st->fallback_done) {
-    st->fallback_done = HTS_TRUE;
-    if (sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_SITEMAP_SRC_GUESSED))
-      queued++;
-  }
-  hts_log_print(opt, LOG_NOTICE, "Sitemap: %d sitemap(s) queued for %s", queued,
-                adr);
-}
-
-hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
-                                const char *fil) {
-  const sitemap_doc *const d = sitemap_find(opt, adr, fil);
-
-  return d != NULL && !d->done ? HTS_TRUE : HTS_FALSE;
-}
-
-/* Handler context: seeding URLs from one document. */
-typedef struct sitemap_ingest_ctx {
-  httrackp *opt;
-  htsmoduleStruct *str;
-  const char *adr; /* host of the document being ingested */
-  int level;
-  hts_boolean is_index;
-  int accepted; /* URLs seeded or documents queued, not merely parsed */
-} sitemap_ingest_ctx;
-
-/* A <loc> of a <urlset>: hand it to the wizard as a top-level seed.
-   The wizard is pointed at the crawl's own start URL, not at the sitemap: the
-   site picks where its sitemap lives, so anchoring travel there would let a
-   root sitemap widen a subtree crawl to the whole host. The URL then becomes
-   its own anchor, exactly as a command-line seed does. */
-static hts_boolean sitemap_seed_url(void *arg, const char *url) {
-  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
-  httrackp *const opt = c->opt;
-  hts_sitemap_state *const st = sitemap_get_state(opt);
-  char BIGSTK buff[HTS_URLMAXSIZE];
-  int before;
-
-  if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) {
-    hts_log_print(opt, LOG_WARNING,
-                  "Sitemap: URL cap reached, ignoring the rest");
-    return HTS_FALSE;
-  }
-  /* strcpybuff aborts rather than truncating: never feed it unchecked input. */
-  if (strlen(url) >= sizeof(buff))
-    return HTS_TRUE;
-  st->nurls++;
-  strcpybuff(buff, url);
-  before = opt->lien_tot;
-  if (htsAddLink(c->str, buff))
-    c->accepted++;
-  if (opt->lien_tot > before)
-    heap_top()->premier = heap_top_index(); /* a seed anchors on itself */
-  return HTS_TRUE;
-}
-
-/* A <loc> of a <sitemapindex>: cross-host children are dropped, so a hostile
-   sitemap cannot aim the fetcher elsewhere. */
-static hts_boolean sitemap_seed_child(void *arg, const char *url) {
-  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
-  char BIGSTK buff[HTS_URLMAXSIZE];
-  lien_adrfil af;
-
-  if (strlen(url) >= sizeof(buff))
-    return HTS_TRUE;
-  strcpybuff(buff, url);
-  if (ident_url_absolute(buff, &af) < 0)
-    return HTS_TRUE;
-  if (!strfield2(af.adr, c->adr)) {
-    hts_log_print(c->opt, LOG_WARNING,
-                  "Sitemap: ignoring off-host child sitemap %s%s", af.adr,
-                  af.fil);
-    return HTS_TRUE;
-  }
-  if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1,
-                    HTS_SITEMAP_SRC_DECLARED))
-    c->accepted++;
-  return HTS_TRUE;
-}
-
-/* The scan classifies the document before the first callback. */
-static hts_boolean sitemap_seed_any(void *arg, const char *url) {
-  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
-
-  return c->is_index ? sitemap_seed_child(arg, url)
-                     : sitemap_seed_url(arg, url);
-}
-
-void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
-                        const char *fil, const char *body, size_t size) {
-  sitemap_doc *const d = sitemap_find(opt, adr, fil);
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-  sitemap_ingest_ctx ctx;
-  int n, anchor, saved_depth;
-
-  if (d == NULL || d->done)
-    return;
-  d->done = HTS_TRUE;
-  /* str->ptr_ is a scratch int owned by the caller, so nothing else moves. */
-  anchor = *str->ptr_;
-  if (st != NULL && st->anchor_adr[0] != '\0' && opt->hash != NULL) {
-    const int i = hash_read((const hash_struct *) opt->hash, st->anchor_adr,
-                            st->anchor_fil, 1);
-
-    if (i >= 0)
-      anchor = i;
-  }
-  *str->ptr_ = anchor;
-  /* Borrow the anchor's position but keep a seed's full depth budget. */
-  saved_depth = heap(anchor)->depth;
-  heap(anchor)->depth = opt->depth + 1;
-  ctx.opt = opt;
-  ctx.str = str;
-  ctx.adr = adr;
-  ctx.level = d->level;
-  ctx.is_index = HTS_FALSE;
-  ctx.accepted = 0;
-
-  n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index,
-                       sitemap_seed_any, &ctx);
-  heap(anchor)->depth = saved_depth;
-  if (n < 0) {
-    hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr,
-                  fil);
-    return;
-  }
-  if (ctx.is_index)
-    hts_log_print(opt, LOG_NOTICE,
-                  "Sitemap: %d of %d child sitemap(s) listed by %s%s",
-                  ctx.accepted, n, adr, fil);
-  else
-    hts_log_print(opt, LOG_NOTICE, "Sitemap: %d of %d URL(s) added from %s%s",
-                  ctx.accepted, n, adr, fil);
-}
-
-void hts_sitemap_free(httrackp *opt) {
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-
-  if (st == NULL)
-    return;
-  while (st->docs != NULL) {
-    sitemap_doc *const next = st->docs->next;
-
-    freet(st->docs);
-    st->docs = next;
-  }
-  freet(opt->sitemap_state);
-  opt->sitemap_state = NULL;
-}
--- a/src/htssitemap.h
+++ b/src/htssitemap.h
@@ -1,108 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 2026 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* HTTrack sitemap ingestion (sitemaps.org 0.9). Internal, not installed.
-   Reads <urlset>/<sitemapindex> documents, plain or gzip-framed, and feeds
-   their <loc> URLs to the crawl as top-level seeds. The whole input is
-   attacker-controlled, so every entry point below is capped. */
-/* ------------------------------------------------------------ */
-
-#ifndef HTS_SITEMAP_DEFH
-#define HTS_SITEMAP_DEFH
-
-#include "htsdefines.h"
-#include "htsopt.h"
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/* Caps. sitemaps.org allows 50000 URLs and 50 MB uncompressed per document;
-   the byte cap sits above that so a conformant sitemap always fits. */
-#define HTS_SITEMAP_MAX_URLS_DOC 50000           /* <loc> per document */
-#define HTS_SITEMAP_MAX_URLS_TOTAL 200000        /* <loc> per mirror */
-#define HTS_SITEMAP_MAX_DOCS 256                 /* documents per mirror */
-#define HTS_SITEMAP_MAX_LEVEL 4                  /* sitemapindex nesting */
-#define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */
-
-/* Who asked for a sitemap document, which decides how far its fetch is gated.
-   The user naming one is the same intent as a start URL; a site declaring one
-   invites the fetch; the well-known location is only ever our guess. */
-typedef enum {
-  HTS_SITEMAP_SRC_USER,     /**< --sitemap-url */
-  HTS_SITEMAP_SRC_DECLARED, /**< a Sitemap: line or a sitemapindex entry */
-  HTS_SITEMAP_SRC_GUESSED   /**< the /sitemap.xml fallback */
-} hts_sitemap_source;
-
-/* Per-URL handler; returning HTS_FALSE stops the scan. */
-typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url);
-
-/* Scan one sitemap document, plain or gzip-framed, handing every acceptable
-   absolute http(s) <loc> URL to `handler`. Stops after `maxurls` URLs, or when
-   the handler refuses. `is_index` (optional) reports a <sitemapindex>, whose
-   URLs are child sitemaps rather than pages. Returns the number of URLs handed
-   out, or -1 when the document could not be decompressed within the caps. */
-int hts_sitemap_scan(const char *body, size_t size, int maxurls,
-                     hts_boolean *is_index, hts_sitemap_handler handler,
-                     void *arg);
-
-/* --- Engine glue (needs a live httrackp). --- */
-
-/* Queue the first sitemap document of the mirror: the explicit --sitemap-url,
-   or the start host's /robots.txt probe for --sitemap. `starturl` is the first
-   command-line seed. No-op when neither option is set. */
-void hts_sitemap_seed(httrackp *opt, const char *starturl);
-
-/* Act on the start host's robots.txt once its rules are installed: queue the
-   Sitemap: URLs it names (newline-separated, from robots_parse), or the
-   well-known /sitemap.xml when it names none. No-op unless --sitemap. */
-void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps);
-
-/* Carry the sitemap marking of (adr,fil) over to the target of a redirect the
-   engine has already queued, so a moved sitemap is still ingested. */
-void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
-                          const char *newadr, const char *newfil);
-
-/* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */
-hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
-                                const char *fil);
-
-/* Ingest a fetched sitemap document (or the robots.txt probe): seed its URLs
-   through the wizard via htsAddLink, and queue nested sitemaps. `str` supplies
-   the parser context of the document being processed. */
-void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
-                        const char *fil, const char *body, size_t size);
-
-/* Release the ingestion state held in opt (NULL-safe, idempotent). */
-void hts_sitemap_free(httrackp *opt);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif
--- a/src/htsthread.c
+++ b/src/htsthread.c
@@ -44,18 +44,9 @@ Please visit our Website: http://www.httrack.com
 #endif
 #endif

-/* Outstanding threads, counted at spawn rather than by the child at entry, so
-   that a caller which spawns and immediately waits still joins them (#747). */
 static int process_chain = 0;
 static htsmutex process_chain_mutex = HTSMUTEX_INIT;

-static void process_chain_add(int delta) {
-  hts_mutexlock(&process_chain_mutex);
-  process_chain += delta;
-  assertf(process_chain >= 0);
-  hts_mutexrelease(&process_chain_mutex);
-}
-
 HTSEXT_API void htsthread_wait(void) {
  htsthread_wait_n(0);
 }
@@ -108,12 +99,20 @@ static void *hts_entry_point(void *tharg)
  void *const arg = s_args->arg;
  void (*fun) (void *arg) = s_args->fun;

-  freet(tharg);
+  free(tharg);
+
+  hts_mutexlock(&process_chain_mutex);
+  process_chain++;
+  assertf(process_chain > 0);
+  hts_mutexrelease(&process_chain_mutex);

  /* run */
  fun(arg);

-  process_chain_add(-1);
+  hts_mutexlock(&process_chain_mutex);
+  process_chain--;
+  assertf(process_chain >= 0);
+  hts_mutexrelease(&process_chain_mutex);
 #ifdef _WIN32
  return 0;
 #else
@@ -123,20 +122,18 @@ static void *hts_entry_point(void *tharg)

 /* create a thread */
 HTSEXT_API int hts_newthread(void (*fun) (void *arg), void *arg) {
-  hts_thread_s *s_args = malloct(sizeof(hts_thread_s));
+  hts_thread_s *s_args = malloc(sizeof(hts_thread_s));

  assertf(s_args != NULL);
  s_args->arg = arg;
  s_args->fun = fun;
-  process_chain_add(1);
 #ifdef _WIN32
  {
    unsigned int idt;
    HANDLE handle =
      (HANDLE) _beginthreadex(NULL, 0, hts_entry_point, s_args, 0, &idt);
    if (handle == 0) {
-      process_chain_add(-1);
-      freet(s_args);
+      free(s_args);
      return -1;
    } else {
      /* detach the thread from the main process so that is can be independent */
@@ -154,8 +151,7 @@ HTSEXT_API int hts_newthread(void (*fun) (void *arg), void *arg) {
        || pthread_attr_setstacksize(&attr, stackSize) != 0
        || (retcode =
            pthread_create(&handle, &attr, hts_entry_point, s_args)) != 0) {
-      process_chain_add(-1);
-      freet(s_args);
+      free(s_args);
      return -1;
    } else {
      /* detach the thread from the main process so that is can be independent */
--- a/src/htstools.c
+++ b/src/htstools.c
@@ -276,21 +276,6 @@ int ident_url_relatif(const char *lien, const char *origin_adr,
  return ok;
 }

-/* Bounded substring search: bodies and archive records carry NUL bytes, so
-   strstr() would stop at the first one. */
-const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
-                       size_t nlen) {
-  size_t i;
-
-  if (nlen == 0 || haylen < nlen)
-    return NULL;
-  for (i = 0; i + nlen <= haylen; i++) {
-    if (hay[i] == *needle && memcmp(hay + i, needle, nlen) == 0)
-      return hay + i;
-  }
-  return NULL;
-}
-
 // créer dans s, à partir du chemin courant curr_fil, le lien vers link (absolu)
 // un ident_url_relatif a déja été fait avant, pour que link ne soit pas un chemin relatif
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
--- a/src/htstools.h
+++ b/src/htstools.h
@@ -61,11 +61,6 @@ typedef struct lien_adrfilsave lien_adrfilsave;
 int ident_url_relatif(const char *lien, const char *origin_adr,
                      const char *origin_fil,
                      lien_adrfil* const adrfil);
-/* Bounded substring search over data that may hold NUL bytes; NULL if absent.
- */
-const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
-                       size_t nlen);
-
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr);
 int link_has_authority(const char *lien);
 int link_has_authorization(const char *lien);
--- a/src/htsweb.c
+++ b/src/htsweb.c
@@ -101,8 +101,8 @@ static void htsweb_sig_brpipe(int code) {
  /* ignore */
 }

-/* Threads that never return; no wait may count on them draining. */
-static int nonjoinable_threads = 0;
+/* Number of background threads */
+static int background_threads = 0;

 /* Server/client ping handling */
 static htsmutex pingMutex = HTSMUTEX_INIT;
@@ -299,19 +299,15 @@ int main(int argc, char *argv[]) {

  /* pinger */
  if (parentPid > 0) {
-    if (hts_newthread(client_ping, (void *) (uintptr_t) parentPid) == 0) {
-#ifndef _WIN32
-      nonjoinable_threads++; /* client_ping() only ever leaves through exit() */
-#endif
-    }
+    hts_newthread(client_ping, (void *) (uintptr_t) parentPid);
+    background_threads++; /* Do not wait for this thread! */
    smallserver_setpinghandler(pingHandler, NULL);
  }

  /* launch */
  ret = help_server(argv[1], defaultPort, bindAddr);

-  /* Drain everything a mirror may still have in flight, the pinger aside. */
-  htsthread_wait_n(nonjoinable_threads);
+  htsthread_wait_n(background_threads - 1);
  hts_uninit();

 #ifdef _WIN32
@@ -386,6 +382,7 @@ void webhttrack_main(char *cmd) {
  commandRunning = 1;
  DEBUG(fprintf(stderr, "commandRunning=1\n"));
  hts_newthread(back_launch_cmd, (void *) strdup(cmd));
+  background_threads++; /* Do not wait for this thread! */
 }

 void webhttrack_lock(void) {
@@ -426,8 +423,8 @@ static int webhttrack_runmain(httrackp * opt, int argc, char **argv) {
  /* Rock'in! */
  ret = hts_main2(argc, argv, opt);

-  /* Wait for pending threads to finish; the pinger and this thread stay. */
-  htsthread_wait_n(nonjoinable_threads + 1);
+  /* Wait for pending threads to finish */
+  htsthread_wait_n(background_threads);

  return ret;
 }
--- a/src/htswizard.c
+++ b/src/htswizard.c
@@ -151,23 +151,6 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag,
  return HTS_FALSE;
 }

-/* The engine's robots.txt verdict for (adr,fil). Under HTS_ROBOTS_SOMETIMES an
-   explicit filter acceptance overrides the ban, which is why the filter outcome
-   is an input; the sitemap fetcher asks the same question outside the wizard.
- */
-hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
-                               hts_boolean filters_decided,
-                               hts_boolean filters_refused) {
-  if (!opt->robots || opt->robotsptr == NULL)
-    return HTS_FALSE;
-  if (checkrobots((robots_wizard *) opt->robotsptr, adr, fil) != -1)
-    return HTS_FALSE;
-  if (filters_decided && !filters_refused &&
-      opt->robots == HTS_ROBOTS_SOMETIMES)
-    return HTS_FALSE;
-  return HTS_TRUE;
-}
-
 static int hts_acceptlink_(httrackp * opt, int ptr,
                           const char *adr, const char *fil, const char *tag,
                           const char *attribute, int *set_prio_to,
@@ -593,26 +576,30 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
      }
    }
    // vérifier robots.txt
-    if (opt->robots && checkrobots(_ROBOTS, adr, fil) == -1) {
+    if (opt->robots) {
+      int r = checkrobots(_ROBOTS, adr, fil);
+
+      if (r == -1) {            // interdiction
 #if DEBUG_ROBOTS
-      printf("robots.txt forbidden: %s%s\n", adr, fil);
+        printf("robots.txt forbidden: %s%s\n", adr, fil);
 #endif
-      if (!hts_robots_forbids(opt, adr, fil,
-                              (!question && filters_answer) ? HTS_TRUE
-                                                            : HTS_FALSE,
-                              (forbidden_url == 1) ? HTS_TRUE : HTS_FALSE)) {
-        if (!forbidden_url) {
-          hts_log_print(
-              opt, LOG_DEBUG,
-              "Warning link followed against robots.txt: link %s at %s%s", l,
-              adr, fil);
+        // question résolue, par les filtres, et mode robot non strict
+        if ((!question) && (filters_answer) &&
+            (opt->robots == HTS_ROBOTS_SOMETIMES) && (forbidden_url != 1)) {
+          r = 0;                // annuler interdiction des robots
+          if (!forbidden_url) {
+            hts_log_print(opt, LOG_DEBUG,
+                          "Warning link followed against robots.txt: link %s at %s%s",
+                          l, adr, fil);
+          }
+        }
+        if (r == -1) {          // interdire
+          forbidden_url = 1;
+          question = 0;
+          hts_log_print(opt, LOG_DEBUG,
+                        "(robots.txt) forbidden link: link %s at %s%s", l, adr,
+                        fil);
        }
-      } else {
-        forbidden_url = 1;
-        question = 0;
-        hts_log_print(opt, LOG_DEBUG,
-                      "(robots.txt) forbidden link: link %s at %s%s", l, adr,
-                      fil);
      }
    }

--- a/src/htswizard.h
+++ b/src/htswizard.h
@@ -49,13 +49,6 @@ typedef struct httrackp httrackp;
 typedef struct lien_url lien_url;
 #endif

-/* The engine's robots.txt verdict for (adr,fil): HTS_TRUE when the fetch is
-   forbidden. `filters_decided`/`filters_refused` carry the filter outcome,
-   which overrides a ban under -s1 (HTS_ROBOTS_SOMETIMES). */
-hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
-                               hts_boolean filters_decided,
-                               hts_boolean filters_refused);
-
 int hts_acceptlink(httrackp * opt, int ptr,
                   const char *adr, const char *fil,
                   const char *tag, const char *attribute,
--- a/src/httrack.c
+++ b/src/httrack.c
@@ -306,8 +306,8 @@ int main(int argc, char **argv) {
    fprintf(stderr, "* %s\n", hts_errmsg(opt));
  }
  global_opt = NULL;
-  htsthread_wait(); /* pending threads still read opt */
  hts_free_opt(opt);
+  htsthread_wait();             /* wait for pending threads */
  hts_uninit();

 #ifdef _WIN32
--- a/src/libhttrack.vcxproj
+++ b/src/libhttrack.vcxproj
@@ -140,7 +140,6 @@
    <ClCompile Include="htszlib.c" />
    <ClCompile Include="htswarc.c" />
    <ClCompile Include="htschanges.c" />
-    <ClCompile Include="htssitemap.c" />
    <ClCompile Include="md5.c" />
    <ClCompile Include="minizip\ioapi.c" />
    <ClCompile Include="minizip\iowin32.c" />
--- a/tests/01_zlib-sitemap.test
+++ b/tests/01_zlib-sitemap.test
@@ -1,10 +0,0 @@
-#!/bin/bash
-#
-# Sitemap parser self-test: <loc> extraction, entity decoding, URL and length
-# rejections, the URL cap, gzip framing and robots.txt Sitemap: records.
-
-set -euo pipefail
-
-out=$(httrack -O /dev/null '-#test=sitemap')
-echo "$out"
-test "$out" = "sitemap self-test OK"
--- a/tests/65_port-siblings.test
+++ b/tests/65_port-siblings.test
@@ -79,29 +79,6 @@ done
 # 65535 is a valid port the old "< 65535" bound refused
 web_accepted 65535

-# --- htsserver returns from main() instead of blocking at exit -------------
-# The exit wait must exclude exactly the threads that never return (#753).
-# $tmp has no lang.def, so the server fails right after announcing and main()
-# reaches the wait on its own; rc, not EXITED, carries the verdict.
-
-web_exits() {
-    local rc=0
-
-    : >"$tmp/exit.log"
-    run_with_timeout 30 htsserver "$tmp" "$@" >"$tmp/exit.log" 2>&1 || rc=$?
-    grep -q "^EXITED" "$tmp/exit.log" ||
-        ! echo "FAIL: #753: htsserver ${*:-(no options)} never finished serving" || exit 1
-    # exactly 1, not merely "not 124": a crash or an assertf abort also escapes
-    # the wait, and would otherwise read as a pass
-    test "$rc" -eq 1 ||
-        ! echo "FAIL: #753: htsserver ${*:-(no options)} exited $rc, want 1" || exit 1
-}
-
-# no pinger: the excluded count went negative
-web_exits
-# a pinger, which never returns and so must stay excluded
-web_exits --ppid $$
-
 # --- proxytrack <proxy-addr:port> <ICP-addr:port> --------------------------
 # A bad argument falls through to the usage screen; it had no range check at
 # all, so 65616 quietly listened on port 80. A valid one binds and blocks.
--- a/tests/90_webhttrack-checkbox-clear.test
+++ b/tests/90_webhttrack-checkbox-clear.test
@@ -149,7 +149,6 @@ BOXES = [
    ("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
    ("toler", "TolerantRequests", "--tolerant", None),
    ("http10", "HTTP10", "--http-10", None),
-    ("sitemap", "Sitemap", "--sitemap", None),
    ("warc", "Warc", "--warc", None),
    ("changes", "Changes", "--changes", None),
    ("norecatch", "NoRecatch", "--do-not-recatch", None),
--- a/tests/95_local-sitemap.test
+++ b/tests/95_local-sitemap.test
@@ -1,134 +0,0 @@
-#!/bin/bash
-#
-# --sitemap seeds the crawl from robots.txt -> sitemapindex -> gzipped urlset.
-# start.html links to nothing, so orphan*.html can only arrive through the
-# sitemap; deep1.html proves the seeds keep a full depth budget under -r2, and
-# the off-host page and child sitemap must both be refused.
-
-set -eu
-
-: "${top_srcdir:=..}"
-
-crawl() { bash "$top_srcdir/tests/local-crawl.sh" "$@"; }
-
-# robots.txt Sitemap: -> index -> .xml.gz, seeds behaving like -r2 seeds. The
-# log assertions pin which route was taken: the two documents are served from
-# both /sitemapdir/index.xml and the well-known /sitemap.xml, so "some sitemap
-# was read" would pass either way. --not-found pins ingestion-only: the sitemap
-# documents feed the crawl but never land in the mirror.
-# --rerun also walks the update path over a mirror holding sitemap seeds.
-crawl --errors 0 --rerun \
-    --found 'sitemapdir/orphan1.html' \
-    --found 'sitemapdir/orphan2.html' \
-    --found 'sitemapdir/deep1.html' \
-    --not-found 'sitemapdir/index.xml' \
-    --not-found 'sitemapdir/pages.xml.gz' \
-    --log-found '2 of 3 URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
-    --log-found '1 of 2 child sitemap\(s\) listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \
-    --log-found 'ignoring off-host child sitemap' \
-    --log-not-found '/sitemap\.xml' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2
-
-# Negative control: without the option nothing but the start page is reached.
-crawl --errors 0 \
-    --found 'sitemapdir/start.html' \
-    --not-found 'sitemapdir/orphan1.html' \
-    --not-found 'sitemapdir/orphan2.html' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2
-
-# Sitemap URLs are not a filter bypass: a -*orphan2* rule still rejects one.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --not-found 'sitemapdir/orphan2.html' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 '-*orphan2*'
-
-# A robots.txt naming no sitemap falls back to the well-known /sitemap.xml.
-# The test server drops its Sitemap: record for this User-Agent.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemap\.xml' \
-    --log-not-found 'sitemapdir/index\.xml' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -F 'nositemap-agent'
-
-# --sitemap-url alone reads the named document and probes nothing.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --found 'sitemapdir/orphan2.html' \
-    --log-not-found 'sitemapdir/index\.xml' \
-    --log-not-found '/sitemap\.xml' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/pages.xml.gz'
-
-# The two options are additive: an unfetchable --sitemap-url is parsed as an
-# empty document and does not stop --sitemap from seeding the crawl.
-crawl --errors-content 1 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-found 'Sitemap: 0 of 0 URL\(s\) added' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/missing.xml'
-
-# The sitemapindex nesting cap stops the chain before its deepest urlset. The
-# positive control is the page one level inside the cap: without it, a cap
-# mutated to 0 (nothing followed) would pass this just as well.
-crawl --errors 0 \
-    --found 'sitemapdir/cap3.html' \
-    --not-found 'sitemapdir/cap4.html' \
-    --log-found 'Sitemap: cap reached' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/chain0.xml'
-
-# A site cannot widen a subtree crawl by putting its sitemap at the root: the
-# page below the start directory is seeded, the one above it is refused.
-crawl --errors 0 \
-    --found 'deep/dir/below.html' \
-    --not-found 'elsewhere/updir.html' \
-    httrack 'BASEURL/deep/dir/start.html' --sitemap -r3 -F 'scopesitemap-agent'
-
-# How far a sitemap fetch is gated depends on who asked for it, so the three
-# cases below have to differ: one "robots is honoured" assertion would hide a
-# regression in either direction. The harness disables robots, hence -s2 here.
-
-# We guessed /sitemap.xml, so a Disallow on it wins.
-crawl --errors 0 \
-    --not-found 'sitemapdir/orphan1.html' \
-    --log-found 'Sitemap: robots.txt forbids' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
-    -F 'denysitemap-agent'
-
-# The site declared it through a Sitemap: line, which the same Disallow does
-# not retract.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-not-found 'Sitemap: robots.txt forbids' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
-    -F 'denydeclared-agent'
-
-# The user named it, which is the same intent as a start URL: not refused
-# either, even though robots.txt forbids that exact path.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-not-found 'Sitemap: robots.txt forbids' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 -s2 -F 'denysitemap-agent' \
-    --sitemap-url 'BASEURL/sitemap.xml'
-
-# A child sitemap is a fetch like any other: a filter rejecting it stops the
-# whole subtree, so the page only it lists is never reached.
-crawl --errors 0 \
-    --not-found 'sitemapdir/gated.html' \
-    --log-found 'Sitemap: filter rule #[0-9]+ refuses' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 '-*filtered.xml*' \
-    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
-
-# Control: without that filter the same chain does reach the page.
-crawl --errors 0 \
-    --found 'sitemapdir/gated.html' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
-
-# A redirected sitemap is still ingested: the marking follows the 301.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-found 'moved\.xml redirects to .*pages\.xml\.gz' \
-    --log-found 'URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/moved.xml'
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -91,7 +91,6 @@ TESTS = \
 	01_engine-xfread.test \
 	01_zlib-acceptencoding.test \
 	01_zlib-warc.test \
-	01_zlib-sitemap.test \
 	01_zlib-warc-cdx.test \
 	01_zlib-warc-wacz.test \
 	01_zlib-contentcodings.test \
@@ -193,7 +192,6 @@ TESTS = \
 	91_webhttrack-directory.test \
 	92_local-proxytrack-ndx-fields.test \
 	93_local-changes.test \
-	94_local-single-file.test \
-	95_local-sitemap.test
+	94_local-single-file.test

 CLEANFILES = check-network_sh.cache
--- a/tests/local-server.py
+++ b/tests/local-server.py
@@ -18,7 +18,6 @@ import base64
 import gzip
 import hashlib
 import os
-import re
 import sys
 import time
 from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
@@ -543,36 +542,8 @@ class Handler(SimpleHTTPRequestHandler):
            return self.fail_cookie(name)
        self.send_html("\tThis is the secret.")

-    # A User-Agent carrying NO_SITEMAP_UA gets a robots.txt with no Sitemap:
-    # record, so a test can drive the /sitemap.xml fallback instead.
-    NO_SITEMAP_UA = "nositemap"
-    # ... and one that additionally Disallows the well-known location, so the
-    # fallback has to be refused by the rules this very body carries.
-    DENY_SITEMAP_UA = "denysitemap"
-    # ... the same Disallow, but with the sitemap also declared: the
-    # declaration is the site inviting the fetch and must win.
-    DENY_DECLARED_UA = "denydeclared"
-    # ... and one that points the sitemap at the site root, to check a subtree
-    # crawl is not widened by where the site chooses to put its sitemap.
-    SCOPE_SITEMAP_UA = "scopesitemap"
-
    def route_robots(self):
-        # The Sitemap: record is group-independent; only --sitemap acts on it.
-        ua = self.headers.get("User-Agent") or ""
-        host = self.headers.get("Host")
-        body = "User-agent: *\nDisallow:\n"
-        if self.DENY_DECLARED_UA in ua:
-            body = (
-                "User-agent: *\nDisallow: /sitemap.xml\n"
-                f"Sitemap: http://{host}/sitemap.xml\n"
-            )
-        elif self.DENY_SITEMAP_UA in ua:
-            body = "User-agent: *\nDisallow: /sitemap.xml\n"
-        elif self.SCOPE_SITEMAP_UA in ua:
-            body += f"Sitemap: http://{host}/scopesitemap.xml\n"
-        elif self.NO_SITEMAP_UA not in ua:
-            body += f"Sitemap: http://{host}/sitemapdir/index.xml\n"
-        body = body.encode()
+        body = b"User-agent: *\nDisallow:\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
@@ -580,132 +551,6 @@ class Handler(SimpleHTTPRequestHandler):
        if self.command != "HEAD":
            self.wfile.write(body)

-    # --- sitemap ingestion (issue #712) ------------------------------------
-    # start.html links to nothing, so orphan*.html are reachable only through
-    # the sitemap. deep1.html proves the seeds keep a full depth budget; the
-    # off-host page <loc> must be dropped by the travel scope, and the off-host
-    # child sitemap by the ingester's same-host rule. The index is served both
-    # from /sitemapdir/ (named by robots.txt) and from the well-known
-    # /sitemap.xml (the fallback).
-
-    def route_sitemap_index(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?>\n'
-            '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
-            f"<sitemap><loc>http://{host}/sitemapdir/pages.xml.gz</loc></sitemap>"
-            "<sitemap><loc>http://sitemap-offhost.invalid/s.xml</loc></sitemap>"
-            "</sitemapindex>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_pages(self):
-        host = self.headers.get("Host")
-        xml = (
-            '<?xml version="1.0" encoding="UTF-8"?>\n'
-            '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
-            f"<url><loc>http://{host}/sitemapdir/orphan1.html</loc></url>"
-            "<url><loc>http://sitemap-offhost.invalid/x.html</loc></url>"
-            f"<url><loc>http://{host}/sitemapdir/orphan2.html</loc></url>"
-            "</urlset>\n"
-        ).encode()
-        self.send_raw(gzip.compress(xml), "application/x-gzip")
-
-    def route_sitemap_start(self):
-        self.send_html("\tNothing links to the sitemap pages.")
-
-    def route_sitemap_orphan1(self):
-        self.send_html('\t<a href="deep1.html">deeper</a>')
-
-    def route_sitemap_orphan2(self):
-        self.send_html("\tSecond orphan.")
-
-    def route_sitemap_deep1(self):
-        self.send_html("\tOne level below an orphan.")
-
-    # chainN is a sitemapindex at nesting level N, listing chain(N+1) and a
-    # urlset capN.xml whose single page is capN.html. Levels up to
-    # HTS_SITEMAP_MAX_LEVEL are followed, so capN.html appears for N below the
-    # cap and stops appearing at it: the pair pins the boundary, which a cap
-    # mutated either way would break.
-    def route_sitemap_chain(self):
-        host = self.headers.get("Host")
-        level = int(self.path.rsplit("/", 1)[-1][len("chain") : -len(".xml")])
-        self.send_raw(
-            (
-                '<?xml version="1.0" encoding="UTF-8"?>\n<sitemapindex>'
-                f"<sitemap><loc>http://{host}/sitemapdir/chain{level + 1}.xml"
-                "</loc></sitemap>"
-                f"<sitemap><loc>http://{host}/sitemapdir/cap{level}.xml"
-                "</loc></sitemap></sitemapindex>\n"
-            ).encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_capset(self):
-        host = self.headers.get("Host")
-        level = self.path.rsplit("/", 1)[-1][len("cap") : -len(".xml")]
-        self.send_raw(
-            (
-                '<?xml version="1.0" encoding="UTF-8"?>\n<urlset>'
-                f"<url><loc>http://{host}/sitemapdir/cap{level}.html</loc></url>"
-                "</urlset>\n"
-            ).encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_cappage(self):
-        self.send_html("\tReached through a nested sitemapindex.")
-
-    def route_sitemap_gatedindex(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?><sitemapindex>'
-            f"<sitemap><loc>http://{host}/sitemapdir/filtered.xml</loc></sitemap>"
-            "</sitemapindex>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_filtered(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
-            f"<url><loc>http://{host}/sitemapdir/gated.html</loc></url>"
-            "</urlset>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_gated(self):
-        self.send_html("\tListed only by the filtered child sitemap.")
-
-    # A moved sitemap: the marking has to follow the redirect.
-    # A root sitemap naming a page below the crawl's start directory and one
-    # above it. Only the first may be seeded when the crawl started at /deep/dir/.
-    def route_sitemap_scope(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
-            f"<url><loc>http://{host}/deep/dir/below.html</loc></url>"
-            f"<url><loc>http://{host}/elsewhere/updir.html</loc></url>"
-            "</urlset>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_deepstart(self):
-        self.send_html("\tA start page in a subdirectory, linking nothing.")
-
-    def route_sitemap_below(self):
-        self.send_html("\tBelow the start directory.")
-
-    def route_sitemap_updir(self):
-        self.send_html("\tAbove the start directory.")
-
-    def route_sitemap_moved(self):
-        self.send_response(301)
-        self.send_header("Location", "/sitemapdir/pages.xml.gz")
-        self.send_header("Content-Length", "0")
-        self.end_headers()
-
    # --- type/extension matrix (issue #267 family) -------------------------

    def send_raw(self, body, content_type, extra_headers=()):
@@ -1894,21 +1739,6 @@ class Handler(SimpleHTTPRequestHandler):
        "/gated/index.php": route_gated_index,
        "/gated/secret.php": route_gated_secret,
        "/robots.txt": route_robots,
-        "/sitemapdir/index.xml": route_sitemap_index,
-        "/sitemap.xml": route_sitemap_index,
-        "/sitemapdir/pages.xml.gz": route_sitemap_pages,
-        "/sitemapdir/start.html": route_sitemap_start,
-        "/sitemapdir/orphan1.html": route_sitemap_orphan1,
-        "/sitemapdir/orphan2.html": route_sitemap_orphan2,
-        "/sitemapdir/deep1.html": route_sitemap_deep1,
-        "/sitemapdir/gatedindex.xml": route_sitemap_gatedindex,
-        "/sitemapdir/filtered.xml": route_sitemap_filtered,
-        "/sitemapdir/gated.html": route_sitemap_gated,
-        "/sitemapdir/moved.xml": route_sitemap_moved,
-        "/scopesitemap.xml": route_sitemap_scope,
-        "/deep/dir/start.html": route_sitemap_deepstart,
-        "/deep/dir/below.html": route_sitemap_below,
-        "/elsewhere/updir.html": route_sitemap_updir,
        "/warcgz/index.html": route_warcgz_index,
        "/warcgz/page.html": route_warcgz_page,
        "/warcgz/data.bin": route_warcgz_data,
@@ -2254,13 +2084,6 @@ class Handler(SimpleHTTPRequestHandler):
            return True
        # Match percent-encoded paths (accented #157 route) by their decoded form.
        handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path))
-        if handler is None:
-            if re.fullmatch(r"/sitemapdir/chain\d+\.xml", path):
-                handler = type(self).route_sitemap_chain
-            elif re.fullmatch(r"/sitemapdir/cap\d+\.xml", path):
-                handler = type(self).route_sitemap_capset
-            elif re.fullmatch(r"/sitemapdir/cap\d+\.html", path):
-                handler = type(self).route_sitemap_cappage
        if handler is not None:
            handler(self)
            return True
--- a/tests/webhttrack-smoke.sh
+++ b/tests/webhttrack-smoke.sh
@@ -46,20 +46,17 @@ cat >"$stubdir/x-www-browser" <<EOF
 echo "stub browser invoked with: \$1" >&2
 # Also fetch an option page and require a rendered title='' tooltip: proves the
 # option template expands and the \${html:} filter escapes into the attribute.
-# option9 additionally proves the WARC and change-report controls and
-# option2 the --single-file pair render with their expanded labels, and
-# option8 the sitemap ones. option2 also pins the absence of an
-# unexpanded key, since the default locale here is French.
+# option9 additionally proves the WARC and change-report controls render with
+# their expanded labels, and option2 the --single-file pair: on field names plus
+# the absence of an unexpanded key, since the default locale here is French.
 opturl="\${1%/}/server/option2.html"
 warcurl="\${1%/}/server/option9.html"
-smurl="\${1%/}/server/option8.html"
 if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html &&
    opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" &&
    printf '%s' "\$opt" | grep -qaF 'name="singlefile"' && printf '%s' "\$opt" | grep -qaF 'name="singlefilemax"' &&
    ! printf '%s' "\$opt" | grep -qaF '\${LANG_SINGLEFILE}' &&
    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC &&
-    printf '%s' "\$warc" | grep -qaF 'name="changes"' && printf '%s' "\$warc" | grep -qaF hts-changes.json &&
-    sm="\$(curl -fsSL --max-time 20 "\$smurl")" && printf '%s' "\$sm" | grep -qaF 'name="sitemapurl"' && printf '%s' "\$sm" | grep -qaF 'name="sitemap"'; then
+    printf '%s' "\$warc" | grep -qaF 'name="changes"' && printf '%s' "\$warc" | grep -qaF hts-changes.json; then
    echo PASS >"$marker"
 else
    echo "FAIL: unexpected response from \$1" >"$marker"
Author	SHA1	Message	Date
Xavier Roche	ef17a145dd	prototype: format the touched lines Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>	2026-07-27 07:47:44 +02:00
Xavier Roche	6eaa4cf36f	prototype: drop the fragment and query htsparse writes after the mark <img src="a.svg#frag"> saved as a.svg#!htsinline#frag, and the pass left the fragment glued to the data: URI. Consume it, and put it back when the reference falls through to a link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>	2026-07-27 07:47:11 +02:00
Xavier Roche	e5ba565b46	prototype: port the singlefile self-test fixture to marked references The fixture fed unmarked HTML to a scanner that no longer exists. Marking it keeps 28 of the 32 assertions; the four that changed are recorded in the diff (quoting is now preserved rather than normalised, and a page with nothing to inline is mark-transparent rather than byte-transparent). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>	2026-07-27 07:39:07 +02:00
Xavier Roche	639690d46e	prototype: --single-file marks references in htsparse instead of rescanning htsparse appends a "#!htsinline" fragment to every reference it saves that --single-file may inline, so the end-of-mirror pass is a substitution over those marks and parses no HTML and no CSS. The fragment keeps the mirror valid if the pass never runs. Measurement prototype for #749. Not for merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>	2026-07-27 07:35:37 +02:00
				`@@ -1 +0,0 @@`
				`<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc></sitemap></sitemapindex>`
				`@@ -1 +0,0 @@`
				`<?xml version="1.0"?><urlset><url><loc>http://h.test/a.html</loc></url><url><loc>https://h.test/b?x=1&y=2</loc></url></urlset>`