htsserver: the session id must gate the request body, not the reply

Every field of a POST body is written straight into the one global key store
the templates and the command dispatcher both read, and "command" from there
reaches the engine. The gate ran after that write and compared "sid" against
"_sid" -- but "_sid" is copied into "sid" beforehand so the templates can
render it, so a request that simply omitted the field compared equal to
itself. Only a wrong id was refused; an absent one passed. Clearing the reply
afterwards does not help either, because the dispatcher sits outside the reply
guard.

Authenticate before parsing instead: scan the raw body for "sid", require at
least one occurrence and reject if any of them differs, and drop the body
untouched when it does not match. That leaves the shared template key alone,
and it closes the dispatcher for free.

A refused request also emitted only a Content-length line, since the status
line for that branch was behind _DEBUG. Any client reads that as a protocol
error, which is how test 68 failed rather than reporting the refusal. Send a
403 instead.

Tests 68 and 77 posted without an id, which is what the engine used to accept,
so both now fetch the one the server renders into the form. Test 78 covers
accept, missing, empty and wrong, asserts the 403, and probes the key store
through ${projname} rather than the suppressed reply -- a reply-only assertion
passes even when the write goes through.

Also fix a leak that hung macOS CI: start() runs inside a command
substitution, so its $! never reached the parent and stop() guarded on an
empty variable, leaving one htsserver per call. Test 77 starts four, which is
exactly the four orphans the runner reported while sitting for half an hour
behind a green test log. Take the pid from the PID= line the server already
announces.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.github/workflows/windows-build.yml

@@ -225,9 +225,8 @@ jobs:
 
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
-          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
-          # webdav-mime needs a reapable background listener, which MSYS cannot give it.
-          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
+          # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -26,12 +26,6 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
-- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
-  pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
-  and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
-  as "marker absent". Capture the reply, assert the status line it must carry
-  (an empty, truncated or redirected one is marker-free too), then match with a
-  here-string.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -54,16 +48,6 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
-- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
-  (`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
-  our own data, wrong for anything read back from a cache, a header or the
-  wire, where it trades a memory smash for a crash on malformed input. Clip
-  with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
-- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
-  a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
-  into the same buffer never appears on it. Before scoping a hardening pass off
-  compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
-  `\bstrcpy\s*\(`, `\bstrcat\s*\(`).
 
 ## C conventions
 - **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -100,17 +84,6 @@ Before pushing, and when reviewing others, don't skim for bugs:
   layout/ABI, cache/wire format, or a security path? A static or unit check
   isn't enough; exercise the wrong behavior at runtime. Claude Code:
   `/review-recipe`.
-- **Poison a canary, never compare it against zero.** Checking that a
-  neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
-  terminator writes — the exact bug the canary is there for. Fill it with a
-  non-zero byte, and prove it by killing both the stray-`'X'` and the
-  stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
-  lands inside the same struct.
-- **Overshoot every destination, not one.** A bounds test that oversizes a
-  single field cannot tell a per-field bound from a one-size-fits-all one, nor
-  from a fix that bounds that field and leaves its neighbours raw. Exercise
-  each destination the path touches, spanning at least two capacities, and
-  check what the code actually emits before writing the expected values.
 
 ## Commits
 - **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

configure.ac

@@ -66,11 +66,10 @@ AC_SUBST(LT_CV_OBJDIR,$lt_cv_objdir)
 AC_SUBST(VERSION_INFO)
 
 ### Default CFLAGS
-# No -Wdeclaration-after-statement: nothing sets -std=, so this builds as gnu17.
 DEFAULT_CFLAGS="-Wall -Wformat -Wformat-security \
 -Wmultichar -Wwrite-strings -Wcast-qual -Wcast-align \
 -Wstrict-prototypes -Wmissing-prototypes \
--Wmissing-declarations \
+-Wmissing-declarations -Wdeclaration-after-statement \
 -Wpointer-arith -Wsequence-point -Wnested-externs \
 -D_REENTRANT"
 AC_SUBST(DEFAULT_CFLAGS)
@@ -78,30 +77,27 @@ DEFAULT_LDFLAGS=""
 AC_SUBST(DEFAULT_LDFLAGS)
 
 ### Additional flags (if supported)
-# -Werror on probes: exit-status-only checks let clang's warn-on-unknown-flag through.
-AX_CHECK_COMPILE_FLAG([-Wparentheses], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wparentheses"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Winit-self], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Winit-self"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Wunused-but-set-parameter], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wunused-but-set-parameter"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Waddress], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Waddress"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Wuninitialized], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wuninitialized"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Wformat=2], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat=2"], [], [-Werror])
-# -Wformat-nonliteral needs -Wformat in the probe or gcc rejects it as ignored.
-AX_CHECK_COMPILE_FLAG([-Wformat-nonliteral], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat-nonliteral"], [], [-Werror -Wformat])
-AX_CHECK_COMPILE_FLAG([-Wmissing-parameter-type], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wmissing-parameter-type"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Wold-style-definition], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wold-style-definition"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Wignored-qualifiers], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wignored-qualifiers"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Wparentheses], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wparentheses"])
+AX_CHECK_COMPILE_FLAG([-Winit-self], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Winit-self"])
+AX_CHECK_COMPILE_FLAG([-Wunused-but-set-parameter], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wunused-but-set-parameter"])
+AX_CHECK_COMPILE_FLAG([-Waddress], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Waddress"])
+AX_CHECK_COMPILE_FLAG([-Wuninitialized], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wuninitialized"])
+AX_CHECK_COMPILE_FLAG([-Wformat=2], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat=2"])
+AX_CHECK_COMPILE_FLAG([-Wformat-nonliteral], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wformat-nonliteral"])
+AX_CHECK_COMPILE_FLAG([-Wmissing-parameter-type], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wmissing-parameter-type"])
+AX_CHECK_COMPILE_FLAG([-Wold-style-definition], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wold-style-definition"])
+AX_CHECK_COMPILE_FLAG([-Wignored-qualifiers], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Wignored-qualifiers"])
 # Make htssafe.h's pointer-dest 'warning' attribute a hard error in our build
 # (migration is at zero; a new char* dest is a regression). gcc/clang each take
 # only their own spelling; downstream keeps the plain warning, not a build break.
-AX_CHECK_COMPILE_FLAG([-Werror=attribute-warning], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=attribute-warning"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-Werror=user-defined-warnings], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=user-defined-warnings"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-fstrict-aliasing -Wstrict-aliasing], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstrict-aliasing -Wstrict-aliasing"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-Werror=attribute-warning], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=attribute-warning"])
+AX_CHECK_COMPILE_FLAG([-Werror=user-defined-warnings], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -Werror=user-defined-warnings"])
+AX_CHECK_COMPILE_FLAG([-fstrict-aliasing -Wstrict-aliasing], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstrict-aliasing -Wstrict-aliasing"])
 AX_CHECK_COMPILE_FLAG([-fstack-protector-strong], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector-strong"],
-	[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"], [], [-Werror])], [-Werror])
-AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"], [], [-Werror])
-AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"], [], [-Werror])
-# No --discard-all: it drops the local symbols naming every static function, so
-# a trace misattributes them to the nearest surviving global. Costs 0.6% size.
+	[AX_CHECK_COMPILE_FLAG([-fstack-protector], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-protector"])])
+AX_CHECK_COMPILE_FLAG([-fstack-clash-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fstack-clash-protection"])
+AX_CHECK_COMPILE_FLAG([-fcf-protection], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -fcf-protection"])
+AX_CHECK_LINK_FLAG([-Wl,--discard-all], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--discard-all"])
 AX_CHECK_LINK_FLAG([-Wl,--no-undefined], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--no-undefined"])
 AX_CHECK_LINK_FLAG([-Wl,-z,relro,-z,now], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,relro,-z,now"])
 AX_CHECK_LINK_FLAG([-Wl,-z,noexecstack], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,-z,noexecstack"])
@@ -123,13 +119,13 @@ AC_SUBST([LIBC_FORCE_LINK])
 ### PIE
 CFLAGS_PIE=""
 LDFLAGS_PIE=""
-AX_CHECK_COMPILE_FLAG([-fpie], [CFLAGS_PIE="-fpie"], [], [-Werror])
+AX_CHECK_COMPILE_FLAG([-fpie -pie], [CFLAGS_PIE="-fpie -pie"])
 AX_CHECK_LINK_FLAG([-pie], [LDFLAGS_PIE="-pie"])
 AC_SUBST([CFLAGS_PIE])
 AC_SUBST([LDFLAGS_PIE])
 
-# Ties a crash trace from a stripped build back to its separate debug symbols.
-AX_CHECK_LINK_FLAG([-Wl,--build-id], [DEFAULT_LDFLAGS="$DEFAULT_LDFLAGS -Wl,--build-id"])
+## Export all symbols for backtraces
+AX_CHECK_COMPILE_FLAG([-rdynamic], [DEFAULT_CFLAGS="$DEFAULT_CFLAGS -rdynamic"])
 
 ### Check for -fvisibility=hidden support
 gl_VISIBILITY

fuzz/Makefile.am

@@ -2,7 +2,7 @@
 if FUZZERS
 noinst_PROGRAMS = fuzz-charset fuzz-meta fuzz-idna fuzz-entities \
 	fuzz-unescape fuzz-filters fuzz-url fuzz-header fuzz-cachendx \
-	fuzz-htsparse fuzz-sitemap
+	fuzz-htsparse
 endif
 
 AM_CPPFLAGS = \
@@ -27,7 +27,6 @@ fuzz_url_SOURCES = fuzz-url.c fuzz.h
 fuzz_header_SOURCES = fuzz-header.c fuzz.h
 fuzz_cachendx_SOURCES = fuzz-cachendx.c fuzz.h
 fuzz_htsparse_SOURCES = fuzz-htsparse.c fuzz.h
-fuzz_sitemap_SOURCES = fuzz-sitemap.c fuzz.h
 
 # List corpus files explicitly: automake does not expand EXTRA_DIST globs.
 EXTRA_DIST = README.md run-fuzzers.sh \
@@ -48,6 +47,4 @@ EXTRA_DIST = README.md run-fuzzers.sh \
 	corpus/cachendx/regress-overadvance.bin \
 	corpus/cachendx/regress-truncated-entry.bin \
 	corpus/htsparse/basic.html corpus/htsparse/script-inscript.html \
-	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html \
-	corpus/sitemap/urlset.xml corpus/sitemap/sitemapindex.xml \
-	corpus/sitemap/truncated.xml corpus/sitemap/urlset.xml.gz
+	corpus/htsparse/meta-usemap.html corpus/htsparse/malformed.html

fuzz/corpus/sitemap/sitemapindex.xml

@@ -1 +0,0 @@
-<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc></sitemap></sitemapindex>

fuzz/corpus/sitemap/truncated.xml

@@ -1 +0,0 @@
-<urlset><loc>http://h.test/x

fuzz/corpus/sitemap/urlset.xml

@@ -1 +0,0 @@
-
<?xml version="1.0"?><urlset><url><loc>http://h.test/a.html</loc></url><url><loc>https://h.test/b?x=1&amp;y=2</loc></url></urlset>

fuzz/fuzz-sitemap.c

@@ -1,60 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 2026 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* Fuzz the sitemap <loc> scanner (htssitemap.c): raw XML, gzip-framed bodies
-   and truncated streams all arrive here straight off the network. */
-#include "fuzz.h"
-#include "htssitemap.h"
-
-static hts_boolean sm_count(void *arg, const char *url) {
-  int *const n = (int *) arg;
-
-  (void) url;
-  (*n)++;
-  return HTS_TRUE;
-}
-
-int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
-  static const int caps[] = {0, 1, 16, HTS_SITEMAP_MAX_URLS_DOC};
-  hts_boolean is_index;
-  char *body;
-  int n = 0, cap;
-
-  if (size == 0)
-    return 0;
-  cap = caps[data[0] % (sizeof(caps) / sizeof(caps[0]))];
-  data++, size--;
-  /* A heap copy of exactly `size` bytes: the scanner must never rely on a
-     terminator, and ASan turns any overread into a report. */
-  body = malloct(size != 0 ? size : 1);
-  memcpy(body, data, size);
-
-  (void) hts_sitemap_scan(body, size, cap, &is_index, sm_count, &n);
-
-  freet(body);
-  return 0;
-}

html/cmdguide.html

@@ -163,26 +163,8 @@ the index" problems disappear.</p>
   <tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
   <tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
   <tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
-  <tr><td><tt>--sitemap (-%m), --sitemap-url URL (-%mu)</tt></td><td>Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.</td></tr>
 </table>
 
-<p>Link-following only finds what something links to. Anything a site publishes
-solely in its sitemap is invisible to HTTrack unless you ask for it.
-<tt>--sitemap</tt> reads the start host's <tt>robots.txt</tt> for
-<tt>Sitemap:</tt> lines and falls back to <tt>/sitemap.xml</tt>;
-<tt>--sitemap-url</tt> names one directly. Nested <tt>sitemapindex</tt> files
-and gzipped <tt>.xml.gz</tt> sitemaps are followed. The URLs found become start
-URLs with the full depth budget, but they still go through your filters and
-scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is
-off by default because a sitemap can list thousands of pages nothing links
-to.</p>
-
-<p>One surprise worth knowing: a sitemap you name with <tt>--sitemap-url</tt>,
-and one the site itself declares in <tt>robots.txt</tt>, are fetched even when
-<tt>robots.txt</tt> disallows that path, because naming or declaring a sitemap
-is an invitation to read it. Only the guessed <tt>/sitemap.xml</tt> obeys a
-<tt>Disallow</tt>. The URLs listed inside are gated normally either way.</p>
-
 <p>The single most common surprise is "only the home page came down." That is
 usually not a scope option at all: it is an off-host redirect. A start URL of
 <tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>

html/httrack.man.html

@@ -87,8 +87,8 @@ offline browser : copy websites to a local directory</p>
 --host-control[=N]</b> ] [ <b>-%P,
 --extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
 --test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
-] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
-[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
+] [ <b>-NN, --structure[=N]</b> ] [ <b>-%N,
+--delayed-type-check</b> ] [ <b>-%D,
 --cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
 ] [ <b>-LN, --long-names[=N]</b> ] [ <b>-KN,
 --keep-links[=N]</b> ] [ <b>-x, --replace-external</b> ] [
@@ -575,22 +575,6 @@ URL per line) (--list &lt;param&gt;)</p></td></tr>
 
 <p>&lt;file&gt; add all scan rules located in this text
 file (one scan rule per line) (--urllist &lt;param&gt;)</p></td></tr>
-<tr valign="top" align="left">
-<td width="9%"></td>
-<td width="4%">
-
-
-<p>-%m</p></td>
-<td width="5%"></td>
-<td width="82%">
-
-
-<p>seed the crawl from the site&rsquo;s sitemap (robots.txt
-Sitemap:, then /sitemap.xml); --sitemap-url URL names one
-explicitly. A sitemap you name, or one the site declares, is
-fetched even under robots.txt Disallow; only the guessed
-/sitemap.xml obeys it. The URLs found still pass every
-filter and scope rule (--sitemap)</p></td></tr>
 </table>
 
 <h3>Build options:

html/server/finished.html

@@ -97,8 +97,9 @@ ${do:end-if}
 </pre>
 
 ${LANG_G8} : 
-${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
-<a href="/website/index.html" target="_new">
+${do:output-mode:html-urlescaped}
+<a href="file://${path}/${projname}/" target="_new">
+${do:output-mode:}
 ${path}/${projname}
 </a></li>
 <ul>

html/server/option10.html

@@ -98,9 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="ftpprox" value="">
-
 ${LANG_PROXYTYPE}
 <select name="proxytype"
 			title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

html/server/option2.html

@@ -98,13 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="errpage" value="">
-<input type="hidden" name="external" value="">
-<input type="hidden" name="hidepwd" value="">
-<input type="hidden" name="hidequery" value="">
-<input type="hidden" name="nopurge" value="">
-
 ${LANG_I33}
 <br>
 <select name="build"

html/server/option2b.html

@@ -103,7 +103,7 @@ ${LANG_Q3}
 <tr><td>
 	<table width="100%">
 	<tr><td align="left">
-	<input type="submit" value="${LANG_OK}"
+	<input type="submit" value="${LANG_OK]" 
 ${do:output-mode:html-urlescaped}
 		onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
 ${do:output-mode:}

html/server/option3.html

@@ -98,9 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="windebug" value="">
-
 ${LANG_I40c}
 <br>
 

html/server/option4.html

@@ -98,11 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="ka" value="">
-<input type="hidden" name="remt" value="">
-<input type="hidden" name="rems" value="">
-
 <table border="0" width="100%" cellspacing="0">
 
 <tr><td>

html/server/option8.html

@@ -98,18 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="cookies" value="">
-<input type="hidden" name="parsejava" value="">
-<input type="hidden" name="updhack" value="">
-<input type="hidden" name="urlhack" value="">
-<input type="hidden" name="keepwww" value="">
-<input type="hidden" name="keepslashes" value="">
-<input type="hidden" name="keepqueryorder" value="">
-<input type="hidden" name="toler" value="">
-<input type="hidden" name="http10" value="">
-<input type="hidden" name="sitemap" value="">
-
 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I58}
@@ -144,17 +132,6 @@ ${listid:robots:LISTDEF_8}
 </select>
 <br><br>
 
-<input type="checkbox" name="sitemap" ${checked:sitemap}
-			title='${html:LANG_SITEMAPTIP}' onMouseOver="info('${html:LANG_SITEMAPTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
-> ${LANG_SITEMAP}
-<br><br>
-
-${LANG_SITEMAPURL}
-<input name="sitemapurl" value="${sitemapurl}" size="40"
-			title='${html:LANG_SITEMAPURLTIP}' onMouseOver="info('${html:LANG_SITEMAPURLTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
->
-<br><br>
-
 <input type="checkbox" name="updhack" ${checked:updhack}
 			title='${html:LANG_I1k}' onMouseOver="info('${html:LANG_I1k}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I62b}

html/server/option9.html

@@ -98,13 +98,6 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
-<!-- clear if not checked -->
-<input type="hidden" name="warc" value="">
-<input type="hidden" name="norecatch" value="">
-<input type="hidden" name="logf" value="">
-<input type="hidden" name="index" value="">
-<input type="hidden" name="index2" value="">
-
 <input type="checkbox" name="cache2" ${checked:cache2}
 			title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I61}

html/server/step2.html

@@ -141,8 +141,6 @@ ${do:copy:KeepSlashes:keepslashes}
 ${do:copy:KeepQueryOrder:keepqueryorder}
 ${do:copy:StripQuery:stripquery}
 ${do:copy:StoreAllInCache:cache2}
-${do:copy:Sitemap:sitemap}
-${do:copy:SitemapUrl:sitemapurl}
 ${do:copy:Warc:warc}
 ${do:copy:WarcFile:warcfile}
 ${do:copy:LogType:logtype}

html/server/step4.html

@@ -114,16 +114,16 @@ ${do:end-if}
 
 ${/* Real commands and ini file generated below */}
 
-<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
+<!-- engine commandline -->
 ${do:output-mode:html}
 <textarea name="command" cols="50" rows="4" style="visibility:hidden">
 httrack \
 	--quiet \
 	--build-top-index \
 	${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
-	${unquoted:urls}
-	${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
-	--path "${arg:path}/${arg:projname}" 
+	${urls}
+	${test:filelist:-%L "}${filelist}${test:filelist:"}
+	--path "${html:path}/${html:projname}" 
 \
 	${test:parseall:--near}
 	${test:link:--test}
@@ -131,7 +131,7 @@ httrack \
 	${test:htmlfirst::--priority=7}
 \
 ${do:if-not-empty:BuildString}
-	--structure "${arg:BuildString}"
+	--structure "${BuildString}"
 ${do:end-if}
 	${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
 \
@@ -150,54 +150,51 @@ ${do:end-if}
 	${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
 	${test:windebug:--debug-headers}
 \
-	${test:connexion:--sockets=}${unquoted:connexion}
+	${test:connexion:--sockets=}${connexion}
 	${test:ka:--keep-alive}
-	${test:timeout:--timeout=}${unquoted:timeout}
+	${test:timeout:--timeout=}${timeout}
 	${test:remt:--host-control=1}
-	${test:retry:--retries=}${unquoted:retry}
-	${test:rate:--min-rate=}${unquoted:rate}
+	${test:retry:--retries=}${retry}
+	${test:rate:--min-rate=}${rate}
 	${test:rems:--host-control=2}
 \
-	${test:depth:--depth=}${unquoted:depth}
-	${test:depth2:--ext-depth=}${unquoted:depth2}
-${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
-	${test:othermax:--max-files=}${unquoted:othermax}
-	${test:maxhtml:--max-files=,}${unquoted:maxhtml}
-	${test:sizemax:--max-size=}${unquoted:sizemax}
-	${test:pausebytes:--max-pause=}${unquoted:pausebytes}
-	${test:maxtime:--max-time=}${unquoted:maxtime}
-	${test:maxrate:--max-rate=}${unquoted:maxrate}
-	${test:maxconn:--connection-per-second=}${unquoted:maxconn}
-	${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
+	${test:depth:--depth=}${depth}
+	${test:depth2:--ext-depth=}${depth2}
+	${test:maxhtml:--max-files=,}${maxhtml}
+	${test:othermax:--max-files=}${othermax}
+	${test:sizemax:--max-files=}${sizemax}
+	${test:pausebytes:--max-pause=}${pausebytes}
+	${test:maxtime:--max-time=}${maxtime}
+	${test:maxrate:--max-rate=}${maxrate}
+	${test:maxconn:--connection-per-second=}${maxconn}
+	${test:maxlinks:--advanced-maxlinks=}${maxlinks}
 \
-	--user-agent "${arg:user}"
-	--footer "${arg:footer}"
+	--user-agent "${html:user}"
+	--footer "${html:footer}"
 \
-	${unquoted:url2}
+	${url2}
 \
-	${ztest:cookies:--cookies=0:}
-	${ztest:parsejava:--parse-java=0:}
+	${test:cookies:--cookies=0:}
+	${test:parsejava:--parse-java=0:}
 	${test:updhack:--updatehack}
 	${test:urlhack:--urlhack=0:--urlhack}
 	${test:keepwww:--keep-www-prefix}
 	${test:keepslashes:--keep-double-slashes}
 	${test:keepqueryorder:--keep-query-order}
-	${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
-	${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
-	${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
+	${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
+	${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
+	${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
 	${test:toler:--tolerant}
 	${test:http10:--http-10}
 	${test:cache2:--store-all-in-cache}
-	${test:sitemap:--sitemap}
-	${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"}
 	${test:warc:--warc}
-	${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
+	${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
 	${test:norecatch:--do-not-recatch}
 	${test:logf:--single-log}
 	${test:logtype:::--extra-log:--debug-log}
 	${test:index:--index=0:}
 	${test:index2:--search-index=0:--search-index}
-	${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
+	${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
 	${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
 </textarea>
 
@@ -216,7 +213,7 @@ ParseAll=${ztest:parseall:0:1}
 HTMLFirst=${ztest:htmlfirst:0:1}
 Cache=${ztest:cache:0:1}
 NoRecatch=${ztest:norecatch:0:1}
-Dos=${dos}
+Dos=${dos
 Index=${ztest:index:0:1}
 WordIndex=${ztest:index2:0:1}
 Log=${ztest:logf:0:1:2}
@@ -242,8 +239,6 @@ KeepSlashes=${ztest:keepslashes:0:1}
 KeepQueryOrder=${ztest:keepqueryorder:0:1}
 StripQuery=${stripquery}
 StoreAllInCache=${ztest:cache2:0:1}
-Sitemap=${ztest:sitemap:0:1}
-SitemapUrl=${sitemapurl}
 Warc=${ztest:warc:0:1}
 WarcFile=${warcfile}
 LogType=${logtype}

lang.def

@@ -1042,11 +1042,3 @@ LANG_WARCFILE
 WARC archive name:
 LANG_WARCFILETIP
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
-LANG_SITEMAP
-Seed the crawl from the site's sitemap
-LANG_SITEMAPTIP
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-LANG_SITEMAPURL
-Sitemap address:
-LANG_SITEMAPURLTIP
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/English.txt

@@ -1012,11 +1012,3 @@ WARC archive name:
 WARC archive name:
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
-Seed the crawl from the site's sitemap
-Seed the crawl from the site's sitemap
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Sitemap address:
-Sitemap address:
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.

lang/Francais.txt

@@ -1012,11 +1012,3 @@ WARC archive name:
 Nom de l'archive WARC :
 Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
 Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.
-Seed the crawl from the site's sitemap
-Partir du plan de site (sitemap)
-Read the site's sitemap (robots.txt Sitemap: lines, then /sitemap.xml) and add every URL it lists as a start URL.
-Lire le plan de site (lignes Sitemap: de robots.txt, puis /sitemap.xml) et ajouter chaque URL listée comme adresse de départ.
-Sitemap address:
-Adresse du plan de site :
-Address of a sitemap to read instead of probing the site; leave blank to probe robots.txt then /sitemap.xml.
-Adresse d'un plan de site à lire au lieu de sonder le site ; laissez vide pour sonder robots.txt puis /sitemap.xml.

libtest/callbacks-example-filenameiisbug.c

@@ -71,9 +71,7 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
       for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
       if (iisBogus[i][j] == '\0'
           && (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
-        /* j bytes matched, so j fit: copying j cannot overrun whatever the
-           table holds, and the tail must survive untouched */
-        memcpy(a, iisBogusReplace[i], (size_t) j);
+        strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
         break;
       }
     }

man/httrack.1

@@ -3,7 +3,7 @@
 .\"
 .\" This file is generated by man/makeman.sh; do not edit by hand.
 .\" SPDX-License-Identifier: GPL-3.0-or-later
-.TH httrack 1 "26 July 2026" "httrack website copier"
+.TH httrack 1 "23 July 2026" "httrack website copier"
 .SH NAME
 httrack \- offline browser : copy websites to a local directory
 .SH SYNOPSIS
@@ -36,7 +36,6 @@ httrack \- offline browser : copy websites to a local directory
 [ \fB\-t, \-\-test\fR ]
 [ \fB\-%L, \-\-list\fR ]
 [ \fB\-%S, \-\-urllist\fR ]
-[ \fB\-%m, \-\-sitemap\fR ]
 [ \fB\-NN, \-\-structure[=N]\fR ]
 [ \fB\-%N, \-\-delayed\-type\-check\fR ]
 [ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
@@ -188,8 +187,6 @@ test all URLs (even forbidden ones) (\-\-test)
 <file> add all URL located in this text file (one URL per line) (\-\-list <param>)
 .IP \-%S
 <file> add all scan rules located in this text file (one scan rule per line) (\-\-urllist <param>)
-.IP \-%m
-seed the crawl from the site's sitemap (robots.txt Sitemap:, then /sitemap.xml); \-\-sitemap\-url URL names one explicitly. A sitemap you name, or one the site declares, is fetched even under robots.txt Disallow; only the guessed /sitemap.xml obeys it. The URLs found still pass every filter and scope rule (\-\-sitemap)
 .SS Build options:
 .IP \-NN
 structure type (0 *original structure, 1+: see below) (\-\-structure[=N])

src/Makefile.am

@@ -32,9 +32,7 @@ AM_LDFLAGS = \
 
 bin_PROGRAMS = proxytrack httrack htsserver
 
-httrack_SOURCES = httrack.c htsbacktrace.c htsbacktrace.h
-# $(DL_LIBS): dladdr() in the crash handler, still in libdl on pre-2.34 glibc.
-httrack_LDADD = $(THREADS_LIBS) $(DL_LIBS) libhttrack.la
+httrack_LDADD = $(THREADS_LIBS) libhttrack.la
 htsserver_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS) libhttrack.la
 proxytrack_LDADD = $(THREADS_LIBS) $(SOCKET_LIBS)
 
@@ -48,8 +46,7 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
 
 lib_LTLIBRARIES = libhttrack.la
 
-htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
-	htscmdline.c htscmdline.h \
+htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
 	htsurlport.c htsurlport.h
 proxytrack_SOURCES = proxy/main.c \
 	proxy/proxytrack.c proxy/store.c \
@@ -63,21 +60,21 @@ whttrackrun_SCRIPTS = webhttrack
 libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscache_selftest.c htsdns_selftest.c htsselftest.c \
 	htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
-	htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
+	htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
-	htsmd5.c htscodec.c htswarc.c htssitemap.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
+	htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
 	htsmodules.c htscharset.c punycode.c htsencoding.c htssniff.c \
 	md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
 	hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
 	htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h  \
-	htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
+	htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
 	htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
 	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
-	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htssitemap.h htsproxy.h htszlib.h  \
+	htstools.h htswizard.h htswrap.h htscodec.h htswarc.h htsproxy.h htszlib.h  \
 	htsstrings.h htsarrays.h httrack-library.h \
 	htscharset.h punycode.h htsencoding.h \
 	htsentities.h htsentities.sh htsbasiccharsets.sh htscodepages.h \
@@ -87,7 +84,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
 libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
 libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
 
-EXTRA_DIST = httrack.h htsstats.h webhttrack \
+EXTRA_DIST = httrack.h webhttrack \
 		version.rc \
 		libhttrack.rc \
 		httrack.rc \

src/htsalias.c

@@ -114,10 +114,6 @@ const char *hts_optalias[][4] = {
    "strip [host/pattern=]key1,key2,... from URLs"},
   {"cookies-file", "-%K", "param1",
    "load extra cookies from a Netscape cookies.txt"},
-  {"sitemap", "-%m", "single",
-   "seed the crawl from the start host's sitemap (robots.txt, then "
-   "/sitemap.xml)"},
-  {"sitemap-url", "-%mu", "param1", "seed the crawl from this sitemap URL"},
   {"warc", "-%r", "single", "write an ISO-28500 WARC/1.1 archive of the crawl"},
   {"warc-file", "-%rf", "param1", "write a WARC archive to the given base name"},
   {"warc-max-size", "-%rs", "param1",

src/htsback.c

@@ -125,22 +125,6 @@ void back_free(struct_back ** sback) {
    above a normal handshake. The last candidate still gets the full timeout. */
 #define HTS_CONNECT_FALLBACK_TIMEOUT 10
 
-void back_read_ftp_result(FILE *fp, htsblk *r) {
-  size_t j = 0;
-
-  if (fscanf(fp, "%d ", &r->statuscode) != 1)
-    r->statuscode = STATUSCODE_INVALID;
-  // an external helper writes this file: stop at capacity, not at EOF
-  while (j + 1 < sizeof(r->msg)) {
-    const int c = fgetc(fp);
-
-    if (c == EOF)
-      break;
-    r->msg[j++] = (char) c;
-  }
-  r->msg[j] = '\0';
-}
-
 int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
                               int timeout) {
   int deadline;
@@ -557,15 +541,9 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
                                const char *ext) {
   // do not use tempnam() but a regular filename
   back->tmpfile_buffer[0] = '\0';
-  if (back->url_sav[0] != '\0') {
-    /* same capacity as url_sav, so truncation drops the extension and aliases
-       the temp name onto the live file that back_finalize_backup() UNLINKs */
-    if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
-      hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
-                    back->url_sav);
-      back->tmpfile_buffer[0] = '\0';
-      return -1;
-    }
+  if (back->url_sav != NULL && back->url_sav[0] != '\0') {
+    snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
+             back->url_sav, ext);
     back->tmpfile = back->tmpfile_buffer;
     if (structcheck(back->tmpfile) != 0) {
       hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
@@ -573,15 +551,8 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
       return -1;
     }
   } else {
-    /* truncation here would collide distinct tmpnameid's onto one name */
-    if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
-                     StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
-                     ext)) {
-      hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
-                    StringBuff(opt->path_html_utf8));
-      back->tmpfile_buffer[0] = '\0';
-      return -1;
-    }
+    snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
+             StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
     back->tmpfile = back->tmpfile_buffer;
   }
   /* OK */
@@ -916,7 +887,8 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
               HTS_STAT.stat_bytes += back[p].r.size;
               HTS_STAT.stat_files++;
               hts_log_print(opt, LOG_TRACE, "added file %s%s => %s",
-                            back[p].url_adr, back[p].url_fil, back[p].url_sav);
+                            back[p].url_adr, back[p].url_fil,
+                            back[p].url_sav != NULL ? back[p].url_sav : "");
             }
             if ((!back[p].r.notmodified) && (opt->is_update)) {
               HTS_STAT.stat_updated_files++;    // page modifiée
@@ -2330,24 +2302,26 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
               && slot_can_be_finalized(opt, &back[i]);
             int may_serialize = slot_can_be_cached_on_disk(&back[i]);
 
-            hts_log_print(
-                opt, LOG_DEBUG,
-                "back[%03d]: may_clean=%d, may_finalize_disk=%d, "
-                "may_serialize=%d:" LF "\t"
-                "finalized(%d), status(%d), locked(%d), delayed(%d), "
-                "test(%d), " LF "\t"
-                "statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), " LF
-                "\t"
-                "contenttype(%s), url(%s%s), save(%s)",
-                i, may_clean, may_finalize, may_serialize, back[i].finalized,
-                back[i].status, back[i].locked, IS_DELAYED_EXT(back[i].url_sav),
-                back[i].testmode, back[i].r.statuscode, (int) back[i].r.size,
-                back[i].r.is_write,
-                may_be_hypertext_mime(opt, back[i].r.contenttype,
-                                      back[i].url_fil),
-                /* */
-                back[i].r.contenttype, back[i].url_adr, back[i].url_fil,
-                back[i].url_sav);
+            hts_log_print(opt, LOG_DEBUG,
+                          "back[%03d]: may_clean=%d, may_finalize_disk=%d, may_serialize=%d:"
+                          LF "\t"
+                          "finalized(%d), status(%d), locked(%d), delayed(%d), test(%d), "
+                          LF "\t"
+                          "statuscode(%d), size(%d), is_write(%d), may_hypertext(%d), "
+                          LF "\t" "contenttype(%s), url(%s%s), save(%s)", i,
+                          may_clean, may_finalize, may_serialize,
+                          back[i].finalized, back[i].status, back[i].locked,
+                          IS_DELAYED_EXT(back[i].url_sav), back[i].testmode,
+                          back[i].r.statuscode, (int) back[i].r.size,
+                          back[i].r.is_write, may_be_hypertext_mime(opt,
+                                                                    back[i].r.
+                                                                    contenttype,
+                                                                    back[i].
+                                                                    url_fil),
+                          /* */
+                          back[i].r.contenttype, back[i].url_adr,
+                          back[i].url_fil,
+                          back[i].url_sav ? back[i].url_sav : "<null>");
           }
         }
       }
@@ -2883,8 +2857,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               // new session
               back[i].r.ssl_con = SSL_new(openssl_ctx);
               if (back[i].r.ssl_con) {
-                /* non-const twin: the OpenSSL macro casts the qualifier away */
-                char *hostname = jump_protocol(back[i].url_adr);
+                const char* hostname = jump_protocol_const(back[i].url_adr);
                 // some servers expect the hostname on the clienthello (SNI TLS extension)
                 SSL_set_tlsext_host_name(back[i].r.ssl_con, hostname);
                 SSL_clear(back[i].r.ssl_con);
@@ -2976,7 +2949,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               back[i].r.msg[0] = '\0';
               strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
               if (!strnotempty(back[i].r.msg)) {
-                htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
+                sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
               }
               deletehttp(&back[i].r);
               back[i].r.soc = INVALID_SOCKET;
@@ -3049,7 +3022,16 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
             FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
                   "rb");
           if (fp) {
-            back_read_ftp_result(fp, &back[i].r);
+            int j = 0;
+
+            fscanf(fp, "%d ", &(back[i].r.statuscode));
+            while(!feof(fp)) {
+              int c = fgetc(fp);
+
+              if (c != EOF)
+                back[i].r.msg[j++] = c;
+            }
+            back[i].r.msg[j++] = '\0';
             fclose(fp);
             UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
             strcpybuff(fconcat
@@ -3350,11 +3332,10 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                   deleteaddr(&back[i].r);
                   if (back[i].r.size < back[i].r.totalsize)
                     back[i].r.statuscode = STATUSCODE_CONNERROR;        // recatch
-                  htsblk_failf(&back[i].r,
-                               "Incorrect length (" LLintP " Bytes, " LLintP
-                               " expected)",
-                               (LLint) back[i].r.size,
-                               (LLint) back[i].r.totalsize);
+                  sprintf(back[i].r.msg,
+                          "Incorrect length (" LLintP " Bytes, " LLintP
+                          " expected)", (LLint) back[i].r.size,
+                          (LLint) back[i].r.totalsize);
                 } else {
                   // Un warning suffira..
                   hts_log_print(opt, LOG_WARNING,

src/htsback.h

@@ -74,10 +74,6 @@ void back_free(struct_back ** sback);
 // backing
 #define BACK_ADD_TEST "(dummy)"
 #define BACK_ADD_TEST2 "(dummy2)"
-/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
-   clipping the message to r->msg. */
-void back_read_ftp_result(FILE *fp, htsblk *r);
-
 int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
                const char *sav);
 int back_available(const struct_back * sback);

src/htsbacktrace.c

@@ -1,238 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 1998 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: crash backtrace printer                                 */
-/* Author: Xavier Roche                                          */
-/* ------------------------------------------------------------ */
-
-/* Before every header: glibc gates dladdr() on it. */
-#if defined(__linux) && !defined(_GNU_SOURCE)
-#define _GNU_SOURCE
-#endif
-
-#include "htsbacktrace.h"
-
-#include "htsglobal.h"
-
-#include <stdlib.h>
-#include <string.h>
-#ifdef _WIN32
-#include <io.h> /* write */
-#endif
-#ifdef HAVE_UNISTD_H
-#include <unistd.h>
-#endif
-#if (defined(__linux) && defined(HAVE_EXECINFO_H))
-#include <dlfcn.h>
-#include <errno.h>
-#include <execinfo.h>
-#include <signal.h>
-#include <time.h>
-#include <sys/wait.h>
-#define USES_BACKTRACE
-#endif
-
-#ifdef USES_BACKTRACE
-#define BT_MAX_FRAMES 64     /* frames we try to name */
-#define BT_MAX_MODULES 8     /* distinct modules, one child each */
-#define BT_HEX_SIZE 19       /* "0x" + 16 nibbles + NUL */
-#define BT_PATH_SIZE 1024    /* module path; longer is skipped */
-#define BT_WAIT_TICKS 300    /* 10ms ticks, shared: cap a slow child */
-#define BT_NO_SYMBOLIZER 127 /* child exit: execvp() found none */
-
-static hts_boolean symbolize_crash = HTS_TRUE;
-
-/* "0x"-prefixed hex: the handler must stay stdio-free. */
-static void print_hex(char *buffer, uintptr_t value) {
-  static const char digits[] = "0123456789abcdef";
-  size_t i = 2, a, b;
-
-  buffer[0] = '0';
-  buffer[1] = 'x';
-  do {
-    buffer[i++] = digits[value & 0xf];
-    value >>= 4;
-  } while (value != 0);
-  buffer[i] = '\0';
-  for (a = 2, b = i - 1; a < b; a++, b--) {
-    const char c = buffer[a];
-
-    buffer[a] = buffer[b];
-    buffer[b] = c;
-  }
-}
-
-/* HTS_FALSE if src does not fit: a truncated module path would point the
-   symbolizer at the wrong file. */
-static hts_boolean copy_bounded(char *dest, size_t size, const char *src) {
-  size_t i;
-
-  for (i = 0; i < size - 1 && src[i] != '\0'; i++) {
-    dest[i] = src[i];
-  }
-  dest[i] = '\0';
-  return src[i] == '\0' ? HTS_TRUE : HTS_FALSE;
-}
-
-/* Run the symbolizer on argv, output on fd, within *budget ticks. HTS_FALSE
-   only if none could be run at all; otherwise silent, the raw trace stands. */
-static hts_boolean spawn_symbolizer(char **argv, int fd, int *budget) {
-  const pid_t pid = fork();
-  int status = 0;
-
-  if (pid == -1)
-    return HTS_FALSE;
-  if (pid == 0) {
-    static char llvm_prog[] = "llvm-symbolizer";
-    static char llvm_opts[] = "-p";
-
-    dup2(fd, 1); /* both symbolizers write on stdout */
-    execvp(argv[0], argv);
-    argv[0] = llvm_prog; /* an LLVM-only install ships no addr2line */
-    argv[1] = llvm_opts;
-    execvp(argv[0], argv);
-    _exit(BT_NO_SYMBOLIZER);
-  }
-  for (; *budget > 0; (*budget)--) {
-    const struct timespec tick = {0, 10 * 1000 * 1000};
-    const pid_t reaped = waitpid(pid, &status, WNOHANG);
-
-    if (reaped == pid)
-      return WIFEXITED(status) && WEXITSTATUS(status) == BT_NO_SYMBOLIZER
-                 ? HTS_FALSE
-                 : HTS_TRUE;
-    if (reaped == -1 && errno != EINTR)
-      return HTS_TRUE;
-    nanosleep(&tick, NULL);
-  }
-  kill(pid, SIGKILL);
-  waitpid(pid, NULL, 0);
-  return HTS_TRUE;
-}
-
-/* Name the frames backtrace_symbols_fd() leaves as module+offset:
-   -fvisibility=hidden keeps them out of .dynsym, but DWARF has them. dladdr()
-   is not formally async-signal-safe; accepted, this path is already fatal. */
-static void symbolize_backtrace(void *const *stack, int size, int fd) {
-  static char prog[] = "addr2line";
-  static char opts[] = "-Cfipa";
-  static char dashe[] = "-e";
-  char hex[BT_MAX_FRAMES][BT_HEX_SIZE];
-  const void *base[BT_MAX_FRAMES];
-  const char *name[BT_MAX_FRAMES];
-  hts_boolean grouped[BT_MAX_FRAMES];
-  char module[BT_PATH_SIZE];
-  char *argv[4 + BT_MAX_FRAMES + 1];
-  int budget = BT_WAIT_TICKS;
-  int i, spawned;
-
-  if (size > BT_MAX_FRAMES)
-    size = BT_MAX_FRAMES;
-
-  for (i = 0; i < size; i++) {
-    Dl_info info;
-
-    grouped[i] = HTS_TRUE; /* skipped unless dladdr() places the frame */
-    if (dladdr(stack[i], &info) == 0 || info.dli_fname == NULL ||
-        info.dli_fname[0] == '\0')
-      continue;
-    base[i] = info.dli_fbase;
-    name[i] = info.dli_fname;
-    print_hex(hex[i], (uintptr_t) ((const char *) stack[i] -
-                                   (const char *) info.dli_fbase));
-    grouped[i] = HTS_FALSE;
-  }
-
-  /* One child per module: addr2line takes a single -e. Each frame is claimed
-     once, so argc cannot exceed argv[]. */
-  for (spawned = 0; spawned < BT_MAX_MODULES; spawned++) {
-    int first, j, argc = 0;
-
-    for (first = 0; first < size && grouped[first]; first++)
-      ;
-    if (first >= size)
-      break;
-    argv[argc++] = prog;
-    argv[argc++] = opts;
-    argv[argc++] = dashe;
-    argv[argc++] = module;
-    for (j = first; j < size; j++) {
-      if (grouped[j] || base[j] != base[first])
-        continue;
-      grouped[j] = HTS_TRUE;
-      argv[argc++] = hex[j];
-    }
-    argv[argc] = NULL;
-    /* access(): skip pseudo-modules like linux-vdso, which have no file and
-       would draw nothing but an addr2line complaint. */
-    if (copy_bounded(module, sizeof(module), name[first]) &&
-        access(module, R_OK) == 0) {
-      const size_t len = strlen(module);
-
-      /* addr2line -a prints offsets only: say which module they are in. */
-      (void) (write(fd, module, len) == (ssize_t) len);
-      (void) (write(fd, ":\n", 2) == 2);
-      if (!spawn_symbolizer(argv, fd, &budget))
-        break; /* no symbolizer: stop at one header */
-    }
-  }
-}
-#endif
-
-void hts_backtrace_init(void) {
-#ifdef USES_BACKTRACE
-  symbolize_crash =
-      getenv("HTTRACK_NO_SYMBOLIZE") == NULL ? HTS_TRUE : HTS_FALSE;
-#endif
-}
-
-void hts_print_backtrace(int fd) {
-#ifdef USES_BACKTRACE
-  void *stack[256];
-  const int size = backtrace(stack, sizeof(stack) / sizeof(stack[0]));
-
-  /* A fault inside the handler lands back here: symbolizing twice interleaves
-     two traces on fd and spends a second budget. */
-  static volatile sig_atomic_t entered = 0;
-
-  if (size != 0) {
-    backtrace_symbols_fd(stack, size, fd);
-    if (symbolize_crash && entered == 0) {
-      entered = 1;
-      symbolize_backtrace(stack, size, fd);
-      entered = 0;
-    }
-  }
-#else
-  const char msg[] = "No stack trace available on this OS :(\n";
-
-  if (write(fd, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
-    /* sorry GCC */
-  }
-#endif
-}

src/htsbacktrace.h

@@ -1,45 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 1998 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: crash backtrace printer                                 */
-/* Author: Xavier Roche                                          */
-/* ------------------------------------------------------------ */
-
-#ifndef HTSBACKTRACE_DEFH
-#define HTSBACKTRACE_DEFH
-
-/* Sample HTTRACK_NO_SYMBOLIZE before any crash: getenv() is not signal-safe.
-   Call once, from the process that installs the fatal-signal handlers. */
-void hts_backtrace_init(void);
-
-/* Write the calling thread's stack to fd, callable from a fatal signal handler:
-   raw frames first, then whatever an external symbolizer can name. Allocates
-   nothing; prints a one-line notice where the OS has no backtrace(). */
-void hts_print_backtrace(int fd);
-
-#endif

src/htscache.c

@@ -142,13 +142,10 @@ struct cache_back_zip_entry {
   int compressionMethod;
 };
 
-/* Clip: strlcpybuff aborts on overflow, which would take the crawl down on a
-   corrupt cache carrying a field wider than ours. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (refvalue)[0] = '\0';                                                    \
-      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
+      strlcpybuff(refvalue, value, refvalue_size);                             \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -632,8 +629,8 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
                 // File exists on disk with declared cache name (this is expected!)
                 if (fexist_utf8(fconv(catbuff, sizeof(catbuff), previous_save))) {       // un fichier existe déja
                   // Expected size ?
-                  const LLint fsize = fsize_utf8(
-                      fconv(catbuff, sizeof(catbuff), previous_save));
+                  const size_t fsize =
+                    fsize_utf8(fconv(catbuff, sizeof(catbuff), previous_save));
                   if (fsize == r.size) {
                     // Target name is the previous name, and the file looks good: nothing to do!
                     if (strcmp(previous_save, target_save) == 0) {
@@ -669,8 +666,7 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
                 // Suppose a broken mirror, with a file being renamed: OK
                 else if (fexist_utf8(fconv(catbuff, sizeof(catbuff), target_save))) {
                   // Expected size ?
-                  const LLint fsize =
-                      fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
+                  const size_t fsize = fsize_utf8(fconv(catbuff, sizeof(catbuff), target_save));
 
                   if (fsize == r.size) {
                     // So far so good
@@ -1444,7 +1440,7 @@ int cache_brstr(char *adr, char *s, size_t s_size) {
 
 /* binput bounded to a NUL-terminated buffer: refuse to start a read at or
    past `end`, so a prior over-advance can't walk a cache-index parse OOB. */
-int cache_binput(const char *adr, const char *end, char *s, int max) {
+int cache_binput(char *adr, const char *end, char *s, int max) {
   if (adr >= end) {
     s[0] = '\0';
     return 0;

src/htscache.h

@@ -93,7 +93,7 @@ void cache_rstr(FILE *fp, char *s, size_t s_size);
 char *cache_rstr_addr(FILE * fp);
 int cache_brstr(char *adr, char *s, size_t s_size);
 /* binput over a NUL-terminated buffer, bounded: no read starts at/past end. */
-int cache_binput(const char *adr, const char *end, char *s, int max);
+int cache_binput(char *adr, const char *end, char *s, int max);
 int cache_brint(char *adr, int *i);
 void cache_rint(FILE * fp, int *i);
 void cache_rLLint(FILE * fp, LLint * i);

src/htscache_selftest.c

@@ -1195,12 +1195,6 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
 
 /* --- read-side corruption injection --------------------------------------- */
 
-/* 100 'A's: a placeholder header line long enough to be overwritten by a
-   forged, over-long X-StatusMessage of the same byte length. */
-#define CORRUPT_LONG_ETAG                                                      \
-  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"                         \
-  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-
 /* canary read back intact after each corruption; victim gets the byte surgery
  */
 #define CORRUPT_ADR "corrupt.example.com"
@@ -1225,24 +1219,6 @@ static void corrupt_build(httrackp *opt) {
   selftest_close(&cache);
 }
 
-/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
-static void corrupt_build_longetag(httrackp *opt) {
-  cache_back cache;
-
-  memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
-  memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
-  remove(reconcile_st_path(opt, "hts-cache/new.zip"));
-  remove(reconcile_st_path(opt, "hts-cache/old.zip"));
-  selftest_open_for_write(&cache, opt);
-  store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
-              "OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
-              strlen(corrupt_body_a));
-  store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
-              "OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
-              corrupt_body_b, strlen(corrupt_body_b));
-  selftest_close(&cache);
-}
-
 /* Like corrupt_build, but the victim carries a 20-char Etag whose header line
    is later overwritten with a forged oversized X-Size (same byte length). */
 static void corrupt_build_etag(httrackp *opt) {
@@ -1427,48 +1403,6 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
   return fail;
 }
 
-/* An over-long field from a foreign cache must clip, not abort: the entry
-   still reads, clipped to capacity, and the canary survives. */
-static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
-                                         size_t wantlastmod, const char *what) {
-  cache_back cache;
-  htsblk v, c;
-  char BIGSTK lv[HTS_URLMAXSIZE * 2];
-  char BIGSTK lc[HTS_URLMAXSIZE * 2];
-  int fail = 0;
-
-  selftest_open_for_read(&cache, opt);
-  lv[0] = lc[0] = '\0';
-  v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
-  if (v.statuscode != 200) {
-    fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
-            v.statuscode);
-    fail++;
-  }
-  if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
-    fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
-            (unsigned) strlen(v.msg), (unsigned) wantmsg);
-    fail++;
-  }
-  if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
-    fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
-            what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
-    fail++;
-  }
-  c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
-  if (c.statuscode != 200) {
-    fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
-            c.statuscode);
-    fail++;
-  }
-  if (v.adr != NULL)
-    freet(v.adr);
-  if (c.adr != NULL)
-    freet(c.adr);
-  selftest_close(&cache);
-  return fail;
-}
-
 /* One zip corruption case: build, patch, then check victim+canary in-session.
  */
 static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1505,31 +1439,6 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
   failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
                                     "garbled deflate stream");
 
-  /* A corrupt cache can hold a field wider than ours. Clipping keeps the
-     entry; aborting would take the crawl down. Overwrite the placeholder Etag
-     line in place, same byte length, so the zip offsets stay intact. */
-  corrupt_build_longetag(opt);
-  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
-                "X-StatusMessage: " /* 17 + 89 = 106 */
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
-                1, 1);
-  failures +=
-      corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
-                                    (size_t) -1, "over-long X-StatusMessage");
-
-  /* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
-     cannot satisfy both. */
-  corrupt_build_longetag(opt);
-  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
-                "Last-Modified: " /* 15 + 91 = 106 */
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
-                1, 1);
-  failures += corrupt_expect_victim_clipped(
-      opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
-      "over-long Last-Modified");
-
   /* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
      truncates negative in the (int) cast the malloc uses: a wraparound alloc.
      cache_add asserts size fits an int, so such a value only reaches the reader

src/htscmdline.c

@@ -1,95 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 1998 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: command line splitter, shared by the engine and         */
-/*       htsserver                                               */
-/* Author: Xavier Roche                                          */
-/* ------------------------------------------------------------ */
-
-#include "htscmdline.h"
-
-#include "htssafe.h"
-
-#include <limits.h>
-#include <stdint.h>
-
-char **hts_split_cmdline(char *cmd, int *nargs) {
-  size_t nsep = 0;
-  size_t capacity;
-  size_t r;
-  size_t w;
-  int argc = 0;
-  hts_boolean quoted = HTS_FALSE;
-  char **argv;
-
-  *nargs = 0;
-
-  /* fold the other separators, so counting them sizes the vector exactly */
-  for (r = 0; cmd[r] != '\0'; r++) {
-    if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
-      cmd[r] = ' ';
-    }
-    if (cmd[r] == ' ') {
-      nsep++;
-    }
-  }
-
-  /* at most one argument per separator, plus the leading one and the NULL */
-  if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
-    return NULL;
-  }
-  capacity = nsep + 2;
-  argv = (char **) malloct(capacity * sizeof(char *));
-  if (argv == NULL) {
-    return NULL;
-  }
-
-  argv[argc++] = cmd;
-  for (r = 0, w = 0; cmd[r] != '\0';) {
-    if (quoted && cmd[r] == '\\' &&
-        (cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
-      r++;
-      cmd[w++] = cmd[r++];
-    } else if (cmd[r] == '\"') {
-      quoted = !quoted;
-      cmd[w++] = cmd[r++];
-    } else if (cmd[r] == ' ' && !quoted) {
-      cmd[w++] = '\0';
-      assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
-      argv[argc++] = cmd + w;
-      r++;
-    } else {
-      cmd[w++] = cmd[r++];
-    }
-  }
-  cmd[w] = '\0';
-  argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
-
-  *nargs = argc;
-  return argv;
-}

src/htscmdline.h

@@ -1,45 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 1998 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: command line splitter, shared by the engine and         */
-/*       htsserver                                               */
-/* Author: Xavier Roche                                          */
-/* ------------------------------------------------------------ */
-
-#ifndef HTSCMDLINE_DEFH
-#define HTSCMDLINE_DEFH
-
-#include "htsglobal.h"
-
-/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
-   argv[0] being the program name and quotes left for the engine to strip.
-   Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
-   entries), or NULL when it cannot be sized or allocated. */
-char **hts_split_cmdline(char *cmd, int *nargs);
-
-#endif

src/htscore.c

@@ -39,7 +39,6 @@ Please visit our Website: http://www.httrack.com
 
 /* File defs */
 #include "htscore.h"
-#include "htssitemap.h"
 #include "htswarc.h"
 
 /* specific definitions */
@@ -448,22 +447,13 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
                           const char *fil) {
   if (!*makeindex_done) {
     if (*makeindex_fp) {
-      /* sized off link_escaped below: at the old flat 1024 a long first link
-         produced a redirect to a clipped URL */
-      char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
+      char BIGSTK tempo[1024];
       if (makeindex_links == 1) {
         char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
         escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
-        /* no redirect beats one pointing at a clipped URL */
-        if (!sprintfbuff(
-                tempo,
-                "<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
-                link_escaped)) {
-          hts_log_print(opt, LOG_WARNING,
-                        "index redirect omitted: first link too long (%s)",
-                        makeindex_firstlink);
-          tempo[0] = '\0';
-        }
+        snprintf(tempo, sizeof(tempo),
+                 "<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
+                 link_escaped);
       } else
         tempo[0] = '\0';
       hts_template_format(*makeindex_fp, template_footer,
@@ -706,19 +696,17 @@ int httpmirror(char *url1, httrackp * opt) {
   // copier adresse(s) dans liste des adresses
   {
     char *a = url1;
-    const LLint list_sz = StringNotEmpty(opt->filelist)
-                              ? fsize_utf8(StringBuff(opt->filelist))
-                              : 0;
-    /* two bytes reserved per list byte; -1 makes an undoublable size refused */
-    const LLint list_room =
-        list_sz > 0 ? (list_sz <= INT64_MAX / 2 ? list_sz * 2 : -1) : 0;
-    const size_t primary_len =
-        llint_grow_size_t(8192 + strlen(url1) * 2, list_room, 0);
+    int primary_len = 8192;
+
+    if (StringNotEmpty(opt->filelist)) {
+      primary_len += max(0, fsize_utf8(StringBuff(opt->filelist)) * 2);
+    }
+    primary_len += (int) strlen(url1) * 2;
 
     // création de la première page, qui contient les liens de base à scanner
     // c'est plus propre et plus logique que d'entrer à la main les liens dans la pile
     // on bénéficie ainsi des vérifications et des tests du robot pour les liens "primaires"
-    primary = primary_len != (size_t) -1 ? (char *) malloct(primary_len) : NULL;
+    primary = (char *) malloct(primary_len);
     if (!primary) {
       printf("PANIC! : Not enough memory [%d]\n", __LINE__);
       XH_extuninit;
@@ -899,7 +887,7 @@ int httpmirror(char *url1, httrackp * opt) {
       }
 
       if (filelist_buff != NULL) {
-        size_t filelist_ptr = 0;
+        int filelist_ptr = 0;
         int n = 0;
         char BIGSTK line[HTS_URLMAXSIZE * 2];
 
@@ -944,22 +932,6 @@ int httpmirror(char *url1, httrackp * opt) {
     heap_top()->premier = heap_top_index();        // premier lien, objet-père=objet              
     heap_top()->precedent = heap_top_index();      // lien précédent
 
-    /* --sitemap: queue the sitemap probe just after the seeds, so its URLs are
-       injected before the crawl gets far. */
-    hts_sitemap_free(opt); /* an earlier mirror may have left a doc list */
-    if (opt->sitemap || StringNotEmpty(opt->sitemap_url)) {
-      char BIGSTK first[HTS_URLMAXSIZE * 2];
-      const char *const eol = strchr(primary, '\n');
-      const size_t len = eol != NULL ? (size_t) (eol - primary) : 0;
-
-      first[0] = '\0';
-      if (len > 0 && len < sizeof(first)) {
-        memcpy(first, primary, len);
-        first[len] = '\0';
-      }
-      hts_sitemap_seed(opt, first);
-    }
-
     // Initialiser cache
     {
       opt->state._hts_in_html_parsing = 4;
@@ -1604,21 +1576,11 @@ int httpmirror(char *url1, httrackp * opt) {
         stre.maketrack_fp = maketrack_fp;
 
         /* Parse */
-        {
-          const int nlinks = opt->lien_tot;
-
-          if (hts_mirror_check_moved(&str, &stre) != 0) {
-            XH_uninit;
-            return -1;
-          }
-          /* A redirect re-queues the target as a fresh link; without carrying
-             the marking over, a moved sitemap is fetched and then ignored. */
-          if (opt->sitemap_state != NULL && opt->lien_tot > nlinks &&
-              hts_sitemap_pending(opt, urladr(), urlfil())) {
-            hts_sitemap_redirect(opt, urladr(), urlfil(), heap_top()->adr,
-                                 heap_top()->fil);
-          }
+        if (hts_mirror_check_moved(&str, &stre) != 0) {
+          XH_uninit;
+          return -1;
         }
+
       }
 
     }                           // if !error
@@ -1636,29 +1598,6 @@ int httpmirror(char *url1, httrackp * opt) {
       /* Load file and decode if necessary, after redirect check. */
       LOAD_IN_MEMORY_IF_NECESSARY();
 
-      /* Sitemap document: turn its <loc> URLs into top-level seeds. They go
-         through htsAddLink, so the wizard's filters and scope rules decide, and
-         this link's max depth leaves them the full budget. */
-      if (opt->sitemap_state != NULL &&
-          hts_sitemap_pending(opt, urladr(), urlfil())) {
-        htsmoduleStruct BIGSTK smstr;
-        int smptr = ptr;
-
-        memset(&smstr, 0, sizeof(smstr));
-        smstr.opt = opt;
-        smstr.sback = sback;
-        smstr.cache = &cache;
-        smstr.hashptr = hashptr;
-        smstr.numero_passe = numero_passe;
-        smstr.ptr_ = &smptr; /* scratch: the ingester retargets the wizard */
-        smstr.addLink = htsAddLink;
-        smstr.url_host = urladr();
-        smstr.url_file = urlfil();
-        smstr.mime = r.contenttype;
-        hts_sitemap_ingest(opt, &smstr, urladr(), urlfil(), r.adr,
-                           r.adr != NULL && r.size > 0 ? (size_t) r.size : 0);
-      }
-
       // ------------------------------------------------------
       // ok, fichier chargé localement
       // ------------------------------------------------------
@@ -1869,9 +1808,6 @@ int httpmirror(char *url1, httrackp * opt) {
 
         if (strnotempty(savename()) == 0) {       // pas de chemin de sauvegarde
           if (strcmp(urlfil(), "/robots.txt") == 0) {     // robots.txt
-            char BIGSTK sitemaps[8192];
-
-            sitemaps[0] = '\0';
             if (r.adr) {
               char BIGSTK infobuff[8192];
 #ifdef IGNORE_RESTRICTIVE_ROBOTS
@@ -1883,8 +1819,7 @@ int httpmirror(char *url1, httrackp * opt) {
 #endif
 
               robots_parse(&robots, urladr(), r.adr, r.size, infobuff,
-                           sizeof(infobuff), keep_root, sitemaps,
-                           sizeof(sitemaps));
+                           sizeof(infobuff), keep_root);
               if (strnotempty(infobuff)) {
                 hts_log_print(opt, LOG_INFO,
                               "Note: robots.txt forbidden links for %s are: %s",
@@ -1894,10 +1829,6 @@ int httpmirror(char *url1, httrackp * opt) {
                               urladr(), infobuff);
               }
             }
-            /* After robots_parse, so the rules this very body carries already
-               gate the sitemap fetch. Runs even on a failed probe, which is
-               what falls back to the well-known location. */
-            hts_sitemap_robots(opt, urladr(), sitemaps);
           }
         } else if (r.is_write) {        // déja sauvé sur disque
           /*
@@ -2314,7 +2245,6 @@ int httpmirror(char *url1, httrackp * opt) {
   // ending
   usercommand(opt, 0, NULL, NULL, NULL, NULL);
   warc_close_opt(opt);
-  hts_sitemap_free(opt);
 
   // désallocation mémoire & buffers
   XH_uninit;
@@ -3702,11 +3632,6 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
   to->warc_cdx = from->warc_cdx;
   to->warc_wacz = from->warc_wacz;
 
-  if (from->sitemap)
-    to->sitemap = from->sitemap;
-  if (StringNotEmpty(from->sitemap_url))
-    StringCopyS(to->sitemap_url, from->sitemap_url);
-
   if (from->pause_max_ms > 0) {
     to->pause_min_ms = from->pause_min_ms;
     to->pause_max_ms = from->pause_max_ms;

src/htscoremain.c

@@ -145,7 +145,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
   int argv_url = -1;            // ==0 : utiliser cache et doit.log
   char *argv_firsturl = NULL;   // utilisé pour nommage par défaut
   char *url = NULL;             // URLS séparées par un espace
-  size_t url_sz = 65535;
+  int url_sz = 65535;
 
   // the parametres
   int httrack_logmode = 3;      // ONE log file
@@ -224,22 +224,21 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
   /* create x_argvblk buffer for transformed command line */
   {
-    size_t current_size = 0;
-    const LLint size = fsize("config");
-    size_t blk_size;
+    int current_size = 0;
+    int size;
     int na;
 
     for(na = 0; na < argc; na++)
-      current_size += strlen(argv[na]) + 1;
-    /* a huge file named "config" must saturate, not wrap, the capacity */
-    blk_size = llint_grow_size_t(current_size, size > 0 ? size : 0, 32768);
-    x_argvblk = blk_size != (size_t) -1 ? (char *) malloct(blk_size) : NULL;
+      current_size += (int) (strlen(argv[na]) + 1);
+    if ((size = fsize("config")) > 0)
+      current_size += size;
+    x_argvblk = (char *) malloct(current_size + 32768);
     if (x_argvblk == NULL) {
       HTS_PANIC_PRINTF("Error, not enough memory");
       htsmain_free();
       return -1;
     }
-    x_argvblk_size = blk_size;
+    x_argvblk_size = (size_t) (current_size + 32768);
     x_argvblk[0] = '\0';
     x_ptr = 0;
 
@@ -1457,29 +1456,20 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                     FILE *fp = FOPEN(argv[na], "rb");
 
                     if (fp != NULL) {
-                      size_t cl = strlen(url);
-                      const size_t fzs = llint_to_size_t(fz);
-                      const size_t capa = llint_grow_size_t(cl, fz, 8192);
+                      int cl = (int) strlen(url);
 
-                      if (capa == (size_t) -1) {
-                        fclose(fp);
-                        HTS_PANIC_PRINTF("File url list too large");
-                        htsmain_free();
-                        return -1;
-                      }
-                      ensureUrlCapacity(url, url_sz, capa);
+                      ensureUrlCapacity(url, url_sz, cl + fz + 8192);
                       if (cl > 0) {     /* don't stick! (3.43) */
                         url[cl] = ' ';
                         cl++;
                       }
-                      if (fread(url + cl, 1, fzs, fp) != fzs) {
-                        fclose(fp);
+                      if (fread(url + cl, 1, fz, fp) != fz) {
                         HTS_PANIC_PRINTF("File url list could not be read");
                         htsmain_free();
                         return -1;
                       }
                       fclose(fp);
-                      *(url + cl + fzs) = '\0';
+                      *(url + cl + fz) = '\0';
                     }
                   }
                 }
@@ -1795,26 +1785,6 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   StringCopy(opt->warc_file, WARC_AUTONAME);
                 }
                 break;
-              case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps
-                if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap
-                  com++;
-                  if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
-                    HTS_PANIC_PRINTF(
-                        "Option sitemap-url needs a blank space and a URL");
-                    htsmain_free();
-                    return -1;
-                  }
-                  na++;
-                  if (strlen(argv[na]) >= HTS_URLMAXSIZE) {
-                    HTS_PANIC_PRINTF("Sitemap URL too long");
-                    htsmain_free();
-                    return -1;
-                  }
-                  StringCopy(opt->sitemap_url, argv[na]);
-                } else { // --sitemap: robots.txt probe, then /sitemap.xml
-                  opt->sitemap = HTS_TRUE;
-                }
-                break;
               case 'Y': // why: explain the filter verdict for a URL, no crawl
                 if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
                   HTS_PANIC_PRINTF("Option why needs a blank space and a URL");
@@ -2326,8 +2296,8 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
       } else {                  // URL/filters
         char catbuff[CATBUFF_SIZE];
-        const size_t urlSize = strlen(argv[na]);
-        const size_t capa = strlen(url) + urlSize + 32;
+        const int urlSize = (int) strlen(argv[na]);
+        const int capa = (int) (strlen(url) + urlSize + 32);
 
         assertf(urlSize < HTS_URLMAXSIZE);
         if (urlSize < HTS_URLMAXSIZE) {

src/htsftp.c

@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
       // folding a nonsense port into 1..65535 fetches one the link never named;
       // an empty "host:" just means the default (#614)
       if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
-        htsblk_failf(&back->r, "Invalid port: %s", a + 1);
+        snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
         back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
         _HALT_FTP return 0;
       }
@@ -262,7 +262,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
     // récupérer adresse résolue
     strcpybuff(back->info, "host name");
     if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
-      htsblk_failf(&back->r, "Unable to get server's address: %s", error);
+      snprintf(back->r.msg, sizeof(back->r.msg),
+               "Unable to get server's address: %s", error);
       back->r.statuscode = STATUSCODE_NON_FATAL;
       _HALT_FTP return 0;
     }
@@ -331,15 +332,18 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
             }
 
           } else {
-            htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
+            snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
+                     linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }
         } else {
-          htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
+          snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
+                   linejmp(line));
           back->r.statuscode = STATUSCODE_INVALID;
         }
       } else {
-        htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
+        snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
+                 linejmp(line));
         back->r.statuscode = STATUSCODE_INVALID;
       }
 
@@ -406,7 +410,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
             }
             // -- fin analyse de l'adresse IP et du port --
           } else {
-            htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
+            snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
+                     linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         } else {
@@ -437,11 +442,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 }
               }
             } else {
-              htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
+              snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
+                       linejmp(line));
               back->r.statuscode = STATUSCODE_INVALID;
             }
           } else {
-            htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
+            snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
+                     linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         }
@@ -547,8 +554,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                   deletesoc(soc_dat);
                   soc_dat = INVALID_SOCKET;
                   //
-                  htsblk_failf(&back->r, "RETR command error: %s",
-                               linejmp(line));
+                  snprintf(back->r.msg, sizeof(back->r.msg),
+                           "RETR command error: %s", linejmp(line));
                   back->r.statuscode = STATUSCODE_INVALID;
                 }               // sinon on est prêts
               } else {
@@ -566,12 +573,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
               back->r.statuscode = STATUSCODE_INVALID;
             }                   // sinon on est prêts
           } else {
-            htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
-                         error);
+            snprintf(back->r.msg, sizeof(back->r.msg),
+                     "Unable to resolve IP %s: %s", adr_ip, error);
             back->r.statuscode = STATUSCODE_INVALID;
           }                     // sinon on est prêts
         } else {
-          htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
+          snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
+                   linejmp(line));
           back->r.statuscode = STATUSCODE_INVALID;
         }                       // sinon on est prêts
 #else
@@ -595,11 +603,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 back->r.statuscode = STATUSCODE_INVALID;
               }
             } else {
-              htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
+              snprintf(back->r.msg, sizeof(back->r.msg),
+                       "RETR command error: %s", linejmp(line));
               back->r.statuscode = STATUSCODE_INVALID;
             }
           } else {
-            htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
+            snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
+                     linejmp(line));
             back->r.statuscode = STATUSCODE_INVALID;
           }
 #ifdef _WIN32
@@ -642,7 +652,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 len = 0;        // fin
                 break;
               case 0:
-                htsblk_failf(&back->r, "Time out (%d)", timeout);
+                snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
+                         timeout);
                 back->r.statuscode = STATUSCODE_INVALID;
                 len = 0;        // fin
                 break;
@@ -705,7 +716,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                 strcpybuff(back->r.msg, "OK");
                 back->r.statuscode = HTTP_OK;
               } else {
-                htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
+                snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
+                         linejmp(line));
                 back->r.statuscode = STATUSCODE_INVALID;
               }
             } else {

src/htsglobal.h

@@ -72,8 +72,7 @@ Please visit our Website: http://www.httrack.com
    HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
    static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
    compiler type-checks the format string at argument index fmt against the
-   varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
-   error signal, so dropping it is a bug; a (void) cast does not silence it. */
+   varargs starting at arg. */
 #ifndef HTS_UNUSED
 #ifdef __GNUC__
 #define HTS_UNUSED __attribute__((unused))
@@ -81,13 +80,10 @@ Please visit our Website: http://www.httrack.com
 #define HTS_STATIC static __attribute__((unused))
 
 #define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
-
-#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
 #else
 #define HTS_UNUSED
 #define HTS_STATIC static
 #define HTS_PRINTF_FUN(fmt, arg)
-#define HTS_CHECK_RESULT
 #endif
 #endif
 
@@ -340,20 +336,16 @@ typedef int INTsys;
 #endif
 
 /* Socket-handle type. An unsigned integer wide enough for a Windows SOCKET;
-   a plain int file descriptor on POSIX. T_SOCP is its printf conversion,
-   '%' included: unsigned __int64 on Win64 must not be printed with "%d". */
+   a plain int file descriptor on POSIX. */
 #ifdef _WIN32
 #if defined(_WIN64)
 
 typedef unsigned __int64 T_SOC;
-#define T_SOCP "%" PRIu64
 #else
 typedef unsigned __int32 T_SOC;
-#define T_SOCP "%" PRIu32
 #endif
 #else
 typedef int T_SOC;
-#define T_SOCP "%d"
 #endif
 
 /* Buffer size for a printed network address (IPv4 or IPv6, NUL included). */

src/htshelp.c

@@ -124,9 +124,7 @@ typedef struct help_wizard_buffers {
   char stropt[2048];        // options
   char stropt2[2048];       // options longues
   char strwild[2048];       // wildcards
-  /* holds all four of the above plus separators: at 4096 a long answer set
-     clipped the filters off the command line */
-  char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
+  char cmd[4096];
   char str[256];
   char *argv[256];
 } help_wizard_buffers;
@@ -158,7 +156,9 @@ void help_wizard(httrackp * opt) {
   char *a;
 
   //
-  if (buffers == NULL) {
+  if (urls == NULL || mainpath == NULL || projname == NULL || stropt == NULL
+      || stropt2 == NULL || strwild == NULL || cmd == NULL || str == NULL
+      || argv == NULL) {
     fprintf(stderr, "* memory exhausted in %s, line %d\n", __FILE__, __LINE__);
     return;
   }
@@ -251,7 +251,6 @@ void help_wizard(httrackp * opt) {
       strcatbuff(stropt2, "--update ");
       break;
     case 0:
-      freet(buffers);
       return;
       break;
     }
@@ -310,23 +309,14 @@ void help_wizard(httrackp * opt) {
       printf("\n");
       if (strlen(stropt) == 1)
         stropt[0] = '\0';       // aucune
-      /* the tail is the filter list, and cmd is split into the argv handed to
-         hts_main() below: a clipped line would silently widen the crawl */
-      if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
-        printf("* command line too long (%d bytes max)\n",
-               (int) sizeof(cmd) - 1);
-        freet(buffers);
-        return;
-      }
+      snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
       printf("---> Wizard command line: httrack %s\n\n", cmd);
       printf("Ready to launch the mirror? (Y/n) :");
       fflush(stdout);
       linput(stdin, str, 250);
       if (strnotempty(str)) {
-        if (!((str[0] == 'y') || (str[0] == 'Y'))) {
-          freet(buffers);
+        if (!((str[0] == 'y') || (str[0] == 'Y')))
           return;
-        }
       }
       printf("\n");
 
@@ -350,7 +340,7 @@ void help_wizard(httrackp * opt) {
   }
 
   /* Free buffers */
-  freet(buffers);
+  free(buffers);
 #undef urls
 #undef mainpath
 #undef projname
@@ -433,8 +423,7 @@ void help_catchurl(const char *dest_path) {
       }
       // former URL!
       {
-        /* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
-        char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
+        char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
 
         inplace_escape_check_url(dest, sizeof(dest));
         snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);
@@ -526,11 +515,6 @@ void help(const char *app, int more) {
     (" %L <file> add all URL located in this text file (one URL per line)");
   infomsg
     (" %S <file> add all scan rules located in this text file (one scan rule per line)");
-  infomsg(" %m  seed the crawl from the site's sitemap (robots.txt Sitemap:, "
-          "then /sitemap.xml); --sitemap-url URL names one explicitly. A "
-          "sitemap you name, or one the site declares, is fetched even under "
-          "robots.txt Disallow; only the guessed /sitemap.xml obeys it. The "
-          "URLs found still pass every filter and scope rule");
   infomsg("");
   infomsg("Build options:");
   infomsg("  NN structure type (0 *original structure, 1+: see below)");

src/htslib.c

@@ -36,7 +36,6 @@ Please visit our Website: http://www.httrack.com
 // Fichier librairie .c
 
 #include "htscore.h"
-#include "htssitemap.h"
 #include "htswarc.h"
 
 /* specific definitions */
@@ -705,16 +704,18 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
   /* Check for errors */
   if (soc == INVALID_SOCKET) {
     if (retour) {
-      if (!strnotempty(retour->msg)) {
+      if (retour->msg) {
+        if (!strnotempty(retour->msg)) {
 #ifdef _WIN32
-        int last_errno = WSAGetLastError();
+          int last_errno = WSAGetLastError();
 
-        htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
+          sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
 #else
-        int last_errno = errno;
+          int last_errno = errno;
 
-        htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
+          sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
 #endif
+        }
       }
     }
   }
@@ -2206,7 +2207,7 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #if DEBUG
       printf("erreur gethostbyname\n");
 #endif
-      if (retour != NULL) {
+      if (retour && retour->msg) {
 #ifdef _WIN32
         snprintf(retour->msg, sizeof(retour->msg),
                  "Unable to get server's address: %s", error);
@@ -2237,17 +2238,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
     DEBUG_W("socket()=%d\n" _(int) soc);
 #endif
     if (soc == INVALID_SOCKET) {
-      if (retour != NULL) {
+      if (retour && retour->msg) {
 #ifdef _WIN32
         int last_errno = WSAGetLastError();
 
-        htsblk_failf(retour, "Unable to create a socket: %s",
-                     strerror(last_errno));
+        sprintf(retour->msg, "Unable to create a socket: %s",
+                strerror(last_errno));
 #else
         int last_errno = errno;
 
-        htsblk_failf(retour, "Unable to create a socket: %s",
-                     strerror(last_errno));
+        sprintf(retour->msg, "Unable to create a socket: %s",
+                strerror(last_errno));
 #endif
       }
       return INVALID_SOCKET;    // erreur création socket impossible
@@ -2261,8 +2262,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
                              &bind_addr, &error) == NULL
           || bind(soc, &SOCaddr_sockaddr(bind_addr), 
                   SOCaddr_size(bind_addr)) != 0) {
-        snprintf(retour->msg, sizeof(retour->msg),
-                 "Unable to bind the specificied server address: %s", error);
+        if (retour && retour->msg) {
+#ifdef _WIN32
+          snprintf(retour->msg, sizeof(retour->msg),
+                   "Unable to bind the specificied server address: %s",
+                   error);
+#else
+          snprintf(retour->msg, sizeof(retour->msg),
+                   "Unable to bind the specificied server address: %s",
+                   error);
+#endif
+        }
         deletesoc(soc);
         return INVALID_SOCKET;
       }
@@ -2309,17 +2319,17 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #if HDEBUG
         printf("unable to connect!\n");
 #endif
-        if (retour != NULL) {
+        if (retour != NULL && retour->msg) {
 #ifdef _WIN32
           const int last_errno = WSAGetLastError();
 
-          htsblk_failf(retour, "Unable to connect to the server: %s",
-                       strerror(last_errno));
+          sprintf(retour->msg, "Unable to connect to the server: %s",
+                  strerror(last_errno));
 #else
           const int last_errno = errno;
 
-          htsblk_failf(retour, "Unable to connect to the server: %s",
-                       strerror(last_errno));
+          sprintf(retour->msg, "Unable to connect to the server: %s",
+                  strerror(last_errno));
 #endif
         }
         /* Close the socket and notify the error!!! */
@@ -2538,15 +2548,6 @@ void fil_simplifie(char *f) {
   }
 }
 
-void htsblk_failf(htsblk *r, const char *fmt, ...) {
-  va_list args;
-
-  va_start(args, fmt);
-  // deliberate clip: the reason is quoted from a remote reply
-  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
-  va_end(args);
-}
-
 // fermer liaison fichier ou socket
 void deletehttp(htsblk * r) {
 #if HTS_DEBUG_CLOSESOCK
@@ -2597,15 +2598,13 @@ void deletesoc(T_SOC soc) {
     if (closesocket(soc) != 0) {
       int err = WSAGetLastError();
 
-      fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
-              strerror(err));
+      fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
     }
 #else
     if (close(soc) != 0) {
       const int err = errno;
 
-      fprintf(stderr, "* error closing socket " T_SOCP ": %s\n", soc,
-              strerror(err));
+      fprintf(stderr, "* error closing socket %d: %s\n", soc, strerror(err));
     }
 #endif
 #if HTS_WIDE_DEBUG
@@ -3018,7 +3017,7 @@ int finput(T_SOC fd, char *s, int max) {
 }
 
 // Like linput, but in memory (optimized)
-int binput(const char *buff, char *s, int max) {
+int binput(char *buff, char *s, int max) {
   int count = 0;
   int destCount = 0;
 
@@ -6020,7 +6019,6 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   StringCopy(opt->strip_query, "");
   StringCopy(opt->cookies_file, "");
   StringCopy(opt->warc_file, "");
-  StringCopy(opt->sitemap_url, "");
   opt->warc_max_size = 0; /* no rotation unless --warc-max-size sets it */
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
@@ -6174,8 +6172,6 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->cookies_file);
     StringFree(opt->why_url);
     StringFree(opt->warc_file);
-    StringFree(opt->sitemap_url);
-    hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */
 
     StringFree(opt->path_html);
     StringFree(opt->path_html_utf8);

src/htslib.h

@@ -199,10 +199,6 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
    etc.). */
 T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
                    int waitconnect, int addr_index, int *addr_count);
-/* Clips the formatted failure reason into r->msg, which also round-trips
-   through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
-void htsblk_failf(htsblk *r, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
-
 HTS_INLINE void deletehttp(htsblk * r);
 HTS_INLINE int deleteaddr(htsblk * r);
 HTS_INLINE void deletesoc(T_SOC soc);
@@ -266,7 +262,7 @@ HTS_INLINE void time_rfc822_local(char *s, struct tm *A);
 
 HTS_INLINE int sendc(htsblk * r, const char *s);
 int finput(T_SOC fd, char *s, int max);
-int binput(const char *buff, char *s, int max);
+int binput(char *buff, char *s, int max);
 int linput(FILE * fp, char *s, int max);
 int linputsoc(T_SOC soc, char *s, int max);
 int linputsoc_t(T_SOC soc, char *s, int max, int timeout);
@@ -611,21 +607,6 @@ static HTS_UNUSED size_t llint_to_size_t(LLint o) {
   }
 }
 
-/* Capacity for @p used bytes plus @p extra more plus @p slack spare;
-   (size_t) -1 if the total exceeds (size_t) -2 or @p extra is negative
-   (llint_to_size_t() would map that to a huge valid-looking size). */
-static HTS_UNUSED size_t llint_grow_size_t(size_t used, LLint extra,
-                                           size_t slack) {
-  const size_t max = (size_t) -2; /* (size_t) -1 is the error value */
-  const size_t e = extra >= 0 ? llint_to_size_t(extra) : (size_t) -1;
-
-  if (e == (size_t) -1 || used > max || slack > max - used ||
-      e > max - used - slack) {
-    return (size_t) -1;
-  }
-  return used + e + slack;
-}
-
 /* dirent() compatibility */
 #ifdef _WIN32
 /* Holds a UTF-8 d_name: MAX_PATH (260) UTF-16 units expand to <=3 bytes each.

src/htsopt.h

@@ -547,13 +547,6 @@ struct httrackp {
                              archive. Tail: ABI */
   hts_boolean warc_wacz; /**< --wacz: package archive+index+pages as a WACZ zip
                               (implies --warc + --warc-cdx). Tail: ABI */
-  hts_boolean sitemap;   /**< --sitemap: probe the start host's robots.txt for
-                              Sitemap: lines, else /sitemap.xml. Tail: ABI */
-  String sitemap_url;    /**< --sitemap-url: sitemap to ingest. Tail: ABI */
-  /* Live state, not an option: copy_htsopt must leave it alone. It sits here
-     rather than in htsoptstate because that struct is embedded by value, so
-     growing it would shift every httrackp field declared after it. */
-  void *sitemap_state; /**< hts_sitemap_state*, or NULL. Tail: ABI */
 };
 
 /* Running statistics for a mirror. */

src/htsparse.c

@@ -4583,7 +4583,7 @@ int hts_wait_delayed(htsmoduleStruct * str, lien_adrfilsave *afs,
             /* seen as in error */
             in_error = back[b].r.statuscode;
             in_error_msg[0] = 0;
-            strncatbuff(in_error_msg, back[b].r.msg, sizeof(in_error_msg) - 1);
+            strncat(in_error_msg, back[b].r.msg, sizeof(in_error_msg) - 1);
             in_error_size = back[b].r.totalsize;
             /* don't break, even with "don't take error pages" switch, because we need to process the slot anyway (and cache the error) */
           }

src/htsproxy.c

@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
   if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
     code = 0;
   if (code < 200 || code >= 300) {
-    htsblk_failf(retour, "proxy CONNECT refused: %s",
-                 strnotempty(line) ? line : "(no status)");
+    snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
+             strnotempty(line) ? line : "(no status)");
     return 0;
   }
 

src/htsrobots.c

@@ -147,8 +147,7 @@ static void robots_blob_add(char *blob, size_t blobsize, char marker,
 
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow, char *sitemaps,
-                  size_t sitemapsize) {
+                  hts_boolean keep_root_disallow) {
   size_t bptr = 0;
   int record = 0;
   char BIGSTK line[1024];
@@ -157,8 +156,6 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
   blob[0] = '\0';
   if (info != NULL && infosize > 0)
     info[0] = '\0';
-  if (sitemaps != NULL && sitemapsize > 0)
-    sitemaps[0] = '\0';
 #if DEBUG_ROBOTS
   printf("robots.txt dump:\n%s\n", body);
 #endif
@@ -175,19 +172,7 @@ void robots_parse(robots_wizard *robots, const char *adr, const char *body,
       line[llen - 1] = '\0';
       llen--;
     }
-    if (sitemaps != NULL && strfield(line, "sitemap:")) {
-      // group-independent record (RFC 9309): collected whatever the group
-      char *a = line + 8;
-
-      while (is_realspace(*a))
-        a++;
-      /* A line at the buffer limit was truncated: a half URL is not one. */
-      if (strnotempty(a) && strlen(line) < sizeof(line) - 3 &&
-          strlen(a) + 2 < sitemapsize - strlen(sitemaps)) {
-        strlcatbuff(sitemaps, a, sitemapsize);
-        strlcatbuff(sitemaps, "\n", sitemapsize);
-      }
-    } else if (strfield(line, "user-agent:")) {
+    if (strfield(line, "user-agent:")) {
       char *a = line + 11;
 
       while (is_realspace(*a))

src/htsrobots.h

@@ -56,12 +56,10 @@ int checkrobots(robots_wizard * robots, const char *adr, const char *fil);
 void checkrobots_free(robots_wizard * robots);
 int checkrobots_set(robots_wizard * robots, const char *adr, const char *data);
 /* Parse robots.txt `body` for `adr`, storing the HTTrack group's rules; `info`
-   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /", and
-   `sitemaps` (optional) collects the Sitemap: URLs, one per line. */
+   gets a disallow summary, `keep_root_disallow` FALSE drops "Disallow: /". */
 void robots_parse(robots_wizard *robots, const char *adr, const char *body,
                   size_t bodysize, char *info, size_t infosize,
-                  hts_boolean keep_root_disallow, char *sitemaps,
-                  size_t sitemapsize);
+                  hts_boolean keep_root_disallow);
 #endif
 
 #endif

src/htssafe.h

@@ -33,7 +33,6 @@ Please visit our Website: http://www.httrack.com
 #ifndef HTSSAFE_DEFH
 #define HTSSAFE_DEFH
 
-#include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -117,15 +116,6 @@ static HTS_UNUSED void abortf_(const char *exp, const char *file, int line) {
 #endif
 #define HTS_IS_NOT_CHAR_BUFFER(VAR) (!HTS_IS_CHAR_BUFFER(VAR))
 
-/* Source capacity for the buff() family, (size_t)-1 when unknown; sizeof of the
-   TYPE keeps a decayed operand ("buf + 1") off -Wsizeof-array-decay. */
-#if (defined(__GNUC__) && !defined(__cplusplus))
-#define HTS_SIZEOF_SRC_(B)                                                     \
-  (HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(__typeof__(B)))
-#else
-#define HTS_SIZEOF_SRC_(B) (HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B))
-#endif
-
 /* Compile-time checks. */
 static HTS_UNUSED void htssafe_compile_time_check_(void) {
   char array[32];
@@ -215,17 +205,19 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #if (defined(__GNUC__) && !defined(__cplusplus))
 
 #define strncatbuff(A, B, N)                                                   \
-  __builtin_choose_expr(HTS_IS_CHAR_BUFFER(A),                                 \
-                        strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), N,  \
-                                      "overflow while appending '" #B          \
-                                      "' to '" #A "'",                         \
-                                      __FILE__, __LINE__),                     \
-                        strncatbuff_ptr_((A), (B), (N)))
+  __builtin_choose_expr(                                                       \
+      HTS_IS_CHAR_BUFFER(A),                                                   \
+      strncat_safe_(A, sizeof(A), B,                                           \
+                    HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N,    \
+                    "overflow while appending '" #B "' to '" #A "'", __FILE__, \
+                    __LINE__),                                                 \
+      strncatbuff_ptr_((A), (B), (N)))
 #else
 #define strncatbuff(A, B, N)                                                   \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strncat(A, B, N)                                                      \
-       : strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), N,                 \
+       : strncat_safe_(A, sizeof(A), B,                                        \
+                       HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N, \
                        "overflow while appending '" #B "' to '" #A "'",        \
                        __FILE__, __LINE__))
 #endif
@@ -240,7 +232,9 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #define strcatbuff(A, B)                                                       \
   __builtin_choose_expr(                                                       \
       HTS_IS_CHAR_BUFFER(A),                                                   \
-      strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), (size_t) -1,          \
+      strncat_safe_(A, sizeof(A), B,                                           \
+                    HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),       \
+                    (size_t) -1,                                               \
                     "overflow while appending '" #B "' to '" #A "'", __FILE__, \
                     __LINE__),                                                 \
       strcatbuff_ptr_((A), (B)))
@@ -248,7 +242,9 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #define strcatbuff(A, B)                                                       \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strcat(A, B)                                                          \
-       : strncat_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B), (size_t) -1,       \
+       : strncat_safe_(A, sizeof(A), B,                                        \
+                       HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),    \
+                       (size_t) -1,                                            \
                        "overflow while appending '" #B "' to '" #A "'",        \
                        __FILE__, __LINE__))
 #endif
@@ -261,17 +257,19 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
 #if (defined(__GNUC__) && !defined(__cplusplus))
 
 #define strcpybuff(A, B)                                                       \
-  __builtin_choose_expr(HTS_IS_CHAR_BUFFER(A),                                 \
-                        strcpy_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B),      \
-                                     "overflow while copying '" #B "' to '" #A \
-                                     "'",                                      \
-                                     __FILE__, __LINE__),                      \
-                        strcpybuff_ptr_((A), (B)))
+  __builtin_choose_expr(                                                       \
+      HTS_IS_CHAR_BUFFER(A),                                                   \
+      strcpy_safe_(A, sizeof(A), B,                                            \
+                   HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),        \
+                   "overflow while copying '" #B "' to '" #A "'", __FILE__,    \
+                   __LINE__),                                                  \
+      strcpybuff_ptr_((A), (B)))
 #else
 #define strcpybuff(A, B)                                                       \
   (HTS_IS_NOT_CHAR_BUFFER(A)                                                   \
        ? strcpy(A, B)                                                          \
-       : strcpy_safe_(A, sizeof(A), B, HTS_SIZEOF_SRC_(B),                     \
+       : strcpy_safe_(A, sizeof(A), B,                                         \
+                      HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),     \
                       "overflow while copying '" #B "' to '" #A "'", __FILE__, \
                       __LINE__))
 #endif
@@ -288,24 +286,24 @@ static char *strncatbuff_ptr_(char *dest, const char *src, size_t n) {
  * Append characters of "B" to "A", "A" having a maximum capacity of "S".
  */
 #define strlcatbuff(A, B, S)                                                   \
-  strncat_safe_(A, S, B, HTS_SIZEOF_SRC_(B), (size_t) -1,                      \
-                "overflow while appending '" #B "' to '" #A "'", __FILE__,     \
-                __LINE__)
+  strncat_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),  \
+                (size_t) -1, "overflow while appending '" #B "' to '" #A "'",  \
+                __FILE__, __LINE__)
 
 /**
  * Append at most "N" characters of "B" to "A", "A" having a maximum capacity
  * of "S".
  */
 #define strlncatbuff(A, B, S, N)                                               \
-  strncat_safe_(A, S, B, HTS_SIZEOF_SRC_(B), N,                                \
-                "overflow while appending '" #B "' to '" #A "'", __FILE__,     \
+  strncat_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),  \
+                N, "overflow while appending '" #B "' to '" #A "'", __FILE__,  \
                 __LINE__)
 
 /**
  * Copy characters of "B" to "A", "A" having a maximum capacity of "S".
  */
 #define strlcpybuff(A, B, S)                                                   \
-  strcpy_safe_(A, S, B, HTS_SIZEOF_SRC_(B),                                    \
+  strcpy_safe_(A, S, B, HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B),   \
                "overflow while copying '" #B "' to '" #A "'", __FILE__,        \
                __LINE__)
 
@@ -424,9 +422,7 @@ static HTS_INLINE HTS_UNUSED htsbuff htsbuff_ptr_(char *buf, size_t cap) {
  */
 static HTS_INLINE HTS_UNUSED void htsbuff_catn(htsbuff *b, const char *s,
                                                size_t n) {
-  /* the (size_t)-1 "no limit" sentinel would reach strnlen as a bound past
-     PTRDIFF_MAX */
-  const size_t add = n != (size_t) -1 ? strnlen(s, n) : strlen(s);
+  const size_t add = strnlen(s, n);
   /* Overflow-safe: keep the (potentially huge) 'add' alone on one side. The
      maintained invariant len < cap makes 'cap - len' >= 1 (no underflow), so
      'add < cap - len' cannot wrap the way 'len + add < cap' could. */
@@ -460,49 +456,6 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
   return b->buf;
 }
 
-/**
- * Callers that deliberately ignore truncation use this instead of
- * slprintfbuff(), so it is not HTS_CHECK_RESULT.
- */
-static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
-    vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
-  int ret;
-
-  assertf(dest != NULL && size != 0);
-  ret = vsnprintf(dest, size, fmt, args);
-  /* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
-  dest[size - 1] = '\0';
-  return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
-}
-
-/**
- * Formatted print into dest (capacity size, NUL included), truncating to fit
- * and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
- * result is the only truncation signal, so it must be acted on. Unlike
- * strcpybuff() it never aborts, so it suits text built from remote input.
- */
-static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
-    slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
-  va_list args;
-  hts_boolean ret;
-
-  va_start(args, fmt);
-  ret = vslprintfbuff(dest, size, fmt, args);
-  va_end(args);
-  return ret;
-}
-
-/**
- * slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
- * On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
- */
-#if (defined(__GNUC__) && !defined(__cplusplus))
-#define sprintfbuff(ARR, ...)                                                  \
-  slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
-#else
-#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
-#endif
-
 /* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
    added bounds checking. freet() also NULLs the freed pointer and tolerates
    NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

src/htsselftest.c

@@ -43,7 +43,6 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
-#include "htsback.h"
 #include "htsdefines.h"
 #include "htslib.h"
 #include "htsalias.h"
@@ -52,14 +51,12 @@ Please visit our Website: http://www.httrack.com
 #include "htscache_selftest.h"
 #include "htsdns_selftest.h"
 #include "htscharset.h"
-#include "htscmdline.h"
 #include "htsencoding.h"
 #include "htsftp.h"
 #include "htsmd5.h"
 #include "htssniff.h"
 #include "htscodec.h"
 #include "htsproxy.h"
-#include "htssitemap.h"
 #include "htswarc.h"
 #if HTS_USEZLIB
 #include "htszlib.h"
@@ -458,12 +455,6 @@ static void basic_selftests(void) {
     // link one level up -> a "../" prefix
     assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
     assertf(strcmp(s, "../a.html") == 0);
-    // an empty current path: the trim used to walk off the front of it, which
-    // "?x" reaches too because the query pre-pass hands on the part before it
-    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
-    assertf(strcmp(s, "dir/page.html") == 0);
-    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
-    assertf(strcmp(s, "dir/page.html") == 0);
   }
 }
 
@@ -490,28 +481,6 @@ static int string_safety_selftests(void) {
   if (strcmp(buf, "abcd") != 0)
     return 1;
 
-  /* A decayed source has no known capacity, so the whole tail must land; a
-     sizeof(char*) capacity would abort here instead. */
-  {
-    char src[32] = "0123456789abcdefghij";
-    char dst[32];
-
-    strcpybuff(dst, src + 1);
-    if (strcmp(dst, "123456789abcdefghij") != 0)
-      return 1;
-  }
-
-  /* Truncating append: stops at N without aborting, what the status-message
-     call sites rely on. */
-  {
-    char dst[10]; /* never sizeof(char*), or MSVC reads it as a pointer */
-
-    dst[0] = '\0';
-    strncatbuff(dst, "abcdefghijkl", sizeof(dst) - 1);
-    if (strcmp(dst, "abcdefghi") != 0)
-      return 1;
-  }
-
   /* strlcpybuff: explicit-capacity copy into a pointer destination, the form
      the migration moves toward */
   {
@@ -577,154 +546,6 @@ static int string_safety_selftests(void) {
       return 1;
   }
 
-  /* sprintfbuff: truncate-and-report. Must never abort (its callers format
-     remote banners) nor write past the array, which the canary catches. */
-  {
-    struct {
-      char dst[8];
-      char canary[8];
-    } s;
-
-    const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
-
-    /* repoison before every call, or an implementation that measures first and
-       writes nothing "passes" the truncating cases on the previous content */
-#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
-
-    memset(&s, '#', sizeof(s));
-    if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
-      return 1;
-
-    /* exact fit: 7 characters plus the NUL */
-    POISON_DST();
-    if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
-      return 1;
-
-    /* one over, then far over: truncated to the prefix, terminated, reported */
-    POISON_DST();
-    if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
-      return 1;
-    POISON_DST();
-    if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
-      return 1;
-
-    /* explicit-capacity form, down to the degenerate size 1 */
-    {
-      char *const p = s.dst;
-
-      POISON_DST();
-      if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
-        return 1;
-      POISON_DST();
-      if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
-        return 1;
-    }
-#undef POISON_DST
-
-    if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
-      return 1;
-  }
-
-  /* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
-     touches nothing else in the block */
-  {
-    htsblk r;
-    char expect[sizeof(r.msg)];
-    char big[4 * sizeof(r.msg)];
-
-    /* contenttype abuts msg, so a one-past-the-end store lands in it rather
-       than in padding. Poison it: a stray NUL is invisible against zeroes,
-       and a stray NUL is exactly what an off-by-one terminator writes. */
-#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
-
-    memset(&r, 0, sizeof(r));
-    memset(r.contenttype, '#', sizeof(r.contenttype));
-    r.statuscode = 1234;
-
-    memset(r.msg, '#', sizeof(r.msg));
-    htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
-    if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
-      return 1;
-
-    /* exact fit: capacity - 1 characters plus the NUL */
-    memset(expect, 'y', sizeof(expect) - 1);
-    expect[sizeof(expect) - 1] = '\0';
-    memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
-    memset(r.msg, '#', sizeof(r.msg));
-    htsblk_failf(&r, "%s", expect);
-    if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
-      return 1;
-
-    /* far over: the expected bytes differ from the cases above, so writing
-       nothing cannot pass on the leftovers */
-    memset(big, 'z', sizeof(big) - 1);
-    big[sizeof(big) - 1] = '\0';
-    memset(expect, 'z', sizeof(expect) - 1);
-    expect[sizeof(expect) - 1] = '\0';
-    memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
-
-    memset(r.msg, '#', sizeof(r.msg));
-    htsblk_failf(&r, "Bad user name: %s", big);
-    if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
-      return 1;
-#undef NEIGHBOURS_INTACT
-  }
-
-  /* back_read_ftp_result: the helper's result file is external input, so an
-     over-long message must stop at msg[]'s capacity */
-  {
-    htsblk r;
-    size_t k;
-
-    /* poisoned so a short message cannot pass on leftovers, and so a stray
-       NUL past msg[] is visible in the neighbour */
-#define FTP_RESULT_CASE(BODY)                                                  \
-  do {                                                                         \
-    FILE *fp_ = tmpfile();                                                     \
-                                                                               \
-    if (fp_ == NULL)                                                           \
-      return 1;                                                                \
-    BODY;                                                                      \
-    rewind(fp_);                                                               \
-    memset(&r, 0, sizeof(r));                                                  \
-    memset(r.msg, '#', sizeof(r.msg));                                         \
-    memset(r.contenttype, '#', sizeof(r.contenttype));                         \
-    back_read_ftp_result(fp_, &r);                                             \
-    fclose(fp_);                                                               \
-    if (r.contenttype[0] != '#')                                               \
-      return 1;                                                                \
-  } while (0)
-
-    /* over capacity: clipped to 79 payload bytes plus the NUL */
-    FTP_RESULT_CASE({
-      fprintf(fp_, "226 ");
-      for (k = 0; k < 4 * sizeof(r.msg); k++)
-        fputc('q', fp_);
-    });
-    if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
-      return 1;
-    for (k = 0; k < sizeof(r.msg) - 1; k++) {
-      if (r.msg[k] != 'q')
-        return 1;
-    }
-
-    /* well under capacity: nothing padded, nothing eaten off the end */
-    FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
-    if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
-      return 1;
-
-    /* a byte over 0x7f must not read as EOF and cut the message short */
-    FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
-    if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
-      return 1;
-
-    /* unparseable status: the message still loads, the code reports failure */
-    FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
-    if (r.statuscode != STATUSCODE_INVALID)
-      return 1;
-#undef FTP_RESULT_CASE
-  }
-
   /* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
      n_eval == 2 on a double-eval macro. */
   {
@@ -1283,128 +1104,6 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
-// hts_split_cmdline(): the vector must grow with the argument count, and a
-// quote inside a value must not end the argument and hand -V to the parser.
-static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
-  char line[512];
-  char **args;
-  int nargs = 0;
-
-  (void) opt;
-  (void) argc;
-  (void) argv;
-
-  // control: every separator splits, and argv[0] is the program name
-  strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(args != NULL && nargs == 6);
-  assertf(args[nargs] == NULL); // callers may walk to the terminator
-  assertf(strcmp(args[0], "httrack") == 0);
-  assertf(strcmp(args[1], "http://x/") == 0);
-  assertf(strcmp(args[2], "--quiet") == 0);
-  assertf(strcmp(args[3], "-c8") == 0);
-  assertf(strcmp(args[4], "-O") == 0);
-  assertf(strcmp(args[5], "out") == 0);
-  freet(args);
-
-  // the template pads with whitespace: empty arguments are kept (the engine
-  // skips them), so the count is one per separator
-  strcpybuff(line, "httrack  --quiet");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 3 && args[1][0] == '\0');
-  assertf(strcmp(args[2], "--quiet") == 0);
-  freet(args);
-
-  // a quoted run keeps both its spaces and its quotes: the engine unquotes
-  strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 4);
-  assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
-  assertf(strcmp(args[3], "-c8") == 0);
-  freet(args);
-
-  // an escaped quote is a literal quote, not the end of the argument: the
-  // engine strips only the outer pair
-  strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 4);
-  assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
-  assertf(strcmp(args[3], "-c8") == 0);
-  freet(args);
-
-  // \\ is a literal backslash, so a Windows path survives
-  strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 3);
-  assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
-  freet(args);
-
-  // outside a quoted run a backslash is literal: the url and wildcard-filter
-  // fields, which the wizard cannot quote, read as before
-  strcpybuff(line, "httrack -*\\** +*.png");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 3);
-  assertf(strcmp(args[1], "-*\\**") == 0);
-  assertf(strcmp(args[2], "+*.png") == 0);
-  freet(args);
-
-  // a quoted run leaves slots unused, so the terminator has to be written and
-  // not inherited: size the vector from a full line first, so freeing it hands
-  // the same chunk back with stale pointers in those slots
-  strcpybuff(line, "httrack a b c d e");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 6);
-  freet(args);
-  strcpybuff(line, "httrack \"a b c d e\"");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 2);
-  assertf(args[nargs] == NULL);
-  freet(args);
-
-  // an unterminated quote protects the rest of the line, as one argument
-  strcpybuff(line, "httrack --footer \"unbalanced -V x");
-  args = hts_split_cmdline(line, &nargs);
-  assertf(nargs == 3);
-  assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
-  freet(args);
-
-  // past the 1024 entries the vector used to hold: distinct arguments, so a
-  // write beyond the allocation cannot read back as the expected parse
-  {
-    const int n = 2000;
-    const size_t size = 16 * (size_t) n + 16;
-    char *big = malloct(size);
-    size_t pos = 0;
-    int i;
-
-    assertf(big != NULL);
-    pos = (size_t) snprintf(big, size, "httrack");
-    assertf(pos < size);
-    for (i = 0; i < n; i++) {
-      // snprintf returns what it wanted to write, so accumulating it blind
-      // would let the next size argument wrap
-      const int len = snprintf(big + pos, size - pos, " a%d", i);
-
-      assertf(len > 0 && (size_t) len < size - pos);
-      pos += (size_t) len;
-    }
-    args = hts_split_cmdline(big, &nargs);
-    assertf(args != NULL && nargs == n + 1);
-    assertf(args[nargs] == NULL);
-    for (i = 0; i < n; i++) {
-      char expect[16];
-
-      snprintf(expect, sizeof(expect), "a%d", i);
-      assertf(strcmp(args[i + 1], expect) == 0);
-    }
-    freet(args);
-    freet(big);
-  }
-
-  printf("cmdline-split self-test OK\n");
-  return 0;
-}
-
 static int st_hashtable(httrackp *opt, int argc, char **argv) {
   char *snum;
   unsigned long count = 0;
@@ -1526,7 +1225,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
       size_t i;
       for (i = bench[loop].offset; i < (size_t) count;
            i += bench[loop].modulus) {
-        int result = 0; /* no final else: an unknown type reports failure */
+        int result;
         FMT();
         if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
           size_t k;
@@ -1600,14 +1299,6 @@ static int st_strsafe(httrackp *opt, int argc, char **argv) {
       htsbuff b = htsbuff_array(small);
 
       htsbuff_cat(&b, src);
-    } else if (strcmp(argv[0], "overflow-src") == 0) {
-      /* Array source with no NUL: its capacity still comes from sizeof(), so
-         the bounded strlen aborts rather than running off the array. */
-      char nonul[6]; /* never sizeof(char*), per the note above */
-      char big[64];
-
-      memset(nonul, src[0], sizeof(nonul));
-      strcpybuff(big, nonul);
     } else {
       strcpybuff(small, src);
     }
@@ -1681,22 +1372,6 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
   if (strcmp(StringBuff(to->warc_file), "run.warc.gz") != 0)
     err = 1;
 
-  /* sitemap pair: the flag latches on, the URL takes the String deep copy */
-  from->sitemap = HTS_TRUE;
-  StringCopy(from->sitemap_url, "http://h.test/sitemap.xml");
-  to->sitemap = HTS_FALSE;
-  StringCopy(to->sitemap_url, "");
-  copy_htsopt(from, to);
-  if (!to->sitemap ||
-      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
-    err = 1;
-  from->sitemap = HTS_FALSE;
-  StringCopy(from->sitemap_url, "");
-  copy_htsopt(from, to);
-  if (!to->sitemap ||
-      strcmp(StringBuff(to->sitemap_url), "http://h.test/sitemap.xml") != 0)
-    err = 1;
-
   /* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
   from->pause_min_ms = 5000;
   from->pause_max_ms = 10000;
@@ -2318,77 +1993,6 @@ static int st_fsize(httrackp *opt, int argc, char **argv) {
   return rc;
 }
 
-/* 4GB+100KB wraps to ~108KB through an int, and needs 33 unsigned bits. A
-   macro, not a static const: MSVC's C mode (/TC) rejects a const object
-   used inside another object's static initializer below (C2099). */
-#define HTS_ST_GROWSIZE_OVER32 (4LL * 1024 * 1024 * 1024 + 100 * 1024)
-
-/* llint_grow_size_t() sizes the buffer holding a whole -%S list file: the
-   result must be the exact 64-bit sum or a clean refusal, never a short one. */
-static int st_growsize(httrackp *opt, int argc, char **argv) {
-  enum { REFUSE, ACCEPT, WIDTH };
-
-  static const struct {
-    size_t used;
-    LLint extra;
-    size_t slack;
-    int want;
-  } cases[] = {
-      {0, 0, 0, ACCEPT},
-      {10, 100, 8192, ACCEPT},
-      {(size_t) -2 - 8, 4, 4, ACCEPT}, /* exact fit, no room to spare */
-      {(size_t) -2, 0, 0, ACCEPT},     /* largest representable capacity */
-      {0, -1, 0, REFUSE},              /* fsize() failure */
-      /* -1 already maps to SIZE_MAX; only this exercises the negative guard */
-      {0, -4096, 0, REFUSE},
-      {(size_t) -1, 1, 0, REFUSE},
-      {(size_t) -2, 0, 1, REFUSE},     /* slack alone overruns */
-      {(size_t) -1 - 8, 4, 4, REFUSE}, /* total would be the error value */
-      {(size_t) -1 - 8, 4, 8, REFUSE},
-      {0, HTS_ST_GROWSIZE_OVER32, 8192,
-       WIDTH}, /* 32-bit size_t can't hold these */
-      {10, HTS_ST_GROWSIZE_OVER32, 8192, WIDTH},
-  };
-
-  size_t k;
-  int rc = 0;
-
-  (void) opt;
-  (void) argc;
-  (void) argv;
-  for (k = 0; k < sizeof(cases) / sizeof(cases[0]); k++) {
-    const size_t used = cases[k].used, slack = cases[k].slack;
-    const LLint extra = cases[k].extra;
-    const size_t got = llint_grow_size_t(used, extra, slack);
-    const hts_boolean refused = got == (size_t) -1 ? HTS_TRUE : HTS_FALSE;
-    const hts_boolean exact =
-        !refused && extra >= 0 && got - used - slack == (size_t) extra;
-    hts_boolean ok;
-
-    switch (cases[k].want) {
-    case ACCEPT:
-      ok = exact;
-      break;
-    case REFUSE:
-      ok = refused;
-      break;
-    default:
-      ok = sizeof(size_t) >= sizeof(LLint) ? exact : refused;
-      break;
-    }
-    if (!ok) {
-      fprintf(stderr,
-              "growsize: grow(" LLintP ", " LLintP ", " LLintP ") = " LLintP
-              " (want %s)\n",
-              (LLint) used, extra, (LLint) slack, (LLint) got,
-              cases[k].want == REFUSE ? "refusal" : "exact sum");
-      rc = 1;
-    }
-  }
-  printf("growsize self-test %s\n", rc == 0 ? "OK" : "FAILED");
-  return rc;
-}
-
 static int st_savename(httrackp *opt, int argc, char **argv) {
   lien_adrfilsave afs;
   cache_back cache;
@@ -2783,20 +2387,17 @@ static int st_cookies(httrackp *opt, int argc, char **argv) {
     static t_cookie ck2;
     htsblk r;
     char host[600];
-    char line[64]; /* treathead NUL-cuts the header in place: never a literal */
 
     memset(&r, 0, sizeof(r));
     memset(host, 'a', sizeof(host) - 1);
     host[sizeof(host) - 1] = '\0';
     ck2.max_len = (int) sizeof(ck2.data);
     ck2.data[0] = '\0';
-    strcpybuff(line, "Set-Cookie: SID=1; path=/");
-    treathead(&ck2, host, "/", &r, line);
+    treathead(&ck2, host, "/", &r, "Set-Cookie: SID=1; path=/");
     if (strnotempty(ck2.data)) // oversize-host cookie was not dropped
       err = 1;
     /* control: a normal host still yields a cookie through treathead */
-    strcpybuff(line, "Set-Cookie: SID=1; path=/");
-    treathead(&ck2, dom, "/", &r, line);
+    treathead(&ck2, dom, "/", &r, "Set-Cookie: SID=1; path=/");
     if (strstr(ck2.data, "SID") == NULL) // guard wrongly dropped a valid cookie
       err = 1;
   }
@@ -3057,32 +2658,6 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
   assertf(strstr(buf, "Refresh") != NULL);
   assertf(strstr(buf, "example.com") != NULL);
 
-  /* a first link whose escaped form overruns the old flat 1024-byte tempo: the
-     redirect must carry the whole URL, not a clipped prefix */
-  {
-    char BIGSTK link[HTS_URLMAXSIZE * 2];
-    char *p = link;
-
-    strcpybuff(link, "http://example.com/");
-    p += strlen(link);
-    memset(p, 'a', 1200);
-    p += 1200;
-    strcpy(p, "/end.html");
-
-    done = 0;
-    fp = fopen(path, "wb");
-    assertf(fp != NULL);
-    hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
-    assertf(fp == NULL);
-    fp = fopen(path, "rb");
-    assertf(fp != NULL);
-    n = fread(buf, 1, sizeof(buf) - 1, fp);
-    fclose(fp);
-    buf[n] = '\0';
-    /* the closing quote proves the URL was not clipped mid-way */
-    assertf(strstr(buf, "/end.html\">") != NULL);
-  }
-
   /* no single link: footer only, no refresh meta */
   done = 0;
   fp = fopen(path, "wb");
@@ -3312,7 +2887,7 @@ static int ae_write_packed(const char *path, int windowBits,
     deflateEnd(&strm);
     return 1;
   }
-  strm.next_in = (const Bytef *) src;
+  strm.next_in = (Bytef *) src;
   strm.avail_in = (uInt) len;
   do {
     size_t n;
@@ -3621,7 +3196,7 @@ static int rb_decide(robots_wizard *r, const char *txt, const char *path) {
   char host[64];
 
   snprintf(host, sizeof(host), "h%d.example", n++);
-  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE, NULL, 0);
+  robots_parse(r, host, txt, strlen(txt), NULL, 0, HTS_TRUE);
   return checkrobots(r, host, path);
 }
 
@@ -3698,267 +3273,6 @@ static int st_robots(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
-/* Collect the URLs a sitemap scan hands out. */
-typedef struct sm_collect {
-  int n;
-  char url[8][HTS_URLMAXSIZE];
-} sm_collect;
-
-static hts_boolean sm_take(void *arg, const char *url) {
-  sm_collect *const c = (sm_collect *) arg;
-
-  if (c->n < (int) (sizeof(c->url) / sizeof(c->url[0])))
-    strcpybuff(c->url[c->n], url);
-  c->n++;
-  return HTS_TRUE;
-}
-
-/* Scan `doc` off a heap buffer with no NUL terminator, so a read past the
-   declared size is an ASan error rather than a silent pass. */
-static int sm_scan(const char *doc, int maxurls, hts_boolean *is_index,
-                   sm_collect *out) {
-  const size_t len = strlen(doc);
-  char *raw = malloct(len);
-  int n;
-
-  memset(out, 0, sizeof(*out));
-  assertf(raw != NULL);
-  memcpy(raw, doc, len);
-  n = hts_sitemap_scan(raw, len, maxurls, is_index, sm_take, out);
-  freet(raw);
-  return n;
-}
-
-static int st_sitemap(httrackp *opt, int argc, char **argv) {
-  sm_collect c;
-  hts_boolean idx;
-  (void) opt;
-  (void) argc;
-  (void) argv;
-
-  /* A urlset yields its <loc> URLs, in order, unescaped. */
-  assertf(sm_scan("<?xml version=\"1.0\"?><urlset>"
-                  "<url><loc>http://h.test/a.html</loc></url>"
-                  "<url><loc>  https://h.test/b?x=1&amp;y=2\n  </loc></url>"
-                  "</urlset>",
-                  100, &idx, &c) == 2);
-  assertf(!idx);
-  assertf(strcmp(c.url[0], "http://h.test/a.html") == 0);
-  assertf(strcmp(c.url[1], "https://h.test/b?x=1&y=2") == 0);
-
-  /* A sitemapindex is flagged: its URLs are child sitemaps, not pages. */
-  assertf(sm_scan("<sitemapindex><sitemap><loc>http://h.test/s2.xml.gz</loc>"
-                  "</sitemap></sitemapindex>",
-                  100, &idx, &c) == 1);
-  assertf(idx);
-
-  /* Root element decides even when the other name appears later as text. */
-  assertf(sm_scan("<urlset><url><loc>http://h.test/a</loc></url>"
-                  "<!-- sitemapindex --></urlset>",
-                  100, &idx, &c) == 1);
-  assertf(!idx);
-
-  /* Numeric character references, decimal and hex, decode to ASCII. */
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#63;b&#x3D;c</loc></urlset>",
-                  100, &idx, &c) == 1);
-  assertf(strcmp(c.url[0], "http://h.test/a?b=c") == 0);
-
-  /* A reference decoding to a control byte is dropped: the shared decoder
-     writes the real character and the URL check refuses it. A reference the
-     decoder cannot represent (&#0;) stays verbatim, like an unknown entity. */
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#10;b</loc></urlset>", 100,
-                  &idx, &c) == 0);
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#9;b</loc></urlset>", 100, &idx,
-                  &c) == 0);
-  assertf(sm_scan("<urlset><loc>http://h.test/a&#0;b</loc></urlset>", 100, &idx,
-                  &c) == 1);
-  assertf(strcmp(c.url[0], "http://h.test/a&#0;b") == 0);
-
-  /* A comment naming the other root element must not flip the verdict. */
-  assertf(sm_scan("<!-- <sitemapindex> --><urlset><url>"
-                  "<loc>http://h.test/p</loc></url></urlset>",
-                  100, &idx, &c) == 1);
-  assertf(!idx);
-  assertf(sm_scan("<?xml version=\"1.0\"?><!-- <urlset> -->"
-                  "<sitemapindex><loc>http://h.test/s</loc></sitemapindex>",
-                  100, &idx, &c) == 1);
-  assertf(idx);
-
-  /* <location> is not <loc>. */
-  assertf(sm_scan("<urlset><location>http://h.test/a</location></urlset>", 100,
-                  &idx, &c) == 0);
-
-  /* Rejected: relative, non-http scheme, embedded space, empty. */
-  assertf(sm_scan("<urlset><loc>/a.html</loc><loc>ftp://h.test/a</loc>"
-                  "<loc>javascript:alert(1)</loc>"
-                  "<loc>http://h.test/a b</loc><loc></loc></urlset>",
-                  100, &idx, &c) == 0);
-
-  /* The URL length bound: one under fits, exactly at it is dropped rather than
-     truncated into a different URL. */
-  {
-    char BIGSTK doc[HTS_URLMAXSIZE * 2];
-    char BIGSTK url[HTS_URLMAXSIZE + 1];
-    size_t i;
-
-    strcpybuff(url, "http://h.test/");
-    for (i = strlen(url); i < HTS_URLMAXSIZE - 1; i++)
-      url[i] = 'a';
-    url[i] = '\0';
-    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
-    assertf(sm_scan(doc, 100, &idx, &c) == 1);
-
-    url[i] = 'a';
-    url[i + 1] = '\0';
-    snprintf(doc, sizeof(doc), "<urlset><loc>%s</loc></urlset>", url);
-    assertf(sm_scan(doc, 100, &idx, &c) == 0);
-  }
-
-  /* The URL cap stops the scan. */
-  assertf(sm_scan("<urlset><loc>http://h.test/1</loc><loc>http://h.test/2</loc>"
-                  "<loc>http://h.test/3</loc></urlset>",
-                  2, &idx, &c) == 2);
-
-  /* The per-document cap at the value the engine actually uses. */
-  {
-    const int many = HTS_SITEMAP_MAX_URLS_DOC + 10;
-    const size_t cap = (size_t) many * 40 + 32;
-    char *big = malloct(cap);
-    size_t off;
-    int i;
-
-    assertf(big != NULL);
-    off = (size_t) snprintf(big, cap, "<urlset>");
-    assertf(off < cap);
-    for (i = 0; i < many; i++) {
-      const int len =
-          snprintf(big + off, cap - off, "<loc>http://h.test/%d</loc>", i);
-
-      assertf(len > 0 && (size_t) len < cap - off);
-      off += (size_t) len;
-    }
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(big, off, HTS_SITEMAP_MAX_URLS_DOC, &idx, sm_take,
-                             &c) == HTS_SITEMAP_MAX_URLS_DOC);
-    /* The handler count, not just the return: a call site hardcoding a smaller
-       cap would still return its own argument. */
-    assertf(c.n == HTS_SITEMAP_MAX_URLS_DOC);
-    freet(big);
-  }
-
-#if HTS_USEZLIB
-  /* A highly compressible document decodes without running away: the ratio
-     budget cannot bind (deflate tops out near 1032:1), so this pins the
-     decompression path itself rather than the 64 MiB ceiling. */
-  {
-    const char *const one = "<url><loc>http://h.test/bomb</loc></url>";
-    const size_t reps = 40000;
-    size_t xlen = 8 + reps * strlen(one) + 10, i;
-    char *x = malloct(xlen + 1);
-    uLongf zlen;
-    char *z;
-    z_stream zs;
-
-    assertf(x != NULL);
-    {
-      size_t w = (size_t) snprintf(x, xlen, "<urlset>");
-      int len;
-
-      assertf(w < xlen);
-      for (i = 0; i < reps; i++) {
-        len = snprintf(x + w, xlen - w, "%s", one);
-        assertf(len > 0 && (size_t) len < xlen - w);
-        w += (size_t) len;
-      }
-      len = snprintf(x + w, xlen - w, "</urlset>");
-      assertf(len > 0 && (size_t) len < xlen - w);
-      w += (size_t) len;
-      xlen = w;
-    }
-    zlen = compressBound((uLong) xlen) + 32;
-    z = malloct((size_t) zlen);
-    assertf(z != NULL);
-    memset(&zs, 0, sizeof(zs));
-    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
-                         Z_DEFAULT_STRATEGY) == Z_OK);
-    zs.next_in = (const Bytef *) x;
-    zs.avail_in = (uInt) xlen;
-    zs.next_out = (Bytef *) z;
-    zs.avail_out = (uInt) zlen;
-    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
-    zlen = (uLongf) zs.total_out;
-    deflateEnd(&zs);
-    /* well over the 4096:1 budget's 1 MiB floor, and far under the 64 MiB cap
-     */
-    assertf(xlen > 1024 * 1024 && (size_t) zlen < xlen / 100);
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(z, (size_t) zlen, 10, &idx, sm_take, &c) == 10);
-    assertf(strcmp(c.url[0], "http://h.test/bomb") == 0);
-    freet(z);
-    freet(x);
-  }
-#endif
-
-  /* An unterminated <loc> at end of buffer must not read past it. */
-  assertf(sm_scan("<urlset><loc>http://h.test/a", 100, &idx, &c) == 0);
-  assertf(sm_scan("<urlset><lo", 100, &idx, &c) == 0);
-
-#if HTS_USEZLIB
-  /* A gzip-framed document is decompressed before scanning. */
-  {
-    const char *const xml =
-        "<urlset><url><loc>http://h.test/gz.html</loc></url></urlset>";
-    uLongf zlen = compressBound((uLong) strlen(xml)) + 32;
-    char *z = malloct((size_t) zlen);
-    z_stream zs;
-
-    assertf(z != NULL);
-    memset(&zs, 0, sizeof(zs));
-    assertf(deflateInit2(&zs, 9, Z_DEFLATED, 16 + MAX_WBITS, 8,
-                         Z_DEFAULT_STRATEGY) == Z_OK);
-    zs.next_in = (const Bytef *) xml;
-    zs.avail_in = (uInt) strlen(xml);
-    zs.next_out = (Bytef *) z;
-    zs.avail_out = (uInt) zlen;
-    assertf(deflate(&zs, Z_FINISH) == Z_STREAM_END);
-    zlen = (uLongf) zs.total_out;
-    deflateEnd(&zs);
-
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(z, (size_t) zlen, 100, &idx, sm_take, &c) == 1);
-    assertf(strcmp(c.url[0], "http://h.test/gz.html") == 0);
-
-    /* Truncated gzip: refused, not scanned as plain text. */
-    memset(&c, 0, sizeof(c));
-    assertf(hts_sitemap_scan(z, 4, 100, &idx, sm_take, &c) == -1);
-    freet(z);
-  }
-#endif
-
-  /* robots.txt: only Sitemap: records, comments stripped, case-insensitive,
-     and group-independent (no User-agent line needed). */
-  /* robots_parse collects Sitemap: whatever the user-agent group, strips the
-     comment and keeps the rules working alongside it. */
-  {
-    const char *const txt = "User-agent: *\nDisallow: /x\n"
-                            "SITEMAP:  http://h.test/s1.xml  # first\n"
-                            "Sitemapper: http://h.test/no.xml\n"
-                            "Sitemap:\thttps://h.test/s2.xml\n";
-    char BIGSTK maps[1024];
-    robots_wizard rb;
-
-    memset(&rb, 0, sizeof(rb));
-    robots_parse(&rb, "h.test", txt, strlen(txt), NULL, 0, HTS_TRUE, maps,
-                 sizeof(maps));
-    assertf(strcmp(maps, "http://h.test/s1.xml\nhttps://h.test/s2.xml\n") == 0);
-    assertf(checkrobots(&rb, "h.test", "/x") == -1);
-    checkrobots_free(&rb);
-  }
-
-  printf("sitemap self-test OK\n");
-  return 0;
-}
-
 /* Connected stream pair over loopback; Windows has no socketpair(). */
 static int st_socketpair(T_SOC sv[2]) {
   struct sockaddr_in sa;
@@ -4056,6 +3370,21 @@ static int st_ftpuser(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* Bounded substring search (records carry NUL bytes; strstr won't do). */
+static const char *warc_memstr(const char *hay, const char *needle,
+                               size_t haylen, size_t nlen) {
+  if (nlen == 0 || haylen < nlen)
+    return NULL;
+  {
+    size_t i;
+    for (i = 0; i + nlen <= haylen; i++) {
+      if (memcmp(hay + i, needle, nlen) == 0)
+        return hay + i;
+    }
+  }
+  return NULL;
+}
+
 /* Slurp a whole file into a malloc'd buffer; sets *len. NULL on error. */
 static unsigned char *warc_slurp(const char *path, size_t *len) {
   FILE *f = FOPEN(path, "rb");
@@ -4133,13 +3462,6 @@ static unsigned char *warc_next_member(const unsigned char **in,
    Content-Length == block length, the \r\n\r\n trailer intact, the response
    body round-trips, and the hop-by-hop Transfer-Encoding is dropped (a real
    Content-Encoding is kept verbatim; see warc-verbatim). */
-/* Argument order kept for the existing call sites; the search itself is the
-   shared hts_memstr. */
-static const char *warc_memstr(const char *hay, const char *needle,
-                               size_t haylen, size_t nlen) {
-  return hts_memstr(hay, haylen, needle, nlen);
-}
-
 static int st_warc(httrackp *opt, int argc, char **argv) {
   char path[HTS_URLMAXSIZE];
   warc_writer *w;
@@ -5375,10 +4697,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   char fpath[HTS_URLMAXSIZE * 2];
   char file[HTS_URLMAXSIZE * 2];
 
-  assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
+  snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
 
   /* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
-  assertf(sprintfbuff(file, "%scookies.txt", fpath));
+  snprintf(file, sizeof(file), "%scookies.txt", fpath);
   {
     FILE *fp = FOPEN(file, "wb");
 
@@ -5388,7 +4710,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
   }
 
   /* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
-  assertf(sprintfbuff(file, "%su@v.txt", fpath));
+  snprintf(file, sizeof(file), "%su@v.txt", fpath);
   {
     FILE *fp = FOPEN(file, "wb");
 
@@ -5410,7 +4732,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
 #endif
 
   (void) UNLINK(file); /* u@v.txt (already gone on Windows) */
-  assertf(sprintfbuff(file, "%scookies.txt", fpath));
+  snprintf(file, sizeof(file), "%scookies.txt", fpath);
   (void) UNLINK(file);
   dir[dirlen] = '\0';
   while (strlen(dir) > base) {
@@ -5484,12 +4806,9 @@ static const struct selftest_entry {
      st_footerfmt},
     {"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
      st_unescape_bounds},
-    {"cmdline-split", "",
-     "webhttrack command-line to argv split (bounds, quoting)",
-     st_cmdlinesplit},
     {"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
-    {"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
-     "bounded string-op self-test", st_strsafe},
+    {"strsafe", "[overflow|overflow-buff [str]]", "bounded string-op self-test",
+     st_strsafe},
     {"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt},
     {"pause", "", "randomized inter-file pause target self-test", st_pause},
     {"relative", "<link> <curr-file>", "relative link between two paths",
@@ -5519,8 +4838,6 @@ static const struct selftest_entry {
     {"sniff", "<content-type> <hex:..|text>", "MIME magic consistency",
      st_sniff},
     {"fsize", "<dir>", "file size past the 2GB signed-32-bit wrap", st_fsize},
-    {"growsize", "", "buffer capacity for a 64-bit file size (no int wrap)",
-     st_growsize},
     {"cache", "<dir>", "cache read/write round-trip self-test", st_cache},
     {"cacheindex", "", "cache-index (.ndx) parse must stay in bounds",
      st_cacheindex},
@@ -5559,8 +4876,6 @@ static const struct selftest_entry {
      st_contentcodings},
     {"robots", "", "robots.txt RFC 9309 Allow/Disallow precedence self-test",
      st_robots},
-    {"sitemap", "",
-     "sitemap <loc> extraction, caps and robots.txt Sitemap:", st_sitemap},
     {"ftp-line", "", "get_ftp_line bounds a hostile FTP reply line",
      st_ftpline},
     {"ftp-userpass", "", "ftp_split_userpass bounds URL userinfo", st_ftpuser},

src/htsserver.c

@@ -363,59 +363,6 @@ static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
   return seen;
 }
 
-/** Append src to the NUL-terminated dst of capacity size (NUL included).
-    False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
-    never aborts, because every piece appended here is client-supplied. */
-static hts_boolean path_append(char *dst, size_t size, const char *src) {
-  const size_t used = strlen(dst);
-  const size_t len = strlen(src);
-
-  /* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
-     len stays alone: "used + len < size" could wrap and pass. */
-  if (len >= size - used) {
-    return HTS_FALSE;
-  }
-  memcpy(dst + used, src, len + 1);
-  return HTS_TRUE;
-}
-
-/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
-static hts_boolean cat_html_escaped(String *dst, char c) {
-  switch (c) {
-  case '<':
-    StringCat(*dst, "&lt;");
-    break;
-  case '>':
-    StringCat(*dst, "&gt;");
-    break;
-  case '&':
-    StringCat(*dst, "&amp;");
-    break;
-  case '\'':
-    StringCat(*dst, "&#39;");
-    break;
-  default:
-    return HTS_FALSE;
-  }
-  return HTS_TRUE;
-}
-
-/* Append the value of a double-quoted command-line argument: escaped for HTML,
-   which the browser undoes when it posts the command line back, and for the
-   argv splitter, which does not. */
-static void cat_cmdline_arg(String *output, const char *value) {
-  const char *a;
-
-  for (a = value; *a != '\0'; a++) {
-    if (*a == '\\' || *a == '\"') {
-      StringCat(*output, "\\");
-    }
-    if (!cat_html_escaped(output, *a)) {
-      StringMemcat(*output, a, 1);
-    }
-  }
-}
-
 int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   int timeout = 30;
   int retour = 0;
@@ -427,9 +374,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   String tmpbuff = STRING_EMPTY;
   String tmpbuff2 = STRING_EMPTY;
   String fspath = STRING_EMPTY;
-  /* Project directory this server set up; the only root /website/ serves from,
-     and deliberately not cleared between requests. */
-  String website = STRING_EMPTY;
   char catbuff[CATBUFF_SIZE];
 
   /* Load strings */
@@ -841,11 +785,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                   if (!structcheck(StringBuff(tmpbuff))) {
                     FILE *fp;
 
-                    /* Both halves of fspath come from posted fields, so a ".."
-                       in them would escape the mirror once served. */
-                    if (strstr(StringBuff(fspath), "..") == NULL) {
-                      StringCopy(website, StringBuff(fspath));
-                    }
                     StringCat(tmpbuff, "winprofile.ini");
                     fp = fopen(StringBuff(tmpbuff), "wb");
                     if (fp != NULL) {
@@ -918,7 +857,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
 
       /* Response */
       if (meth) {
-        hts_boolean virtualpath = HTS_FALSE;
+        int virtualpath = 0;
         char *pos;
         char *url = strchr(line1, ' ');
 
@@ -929,11 +868,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           char *qpos;
 
           /* get the URL */
-          fsfile[0] = '\0';
           if (error_redirect == NULL) {
             if ((qpos = strchr(url, '?'))) {
               *qpos = '\0';
             }
+            fsfile[0] = '\0';
             if (strcmp(url, "/") == 0) {
               file = "/server/index.html";
               meth = 2;
@@ -946,7 +885,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           }
 
           if (strncmp(file, "/website/", 9) == 0) {
-            virtualpath = HTS_TRUE;
+            virtualpath = 1;
           }
 
           /* override */
@@ -960,26 +899,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             }
           }
 
-          /* the override above may have swapped a mirror path for a GUI page */
-          virtualpath = strncmp(file, "/website/", 9) == 0;
+          if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
+            if (strncmp(file, "/website/", 9) != 0) {
+              sprintf(fsfile, "%shtml%s", path, file);
+            } else {
+              intptr_t adr = 0;
 
-          if (!virtualpath) {
-            if (!path_append(fsfile, sizeof(fsfile), path) ||
-                !path_append(fsfile, sizeof(fsfile), "html") ||
-                !path_append(fsfile, sizeof(fsfile), file)) {
-              fsfile[0] = '\0';
-            }
-          } else if (StringNotEmpty(website)) {
-            /* Never the posted "projpath": a client root reads any file. */
-            if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
-                !path_append(fsfile, sizeof(fsfile), "/") ||
-                !path_append(fsfile, sizeof(fsfile), file + 9)) {
-              fsfile[0] = '\0';
+              if (coucal_readptr(NewLangList, "projpath", &adr)) {
+                sprintf(fsfile, "%s%s", (char *) adr, file + 9);
+              }
             }
           }
 
-          /* path itself may hold ".." (webhttrack passes "<bin>/../share"), so
-             only the untrusted halves are checked: file here, website above. */
           if (fsfile[0] && strstr(file, "..") == NULL
               && (fp = fopen(fsfile, "rb"))) {
             char ok[] =
@@ -1034,9 +965,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                 StringCat(headers, "\r\n");
               }
               coucal_write(NewLangList, "redirect", (intptr_t) NULL);
-            } else if (!virtualpath && is_html(file)) {
-              /* GUI templates only: ${_sid} in a mirrored page would hand the
-                 crawled site the session id that authenticates commands */
+            } else if (is_html(file)) {
               int outputmode = 0;
 
               StringMemcat(headers, ok, sizeof(ok) - 1);
@@ -1064,7 +993,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                     int p;
                     int format = 0;
                     int listDefault = 0;
-                    hts_boolean unquoted = HTS_FALSE;
 
                     name[0] = '\0';
                     strlncatbuff(name, str, sizeof(name_), n);
@@ -1074,12 +1002,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                     } else if ((p = strfield(name, "html:"))) {
                       name += p;
                       format = 1;
-                    } else if ((p = strfield(name, "unquoted:"))) {
-                      name += p;
-                      unquoted = HTS_TRUE;
-                    } else if ((p = strfield(name, "arg:"))) {
-                      name += p;
-                      format = 5;
                     } else if ((p = strfield(name, "list:"))) {
                       name += p;
                       format = 2;
@@ -1216,8 +1138,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                        test:<if ==0>:<if ==1>:<if == 2>..
                        ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
                      */
-                    else if ((p = strfield(name, "test:")) ||
-                             (p = strfield(name, "ztest:"))) {
+                    else if ((p = strfield(name, "test:"))
+                             || (p = strfield(name, "ztest:"))) {
                       intptr_t adr = 0;
                       char *pos2;
                       int ztest = (name[0] == 'z');
@@ -1320,12 +1242,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                         }
                       }
                     }
-                    /* consumed here: it shares nothing with the list and
-                       option formats below */
-                    if (format == 5 && langstr != NULL && outputmode != -1) {
-                      cat_cmdline_arg(&output, langstr);
-                      langstr = NULL;
-                    }
                     if (langstr && outputmode != -1) {
                       switch (format) {
                       case 0:
@@ -1343,18 +1259,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                                 StringMemcat(output, &c, 1);
                               }
                               a += 2;
-                            } else if (unquoted && a[0] == '\"') {
-                              /* the browser posts an entity back as a raw
-                                 quote, which would open a quoted run in the
-                                 argv splitter; a URI cannot hold one anyway */
-                              StringCat(output, "%22");
-                            } else if (outputmode &&
-                                       cat_html_escaped(&output, a[0])) {
-                              /* appended as an entity */
+                            } else if (outputmode && a[0] == '<') {
+                              StringCat(output, "&lt;");
+                            } else if (outputmode && a[0] == '>') {
+                              StringCat(output, "&gt;");
+                            } else if (outputmode && a[0] == '&') {
+                              StringCat(output, "&amp;");
+                            } else if (outputmode && a[0] == '\'') {
+                              StringCat(output, "&#39;");
                             } else if (outputmode == 3 && a[0] == ' ') {
                               StringCat(output, "%20");
-                            } else if (outputmode >= 2 &&
-                                       ((unsigned char) a[0]) < 32) {
+                            } else if (outputmode >= 2
+                                       && ((unsigned char) a[0]) < 32) {
                               char tmp[32];
 
                               sprintf(tmp, "%%%02x", (unsigned char) a[0]);
@@ -1415,10 +1331,20 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                               }
                               StringClear(tmpbuff);
                               break;
+                            case '<':
+                              StringCat(tmpbuff, "&lt;");
+                              break;
+                            case '>':
+                              StringCat(tmpbuff, "&gt;");
+                              break;
+                            case '&':
+                              StringCat(tmpbuff, "&amp;");
+                              break;
+                            case '\'':
+                              StringCat(tmpbuff, "&#39;");
+                              break;
                             default:
-                              if (!cat_html_escaped(&tmpbuff, *fstr)) {
-                                StringMemcat(tmpbuff, fstr, 1);
-                              }
+                              StringMemcat(tmpbuff, fstr, 1);
                               break;
                             }
                             fstr++;
@@ -1458,9 +1384,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               }
 #endif
             } else {
-              if (is_html(file)) {
-                StringMemcat(headers, ok, sizeof(ok) - 1);
-              } else if (is_text(file)) {
+              if (is_text(file)) {
                 StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
               } else if (is_js(file)) {
                 StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
@@ -1567,7 +1491,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
   StringFree(tmpbuff);
   StringFree(tmpbuff2);
   StringFree(fspath);
-  StringFree(website);
 
   if (buffer)
     free(buffer);

src/htssitemap.c

@@ -1,615 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 2026 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: sitemap ingestion (sitemaps.org 0.9)                   */
-/* Author: Xavier Roche                                         */
-/* ------------------------------------------------------------ */
-
-#define HTS_INTERNAL_BYTECODE
-
-#include "htscore.h"
-#include "htssitemap.h"
-
-#include "htsbase.h"
-#include "htscodec.h"
-#include "htsencoding.h"
-#include "htsfilters.h"
-#include "htshash.h"
-#include "htsmodules.h"
-#include "htslib.h"
-#include "htsrobots.h"
-#include "htssafe.h"
-#include "htstools.h"
-
-#include <ctype.h>
-#include <string.h>
-
-/* One queued sitemap document awaiting ingestion. */
-typedef struct sitemap_doc {
-  char adr[HTS_URLMAXSIZE];
-  char fil[HTS_URLMAXSIZE];
-  int level;
-  hts_sitemap_source src;
-  hts_boolean done;
-  struct sitemap_doc *next;
-} sitemap_doc;
-
-struct hts_sitemap_state {
-  sitemap_doc *docs;
-  int ndocs; /* documents queued, capped by HTS_SITEMAP_MAX_DOCS */
-  int nurls; /* URLs seeded, capped by HTS_SITEMAP_MAX_URLS_TOTAL */
-  hts_boolean probe_done;    /* the robots.txt probe has been answered */
-  hts_boolean fallback_done; /* the /sitemap.xml fallback was already queued */
-  /* The crawl's own start URL. Seeded URLs are judged against it, so a site
-     cannot widen a subtree crawl by putting its sitemap at the root. */
-  char anchor_adr[HTS_URLMAXSIZE];
-  char anchor_fil[HTS_URLMAXSIZE];
-};
-typedef struct hts_sitemap_state hts_sitemap_state;
-
-/* --------------------------------------------------------------------- */
-/* Document parsing (no engine state: fuzzable and self-testable)         */
-/* --------------------------------------------------------------------- */
-
-/* Accept only an absolute http(s) URL with no space or control byte. */
-static hts_boolean sitemap_url_ok(const char *url) {
-  const char *p;
-
-  if (!strfield(url, "http://") && !strfield(url, "https://"))
-    return HTS_FALSE;
-  for (p = url; *p != '\0'; p++) {
-    if ((unsigned char) *p <= ' ' || (unsigned char) *p == 0x7f)
-      return HTS_FALSE;
-  }
-  return HTS_TRUE;
-}
-
-/* Skip to the character after the next '>' at or after p, or NULL. */
-static const char *sitemap_tag_end(const char *p, const char *end) {
-  while (p < end && *p != '>')
-    p++;
-  return p < end ? p + 1 : NULL;
-}
-
-/* HTS_TRUE when the document's root element is `name`. Skips the XML
-   declaration, comments and processing instructions first, so a comment
-   mentioning the other root element cannot decide the document type. */
-static hts_boolean sitemap_root_is(const char *doc, size_t size,
-                                   const char *name) {
-  const size_t nlen = strlen(name);
-  size_t i = 0;
-
-  if (size >= 3 && memcmp(doc, "\xef\xbb\xbf", 3) == 0)
-    i = 3; /* UTF-8 BOM */
-  while (i < size) {
-    if (isspace((unsigned char) doc[i])) {
-      i++;
-    } else if (doc[i] != '<') {
-      return HTS_FALSE; /* character data before any element: not XML */
-    } else if (i + 4 <= size && memcmp(doc + i, "<!--", 4) == 0) {
-      const char *const e = hts_memstr(doc + i, size - i, "-->", 3);
-
-      if (e == NULL)
-        return HTS_FALSE;
-      i = (size_t) (e - doc) + 3;
-    } else if (i + 2 <= size && (doc[i + 1] == '?' || doc[i + 1] == '!')) {
-      while (i < size && doc[i] != '>')
-        i++;
-      i++;
-    } else {
-      size_t j = i + 1;
-
-      /* an optional namespace prefix: <sm:sitemapindex> is the same element */
-      while (j < size && doc[j] != ':' && doc[j] != '>' &&
-             !isspace((unsigned char) doc[j]))
-        j++;
-      if (j >= size || doc[j] != ':')
-        j = i + 1;
-      else
-        j++;
-      return j + nlen <= size && memcmp(doc + j, name, nlen) == 0 &&
-                     (j + nlen == size ||
-                      isspace((unsigned char) doc[j + nlen]) ||
-                      doc[j + nlen] == '>' || doc[j + nlen] == '/')
-                 ? HTS_TRUE
-                 : HTS_FALSE;
-    }
-  }
-  return HTS_FALSE;
-}
-
-/* Decompress a gzip-framed body into a fresh buffer. The 64 MiB cap is what
-   binds in practice; deflate tops out near 1032:1, so the tree's codec budget
-   only matters as the shared policy for a coding that could go further. */
-static char *sitemap_gunzip(const char *body, size_t size, size_t *outsize) {
-  const LLint budget = hts_codec_maxout((LLint) size);
-  size_t cap = budget < (LLint) HTS_SITEMAP_MAX_BYTES
-                   ? (size_t) budget
-                   : (size_t) HTS_SITEMAP_MAX_BYTES;
-  char *out;
-  size_t n;
-
-  if (cap == 0)
-    return NULL;
-  out = malloct(cap + 1);
-  if (out == NULL)
-    return NULL;
-  n = hts_codec_head(HTS_CODEC_DEFLATE, body, size, out, cap);
-  if (n == 0) {
-    freet(out);
-    return NULL;
-  }
-  out[n] = '\0';
-  *outsize = n;
-  return out;
-}
-
-int hts_sitemap_scan(const char *body, size_t size, int maxurls,
-                     hts_boolean *is_index, hts_sitemap_handler handler,
-                     void *arg) {
-  char *unpacked = NULL;
-  const char *doc;
-  const char *end;
-  const char *p;
-  int n = 0;
-
-  if (is_index != NULL)
-    *is_index = HTS_FALSE;
-  if (body == NULL || size < 2 || handler == NULL)
-    return 0;
-
-  /* Content-Encoding gzip is undone upstream; only the container is left. */
-  if ((unsigned char) body[0] == 0x1f && (unsigned char) body[1] == 0x8b) {
-    unpacked = sitemap_gunzip(body, size, &size);
-    if (unpacked == NULL)
-      return -1;
-    doc = unpacked;
-  } else {
-    if (size > (size_t) HTS_SITEMAP_MAX_BYTES)
-      size = (size_t) HTS_SITEMAP_MAX_BYTES;
-    doc = body;
-  }
-  end = doc + size;
-
-  /* Set before the first callback: the handler reads the verdict. */
-  if (is_index != NULL)
-    *is_index = sitemap_root_is(doc, size, "sitemapindex");
-
-  for (p = doc; n < maxurls;) {
-    const char *loc = hts_memstr(p, (size_t) (end - p), "<loc", 4);
-    const char *val;
-    const char *stop;
-    size_t len;
-    char BIGSTK url[HTS_URLMAXSIZE];
-
-    if (loc == NULL)
-      break;
-    /* "<loc>" or "<loc xmlns:..>", never "<location>" */
-    if (loc + 4 >= end || (loc[4] != '>' && !isspace((unsigned char) loc[4]))) {
-      p = loc + 4;
-      continue;
-    }
-    val = sitemap_tag_end(loc + 4, end);
-    if (val == NULL)
-      break;
-    for (stop = val; stop < end && *stop != '<'; stop++)
-      ;
-    /* No closing tag: truncated document, so the value may be a partial URL. */
-    if (stop == end)
-      break;
-    p = stop;
-    while (val < stop && isspace((unsigned char) *val))
-      val++;
-    while (stop > val && isspace((unsigned char) *(stop - 1)))
-      stop--;
-    len = (size_t) (stop - val);
-    /* Overflow-safe: the untrusted length alone against the room left. */
-    if (len == 0 || len >= sizeof(url))
-      continue;
-    memcpy(url, val, len);
-    url[len] = '\0';
-    /* hts_unescapeEntities decodes in place and tolerates src == dest; a
-       reference to a control byte survives as one and sitemap_url_ok drops it.
-     */
-    if (hts_unescapeEntities(url, url, sizeof(url)) != 0 ||
-        !sitemap_url_ok(url))
-      continue;
-    n++;
-    if (!handler(arg, url))
-      break;
-  }
-
-  if (unpacked != NULL)
-    freet(unpacked);
-  return n;
-}
-
-/* --------------------------------------------------------------------- */
-/* Engine glue                                                           */
-/* --------------------------------------------------------------------- */
-
-static hts_sitemap_state *sitemap_get_state(httrackp *opt) {
-  if (opt->sitemap_state == NULL)
-    opt->sitemap_state = calloct(1, sizeof(hts_sitemap_state));
-  return (hts_sitemap_state *) opt->sitemap_state;
-}
-
-static sitemap_doc *sitemap_find(httrackp *opt, const char *adr,
-                                 const char *fil) {
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-  sitemap_doc *d;
-
-  if (st == NULL)
-    return NULL;
-  for (d = st->docs; d != NULL; d = d->next) {
-    if (strfield2(d->adr, adr) && strcmp(d->fil, fil) == 0)
-      return d;
-  }
-  return NULL;
-}
-
-/* Who asked for this document decides how far it is gated. The wizard proper
-   is not usable here: it wants a referring link, and its up/down travel rules
-   would judge a child sitemap against the parent sitemap's directory. */
-static hts_boolean sitemap_fetch_allowed(httrackp *opt, const char *adr,
-                                         const char *fil,
-                                         hts_sitemap_source src) {
-  /* adr and fil are each capped just under HTS_URLMAXSIZE, and lfull prefixes
-     a scheme and a slash on top of both: 2 * HTS_URLMAXSIZE does not fit. */
-  char BIGSTK l[HTS_URLMAXSIZE * 2 + 16], lfull[HTS_URLMAXSIZE * 2 + 16];
-  int jokdepth = 0, jok;
-
-  hts_boolean refused;
-
-  /* The user naming a sitemap is the same intent as naming a start URL, which
-     the wizard admits unconditionally. */
-  if (src == HTS_SITEMAP_SRC_USER)
-    return HTS_TRUE;
-  strcpybuff(l, jump_identification_const(adr));
-  if (*fil != '/')
-    strcatbuff(l, "/");
-  strcatbuff(l, fil);
-  strcpybuff(lfull, link_has_authority(adr) ? "" : "http://");
-  strcatbuff(lfull, adr);
-  if (*fil != '/')
-    strcatbuff(lfull, "/");
-  strcatbuff(lfull, fil);
-  jok = fa_strjoker_dual(0, *opt->filters.filters, *opt->filters.filptr, lfull,
-                         l, NULL, NULL, &jokdepth);
-  refused = (jok == -1) ? HTS_TRUE : HTS_FALSE;
-  if (refused) {
-    hts_log_print(opt, LOG_NOTICE, "Sitemap: filter rule #%d refuses %s%s",
-                  jokdepth + 1, adr, fil);
-    return HTS_FALSE;
-  }
-  /* A Sitemap: line, or a sitemapindex entry, is the site inviting the fetch;
-     a Disallow elsewhere in the same file does not retract it. The well-known
-     location is only ever a guess, so there a Disallow wins. */
-  if (src == HTS_SITEMAP_SRC_GUESSED &&
-      hts_robots_forbids(opt, adr, fil, (jok != 0) ? HTS_TRUE : HTS_FALSE,
-                         refused)) {
-    hts_log_print(opt, LOG_NOTICE, "Sitemap: robots.txt forbids %s%s", adr,
-                  fil);
-    return HTS_FALSE;
-  }
-  return HTS_TRUE;
-}
-
-/* Record the link with save="" so the body stays in memory: a sitemap is
-   ingested, never mirrored. */
-static hts_boolean sitemap_queue_(httrackp *opt, const char *adr,
-                                  const char *fil, int level,
-                                  hts_sitemap_source src, hts_boolean link_it) {
-  hts_sitemap_state *const st = sitemap_get_state(opt);
-  sitemap_doc *d;
-
-  if (st == NULL)
-    return HTS_FALSE;
-  if (st->ndocs >= HTS_SITEMAP_MAX_DOCS || level > HTS_SITEMAP_MAX_LEVEL) {
-    hts_log_print(opt, LOG_WARNING, "Sitemap: cap reached, skipping %s%s", adr,
-                  fil);
-    return HTS_FALSE;
-  }
-  if (strlen(adr) >= sizeof(d->adr) || strlen(fil) >= sizeof(d->fil))
-    return HTS_FALSE;
-  if (sitemap_find(opt, adr, fil) != NULL)
-    return HTS_FALSE;
-  if (!sitemap_fetch_allowed(opt, adr, fil, src))
-    return HTS_FALSE;
-  d = calloct(1, sizeof(sitemap_doc));
-  if (d == NULL)
-    return HTS_FALSE;
-  strcpybuff(d->adr, adr);
-  strcpybuff(d->fil, fil);
-  d->level = level;
-  d->src = src;
-  d->next = st->docs;
-  st->docs = d;
-  st->ndocs++;
-
-  if (!link_it)
-    return HTS_TRUE;
-  if (!hts_record_link(opt, adr, fil, "", "", "", NULL))
-    return HTS_FALSE;
-  heap_top()->testmode = 0;
-  heap_top()->link_import = 0;
-  heap_top()->depth = opt->depth + 1;
-  heap_top()->pass2 = 0;
-  heap_top()->retry = opt->retry;
-  heap_top()->premier = heap_top_index();
-  heap_top()->precedent = heap_top_index();
-  hts_log_print(opt, LOG_INFO, "Sitemap: queued %s%s", adr, fil);
-  return HTS_TRUE;
-}
-
-static hts_boolean sitemap_queue(httrackp *opt, const char *adr,
-                                 const char *fil, int level,
-                                 hts_sitemap_source src) {
-  return sitemap_queue_(opt, adr, fil, level, src, HTS_TRUE);
-}
-
-void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
-                          const char *newadr, const char *newfil) {
-  sitemap_doc *const d = sitemap_find(opt, adr, fil);
-
-  if (d == NULL || d->done)
-    return;
-  d->done = HTS_TRUE; /* the body lives at the target now */
-  /* The engine already queued the target link, so only the marking moves. */
-  (void) sitemap_queue_(opt, newadr, newfil, d->level, d->src, HTS_FALSE);
-  hts_log_print(opt, LOG_NOTICE, "Sitemap: %s%s redirects to %s%s", adr, fil,
-                newadr, newfil);
-}
-
-void hts_sitemap_seed(httrackp *opt, const char *starturl) {
-  char BIGSTK url[HTS_URLMAXSIZE * 2];
-  lien_adrfil af;
-
-  if (StringNotEmpty(opt->sitemap_url)) {
-    if (strlen(StringBuff(opt->sitemap_url)) >= sizeof(url)) {
-      hts_log_print(opt, LOG_ERROR, "Sitemap URL too long");
-    } else {
-      strcpybuff(url, StringBuff(opt->sitemap_url));
-      if (strstr(url, ":/") == NULL)
-        hts_log_print(opt, LOG_ERROR, "Sitemap URL must be absolute: %s", url);
-      else if (ident_url_absolute(url, &af) >= 0)
-        (void) sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_USER);
-    }
-  }
-  if (starturl == NULL || starturl[0] == '\0' ||
-      strlen(starturl) >= sizeof(url))
-    return;
-  strcpybuff(url, starturl);
-  if (ident_url_absolute(url, &af) < 0)
-    return;
-  {
-    hts_sitemap_state *const st = sitemap_get_state(opt);
-
-    if (st != NULL && strlen(af.adr) < sizeof(st->anchor_adr) &&
-        strlen(af.fil) < sizeof(st->anchor_fil)) {
-      strcpybuff(st->anchor_adr, af.adr);
-      strcpybuff(st->anchor_fil, af.fil);
-    }
-  }
-  if (!opt->sitemap)
-    return;
-  /* Answered in hts_sitemap_robots, once the parsed rules are installed. */
-  if (hts_record_link(opt, af.adr, "/robots.txt", "", "", "", NULL)) {
-    heap_top()->testmode = 0;
-    heap_top()->link_import = 0;
-    heap_top()->depth = 0;
-    heap_top()->pass2 = 0;
-    heap_top()->retry = opt->retry;
-    heap_top()->premier = heap_top_index();
-    heap_top()->precedent = heap_top_index();
-    /* Claim the host so the parser does not queue robots.txt a second time. */
-    if (opt->robotsptr != NULL)
-      (void) checkrobots_set((robots_wizard *) opt->robotsptr, af.adr, "");
-  }
-}
-
-void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps) {
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-  int queued = 0;
-
-  if (st == NULL || !opt->sitemap || st->probe_done ||
-      !strfield2(st->anchor_adr, adr))
-    return;
-  st->probe_done = HTS_TRUE;
-  if (sitemaps != NULL) {
-    const char *p = sitemaps;
-
-    while (*p != '\0') {
-      const char *const eol = strchr(p, '\n');
-      const size_t len = eol != NULL ? (size_t) (eol - p) : strlen(p);
-      char BIGSTK line[HTS_URLMAXSIZE];
-      lien_adrfil af;
-
-      if (len > 0 && len < sizeof(line)) {
-        memcpy(line, p, len);
-        line[len] = '\0';
-        /* Same host: a Sitemap: line must not aim the fetcher elsewhere. */
-        if (sitemap_url_ok(line) && ident_url_absolute(line, &af) >= 0 &&
-            strfield2(af.adr, adr) &&
-            sitemap_queue(opt, af.adr, af.fil, 0, HTS_SITEMAP_SRC_DECLARED))
-          queued++;
-      }
-      if (eol == NULL)
-        break;
-      p = eol + 1;
-    }
-  }
-  if (queued == 0 && !st->fallback_done) {
-    st->fallback_done = HTS_TRUE;
-    if (sitemap_queue(opt, adr, "/sitemap.xml", 0, HTS_SITEMAP_SRC_GUESSED))
-      queued++;
-  }
-  hts_log_print(opt, LOG_NOTICE, "Sitemap: %d sitemap(s) queued for %s", queued,
-                adr);
-}
-
-hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
-                                const char *fil) {
-  const sitemap_doc *const d = sitemap_find(opt, adr, fil);
-
-  return d != NULL && !d->done ? HTS_TRUE : HTS_FALSE;
-}
-
-/* Handler context: seeding URLs from one document. */
-typedef struct sitemap_ingest_ctx {
-  httrackp *opt;
-  htsmoduleStruct *str;
-  const char *adr; /* host of the document being ingested */
-  int level;
-  hts_boolean is_index;
-  int accepted; /* URLs seeded or documents queued, not merely parsed */
-} sitemap_ingest_ctx;
-
-/* A <loc> of a <urlset>: hand it to the wizard as a top-level seed.
-   The wizard is pointed at the crawl's own start URL, not at the sitemap: the
-   site picks where its sitemap lives, so anchoring travel there would let a
-   root sitemap widen a subtree crawl to the whole host. The URL then becomes
-   its own anchor, exactly as a command-line seed does. */
-static hts_boolean sitemap_seed_url(void *arg, const char *url) {
-  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
-  httrackp *const opt = c->opt;
-  hts_sitemap_state *const st = sitemap_get_state(opt);
-  char BIGSTK buff[HTS_URLMAXSIZE];
-  int before;
-
-  if (st == NULL || st->nurls >= HTS_SITEMAP_MAX_URLS_TOTAL) {
-    hts_log_print(opt, LOG_WARNING,
-                  "Sitemap: URL cap reached, ignoring the rest");
-    return HTS_FALSE;
-  }
-  /* strcpybuff aborts rather than truncating: never feed it unchecked input. */
-  if (strlen(url) >= sizeof(buff))
-    return HTS_TRUE;
-  st->nurls++;
-  strcpybuff(buff, url);
-  before = opt->lien_tot;
-  if (htsAddLink(c->str, buff))
-    c->accepted++;
-  if (opt->lien_tot > before)
-    heap_top()->premier = heap_top_index(); /* a seed anchors on itself */
-  return HTS_TRUE;
-}
-
-/* A <loc> of a <sitemapindex>: cross-host children are dropped, so a hostile
-   sitemap cannot aim the fetcher elsewhere. */
-static hts_boolean sitemap_seed_child(void *arg, const char *url) {
-  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
-  char BIGSTK buff[HTS_URLMAXSIZE];
-  lien_adrfil af;
-
-  if (strlen(url) >= sizeof(buff))
-    return HTS_TRUE;
-  strcpybuff(buff, url);
-  if (ident_url_absolute(buff, &af) < 0)
-    return HTS_TRUE;
-  if (!strfield2(af.adr, c->adr)) {
-    hts_log_print(c->opt, LOG_WARNING,
-                  "Sitemap: ignoring off-host child sitemap %s%s", af.adr,
-                  af.fil);
-    return HTS_TRUE;
-  }
-  if (sitemap_queue(c->opt, af.adr, af.fil, c->level + 1,
-                    HTS_SITEMAP_SRC_DECLARED))
-    c->accepted++;
-  return HTS_TRUE;
-}
-
-/* The scan classifies the document before the first callback. */
-static hts_boolean sitemap_seed_any(void *arg, const char *url) {
-  sitemap_ingest_ctx *const c = (sitemap_ingest_ctx *) arg;
-
-  return c->is_index ? sitemap_seed_child(arg, url)
-                     : sitemap_seed_url(arg, url);
-}
-
-void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
-                        const char *fil, const char *body, size_t size) {
-  sitemap_doc *const d = sitemap_find(opt, adr, fil);
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-  sitemap_ingest_ctx ctx;
-  int n, anchor, saved_depth;
-
-  if (d == NULL || d->done)
-    return;
-  d->done = HTS_TRUE;
-  /* str->ptr_ is a scratch int owned by the caller, so nothing else moves. */
-  anchor = *str->ptr_;
-  if (st != NULL && st->anchor_adr[0] != '\0' && opt->hash != NULL) {
-    const int i = hash_read((const hash_struct *) opt->hash, st->anchor_adr,
-                            st->anchor_fil, 1);
-
-    if (i >= 0)
-      anchor = i;
-  }
-  *str->ptr_ = anchor;
-  /* Borrow the anchor's position but keep a seed's full depth budget. */
-  saved_depth = heap(anchor)->depth;
-  heap(anchor)->depth = opt->depth + 1;
-  ctx.opt = opt;
-  ctx.str = str;
-  ctx.adr = adr;
-  ctx.level = d->level;
-  ctx.is_index = HTS_FALSE;
-  ctx.accepted = 0;
-
-  n = hts_sitemap_scan(body, size, HTS_SITEMAP_MAX_URLS_DOC, &ctx.is_index,
-                       sitemap_seed_any, &ctx);
-  heap(anchor)->depth = saved_depth;
-  if (n < 0) {
-    hts_log_print(opt, LOG_ERROR, "Sitemap: could not decompress %s%s", adr,
-                  fil);
-    return;
-  }
-  if (ctx.is_index)
-    hts_log_print(opt, LOG_NOTICE,
-                  "Sitemap: %d of %d child sitemap(s) listed by %s%s",
-                  ctx.accepted, n, adr, fil);
-  else
-    hts_log_print(opt, LOG_NOTICE, "Sitemap: %d of %d URL(s) added from %s%s",
-                  ctx.accepted, n, adr, fil);
-}
-
-void hts_sitemap_free(httrackp *opt) {
-  hts_sitemap_state *const st = (hts_sitemap_state *) opt->sitemap_state;
-
-  if (st == NULL)
-    return;
-  while (st->docs != NULL) {
-    sitemap_doc *const next = st->docs->next;
-
-    freet(st->docs);
-    st->docs = next;
-  }
-  freet(opt->sitemap_state);
-  opt->sitemap_state = NULL;
-}

src/htssitemap.h

@@ -1,108 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 2026 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* HTTrack sitemap ingestion (sitemaps.org 0.9). Internal, not installed.
-   Reads <urlset>/<sitemapindex> documents, plain or gzip-framed, and feeds
-   their <loc> URLs to the crawl as top-level seeds. The whole input is
-   attacker-controlled, so every entry point below is capped. */
-/* ------------------------------------------------------------ */
-
-#ifndef HTS_SITEMAP_DEFH
-#define HTS_SITEMAP_DEFH
-
-#include "htsdefines.h"
-#include "htsopt.h"
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/* Caps. sitemaps.org allows 50000 URLs and 50 MB uncompressed per document;
-   the byte cap sits above that so a conformant sitemap always fits. */
-#define HTS_SITEMAP_MAX_URLS_DOC 50000           /* <loc> per document */
-#define HTS_SITEMAP_MAX_URLS_TOTAL 200000        /* <loc> per mirror */
-#define HTS_SITEMAP_MAX_DOCS 256                 /* documents per mirror */
-#define HTS_SITEMAP_MAX_LEVEL 4                  /* sitemapindex nesting */
-#define HTS_SITEMAP_MAX_BYTES (64 * 1024 * 1024) /* decompressed document */
-
-/* Who asked for a sitemap document, which decides how far its fetch is gated.
-   The user naming one is the same intent as a start URL; a site declaring one
-   invites the fetch; the well-known location is only ever our guess. */
-typedef enum {
-  HTS_SITEMAP_SRC_USER,     /**< --sitemap-url */
-  HTS_SITEMAP_SRC_DECLARED, /**< a Sitemap: line or a sitemapindex entry */
-  HTS_SITEMAP_SRC_GUESSED   /**< the /sitemap.xml fallback */
-} hts_sitemap_source;
-
-/* Per-URL handler; returning HTS_FALSE stops the scan. */
-typedef hts_boolean (*hts_sitemap_handler)(void *arg, const char *url);
-
-/* Scan one sitemap document, plain or gzip-framed, handing every acceptable
-   absolute http(s) <loc> URL to `handler`. Stops after `maxurls` URLs, or when
-   the handler refuses. `is_index` (optional) reports a <sitemapindex>, whose
-   URLs are child sitemaps rather than pages. Returns the number of URLs handed
-   out, or -1 when the document could not be decompressed within the caps. */
-int hts_sitemap_scan(const char *body, size_t size, int maxurls,
-                     hts_boolean *is_index, hts_sitemap_handler handler,
-                     void *arg);
-
-/* --- Engine glue (needs a live httrackp). --- */
-
-/* Queue the first sitemap document of the mirror: the explicit --sitemap-url,
-   or the start host's /robots.txt probe for --sitemap. `starturl` is the first
-   command-line seed. No-op when neither option is set. */
-void hts_sitemap_seed(httrackp *opt, const char *starturl);
-
-/* Act on the start host's robots.txt once its rules are installed: queue the
-   Sitemap: URLs it names (newline-separated, from robots_parse), or the
-   well-known /sitemap.xml when it names none. No-op unless --sitemap. */
-void hts_sitemap_robots(httrackp *opt, const char *adr, const char *sitemaps);
-
-/* Carry the sitemap marking of (adr,fil) over to the target of a redirect the
-   engine has already queued, so a moved sitemap is still ingested. */
-void hts_sitemap_redirect(httrackp *opt, const char *adr, const char *fil,
-                          const char *newadr, const char *newfil);
-
-/* HTS_TRUE when (adr,fil) is a queued sitemap document awaiting ingestion. */
-hts_boolean hts_sitemap_pending(httrackp *opt, const char *adr,
-                                const char *fil);
-
-/* Ingest a fetched sitemap document (or the robots.txt probe): seed its URLs
-   through the wizard via htsAddLink, and queue nested sitemaps. `str` supplies
-   the parser context of the document being processed. */
-void hts_sitemap_ingest(httrackp *opt, htsmoduleStruct *str, const char *adr,
-                        const char *fil, const char *body, size_t size);
-
-/* Release the ingestion state held in opt (NULL-safe, idempotent). */
-void hts_sitemap_free(httrackp *opt);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif

src/htsstats.h

@@ -1,63 +0,0 @@
-/* ------------------------------------------------------------ */
-/*
-HTTrack Website Copier, Offline Browser for Windows and Unix
-Copyright (C) 1998 Xavier Roche and other contributors
-
-SPDX-License-Identifier: GPL-3.0-or-later
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-Ethical use: we kindly ask that you NOT use this software to harvest email
-addresses or to collect any other private information about people. Doing so
-would dishonor our work and waste the many hours we have spent on it.
-
-Please visit our Website: http://www.httrack.com
-*/
-
-/* ------------------------------------------------------------ */
-/* File: in-progress display rows, shared by httrack and htsserver .h */
-/* Author: Xavier Roche                                         */
-/* ------------------------------------------------------------ */
-
-#ifndef HTS_STATS_DEFH
-#define HTS_STATS_DEFH
-
-#include "htsglobal.h"
-
-/* One row of the "in progress" display, shared so the CLI (httrack) and the
-   web GUI (htsserver) cannot drift apart: each fills its own array. */
-
-#define NStatsBuffer 14
-
-#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
-#define HTS_DEF_FWSTRUCT_t_StatsBuffer
-typedef struct t_StatsBuffer t_StatsBuffer;
-#endif
-struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[288];                         // a short label plus back->info[256]
-  char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
-  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
-  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived; // pour disabled
-};
-
-#endif

src/htstools.c

@@ -276,21 +276,6 @@ int ident_url_relatif(const char *lien, const char *origin_adr,
   return ok;
 }
 
-/* Bounded substring search: bodies and archive records carry NUL bytes, so
-   strstr() would stop at the first one. */
-const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
-                       size_t nlen) {
-  size_t i;
-
-  if (nlen == 0 || haylen < nlen)
-    return NULL;
-  for (i = 0; i + nlen <= haylen; i++) {
-    if (hay[i] == *needle && memcmp(hay + i, needle, nlen) == 0)
-      return hay + i;
-  }
-  return NULL;
-}
-
 // créer dans s, à partir du chemin courant curr_fil, le lien vers link (absolu)
 // un ident_url_relatif a déja été fait avant, pour que link ne soit pas un chemin relatif
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
@@ -323,10 +308,8 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
   // copy only the current path
   curr = _curr;
   strlcpybuff(curr, curr_fil, sizeof(_curr));
-  if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
-    // an empty path has no last character: curr-1 would read before the buffer
-    a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
-  }
+  if ((a = strchr(curr, '?')) == NULL)  // couper au ? (params)
+    a = curr + strlen(curr) - 1;        // pas de params: aller à la fin
   while((*a != '/') && (a > curr))
     a--;                        // chercher dernier / du chemin courant
   if (*a == '/')
@@ -1335,19 +1318,19 @@ HTSEXT_API hts_boolean hts_findnext(find_handle find) {
   if (find) {
 #ifdef _WIN32
     if ((FindNextFileA(find->handle, &find->hdata)))
-      return HTS_TRUE;
+      return 1;
 #else
     char catbuff[CATBUFF_SIZE];
 
     memset(&(find->filestat), 0, sizeof(find->filestat));
     if ((find->dirp = readdir(find->hdir)))
-      if (!STAT(
-              concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name),
-              &find->filestat))
-        return HTS_TRUE;
+      if (find->dirp->d_name)
+        if (!STAT
+            (concat(catbuff, sizeof(catbuff), find->path, find->dirp->d_name), &find->filestat))
+          return 1;
 #endif
   }
-  return HTS_FALSE;
+  return 0;
 }
 
 HTSEXT_API int hts_findclose(find_handle find) {

src/htstools.h

@@ -61,11 +61,6 @@ typedef struct lien_adrfilsave lien_adrfilsave;
 int ident_url_relatif(const char *lien, const char *origin_adr,
                       const char *origin_fil,
                       lien_adrfil* const adrfil);
-/* Bounded substring search over data that may hold NUL bytes; NULL if absent.
- */
-const char *hts_memstr(const char *hay, size_t haylen, const char *needle,
-                       size_t nlen);
-
 int lienrelatif(char *s, size_t ssize, const char *link, const char *curr);
 int link_has_authority(const char *lien);
 int link_has_authorization(const char *lien);

src/htsweb.c

@@ -62,7 +62,6 @@ Please visit our Website: http://www.httrack.com
 #include "htsmd5.c"
 #include "md5.c"
 
-#include "htscmdline.h"
 #include "htsserver.h"
 #include "htsurlport.h"
 #include "htsweb.h"
@@ -320,8 +319,10 @@ int main(int argc, char *argv[]) {
 static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
 static void back_launch_cmd(void *pP) {
   char *cmd = (char *) pP;
-  char **argv;
+  char **argv = (char **) malloct(1024 * sizeof(char *));
   int argc = 0;
+  int i = 0;
+  int g = 0;
 
   //
   httrackp *opt;
@@ -332,19 +333,28 @@ static void back_launch_cmd(void *pP) {
   commandReturnCmdl = strdup(cmd);
 
   /* split */
-  argv = hts_split_cmdline(cmd, &argc);
-  if (argv == NULL) {
-    if (commandReturnMsg)
-      free(commandReturnMsg);
-    commandReturnMsg = strdup("could not parse the command line");
-    commandReturn = -1;
-    commandRunning = 0;
-    commandEnd = 1;
-    free(cmd);
-    return;
+  argv[0] = strdup("webhttrack");
+  argv[1] = cmd;
+  argc++;
+  i = 0;
+  while(cmd[i]) {
+    if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
+      cmd[i] = ' ';
+    }
+    i++;
+  }
+  i = 0;
+  while(cmd[i]) {
+    if (cmd[i] == '\"')
+      g = !g;
+    if (cmd[i] == ' ') {
+      if (!g) {
+        cmd[i] = '\0';
+        argv[argc++] = cmd + i + 1;
+      }
+    }
+    i++;
   }
-  /* drop the program name the posted command line carries */
-  argv[0] = strdupt("webhttrack");
 
   /* init */
   hts_init();
@@ -373,7 +383,6 @@ static void back_launch_cmd(void *pP) {
 
   /* free */
   free(cmd);
-  freet(argv[0]);
   freet(argv);
   return;
 }
@@ -673,10 +682,10 @@ int __cdecl htsshow_loop(t_hts_callbackarg * carg, httrackp * opt, lien_back * b
                     char *eps = strchr(back[i].url_adr, '/');
                     int count;
 
-                    if (ep != NULL && eps != NULL && ep < eps &&
-                        (count = (int) (ep - back[i].url_adr)) < 4) {
+                    if (ep != NULL && ep < eps
+                        && (count = (int) (ep - back[i].url_adr)) < 4) {
                       proto[0] = '\0';
-                      strncatbuff(proto, back[i].url_adr, count);
+                      strncat(proto, back[i].url_adr, count);
                     }
                   }
                   snprintf(StatsBuffer[index].state, sizeof(StatsBuffer[index].state),

src/htsweb.h

@@ -35,10 +35,26 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
-#include "htsstats.h"
 
+#define NStatsBuffer     14
 #define MAX_LEN_INPROGRESS 40
 
+typedef struct t_StatsBuffer {
+  char name[1024];
+  char file[1024];
+  char state[256];
+  char url_sav[HTS_URLMAXSIZE * 2];     // pour cancel
+  char url_adr[HTS_URLMAXSIZE * 2];
+  char url_fil[HTS_URLMAXSIZE * 2];
+  LLint size;
+  LLint sizetot;
+  int offset;
+  //
+  int back;
+  //
+  int actived;                  // pour disabled
+} t_StatsBuffer;
+
 typedef struct t_InpInfo {
   int ask_refresh;
   int refresh;

src/htswizard.c

@@ -151,23 +151,6 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag,
   return HTS_FALSE;
 }
 
-/* The engine's robots.txt verdict for (adr,fil). Under HTS_ROBOTS_SOMETIMES an
-   explicit filter acceptance overrides the ban, which is why the filter outcome
-   is an input; the sitemap fetcher asks the same question outside the wizard.
- */
-hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
-                               hts_boolean filters_decided,
-                               hts_boolean filters_refused) {
-  if (!opt->robots || opt->robotsptr == NULL)
-    return HTS_FALSE;
-  if (checkrobots((robots_wizard *) opt->robotsptr, adr, fil) != -1)
-    return HTS_FALSE;
-  if (filters_decided && !filters_refused &&
-      opt->robots == HTS_ROBOTS_SOMETIMES)
-    return HTS_FALSE;
-  return HTS_TRUE;
-}
-
 static int hts_acceptlink_(httrackp * opt, int ptr,
                            const char *adr, const char *fil, const char *tag,
                            const char *attribute, int *set_prio_to,
@@ -593,26 +576,30 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
       }
     }
     // vérifier robots.txt
-    if (opt->robots && checkrobots(_ROBOTS, adr, fil) == -1) {
+    if (opt->robots) {
+      int r = checkrobots(_ROBOTS, adr, fil);
+
+      if (r == -1) {            // interdiction
 #if DEBUG_ROBOTS
-      printf("robots.txt forbidden: %s%s\n", adr, fil);
+        printf("robots.txt forbidden: %s%s\n", adr, fil);
 #endif
-      if (!hts_robots_forbids(opt, adr, fil,
-                              (!question && filters_answer) ? HTS_TRUE
-                                                            : HTS_FALSE,
-                              (forbidden_url == 1) ? HTS_TRUE : HTS_FALSE)) {
-        if (!forbidden_url) {
-          hts_log_print(
-              opt, LOG_DEBUG,
-              "Warning link followed against robots.txt: link %s at %s%s", l,
-              adr, fil);
+        // question résolue, par les filtres, et mode robot non strict
+        if ((!question) && (filters_answer) &&
+            (opt->robots == HTS_ROBOTS_SOMETIMES) && (forbidden_url != 1)) {
+          r = 0;                // annuler interdiction des robots
+          if (!forbidden_url) {
+            hts_log_print(opt, LOG_DEBUG,
+                          "Warning link followed against robots.txt: link %s at %s%s",
+                          l, adr, fil);
+          }
+        }
+        if (r == -1) {          // interdire
+          forbidden_url = 1;
+          question = 0;
+          hts_log_print(opt, LOG_DEBUG,
+                        "(robots.txt) forbidden link: link %s at %s%s", l, adr,
+                        fil);
         }
-      } else {
-        forbidden_url = 1;
-        question = 0;
-        hts_log_print(opt, LOG_DEBUG,
-                      "(robots.txt) forbidden link: link %s at %s%s", l, adr,
-                      fil);
       }
     }
 

src/htswizard.h

@@ -49,13 +49,6 @@ typedef struct httrackp httrackp;
 typedef struct lien_url lien_url;
 #endif
 
-/* The engine's robots.txt verdict for (adr,fil): HTS_TRUE when the fetch is
-   forbidden. `filters_decided`/`filters_refused` carry the filter outcome,
-   which overrides a ban under -s1 (HTS_ROBOTS_SOMETIMES). */
-hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
-                               hts_boolean filters_decided,
-                               hts_boolean filters_refused);
-
 int hts_acceptlink(httrackp * opt, int ptr,
                    const char *adr, const char *fil,
                    const char *tag, const char *attribute,

src/httrack.c

@@ -46,7 +46,6 @@ Please visit our Website: http://www.httrack.com
 #include "httrack.h"
 #include "htslib.h"
 #include "htscharset.h" // after htslib.h: winsock2.h must precede windows.h
-#include "htsbacktrace.h"
 
 /* Static definitions */
 static int fexist(const char *s);
@@ -73,6 +72,10 @@ static int linput(FILE * fp, char *s, int max);
 #include <sys/ioctl.h>
 #endif
 #include <ctype.h>
+#if (defined(__linux) && defined(HAVE_EXECINFO_H))
+#include <execinfo.h>
+#define USES_BACKTRACE
+#endif
 /* END specific definitions */
 
 static void __cdecl htsshow_init(t_hts_callbackarg * carg);
@@ -230,7 +233,8 @@ static hts_boolean vt_size_refresh(void) {
 */
 #define STYLE_STATVALUES VT_BOLD
 #define STYLE_STATTEXT   VT_UNBOLD
-#define STYLE_STATRESET VT_UNBOLD
+#define STYLE_STATRESET  VT_UNBOLD
+#define NStatsBuffer     14
 /* Rows the stats block and "Current job" take above the in-progress list. */
 #define NStatsHeaderRows 7
 
@@ -876,6 +880,21 @@ static void sig_doback(int blind) {     // mettre en backing
 #undef FD_ERR
 #define FD_ERR 2
 
+static void print_backtrace(void) {
+#ifdef USES_BACKTRACE
+  void *stack[256];
+  const int size = backtrace(stack, sizeof(stack)/sizeof(stack[0]));
+  if (size != 0) {
+    backtrace_symbols_fd(stack, size, FD_ERR);
+  }
+#else
+  const char msg[] = "No stack trace available on this OS :(\n";
+  if (write(FD_ERR, msg, sizeof(msg) - 1) != sizeof(msg) - 1) {
+    /* sorry GCC */
+  }
+#endif
+}
+
 static size_t print_num(char *buffer, int num) {
   size_t i, j;
   if (num < 0) {
@@ -909,7 +928,7 @@ static void sig_fatal(int code) {
   size += print_num(&buffer[size], code);
   buffer[size++] = '\n';
   (void) (write(FD_ERR, buffer, size) == size);
-  hts_print_backtrace(FD_ERR);
+  print_backtrace();
   (void) (write(FD_ERR, msgreport, sizeof(msgreport) - 1)
     == sizeof(msgreport) - 1);
   abort();
@@ -932,7 +951,6 @@ static void sig_leave(int code) {
 }
 
 static void signal_handlers(void) {
-  hts_backtrace_init();
 #ifdef _WIN32
   signal(SIGINT, sig_leave);    // ^C
   signal(SIGTERM, sig_finish);  // kill <process>

src/httrack.h

@@ -36,7 +36,26 @@ Please visit our Website: http://www.httrack.com
 #include "htsglobal.h"
 #include "htscore.h"
 #include "htssafe.h"
-#include "htsstats.h"
+
+#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
+#define HTS_DEF_FWSTRUCT_t_StatsBuffer
+typedef struct t_StatsBuffer t_StatsBuffer;
+#endif
+struct t_StatsBuffer {
+  char name[1024];
+  char file[1024];
+  char state[256];
+  char BIGSTK url_sav[HTS_URLMAXSIZE * 2];      // pour cancel
+  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
+  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
+  LLint size;
+  LLint sizetot;
+  int offset;
+  //
+  int back;
+  //
+  int actived;                  // pour disabled
+};
 
 #ifndef HTS_DEF_FWSTRUCT_t_InpInfo
 #define HTS_DEF_FWSTRUCT_t_InpInfo

src/httrack.vcxproj

@@ -100,7 +100,6 @@
 
   <ItemGroup>
     <ClCompile Include="httrack.c" />
-    <ClCompile Include="htsbacktrace.c" />
   </ItemGroup>
 
   <!-- Pulls in libhttrack.lib and builds it first. -->

src/libhttrack.vcxproj

@@ -59,10 +59,9 @@
   <ItemDefinitionGroup>
     <ClCompile>
       <!-- LIBHTTRACK_EXPORTS turns HTSEXT_API into __declspec(dllexport); ZLIB_DLL
-           imports zlib from its DLL, ZLIB_CONST makes its next_in const as the
-           autotools build does. HTS_USEBROTLI/HTS_USEZSTD enable the br and zstd
-           content codings. Windows 7 floor, matching WinHTTrack. -->
-      <PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;ZLIB_CONST;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+           imports zlib from its DLL. HTS_USEBROTLI/HTS_USEZSTD enable the br and
+           zstd content codings. Windows 7 floor, matching WinHTTrack. -->
+      <PreprocessorDefinitions>WIN32;_WINDOWS;_MBCS;_USRDLL;LIBHTTRACK_EXPORTS;ZLIB_DLL;HTS_USEBROTLI=1;HTS_USEZSTD=1;WINVER=0x0601;_WIN32_WINNT=0x0601;_CRT_SECURE_NO_WARNINGS;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>$(MSBuildThisFileDirectory);$(MSBuildThisFileDirectory)coucal;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <WarningLevel>Level3</WarningLevel>
       <MultiProcessorCompilation>true</MultiProcessorCompilation>
@@ -122,7 +121,6 @@
     <ClCompile Include="htshelp.c" />
     <ClCompile Include="htsindex.c" />
     <ClCompile Include="htslib.c" />
-    <ClCompile Include="htscmdline.c" />
     <ClCompile Include="htsurlport.c" />
     <ClCompile Include="htsmd5.c" />
     <ClCompile Include="htsmodules.c" />
@@ -138,7 +136,6 @@
     <ClCompile Include="htswrap.c" />
     <ClCompile Include="htszlib.c" />
     <ClCompile Include="htswarc.c" />
-    <ClCompile Include="htssitemap.c" />
     <ClCompile Include="md5.c" />
     <ClCompile Include="minizip\ioapi.c" />
     <ClCompile Include="minizip\iowin32.c" />

src/proxy/proxytrack.c

@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     PT_Element elt = PT_ElementNew();
 
     elt->statuscode = 405;
-    strcpybuff(elt->msg, "Method Not Allowed");
+    strcpy(elt->msg, "Method Not Allowed");
     return elt;
   }
 
@@ -771,7 +771,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
                   PT_ReadIndex(indexes, StringBuff(itemUrl) + 1, FETCH_HEADERS);
                 if (file != NULL && file->statuscode == HTTP_OK) {
                   size = file->size;
-                  if (file->lastmodified[0] != '\0') {
+                  if (file->lastmodified) {
                     timestamp = get_time_rfc822(file->lastmodified);
                   }
                   if (timestamp == (time_t) 0) {
@@ -785,7 +785,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
                     }
                     timestamp = timestampRep;
                   }
-                  if (file->contenttype[0] != '\0') {
+                  if (file->contenttype) {
                     mimeType = file->contenttype;
                   }
                 }
@@ -822,7 +822,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     elt->statuscode = 207;      /* Multi-Status */
     strcpy(elt->charset, "utf-8");
     strcpy(elt->contenttype, "text/xml");
-    strcpybuff(elt->msg, "Multi-Status");
+    strcpy(elt->msg, "Multi-Status");
     StringFree(response);
 
     fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
@@ -879,7 +879,7 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
     elt->statuscode = HTTP_OK;
     strcpy(elt->charset, "iso-8859-1");
     strcpy(elt->contenttype, "text/html");
-    strcpybuff(elt->msg, "OK");
+    strcpy(elt->msg, "OK");
     StringFree(html);
     return elt;
   }

src/proxy/store.c

@@ -407,7 +407,7 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
     elt->statuscode = HTTP_OK;
     strcpy(elt->charset, "iso-8859-1");
     strcpy(elt->contenttype, "text/html");
-    strcpybuff(elt->msg, "OK");
+    strcpy(elt->msg, "OK");
     StringFree(html);
     return elt;
   }
@@ -579,10 +579,9 @@ PT_Index PT_LoadCache(const char *filename) {
         if (chain != NULL) {
           const char *scheme = link_has_authority(chain->name) ? "" : "http://";
 
-          /* dropped rather than truncated: empty already reads as "unset" */
-          if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
-                           (const char *) chain->name))
-            index->slots.common.startUrl[0] = '\0';
+          snprintf(index->slots.common.startUrl,
+                   sizeof(index->slots.common.startUrl), "%s%s", scheme,
+                   (const char *) chain->name);
         }
       }
     }
@@ -828,18 +827,6 @@ PT_Element PT_ElementNew(void) {
   return r;
 }
 
-/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
-static void PT_Element_failf(PT_Element r, const char *fmt, ...)
-    HTS_PRINTF_FUN(2, 3);
-
-static void PT_Element_failf(PT_Element r, const char *fmt, ...) {
-  va_list args;
-
-  va_start(args, fmt);
-  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
-  va_end(args);
-}
-
 PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
   if (index != NULL && SAFE_INDEX(index)) {
     return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
@@ -878,17 +865,12 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
   sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
   (headersSize) += (int) strlen(headers + headersSize); \
 } while(0)
-/* refvalue_size is mandatory: the cache line is bounded only by the line
-   buffer, not by the destination. Clip rather than reject, since an
-   engine-written field can be wider than ours. */
-#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
-  do {                                                                         \
-    if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (refvalue)[0] = '\0';                                                    \
-      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
-      line[0] = '\0';                                                          \
-    }                                                                          \
-  } while (0)
+#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
+  if (line[0] != '\0' && strfield2(line, refline)) { \
+    strcpy(refvalue, value); \
+    line[0] = '\0'; \
+	} \
+} while(0)
 #define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
   if (line[0] != '\0' && strfield2(line, refline)) { \
     int intval = 0; \
@@ -990,12 +972,9 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
                     const char *scheme =
                         link_has_authority(filenameIndex) ? "" : "http://";
 
-                    /* dropped rather than truncated; try the next entry */
-                    if (sprintfbuff(index->startUrl, "%s%s", scheme,
-                                    filenameIndex))
-                      firstSeen = 1;
-                    else
-                      index->startUrl[0] = '\0';
+                    firstSeen = 1;
+                    snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
+                             scheme, filenameIndex);
                   }
                 }
               } else {
@@ -1091,23 +1070,16 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                 value++;
               ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
               ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
-              ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
-                                   sizeof(r->msg));
+              ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg);     // msg
               ZIP_READFIELD_INT(line, value, "X-Size", r->size);        // size
-              ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
-                                   sizeof(r->contenttype));
-              ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
-                                   sizeof(r->charset));
-              ZIP_READFIELD_STRING(line, value, "Last-Modified",
-                                   r->lastmodified, sizeof(r->lastmodified));
-              ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
-                                   sizeof(r->etag));
-              ZIP_READFIELD_STRING(line, value, "Location", r->location,
-                                   sizeof(location_default));
+              ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype);        // contenttype
+              ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset);       // contenttype
+              ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified);      // last-modified
+              ZIP_READFIELD_STRING(line, value, "Etag", r->etag);       // Etag
+              ZIP_READFIELD_STRING(line, value, "Location", r->location);       // 'location' pour moved
               ZIP_READFIELD_STRING(line, value, "Content-Disposition",
-                                   r->cdispo, sizeof(r->cdispo));
-              ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
-                                   sizeof(previous_save_));
+                                   r->cdispo); // Content-disposition
+              ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_);      // Original save filename
               if (line[0] != '\0') {
                 int len = r->headers ? ((int) strlen(r->headers)) : 0;
                 int nlen =
@@ -1163,9 +1135,8 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                   snprintf(previous_save, sizeof(previous_save), "%s%s",
                            index->path, previous_save_ + index->fixedPath);
                 } else {
-                  PT_Element_failf(
-                      r, "Bogus fixePath prefix for %s (prefixLen=%d)",
-                      previous_save_, (int) index->fixedPath);
+                  snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
+                          previous_save_, (int) index->fixedPath);
                   r->statuscode = STATUSCODE_INVALID;
                 }
               } else {
@@ -1184,7 +1155,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
             // Peut-on stocker le fichier directement sur disque?
             if (ok) {
               if (r->msg[0] == '\0') {
-                strcpybuff(r->msg, "Cache Read Error : Unexpected error");
+                strcpy(r->msg, "Cache Read Error : Unexpected error");
               }
             } else {            // lire en mémoire
 
@@ -1202,27 +1173,24 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                           int last_errno = errno;
 
                           r->statuscode = STATUSCODE_INVALID;
-                          PT_Element_failf(r,
-                                           "Read error in cache disk data: %s",
-                                           strerror(last_errno));
+                          sprintf(r->msg, "Read error in cache disk data: %s",
+                                  strerror(last_errno));
                         }
                         r->adr[r->size] = '\0';
                       } else {
                         r->statuscode = STATUSCODE_INVALID;
-                        strcpybuff(r->msg,
-                                   "Read error (memory exhausted) from cache");
+                        strcpy(r->msg,
+                               "Read error (memory exhausted) from cache");
                       }
                       fclose(fp);
                     } else {
                       r->statuscode = STATUSCODE_INVALID;
-                      PT_Element_failf(
-                          r, "Read error (can't open '%s') from cache",
-                          file_convert(catbuff, sizeof(catbuff),
-                                       previous_save));
+                      snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
+                              file_convert(catbuff, sizeof(catbuff), previous_save));
                     }
                   } else {
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpybuff(r->msg, "Cached file name is invalid");
+                    strcpy(r->msg, "Cached file name is invalid");
                   }
                 }
               } else {
@@ -1234,12 +1202,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
                       free(r->adr);
                       r->adr = NULL;
                       r->statuscode = STATUSCODE_INVALID;
-                      strcpybuff(r->msg, "Cache Read Error : Read Data");
+                      strcpy(r->msg, "Cache Read Error : Read Data");
                     } else
                       *(r->adr + r->size) = '\0';
                   } else {      // erreur
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpybuff(r->msg, "Cache Memory Error");
+                    strcpy(r->msg, "Cache Memory Error");
                   }
                 }
               }
@@ -1247,21 +1215,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
           }                     // si save==null, ne rien charger (juste en tête)
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpybuff(r->msg, "Cache Read Error : Read Header Data");
+          strcpy(r->msg, "Cache Read Error : Read Header Data");
         }
         unzCloseCurrentFile(index->zFile);
       } else {
         r->statuscode = STATUSCODE_INVALID;
-        strcpybuff(r->msg, "Cache Read Error : Open File");
+        strcpy(r->msg, "Cache Read Error : Open File");
       }
 
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpybuff(r->msg, "Cache Read Error : Bad Offset");
+      strcpy(r->msg, "Cache Read Error : Bad Offset");
     }
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpybuff(r->msg, "File Cache Entry Not Found");
+    strcpy(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -1602,11 +1570,9 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
                   const char *scheme =
                       link_has_authority(line) ? "" : "http://";
 
-                  /* dropped rather than truncated; try the next entry */
-                  if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
-                    firstSeen = 1;
-                  else
-                    index->startUrl[0] = '\0';
+                  firstSeen = 1;
+                  snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
+                           scheme, line);
                 }
               }
 
@@ -1827,9 +1793,8 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                 snprintf(previous_save, sizeof(previous_save), "%s%s",
                          index->path, previous_save_ + index->fixedPath);
               } else {
-                PT_Element_failf(r,
-                                 "Bogus fixePath prefix for %s (prefixLen=%d)",
-                                 previous_save_, (int) index->fixedPath);
+                snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
+                        previous_save_, (int) index->fixedPath);
                 r->statuscode = STATUSCODE_INVALID;
               }
             } else {
@@ -1855,18 +1820,17 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                 if (r->adr != NULL) {
                   if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
                     r->statuscode = STATUSCODE_INVALID;
-                    strcpybuff(r->msg, "Read error in cache disk data");
+                    strcpy(r->msg, "Read error in cache disk data");
                   }
                   r->adr[r->size] = '\0';
                 } else {
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpybuff(r->msg,
-                             "Read error (memory exhausted) from cache");
+                  strcpy(r->msg, "Read error (memory exhausted) from cache");
                 }
                 fclose(fp);
               } else {
                 r->statuscode = STATUSCODE_INVALID;
-                strcpybuff(r->msg, "Previous cache file not found (2)");
+                strcpy(r->msg, "Previous cache file not found (2)");
               }
             }
           } else {
@@ -1878,30 +1842,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
                   free(r->adr);
                   r->adr = NULL;
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpybuff(r->msg, "Cache Read Error : Read Data");
+                  strcpy(r->msg, "Cache Read Error : Read Data");
                 } else
                   r->adr[r->size] = '\0';
               } else {          // erreur
                 r->statuscode = STATUSCODE_INVALID;
-                strcpybuff(r->msg, "Cache Memory Error");
+                strcpy(r->msg, "Cache Memory Error");
               }
             }
           }
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpybuff(r->msg, "Cache Read Error : Bad Data");
+          strcpy(r->msg, "Cache Read Error : Bad Data");
         }
       } else {                  // erreur
         r->statuscode = STATUSCODE_INVALID;
-        strcpybuff(r->msg, "Cache Read Error : Read Header");
+        strcpy(r->msg, "Cache Read Error : Read Header");
       }
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpybuff(r->msg, "Cache Read Error : Seek Failed");
+      strcpy(r->msg, "Cache Read Error : Seek Failed");
     }
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpybuff(r->msg, "File Cache Entry Not Found");
+    strcpy(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -2164,15 +2128,12 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
   return 0;
 }
 
-/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
-#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)   \
-  do {                                                                         \
-    if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (refvalue)[0] = '\0';                                                    \
-      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
-      line[0] = '\0';                                                          \
-    }                                                                          \
-  } while (0)
+#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
+  if (line[0] != '\0' && strfield2(line, refline)) { \
+    strcpy(refvalue, value); \
+    line[0] = '\0'; \
+	} \
+} while(0)
 #define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
   if (line[0] != '\0' && strfield2(line, refline)) { \
     int intval = 0; \
@@ -2230,7 +2191,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
           }
           if ((pos = getArcField(index->line, 2)) != NULL) {
             r->msg[0] = '\0';
-            strncatbuff(r->msg, pos, sizeof(r->msg) - 1);
+            strncat(r->msg, pos, sizeof(pos) - 1);
           }
           while(linput(index->file, index->line, sizeof(index->line) - 1)
                 && index->line[0] != '\0') {
@@ -2241,16 +2202,11 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
               *value = '\0';
               for(value++; *value == ' ' || *value == '\t'; value++) ;
               HTTP_READFIELD_INT(line, value, "Content-Length", r->size);       // size
-              HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
-                                    sizeof(r->contenttype));
-              HTTP_READFIELD_STRING(line, value, "Last-Modified",
-                                    r->lastmodified, sizeof(r->lastmodified));
-              HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
-                                    sizeof(r->etag));
-              HTTP_READFIELD_STRING(line, value, "Location", r->location,
-                                    sizeof(location_default));
-              HTTP_READFIELD_STRING(line, value, "Content-Disposition",
-                                    r->cdispo, sizeof(r->cdispo));
+              HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype);       // contenttype
+              HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified);     // last-modified
+              HTTP_READFIELD_STRING(line, value, "Etag", r->etag);      // Etag
+              HTTP_READFIELD_STRING(line, value, "Location", r->location);      // 'location' pour moved
+              HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo);     // Content-disposition
               if (line[0] != '\0') {
                 int len = r->headers ? ((int) strlen(r->headers)) : 0;
                 int nlen =
@@ -2290,7 +2246,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
                 fetchSize = dataLength - metaSize;
               } else if (fetchSize > dataLength - metaSize) {
                 r->statuscode = STATUSCODE_INVALID;
-                strcpybuff(r->msg, "Cache Read Error : Truncated Data");
+                strcpy(r->msg, "Cache Read Error : Truncated Data");
               }
               r->size = 0;
               if (r->statuscode != STATUSCODE_INVALID) {
@@ -2303,13 +2259,12 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
                     int last_errno = errno;
 
                     r->statuscode = STATUSCODE_INVALID;
-                    PT_Element_failf(r, "Read error in cache disk data: %s",
-                                     strerror(last_errno));
+                    sprintf(r->msg, "Read error in cache disk data: %s",
+                            strerror(last_errno));
                   }
                 } else {
                   r->statuscode = STATUSCODE_INVALID;
-                  strcpybuff(r->msg,
-                             "Read error (memory exhausted) from cache");
+                  strcpy(r->msg, "Read error (memory exhausted) from cache");
                 }
               }
             }
@@ -2317,21 +2272,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
 
         } else {
           r->statuscode = STATUSCODE_INVALID;
-          strcpybuff(r->msg, "Cache Read Error : Read Header Error");
+          strcpy(r->msg, "Cache Read Error : Read Header Error");
         }
 
       } else {
         r->statuscode = STATUSCODE_INVALID;
-        strcpybuff(r->msg, "Cache Read Error : Read Header Error");
+        strcpy(r->msg, "Cache Read Error : Read Header Error");
       }
     } else {
       r->statuscode = STATUSCODE_INVALID;
-      strcpybuff(r->msg, "Cache Read Error : Seek Error");
+      strcpy(r->msg, "Cache Read Error : Seek Error");
     }
 
   } else {
     r->statuscode = STATUSCODE_INVALID;
-    strcpybuff(r->msg, "File Cache Entry Not Found");
+    strcpy(r->msg, "File Cache Entry Not Found");
   }
   if (r->location[0] != '\0') {
     r->location = strdup(r->location);
@@ -2380,18 +2335,8 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
   FILE *const fp = st->fp;
   struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
-  struct tm unknown_date;
   int size_headers;
 
-  /* a cached entry with no parseable Last-Modified must not take the writer
-     down; the epoch is the conventional "date unknown" */
-  if (tm == NULL) {
-    memset(&unknown_date, 0, sizeof(unknown_date));
-    unknown_date.tm_year = 70;
-    unknown_date.tm_mday = 1;
-    tm = &unknown_date;
-  }
-
   sprintf(st->headers,
           "HTTP/1.0 %d %s"
           "\r\n"

src/webhttrack.vcxproj

@@ -97,7 +97,6 @@
   <ItemGroup>
     <ClCompile Include="htsserver.c" />
     <ClCompile Include="htsweb.c" />
-    <ClCompile Include="htscmdline.c" />
     <ClCompile Include="htsurlport.c" />
   </ItemGroup>
 

tests/01_engine-cmdline-split.test

@@ -1,8 +0,0 @@
-#!/bin/bash
-#
-
-set -euo pipefail
-
-# webhttrack posts its command line as one string: the argv split must grow past
-# 1024 arguments and keep a quote inside a value out of the option parser.
-httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

tests/01_engine-growsize.test

@@ -1,25 +0,0 @@
-#!/bin/bash
-#
-# Buffer capacity for a 64-bit file size ('httrack -#test=growsize'): a -%S list
-# file past 4GB must size its buffer exactly or be refused, never wrap.
-
-set -euo pipefail
-
-out=$(httrack -#test=growsize)
-echo "$out"
-test "$out" == "growsize self-test OK"
-
-tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_growsize.XXXXXX")
-trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
-
-echo '<html><body>hi</body></html>' >"$tmp/index.html"
-printf -- '-*/zzmarker*\n' >"$tmp/rules.txt"
-
-# the rules file lands in the URL/filter string, echoed back by the banner
-run=$(httrack -O "$tmp/out" --quiet -n "-%S" "$tmp/rules.txt" \
-    "file://$tmp/index.html" 2>&1) || true
-printf '%s\n' "$run" | grep -q 'zzmarker' || {
-    echo "FAIL: -%S rules file was not loaded"
-    printf '%s\n' "$run"
-    exit 1
-}

tests/01_engine-strsafe.test

@@ -47,18 +47,3 @@ case "$err" in
     exit 1
     ;;
 esac
-
-# An array source with no NUL must abort on the source bound rather than run
-# off the array: the array half of the source-capacity selection.
-err=$(httrack -#test=strsafe overflow-src "x" 2>&1) || true
-case "$err" in
-*"strsafe: NOT aborted"*)
-    echo "unterminated source array was NOT caught" >&2
-    exit 1
-    ;;
-*"size < sizeof_source"*) ;;
-*)
-    echo "expected htssafe source-bound abort, got: $err" >&2
-    exit 1
-    ;;
-esac

tests/01_zlib-sitemap.test

@@ -1,10 +0,0 @@
-#!/bin/bash
-#
-# Sitemap parser self-test: <loc> extraction, entity decoding, URL and length
-# rejections, the URL cap, gzip framing and robots.txt Sitemap: records.
-
-set -euo pipefail
-
-out=$(httrack -O /dev/null '-#test=sitemap')
-echo "$out"
-test "$out" = "sitemap self-test OK"

tests/53_local-proxytrack-arc-reason.test

@@ -1,32 +0,0 @@
-#!/bin/bash
-# An ARC entry's HTTP reason phrase must survive a proxytrack --convert
-# round-trip; it used to be clipped to sizeof(char*) - 1 bytes.
-
-set -euo pipefail
-
-dir=$(mktemp -d)
-trap 'rm -rf "$dir"' EXIT
-
-printf 'HTTP/1.1 404 Not Found Here At All\r\nContent-Type: text/html\r\nLast-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
-printf 'hello' >"$dir/body"
-alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
-
-# ARC 1.0: filedesc record and version block, then per entry
-# <nl> <URL-record> <nl> <headers> <body>; the record's last field is that length.
-{
-    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
-    printf '2 0 test\n'
-    printf '\n\n'
-    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 404 - - 0 t.arc %d\n' "$alen"
-    cat "$dir/hdr" "$dir/body"
-} >"$dir/in.arc"
-
-proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1
-
-# HTTP/1.0 is the writer's own prefix (the input says 1.1), so a verbatim copy
-# of the input cannot satisfy this match.
-grep -aqF 'HTTP/1.0 404 Not Found Here At All' "$dir/out.arc" || {
-    echo "reason phrase lost in the ARC round-trip:" >&2
-    grep -a '^HTTP/' "$dir/out.arc" >&2 || echo "(no status line at all)" >&2
-    exit 1
-}

tests/77_webhttrack-redirect.test

@@ -146,13 +146,11 @@ sid=$(scrape_sid "${port}")
 resp=$(post_redirect '/foo
 X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
 stop
-# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
-# pipefail then turns the hit into a silent pass.
-grep -qi 'X-Injected' <<<"${resp}" &&
+printf '%s' "${resp}" | grep -qi 'X-Injected' &&
     fail "CRLF in the redirect value reached the response"
-test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
+test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
     fail "CRLF redirect still emitted a Location header"
-test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
+test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
     fail "CRLF redirect did not yield exactly one status line"
 
 # Loopback unless asked otherwise. Assert the socket, not the announcement.

tests/78_webhttrack-sid.test

@@ -103,10 +103,9 @@ sid=$(scrape_sid "${port}")
 test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
 
 # Accept: the legitimate flow still works. "redirect" is the cheapest field
-# with a reply that is visible in the headers. Matched from a here-string, not a
-# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
+# with a reply that is visible in the headers.
 resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
-grep -q '^Location: /accepted' <<<"${resp}" ||
+printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
     fail "a body carrying the correct sid was refused"
 
 # Refuse: missing and wrong. A missing one is the regression under test — the
@@ -115,12 +114,12 @@ grep -q '^Location: /accepted' <<<"${resp}" ||
 for bad in "redirect=/nosid" "sid=&redirect=/empty" \
     "sid=00000000000000000000000000000000&redirect=/wrong"; do
     resp=$(request "${port}" "${bad}")
-    grep -q '^Location:' <<<"${resp}" &&
+    printf '%s' "${resp}" | grep -q '^Location:' &&
         fail "body accepted without a valid sid: ${bad}"
     # A refusal has to be a well-formed reply, not a headerless fragment: the
     # release build used to emit only Content-length, which reads as a protocol
     # error to any client and hides the reason.
-    grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
+    printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
         fail "refusal was not a 403: ${bad}"
 done
 
@@ -128,23 +127,16 @@ done
 # applied to one global key store that later requests render from, and the
 # command dispatcher reads it from there. Probe the store itself — step3
 # interpolates ${projname} into its title — rather than the refused reply.
-store_page() {
-    # a dead probe or a non-page reply must not read as "the store was not written"
-    page=$(request "$1" "" step3) || fail "the key store probe failed"
-    grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
-        fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
-}
+title() { request "$1" "" step3; }
 
 request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
-store_page "${port}"
-grep -q 'UNAUTHWRITE' <<<"${page}" &&
+title "${port}" | grep -q 'UNAUTHWRITE' &&
     fail "a body without a valid sid was written to the key store"
 
 # Paired accept case: without it the assertion above passes even if the server
 # simply ignores every body, which would prove nothing.
 request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
-store_page "${port}"
-grep -q 'AUTHWRITE' <<<"${page}" ||
+title "${port}" | grep -q 'AUTHWRITE' ||
     fail "a body carrying the correct sid was not written to the key store"
 
 echo "PASS"

tests/79_local-proxytrack-webdav-mime.test

@@ -1,90 +0,0 @@
-#!/bin/bash
-#
-# proxytrack's WebDAV PROPFIND fallback must behave identically whether or not
-# the cache entry supplies Content-Type and Last-Modified.
-
-set -euo pipefail
-
-: "${top_srcdir:=..}"
-# shellcheck source=tests/testlib.sh
-. "$top_srcdir/tests/testlib.sh"
-
-python=$(find_python) || {
-    echo "python3 missing, skipping"
-    exit 77
-}
-command -v curl >/dev/null 2>&1 || {
-    echo "curl missing, skipping"
-    exit 77
-}
-# First test to run proxytrack as a live server, and MSYS cannot reap a native
-# listener: the orphan wedged the whole Windows suite for 48 minutes (#595).
-if is_windows; then
-    echo "windows: cannot reap a backgrounded proxytrack, skipping"
-    exit 77
-fi
-
-dir=$(mktemp -d)
-ptpid=
-cleanup() {
-    stop_server "$ptpid"
-    rm -rf "$dir"
-}
-trap cleanup EXIT
-
-# Neither header is present, so file->contenttype and ->lastmodified stay "".
-printf 'HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
-printf 'hello' >"$dir/body"
-alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
-{
-    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
-    printf '2 0 test\n'
-    printf '\n\n'
-    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
-    cat "$dir/hdr" "$dir/body"
-} >"$dir/in.arc"
-
-freeport() {
-    "$python" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'
-}
-proxyport=$(freeport)
-icpport=$(freeport)
-
-proxytrack "127.0.0.1:$proxyport" "127.0.0.1:$icpport" "$dir/in.arc" >"$dir/pt.log" 2>&1 &
-ptpid=$!
-waited=0
-until grep -qE "HTTP Proxy installed on|Unable to (initialize a temporary server|create the server)" "$dir/pt.log"; do
-    kill -0 "$ptpid" 2>/dev/null || {
-        echo "FAIL: proxytrack exited before listening"
-        cat "$dir/pt.log"
-        exit 1
-    }
-    test "$waited" -lt 50 || {
-        echo "FAIL: proxytrack never announced its listen port"
-        exit 1
-    }
-    sleep 0.1
-    waited=$((waited + 1))
-done
-grep -q "HTTP Proxy installed on" "$dir/pt.log" || {
-    echo "FAIL: proxytrack failed to bind"
-    cat "$dir/pt.log"
-    exit 1
-}
-
-# --max-time: an unbounded read here would wedge the runner rather than fail
-curl -s --max-time 30 -X PROPFIND -H "Depth: 1" \
-    "http://127.0.0.1:$proxyport/webdav/example.com/" >"$dir/resp.xml"
-
-grep -q '<getcontenttype>application/octet-stream</getcontenttype>' "$dir/resp.xml" || {
-    echo "FAIL: missing Content-Type did not fall back to application/octet-stream"
-    cat "$dir/resp.xml"
-    exit 1
-}
-grep -q '<getlastmodified>Wed, 01 Jan 2025 00:00:00 GMT</getlastmodified>' "$dir/resp.xml" || {
-    echo "FAIL: missing Last-Modified did not fall back to the index timestamp"
-    cat "$dir/resp.xml"
-    exit 1
-}
-
-echo "OK: WebDAV listing falls back correctly for an entry with no Content-Type/Last-Modified"

tests/80_engine-crash-symbolize.test

@@ -1,97 +0,0 @@
-#!/bin/bash
-# A fatal signal prints the raw backtrace, then names the frames
-# backtrace_symbols_fd() leaves as module+offset (-fvisibility=hidden hides them).
-set -eu
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-if is_windows; then
-    echo "no backtrace() on Windows; skipping" >&2
-    exit 77
-fi
-command -v httrack >/dev/null || {
-    echo "could not find httrack" >&2
-    exit 1
-}
-
-tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_symbolize.XXXXXX") || exit 1
-trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM
-out="${tmpdir}/trace"
-raw="${tmpdir}/trace-optout"
-overflow="this string is far too long for the buffer"
-# A frame glibc could not name. Stop at the offset: it prints the trailing
-# "[0xADDR]" with or without a leading space depending on its version.
-rawframe='(+0x[0-9a-f][0-9a-f]*)'
-
-# The strsafe selftest aborts on purpose, which lands in sig_fatal.
-rc=0
-run_with_timeout 60 httrack -#test=strsafe overflow "$overflow" >"$out" 2>&1 || rc=$?
-test "$rc" -ne 124 || {
-    echo "the crash handler did not finish within the deadline" >&2
-    exit 1
-}
-
-grep -q '^Caught signal ' "$out" || {
-    echo "no 'Caught signal' line:" >&2
-    cat "$out" >&2
-    exit 1
-}
-if grep -q 'No stack trace available on this OS' "$out"; then
-    echo "no backtrace() on this platform; skipping" >&2
-    exit 77
-fi
-# Unconditional and first: symbolizing must never cost the raw trace.
-grep -q "$rawframe" "$out" || {
-    echo "raw module+offset frames are gone:" >&2
-    cat "$out" >&2
-    exit 1
-}
-
-command -v addr2line >/dev/null || {
-    echo "addr2line not installed; skipping" >&2
-    exit 77
-}
-
-# Control: resolve the same frames ourselves, so a stripped build cannot turn
-# the assertion below into a vacuous pass.
-oracle="${tmpdir}/oracle"
-sed -n 's/^\([^(]*\)(+\(0x[0-9a-f]*\)).*$/\1 \2/p' "$out" |
-    while read -r mod off; do
-        test -f "$mod" || continue
-        addr2line -Cfip -e "$mod" "$off" 2>/dev/null || true
-    done >"$oracle"
-grep -qE 'st_strsafe|strcpy_safe_' "$oracle" || {
-    echo "addr2line names no hidden symbol in this build; skipping" >&2
-    exit 77
-}
-
-# The payload. Dropping the raw frames first is what makes it specific: both
-# names are static, so .dynsym could never have carried them. Not anchored on
-# the "0xOFF:" line, the inline chain puts the name on either half.
-grep -v "$rawframe" "$out" | grep -qE 'st_strsafe|strcpy_safe_' || {
-    echo "the handler did not name the hidden frames:" >&2
-    cat "$out" >&2
-    exit 1
-}
-
-# Opt-out: raw trace kept, nothing spawned.
-export HTTRACK_NO_SYMBOLIZE=1
-rc=0
-run_with_timeout 60 httrack -#test=strsafe overflow "$overflow" >"$raw" 2>&1 || rc=$?
-unset HTTRACK_NO_SYMBOLIZE
-test "$rc" -ne 124 || {
-    echo "the opt-out run did not finish within the deadline" >&2
-    exit 1
-}
-grep -q "$rawframe" "$raw" || {
-    echo "the opt-out lost the raw trace:" >&2
-    cat "$raw" >&2
-    exit 1
-}
-! grep -q '^0x[0-9a-f]*: ' "$raw" || {
-    echo "the opt-out still symbolized:" >&2
-    cat "$raw" >&2
-    exit 1
-}

tests/81_webhttrack-maxsize.test

@@ -1,142 +0,0 @@
-#!/bin/bash
-#
-# The wizard's size fields must render as distinct caps: sizemax is -M, othermax
-# then maxhtml share -m.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-python=$(find_python) || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_maxsize.XXXXXX") || fail "no tmpdir"
-srvlog=$(mktemp)
-srv=
-cleanup() {
-    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
-    srv=
-    rm -rf "${work}" "${srvlog}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
-# ~/.httrack.ini out of it.
-sport=$("${python}" -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()')
-(
-    trap '' TERM TTOU
-    export HOME="${work}"
-    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
-) &
-srv=$!
-for _ in $(seq 1 40); do
-    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
-    kill -0 "${srv}" 2>/dev/null || break
-    sleep 0.25
-done
-test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
-
-# Audit what step4.html renders back; without command_do nothing is launched.
-"${python}" - "${url}" <<'PY' || fail "wizard rendered the wrong options (see above)"
-import re, sys, urllib.parse, urllib.request
-
-url = sys.argv[1]
-html, other, site = "111111", "222222", "333333"
-
-
-def post(fields):
-    # The body is refused without the session id the server puts in each form.
-    form = urllib.request.urlopen(url + "server/index.html", timeout=20).read()
-    m = re.search(rb'name="sid" value="([0-9a-f]+)"', form)
-    if not m:
-        raise SystemExit("no session id in server/index.html")
-    fields = [("sid", m.group(1).decode())] + fields
-    body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in fields)
-    req = urllib.request.Request(url + "server/step4.html",
-                                 data=body.encode("latin-1"), method="POST")
-    return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
-
-
-def textarea(page, name):
-    m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
-    if m is None:
-        sys.exit("no %s textarea in the rendered step4.html" % name)
-    return m.group(1)
-
-
-def want(ok, msg, ctx):
-    if not ok:
-        sys.exit("%s\nrendered:%s" % (msg, ctx))
-
-
-page = post([("maxhtml", html), ("othermax", other), ("sizemax", site),
-             ("dos", "2")])
-cmd = textarea(page, "command")
-want("--max-size=" + site in cmd, "site cap not emitted as --max-size", cmd)
-want("--max-files=" + site not in cmd, "site cap still an --max-files", cmd)
-# Positive controls: both per-file caps still ride -m, one option each.
-want("--max-files=" + other in cmd, "non-HTML cap not emitted as --max-files",
-     cmd)
-want("--max-files=," + html in cmd, "HTML cap not emitted as --max-files=,",
-     cmd)
-want(cmd.count("--max-files=") == 2, "unexpected --max-files count", cmd)
-want(cmd.index("--max-files=" + other) < cmd.index("--max-files=," + html),
-     "the bare --max-files must precede the --max-files=, form", cmd)
-
-# winprofile.ini feeds the same three caps to the Windows GUI, one key each.
-ini = textarea(page, "winprofile").replace("\r\n", "\n")  # the ini is CRLF
-want("\nDos=2" in ini, "Dos not written from the dos field", ini)
-want("\nMaxHtml=" + html + "\n" in ini, "MaxHtml not written from maxhtml", ini)
-want("\nMaxOther=" + other + "\n" in ini, "MaxOther not written from othermax",
-     ini)
-want("\nMaxAll=" + site + "\n" in ini, "MaxAll not written from sizemax", ini)
-
-# A bare --max-files= (an unguarded empty field) parses as -m with no digits,
-# silently clearing the html cap.
-page = post([("maxhtml", ""), ("othermax", ""), ("sizemax", ""), ("dos", "2")])
-cmd = textarea(page, "command")
-want("--max-rate=" in cmd, "empty size fields rendered no command line", cmd)
-want("--max-files" not in cmd and "--max-size" not in cmd,
-     "empty size fields still rendered a size option", cmd)
-
-# A mistyped ${LANG_OK] key renders the OK button's label empty.
-opts = urllib.request.urlopen(url + "server/option2b.html",
-                              timeout=20).read().decode("latin-1")
-want("LANG_OK" not in opts, "unexpanded LANG_OK in option2b.html", "")
-want('<input type="submit" value="OK"' in opts, "no OK label in option2b.html",
-     "")
-PY
-
-cleanup
-
-# A bare -m<n> resets the html limit, so only the bare-then-comma order keeps
-# both caps live. basic.html is 487 bytes, well past the 10-byte html cap.
-crawl() {
-    bash "${distdir}/tests/local-crawl.sh" "$@"
-}
-crawl --errors 1 --log-found 'File too big' \
-    httrack 'BASEURL/simple/basic.html' '--max-files=,10'
-crawl --errors 1 --log-found 'File too big' \
-    httrack 'BASEURL/simple/basic.html' '--max-files=500000' '--max-files=,10'
-crawl --errors 0 --found 'simple/basic.html' --log-not-found 'File too big' \
-    httrack 'BASEURL/simple/basic.html' '--max-files=,10' '--max-files=500000'
-
-echo "PASS"

tests/82_webhttrack-browse-links.test

@@ -1,135 +0,0 @@
-#!/bin/bash
-#
-# A browser will not follow an http: page to a file: URL, so the GUI's "browse
-# the mirror" link has to go through the server's own /website/ route.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-python=$(find_python) || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-# Control: on a path that does not resolve, grep "passes" without reading anything.
-test -f "${distdir}/html/server/finished.html" || fail "no GUI pages under ${distdir}"
-bad=$(grep -l 'file://' "${distdir}"/html/server/finished.html || true)
-test -z "${bad}" || fail "file: link left in: ${bad}"
-
-work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_browse.XXXXXX") || fail "no tmpdir"
-srvlog=$(mktemp)
-srv=
-srvpid=
-cleanup() {
-    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
-    test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
-    rm -rf "${work}" "${srvlog}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-# Stand in for a finished mirror: the crawl itself is not under test.
-proj="${work}/websites/proj"
-mkdir -p "${proj}"
-printf 'MIRROR-PROBE-OK\n' >"${proj}/probe.txt"
-printf '<html><body>MIRROR-INDEX-OK</body></html>\n' >"${proj}/index.html"
-
-# An isolated HOME keeps a stray ~/.httrack.ini out of the server's settings.
-sport=$("${python}" -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()')
-(
-    trap '' TERM TTOU
-    export HOME="${work}"
-    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
-) &
-srv=$!
-for _ in $(seq 1 40); do
-    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
-    kill -0 "${srv}" 2>/dev/null || break
-    sleep 0.25
-done
-test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
-srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
-
-# htsserver resolves the posted paths itself, so hand it native ones.
-"${python}" - "${url}" "$(nativepath "${work}/websites")" <<'PY' || fail "browse-link checks failed"
-import re, sys, urllib.error, urllib.parse, urllib.request
-
-url, base = sys.argv[1].rstrip("/"), sys.argv[2]
-rc = 0
-
-
-def check(ok, what):
-    global rc
-    print(("ok: " if ok else "FAIL: ") + what)
-    if not ok:
-        rc = 1
-
-
-def get(path):
-    # The UI is served ISO-8859-1, so stay in bytes.
-    return urllib.request.urlopen(url + path, timeout=20).read()
-
-
-# No crawl has run, so no commandRunning/commandEnd override swaps the page out.
-finished = get("/server/finished.html").decode("latin-1")
-# Control: an empty or unrendered page would pass "no file:" on its own.
-check("HTTrack Website Copier" in finished, "finished.html rendered")
-check("file://" not in finished, "finished.html has no file: link")
-
-# Control: /website/ 404s until a project is set, so the fetches below are real.
-try:
-    get("/website/probe.txt")
-    check(False, "/website/ served with no project set")
-except urllib.error.HTTPError as e:
-    check(e.code == 404, "/website/ is bound to a project (got %d)" % e.code)
-
-# Point the server at the mirror the way step4.html does; a body without the
-# session id is refused outright. Only a profile save records the served root,
-# so post one; command_do=save runs no crawl.
-sid = re.search(r'name="sid" value="([0-9a-f]+)"', finished)
-if sid is None:
-    print("FAIL: no sid in the rendered form")
-    sys.exit(1)
-fields = [("sid", sid.group(1)), ("path", base), ("projname", "proj"),
-          ("command", "httrack"), ("command_do", "save"), ("winprofile", "x")]
-body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="")) for k, v in fields)
-urllib.request.urlopen(urllib.request.Request(
-    url + "/server/step4.html", data=body.encode("latin-1"), method="POST"),
-    timeout=20).read()
-
-check(get("/website/probe.txt") == b"MIRROR-PROBE-OK\n", "mirror file over http")
-check("MIRROR-INDEX-OK" in get("/website/index.html").decode("latin-1"),
-      "mirror index over http")
-
-body = get("/server/finished.html").decode("latin-1")
-# Match the anchor by its mirror-path label: the list below it links /website/
-# too, so a bare "is the route mentioned" check passes on the unfixed page.
-check(re.search(r'href="/website/index\.html"[^>]*>\s*' +
-                re.escape(base + "/proj"), body) is not None,
-      "the mirror-path link points at the served mirror")
-check("file://" not in body, "finished.html has no file: link")
-
-sys.exit(rc)
-PY
-
-# A leaked htsserver wedges the parallel harness behind a green log.
-cleanup
-! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
-
-echo "PASS"

tests/83_webhttrack-argescape.test

@@ -1,148 +0,0 @@
-#!/bin/bash
-#
-# The wizard renders its httrack command line as one string, which the engine
-# splits back into argv: a double quote in a field must reach the split escaped,
-# or the rest of the value is parsed as fresh options (-V runs a shell command).
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-command -v python3 >/dev/null || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-srv=
-log=$(mktemp)
-cleanup() {
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    rm -f "${log}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-freeport() {
-    python3 -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()'
-}
-
-# Echo the announced URL. Runs in a command substitution, so it is a
-# subshell and cannot export the pid: the caller reads it back with srvpid.
-start() {
-    local port url
-    port=$(freeport)
-    : >"${log}"
-    (
-        trap '' TERM TTOU
-        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
-    ) &
-    for _ in $(seq 1 40); do
-        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
-        sleep 0.25
-    done
-    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
-    echo "${url}"
-}
-
-srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
-
-portof() { echo "${1##*:}" | tr -d /; }
-
-# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
-# default index). Prints the reply.
-request() {
-    python3 -c 'import socket, sys
-port, body = int(sys.argv[1]), sys.argv[2]
-page = sys.argv[3] if len(sys.argv) > 3 else "index"
-if body:
-    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
-           "Content-type: application/x-www-form-urlencoded\r\n"
-           "Content-length: %d\r\n\r\n%s" % (len(body), body))
-else:
-    req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
-s = socket.create_connection(("127.0.0.1", port), 10)
-s.settimeout(30)
-s.sendall(req.encode())
-out = b""
-while True:
-    b = s.recv(65536)
-    if not b:
-        break
-    out += b
-s.close()
-sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
-}
-
-scrape_sid() {
-    request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
-}
-
-# urlencode the key=value pairs given as arguments
-formencode() {
-    python3 -c 'import sys, urllib.parse
-print(urllib.parse.urlencode([tuple(a.split("=", 1)) for a in sys.argv[1:]]))' "$@"
-}
-
-url=$(start)
-port=$(portof "${url}")
-srv=$(srvpid)
-test -n "${srv}" || fail "htsserver did not report its pid"
-
-sid=$(scrape_sid "${port}")
-test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
-
-# Fill the wizard fields the command line quotes, then read back the generated
-# command line: the user-agent carries a break-out attempt, the footer a
-# backslash (which must survive the escape round trip), the project name a plain
-# value.
-body=$(formencode "sid=${sid}" 'user=Moz" -V "touch /tmp/pwn' 'footer=a\b"c' \
-    "path=/tmp/p" "projname=plain proj" 'urls=http://x/a"b' 'url2=+*.png"')
-request "${port}" "${body}" >/dev/null
-cmdline=$(request "${port}" "" step4 |
-    sed -n '/<textarea name="command"/,/<\/textarea>/p')
-
-# Control: without the fields in the page every assertion below is vacuous.
-grep -q -- '--user-agent' <<<"${cmdline}" ||
-    fail "no --user-agent in the generated command line (probe blind)"
-
-# A plain value is passed through untouched: the escaping must not mangle the
-# ordinary case.
-grep -qF -- '--path "/tmp/p/plain proj"' <<<"${cmdline}" ||
-    fail "a plain quoted value was not passed through: ${cmdline}"
-
-# The quote is escaped, so the split keeps it inside the value...
-grep -qF -- '--user-agent "Moz\" -V \"touch /tmp/pwn"' <<<"${cmdline}" ||
-    fail "the quote in the user-agent was not escaped: ${cmdline}"
-
-# ...and the raw form, which would end the argument and hand -V to the option
-# parser, is gone.
-grep -qF -- '--user-agent "Moz" -V "touch' <<<"${cmdline}" &&
-    fail "the user-agent still closes its argument early: ${cmdline}"
-
-# A backslash is escaped too, or the split would eat it along with the quote
-# that follows.
-grep -qF -- '--footer "a\\b\"c"' <<<"${cmdline}" ||
-    fail "the backslash in the footer was not escaped: ${cmdline}"
-
-# The url and filter fields sit outside quotes, where a backslash cannot escape
-# anything: one quote there flips the parity of every quote after it, so the
-# escaping above would protect nothing. They must not emit a raw quote at all.
-grep -qF -- 'http://x/a%22b' <<<"${cmdline}" ||
-    fail "the quote in the url field was not neutralised: ${cmdline}"
-grep -qF -- '+*.png%22' <<<"${cmdline}" ||
-    fail "the quote in the filter field was not neutralised: ${cmdline}"
-grep -qF -- 'http://x/a"b' <<<"${cmdline}" &&
-    fail "the url field still emits a raw quote: ${cmdline}"
-
-echo "PASS"

tests/84_webhttrack-mirror-verbatim.test

@@ -1,167 +0,0 @@
-#!/bin/bash
-#
-# htsserver serves the crawled mirror under /website/ alongside its own GUI:
-# mirrored pages must skip the ${...} expander, or ${_sid} leaks the session id.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-command -v python3 >/dev/null || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-log=$(mktemp)
-work=$(mktemp -d)
-csrv=
-# start() runs in a command substitution, so its $! never reaches this shell. A
-# missed kill leaves an orphan holding the CI job open long after a green suite.
-srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
-cleanup() {
-    local pid
-    pid=$(srvpid)
-    test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
-    test -z "${csrv}" || kill -9 "${csrv}" 2>/dev/null || true
-    wait "${csrv}" 2>/dev/null || true # absorb bash's async "Killed" notice
-    rm -rf "${log}" "${work}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-freeport() {
-    python3 -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()'
-}
-
-# Echo the announced URL.
-start() {
-    local port url
-    port=$(freeport)
-    : >"${log}"
-    (
-        trap '' TERM TTOU
-        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
-    ) &
-    for _ in $(seq 1 40); do
-        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
-        sleep 0.25
-    done
-    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
-    echo "${url}"
-}
-
-portof() { echo "${1##*:}" | tr -d /; }
-
-# GET $2 from 127.0.0.1:$1, headers into $3 and the body, byte for byte, into $4.
-fetch() {
-    python3 -c 'import socket, sys
-s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
-s.settimeout(20)
-s.sendall(("GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % sys.argv[2]).encode())
-out = b""
-while True:
-    b = s.recv(65536)
-    if not b:
-        break
-    out += b
-s.close()
-head, _, body = out.partition(b"\r\n\r\n")
-open(sys.argv[3], "wb").write(head)
-open(sys.argv[4], "wb").write(body)' "$1" "$2" "$3" "$4"
-}
-
-# POST the remaining args as urlencoded key=value fields to 127.0.0.1:$1.
-post() {
-    local port=$1
-    shift
-    python3 -c 'import socket, sys, urllib.parse
-body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe=""))
-                for k, v in (a.split("=", 1) for a in sys.argv[2:])).encode()
-s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
-s.settimeout(30)
-s.sendall(b"POST /step4.html HTTP/1.0\r\nHost: 127.0.0.1\r\n"
-          b"Content-type: application/x-www-form-urlencoded\r\n"
-          b"Content-length: %d\r\n\r\n" % len(body) + body)
-while s.recv(65536):
-    pass
-s.close()' "${port}" "$@" >/dev/null
-}
-
-# LF-only, and a line ending in a backslash: the expander rewrites both, so a
-# mangled reply fails the byte comparison even where no directive is present.
-mirror="${work}/proj/hostile.html"
-hdr="${work}/hdr"
-body="${work}/body"
-
-# The server merges $HOME/.httrack.ini into the same store on the first request;
-# point it somewhere empty so a developer's own file cannot shadow the fields.
-export HOME="${work}"
-
-url=$(start)
-port=$(portof "${url}")
-
-# Positive control: the GUI's own templates must still expand. This is also
-# where the real token comes from, so its absence below can be asserted.
-fetch "${port}" /server/index.html "${hdr}" "${body}"
-sid=$(sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' "${body}" | head -1)
-test "${#sid}" -eq 32 || fail "GUI page did not expand \${sid} (got '${sid}')"
-
-# /website/ serves only the root the server itself recorded, so save a profile
-# (no command_do=start, so nothing crawls) to create it, then plant the file.
-post "${port}" "sid=${sid}" command=httrack command_do=save winprofile=x \
-    "path=${work}" projname=proj
-test -f "${work}/proj/hts-cache/winprofile.ini" ||
-    fail "profile save did not create the project: $(cat "${log}")"
-mkdir -p "$(dirname "${mirror}")"
-# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
-printf 'Hostile mirrored page.\nsid=${_sid} copy=${sid}\ntrailing backslash: \\\n' \
-    >"${mirror}"
-
-fetch "${port}" /website/hostile.html "${hdr}" "${body}"
-grep -q '^HTTP/1\.0 200 ' "${hdr}" || fail "mirrored page not served: $(head -1 "${hdr}")"
-grep -qF "${sid}" "${body}" &&
-    fail "the session id was expanded into mirrored content"
-# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
-grep -qF '${_sid}' "${body}" || fail "\${_sid} did not survive verbatim"
-# shellcheck disable=SC2016
-grep -qF '${sid}' "${body}" || fail "\${sid} did not survive verbatim"
-cmp -s "${mirror}" "${body}" || fail "mirrored file not served byte for byte"
-# The mirror stays browsable: verbatim must not mean served as a download.
-grep -qi '^Content-type: text/html' "${hdr}" ||
-    fail "mirrored page lost its text/html type: $(cat "${hdr}")"
-
-# The other direction: while a crawl runs, every .html request is overridden to
-# the GUI's own refresh page, so a /website/ URL stops naming mirrored content.
-# /trickle/ dribbles for a minute, which holds the crawl open for the probe.
-clog="${work}/content.log"
-python3 "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
-csrv=$!
-for _ in $(seq 1 40); do
-    cport=$(sed -n 's/^PORT //p' "${clog}") && test -n "${cport}" && break
-    kill -0 "${csrv}" 2>/dev/null || break
-    sleep 0.25
-done
-test -n "${cport:-}" || fail "content server did not come up: $(cat "${clog}")"
-
-post "${port}" "sid=${sid}" "path=${work}" projname=crawl winprofile=x \
-    command_do=start \
-    "command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/trickle/ -O ${work}/crawl"
-
-fetch "${port}" /website/hostile.html "${hdr}" "${body}"
-grep -q '^HTTP/1\.0 200 ' "${hdr}" ||
-    fail "running crawl: /website/ was not overridden to the GUI page: $(head -1 "${hdr}")"
-grep -qF "'crawl' - HTTrack Website Copier" "${body}" ||
-    fail "the overridden GUI page was not the expanded refresh page"
-
-echo "PASS"

tests/85_webhttrack-projpath.test

@@ -1,183 +0,0 @@
-#!/bin/bash
-#
-# /website/ is served from the project directory htsserver set up itself, never
-# from a root the request body names, and composing that path must stay bounded.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-command -v python3 >/dev/null || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-srv=
-log=$(mktemp)
-base=$(mktemp -d)
-cleanup() {
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    rm -f "${log}"
-    rm -rf "${base}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-freeport() {
-    python3 -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()'
-}
-
-# Echo the announced URL. Runs in a command substitution, so it is a
-# subshell and cannot export the pid: the caller reads it back with srvpid.
-start() {
-    local port url
-    port=$(freeport)
-    : >"${log}"
-    (
-        trap '' TERM TTOU
-        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
-    ) &
-    for _ in $(seq 1 40); do
-        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
-        sleep 0.25
-    done
-    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
-    echo "${url}"
-}
-
-# The server reports its own pid; the aliveness assertion below hangs off it.
-srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
-
-alive() { kill -0 "$1" 2>/dev/null; }
-
-portof() { echo "${1##*:}" | tr -d /; }
-
-# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
-# $2 is empty. Prints the reply.
-request() {
-    python3 -c 'import socket, sys
-port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
-if path:
-    req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
-else:
-    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
-           "Content-type: application/x-www-form-urlencoded\r\n"
-           "Content-length: %d\r\n\r\n%s" % (len(body), body))
-s = socket.create_connection(("127.0.0.1", port), 10)
-s.settimeout(30)
-s.sendall(req.encode())
-out = b""
-while True:
-    b = s.recv(65536)
-    if not b:
-        break
-    out += b
-s.close()
-sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
-}
-
-get() { request "$1" "$2" ""; }
-post() { request "$1" "" "$2"; }
-
-# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
-# reads as marker-absent, and so do a truncated body and a 302 to the file.
-reply=
-fetch() {
-    reply=$(get "$1" "$2") || fail "the request for $2 failed"
-    grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
-        fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
-}
-
-url=$(start)
-port=$(portof "${url}")
-srv=$(srvpid)
-test -n "${srv}" || fail "htsserver did not report its pid"
-
-# Every request body is gated by the session id (78_webhttrack-sid.test).
-sid=$(get "${port}" /server/index.html |
-    sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
-test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
-
-# A file the mirror must never expose, next to the project that may.
-echo "SECRETMARKER" >"${base}/secret.txt"
-mkdir -p "${base}/proj"
-echo "LOGMARKER" >"${base}/proj/hts-log.txt"
-
-# error_redirect is the branch that skipped the fsfile clearing, so fail the save
-# with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
-# any successful save: commandEnd then swaps the error page for the finished one.
-get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
-saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
-grep -q '^Location: /server/error.html' <<<"${saved}" ||
-    fail "the refused save did not redirect to the error page"
-
-# No project yet, so no root to serve from.
-post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
-fetch "${port}" /website/secret.txt 404
-grep -q SECRETMARKER <<<"${reply}" &&
-    fail "a posted projpath served a file outside any project"
-
-# Positive control: step4's "save settings" flow registers the project without
-# crawling, and browsing its mirror keeps working. Without it the assertions
-# around it would pass on a server that never serves /website/ at all.
-body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
-body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
-post "${port}" "${body}" >/dev/null
-test -f "${base}/proj/hts-cache/winprofile.ini" ||
-    fail "the project was not registered: $(cat "${log}")"
-fetch "${port}" /website/hts-log.txt 200
-grep -q LOGMARKER <<<"${reply}" ||
-    fail "the registered project's mirror is not served"
-
-# Same request with the root repointed: the project is legitimate, projpath is
-# not what decides where the bytes come from.
-post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
-fetch "${port}" /website/secret.txt 404
-grep -q SECRETMARKER <<<"${reply}" &&
-    fail "a posted projpath repointed the served root"
-post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
-fetch "${port}" /website/passwd 404
-grep -q '^root:' <<<"${reply}" &&
-    fail "a posted projpath read an arbitrary system file"
-
-# A ".." anywhere in the recorded root would escape the mirror on every later
-# request, so the save must be refused and the previous root kept.
-post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
-fetch "${port}" /website/secret.txt 404
-grep -q SECRETMARKER <<<"${reply}" &&
-    fail "a '..' in the saved project path escaped the mirror root"
-fetch "${port}" /website/hts-log.txt 200
-grep -q LOGMARKER <<<"${reply}" ||
-    fail "rejecting the '..' root also lost the previous one"
-
-# The root is now the server's own, but it is still built from two posted
-# fields: 800-odd bytes of them used to be sprintf'd into a 1024-byte buffer.
-seg=$(printf '%0200d' 0)
-longpath="${base}/${seg}/${seg}/${seg}/${seg}"
-fspath="${longpath}/proj"
-# structcheck() refuses a root over HTS_URLMAXSIZE, so the URL carries the rest.
-test "$((${#fspath} + 11))" -le 1024 ||
-    fail "the long project path (${#fspath}) would not pass structcheck"
-longurl=$(printf '%0800d' 0)
-test "$((${#fspath} + 1 + ${#longurl}))" -gt 1024 ||
-    fail "the composed path (${#fspath} + ${#longurl}) would not overflow"
-post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${longpath}&projname=proj" >/dev/null
-test -f "${fspath}/hts-cache/winprofile.ini" ||
-    fail "the long-path project was not registered: $(cat "${log}")"
-get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
-alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
-# Not just alive: still answering.
-fetch "${port}" /server/index.html 200
-
-echo "PASS"

tests/86_local-proxytrack-cache-longfields.test

@@ -1,111 +0,0 @@
-#!/bin/bash
-# A cached header longer than proxytrack's field must be clipped to that field,
-# not overflow into the next one (contenttype[64] abuts the location pointer).
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-dir=$(mktemp -d)
-trap 'rm -rf "$dir"' EXIT
-
-pad() { printf "%${1}s" '' | tr ' ' "$2"; }
-
-# Longest surviving run of char $2 in file $1, or 0.
-runlen() {
-    grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
-}
-
-# --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
-# Each header overshoots its own destination, so a per-field clip is the only
-# way to get every expected length right at once.
-{
-    printf 'HTTP/1.1 200 OK\r\n'
-    printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)"
-    printf 'Etag: %s\r\n' "$(pad 400 E)"
-    printf 'Content-Disposition: %s\r\n' "$(pad 400 D)"
-    printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
-    printf 'Content-Length: 5\r\n\r\n'
-} >"$dir/hdr"
-printf 'hello' >"$dir/body"
-alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
-{
-    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
-    printf '2 0 test\n'
-    printf '\n\n'
-    printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
-    cat "$dir/hdr" "$dir/body"
-} >"$dir/in.arc"
-
-proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
-    echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2
-    exit 1
-}
-test -s "$dir/out.arc" || {
-    echo "FAIL: ARC entry dropped instead of clipped" >&2
-    exit 1
-}
-# Only contenttype survives into the ARC output; the over-long Etag and
-# Content-Disposition above are still load-bearing, since bounding just one
-# field leaves the others smashing the element (the run above would crash).
-got=$(runlen "$dir/out.arc" A)
-test "${got:-0}" = 54 || {
-    echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2
-    exit 1
-}
-
-# --- ZIP reader (ZIP_READFIELD_STRING) --------------------------------------
-python=$(find_python) || {
-    echo "python3 missing; ARC half only" >&2
-    exit 0
-}
-
-"$python" - "$dir/in.zip" <<'EOF'
-import sys, zipfile
-
-body = b"hello world"
-meta = (
-    "X-In-Cache: 1\r\n"
-    "X-StatusCode: 200\r\n"
-    "X-StatusMessage: ok%s\r\n"
-    "X-Size: %d\r\n"
-    "Content-Type: text/html%s\r\n"
-    "X-Charset: %s\r\n"
-    "Etag: %s\r\n"
-    "Content-Disposition: %s\r\n"
-    # a parseable date is required: the ARC writer dereferences it unchecked
-    "Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
-    "X-Save: out/page.html\r\n"
-    % ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400)
-).encode()
-
-zi = zipfile.ZipInfo("example.com/page.html")
-zi.compress_type = zipfile.ZIP_STORED
-zi.extra = meta
-with zipfile.ZipFile(sys.argv[1], "w") as z:
-    z.writestr(zi, body)
-EOF
-
-proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || {
-    echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2
-    exit 1
-}
-test -s "$dir/out2.arc" || {
-    echo "FAIL: zip entry dropped instead of clipped" >&2
-    exit 1
-}
-# three destinations, two capacities: msg[1024] keeps all 400 M's where a
-# one-size-fits-all clip would cut it to 63 like the others
-while read -r ch want; do
-    got=$(runlen "$dir/out2.arc" "$ch")
-    test "${got:-0}" = "$want" || {
-        echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2
-        exit 1
-    }
-done <<'EOF'
-A 54
-C 63
-M 400
-EOF

tests/87_local-proxytrack-nodate.test

@@ -1,51 +0,0 @@
-#!/bin/bash
-# A cached entry whose Last-Modified is missing or unparseable must still
-# convert: the ARC writer used to dereference the parsed date unchecked.
-
-set -euo pipefail
-
-dir=$(mktemp -d)
-trap 'rm -rf "$dir"' EXIT
-
-# $1 Last-Modified value (empty = omit the header), $2 expected archive date,
-# $3 label. The date is asserted exactly: a guard that fires unconditionally
-# would clobber the valid case, and one that skips the year/day would emit a
-# month and day of 00.
-run_case() {
-    lastmod=$1
-    wantdate=$2
-    what=$3
-
-    {
-        printf 'HTTP/1.1 200 OK\r\n'
-        printf 'Content-Type: text/html\r\n'
-        test -z "$lastmod" || printf 'Last-Modified: %s\r\n' "$lastmod"
-        printf 'Content-Length: 5\r\n\r\n'
-    } >"$dir/hdr"
-    printf 'hello' >"$dir/body"
-    alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
-    {
-        printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
-        printf '2 0 test\n'
-        printf '\n\n'
-        printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
-        cat "$dir/hdr" "$dir/body"
-    } >"$dir/in.arc"
-
-    proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
-        echo "FAIL: proxytrack crashed on $what" >&2
-        exit 1
-    }
-    grep -aq "^http://example.com/p.html 0.0.0.0 ${wantdate} " "$dir/out.arc" || {
-        echo "FAIL: $what: expected archive date ${wantdate}, got:" >&2
-        grep -a '^http://example.com/' "$dir/out.arc" >&2 || echo "(entry dropped)" >&2
-        exit 1
-    }
-}
-
-# the epoch stands in for "date unknown"
-run_case '' 19700101000000 'a missing Last-Modified'
-run_case 'not a date at all' 19700101000000 'an unparseable Last-Modified'
-run_case '0' 19700101000000 'a bare 0 Last-Modified'
-# a parseable date must still come through untouched
-run_case 'Sun, 06 Nov 1994 08:49:37 GMT' 19941106084937 'a valid Last-Modified'

tests/89_local-sitemap.test

@@ -1,134 +0,0 @@
-#!/bin/bash
-#
-# --sitemap seeds the crawl from robots.txt -> sitemapindex -> gzipped urlset.
-# start.html links to nothing, so orphan*.html can only arrive through the
-# sitemap; deep1.html proves the seeds keep a full depth budget under -r2, and
-# the off-host page and child sitemap must both be refused.
-
-set -eu
-
-: "${top_srcdir:=..}"
-
-crawl() { bash "$top_srcdir/tests/local-crawl.sh" "$@"; }
-
-# robots.txt Sitemap: -> index -> .xml.gz, seeds behaving like -r2 seeds. The
-# log assertions pin which route was taken: the two documents are served from
-# both /sitemapdir/index.xml and the well-known /sitemap.xml, so "some sitemap
-# was read" would pass either way. --not-found pins ingestion-only: the sitemap
-# documents feed the crawl but never land in the mirror.
-# --rerun also walks the update path over a mirror holding sitemap seeds.
-crawl --errors 0 --rerun \
-    --found 'sitemapdir/orphan1.html' \
-    --found 'sitemapdir/orphan2.html' \
-    --found 'sitemapdir/deep1.html' \
-    --not-found 'sitemapdir/index.xml' \
-    --not-found 'sitemapdir/pages.xml.gz' \
-    --log-found '2 of 3 URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
-    --log-found '1 of 2 child sitemap\(s\) listed by 127\.0\.0\.1:[0-9]+/sitemapdir/index\.xml' \
-    --log-found 'ignoring off-host child sitemap' \
-    --log-not-found '/sitemap\.xml' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2
-
-# Negative control: without the option nothing but the start page is reached.
-crawl --errors 0 \
-    --found 'sitemapdir/start.html' \
-    --not-found 'sitemapdir/orphan1.html' \
-    --not-found 'sitemapdir/orphan2.html' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2
-
-# Sitemap URLs are not a filter bypass: a -*orphan2* rule still rejects one.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --not-found 'sitemapdir/orphan2.html' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 '-*orphan2*'
-
-# A robots.txt naming no sitemap falls back to the well-known /sitemap.xml.
-# The test server drops its Sitemap: record for this User-Agent.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-found 'listed by 127\.0\.0\.1:[0-9]+/sitemap\.xml' \
-    --log-not-found 'sitemapdir/index\.xml' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -F 'nositemap-agent'
-
-# --sitemap-url alone reads the named document and probes nothing.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --found 'sitemapdir/orphan2.html' \
-    --log-not-found 'sitemapdir/index\.xml' \
-    --log-not-found '/sitemap\.xml' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/pages.xml.gz'
-
-# The two options are additive: an unfetchable --sitemap-url is parsed as an
-# empty document and does not stop --sitemap from seeding the crawl.
-crawl --errors-content 1 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-found 'Sitemap: 0 of 0 URL\(s\) added' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/missing.xml'
-
-# The sitemapindex nesting cap stops the chain before its deepest urlset. The
-# positive control is the page one level inside the cap: without it, a cap
-# mutated to 0 (nothing followed) would pass this just as well.
-crawl --errors 0 \
-    --found 'sitemapdir/cap3.html' \
-    --not-found 'sitemapdir/cap4.html' \
-    --log-found 'Sitemap: cap reached' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/chain0.xml'
-
-# A site cannot widen a subtree crawl by putting its sitemap at the root: the
-# page below the start directory is seeded, the one above it is refused.
-crawl --errors 0 \
-    --found 'deep/dir/below.html' \
-    --not-found 'elsewhere/updir.html' \
-    httrack 'BASEURL/deep/dir/start.html' --sitemap -r3 -F 'scopesitemap-agent'
-
-# How far a sitemap fetch is gated depends on who asked for it, so the three
-# cases below have to differ: one "robots is honoured" assertion would hide a
-# regression in either direction. The harness disables robots, hence -s2 here.
-
-# We guessed /sitemap.xml, so a Disallow on it wins.
-crawl --errors 0 \
-    --not-found 'sitemapdir/orphan1.html' \
-    --log-found 'Sitemap: robots.txt forbids' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
-    -F 'denysitemap-agent'
-
-# The site declared it through a Sitemap: line, which the same Disallow does
-# not retract.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-not-found 'Sitemap: robots.txt forbids' \
-    httrack 'BASEURL/sitemapdir/start.html' --sitemap -r2 -s2 \
-    -F 'denydeclared-agent'
-
-# The user named it, which is the same intent as a start URL: not refused
-# either, even though robots.txt forbids that exact path.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-not-found 'Sitemap: robots.txt forbids' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 -s2 -F 'denysitemap-agent' \
-    --sitemap-url 'BASEURL/sitemap.xml'
-
-# A child sitemap is a fetch like any other: a filter rejecting it stops the
-# whole subtree, so the page only it lists is never reached.
-crawl --errors 0 \
-    --not-found 'sitemapdir/gated.html' \
-    --log-found 'Sitemap: filter rule #[0-9]+ refuses' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 '-*filtered.xml*' \
-    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
-
-# Control: without that filter the same chain does reach the page.
-crawl --errors 0 \
-    --found 'sitemapdir/gated.html' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/gatedindex.xml'
-
-# A redirected sitemap is still ingested: the marking follows the 301.
-crawl --errors 0 \
-    --found 'sitemapdir/orphan1.html' \
-    --log-found 'moved\.xml redirects to .*pages\.xml\.gz' \
-    --log-found 'URL\(s\) added from 127\.0\.0\.1:[0-9]+/sitemapdir/pages\.xml\.gz' \
-    httrack 'BASEURL/sitemapdir/start.html' -r2 \
-    --sitemap-url 'BASEURL/sitemapdir/moved.xml'

tests/90_webhttrack-checkbox-clear.test

@@ -1,196 +0,0 @@
-#!/bin/bash
-#
-# An unchecked box posts nothing and the stored "1" survives: hence a hidden
-# companion per box, plus a disabling flag for the default-on ones.
-
-set -euo pipefail
-
-testdir=$(cd "$(dirname "$0")" && pwd)
-distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
-distdir=$(cd "${distdir}" && pwd)
-# shellcheck source=tests/testlib.sh
-. "${testdir}/testlib.sh"
-
-fail() {
-    echo "FAIL: $*" >&2
-    exit 1
-}
-
-command -v htsserver >/dev/null || fail "no htsserver in PATH"
-python=$(find_python) || {
-    echo "python3 not found; skipping" >&2
-    exit 77
-}
-
-work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
-srvlog=$(mktemp)
-srv=
-srvpid=
-cleanup() {
-    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
-    test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
-    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
-    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
-    rm -rf "${work}" "${srvlog}"
-}
-trap cleanup EXIT HUP INT QUIT PIPE TERM
-
-# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
-sport=$("${python}" -c 'import socket
-s = socket.socket()
-s.bind(("127.0.0.1", 0))
-print(s.getsockname()[1])
-s.close()')
-(
-    trap '' TERM TTOU
-    export HOME="${work}"
-    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
-) &
-srv=$!
-for _ in $(seq 1 40); do
-    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
-    kill -0 "${srv}" 2>/dev/null || break
-    sleep 0.25
-done
-test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
-srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
-
-"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
-import glob, os, re, sys, urllib.parse, urllib.request
-
-url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
-rc = 0
-
-
-def check(ok, what):
-    global rc
-    print(("ok: " if ok else "FAIL: ") + what)
-    if not ok:
-        rc = 1
-
-
-# cache/cache2 are left to the separate rework of the dead "Cache" seed.
-skip = {"cache", "cache2"}
-page_of = {}
-boxes = 0
-for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
-    page = open(path, "rb").read().decode("latin-1")
-    cleared = dict((m.group(1), m.start()) for m in
-                   re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
-    for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
-        boxes += 1
-        if m.group(1) in skip:
-            continue
-        page_of[m.group(1)] = os.path.basename(path)
-        at = cleared.get(m.group(1))
-        check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
-              % (os.path.basename(path), m.group(1)))
-# Control: a regex that stopped matching would leave nothing to assert on.
-check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
-
-sid = None
-
-
-def get(path):
-    # The UI is served ISO-8859-1, so decode, do not assume UTF-8.
-    return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
-
-
-def post(fields):
-    global sid
-    if sid is None:
-        m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
-        if m is None:
-            sys.exit("no session id in server/index.html")
-        sid = m.group(1)
-    body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
-                    [("sid", sid)] + fields)
-    req = urllib.request.Request(url + "/server/step4.html",
-                                 data=body.encode("latin-1"), method="POST")
-    return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
-
-
-def textarea(page, name):
-    m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
-    if m is None:
-        sys.exit("no %s textarea in the rendered step4.html" % name)
-    return m.group(1)
-
-
-def checked(page, name):
-    return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
-
-
-# What each state puts on the command line (None: nothing) and its profile key.
-# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
-# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
-BOXES = [
-    # field           profile key           set                          cleared
-    ("parseall", "ParseAll", "--near", None),
-    ("link", "Near", "--test", None),
-    ("testall", "Test", "--extended-parsing", None),
-    # htmlfirst emits --priority=7, which the scan-priority list also emits.
-    ("htmlfirst", "HTMLFirst", None, None),
-    ("errpage", "NoErrorPages", "--generate-errors=0", None),
-    ("external", "NoExternalPages", "--replace-external", None),
-    ("hidepwd", "NoPwdInPages", "--disable-passwords", None),
-    ("hidequery", "NoQueryStrings", "--include-query-string=0", None),
-    ("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
-    ("windebug", None, "--debug-headers", None),
-    ("ka", "KeepAlive", "--keep-alive", None),
-    ("remt", "RemoveTimeout", "--host-control=1", None),
-    ("rems", "RemoveRateout", "--host-control=2", None),
-    ("cookies", "Cookies", None, "--cookies=0"),
-    ("parsejava", "ParseJava", None, "--parse-java=0"),
-    ("updhack", "UpdateHack", "--updatehack", None),
-    ("urlhack", "URLHack", "--urlhack", None),
-    ("keepwww", "KeepWww", "--keep-www-prefix", None),
-    ("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
-    ("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
-    ("toler", "TolerantRequests", "--tolerant", None),
-    ("http10", "HTTP10", "--http-10", None),
-    ("sitemap", "Sitemap", "--sitemap", None),
-    ("warc", "Warc", "--warc", None),
-    ("norecatch", "NoRecatch", "--do-not-recatch", None),
-    ("logf", "Log", "--single-log", None),
-    ("index", "Index", None, None),
-    ("index2", "WordIndex", "--search-index", None),
-    ("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
-]
-
-for field, key, when_set, when_clear in BOXES:
-    for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
-                                          ("", when_clear, when_set, "0")):
-        rendered = post([(field, value)])
-        cmd = textarea(rendered, "command").split()
-        label = "%s %s" % (field, "set" if value else "cleared")
-        if want:
-            check(want in cmd, "%s emits %s" % (label, want))
-        if unwanted:
-            check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
-        if key:
-            # The Windows GUI reads the same option out of the profile.
-            check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
-                  "%s writes %s=%s" % (label, key, stored))
-        check(checked(page_of[field], field) == bool(value),
-              "%s draws %s box" % (label, "a ticked" if value else "an empty"))
-missing = sorted(set(page_of) - set(b[0] for b in BOXES))
-check(not missing, "every checkbox is exercised (missing %s)" % missing)
-
-# A browser posts the companion and the ticked box under one name, so the fix
-# rests on the body loop keeping the last value; both orders pin that down.
-cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
-check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
-check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
-cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
-check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
-check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
-
-sys.exit(rc)
-PY
-
-# A leaked htsserver wedges the parallel harness behind a green log.
-cleanup
-! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
-
-echo "PASS"

tests/Makefile.am

@@ -32,7 +32,6 @@ TESTS = \
 	00_runnable.test \
 	01_engine-charset.test \
 	01_engine-cmdline.test \
-	01_engine-cmdline-split.test \
 	01_engine-cookies.test \
 	01_engine-copyopt.test \
 	01_engine-crange.test \
@@ -65,7 +64,6 @@ TESTS = \
 	01_engine-reconcile.test \
 	01_engine-expandhome.test \
 	01_engine-fsize.test \
-	01_engine-growsize.test \
 	01_engine-redirect.test \
 	01_engine-longpath-io.test \
 	01_engine-mirror-io.test \
@@ -90,7 +88,6 @@ TESTS = \
 	01_engine-xfread.test \
 	01_zlib-acceptencoding.test \
 	01_zlib-warc.test \
-	01_zlib-sitemap.test \
 	01_zlib-warc-cdx.test \
 	01_zlib-warc-wacz.test \
 	01_zlib-contentcodings.test \
@@ -150,7 +147,6 @@ TESTS = \
 	50_local-contentcodings.test \
 	51_local-update-codec.test \
 	52_local-socks5.test \
-	53_local-proxytrack-arc-reason.test \
 	53_local-proxytrack-cache-corrupt.test \
 	54_local-update-truncate-purge.test \
 	55_local-chunked.test \
@@ -176,17 +172,6 @@ TESTS = \
 	75_engine-longpath-posix.test \
 	76_cli-resize.test \
 	77_webhttrack-redirect.test \
-	78_webhttrack-sid.test \
-	79_local-proxytrack-webdav-mime.test \
-	80_engine-crash-symbolize.test \
-	81_webhttrack-maxsize.test \
-	82_webhttrack-browse-links.test \
-	83_webhttrack-argescape.test \
-	84_webhttrack-mirror-verbatim.test \
-	85_webhttrack-projpath.test \
-	86_local-proxytrack-cache-longfields.test \
-	87_local-proxytrack-nodate.test \
-	89_local-sitemap.test \
-	90_webhttrack-checkbox-clear.test
+	78_webhttrack-sid.test
 
 CLEANFILES = check-network_sh.cache

tests/local-server.py

@@ -18,7 +18,6 @@ import base64
 import gzip
 import hashlib
 import os
-import re
 import sys
 import time
 from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
@@ -543,36 +542,8 @@ class Handler(SimpleHTTPRequestHandler):
             return self.fail_cookie(name)
         self.send_html("\tThis is the secret.")
 
-    # A User-Agent carrying NO_SITEMAP_UA gets a robots.txt with no Sitemap:
-    # record, so a test can drive the /sitemap.xml fallback instead.
-    NO_SITEMAP_UA = "nositemap"
-    # ... and one that additionally Disallows the well-known location, so the
-    # fallback has to be refused by the rules this very body carries.
-    DENY_SITEMAP_UA = "denysitemap"
-    # ... the same Disallow, but with the sitemap also declared: the
-    # declaration is the site inviting the fetch and must win.
-    DENY_DECLARED_UA = "denydeclared"
-    # ... and one that points the sitemap at the site root, to check a subtree
-    # crawl is not widened by where the site chooses to put its sitemap.
-    SCOPE_SITEMAP_UA = "scopesitemap"
-
     def route_robots(self):
-        # The Sitemap: record is group-independent; only --sitemap acts on it.
-        ua = self.headers.get("User-Agent") or ""
-        host = self.headers.get("Host")
-        body = "User-agent: *\nDisallow:\n"
-        if self.DENY_DECLARED_UA in ua:
-            body = (
-                "User-agent: *\nDisallow: /sitemap.xml\n"
-                f"Sitemap: http://{host}/sitemap.xml\n"
-            )
-        elif self.DENY_SITEMAP_UA in ua:
-            body = "User-agent: *\nDisallow: /sitemap.xml\n"
-        elif self.SCOPE_SITEMAP_UA in ua:
-            body += f"Sitemap: http://{host}/scopesitemap.xml\n"
-        elif self.NO_SITEMAP_UA not in ua:
-            body += f"Sitemap: http://{host}/sitemapdir/index.xml\n"
-        body = body.encode()
+        body = b"User-agent: *\nDisallow:\n"
         self.send_response(200)
         self.send_header("Content-Type", "text/plain")
         self.send_header("Content-Length", str(len(body)))
@@ -580,132 +551,6 @@ class Handler(SimpleHTTPRequestHandler):
         if self.command != "HEAD":
             self.wfile.write(body)
 
-    # --- sitemap ingestion (issue #712) ------------------------------------
-    # start.html links to nothing, so orphan*.html are reachable only through
-    # the sitemap. deep1.html proves the seeds keep a full depth budget; the
-    # off-host page <loc> must be dropped by the travel scope, and the off-host
-    # child sitemap by the ingester's same-host rule. The index is served both
-    # from /sitemapdir/ (named by robots.txt) and from the well-known
-    # /sitemap.xml (the fallback).
-
-    def route_sitemap_index(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?>\n'
-            '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
-            f"<sitemap><loc>http://{host}/sitemapdir/pages.xml.gz</loc></sitemap>"
-            "<sitemap><loc>http://sitemap-offhost.invalid/s.xml</loc></sitemap>"
-            "</sitemapindex>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_pages(self):
-        host = self.headers.get("Host")
-        xml = (
-            '<?xml version="1.0" encoding="UTF-8"?>\n'
-            '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
-            f"<url><loc>http://{host}/sitemapdir/orphan1.html</loc></url>"
-            "<url><loc>http://sitemap-offhost.invalid/x.html</loc></url>"
-            f"<url><loc>http://{host}/sitemapdir/orphan2.html</loc></url>"
-            "</urlset>\n"
-        ).encode()
-        self.send_raw(gzip.compress(xml), "application/x-gzip")
-
-    def route_sitemap_start(self):
-        self.send_html("\tNothing links to the sitemap pages.")
-
-    def route_sitemap_orphan1(self):
-        self.send_html('\t<a href="deep1.html">deeper</a>')
-
-    def route_sitemap_orphan2(self):
-        self.send_html("\tSecond orphan.")
-
-    def route_sitemap_deep1(self):
-        self.send_html("\tOne level below an orphan.")
-
-    # chainN is a sitemapindex at nesting level N, listing chain(N+1) and a
-    # urlset capN.xml whose single page is capN.html. Levels up to
-    # HTS_SITEMAP_MAX_LEVEL are followed, so capN.html appears for N below the
-    # cap and stops appearing at it: the pair pins the boundary, which a cap
-    # mutated either way would break.
-    def route_sitemap_chain(self):
-        host = self.headers.get("Host")
-        level = int(self.path.rsplit("/", 1)[-1][len("chain") : -len(".xml")])
-        self.send_raw(
-            (
-                '<?xml version="1.0" encoding="UTF-8"?>\n<sitemapindex>'
-                f"<sitemap><loc>http://{host}/sitemapdir/chain{level + 1}.xml"
-                "</loc></sitemap>"
-                f"<sitemap><loc>http://{host}/sitemapdir/cap{level}.xml"
-                "</loc></sitemap></sitemapindex>\n"
-            ).encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_capset(self):
-        host = self.headers.get("Host")
-        level = self.path.rsplit("/", 1)[-1][len("cap") : -len(".xml")]
-        self.send_raw(
-            (
-                '<?xml version="1.0" encoding="UTF-8"?>\n<urlset>'
-                f"<url><loc>http://{host}/sitemapdir/cap{level}.html</loc></url>"
-                "</urlset>\n"
-            ).encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_cappage(self):
-        self.send_html("\tReached through a nested sitemapindex.")
-
-    def route_sitemap_gatedindex(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?><sitemapindex>'
-            f"<sitemap><loc>http://{host}/sitemapdir/filtered.xml</loc></sitemap>"
-            "</sitemapindex>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_filtered(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
-            f"<url><loc>http://{host}/sitemapdir/gated.html</loc></url>"
-            "</urlset>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_gated(self):
-        self.send_html("\tListed only by the filtered child sitemap.")
-
-    # A moved sitemap: the marking has to follow the redirect.
-    # A root sitemap naming a page below the crawl's start directory and one
-    # above it. Only the first may be seeded when the crawl started at /deep/dir/.
-    def route_sitemap_scope(self):
-        host = self.headers.get("Host")
-        self.send_raw(
-            '<?xml version="1.0" encoding="UTF-8"?><urlset>'
-            f"<url><loc>http://{host}/deep/dir/below.html</loc></url>"
-            f"<url><loc>http://{host}/elsewhere/updir.html</loc></url>"
-            "</urlset>\n".encode(),
-            "application/xml",
-        )
-
-    def route_sitemap_deepstart(self):
-        self.send_html("\tA start page in a subdirectory, linking nothing.")
-
-    def route_sitemap_below(self):
-        self.send_html("\tBelow the start directory.")
-
-    def route_sitemap_updir(self):
-        self.send_html("\tAbove the start directory.")
-
-    def route_sitemap_moved(self):
-        self.send_response(301)
-        self.send_header("Location", "/sitemapdir/pages.xml.gz")
-        self.send_header("Content-Length", "0")
-        self.end_headers()
-
     # --- type/extension matrix (issue #267 family) -------------------------
 
     def send_raw(self, body, content_type, extra_headers=()):
@@ -1733,21 +1578,6 @@ class Handler(SimpleHTTPRequestHandler):
         "/gated/index.php": route_gated_index,
         "/gated/secret.php": route_gated_secret,
         "/robots.txt": route_robots,
-        "/sitemapdir/index.xml": route_sitemap_index,
-        "/sitemap.xml": route_sitemap_index,
-        "/sitemapdir/pages.xml.gz": route_sitemap_pages,
-        "/sitemapdir/start.html": route_sitemap_start,
-        "/sitemapdir/orphan1.html": route_sitemap_orphan1,
-        "/sitemapdir/orphan2.html": route_sitemap_orphan2,
-        "/sitemapdir/deep1.html": route_sitemap_deep1,
-        "/sitemapdir/gatedindex.xml": route_sitemap_gatedindex,
-        "/sitemapdir/filtered.xml": route_sitemap_filtered,
-        "/sitemapdir/gated.html": route_sitemap_gated,
-        "/sitemapdir/moved.xml": route_sitemap_moved,
-        "/scopesitemap.xml": route_sitemap_scope,
-        "/deep/dir/start.html": route_sitemap_deepstart,
-        "/deep/dir/below.html": route_sitemap_below,
-        "/elsewhere/updir.html": route_sitemap_updir,
         "/warcgz/index.html": route_warcgz_index,
         "/warcgz/page.html": route_warcgz_page,
         "/warcgz/data.bin": route_warcgz_data,
@@ -2072,13 +1902,6 @@ class Handler(SimpleHTTPRequestHandler):
             return True
         # Match percent-encoded paths (accented #157 route) by their decoded form.
         handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path))
-        if handler is None:
-            if re.fullmatch(r"/sitemapdir/chain\d+\.xml", path):
-                handler = type(self).route_sitemap_chain
-            elif re.fullmatch(r"/sitemapdir/cap\d+\.xml", path):
-                handler = type(self).route_sitemap_capset
-            elif re.fullmatch(r"/sitemapdir/cap\d+\.html", path):
-                handler = type(self).route_sitemap_cappage
         if handler is not None:
             handler(self)
             return True

tests/webhttrack-smoke.sh

@@ -46,14 +46,12 @@ cat >"$stubdir/x-www-browser" <<EOF
 echo "stub browser invoked with: \$1" >&2
 # Also fetch an option page and require a rendered title='' tooltip: proves the
 # option template expands and the \${html:} filter escapes into the attribute.
-# option9/option8 additionally prove the WARC and sitemap controls render.
+# option9 additionally proves the WARC control renders with its expanded label.
 opturl="\${1%/}/server/option2.html"
 warcurl="\${1%/}/server/option9.html"
-smurl="\${1%/}/server/option8.html"
 if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html &&
     opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" &&
-    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC &&
-    sm="\$(curl -fsSL --max-time 20 "\$smurl")" && printf '%s' "\$sm" | grep -qaF 'name="sitemapurl"' && printf '%s' "\$sm" | grep -qaF 'name="sitemap"'; then
+    warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then
     echo PASS >"$marker"
 else
     echo "FAIL: unexpected response from \$1" >"$marker"