An .ndx entry's two URL halves overflow the buffer they share (#744)

Both binput calls pass HTS_URLMAXSIZE, but they write into one
line[HTS_URLMAXSIZE * 2] and binput puts its NUL at s[max]. A first field
that fills its whole bound leaves the second writing line[2048], one past
the array. ASan reports a stack-buffer-overflow on a crafted .ndx. Bound
the second by what the first left.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

.github/workflows/windows-build.yml

@@ -226,8 +226,9 @@ jobs:
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
           # footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
-          # webdav-mime needs a reapable background listener, which MSYS cannot give it.
-          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
+          # webdav-mime needs a reapable background listener, which MSYS cannot give it;
+          # badmtime needs a filesystem that stores an mtime past gmtime's range.
+          expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test"
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -26,6 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   check`, or `PATH="<bld>/src:$PATH"` for a manual run.
 - Give new `.test` scripts `set -e`: the older ones predate the rule, so several
   `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
+- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
+  pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
+  and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
+  as "marker absent". Capture the reply, assert the status line it must carry
+  (an empty, truncated or redirected one is marker-free too), then match with a
+  here-string.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -48,6 +54,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
+- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
+  (`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
+  our own data, wrong for anything read back from a cache, a header or the
+  wire, where it trades a memory smash for a crash on malformed input. Clip
+  with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
+- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
+  a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
+  into the same buffer never appears on it. Before scoping a hardening pass off
+  compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
+  `\bstrcpy\s*\(`, `\bstrcat\s*\(`).
 
 ## C conventions
 - **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -84,6 +100,17 @@ Before pushing, and when reviewing others, don't skim for bugs:
   layout/ABI, cache/wire format, or a security path? A static or unit check
   isn't enough; exercise the wrong behavior at runtime. Claude Code:
   `/review-recipe`.
+- **Poison a canary, never compare it against zero.** Checking that a
+  neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
+  terminator writes — the exact bug the canary is there for. Fill it with a
+  non-zero byte, and prove it by killing both the stray-`'X'` and the
+  stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
+  lands inside the same struct.
+- **Overshoot every destination, not one.** A bounds test that oversizes a
+  single field cannot tell a per-field bound from a one-size-fits-all one, nor
+  from a fix that bounds that field and leaves its neighbours raw. Exercise
+  each destination the path touches, spanning at least two capacities, and
+  check what the code actually emits before writing the expected values.
 
 ## Commits
 - **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

html/server/option10.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ftpprox" value="">
+
 ${LANG_PROXYTYPE}
 <select name="proxytype"
 			title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

html/server/option2.html

@@ -98,6 +98,13 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="errpage" value="">
+<input type="hidden" name="external" value="">
+<input type="hidden" name="hidepwd" value="">
+<input type="hidden" name="hidequery" value="">
+<input type="hidden" name="nopurge" value="">
+
 ${LANG_I33}
 <br>
 <select name="build"

html/server/option3.html

@@ -98,6 +98,9 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="windebug" value="">
+
 ${LANG_I40c}
 <br>
 

html/server/option4.html

@@ -98,6 +98,11 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="ka" value="">
+<input type="hidden" name="remt" value="">
+<input type="hidden" name="rems" value="">
+
 <table border="0" width="100%" cellspacing="0">
 
 <tr><td>

html/server/option8.html

@@ -98,6 +98,17 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="cookies" value="">
+<input type="hidden" name="parsejava" value="">
+<input type="hidden" name="updhack" value="">
+<input type="hidden" name="urlhack" value="">
+<input type="hidden" name="keepwww" value="">
+<input type="hidden" name="keepslashes" value="">
+<input type="hidden" name="keepqueryorder" value="">
+<input type="hidden" name="toler" value="">
+<input type="hidden" name="http10" value="">
+
 <input type="checkbox" name="cookies" ${checked:cookies}
 			title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I58}

html/server/option9.html

@@ -98,6 +98,13 @@ ${do:end-if}
 <input type="hidden" name="redirect" value="">
 <input type="hidden" name="closeme" value="">
 
+<!-- clear if not checked -->
+<input type="hidden" name="warc" value="">
+<input type="hidden" name="norecatch" value="">
+<input type="hidden" name="logf" value="">
+<input type="hidden" name="index" value="">
+<input type="hidden" name="index2" value="">
+
 <input type="checkbox" name="cache2" ${checked:cache2}
 			title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
 > ${LANG_I61}

html/server/step4.html

@@ -114,7 +114,7 @@ ${do:end-if}
 
 ${/* Real commands and ini file generated below */}
 
-<!-- engine commandline -->
+<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
 ${do:output-mode:html}
 <textarea name="command" cols="50" rows="4" style="visibility:hidden">
 httrack \
@@ -175,8 +175,8 @@ ${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one *
 \
 	${unquoted:url2}
 \
-	${test:cookies:--cookies=0:}
-	${test:parsejava:--parse-java=0:}
+	${ztest:cookies:--cookies=0:}
+	${ztest:parsejava:--parse-java=0:}
 	${test:updhack:--updatehack}
 	${test:urlhack:--urlhack=0:--urlhack}
 	${test:keepwww:--keep-www-prefix}

libtest/callbacks-example-filenameiisbug.c

@@ -71,7 +71,9 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
       for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
       if (iisBogus[i][j] == '\0'
           && (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
-        strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
+        /* j bytes matched, so j fit: copying j cannot overrun whatever the
+           table holds, and the tail must survive untouched */
+        memcpy(a, iisBogusReplace[i], (size_t) j);
         break;
       }
     }

src/Makefile.am

@@ -48,7 +48,7 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
 
 lib_LTLIBRARIES = libhttrack.la
 
-htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
+htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
 	htscmdline.c htscmdline.h \
 	htsurlport.c htsurlport.h
 proxytrack_SOURCES = proxy/main.c \
@@ -87,7 +87,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
 libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
 libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
 
-EXTRA_DIST = httrack.h webhttrack \
+EXTRA_DIST = httrack.h htsstats.h webhttrack \
 		version.rc \
 		libhttrack.rc \
 		httrack.rc \

src/htscache.c

@@ -142,10 +142,12 @@ struct cache_back_zip_entry {
   int compressionMethod;
 };
 
+/* A corrupt cache can carry a field wider than ours; clipping it keeps the
+   entry, where aborting would take the crawl down. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      strlcpybuff(refvalue, value, refvalue_size);                             \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)

src/htscache_selftest.c

@@ -1195,6 +1195,12 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
 
 /* --- read-side corruption injection --------------------------------------- */
 
+/* 100 'A's: a placeholder header line long enough to be overwritten by a
+   forged, over-long X-StatusMessage of the same byte length. */
+#define CORRUPT_LONG_ETAG                                                      \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"                         \
+  "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+
 /* canary read back intact after each corruption; victim gets the byte surgery
  */
 #define CORRUPT_ADR "corrupt.example.com"
@@ -1219,6 +1225,24 @@ static void corrupt_build(httrackp *opt) {
   selftest_close(&cache);
 }
 
+/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
+static void corrupt_build_longetag(httrackp *opt) {
+  cache_back cache;
+
+  memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
+  memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
+  remove(reconcile_st_path(opt, "hts-cache/new.zip"));
+  remove(reconcile_st_path(opt, "hts-cache/old.zip"));
+  selftest_open_for_write(&cache, opt);
+  store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
+              "OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
+              strlen(corrupt_body_a));
+  store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
+              "OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
+              corrupt_body_b, strlen(corrupt_body_b));
+  selftest_close(&cache);
+}
+
 /* Like corrupt_build, but the victim carries a 20-char Etag whose header line
    is later overwritten with a forged oversized X-Size (same byte length). */
 static void corrupt_build_etag(httrackp *opt) {
@@ -1403,6 +1427,48 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
   return fail;
 }
 
+/* An over-long field from a foreign cache must clip, not abort: the entry
+   still reads, clipped to capacity, and the canary survives. */
+static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
+                                         size_t wantlastmod, const char *what) {
+  cache_back cache;
+  htsblk v, c;
+  char BIGSTK lv[HTS_URLMAXSIZE * 2];
+  char BIGSTK lc[HTS_URLMAXSIZE * 2];
+  int fail = 0;
+
+  selftest_open_for_read(&cache, opt);
+  lv[0] = lc[0] = '\0';
+  v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
+  if (v.statuscode != 200) {
+    fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
+            v.statuscode);
+    fail++;
+  }
+  if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
+    fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
+            (unsigned) strlen(v.msg), (unsigned) wantmsg);
+    fail++;
+  }
+  if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
+    fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
+            what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
+    fail++;
+  }
+  c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
+  if (c.statuscode != 200) {
+    fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
+            c.statuscode);
+    fail++;
+  }
+  if (v.adr != NULL)
+    freet(v.adr);
+  if (c.adr != NULL)
+    freet(c.adr);
+  selftest_close(&cache);
+  return fail;
+}
+
 /* One zip corruption case: build, patch, then check victim+canary in-session.
  */
 static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1439,6 +1505,31 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
   failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
                                     "garbled deflate stream");
 
+  /* A corrupt cache can hold a field wider than ours. Clipping keeps the
+     entry; aborting would take the crawl down. Overwrite the placeholder Etag
+     line in place, same byte length, so the zip offsets stay intact. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "X-StatusMessage: " /* 17 + 89 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures +=
+      corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
+                                    (size_t) -1, "over-long X-StatusMessage");
+
+  /* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
+     cannot satisfy both. */
+  corrupt_build_longetag(opt);
+  corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
+                "Last-Modified: " /* 15 + 91 = 106 */
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+                "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+                1, 1);
+  failures += corrupt_expect_victim_clipped(
+      opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
+      "over-long Last-Modified");
+
   /* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
      truncates negative in the (int) cast the malloc uses: a wraparound alloc.
      cache_add asserts size fits an int, so such a value only reaches the reader

src/htshelp.c

@@ -433,7 +433,8 @@ void help_catchurl(const char *dest_path) {
       }
       // former URL!
       {
-        char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
+        /* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
+        char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
 
         inplace_escape_check_url(dest, sizeof(dest));
         snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);

src/htslib.c

@@ -2537,15 +2537,6 @@ void fil_simplifie(char *f) {
   }
 }
 
-void htsblk_failf(htsblk *r, const char *fmt, ...) {
-  va_list args;
-
-  va_start(args, fmt);
-  // deliberate clip: the reason is quoted from a remote reply
-  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
-  va_end(args);
-}
-
 // fermer liaison fichier ou socket
 void deletehttp(htsblk * r) {
 #if HTS_DEBUG_CLOSESOCK

src/htslib.h

@@ -201,7 +201,8 @@ T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
                    int waitconnect, int addr_index, int *addr_count);
 /* Clips the formatted failure reason into r->msg, which also round-trips
    through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
-void htsblk_failf(htsblk *r, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
+#define htsblk_failf(R, ...)                                                   \
+  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
 
 HTS_INLINE void deletehttp(htsblk * r);
 HTS_INLINE int deleteaddr(htsblk * r);

src/htssafe.h

@@ -460,6 +460,25 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
   return b->buf;
 }
 
+/**
+ * Copy src into dest (capacity size, NUL included), truncating to fit and
+ * always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
+ * value read back from a cache, a header or the wire, where refusing the whole
+ * record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
+ * callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
+ */
+static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
+                                                     const char *src) {
+  size_t len, copy;
+
+  assertf(dest != NULL && src != NULL && size != 0);
+  len = strlen(src);
+  copy = len < size ? len : size - 1;
+  memcpy(dest, src, copy);
+  dest[copy] = '\0';
+  return copy == len ? HTS_TRUE : HTS_FALSE;
+}
+
 /**
  * Callers that deliberately ignore truncation use this instead of
  * slprintfbuff(), so it is not HTS_CHECK_RESULT.
@@ -469,6 +488,9 @@ static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
   int ret;
 
   assertf(dest != NULL && size != 0);
+  /* a vsnprintf failing outright may write nothing at all, leaving whatever
+     the caller had on the stack for it to publish */
+  dest[0] = '\0';
   ret = vsnprintf(dest, size, fmt, args);
   /* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
   dest[size - 1] = '\0';
@@ -492,6 +514,20 @@ static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
   return ret;
 }
 
+/**
+ * slprintfbuff() for diagnostics quoting remote or client text, which are
+ * meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
+ * cast on slprintfbuff() is no substitute, GCC warns through it.
+ */
+static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
+    char *dest, size_t size, const char *fmt, ...) {
+  va_list args;
+
+  va_start(args, fmt);
+  (void) vslprintfbuff(dest, size, fmt, args);
+  va_end(args);
+}
+
 /**
  * slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
  * On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.

src/htsselftest.c

@@ -457,6 +457,12 @@ static void basic_selftests(void) {
     // link one level up -> a "../" prefix
     assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
     assertf(strcmp(s, "../a.html") == 0);
+    // an empty current path: the trim used to walk off the front of it, which
+    // "?x" reaches too because the query pre-pass hands on the part before it
+    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
+    assertf(strcmp(s, "dir/page.html") == 0);
+    assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
+    assertf(strcmp(s, "dir/page.html") == 0);
   }
 }
 
@@ -618,6 +624,66 @@ static int string_safety_selftests(void) {
       return 1;
   }
 
+  /* strclipbuff: truncate-and-report, never abort. Same canary shape; the
+     destination is poisoned before every call so a case cannot pass on the
+     previous one's bytes. */
+  {
+    struct {
+      char dst[8];
+      char canary[8];
+    } s;
+
+    memset(&s, '#', sizeof(s));
+#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
+
+    /* well under capacity: no padding, nothing eaten off the end */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "abc") || strcmp(s.dst, "abc") != 0)
+      return 1;
+
+    /* exact fit: capacity - 1 characters plus the NUL */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "1234567") ||
+        strcmp(s.dst, "1234567") != 0)
+      return 1;
+
+    /* one over, then far over: clipped, terminated, reported. The expected
+       bytes differ from the case above, so a write-nothing implementation
+       cannot pass on the leftovers. */
+    POISON_DST();
+    if (strclipbuff(s.dst, sizeof(s.dst), "abcdefgh") ||
+        strcmp(s.dst, "abcdefg") != 0)
+      return 1;
+    POISON_DST();
+    if (strclipbuff(s.dst, sizeof(s.dst), "0123456789abcdef") ||
+        strcmp(s.dst, "0123456") != 0)
+      return 1;
+
+    /* degenerate capacity 1: only the NUL fits. Capacity 2 pins the boundary
+       between that and the sizes above: one character plus the NUL. */
+    POISON_DST();
+    if (strclipbuff(s.dst, 1, "x") || s.dst[0] != '\0')
+      return 1;
+    POISON_DST();
+    if (strclipbuff(s.dst, 2, "yz") || strcmp(s.dst, "y") != 0)
+      return 1;
+
+    /* the empty string fits any non-zero capacity */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "") || s.dst[0] != '\0')
+      return 1;
+
+    /* a byte over 0x7f must not end the copy early */
+    POISON_DST();
+    if (!strclipbuff(s.dst, sizeof(s.dst), "\xff\xfe") ||
+        strcmp(s.dst, "\xff\xfe") != 0)
+      return 1;
+#undef POISON_DST
+
+    if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
+      return 1;
+  }
+
   /* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
      touches nothing else in the block */
   {
@@ -1519,7 +1585,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
       size_t i;
       for (i = bench[loop].offset; i < (size_t) count;
            i += bench[loop].modulus) {
-        int result;
+        int result = 0; /* no final else: an unknown type reports failure */
         FMT();
         if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
           size_t k;

src/htsserver.c

@@ -318,6 +318,18 @@ typedef struct {
   error_redirect = "/server/error.html"; \
 } while(0)
 
+/* Longest error message shown on the error page; the rest is clipped. */
+#define ERROR_MESSAGE_MAX 1024
+
+/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
+   whose length the client picks. */
+#define SET_ERRORF(...)                                                        \
+  do {                                                                         \
+    char errbuf[ERROR_MESSAGE_MAX];                                            \
+    slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__);                    \
+    SET_ERROR(errbuf);                                                         \
+  } while (0)
+
 /* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
    so anything near this is already invalid and is rejected unread. */
 #define SID_VALUE_MAX 64
@@ -725,7 +737,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
           fp = fopen(StringBuff(fspath), "rb");
           if (fp) {
             /* Read file */
-            while(!feof(fp)) {
+            while (!feof(fp) && !ferror(fp)) {
               char *str = line;
               char *pos;
 
@@ -879,28 +891,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                           commandEnd = 1;
                         }
                       } else {
-                        char tmp[1024];
-
-                        sprintf(tmp,
-                                "Unable to write %d bytes in the the init file %s",
-                                count, StringBuff(fspath));
-                        SET_ERROR(tmp);
+                        SET_ERRORF(
+                            "Unable to write %d bytes in the the init file %s",
+                            count, StringBuff(fspath));
                       }
                       fclose(fp);
                     } else {
-                      char tmp[1024];
-
-                      sprintf(tmp, "Unable to create the init file %s",
-                              StringBuff(fspath));
-                      SET_ERROR(tmp);
+                      SET_ERRORF("Unable to create the init file %s",
+                                 StringBuff(fspath));
                     }
                   } else {
-                    char tmp[1024];
-
-                    sprintf(tmp,
-                            "Unable to create the directory structure in %s",
-                            StringBuff(fspath));
-                    SET_ERROR(tmp);
+                    SET_ERRORF("Unable to create the directory structure in %s",
+                               StringBuff(fspath));
                   }
 
                 } else {
@@ -978,10 +980,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
             }
           }
 
-          /* path itself may hold ".." (webhttrack passes "<bin>/../share"), so
-             only the untrusted halves are checked: file here, website above. */
-          if (fsfile[0] && strstr(file, "..") == NULL
-              && (fp = fopen(fsfile, "rb"))) {
+          /* Regular files only: reading a directory or FIFO never ends, and
+             "path" may hold "..", so only the untrusted halves are checked. */
+          if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
+              (fp = fopen(fsfile, "rb"))) {
             char ok[] =
               "HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
               "Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
@@ -1040,7 +1042,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               int outputmode = 0;
 
               StringMemcat(headers, ok, sizeof(ok) - 1);
-              while(!feof(fp)) {
+              while (!feof(fp) && !ferror(fp)) {
                 char *str = line;
                 int prevlen = (int) StringLength(output);
                 int nocr = 0;
@@ -1474,14 +1476,15 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
               while(!feof(fp)) {
                 int n = (int) fread(line, 1, sizeof(line) - 2, fp);
 
-                if (n > 0) {
-                  StringMemcat(output, line, n);
+                if (n <= 0) {
+                  break; /* short read: EOF or error, never a retry */
                 }
+                StringMemcat(output, line, n);
               }
             }
             fclose(fp);
-          } else if (strcmp(file, "/ping") == 0
-                     || strncmp(file, "/ping?", 6) == 0) {
+          } else if (strcmp(file, "/ping") == 0 ||
+                     strncmp(file, "/ping?", 6) == 0) {
             char error_hdr[] =
               "HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
               "Content-type: text/html\r\n";

src/htsstats.h

@@ -0,0 +1,63 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: in-progress display rows, shared by httrack and htsserver .h */
+/* Author: Xavier Roche                                         */
+/* ------------------------------------------------------------ */
+
+#ifndef HTS_STATS_DEFH
+#define HTS_STATS_DEFH
+
+#include "htsglobal.h"
+
+/* One row of the "in progress" display, shared so the CLI (httrack) and the
+   web GUI (htsserver) cannot drift apart: each fills its own array. */
+
+#define NStatsBuffer 14
+
+#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
+#define HTS_DEF_FWSTRUCT_t_StatsBuffer
+typedef struct t_StatsBuffer t_StatsBuffer;
+#endif
+struct t_StatsBuffer {
+  char name[1024];
+  char file[1024];
+  char state[288];                         // a short label plus back->info[256]
+  char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
+  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
+  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
+  LLint size;
+  LLint sizetot;
+  int offset;
+  //
+  int back;
+  //
+  int actived; // pour disabled
+};
+
+#endif

src/htstools.c

@@ -308,8 +308,10 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
   // copy only the current path
   curr = _curr;
   strlcpybuff(curr, curr_fil, sizeof(_curr));
-  if ((a = strchr(curr, '?')) == NULL)  // couper au ? (params)
-    a = curr + strlen(curr) - 1;        // pas de params: aller à la fin
+  if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
+    // an empty path has no last character: curr-1 would read before the buffer
+    a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
+  }
   while((*a != '/') && (a > curr))
     a--;                        // chercher dernier / du chemin courant
   if (*a == '/')

src/htsweb.h

@@ -35,26 +35,10 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsstats.h"
 
-#define NStatsBuffer     14
 #define MAX_LEN_INPROGRESS 40
 
-typedef struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[288];                      // a short label plus back->info[256]
-  char url_sav[HTS_URLMAXSIZE * 2];     // pour cancel
-  char url_adr[HTS_URLMAXSIZE * 2];
-  char url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-} t_StatsBuffer;
-
 typedef struct t_InpInfo {
   int ask_refresh;
   int refresh;

src/httrack.c

@@ -230,8 +230,7 @@ static hts_boolean vt_size_refresh(void) {
 */
 #define STYLE_STATVALUES VT_BOLD
 #define STYLE_STATTEXT   VT_UNBOLD
-#define STYLE_STATRESET  VT_UNBOLD
-#define NStatsBuffer     14
+#define STYLE_STATRESET VT_UNBOLD
 /* Rows the stats block and "Current job" take above the in-progress list. */
 #define NStatsHeaderRows 7
 

src/httrack.h

@@ -36,26 +36,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsglobal.h"
 #include "htscore.h"
 #include "htssafe.h"
-
-#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
-#define HTS_DEF_FWSTRUCT_t_StatsBuffer
-typedef struct t_StatsBuffer t_StatsBuffer;
-#endif
-struct t_StatsBuffer {
-  char name[1024];
-  char file[1024];
-  char state[288]; // a short label plus back->info[256]
-  char BIGSTK url_sav[HTS_URLMAXSIZE * 2];      // pour cancel
-  char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
-  char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
-  LLint size;
-  LLint sizetot;
-  int offset;
-  //
-  int back;
-  //
-  int actived;                  // pour disabled
-};
+#include "htsstats.h"
 
 #ifndef HTS_DEF_FWSTRUCT_t_InpInfo
 #define HTS_DEF_FWSTRUCT_t_InpInfo

src/proxy/proxytrack.c

@@ -820,8 +820,8 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
     elt->size = StringLength(response);
     elt->adr = StringAcquire(&response);
     elt->statuscode = 207;      /* Multi-Status */
-    strcpy(elt->charset, "utf-8");
-    strcpy(elt->contenttype, "text/xml");
+    strcpybuff(elt->charset, "utf-8");
+    strcpybuff(elt->contenttype, "text/xml");
     strcpybuff(elt->msg, "Multi-Status");
     StringFree(response);
 
@@ -877,8 +877,8 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
     elt->size = StringLength(html);
     elt->adr = StringAcquire(&html);
     elt->statuscode = HTTP_OK;
-    strcpy(elt->charset, "iso-8859-1");
-    strcpy(elt->contenttype, "text/html");
+    strcpybuff(elt->charset, "iso-8859-1");
+    strcpybuff(elt->contenttype, "text/html");
     strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;

src/proxy/proxytrack.h

@@ -368,7 +368,11 @@ HTS_UNUSED static struct tm PT_GetTime(time_t t) {
   if (tm != NULL)
     return *tm;
   else {
+    /* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
+       day of "00"; the epoch is the conventional "date unknown" */
     memset(&tmbuf, 0, sizeof(tmbuf));
+    tmbuf.tm_year = 70;
+    tmbuf.tm_mday = 1;
     return tmbuf;
   }
 }

src/proxy/store.c

@@ -405,8 +405,8 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
     elt->size = StringLength(html);
     elt->adr = StringAcquire(&html);
     elt->statuscode = HTTP_OK;
-    strcpy(elt->charset, "iso-8859-1");
-    strcpy(elt->contenttype, "text/html");
+    strcpybuff(elt->charset, "iso-8859-1");
+    strcpybuff(elt->contenttype, "text/html");
     strcpybuff(elt->msg, "OK");
     StringFree(html);
     return elt;
@@ -829,16 +829,8 @@ PT_Element PT_ElementNew(void) {
 }
 
 /* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
-static void PT_Element_failf(PT_Element r, const char *fmt, ...)
-    HTS_PRINTF_FUN(2, 3);
-
-static void PT_Element_failf(PT_Element r, const char *fmt, ...) {
-  va_list args;
-
-  va_start(args, fmt);
-  (void) vslprintfbuff(r->msg, sizeof(r->msg), fmt, args);
-  va_end(args);
-}
+#define PT_Element_failf(R, ...)                                               \
+  slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
 
 PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
   if (index != NULL && SAFE_INDEX(index)) {
@@ -879,13 +871,11 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
   (headersSize) += (int) strlen(headers + headersSize); \
 } while(0)
 /* refvalue_size is mandatory: the cache line is bounded only by the line
-   buffer, not by the destination. Clip rather than reject, since an
-   engine-written field can be wider than ours. */
+   buffer, not by the destination. */
 #define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)    \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (refvalue)[0] = '\0';                                                    \
-      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -1042,7 +1032,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
   previous_save[0] = previous_save_[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  strcpy(r->location, "");
+  r->location[0] = '\0';
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -1127,7 +1117,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
             int pathLen = (int) strlen(index->path);
 
             if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) {    // old (<3.40) buggy format
-              strcpy(previous_save, previous_save_);
+              strcpybuff(previous_save, previous_save_);
             }
             // relative ? (hack)
             else if (index->safeCache || (previous_save_[0] != '/'      // /home/foo/bar.gif
@@ -1554,7 +1544,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
             cache->version = (int) (firstline[8] - '0');        // cache 1.x
             if (cache->version <= 5) {
               a += cache_brstr(a, firstline, sizeof(firstline));
-              strcpy(cache->lastmodified, firstline);
+              strcpybuff(cache->lastmodified, firstline);
             } else {
               fclose(cache->dat);
               cache->dat = NULL;
@@ -1571,7 +1561,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
         } else {                // Vieille version du cache
           /* */
           cache->version = 0;   // cache 1.0
-          strcpy(cache->lastmodified, firstline);
+          strcpybuff(cache->lastmodified, firstline);
         }
 
         /* Create hash table for the cache (MUCH FASTER!) */
@@ -1587,7 +1577,14 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
               a++;
               /* read "host/file" */
               a += binput(a, line, HTS_URLMAXSIZE);
-              a += binput(a, line + strlen(line), HTS_URLMAXSIZE);
+              {
+                /* binput writes its NUL at s[max], so the second field must be
+                   bounded by what is left of line[], not by the same constant
+                 */
+                const size_t used = strlen(line);
+
+                a += binput(a, line + used, (int) (sizeof(line) - used - 1));
+              }
               /* read position */
               a += binput(a, linepos, 200);
               sscanf(linepos, "%d", &pos);
@@ -1684,7 +1681,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
   previous_save[0] = previous_save_[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  strcpy(r->location, "");
+  r->location[0] = '\0';
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(cache->hash, url, &hash_pos);
@@ -1791,7 +1788,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
           int pathLen = (int) strlen(index->path);
 
           if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) {      // old (<3.40) buggy format
-            strcpy(previous_save, previous_save_);
+            strcpybuff(previous_save, previous_save_);
           }
           // relative ? (hack)
           else if (index->safeCache || (previous_save_[0] != '/'        // /home/foo/bar.gif
@@ -2168,8 +2165,7 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
 #define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size)   \
   do {                                                                         \
     if (line[0] != '\0' && strfield2(line, refline)) {                         \
-      (refvalue)[0] = '\0';                                                    \
-      strlncatbuff(refvalue, value, refvalue_size, (refvalue_size) - 1);       \
+      (void) strclipbuff(refvalue, refvalue_size, value);                      \
       line[0] = '\0';                                                          \
     }                                                                          \
   } while (0)
@@ -2208,7 +2204,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
   location_default[0] = '\0';
   memset(r, 0, sizeof(_PT_Element));
   r->location = location_default;
-  strcpy(r->location, "");
+  r->location[0] = '\0';
   if (strncmp(url, "http://", 7) == 0)
     url += 7;
   hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -2380,8 +2376,18 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
   FILE *const fp = st->fp;
   struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
+  struct tm unknown_date;
   int size_headers;
 
+  /* a cached entry with no parseable Last-Modified must not take the writer
+     down; the epoch is the conventional "date unknown" */
+  if (tm == NULL) {
+    memset(&unknown_date, 0, sizeof(unknown_date));
+    unknown_date.tm_year = 70;
+    unknown_date.tm_mday = 1;
+    tm = &unknown_date;
+  }
+
   sprintf(st->headers,
           "HTTP/1.0 %d %s"
           "\r\n"
@@ -2417,7 +2423,7 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
   if (element->adr != NULL) {
     domd5mem(element->adr, element->size, st->md5, 1);
   } else {
-    strcpy(st->md5, "-");
+    strcpybuff(st->md5, "-");
   }
   fprintf(fp,
           /* nl */

tests/77_webhttrack-redirect.test

@@ -146,11 +146,13 @@ sid=$(scrape_sid "${port}")
 resp=$(post_redirect '/foo
 X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
 stop
-printf '%s' "${resp}" | grep -qi 'X-Injected' &&
+# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
+# pipefail then turns the hit into a silent pass.
+grep -qi 'X-Injected' <<<"${resp}" &&
     fail "CRLF in the redirect value reached the response"
-test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
+test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
     fail "CRLF redirect still emitted a Location header"
-test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
+test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
     fail "CRLF redirect did not yield exactly one status line"
 
 # Loopback unless asked otherwise. Assert the socket, not the announcement.

tests/78_webhttrack-sid.test

@@ -103,9 +103,10 @@ sid=$(scrape_sid "${port}")
 test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
 
 # Accept: the legitimate flow still works. "redirect" is the cheapest field
-# with a reply that is visible in the headers.
+# with a reply that is visible in the headers. Matched from a here-string, not a
+# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
 resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
-printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
+grep -q '^Location: /accepted' <<<"${resp}" ||
     fail "a body carrying the correct sid was refused"
 
 # Refuse: missing and wrong. A missing one is the regression under test — the
@@ -114,12 +115,12 @@ printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
 for bad in "redirect=/nosid" "sid=&redirect=/empty" \
     "sid=00000000000000000000000000000000&redirect=/wrong"; do
     resp=$(request "${port}" "${bad}")
-    printf '%s' "${resp}" | grep -q '^Location:' &&
+    grep -q '^Location:' <<<"${resp}" &&
         fail "body accepted without a valid sid: ${bad}"
     # A refusal has to be a well-formed reply, not a headerless fragment: the
     # release build used to emit only Content-length, which reads as a protocol
     # error to any client and hides the reason.
-    printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
+    grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
         fail "refusal was not a 403: ${bad}"
 done
 
@@ -127,16 +128,23 @@ done
 # applied to one global key store that later requests render from, and the
 # command dispatcher reads it from there. Probe the store itself — step3
 # interpolates ${projname} into its title — rather than the refused reply.
-title() { request "$1" "" step3; }
+store_page() {
+    # a dead probe or a non-page reply must not read as "the store was not written"
+    page=$(request "$1" "" step3) || fail "the key store probe failed"
+    grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
+        fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
+}
 
 request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
-title "${port}" | grep -q 'UNAUTHWRITE' &&
+store_page "${port}"
+grep -q 'UNAUTHWRITE' <<<"${page}" &&
     fail "a body without a valid sid was written to the key store"
 
 # Paired accept case: without it the assertion above passes even if the server
 # simply ignores every body, which would prove nothing.
 request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
-title "${port}" | grep -q 'AUTHWRITE' ||
+store_page "${port}"
+grep -q 'AUTHWRITE' <<<"${page}" ||
     fail "a body carrying the correct sid was not written to the key store"
 
 echo "PASS"

tests/85_webhttrack-projpath.test

@@ -90,6 +90,15 @@ sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
 get() { request "$1" "$2" ""; }
 post() { request "$1" "" "$2"; }
 
+# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
+# reads as marker-absent, and so do a truncated body and a 302 to the file.
+reply=
+fetch() {
+    reply=$(get "$1" "$2") || fail "the request for $2 failed"
+    grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
+        fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
+}
+
 url=$(start)
 port=$(portof "${url}")
 srv=$(srvpid)
@@ -109,13 +118,14 @@ echo "LOGMARKER" >"${base}/proj/hts-log.txt"
 # with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
 # any successful save: commandEnd then swaps the error page for the finished one.
 get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
-post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p" |
-    grep -q '^Location: /server/error.html' ||
+saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
+grep -q '^Location: /server/error.html' <<<"${saved}" ||
     fail "the refused save did not redirect to the error page"
 
 # No project yet, so no root to serve from.
 post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
-get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
+fetch "${port}" /website/secret.txt 404
+grep -q SECRETMARKER <<<"${reply}" &&
     fail "a posted projpath served a file outside any project"
 
 # Positive control: step4's "save settings" flow registers the project without
@@ -126,24 +136,29 @@ body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
 post "${port}" "${body}" >/dev/null
 test -f "${base}/proj/hts-cache/winprofile.ini" ||
     fail "the project was not registered: $(cat "${log}")"
-get "${port}" /website/hts-log.txt | grep -q LOGMARKER ||
+fetch "${port}" /website/hts-log.txt 200
+grep -q LOGMARKER <<<"${reply}" ||
     fail "the registered project's mirror is not served"
 
 # Same request with the root repointed: the project is legitimate, projpath is
 # not what decides where the bytes come from.
 post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
-get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
+fetch "${port}" /website/secret.txt 404
+grep -q SECRETMARKER <<<"${reply}" &&
     fail "a posted projpath repointed the served root"
 post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
-get "${port}" /website/passwd | grep -q '^root:' &&
+fetch "${port}" /website/passwd 404
+grep -q '^root:' <<<"${reply}" &&
     fail "a posted projpath read an arbitrary system file"
 
 # A ".." anywhere in the recorded root would escape the mirror on every later
 # request, so the save must be refused and the previous root kept.
 post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
-get "${port}" /website/secret.txt | grep -q SECRETMARKER &&
+fetch "${port}" /website/secret.txt 404
+grep -q SECRETMARKER <<<"${reply}" &&
     fail "a '..' in the saved project path escaped the mirror root"
-get "${port}" /website/hts-log.txt | grep -q LOGMARKER ||
+fetch "${port}" /website/hts-log.txt 200
+grep -q LOGMARKER <<<"${reply}" ||
     fail "rejecting the '..' root also lost the previous one"
 
 # The root is now the server's own, but it is still built from two posted
@@ -162,7 +177,7 @@ test -f "${fspath}/hts-cache/winprofile.ini" ||
     fail "the long-path project was not registered: $(cat "${log}")"
 get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
 alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
-get "${port}" /server/index.html | grep -q '200 OK' ||
-    fail "the server stopped answering after the over-long project path"
+# Not just alive: still answering.
+fetch "${port}" /server/index.html 200
 
 echo "PASS"

tests/87_local-proxytrack-nodate.test

@@ -0,0 +1,51 @@
+#!/bin/bash
+# A cached entry whose Last-Modified is missing or unparseable must still
+# convert: the ARC writer used to dereference the parsed date unchecked.
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+# $1 Last-Modified value (empty = omit the header), $2 expected archive date,
+# $3 label. The date is asserted exactly: a guard that fires unconditionally
+# would clobber the valid case, and one that skips the year/day would emit a
+# month and day of 00.
+run_case() {
+    lastmod=$1
+    wantdate=$2
+    what=$3
+
+    {
+        printf 'HTTP/1.1 200 OK\r\n'
+        printf 'Content-Type: text/html\r\n'
+        test -z "$lastmod" || printf 'Last-Modified: %s\r\n' "$lastmod"
+        printf 'Content-Length: 5\r\n\r\n'
+    } >"$dir/hdr"
+    printf 'hello' >"$dir/body"
+    alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+    {
+        printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+        printf '2 0 test\n'
+        printf '\n\n'
+        printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+        cat "$dir/hdr" "$dir/body"
+    } >"$dir/in.arc"
+
+    proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
+        echo "FAIL: proxytrack crashed on $what" >&2
+        exit 1
+    }
+    grep -aq "^http://example.com/p.html 0.0.0.0 ${wantdate} " "$dir/out.arc" || {
+        echo "FAIL: $what: expected archive date ${wantdate}, got:" >&2
+        grep -a '^http://example.com/' "$dir/out.arc" >&2 || echo "(entry dropped)" >&2
+        exit 1
+    }
+}
+
+# the epoch stands in for "date unknown"
+run_case '' 19700101000000 'a missing Last-Modified'
+run_case 'not a date at all' 19700101000000 'an unparseable Last-Modified'
+run_case '0' 19700101000000 'a bare 0 Last-Modified'
+# a parseable date must still come through untouched
+run_case 'Sun, 06 Nov 1994 08:49:37 GMT' 19941106084937 'a valid Last-Modified'

tests/88_local-proxytrack-badmtime.test

@@ -0,0 +1,69 @@
+#!/bin/bash
+# A cache file whose mtime is beyond what gmtime can represent must not put a
+# day of "00" in the ARC filedesc date: PT_GetTime fell back to an all-zero tm.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+rc=0
+"$python" - "$dir/in.zip" <<'EOF' || rc=$?
+import os, sys, zipfile
+
+body = b"hello world"
+meta = (
+    "X-In-Cache: 1\r\n"
+    "X-StatusCode: 200\r\n"
+    "X-StatusMessage: OK\r\n"
+    "X-Size: %d\r\n"
+    "Content-Type: text/html\r\n"
+    "Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
+    "X-Save: out/page.html\r\n" % len(body)
+).encode()
+
+zi = zipfile.ZipInfo("example.com/page.html")
+zi.compress_type = zipfile.ZIP_STORED
+zi.extra = meta
+with zipfile.ZipFile(sys.argv[1], "w") as z:
+    z.writestr(zi, body)
+
+# past gmtime's range; the index timestamp is this file's mtime
+huge = 4611686018427387903
+try:
+    os.utime(sys.argv[1], (huge, huge))
+except OSError:
+    sys.exit(77)
+if int(os.stat(sys.argv[1]).st_mtime) < huge:
+    sys.exit(77)  # filesystem clamped it, nothing to test
+EOF
+test "$rc" -eq 0 || {
+    test "$rc" -eq 77 || {
+        echo "FAIL: could not build the cache (python exit $rc)" >&2
+        exit 1
+    }
+    echo "filesystem will not hold an out-of-range mtime; skipping" >&2
+    exit 77
+}
+
+proxytrack --convert "$dir/out.arc" "$dir/in.zip" >/dev/null 2>&1 || {
+    echo "FAIL: proxytrack failed on a cache with an out-of-range mtime" >&2
+    exit 1
+}
+
+# field 3 of the filedesc line is YYYYMMDDhhmmss; the epoch stands in for
+# "unknown", and 14 digits with a real day is the point (00 is what broke)
+date=$(head -n1 "$dir/out.arc" | awk '{ print $3 }')
+test "$date" = 19700101000000 || {
+    echo "FAIL: filedesc date is '$date', expected 19700101000000" >&2
+    exit 1
+}

tests/89_webhttrack-error-overflow.test

@@ -0,0 +1,174 @@
+#!/bin/bash
+#
+# The "save settings" failure messages quote a project path the request body
+# supplies, so composing them must clip instead of running off a fixed buffer.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+# ERROR_MESSAGE_MAX - 1 in htsserver.c.
+msgmax=1023
+
+srv=
+log=$(mktemp)
+# Physical path: macOS resolves TMPDIR through /var -> private/var, and the
+# paths built below sit within a few bytes of its 1024-byte PATH_MAX.
+base=$(cd "$(mktemp -d)" && pwd -P)
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    rm -f "${log}"
+    rm -rf "${base}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL. Runs in a command substitution, so it is a
+# subshell and cannot export the pid: the caller reads it back with srvpid.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU XFSZ
+        # Caps regular-file writes, which is how the failing-write branch below
+        # is reached without /dev/full; the announce rides a pipe, exempt.
+        ulimit -f 8
+        exec htsserver "${distdir}/" --port "${port}" 2>&1
+    ) | cat >"${log}" &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
+
+alive() { kill -0 "$1" 2>/dev/null; }
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
+# $2 is empty. Prints the reply.
+request() {
+    python3 -c 'import socket, sys
+port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
+if path:
+    req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
+else:
+    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+           "Content-type: application/x-www-form-urlencoded\r\n"
+           "Content-length: %d\r\n\r\n%s" % (len(body), body))
+s = socket.create_connection(("127.0.0.1", port), 10)
+s.settimeout(30)
+s.sendall(req.encode())
+out = b""
+while True:
+    b = s.recv(65536)
+    if not b:
+        break
+    out += b
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
+}
+
+get() { request "$1" "$2" ""; }
+post() { request "$1" "" "$2"; }
+
+# Echo a path under the root $1 exactly $2 bytes long, in NAME_MAX-sized parts.
+padpath() {
+    local root=$1 want=$2 p=$1 last
+    while test $((${#p} + 203)) -lt "${want}"; do
+        p="${p}/$(printf '%0200d' 0)"
+    done
+    last=$((want - ${#p} - 1))
+    test "${last}" -ge 1 || fail "cannot build a ${want}-byte path under ${root}"
+    printf '%s/%0*d' "${p}" "${last}" 0
+}
+
+# Fail unless the error page renders the message opening with $1 about project
+# path $2, clipped to msgmax. error.html puts ${error} alone on its own line.
+check_clipped() {
+    local prefix=$1 full=$((${#1} + ${#2})) page line shown
+    test "${full}" -gt "${msgmax}" ||
+        fail "a ${full}-byte message fits in ${msgmax}: it cannot show a clip"
+    page=$(get "${port}" /server/error.html | tr -d '\r')
+    line=$(printf '%s\n' "${page}" | grep "^${prefix}") || {
+        shown=$(printf '%s\n' "${page}" | grep '^Unable to ' | cut -c1-80 || true)
+        fail "the error page reports '${shown:-nothing}', want: ${prefix}"
+    }
+    test "${#line}" -eq "${msgmax}" ||
+        fail "the message rendered ${#line} bytes, want ${full} clipped to ${msgmax}"
+}
+
+url=$(start)
+port=$(portof "${url}")
+srv=$(srvpid)
+test -n "${srv}" || fail "htsserver did not report its pid"
+
+# Every request body is gated by the session id (78_webhttrack-sid.test).
+sid=$(get "${port}" /server/index.html |
+    sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
+
+save() {
+    post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=$2&path=$1&projname=$3"
+}
+
+# A project path too long to be created at all.
+path=$(padpath "${base}/nodir" 1098)
+save "${path}" x p >/dev/null
+alive "${srv}" ||
+    fail "an over-long project path crashed the server: $(cat "${log}")"
+check_clipped "Unable to create the directory structure in " "${path}/p"
+
+# A creatable project path whose hts-cache is not a directory, so the init file
+# under it cannot be opened.
+path=$(padpath "${base}/isdir" 993)
+mkdir -p "${path}/p"
+ln -s /dev/null "${path}/p/hts-cache"
+save "${path}" x p >/dev/null
+alive "${srv}" || fail "an unwritable init file crashed the server: $(cat "${log}")"
+check_clipped "Unable to create the init file " "${path}/p"
+
+# The init file opens, then the write fails: the profile is past both stdio's
+# buffer, so fwrite() reaches the descriptor, and the server's file-size limit.
+path=$(padpath "${base}/full" 993)
+mkdir -p "${path}/p/hts-cache"
+profile=$(printf '%01024d' 0)
+profile=${profile}${profile}${profile}${profile}
+profile=${profile}${profile}${profile}${profile}
+test "${#profile}" -eq 16384 || fail "built a ${#profile}-byte profile, want 16384"
+save "${path}" "${profile}" p >/dev/null
+alive "${srv}" || fail "a short init file write crashed the server: $(cat "${log}")"
+written=$(wc -c <"${path}/p/hts-cache/winprofile.ini" | tr -d ' ')
+test "${written}" -lt "${#profile}" ||
+    fail "the file-size limit did not stop the write (${written} bytes landed)"
+check_clipped "Unable to write ${#profile} bytes in the the init file " "${path}/p"
+
+get "${port}" /server/index.html | grep -q '200 OK' ||
+    fail "the server stopped answering after the refused saves"
+
+echo "PASS"

tests/90_webhttrack-checkbox-clear.test

@@ -0,0 +1,195 @@
+#!/bin/bash
+#
+# An unchecked box posts nothing and the stored "1" survives: hence a hidden
+# companion per box, plus a disabling flag for the default-on ones.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
+srvlog=$(mktemp)
+srv=
+srvpid=
+cleanup() {
+    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
+    test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
+    rm -rf "${work}" "${srvlog}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
+sport=$("${python}" -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()')
+(
+    trap '' TERM TTOU
+    export HOME="${work}"
+    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
+) &
+srv=$!
+for _ in $(seq 1 40); do
+    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
+    kill -0 "${srv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
+srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
+
+"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
+import glob, os, re, sys, urllib.parse, urllib.request
+
+url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
+rc = 0
+
+
+def check(ok, what):
+    global rc
+    print(("ok: " if ok else "FAIL: ") + what)
+    if not ok:
+        rc = 1
+
+
+# cache/cache2 are left to the separate rework of the dead "Cache" seed.
+skip = {"cache", "cache2"}
+page_of = {}
+boxes = 0
+for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
+    page = open(path, "rb").read().decode("latin-1")
+    cleared = dict((m.group(1), m.start()) for m in
+                   re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
+    for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
+        boxes += 1
+        if m.group(1) in skip:
+            continue
+        page_of[m.group(1)] = os.path.basename(path)
+        at = cleared.get(m.group(1))
+        check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
+              % (os.path.basename(path), m.group(1)))
+# Control: a regex that stopped matching would leave nothing to assert on.
+check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
+
+sid = None
+
+
+def get(path):
+    # The UI is served ISO-8859-1, so decode, do not assume UTF-8.
+    return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
+
+
+def post(fields):
+    global sid
+    if sid is None:
+        m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
+        if m is None:
+            sys.exit("no session id in server/index.html")
+        sid = m.group(1)
+    body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
+                    [("sid", sid)] + fields)
+    req = urllib.request.Request(url + "/server/step4.html",
+                                 data=body.encode("latin-1"), method="POST")
+    return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
+
+
+def textarea(page, name):
+    m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
+    if m is None:
+        sys.exit("no %s textarea in the rendered step4.html" % name)
+    return m.group(1)
+
+
+def checked(page, name):
+    return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
+
+
+# What each state puts on the command line (None: nothing) and its profile key.
+# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
+# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
+BOXES = [
+    # field           profile key           set                          cleared
+    ("parseall", "ParseAll", "--near", None),
+    ("link", "Near", "--test", None),
+    ("testall", "Test", "--extended-parsing", None),
+    # htmlfirst emits --priority=7, which the scan-priority list also emits.
+    ("htmlfirst", "HTMLFirst", None, None),
+    ("errpage", "NoErrorPages", "--generate-errors=0", None),
+    ("external", "NoExternalPages", "--replace-external", None),
+    ("hidepwd", "NoPwdInPages", "--disable-passwords", None),
+    ("hidequery", "NoQueryStrings", "--include-query-string=0", None),
+    ("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
+    ("windebug", None, "--debug-headers", None),
+    ("ka", "KeepAlive", "--keep-alive", None),
+    ("remt", "RemoveTimeout", "--host-control=1", None),
+    ("rems", "RemoveRateout", "--host-control=2", None),
+    ("cookies", "Cookies", None, "--cookies=0"),
+    ("parsejava", "ParseJava", None, "--parse-java=0"),
+    ("updhack", "UpdateHack", "--updatehack", None),
+    ("urlhack", "URLHack", "--urlhack", None),
+    ("keepwww", "KeepWww", "--keep-www-prefix", None),
+    ("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
+    ("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
+    ("toler", "TolerantRequests", "--tolerant", None),
+    ("http10", "HTTP10", "--http-10", None),
+    ("warc", "Warc", "--warc", None),
+    ("norecatch", "NoRecatch", "--do-not-recatch", None),
+    ("logf", "Log", "--single-log", None),
+    ("index", "Index", None, None),
+    ("index2", "WordIndex", "--search-index", None),
+    ("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
+]
+
+for field, key, when_set, when_clear in BOXES:
+    for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
+                                          ("", when_clear, when_set, "0")):
+        rendered = post([(field, value)])
+        cmd = textarea(rendered, "command").split()
+        label = "%s %s" % (field, "set" if value else "cleared")
+        if want:
+            check(want in cmd, "%s emits %s" % (label, want))
+        if unwanted:
+            check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
+        if key:
+            # The Windows GUI reads the same option out of the profile.
+            check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
+                  "%s writes %s=%s" % (label, key, stored))
+        check(checked(page_of[field], field) == bool(value),
+              "%s draws %s box" % (label, "a ticked" if value else "an empty"))
+missing = sorted(set(page_of) - set(b[0] for b in BOXES))
+check(not missing, "every checkbox is exercised (missing %s)" % missing)
+
+# A browser posts the companion and the ticked box under one name, so the fix
+# rests on the body loop keeping the last value; both orders pin that down.
+cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
+check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
+check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
+cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
+check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
+check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
+
+sys.exit(rc)
+PY
+
+# A leaked htsserver wedges the parallel harness behind a green log.
+cleanup
+! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
+
+echo "PASS"

tests/91_webhttrack-directory.test

@@ -0,0 +1,181 @@
+#!/bin/bash
+#
+# fopen() succeeds on a directory on POSIX and reading one never reaches EOF; a
+# FIFO blocks in fopen() instead. Either used to wedge the single-threaded server.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+srv=
+log=$(mktemp)
+base=$(mktemp -d)
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    rm -f "${log}"
+    rm -rf "${base}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+# First line only. A "| head -1" would close the pipe early and, under pipefail,
+# SIGPIPE the producer into a spurious failure.
+firstline() { echo "${1%%$'\n'*}"; }
+
+freeport() {
+    python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()'
+}
+
+# Echo the announced URL. Runs in a command substitution, so it is a
+# subshell and cannot export the pid: the caller reads it back with srvpid.
+start() {
+    local port url
+    port=$(freeport)
+    : >"${log}"
+    (
+        trap '' TERM TTOU
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+    echo "${url}"
+}
+
+# The server reports its own pid; the aliveness assertion below hangs off it.
+srvpid() { firstline "$(sed -n 's/^PID=//p' "${log}")"; }
+
+alive() { kill -0 "$1" 2>/dev/null; }
+
+portof() { echo "${1##*:}" | tr -d /; }
+
+# GET the path $2 from 127.0.0.1:$1, or POST the body $3 to / when $2 is empty.
+# The socket timeout is what makes an unfixed tree fail rather than wedge CI.
+request() {
+    python3 -c 'import socket, sys
+port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
+if path:
+    req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
+else:
+    req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
+           "Content-type: application/x-www-form-urlencoded\r\n"
+           "Content-length: %d\r\n\r\n%s" % (len(body), body))
+s = socket.create_connection(("127.0.0.1", port), 10)
+s.settimeout(15)
+s.sendall(req.encode())
+out = b""
+try:
+    while True:
+        b = s.recv(65536)
+        if not b:
+            break
+        out += b
+except socket.timeout:
+    sys.stderr.write("timed out after %d bytes\n" % len(out))
+    sys.exit(9)
+s.close()
+sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
+}
+
+# GET $2, or fail with $3 when the reply never comes.
+get() {
+    local out
+    out=$(request "$1" "$2" "") || fail "$2 did not answer: $3"
+    echo "${out}"
+}
+
+post() { request "$1" "" "$2"; }
+
+url=$(start)
+port=$(portof "${url}")
+srv=$(srvpid)
+test -n "${srv}" || fail "htsserver did not report its pid"
+
+# Needs no session id and no project: html/server is a directory of the install.
+reply=$(get "${port}" /server/ "an unauthenticated request wedged the server")
+case "${reply}" in
+"HTTP/1.0 404 "*) ;;
+*) fail "a GUI directory did not answer 404: $(firstline "${reply}")" ;;
+esac
+
+# Every request body is gated by the session id.
+reply=$(get "${port}" /server/index.html "the GUI is not served")
+sid=$(firstline "$(echo "${reply}" |
+    sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p')")
+test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
+
+mkdir -p "${base}/proj/sub"
+echo "PROBEMARKER" >"${base}/proj/probe.txt"
+
+# mkfifo and ln -s may not work on a Windows checkout, so both stay optional.
+refuse=(/website/ /website/sub/)
+if mkfifo "${base}/proj/fifo.txt" 2>/dev/null; then
+    refuse+=(/website/fifo.txt)
+fi
+ln -s probe.txt "${base}/proj/link.txt" 2>/dev/null || true
+
+# step4's "save settings" flow registers the project without crawling, and that
+# is what arms the /website/ root.
+body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
+body="${body}&path=${base}&projname=proj"
+post "${port}" "${body}" >/dev/null || fail "the save request did not answer"
+test -f "${base}/proj/hts-cache/winprofile.ini" ||
+    fail "the project was not registered: $(cat "${log}")"
+
+# Positive control: every assertion below would pass on a server serving nothing.
+served() {
+    case "$(get "$1" "$2" "$3")" in
+    *PROBEMARKER*) ;;
+    *) fail "$3" ;;
+    esac
+}
+served "${port}" /website/probe.txt "the registered project's mirror is not served"
+if test -L "${base}/proj/link.txt"; then
+    # The guard stats rather than lstats, so a symlinked mirror file still serves.
+    served "${port}" /website/link.txt "a symlink to a mirror file is not served"
+fi
+
+# /website/ is where the GUI's own "browse mirrored site" link points.
+for path in "${refuse[@]}"; do
+    reply=$(get "${port}" "${path}" "the server wedged on ${path}")
+    case "${reply}" in
+    "HTTP/1.0 404 "*) ;;
+    *) fail "${path} did not answer 404: $(firstline "${reply}")" ;;
+    esac
+done
+
+# The accept loop is single-threaded: one spin denies every later request.
+alive "${srv}" || fail "the server died: $(cat "${log}")"
+served "${port}" /website/probe.txt "the server stopped answering after a directory request"
+
+# This fopen() is reached before any guard, so its read loop must stop on ferror().
+mkdir -p "${base}/proj2/hts-cache/winprofile.ini"
+reply=$(post "${port}" "sid=${sid}&path=${base}&loadprojname=proj2") ||
+    fail "a directory winprofile.ini wedged the server"
+case "${reply}" in
+"HTTP/1.0 3"*) ;;
+*) fail "loading a project did not redirect: $(firstline "${reply}")" ;;
+esac
+
+alive "${srv}" || fail "the server died: $(cat "${log}")"
+served "${port}" /website/probe.txt "the server stopped answering after a project load"
+
+echo "PASS"

tests/92_local-proxytrack-ndx-fields.test

@@ -0,0 +1,37 @@
+#!/bin/bash
+# The two halves of an .ndx entry's URL share one buffer, so the second must be
+# bounded by what the first left. The sanitizer CI build turns a regression
+# here into a hard stack-buffer-overflow.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+# each field fills binput's own bound, so together they run one past line[]
+"$python" - "$dir/foo.ndx" <<'EOF'
+import sys
+
+with open(sys.argv[1], "w") as f:
+    f.write("CACHE-1.4\n")
+    f.write("Wed, 01 Jan 2025 00:00:00 GMT\n")
+    f.write("h" * 1024 + "\n" + "f" * 1024 + "\n" + "0\n")
+    # a trailing entry keeps the walk inside the buffer, so this exercises the
+    # field bound alone
+    f.write("tail/entry\nx\n1\n")
+EOF
+: >"$dir/foo.dat" # the reader only proceeds when the sibling .dat opens
+
+proxytrack --convert "$dir/out.arc" "$dir/foo.ndx" >/dev/null 2>&1 || {
+    echo "FAIL: proxytrack crashed on an .ndx with two full-length URL fields" >&2
+    exit 1
+}

tests/Makefile.am

@@ -183,6 +183,12 @@ TESTS = \
 	83_webhttrack-argescape.test \
 	84_webhttrack-mirror-verbatim.test \
 	85_webhttrack-projpath.test \
-	86_local-proxytrack-cache-longfields.test
+	86_local-proxytrack-cache-longfields.test \
+	87_local-proxytrack-nodate.test \
+	88_local-proxytrack-badmtime.test \
+	89_webhttrack-error-overflow.test \
+	90_webhttrack-checkbox-clear.test \
+	91_webhttrack-directory.test \
+	92_local-proxytrack-ndx-fields.test
 
 CLEANFILES = check-network_sh.cache