A failed re-fetch overwrote the mirrored file with the aborted read's debris (#763)

* A failed re-fetch overwrote the mirrored file with the aborted read's debris

A transfer that dies before a complete response has no body, yet the save path
still consulted r.adr, which at that point holds whatever the aborted header
read left behind: raw status-line bytes, or an empty buffer that truncated the
file to zero on macOS. Require a successful transfer, as the empty-body half of
the condition already did.

Closes #748

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the failed re-fetch keeps its bytes on every platform

Test 93 filtered reset.bin out of its bucket lists because a connection killed
before the status line surfaced differently per platform. It no longer does, so
assert the resource like any other: unchanged, with the bytes pass 1 mirrored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: make the re-fetch assertions catch a crawl that never re-fetched

The new test passed with no second pass at all, so a regression that stopped
re-fetching would have looked green. Assert the failure the fixture provokes,
give stay.bin a fresh pass-2 body so a fix that stopped overwriting anything
fails, and compare reset.bin by checksum rather than by length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: assert the retry exhaustion, not the per-platform failure message

The message a cut connection produces depends on whether any bytes arrived, so
matching it would fail on a runner that sees none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

configure.ac

@@ -173,10 +173,7 @@ AC_CHECK_TYPE(sa_family_t, [], [AC_DEFINE([sa_family_t], [uint16_t], [sa_family_
 AX_CHECK_ALIGNED_ACCESS_REQUIRED
 
 # check for various headers
-AC_CHECK_HEADERS([execinfo.h sys/ioctl.h sys/random.h])
-
-## CSPRNG for the --single-file mark; /dev/urandom is the fallback
-AC_CHECK_FUNCS([getrandom])
+AC_CHECK_HEADERS([execinfo.h sys/ioctl.h])
 
 ### zlib (mandatory)
 CHECK_ZLIB()

fuzz/corpus/singlefile/img-src.html

@@ -1,4 +1,3 @@
-<img src="a.png#!htsinline">
-<img src=big.png#!htsinline alt=over-cap>
-<img src="../escape.png#!htsinline"><img src="/abs.png#!htsinline">
-<img src="data:,x#!htsinline">
+<img src="a.png">
+<img src=big.png alt=over-cap>
+<img src="../escape.png"><img src="/abs.png"><img src="data:,x">

fuzz/corpus/singlefile/link-rel.html

@@ -1,3 +1,4 @@
-<link rel="stylesheet" href="s.css#!htsinline">
-<link rel="icon" href="a.png#!htsinline">
-<link rel="canonical" href="other.html#!htsinline">
+<link rel="stylesheet" href="s.css">
+<link rel=icon href=a.png>
+<link rel="next" href="p2.html">
+<link rel="preload" href="j.js">

fuzz/corpus/singlefile/malformed.html

@@ -1,3 +1,5 @@
-<p a=1 b=2 c=3 d=4 e=5 f=6 g=7 h=8 i=9 j=10 title="> <img src=a.png#!htsinline>">x</p>
-<!--><img src="a.png#!htsinline">
-<img src="a.png#!htsinline" >
+<img src="unterminated.png
+<div style="background:url(a.png">
+<style>@import url(
+<!-- unterminated comment
+<a href=

fuzz/corpus/singlefile/many-attrs.html

@@ -0,0 +1 @@
+<img src="a.png" a0="v" a1="v" a2="v" a3="v" a4="v" a5="v" a6="v" a7="v" a8="v" a9="v" a10="v" a11="v" a12="v" a13="v" a14="v" a15="v" a16="v" a17="v" a18="v" a19="v" a20="v" a21="v" a22="v" a23="v" a24="v" a25="v" a26="v" a27="v" a28="v" a29="v" a30="v" a31="v" a32="v" a33="v" a34="v" a35="v" a36="v" a37="v" a38="v" a39="v" a40="v" a41="v" a42="v" a43="v" a44="v" a45="v" a46="v" a47="v" a48="v" a49="v" a50="v" a51="v" a52="v" a53="v" a54="v" a55="v" a56="v" a57="v" a58="v" a59="v" a60="v" a61="v" a62="v" a63="v" a64="v" a65="v" a66="v" a67="v" a68="v" a69="v">

fuzz/corpus/singlefile/mark-at-eof.html

@@ -1 +0,0 @@
-a.png#!htsinline

fuzz/corpus/singlefile/mark-degenerate.html

@@ -1,5 +0,0 @@
-#!htsinline#!htsinline#!htsinline
-=#!htsinline=
-"#!htsinline"
-a.png#!htsinlin
-#!htsinlin

fuzz/corpus/singlefile/mark-only.html

@@ -1 +0,0 @@
-#!htsinline

fuzz/corpus/singlefile/rawtext.html

@@ -1,3 +1,4 @@
-<script src="j.js#!htsinline"></script>
-<script>var u="a.png#!htsinline";</script>
-<textarea>a.png#!htsinline</textarea>
+<script>var s="</scripting>"; if(a</b) x=1;</script>
+<script src="j.js"></script>
+<textarea></textareas></textarea>
+<title></titles></title>

fuzz/corpus/singlefile/srcset.html

@@ -1 +1,2 @@
-<img srcset="a.png#!htsinline 1x, big.png#!htsinline 2x, a.png#!htsinline 480w">
+<img srcset="a.png 1x, big.png 2x, a.png 100w">
+<source srcset="a.png,, a.png 2x," src="a.png">

fuzz/corpus/singlefile/style-attr.html

@@ -1,2 +1,2 @@
-<div style="background:url(a.png#!htsinline)"></div>
-<div style='background:url("a.png#!htsinline")'></div>
+<div style="background:url(a.png);list-style:url('a.png')"></div>
+<p style='background:url("a.png")'>x</p>

fuzz/corpus/singlefile/style-block.html

@@ -1,4 +1,5 @@
-<style>@import "s.css#!htsinline";
-@import url(sub/b.css#!htsinline);
-div{background:url(a.png#!htsinline)}
+<style>@import "s.css";
+@import url(sub/b.css);
+div{background:url(a.png)}
+/* url(a.png) */ p:after{content:"url(a.png)"}
 </style>

fuzz/corpus/singlefile/traversal.html

@@ -1,4 +0,0 @@
-<img src="../../../../etc/passwd#!htsinline">
-<img src="a.png#!htsinline#frag?q=1">
-<img src="sub/../a.png#!htsinline">
-<img src="./a.png#!htsinline">

fuzz/fuzz-singlefile.c

@@ -95,13 +95,10 @@ static void sf_init(void) {
   sf_write("a.png", png, sizeof(png) - 1);
   sf_write("big.png", big, sizeof(big));
   sf_text("j.js", "var x=1;\n");
-  /* Marked, so an inlined stylesheet recurses into its own marks and its
-     un-inlinable reference is rebased; unmarked assets leave the target as a
-     bare scan that reaches nothing. */
-  sf_text("s.css", "@import url(sub/b.css" SINGLEFILE_MARK ");\n"
-                   "div{background:url(a.png" SINGLEFILE_MARK ")}\n"
-                   "p{background:url(big.png" SINGLEFILE_MARK ")}\n");
-  sf_text("sub/b.css", "p{background:url(../a.png" SINGLEFILE_MARK ")}\n");
+  /* @import plus a url(), so an inlined stylesheet recurses and its own
+     relative reference is rebased. */
+  sf_text("s.css", "@import url(sub/b.css);\ndiv{background:url(a.png)}\n");
+  sf_text("sub/b.css", "p{background:url(../a.png)}\n");
 }
 
 int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {

src/htscodec.c

@@ -72,7 +72,7 @@ hts_codec hts_codec_parse(const char *encoding) {
     return HTS_CODEC_IDENTITY;
   if (strfield2(encoding, "gzip") || strfield2(encoding, "x-gzip") ||
       strfield2(encoding, "deflate") || strfield2(encoding, "x-deflate"))
-    return HTS_USEZLIB ? HTS_CODEC_DEFLATE : HTS_CODEC_UNSUPPORTED;
+    return HTS_CODEC_DEFLATE;
   if (strfield2(encoding, "br"))
     return HTS_USEBROTLI ? HTS_CODEC_BROTLI : HTS_CODEC_UNSUPPORTED;
   if (strfield2(encoding, "zstd"))
@@ -98,16 +98,11 @@ hts_codec hts_codec_parse(const char *encoding) {
 const char *hts_acceptencoding(hts_boolean compressible, hts_boolean secure) {
   if (!compressible)
     return "identity";
-#if HTS_USEZLIB
   /* br and zstd over TLS only, as browsers do: a cleartext intermediary that
      rewrites a coding it can not read would corrupt the mirror. */
   if (secure)
     return "gzip, deflate" HTS_AE_BROTLI HTS_AE_ZSTD ", identity;q=0.9";
   return "gzip, deflate, identity;q=0.9";
-#else
-  (void) secure;
-  return "identity";
-#endif
 }
 
 hts_boolean hts_codec_is_archive_ext(hts_codec codec, const char *ext) {
@@ -300,11 +295,7 @@ int hts_codec_unpack(hts_codec codec, const char *filename,
     return -1;
   switch (codec) {
   case HTS_CODEC_DEFLATE:
-#if HTS_USEZLIB
     return hts_zunpack(filename, newfile);
-#else
-    return -1;
-#endif
   case HTS_CODEC_BROTLI:
   case HTS_CODEC_ZSTD:
     break;
@@ -348,10 +339,8 @@ size_t hts_codec_head(hts_codec codec, const void *in, size_t in_len, void *out,
   if (in == NULL || in_len == 0 || out == NULL || out_len == 0)
     return 0;
   switch (codec) {
-#if HTS_USEZLIB
   case HTS_CODEC_DEFLATE:
     return hts_zhead(in, in_len, out, out_len);
-#endif
 #if HTS_USEBROTLI
   case HTS_CODEC_BROTLI:
     return codec_head_brotli(in, in_len, out, out_len);

src/htscore.c

@@ -1980,10 +1980,9 @@ int httpmirror(char *url1, httrackp * opt) {
           }
 
           // ATTENTION C'EST ICI QU'ON SAUVE LE FICHIER!!
-          // An empty body must not overwrite the file when the transfer failed
-          // (statuscode <= 0, e.g. an -M hard-stop): it would truncate a good
-          // copy to 0 (#77 follow-up).
-          if (r.adr != NULL || (r.size == 0 && r.statuscode > 0)) {
+          // A failed transfer has no body: r.adr holds debris from the aborted
+          // read, which would destroy the copy being re-fetched (#748).
+          if (r.statuscode > 0 && (r.adr != NULL || r.size == 0)) {
             file_notify(opt, urladr(), urlfil(), savename(), 1, 1, r.notmodified);
             if (filesave(opt, r.adr, (int) r.size, savename(), urladr(), urlfil()) !=
                 0) {
@@ -2693,7 +2692,11 @@ HTSEXT_API int structcheck(const char *path) {
       if (!S_ISDIR(st.st_mode)) {
 #if HTS_REMOVE_ANNOYING_INDEX
         if (S_ISREG(st.st_mode)) {      /* Regular file in place ; move it and create directory */
-          sprintf(tmpbuf, "%s.txt", file);
+          /* bounded here, not by the path-length guard far above */
+          if (!sprintfbuff(tmpbuf, "%s.txt", file)) {
+            errno = ENAMETOOLONG;
+            return -1;
+          }
           if (rename(file, tmpbuf) != 0) {      /* Can't rename regular file */
             return -1;
           }
@@ -2801,7 +2804,11 @@ HTSEXT_API int structcheck_utf8(const char *path) {
       if (!S_ISDIR(st.st_mode)) {
 #if HTS_REMOVE_ANNOYING_INDEX
         if (S_ISREG(st.st_mode)) {      /* Regular file in place ; move it and create directory */
-          sprintf(tmpbuf, "%s.txt", file);
+          /* bounded here, not by the path-length guard far above */
+          if (!sprintfbuff(tmpbuf, "%s.txt", file)) {
+            errno = ENAMETOOLONG;
+            return -1;
+          }
           if (RENAME(file, tmpbuf) != 0) {      /* Can't rename regular file */
             return -1;
           }
@@ -3821,11 +3828,12 @@ int htsAddLink(htsmoduleStruct * str, char *link) {
         strcpybuff(codebase, heap(ptr)->fil);
       else
         strcpybuff(codebase, heap(heap(ptr)->precedent)->fil);
-      a = codebase + strlen(codebase) - 1;
+      // empty codebase has no last char; codebase-1 would underflow
+      a = codebase[0] != '\0' ? codebase + strlen(codebase) - 1 : codebase;
       while((*a) && (*a != '/') && (a > codebase))
         a--;
       if (*a == '/')
-        *(a + 1) = '\0';        // couper
+        *(a + 1) = '\0';        // cut
     } else {                    // couper http:// éventuel
       if (strfield(codebase, "http://")) {
         char BIGSTK tempo[HTS_URLMAXSIZE * 2];

src/htscoremain.c

@@ -1221,16 +1221,6 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                   opt->mimehtml = 0;
                   com++;
                 }
-                if (opt->mimehtml && opt->single_file) {
-                  HTS_PANIC_PRINTF(
-                      "-%M and --single-file are two ways to make one "
-                      "self-contained file, so pick one: MIME (-%M) carries "
-                      "text parts without the base64 tax and stores a shared "
-                      "asset once; single-file HTML opens anywhere by "
-                      "double-click.");
-                  htsmain_free();
-                  return -1;
-                }
                 break;
               case 'k':
                 opt->nokeepalive = 0;
@@ -1842,16 +1832,6 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                     com++;
                   }
                 }
-                if (opt->single_file && opt->mimehtml) {
-                  HTS_PANIC_PRINTF(
-                      "-%M and --single-file are two ways to make one "
-                      "self-contained file, so pick one: MIME (-%M) carries "
-                      "text parts without the base64 tax and stores a shared "
-                      "asset once; single-file HTML opens anywhere by "
-                      "double-click.");
-                  htsmain_free();
-                  return -1;
-                }
                 break;
               case 'm': // sitemap / sitemap-url: seed the crawl from sitemaps
                 if (*(com + 1) == 'u') { // --sitemap-url URL: explicit sitemap

src/htslib.c

@@ -1131,12 +1131,10 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
 
       // Compression accepted ?
       if (retour->req.http11) {
-        hts_boolean compressible = HTS_FALSE;
+        hts_boolean compressible =
+            (!retour->req.range_used && !retour->req.nocompression);
         hts_boolean secure = HTS_FALSE;
 
-#if HTS_USEZLIB
-        compressible = (!retour->req.range_used && !retour->req.nocompression);
-#endif
 #if HTS_USEOPENSSL
         secure = retour->ssl ? HTS_TRUE : HTS_FALSE;
 #endif
@@ -6037,7 +6035,6 @@ HTSEXT_API httrackp *hts_create_opt(void) {
   opt->changes_state = NULL;
   opt->single_file = HTS_FALSE;
   opt->single_file_max_size = SINGLEFILE_DEFAULT_MAX_SIZE;
-  opt->singlefile_state = NULL;
   StringCopy(opt->why_url, "");
   opt->pause_min_ms = 0;
   opt->pause_max_ms = 0;
@@ -6192,7 +6189,6 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
     StringFree(opt->warc_file);
     StringFree(opt->sitemap_url);
     hts_sitemap_free(opt); /* backstop: httpmirror's early-return paths */
-    singlefile_free(opt);
 
     hts_changes_free_opt(opt);
 

src/htsname.c

@@ -43,9 +43,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsencoding.h"
 #include "htssniff.h"
 #include "htscodec.h"
-#if HTS_USEZLIB
 #include "htszlib.h"
-#endif
 #include <ctype.h>
 #include <limits.h>
 

src/htsopt.h

@@ -564,8 +564,6 @@ struct httrackp {
      rather than in htsoptstate because that struct is embedded by value, so
      growing it would shift every httrackp field declared after it. */
   void *sitemap_state; /**< hts_sitemap_state*, or NULL. Tail: ABI */
-  void *singlefile_state; /**< hts_singlefile_state*, or NULL; holds this run's
-                               mark secret, never persisted. Tail: ABI */
 };
 
 /* Running statistics for a mirror. */

src/htsparse.c

@@ -59,7 +59,6 @@ Please visit our Website: http://www.httrack.com
 
 // parser
 #include "htsparse.h"
-#include "htssinglefile.h"
 #include "htsback.h"
 
 // arrays
@@ -385,7 +384,6 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
     FILE *fp = NULL;                  // fichier écrit localement 
     const char *html = r->adr;        // pointeur (on parcours)
     const char *lastsaved;            // adresse du dernier octet sauvé + 1
-    hts_boolean sf_may_mark = HTS_FALSE; // --single-file: mark this one?
 
     hts_log_print(opt, LOG_DEBUG, "scanning file %s%s (%s)..", urladr(), urlfil(),
                   savename());
@@ -401,13 +399,6 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
           r->adr[i] = ' ';
         }
       }
-      /* --single-file: RFC 2046 makes the generator responsible for a
-         boundary that cannot collide with what it encapsulates. Same duty
-         here, and the same negligible odds: only chance can put this run's
-         secret in a fetched document, since no site can spell it. */
-      if (opt->single_file) {
-        sf_may_mark = singlefile_may_mark(opt, r->adr, (size_t) r->size);
-      }
     }
 
     // Indexing!
@@ -3052,32 +3043,10 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
 
                         if ((opt->getmode & HTS_GETMODE_HTML) && (ptr > 0)) {
 
-                          /* --single-file: tag the reference for the
-                             end-of-mirror pass. A fragment, so a mirror left
-                             marked by an interrupted run still browses. */
-                          const char sf_class =
-                              (opt->single_file && sf_may_mark && !in_media &&
-                               p_type == 0 && !p_searchMETAURL)
-                                  ? singlefile_ref_class(
-                                        intag_start_valid ? intag_name : NULL,
-                                        tag_attr_start)
-                                  : 0;
-                          const size_t sf_start = TypedArraySize(output_buffer);
-
                           // écrire le lien modifié, relatif
                           // Note: escape all chars, even >127 (no UTF)
                           HT_ADD_HTMLESCAPED_FULL(tempo);
 
-                          /* Measured on what was appended, not on tempo: the
-                             escape above is free to change the length. */
-                          if (sf_class != 0) {
-                            char sf_mark[SINGLEFILE_MARK_MAX];
-
-                            HT_ADD(singlefile_mark(
-                                opt, sf_mark, sizeof(sf_mark), sf_class,
-                                TypedArraySize(output_buffer) - sf_start));
-                          }
-
                           // Add query-string, for informational purpose only
                           // Useless, because all parameters-pages are saved into different targets
                           if (opt->includequery) {

src/htsselftest.c

@@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsglobal.h"
 #include "htscore.h"
+#include "htsmodules.h"
 #include "htsback.h"
 #include "htsdefines.h"
 #include "htslib.h"
@@ -63,9 +64,7 @@ Please visit our Website: http://www.httrack.com
 #include "htswarc.h"
 #include "htschanges.h"
 #include "htssinglefile.h"
-#if HTS_USEZLIB
 #include "htszlib.h"
-#endif
 #if HTS_USEZSTD
 #include <zstd.h>
 #endif
@@ -2623,6 +2622,68 @@ static int st_savename(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* an empty fil started htsAddLink's codebase walk before the buffer (#730) */
+static int st_addlink(httrackp *opt, int argc, char **argv) {
+  htsmoduleStruct BIGSTK str;
+  cache_back cache;
+  struct_back *sback;
+  hash_struct hash;
+  int ptr = 0;
+  int i;
+
+  (void) argc;
+  (void) argv;
+
+  memset(&cache, 0, sizeof(cache));
+  cache.hashtable = (void *) coucal_new(0);
+  sback = back_new(opt, opt->maxsoc * 32 + 1024);
+  /* same wiring as hts_mirror (htscore.c) */
+  hash_init(opt, &hash, opt->urlhack);
+  hash.liens = (const lien_url *const *const *) &opt->liens;
+  opt->hash = &hash;
+  hts_record_init(opt);
+
+  memset(&str, 0, sizeof(str));
+  str.opt = opt;
+  str.sback = sback;
+  str.cache = &cache;
+  str.hashptr = &hash;
+  str.ptr_ = &ptr;
+  str.addLink = htsAddLink;
+
+  /* [0] is the underflow; [1] and [2] are controls that the trim is unchanged.
+     A query-only link is the one that notices the trim at all: for the others
+     ident_url_relatif() re-derives the directory from the path it is given. */
+  for (i = 0; i < 3; i++) {
+    static const char *const fil[3] = {"", "/dir/page.html", "/dir/page.html"};
+    static const char *const lnk[3] = {"sub/page.html", "sub/page.html",
+                                       "?x=1"};
+    static const char *const want[3] = {
+        "untouched", "http://www.example.com/dir/sub/page.html",
+        "http://www.example.com/dir/?x=1"};
+    char BIGSTK loc[HTS_URLMAXSIZE * 2];
+    char BIGSTK link[HTS_URLMAXSIZE];
+
+    strcpybuff(loc, "untouched");
+    strcpybuff(link, lnk[i]);
+    str.localLink = loc;
+    str.localLinkSize = (int) sizeof(loc);
+    if (!hts_record_link(opt, "www.example.com", fil[i], "", "", "", ""))
+      return 1;
+    ptr = heap_top_index();
+    str.url_host = heap(ptr)->adr;
+    str.url_file = heap(ptr)->fil;
+    assertf(htsAddLink(&str, link) == 0); /* refused by the wizard either way */
+    if (strcmp(loc, want[i]) != 0) {
+      fprintf(stderr, "addlink[%d]: got '%s' want '%s'\n", i, loc, want[i]);
+      return 1;
+    }
+  }
+
+  printf("addlink self-test OK\n");
+  return 0;
+}
+
 static int st_cache(httrackp *opt, int argc, char **argv) {
   int err;
 
@@ -3264,6 +3325,81 @@ static int st_topindex(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+/* Build a path of exactly len chars under base; returns that length. */
+static size_t st_structcheck_longpath(char *dst, size_t dstsize,
+                                      const char *base, size_t len) {
+  size_t n = strlen(base);
+
+  assertf(len < dstsize && n + 2 <= len);
+  memmove(dst, base, n);
+  while (n < len) {
+    size_t seg = len - n - 1;
+
+    if (seg > 200) /* stay under the usual 255-byte component limit */
+      seg = len - n == 202 ? 199 : 200; /* never leave a bare separator */
+    dst[n++] = '/';
+    memset(dst + n, 'x', seg);
+    n += seg;
+  }
+  dst[n] = '\0';
+  return n;
+}
+
+/* The path guard, and the <name>.txt rename structcheck() performs when a
+   regular file sits where a directory has to go (#745). */
+static int st_structcheck(httrackp *opt, int argc, char **argv) {
+  char BIGSTK path[HTS_URLMAXSIZE * 2];
+  char BIGSTK target[HTS_URLMAXSIZE * 2];
+  FILE *fp;
+
+  (void) opt;
+  if (argc < 1) {
+    fprintf(stderr, "usage: -#test=structcheck <writable directory>\n");
+    return 1;
+  }
+
+  /* over the guard: refused before a single directory is created */
+  st_structcheck_longpath(path, sizeof(path), argv[0], HTS_URLMAXSIZE + 1);
+  errno = 0;
+  assertf(structcheck(path) == -1);
+  assertf(errno == EINVAL);
+  errno = 0;
+  assertf(structcheck_utf8(path) == -1);
+  assertf(errno == EINVAL);
+  {
+    char *const sep = strchr(path + strlen(argv[0]) + 1, '/');
+
+    assertf(sep != NULL);
+    sep[1] = '\0'; /* the outermost component it would have created */
+    assertf(!dir_exists(path));
+  }
+
+  /* a regular file where a directory belongs is renamed to <name>.txt */
+  snprintf(path, sizeof(path), "%s/sc", argv[0]);
+  fp = fopen(path, "wb");
+  assertf(fp != NULL);
+  fclose(fp);
+  snprintf(path, sizeof(path), "%s/sc/sub/", argv[0]);
+  assertf(structcheck(path) == 0);
+  assertf(dir_exists(path));
+  snprintf(target, sizeof(target), "%s/sc.txt", argv[0]);
+  assertf(fexist(target));
+
+  /* the utf-8 entry point carries the same rename */
+  snprintf(path, sizeof(path), "%s/u8", argv[0]);
+  fp = FOPEN(path, "wb");
+  assertf(fp != NULL);
+  fclose(fp);
+  snprintf(path, sizeof(path), "%s/u8/sub/", argv[0]);
+  assertf(structcheck_utf8(path) == 0);
+  assertf(dir_exists(path));
+  snprintf(target, sizeof(target), "%s/u8.txt", argv[0]);
+  assertf(fexist_utf8(target));
+
+  printf("structcheck self-test OK\n");
+  return 0;
+}
+
 /* Each inplace_escape_*() must equal escape_*() on a copy. */
 static int st_inplace_escape(httrackp *opt, int argc, char **argv) {
   /* >255 bytes forces the helper's malloct path, not the stack buffer */
@@ -3377,7 +3513,6 @@ static int st_status(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
-#if HTS_USEZLIB
 /* Deflate src->path at windowBits (16+ gzip, + zlib, - raw); 0 on success. */
 static int ae_write_packed(const char *path, int windowBits,
                            const unsigned char *src, size_t len) {
@@ -3451,7 +3586,6 @@ static int ae_write_collision(const char *path, const unsigned char *src,
   freet(buf);
   return ok ? 0 : 1;
 }
-#endif
 
 /* Write src[0..len) to path as-is; 0 on success. */
 static int ae_write_raw(const char *path, const unsigned char *src,
@@ -3501,7 +3635,6 @@ static int st_acceptencoding(httrackp *opt, int argc, char **argv) {
   assertf(strstr(on, "br") == NULL && strstr(on, "zstd") == NULL);
   assertf((strstr(tls, ", br") != NULL) == (HTS_USEBROTLI != 0));
   assertf((strstr(tls, "zstd") != NULL) == (HTS_USEZSTD != 0));
-#if HTS_USEZLIB
   if (argc >= 1) {
     static const int windowBits[] = {16 + MAX_WBITS, MAX_WBITS, -MAX_WBITS};
     const unsigned char small[] =
@@ -3580,10 +3713,6 @@ static int st_acceptencoding(httrackp *opt, int argc, char **argv) {
     }
     freet(body);
   }
-#else
-  (void) argc;
-  (void) argv;
-#endif
   printf("acceptencoding self-test OK: %s\n", on);
   return 0;
 }
@@ -3928,7 +4057,6 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) {
     freet(big);
   }
 
-#if HTS_USEZLIB
   /* A highly compressible document decodes without running away: the ratio
      budget cannot bind (deflate tops out near 1032:1), so this pins the
      decompression path itself rather than the 64 MiB ceiling. */
@@ -3979,13 +4107,11 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) {
     freet(z);
     freet(x);
   }
-#endif
 
   /* An unterminated <loc> at end of buffer must not read past it. */
   assertf(sm_scan("<urlset><loc>http://h.test/a", 100, &idx, &c) == 0);
   assertf(sm_scan("<urlset><lo", 100, &idx, &c) == 0);
 
-#if HTS_USEZLIB
   /* A gzip-framed document is decompressed before scanning. */
   {
     const char *const xml =
@@ -4015,7 +4141,6 @@ static int st_sitemap(httrackp *opt, int argc, char **argv) {
     assertf(hts_sitemap_scan(z, 4, 100, &idx, sm_take, &c) == -1);
     freet(z);
   }
-#endif
 
   /* robots.txt: only Sitemap: records, comments stripped, case-insensitive,
      and group-independent (no User-agent line needed). */
@@ -5201,47 +5326,11 @@ static hts_boolean sf_try_put(const char *dir, const char *rel,
   return HTS_TRUE;
 }
 
-/* Expand a fixture: \001<ref>\002 becomes <ref> plus this run's mark for it.
-   The secret is random per run, so a fixture cannot spell a mark itself. */
-static void sf_expand_fixture(httrackp *opt, const char *in, size_t len,
-                              String *out) {
-  size_t i, start = 0;
-
-  StringClear(*out);
-  for (i = 0; i < len; i++) {
-    if (in[i] == '\001') {
-      start = StringLength(*out);
-    } else if (in[i] == '\002') {
-      char mark[SINGLEFILE_MARK_MAX];
-
-      StringCat(*out,
-                singlefile_mark(opt, mark, sizeof(mark), SINGLEFILE_CLASS_ANY,
-                                StringLength(*out) - start));
-    } else {
-      StringAddchar(*out, in[i]);
-    }
-  }
-}
-
 static void sf_put(const char *dir, const char *rel, const void *data,
                    size_t len) {
   assertf(sf_try_put(dir, rel, data, len));
 }
 
-/* sf_put() for a text fixture, expanding its \001<ref>\002 delimiters. Binary
-   fixtures must not go through here: sf_png carries those very bytes. */
-static size_t sf_put_marked(httrackp *opt, const char *dir, const char *rel,
-                            const void *data, size_t len) {
-  String body = STRING_EMPTY;
-  size_t written;
-
-  sf_expand_fixture(opt, (const char *) data, len, &body);
-  assertf(sf_try_put(dir, rel, StringBuff(body), StringLength(body)));
-  written = StringLength(body);
-  StringFree(body);
-  return written;
-}
-
 /* Number of times needle occurs in hay. */
 static int sf_count(const char *hay, const char *needle) {
   const size_t l = strlen(needle);
@@ -5351,80 +5440,75 @@ static const char sf_png[] = "\x89PNG\r\n\x1a\n\x00\x01\x02\xff";
 
 static const char sf_page[] =
     "<html><head>\n"
-    "<link rel=\"stylesheet\" href=\"\001css/main.css\002\">\n"
-    "<link rel=\"canonical\" href=\"\001other.html\002\">\n"
+    "<link rel=\"stylesheet\" href=\"css/main.css\">\n"
+    "<link rel=\"canonical\" href=\"other.html\">\n"
     "<title>t</title>\n"
-    "<style>body { background: url(\"\001img/a%20b.png\002\"); }</style>\n"
+    "<style>body { background: url(\"img/a%20b.png\"); }</style>\n"
     "</head><body>\n"
-    "<img src=\"\001img/a%20b.png\002\" srcset=\"\001img/a%20b.png\002 "
-    "1x, \001img/big.png\002 2x\">\n"
-    "<link rel=\"icon\" href=\"\001icon.png\002\">\n"
-    "<link rel=\"preload\" as=\"font\" href=\"\001font/f.woff2\002\">\n"
+    "<img src=\"img/a%20b.png\" srcset=\"img/a%20b.png 1x, img/big.png 2x\">\n"
+    "<link rel=\"icon\" href=\"icon.png\">\n"
+    "<link rel=\"preload\" as=\"font\" href=\"font/f.woff2\">\n"
     "<img src=\"data:image/gif;base64,QUJD\">\n"
     /* Each has a real file where its guard's removal would land it; without
        that they stay links either way, the target merely being absent. */
     "<img src=\"http://example.com/x.png\">\n"
     "<img src=\"//example.com/x.png\">\n"
-    "<input type=\"image\" src=\"\001img/in.png\002\">\n"
+    "<input type=\"image\" src=\"img/in.png\">\n"
     /* Lazy loading: src is the placeholder, the real image rides data-src. */
-    "<img src=\"\001img/ph.png\002\" data-src=\"\001img/lz.png\002\" "
-    "data-srcset=\"\001img/lz2.png\002 2x\" "
-    "lowsrc=\"\001img/low.png\002\">\n"
-    "<object data=\"\001img/ob.png\002\"></object>\n"
-    "<embed src=\"\001img/em.png\002\">\n"
-    "<img data-src=\"\001other.html\002\">\n"
+    "<img src=\"img/ph.png\" data-src=\"img/lz.png\" "
+    "data-srcset=\"img/lz2.png 2x\" lowsrc=\"img/low.png\">\n"
+    "<object data=\"img/ob.png\"></object>\n"
+    "<embed src=\"img/em.png\">\n"
+    "<img data-src=\"other.html\">\n"
     /* What a first pass emits: re-resolving it is what a second pass must not
        do, and the fallback type would inline whatever the walk found. */
     "<link rel=\"stylesheet\" href=\"data:text/css;base64,QUJD\">\n"
-    "<video poster=\"\001img/po.png\002\" controls>"
-    "<source src=\"\001v.mp4\002\" type=\"video/mp4\"></video>\n"
-    "<svg><image href=\"\001img/sv.png\002\"/></svg>\n"
-    "<table background=\"\001img/bg.png\002\"><tr><td>x</td></tr></table>\n"
+    "<video poster=\"img/po.png\" controls>"
+    "<source src=\"v.mp4\" type=\"video/mp4\"></video>\n"
+    "<svg><image href=\"img/sv.png\"/></svg>\n"
+    "<table background=\"img/bg.png\"><tr><td>x</td></tr></table>\n"
     /* The second is what bites: drop the clamp and its leading ".." lands it
        back on <root>/img/a b.png. The first can only 404 either way. */
-    "<img src=\"\001../escape.png\002\">\n"
-    "<img src=\"\001../img/a%20b.png\002\">\n"
+    "<img src=\"../escape.png\">\n"
+    "<img src=\"../img/a%20b.png\">\n"
     "<a href=\"img/a%20b.png\">link</a>\n"
-    "<script src=\"\001js/app.js\002\"></script>\n"
+    "<script src=\"js/app.js\"></script>\n"
     "<script>var s = \"</scripting>\"; var t = \"<img src='img/a%20b.png'>\";"
     "</script>\n"
-    "<img src=\"\001missing.png\002\" >\n"
-    "<!--><img src=\"\001img/a%20b.png\002\">\n"
-    "<div style=\"background:url(\001img/a%20b.png\002)\"></div>\n"
-    "<div style='content:\"x\"; "
-    "background:url(\001img/a%20b.png\002)'></div>\n"
+    "<img src=\"missing.png\" >\n"
+    "<!--><img src=\"img/a%20b.png\">\n"
+    "<div style=\"background:url(img/a%20b.png)\"></div>\n"
+    "<div style='content:\"x\"; background:url(img/a%20b.png)'></div>\n"
     "</body></html>\n";
 
 /* Lay a small mirror down under root. */
-static void sf_fixture(httrackp *opt, const char *root) {
+static void sf_fixture(const char *root) {
   /* The over-cap url() is what drives the rebase fallback: a reference an
      inlined stylesheet could not embed has to come out relative to the page,
      not to the stylesheet, or it dangles. */
   static const char css[] =
-      "@import \"\001sub/nested.css\002\";\n"
-      "@import url(\"\001sub/two.css\002\");\n"
+      "@import \"sub/nested.css\";\n"
+      "@import url(\"sub/two.css\");\n"
       "@import \"a\\\"url(../img/a b.png)b.css\";\n"
-      "@font-face { font-family: f; src: url(\001../font/f.woff2\002); }\n"
-      "body { background: url(\001../img/a%20b.png\002); }\n"
-      "div { background: url(\001../img/big.png\002); }\n"
+      "@font-face { font-family: f; src: url(../font/f.woff2); }\n"
+      "body { background: url(../img/a b.png); }\n"
+      "div { background: url(../img/big.png); }\n"
       "/* url(../img/never.png) */\n";
-  static const char nested[] =
-      "div { background: url(\001../../img/a%20b.png\002); }\n";
-  static const char two[] =
-      "p { background: url(\001../../img/a%20b.png\002); }\n";
-  static const char deep[] = "<html><head><link rel=\"stylesheet\" "
-                             "href=\"\001../../css/main.css\002\">\n"
-                             "</head><body>d</body></html>\n";
+  static const char nested[] = "div { background: url(../../img/a b.png); }\n";
+  static const char two[] = "p { background: url(../../img/a b.png); }\n";
+  static const char deep[] =
+      "<html><head><link rel=\"stylesheet\" href=\"../../css/main.css\">\n"
+      "</head><body>d</body></html>\n";
   static const char js[] = "var app = 1;\n";
   char big[4096];
 
   memset(big, 'B', sizeof(big));
-  sf_put_marked(opt, root, "page.html", sf_page, sizeof(sf_page) - 1);
-  sf_put_marked(opt, root, "deep/sub/page.html", deep, sizeof(deep) - 1);
+  sf_put(root, "page.html", sf_page, sizeof(sf_page) - 1);
+  sf_put(root, "deep/sub/page.html", deep, sizeof(deep) - 1);
   sf_put(root, "other.html", "<html>o</html>", 14);
-  sf_put_marked(opt, root, "css/main.css", css, sizeof(css) - 1);
-  sf_put_marked(opt, root, "css/sub/nested.css", nested, sizeof(nested) - 1);
-  sf_put_marked(opt, root, "css/sub/two.css", two, sizeof(two) - 1);
+  sf_put(root, "css/main.css", css, sizeof(css) - 1);
+  sf_put(root, "css/sub/nested.css", nested, sizeof(nested) - 1);
+  sf_put(root, "css/sub/two.css", two, sizeof(two) - 1);
   sf_put(root, "js/app.js", js, sizeof(js) - 1);
   sf_put(root, "img/a b.png", sf_png, SF_PNG_LEN);
   sf_put(root, "img/big.png", big, sizeof(big));
@@ -5462,7 +5546,7 @@ static void sf_fixture(httrackp *opt, const char *root) {
     }
     StringCat(wide,
               " title=\"> <img src=img/a%20b.png> \">end</p></body></html>");
-    sf_put_marked(opt, root, "wide.html", StringBuff(wide), StringLength(wide));
+    sf_put(root, "wide.html", StringBuff(wide), StringLength(wide));
     StringFree(wide);
   }
   sf_put(root, "v.mp4",
@@ -5488,7 +5572,7 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
   sf_put(argv[0], "escape.png", sf_png,
          SF_PNG_LEN); /* just outside the mirror */
   fconcat(root, sizeof(root), argv[0], "mirror/");
-  sf_fixture(opt, root);
+  sf_fixture(root);
   fconcat(page, sizeof(page), root, "page.html");
 
   /* Cap between the small assets and big.png. */
@@ -5556,10 +5640,13 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
   sf_check(strstr(out, "var t = \"<img src='img/a%20b.png'>\";") != NULL,
            "script body rewritten past a </scripting> lookalike");
 
-  /* Only the marked reference is touched: the value keeps its own quoting, so
-     nothing can be emitted that the attribute could not already hold. */
-  sf_check(strstr(out, "style='content:\"x\"; background:url(data:") != NULL,
-           "style attribute re-quoted instead of substituted in place");
+  /* Nothing an attribute value cannot hold: url() stays unquoted, and a quote
+     that was already in the CSS is escaped. */
+  sf_check(strstr(out, "url(\"data:") == NULL,
+           "a quoted url() would end a style attribute");
+  sf_check(strstr(out, "style=\"content:&quot;x&quot;; background:url(data:") !=
+               NULL,
+           "quote inside a rewritten style attribute not escaped");
 
   sf_check(strstr(out, "img/big.png 2x") != NULL, "over-cap asset inlined");
   sf_check(strstr(out, " 1x") != NULL, "srcset descriptor lost");
@@ -5579,7 +5666,7 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
              "url() inside a CSS comment was rewritten");
     sf_check(strstr(css, "url(data:font/woff2;base64,") != NULL,
              "@font-face src not inlined");
-    sf_check(strstr(css, "@import url(\"data:text/css;base64,") != NULL,
+    sf_check(strstr(css, "@import url(data:text/css;base64,") != NULL,
              "@import url() form not inlined");
     sf_check(strstr(css, "url(../img/a b.png)b.css") != NULL,
              "url() inside a string with an escaped quote was rewritten");
@@ -5643,7 +5730,7 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
 
   /* Same page, a cap above big.png: it now inlines. */
   fconcat(root, sizeof(root), argv[0], "mirror2/");
-  sf_fixture(opt, root);
+  sf_fixture(root);
   fconcat(page, sizeof(page), root, "page.html");
   opt->single_file_max_size = 1024 * 1024;
   sf_check(singlefile_rewrite_file(opt, root, page),
@@ -5657,37 +5744,21 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
   /* Nothing inlines, so the rewriter must be byte transparent. Assert on its
      output: the file it declined to write could not have changed regardless. */
   fconcat(root, sizeof(root), argv[0], "mirror3/");
-  sf_fixture(opt, root);
+  sf_fixture(root);
   fconcat(page, sizeof(page), root, "page.html");
   opt->single_file_max_size = 1;
-  /* The page is still rewritten: nothing inlines, but the marks must go. */
-  sf_check(singlefile_rewrite_file(opt, root, page),
-           "one-byte cap left the marks in place");
-  {
-    char *capped = readfile_utf8(page);
-
-    /* Only the two data: URIs the fixture itself ships. */
-    sf_check(capped != NULL && sf_count(capped, ";base64,") == 2,
-             "one-byte cap inlined");
-    freet(capped);
-  }
+  sf_check(!singlefile_rewrite_file(opt, root, page), "one-byte cap inlined");
   {
     String verbatim = STRING_EMPTY;
 
     StringClear(verbatim);
-    String marked = STRING_EMPTY;
-
-    sf_expand_fixture(opt, sf_page, sizeof(sf_page) - 1, &marked);
-    (void) singlefile_rewrite_html(opt, root, page, StringBuff(marked),
-                                   StringLength(marked),
+    (void) singlefile_rewrite_html(opt, root, page, sf_page,
+                                   sizeof(sf_page) - 1,
                                    SINGLEFILE_MAX_PAGE_SIZE, &verbatim);
-    /* Mark-transparent, not byte-transparent: a reference that cannot be
-       inlined loses its mark and keeps everything else. */
-    sf_check(StringLength(verbatim) < StringLength(marked),
-             "a page with nothing to inline kept its marks");
-    sf_check(strstr(StringBuff(verbatim), singlefile_intro(opt)) == NULL,
-             "an un-inlinable reference kept its mark");
-    StringFree(marked);
+    sf_check(StringLength(verbatim) == sizeof(sf_page) - 1 &&
+                 memcmp(StringBuff(verbatim), sf_page, sizeof(sf_page) - 1) ==
+                     0,
+             "a page with nothing to inline was re-serialized differently");
     StringFree(verbatim);
   }
   (void) outlen;
@@ -5696,29 +5767,26 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
      run is the control: it proves the fan-out is real, so the small one was
      cut short by the budget and not by the fixture. */
   {
-    static const char bomb_css[] =
-        "@import \"\001b.css\002\";@import \"\001b.css\002\";"
-        "@import \"\001b.css\002\";@import \"\001b.css\002\";\n";
-    static const char bomb_html[] =
-        "<html><head>"
-        "<link rel=\"stylesheet\" href=\"\001b.css\002\">"
-        "</head></html>\n";
-    size_t css_len;
-    String small = STRING_EMPTY, large = STRING_EMPTY, bomb = STRING_EMPTY;
+    static const char bomb_css[] = "@import \"b.css\";@import \"b.css\";"
+                                   "@import \"b.css\";@import \"b.css\";\n";
+    static const char bomb_html[] = "<html><head>"
+                                    "<link rel=\"stylesheet\" href=\"b.css\">"
+                                    "</head></html>\n";
+    const size_t css_len = sizeof(bomb_css) - 1;
+    String small = STRING_EMPTY, large = STRING_EMPTY;
 
     fconcat(root, sizeof(root), argv[0], "bomb/");
-    css_len = sf_put_marked(opt, root, "b.css", bomb_css, sizeof(bomb_css) - 1);
+    sf_put(root, "b.css", bomb_css, css_len);
     fconcat(page, sizeof(page), root, "page.html");
     opt->single_file_max_size = 1024 * 1024;
     StringClear(small);
     StringClear(large);
-    sf_expand_fixture(opt, bomb_html, sizeof(bomb_html) - 1, &bomb);
-    (void) singlefile_rewrite_html(opt, root, page, StringBuff(bomb),
-                                   StringLength(bomb), (LLint) css_len * 3,
+    (void) singlefile_rewrite_html(opt, root, page, bomb_html,
+                                   sizeof(bomb_html) - 1, (LLint) css_len * 3,
                                    &small);
-    (void) singlefile_rewrite_html(opt, root, page, StringBuff(bomb),
-                                   StringLength(bomb), SINGLEFILE_MAX_PAGE_SIZE,
-                                   &large);
+    (void) singlefile_rewrite_html(opt, root, page, bomb_html,
+                                   sizeof(bomb_html) - 1,
+                                   SINGLEFILE_MAX_PAGE_SIZE, &large);
     sf_check(StringLength(large) > 4096, "the @import bomb did not fan out");
     sf_check(StringLength(small) < StringLength(large) / 8,
              "the per-page budget did not cut the fan-out short");
@@ -5728,7 +5796,6 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
              "the per-page budget was not charged as each asset was taken");
     StringFree(small);
     StringFree(large);
-    StringFree(bomb);
   }
 
   if (!sf_colon_ok)
@@ -6147,6 +6214,116 @@ static int st_changes(httrackp *opt, int argc, char **argv) {
   return err;
 }
 
+/* #747: a thread is outstanding from the moment hts_newthread() returns, not
+   from the moment it starts running, or a wait right after the spawn joins
+   nothing. One thread per round is what makes the old bug visible: the wait
+   had to find the counter at zero, and a batch of spawns gives the earlier
+   threads time to raise it. Unfixed, one round in two caught it, so the round
+   count is what turns that into a reliable failure. */
+#define THREADWAIT_N 8
+#define THREADWAIT_ROUNDS 16
+#define THREADWAIT_SLEEP_MS 50
+#define THREADWAIT_GATE_MS 10000
+
+static htsmutex threadwait_lock = HTSMUTEX_INIT;
+static int threadwait_done = 0;
+static hts_boolean threadwait_gated = HTS_FALSE;
+
+static int threadwait_count(void) {
+  int n;
+
+  hts_mutexlock(&threadwait_lock);
+  n = threadwait_done;
+  hts_mutexrelease(&threadwait_lock);
+  return n;
+}
+
+static void threadwait_thread(void *arg) {
+  (void) arg;
+  Sleep(THREADWAIT_SLEEP_MS);
+  hts_mutexlock(&threadwait_lock);
+  threadwait_done++;
+  hts_mutexrelease(&threadwait_lock);
+}
+
+/* Stays outstanding until the gate clears, so wait_n() can be asked to leave a
+   known number of live threads behind. Bounded: a wait_n() that wrongly drains
+   them would otherwise never return, and hang the suite instead of failing. */
+static void threadwait_gated_thread(void *arg) {
+  int waited;
+
+  (void) arg;
+  for (waited = 0; waited < THREADWAIT_GATE_MS; waited += 10) {
+    hts_boolean gated;
+
+    hts_mutexlock(&threadwait_lock);
+    gated = threadwait_gated;
+    hts_mutexrelease(&threadwait_lock);
+    if (!gated)
+      break;
+    Sleep(10);
+  }
+  hts_mutexlock(&threadwait_lock);
+  threadwait_done++;
+  hts_mutexrelease(&threadwait_lock);
+}
+
+static int st_threadwait(httrackp *opt, int argc, char **argv) {
+  int err = 0;
+  int i, round;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+  /* htsthread_wait() joins a thread spawned just before it */
+  for (round = 0; round < THREADWAIT_ROUNDS && !err; round++) {
+    hts_mutexlock(&threadwait_lock);
+    threadwait_done = 0;
+    hts_mutexrelease(&threadwait_lock);
+    if (hts_newthread(threadwait_thread, NULL) != 0) {
+      fprintf(stderr, "threadwait: cannot spawn\n");
+      return 1;
+    }
+    htsthread_wait();
+    if (threadwait_count() != 1) {
+      fprintf(stderr, "threadwait: round %d returned before the thread ran\n",
+              round);
+      err = 1;
+    }
+  }
+
+  /* htsthread_wait_n(n) leaves n behind rather than draining everything */
+  hts_mutexlock(&threadwait_lock);
+  threadwait_done = 0;
+  threadwait_gated = HTS_TRUE;
+  hts_mutexrelease(&threadwait_lock);
+  for (i = 0; i < THREADWAIT_N; i++) {
+    if (hts_newthread(threadwait_gated_thread, NULL) != 0) {
+      fprintf(stderr, "threadwait: cannot spawn a gated thread\n");
+      return 1;
+    }
+  }
+  htsthread_wait_n(THREADWAIT_N);
+  if (threadwait_count() != 0) {
+    fprintf(stderr, "threadwait: wait_n(%d) joined %d gated threads\n",
+            THREADWAIT_N, threadwait_count());
+    err = 1;
+  }
+  hts_mutexlock(&threadwait_lock);
+  threadwait_gated = HTS_FALSE;
+  hts_mutexrelease(&threadwait_lock);
+  htsthread_wait();
+  if (threadwait_count() != THREADWAIT_N) {
+    fprintf(stderr, "threadwait: wait left %d/%d gated threads running\n",
+            THREADWAIT_N - threadwait_count(), THREADWAIT_N);
+    err = 1;
+  }
+
+  printf("threadwait self-test: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
 #define CHANGES_RACE_FILES 8
 #define CHANGES_RACE_ROUNDS 400
 
@@ -6161,11 +6338,8 @@ static void changes_race_notify(httrackp *opt, int n) {
   hts_changes_notify(opt, "race.example", fil, save, HTS_TRUE, HTS_FALSE);
 }
 
-/* htsthread_wait() counts a thread only once it is running, so it can return
-   before any of them started; join on our own counter instead. */
 static htsmutex changes_race_lock = HTSMUTEX_INIT;
 static int changes_race_started = 0;
-static int changes_race_live = 0;
 
 static int changes_race_count(int *which) {
   int n;
@@ -6185,9 +6359,6 @@ static void changes_race_thread(void *arg) {
   hts_mutexrelease(&changes_race_lock);
   for (i = 0; i < CHANGES_RACE_ROUNDS; i++)
     changes_race_notify(opt, i % CHANGES_RACE_FILES);
-  hts_mutexlock(&changes_race_lock);
-  changes_race_live--;
-  hts_mutexrelease(&changes_race_lock);
 }
 
 /* A transfer thread the crawl never joins (FTP) reaches hts_changes_notify()
@@ -6242,7 +6413,6 @@ static int st_changes_race(httrackp *opt, int argc, char **argv) {
      threads reaching a fresh one together race on the init itself. */
   hts_mutexlock(&changes_race_lock);
   changes_race_started = 0;
-  changes_race_live = 4;
   hts_mutexrelease(&changes_race_lock);
   for (i = 0; i < 4; i++) {
     if (hts_newthread(changes_race_thread, opt) != 0) {
@@ -6256,8 +6426,7 @@ static int st_changes_race(httrackp *opt, int argc, char **argv) {
   for (i = 0; i < 64; i++)
     hts_changes_report(opt, &out);
   hts_changes_close_opt(opt);
-  while (changes_race_count(&changes_race_live) > 0)
-    Sleep(10);
+  htsthread_wait();
 
   /* Sealed: a straggler must be dropped, not start a report nobody writes. */
   changes_race_notify(opt, CHANGES_RACE_FILES + 1);
@@ -6335,6 +6504,8 @@ static const struct selftest_entry {
      st_changes},
     {"changes-race", "<dir>", "--changes under a late transfer thread (#714)",
      st_changes_race},
+    {"threadwait", "", "htsthread_wait() joins threads spawned just before it",
+     st_threadwait},
     {"pause", "", "randomized inter-file pause target self-test", st_pause},
     {"relative", "<link> <curr-file>", "relative link between two paths",
      st_relative},
@@ -6365,6 +6536,8 @@ static const struct selftest_entry {
     {"fsize", "<dir>", "file size past the 2GB signed-32-bit wrap", st_fsize},
     {"growsize", "", "buffer capacity for a 64-bit file size (no int wrap)",
      st_growsize},
+    {"addlink", "", "htsAddLink codebase walk over an empty current path",
+     st_addlink},
     {"cache", "<dir>", "cache read/write round-trip self-test", st_cache},
     {"cacheindex", "", "cache-index (.ndx) parse must stay in bounds",
      st_cacheindex},
@@ -6388,6 +6561,9 @@ static const struct selftest_entry {
     {"useragent", "", "default User-Agent self-test", st_useragent},
     {"makeindex", "[dir]", "hts_finish_makeindex footer/refresh self-test",
      st_makeindex},
+    {"structcheck", "<dir>",
+     "structcheck path guard and the <name>.txt rename it performs",
+     st_structcheck},
     {"topindex", "[dir]",
      "hts_buildtopindex charset handling of a non-ASCII project dir",
      st_topindex},

src/htssinglefile.h

@@ -55,62 +55,18 @@ extern "C" {
    so a few hundred bytes of hostile CSS can otherwise ask for gigabytes. */
 #define SINGLEFILE_MAX_PAGE_SIZE (64 * 1024 * 1024)
 
-/* The mark htsparse appends to a saved reference the pass may inline:
-
-     #!<16-hex secret>.<class>.<len>
-
-   A fragment, so a mirror left marked by an interrupted run still browses.
-   The secret is 64 CSPRNG bits drawn once per run and never written to disk,
-   which is what makes the mark unforgeable: a site cannot spell one, so no
-   sanitiser has to keep hostile bytes away from it. <len> is the byte length
-   of the reference text immediately preceding the mark, so the pass never has
-   to guess where that reference starts. <class> is the context htsparse saw,
-   checked against the resolved type so a mismatch fails loudly. */
-#define SINGLEFILE_MARK_INTRO "#!"
-#define SINGLEFILE_SECRET_HEX 16
-
-/* <class>: what the referencing context expects. */
-#define SINGLEFILE_CLASS_ANY '-'
-#define SINGLEFILE_CLASS_CSS 'c'
-#define SINGLEFILE_CLASS_JS 'j'
-
-/* Enough for the intro, the secret, both separators and a 20-digit length. */
-#define SINGLEFILE_MARK_MAX 64
-
-/* Release this run's secret. */
-void singlefile_free(httrackp *opt);
-
-/* "#!<secret>" for this run, or NULL if no CSPRNG was available (in which case
-   nothing may be marked). Draws the secret on first use. */
-const char *singlefile_intro(httrackp *opt);
-
-/* HTS_FALSE if [body,len) already contains this run's intro, in which case the
-   document must not be marked: RFC 2046 gives the generator the same duty for
-   a MIME boundary. Only a 2^-64 coincidence can trip it, since the intro is
-   not something fetched content can spell. */
-hts_boolean singlefile_may_mark(httrackp *opt, const char *body, size_t len);
-
-/* Write the mark for a reference of reflen bytes into buf; returns buf. */
-const char *singlefile_mark(httrackp *opt, char *buf, size_t bufsize, char cls,
-                            size_t reflen);
-
-/* The class a reference in this context may inline as, or 0 to leave it alone.
-   tag_name points just past the '<' of the enclosing start tag, or NULL when
-   there is none (inside a stylesheet or a script); attr at the attribute name.
-   Everything htsparse detects is inlinable unless it names a page. */
-char singlefile_ref_class(const char *tag_name, const char *attr);
-
-/* Rewrite every HTML page the mirror produced, then strip the marks left in
-   the assets. No-op unless opt->single_file; call once the tree is final,
-   after the update purge. */
+/* Rewrite every HTML page the mirror produced. No-op unless opt->single_file;
+   call once the tree is final, after the update purge. */
 void singlefile_process_mirror(httrackp *opt);
 
-/* Expand the marks in the document held in memory, appending the result to
-   out. root is the mirror directory that references may not escape; page_path
-   is the document's own path under it (both UTF-8, '/' or native separators).
-   page_budget caps the total inlined bytes, since a nested @import fans out
+/* Rewrite one HTML document held in memory, appending the result to out.
+   root is the mirror directory that references may not escape; page_path is
+   the document's own path under it (both UTF-8, '/' or native separators).
+   page_budget caps the total inlined bytes, since nested @import fans out
    multiplicatively; the mirror pass passes SINGLEFILE_MAX_PAGE_SIZE.
-   Returns HTS_TRUE if at least one reference was replaced. */
+   Returns HTS_TRUE if at least one reference was replaced; out may still
+   differ from the input when that is HTS_FALSE, since a style or srcset value
+   is re-serialized in place. */
 hts_boolean singlefile_rewrite_html(httrackp *opt, const char *root,
                                     const char *page_path, const char *html,
                                     size_t html_len, LLint page_budget,

src/htstools.c

@@ -41,14 +41,8 @@ Please visit our Website: http://www.httrack.com
 #include "htsstrings.h"
 #include "htscharset.h"
 #ifdef _WIN32
-/* before <stdlib.h>, which is what declares rand_s under it */
-#define _CRT_RAND_S
 #include "windows.h"
 #else
-#include <errno.h>
-#ifdef HAVE_SYS_RANDOM_H
-#include <sys/random.h>
-#endif
 #include <dirent.h>
 #ifdef HAVE_UNISTD_H
 #include <unistd.h>
@@ -1450,50 +1444,6 @@ HTSEXT_API hts_boolean hts_findissystem(find_handle find) {
   return 0;
 }
 
-hts_boolean hts_random_bytes(void *buf, size_t len) {
-  unsigned char *const p = (unsigned char *) buf;
-
-#ifdef _WIN32
-  size_t i;
-
-  /* rand_s() is the CRT's wrapper over the system CSPRNG, and unlike
-     BCryptGenRandom it needs no extra import library. */
-  for (i = 0; i < len; i += sizeof(unsigned int)) {
-    const size_t left = len - i;
-    unsigned int v;
-
-    if (rand_s(&v) != 0)
-      return HTS_FALSE;
-    memcpy(p + i, &v, left < sizeof(v) ? left : sizeof(v));
-  }
-  return HTS_TRUE;
-#else
-  size_t got = 0;
-  FILE *fp;
-
-#ifdef HAVE_GETRANDOM
-  while (got < len) {
-    const ssize_t n = getrandom(p + got, len - got, 0);
-
-    if (n < 0) {
-      if (errno == EINTR)
-        continue;
-      break; /* pre-3.17 kernel or a seccomp filter: try /dev/urandom */
-    }
-    got += (size_t) n;
-  }
-  if (got == len)
-    return HTS_TRUE;
-#endif
-  fp = fopen("/dev/urandom", "rb");
-  if (fp == NULL)
-    return HTS_FALSE;
-  got += fread(p + got, 1, len - got, fp);
-  fclose(fp);
-  return got == len ? HTS_TRUE : HTS_FALSE;
-#endif
-}
-
 hts_boolean hts_rename_over(const char *src, const char *dst) {
   char csrc[CATBUFF_SIZE], cdst[CATBUFF_SIZE];
 

src/htstools.h

@@ -141,10 +141,6 @@ HTSEXT_API hts_boolean hts_findissystem(find_handle find);
    paths are fconv()'d. */
 hts_boolean hts_rename_over(const char *src, const char *dst);
 
-/* Fill buf with len cryptographically strong random bytes. HTS_FALSE when no
-   such source is available: the caller must fail, never fall back to rand(). */
-hts_boolean hts_random_bytes(void *buf, size_t len);
-
 #endif
 
 #endif

src/htsweb.c

@@ -101,8 +101,8 @@ static void htsweb_sig_brpipe(int code) {
   /* ignore */
 }
 
-/* Number of background threads */
-static int background_threads = 0;
+/* Threads that never return; no wait may count on them draining. */
+static int nonjoinable_threads = 0;
 
 /* Server/client ping handling */
 static htsmutex pingMutex = HTSMUTEX_INIT;
@@ -224,9 +224,7 @@ int main(int argc, char *argv[]) {
 #ifdef HTS_USESWF
   smallserver_setkey("USESWF", "1");
 #endif
-#ifdef HTS_USEZLIB
   smallserver_setkey("USEZLIB", "1");
-#endif
 #ifdef _WIN32
   smallserver_setkey("WIN32", "1");
 #endif
@@ -299,15 +297,19 @@ int main(int argc, char *argv[]) {
 
   /* pinger */
   if (parentPid > 0) {
-    hts_newthread(client_ping, (void *) (uintptr_t) parentPid);
-    background_threads++; /* Do not wait for this thread! */
+    if (hts_newthread(client_ping, (void *) (uintptr_t) parentPid) == 0) {
+#ifndef _WIN32
+      nonjoinable_threads++; /* client_ping() only ever leaves through exit() */
+#endif
+    }
     smallserver_setpinghandler(pingHandler, NULL);
   }
 
   /* launch */
   ret = help_server(argv[1], defaultPort, bindAddr);
 
-  htsthread_wait_n(background_threads - 1);
+  /* Drain everything a mirror may still have in flight, the pinger aside. */
+  htsthread_wait_n(nonjoinable_threads);
   hts_uninit();
 
 #ifdef _WIN32
@@ -382,7 +384,6 @@ void webhttrack_main(char *cmd) {
   commandRunning = 1;
   DEBUG(fprintf(stderr, "commandRunning=1\n"));
   hts_newthread(back_launch_cmd, (void *) strdup(cmd));
-  background_threads++; /* Do not wait for this thread! */
 }
 
 void webhttrack_lock(void) {
@@ -423,8 +424,8 @@ static int webhttrack_runmain(httrackp * opt, int argc, char **argv) {
   /* Rock'in! */
   ret = hts_main2(argc, argv, opt);
 
-  /* Wait for pending threads to finish */
-  htsthread_wait_n(background_threads);
+  /* Wait for pending threads to finish; the pinger and this thread stay. */
+  htsthread_wait_n(nonjoinable_threads + 1);
 
   return ret;
 }

src/htswizard.c

@@ -61,6 +61,11 @@ Please visit our Website: http://www.httrack.com
     assertf((*opt->filters.filptr) < opt->maxfilter);                          \
   } while (0)
 
+typedef struct htspair_t {
+  const char *tag;
+  const char *attr;
+} htspair_t;
+
 /* "embedded" */
 htspair_t hts_detect_embed[] = {
   {"img", "src"},
@@ -128,13 +133,11 @@ int hts_acceptlink(httrackp * opt, int ptr,
   return forbidden_url;
 }
 
-hts_boolean hts_cmp_tag_token(const char *tag, const char *cmp) {
+static int cmp_token(const char *tag, const char *cmp) {
   int p;
 
-  return (tag != NULL && strncasecmp(tag, cmp, (p = (int) strlen(cmp))) == 0 &&
-          !isalnum((unsigned char) tag[p]))
-             ? HTS_TRUE
-             : HTS_FALSE;
+  return (strncasecmp(tag, cmp, (p = (int) strlen(cmp))) == 0
+          && !isalnum((unsigned char) tag[p]));
 }
 
 /* TRUE if (tag, attribute) matches an embedded-asset pair in the table */
@@ -142,8 +145,7 @@ static hts_boolean is_embed_pair(const htspair_t *table, const char *tag,
                                  const char *attribute) {
   int i;
   for (i = 0; table[i].tag != NULL; i++) {
-    if (hts_cmp_tag_token(tag, table[i].tag) &&
-        hts_cmp_tag_token(attribute, table[i].attr))
+    if (cmp_token(tag, table[i].tag) && cmp_token(attribute, table[i].attr))
       return HTS_TRUE;
   }
   return HTS_FALSE;

src/htswizard.h

@@ -56,19 +56,6 @@ hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
                                hts_boolean filters_decided,
                                hts_boolean filters_refused);
 
-/* A (tag, attribute) pair naming a reference kind. */
-#ifndef HTS_DEF_DEFSTRUCT_htspair_t
-#define HTS_DEF_DEFSTRUCT_htspair_t
-
-typedef struct htspair_t {
-  const char *tag;
-  const char *attr;
-} htspair_t;
-#endif
-
-/* HTS_TRUE if tag starts with the whole token cmp; NULL tag never matches. */
-hts_boolean hts_cmp_tag_token(const char *tag, const char *cmp);
-
 int hts_acceptlink(httrackp * opt, int ptr,
                    const char *adr, const char *fil,
                    const char *tag, const char *attribute,

src/htszlib.c

@@ -40,7 +40,6 @@ Please visit our Website: http://www.httrack.com
 #include "htscodec.h"
 #include "htszlib.h"
 
-#if HTS_USEZLIB
 /* zlib */
 /*
 #include <zlib.h>
@@ -274,4 +273,3 @@ const char *hts_get_zerror(int err) {
     break;
   }
 }
-#endif

tests/01_engine-addlink.test

@@ -0,0 +1,7 @@
+#!/bin/bash
+#
+# an empty current path underflowed htsAddLink's codebase walk (#730).
+
+set -euo pipefail
+
+httrack -O /dev/null -#test=addlink | grep -q "addlink self-test OK"

tests/01_engine-structcheck.test

@@ -0,0 +1,12 @@
+#!/bin/bash
+#
+# structcheck(): the path-length guard, and the <name>.txt rename that once
+# built its target with an unbounded sprintf (#745).
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+httrack -O /dev/null -#test=structcheck "$dir" |
+    grep -q "structcheck self-test OK"

tests/01_engine-threadwait.test

@@ -0,0 +1,28 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_threadwait_st.XXXXXX") || exit 1
+trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM
+
+# No pipe into grep: SIGPIPE would mask a failing exit status.
+expect_ok() {
+    local label="$1" out
+    shift
+    out=$("$@" 2>&1) || {
+        echo "FAIL: ${label} exited non-zero: ${out}"
+        exit 1
+    }
+    case "$out" in
+    *"${label}: OK"*) ;;
+    *)
+        echo "FAIL: ${out}"
+        exit 1
+        ;;
+    esac
+}
+
+# A thread is outstanding from the moment hts_newthread() returns, so a wait
+# that follows the spawn joins it, and wait_n(n) still leaves n behind (#747).
+expect_ok "threadwait self-test" httrack -O "${tmpdir}/o1" -#test=threadwait

tests/65_port-siblings.test

@@ -79,6 +79,29 @@ done
 # 65535 is a valid port the old "< 65535" bound refused
 web_accepted 65535
 
+# --- htsserver returns from main() instead of blocking at exit -------------
+# The exit wait must exclude exactly the threads that never return (#753).
+# $tmp has no lang.def, so the server fails right after announcing and main()
+# reaches the wait on its own; rc, not EXITED, carries the verdict.
+
+web_exits() {
+    local rc=0
+
+    : >"$tmp/exit.log"
+    run_with_timeout 30 htsserver "$tmp" "$@" >"$tmp/exit.log" 2>&1 || rc=$?
+    grep -q "^EXITED" "$tmp/exit.log" ||
+        ! echo "FAIL: #753: htsserver ${*:-(no options)} never finished serving" || exit 1
+    # exactly 1, not merely "not 124": a crash or an assertf abort also escapes
+    # the wait, and would otherwise read as a pass
+    test "$rc" -eq 1 ||
+        ! echo "FAIL: #753: htsserver ${*:-(no options)} exited $rc, want 1" || exit 1
+}
+
+# no pinger: the excluded count went negative
+web_exits
+# a pinger, which never returns and so must stay excluded
+web_exits --ppid $$
+
 # --- proxytrack <proxy-addr:port> <ICP-addr:port> --------------------------
 # A bad argument falls through to the usage screen; it had no range check at
 # all, so 65616 quietly listened on port 80. A valid one binds and blocks.

tests/93_local-changes.test

@@ -94,10 +94,7 @@ expect_counts_match() {
     echo "OK"
 }
 lines() { printf '%s\n' "$@" | sort; }
-# A connection killed before the status line surfaces differently per platform
-# (macOS truncates the file, #748; Linux leaves it), so reset.bin is asserted on
-# its own and kept out of the exact lists.
-listed_but_reset() { listed "$1" | grep -v '/reset\.bin$' | sort; }
+digest() { cksum <"$1" | tr -d '[:space:]'; }
 
 # --- pass 1: nothing to compare against --------------------------------------
 httrack "${common[@]}" -O "$out" --purge-old=0 "${base}/changes/index.html" \
@@ -120,6 +117,7 @@ grep -aq "first crawl" "${out}/hts-log.txt" || {
 }
 
 host="127.0.0.1_${port}"
+resetdigest=$(digest "${out}/${host}/changes/reset.bin")
 # A leftover from some earlier failed attempt, at a name pass 2 fetches for the
 # first time: on disk, but never part of the previous mirror, so it is new.
 printf 'leftover junk' >"${out}/${host}/changes/fresh.html"
@@ -144,21 +142,19 @@ expect "gone is doomed.html and the old redirect target" \
 expect "changed is exactly the four moved resources" \
     "$(lines "${host}/changes/coded.bin" "${host}/changes/index.html" \
         "${host}/changes/moved.bin" "${host}/changes/moved.html")" \
-    "$(listed_but_reset changed)"
+    "$(listed changed | sort)"
 # Re-served byte for byte, 200, no Last-Modified, no ETag. flaky.bin gets there
 # through a failed transfer and a retry, so its file is notified twice;
 # redirtarget.html arrives behind a 302. sized.html has a byte-identical
 # payload, but the rewritten link to the renamed redirect target makes the file
-# on disk change length.
-expect "unchanged is exactly the six stable resources" \
+# on disk change length. reset.bin never completes a transfer (#746), so its
+# previous copy stands.
+expect "unchanged is exactly the seven stable resources" \
     "$(lines "${host}/changes/codedstable.bin" "${host}/changes/flaky.bin" \
-        "${host}/changes/redirtarget.html" "${host}/changes/sized.html" \
-        "${host}/changes/stable.bin" "${host}/changes/stable.html")" \
-    "$(listed_but_reset unchanged)"
-# reset.bin never completes a transfer, so it drops out of new.lst with its
-# mirrored copy still there. Whatever else it is, it is not a deletion.
-expect "a failed re-fetch is never reported gone" "" \
-    "$(listed gone | grep '/reset\.bin$' || true)"
+        "${host}/changes/redirtarget.html" "${host}/changes/reset.bin" \
+        "${host}/changes/sized.html" "${host}/changes/stable.bin" \
+        "${host}/changes/stable.html")" \
+    "$(listed unchanged | sort)"
 expect_counts_match "pass 2 counts match its lists"
 
 # Every mirrored file appears once and only once, across all four buckets.
@@ -192,19 +188,15 @@ test -f "${out}/${host}/changes/doomed.html" || {
     exit 1
 }
 echo "OK"
-printf '[a failed transfer keeps its file] ..\t'
-test -f "${out}/${host}/changes/reset.bin" || {
-    echo "FAIL: reset.bin lost its previous copy"
-    exit 1
-}
-echo "OK"
+expect "a failed transfer keeps its bytes" "$resetdigest" \
+    "$(digest "${out}/${host}/changes/reset.bin")"
 
 # --- pass 3: the same run with purging on ------------------------------------
 httrack "${common[@]}" -O "$out" --update "${base}/changes/index.html" \
     >"${tmpdir}/log3" 2>&1
 expect "the report says files were purged" "true" "$(field purged)"
 expect "gone is the page that only pass 2 linked" \
-    "${host}/changes/transient.html" "$(listed_but_reset gone)"
+    "${host}/changes/transient.html" "$(listed gone | sort)"
 printf '[a purged file is off disk] ..\t'
 test ! -f "${out}/${host}/changes/transient.html" || {
     echo "FAIL: transient.html survived --purge-old"

tests/94_local-single-file.test

@@ -37,11 +37,6 @@ cat >"${doc}/index.html" <<'EOF'
 <img src="pixel.svg" data-src="lazy.svg" alt="l">
 <script src="app.js"></script>
 <a href="other.html">other</a>
-<p>id=pixel.svg#!htsinline end</p>
-<link rel="stylesheet" href="style_noext">
-<script src="app_noext"></script>
-<img src="pixel.svg#icon">
-<a href="pixel.svg">raw</a>
 </body></html>
 EOF
 cat >"${doc}/other.html" <<'EOF'
@@ -55,8 +50,6 @@ cat >"${doc}/nested.css" <<'EOF'
 div { background: url(pixel.svg); }
 EOF
 printf 'var app = 1;\n' >"${doc}/app.js"
-printf 'body { color: red; }\n' >"${doc}/style_noext"
-printf 'var noext = 1;\n' >"${doc}/app_noext"
 printf '<svg xmlns="http://www.w3.org/2000/svg"><rect width="1" height="1"/></svg>\n' \
     >"${doc}/pixel.svg"
 printf '<svg xmlns="http://www.w3.org/2000/svg"><circle r="1"/></svg>\n' \
@@ -264,107 +257,12 @@ grep -q 'data:image/svg+xml;base64,' "$badpage" || {
     echo "FAIL: a rejected cap did not fall back to the default"
     exit 1
 }
-echo OK
-
-printf '[a page cannot forge a mark] ..\t'
-# The page ships what the old in-band marker looked like, against a real
-# mirrored asset. The mark now carries a per-run secret the site cannot spell,
-# so the text is not a mark and is not even rewritten: byte-identical to the
-# mirror taken without --single-file. pixel.svg is inlined elsewhere on the
-# page, so the encoder was demonstrably willing.
-forged=$(sed -n 's/.*<p>\(id=[^<]*\)<\/p>.*/\1/p' "$page")
-test "$forged" = 'id=pixel.svg#!htsinline end' || {
-    echo "FAIL: the page was altered: $forged"
-    exit 1
-}
-grep -q 'data:image/svg+xml;base64,' "$page" || {
-    echo "FAIL: nothing was inlined at all, so the probe proves nothing"
-    exit 1
-}
-echo OK
-
-printf '[a mark cannot ride in on a header] ..\t'
-# htsparse echoes the Content-Type charset into a <meta>; a forged mark there
-# bypasses anything that scans the body. Unforgeable means the channel does
-# not matter, so this asserts on the channel the body sweep could never see.
-grep -q 'charset=127' "$page" || true
-if grep -q 'content="text/html;charset=data:' "$page"; then
-    echo "FAIL: a charset-borne mark was expanded"
+if grep -q 'src="big.svg"' "$badpage"; then
+    echo "FAIL: the default cap did not cover big.svg"
     exit 1
 fi
 echo OK
 
-printf '[assets keep no marks] ..\t'
-if grep -rlF '#!' "${out}/127.0.0.1_${port}" 2>/dev/null | grep -q .; then
-    marked=$(grep -rlF '#!' "${out}/127.0.0.1_${port}" | tr '\n' ' ')
-    # the fixture's own literal is expected in index.html only
-    test "$marked" = "${out}/127.0.0.1_${port}/index.html " || {
-        echo "FAIL: marks survived in $marked"
-        exit 1
-    }
-fi
-echo OK
-
-printf '[an unrecognised type still inlines from its context] ..\t'
-# style_noext and app_noext have no usable extension, so only the referencing
-# context says what they are. plain.css/plain.js are the control: they inline
-# either way, so a failure here is the context path and not the encoder.
-grep -q 'href="data:text/css;base64,' "$page" || {
-    echo "FAIL: no stylesheet inlined at all"
-    exit 1
-}
-if grep -q 'href="style_noext"' "$page"; then
-    echo "FAIL: extensionless stylesheet was not inlined"
-    exit 1
-fi
-if grep -q 'src="app_noext"'  "$page"; then
-    echo "FAIL: extensionless script was not inlined"
-    exit 1
-fi
-echo OK
-
-printf '[a fragment survives onto the data: URI] ..\t'
-# An SVG fragment selects a view, so dropping it renders the wrong thing (#766).
-grep -q 'data:image/svg+xml;base64,[A-Za-z0-9+/=]*#icon' "$page" || {
-    echo "FAIL: the fragment was dropped"
-    exit 1
-}
-echo OK
-
-printf '[a page link is not inlined even when it names an asset] ..\t'
-# <a href> to an image: only the deny rule stops this, since the MIME gate
-# would happily take it. Without the rule the anchor becomes a data: URI.
-grep -q '<a href="pixel.svg">raw</a>' "$page" || {
-    echo "FAIL: an anchor to an inlinable asset was rewritten"
-    exit 1
-}
-echo OK
-
-printf '[-%%M refuses to pair with --single-file] ..\t'
-both=$(httrack -O "${tmpdir}/both" --quiet --robots=0 --retries=0 -%M \
-    --single-file "${base}/index.html" 2>&1 || true)
-case "$both" in
-*"pick one"*) echo OK ;;
-*)
-    echo "FAIL: the combination was accepted: $both"
-    exit 1
-    ;;
-esac
-
-printf '[--changes and --single-file together] ..\t'
-# Both hook end-of-mirror. Expansion must run first, so the report sees the
-# inlined size rather than the marked intermediate.
-chg="${tmpdir}/chg"
-mkdir "$chg"
-httrack "${common[@]}" --single-file --changes "${base}/index.html" \
-    >"${tmpdir}/log6" 2>&1
-chgpage="${chg}/127.0.0.1_${port}/index.html"
-test ! -f "$chgpage" || grep -q 'base64,' "$chgpage" || {
-    echo "FAIL: --changes suppressed the inlining"
-    exit 1
-}
-echo OK
-
 printf '[the rewritten page keeps the mirror file mode] ..\t'
 # The rewrite spools to a temp and renames, bypassing the engine's chmod.
 if is_windows; then

tests/96_local-refetch-keep.test

@@ -0,0 +1,28 @@
+#!/bin/bash
+#
+# A re-fetch cut mid-header must leave the mirrored file alone (#748): unfixed,
+# the aborted read's leftover buffer lands on it. err.bin is the control, an
+# HTTP 500 on the same resource already keeping its copy; stay.bin answers
+# normally with a new body, so a fix that stopped overwriting anything fails.
+#
+# Purging is off: an update purge deletes both files for an unrelated reason
+# (#746), which would hide what this asserts.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" \
+    --rerun-args '--update --purge-old=0' \
+    --log-found 'after 2 retries at link .*keep/page\.html' \
+    --log-found 'after 2 retries at link .*keep/data\.bin' \
+    --file-matches 'keep/page.html' 'KEEP-PAGE-V1' \
+    --file-not-matches 'keep/page.html' 'HTTP/1\.0 200' \
+    --file-matches 'keep/data.bin' 'KEEP-BIN-V1' \
+    --file-not-matches 'keep/data.bin' 'HTTP/1\.0 200' \
+    --file-min-bytes 'keep/data.bin' 2048 \
+    --file-matches 'keep/err.bin' 'KEEP-ERR-V1' \
+    --file-min-bytes 'keep/err.bin' 2048 \
+    --file-matches 'keep/stay.bin' 'KEEP-STAY-V2' \
+    --file-min-bytes 'keep/stay.bin' 2048 \
+    httrack 'BASEURL/keep/index.html' --retries=2

tests/Makefile.am

@@ -30,6 +30,7 @@ TEST_EXTENSIONS = .test
 TEST_LOG_COMPILER = $(BASH)
 TESTS = \
 	00_runnable.test \
+	01_engine-addlink.test \
 	01_engine-changes.test \
 	01_engine-charset.test \
 	01_engine-cmdline.test \
@@ -81,7 +82,9 @@ TESTS = \
 	01_engine-status.test \
 	01_engine-stripquery.test \
 	01_engine-strsafe.test \
+	01_engine-structcheck.test \
 	01_engine-syscharset.test \
+	01_engine-threadwait.test \
 	01_engine-topindex.test \
 	01_engine-urlhack.test \
 	01_engine-unescape-bounds.test \
@@ -194,6 +197,7 @@ TESTS = \
 	92_local-proxytrack-ndx-fields.test \
 	93_local-changes.test \
 	94_local-single-file.test \
-	95_local-sitemap.test
+	95_local-sitemap.test \
+	96_local-refetch-keep.test
 
 CLEANFILES = check-network_sh.cache

tests/local-server.py

@@ -1000,6 +1000,62 @@ class Handler(SimpleHTTPRequestHandler):
         v = 1 if self.refetch_pass() == 1 else 2
         self.send_raw(b"<html><body><p>STAY-V%d</p></body></html>" % v, "text/html")
 
+    # --- re-fetch cut mid-header, so nothing is stored (#746, #748) ---------
+    KEEP_PAGE = b"<html><body><p>KEEP-PAGE-V1</p></body></html>"
+    KEEP_BIN = b"KEEP-BIN-V1\n" + b"\x51\x52\x53\x54" * 512
+    KEEP_ERR = b"KEEP-ERR-V1\n" + b"\x61\x62\x63\x64" * 512
+
+    def send_cut_headers(self):
+        """Hang up mid-header: the response never becomes parseable."""
+        self.close_connection = True
+        try:
+            self.wfile.write(b"HTTP/1.0 200 OK\r\nContent-Ty")
+            self.wfile.flush()
+        except OSError:
+            pass
+        self.connection.close()
+
+    def route_keep_index(self):
+        self.refetch_pass()
+        self.send_html(
+            '\t<a href="page.html">page</a>\n'
+            '\t<a href="data.bin">data</a>\n'
+            '\t<a href="err.bin">err</a>\n'
+            '\t<a href="stay.bin">stay</a>\n'
+        )
+
+    def route_keep_page(self):
+        if self.refetch_pass() == 1:
+            self.send_raw(self.KEEP_PAGE, "text/html")
+        else:
+            self.send_cut_headers()
+
+    def route_keep_data(self):
+        if self.refetch_pass() == 1:
+            self.send_raw(self.KEEP_BIN, "application/octet-stream")
+        else:
+            self.send_cut_headers()
+
+    # Control: an HTTP error on the same resource already keeps the copy, so
+    # the cut-header routes above must end up indistinguishable from it.
+    def route_keep_err(self):
+        if self.refetch_pass() == 1:
+            self.send_raw(self.KEEP_ERR, "application/octet-stream")
+        else:
+            self.send_response(500)
+            self.send_header("Content-Type", "text/html")
+            self.send_header("Content-Length", "0")
+            self.end_headers()
+
+    # Control: answers normally, with a new body on pass 2, so a fix that
+    # stopped overwriting mirrored files altogether would be caught.
+    def route_keep_stay(self):
+        v = 1 if self.refetch_pass() == 1 else 2
+        self.send_raw(
+            b"KEEP-STAY-V%d\n" % v + b"\x71\x72\x73\x74" * 512,
+            "application/octet-stream",
+        )
+
     # Echo what httrack advertised, so a crawl can assert the header.
     def route_codec_ae(self):
         self.send_raw(
@@ -1950,6 +2006,11 @@ class Handler(SimpleHTTPRequestHandler):
         "/uptrunc/page.html": route_uptrunc_page,
         "/uptrunc/file.bin": route_uptrunc_file,
         "/uptrunc/stay.html": route_uptrunc_stay,
+        "/keep/index.html": route_keep_index,
+        "/keep/page.html": route_keep_page,
+        "/keep/data.bin": route_keep_data,
+        "/keep/err.bin": route_keep_err,
+        "/keep/stay.bin": route_keep_stay,
         "/types/index.html": route_types_index,
         "/types/control.php": route_types,
         "/types/photo.png": route_types,