url_savename never clamps the final path segment, only the whole path (#861)

* url_savename never clamps the final path segment, only the whole path

Inside url_savename's over-ceiling shortening, every directory segment is cut
to MAX_SEG_LEN but the name itself is copied under the whole-path budget alone.
A single component could therefore run to ~226 characters on Windows while its
parent directories were held to 48, and once the directories had eaten the
budget the cut landed mid-name and took the extension with it, leaving a page
saved with no .html at all.

Clamp the name like any other segment, and reserve a plain extension across the
cut the way #623 already reserves the ".delayed" marker.

75_engine-longpath-posix's Windows branch asserted the clamp all along; it had
never run until #847 broadened the CI glob, so this also drops the expected_skip
that filed the failure as #852.

Closes #852

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix 01_engine-savename's Windows expectation for the clamped name

The Windows arm pinned the pre-fix output: 210 characters of name and no
extension at all. It now takes the 48-char segment clamp with its ".html"
intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitattributes

@@ -7,7 +7,5 @@
 *.test text eol=lf
 *.sh text eol=lf
 
-# Every new test appends to the TESTS list, so two branches in flight always
-# collide on its last line. A union merge keeps both entries; automake does not
-# care about order.
-tests/Makefile.am merge=union
+# Scoped to this file so a union can never touch build logic.
+tests/tests-list.mk merge=union

.github/workflows/windows-build.yml

@@ -209,9 +209,11 @@ jobs:
           : >"$progress"
 
           pass=0 fail=0 skip=0 failed="" skipped="" deadline=0
-          for t in 00_runnable.test 01_engine-*.test 01_zlib-*.test \
-                   *_local-*.test 13_crawl_proxy_https.test 58_watchdog.test \
-                   60_crawl-log-salvage.test; do
+          # Globbed, not enumerated: a new NNN_engine-*.test or NNN_local-*.test
+          # is picked up automatically instead of silently getting zero coverage.
+          for t in 00_runnable.test *_engine-*.test *_zlib-*.test \
+                   *_local-*.test *_watchdog*.test *_crawl_proxy_https.test \
+                   *_crawl-log-salvage.test; do
             elapsed=$((SECONDS - started))
             if [ "$elapsed" -ge "$suite_deadline" ]; then
               echo "::error::suite deadline: ${elapsed}s elapsed, stopping before $t"
@@ -258,17 +260,38 @@ jobs:
 
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
+          # One name per line, so two branches each appending one don't collide on the
+          # same line; compared as a sorted set below, so glob discovery order can't
+          # cause a false mismatch either.
           # footer-overflow and purge-longpath skip on Windows (need a path past MAX_PATH);
           # crange pending #581;
-          # webdav-mime needs a reapable background listener, which MSYS cannot give it;
+          # webdav-default and webdav-mime need a reapable background listener, which MSYS cannot give them;
           # badmtime needs a filesystem that stores an mtime past gmtime's range;
           # single-file ends on a GUI half needing htsserver, which this job does not build;
-          # update-304-leak needs a LeakSanitizer build, which MSVC has no equivalent of.
-          expected_skips=" 01_engine-footer-overflow.test 100_local-purge-longpath.test 114_local-update-304-leak.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test 94_local-single-file.test"
+          # update-304-leak needs a LeakSanitizer build, which MSVC has no equivalent of;
+          # crash-symbolize needs backtrace(), which Windows has no equivalent of.
+          expected_skips="01_engine-footer-overflow.test
+          100_local-purge-longpath.test
+          114_local-update-304-leak.test
+          120_local-proxytrack-webdav-default.test
+          48_local-crange-memresume.test
+          71_local-crange-repaircache.test
+          79_local-proxytrack-webdav-mime.test
+          80_engine-crash-symbolize.test
+          88_local-proxytrack-badmtime.test
+          94_local-single-file.test"
           # First, or the deadline reads as an unexplained shortfall in the gates below.
           [ "$deadline" -eq 0 ] || { echo "::error::suite did not finish within ${suite_deadline}s"; exit 1; }
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
-          [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
+          # Word-split on whitespace (space-joined $skipped, newline-joined
+          # expected_skips both work) and sort, so the compare is a set, not a string.
+          got=$(printf '%s\n' $skipped | sort)
+          want=$(printf '%s\n' $expected_skips | sort)
+          if [ "$got" != "$want" ]; then
+            echo "::error::skip set changed from expected; - missing, + newly skipped"
+            diff -u <(echo "$want") <(echo "$got") | tail -n +3 | sed 's/^/      /'
+            exit 1
+          fi
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }
 
       - name: Upload the test logs

AGENTS.md

@@ -19,7 +19,7 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   (`request_queue_size`) so macOS/BSD don't drop connections under a parallel
   `-c16` bigcrawl the way Python's default backlog of 5 did.
   Or run `sh build.sh` to do bootstrap + configure + make in one shot.
-- A `tests/NN_*.test` runs only if listed in `tests/Makefile.am`'s `TESTS`; an
+- A `tests/NN_*.test` runs only if listed in `tests/tests-list.mk`; an
   unregistered file is silently skipped.
 - `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
   not — an installed `/usr/bin/httrack` then shadows your build. Run via `make

src/htsalias.c

@@ -501,13 +501,11 @@ int optinclude_file(const char *name, int *argc, char **argv, char *x_argvblk,
       /* read line */
       linput(fp, line, 250);
       hts_lowcase(line);
+      /* trim first: a blank line is skipped, not parsed as an option */
+      hts_rtrim(line, HTS_REALSPACES);
       if (strnotempty(line)) {
         /* no comment line: # // ; */
         if (strchr("#/;", line[0]) == NULL) {
-          /* right trim */
-          a = line + strlen(line) - 1;
-          while(is_realspace(*a))
-            *(a--) = '\0';
           /* jump "set " and spaces */
           a = line;
           while(is_realspace(*a))
@@ -577,7 +575,6 @@ int optinclude_file(const char *name, int *argc, char **argv, char *x_argvblk,
             }
           }
         }
-
       }
     }
     fclose(fp);

src/htsback.c

@@ -483,18 +483,27 @@ int back_cleanup_background(httrackp * opt, cache_back * cache,
 #ifndef HTS_NO_BACK_ON_DISK
       /* temporarily serialize the entry on disk */
       {
-        int fsz = (int) strlen(back[i].url_sav);
-        char *filename = malloc(fsz + 8 + 1);
+        /* +16: room for the ".tmp" the url_sav form appends to a full-length
+           save name, so one buffer holds both shapes */
+        char BIGSTK tmpname[HTS_URLMAXSIZE * 2 + 16];
+        char *filename;
+        hts_boolean named;
+
+        /* the -p0 name is not derived from url_sav, so it needs a buffer of
+           its own size rather than the save name's */
+        if (opt->getmode != 0) {
+          named =
+              slprintfbuff(tmpname, sizeof(tmpname), "%s.tmp", back[i].url_sav);
+        } else {
+          named = slprintfbuff(tmpname, sizeof(tmpname), "%stmpfile%d.tmp",
+                               StringBuff(opt->path_html_utf8),
+                               opt->state.tmpnameid++);
+        }
+        filename = named ? strdupt(tmpname) : NULL;
 
         if (filename != NULL) {
           FILE *fp;
 
-          if (opt->getmode != 0) {
-            sprintf(filename, "%s.tmp", back[i].url_sav);
-          } else {
-            sprintf(filename, "%stmpfile%d.tmp",
-                    StringBuff(opt->path_html_utf8), opt->state.tmpnameid++);
-          }
           /* Security check */
           if (fexist_utf8(filename)) {
             hts_log_print(opt, LOG_WARNING,
@@ -525,11 +534,12 @@ int back_cleanup_background(httrackp * opt, cache_back * cache,
                           "file does not exist");
           }
           if (filename != NULL)
-            free(filename);
+            freet(filename);
         } else {
           hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
-                        "engine: warning: serialize error for %s%s: memory full",
-                        back[i].url_adr, back[i].url_fil);
+                        "engine: warning: serialize error for %s%s: %s",
+                        back[i].url_adr, back[i].url_fil,
+                        named ? "memory full" : "temporary filename too long");
         }
       }
 #else
@@ -646,6 +656,42 @@ int back_nsoc_overall(const struct_back * sback) {
   return n;
 }
 
+/* Subdirectory holding a mirrored file's temporaries, beside it. url_savename()
+   maps '~' to '_', so no URL can ever be mirrored inside it (#774, #842). */
+#define HTS_TMPDIR "~hts-tmp"
+
+/* Build save's temporary as <dir>/<HTS_TMPDIR>/<name>.<ext>. Appending the
+   extension to save instead put it in the mirror namespace, so a site serving
+   <path>.bak had its copy taken as the backup and then unlinked (#774).
+   HTS_FALSE (dest emptied) if it would not fit. Note: utf-8. */
+static hts_boolean back_tmpname(char *dest, size_t size, const char *save,
+                                const char *ext) {
+  const char *const slash = strrchr(save, '/');
+  const int dirlen = slash != NULL ? (int) (slash - save) + 1 : 0;
+
+  if (!slprintfbuff(dest, size, "%.*s" HTS_TMPDIR "/%s.%s", dirlen, save,
+                    slash != NULL ? slash + 1 : save, ext)) {
+    dest[0] = '\0';
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* Note: utf-8 */
+void back_tmpdir_drop(const char *tmp) {
+  char BIGSTK dir[HTS_URLMAXSIZE * 2];
+  const char *slash;
+
+  if (tmp == NULL || (slash = strrchr(tmp, '/')) == NULL)
+    return;
+  if (!strclipbuff(dir, sizeof(dir), tmp))
+    return;
+  dir[slash - tmp] = '\0';
+  slash = strrchr(dir, '/');
+  if (strcmp(slash != NULL ? slash + 1 : dir, HTS_TMPDIR) == 0)
+    (void) RMDIR(dir);
+}
+
 /* generate temporary file on lien_back */
 /* Note: utf-8 */
 static int create_back_tmpfile(httrackp *opt, lien_back *const back,
@@ -653,23 +699,25 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
   // do not use tempnam() but a regular filename
   back->tmpfile_buffer[0] = '\0';
   if (back->url_sav[0] != '\0') {
-    /* same capacity as url_sav, so truncation drops the extension and aliases
-       the temp name onto the live file that back_finalize_backup() UNLINKs */
-    if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
+    if (!back_tmpname(back->tmpfile_buffer, sizeof(back->tmpfile_buffer),
+                      back->url_sav, ext)) {
       hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
                     back->url_sav);
-      back->tmpfile_buffer[0] = '\0';
       return -1;
     }
     back->tmpfile = back->tmpfile_buffer;
     if (structcheck(back->tmpfile) != 0) {
       hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
                     back->tmpfile);
+      back->tmpfile_buffer[0] = '\0';
+      back->tmpfile = NULL;
       return -1;
     }
   } else {
+    /* same directory as the named case, so back_tmpdir_drop() only removes one
+       the engine made (#842) */
     /* truncation here would collide distinct tmpnameid's onto one name */
-    if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
+    if (!sprintfbuff(back->tmpfile_buffer, "%s/" HTS_TMPDIR "/tmp%d.%s",
                      StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
                      ext)) {
       hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
@@ -678,6 +726,13 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
       return -1;
     }
     back->tmpfile = back->tmpfile_buffer;
+    if (structcheck(back->tmpfile) != 0) {
+      hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
+                    back->tmpfile);
+      back->tmpfile_buffer[0] = '\0';
+      back->tmpfile = NULL;
+      return -1;
+    }
   }
   /* OK */
   hts_log_print(opt, LOG_TRACE, "produced temporary name %s", back->tmpfile);
@@ -697,6 +752,10 @@ void back_refetch_backup(httrackp *opt, lien_back *const back) {
         hts_log_print(opt, LOG_WARNING, "replacing leftover backup %s",
                       back->tmpfile);
       saved = hts_rename_over(opt, back->url_sav, back->tmpfile);
+      /* Another slot sharing the directory may have removed it between the
+         structcheck above and the rename: recreate it and try once more. */
+      if (!saved && structcheck(back->tmpfile) == 0)
+        saved = hts_rename_over(opt, back->url_sav, back->tmpfile);
     }
     if (!saved) {
       hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
@@ -720,16 +779,22 @@ static hts_boolean back_transfer_failed(const int statuscode) {
   }
 }
 
-/* Commit or restore a re-fetch backup (#77 follow-up): a re-fetch over an
-   existing file moved the good copy to back->tmpfile before truncating url_sav.
-   commit keeps the new file and drops the backup; else restore it so an aborted
-   transfer leaves the previous copy intact. Skips the zlib .z temp. */
-static void back_finalize_backup(httrackp *opt, lien_back *const back,
-                                 const hts_boolean commit) {
+hts_boolean back_finalize_backup(httrackp *opt, lien_back *const back,
+                                 hts_boolean commit) {
+  const hts_boolean wanted = commit;
+
   if (back->tmpfile == NULL || back->r.compressed)
-    return;
+    return HTS_TRUE;
+  /* Nothing to commit to: filecreate() can fail after the backup was taken,
+     and dropping it then loses both copies (#775). */
+  if (commit && !fexist_utf8(back->url_sav)) {
+    hts_log_print(opt, LOG_WARNING, "%s was never created; restoring %s",
+                  back->url_sav, back->tmpfile);
+    commit = HTS_FALSE;
+  }
   if (commit) {
     (void) UNLINK(back->tmpfile); /* new copy is good; drop the backup */
+    back_tmpdir_drop(back->tmpfile);
   } else {
     if (back->r.out != NULL) {
       fclose(back->r.out);
@@ -741,8 +806,17 @@ static void back_finalize_backup(httrackp *opt, lien_back *const back,
       hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
                     "could not restore %s; previous copy kept as %s",
                     back->url_sav, back->tmpfile);
+    else
+      back_tmpdir_drop(back->tmpfile);
   }
   back->tmpfile = NULL;
+  return commit == wanted ? HTS_TRUE : HTS_FALSE;
+}
+
+/* A chunked body is framed by its terminating zero-length chunk (#840);
+   chunk_blocksize is reset per response and reaches -1 only once it is seen. */
+static hts_boolean back_chunked_unterminated(const lien_back *const back) {
+  return back->is_chunk && back->chunk_blocksize != -1 ? HTS_TRUE : HTS_FALSE;
 }
 
 // objet (lien) téléchargé ou transféré depuis le cache
@@ -764,9 +838,22 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
     /* Don't store broken files. Note: check is done before compression.
        If the file is partial, the next run will attempt to continue it with compression too.
      */
-    if (back[p].r.totalsize >= 0 && back[p].r.statuscode > 0
-        && back[p].r.size != back[p].r.totalsize && !opt->tolerant) {
-      if (back[p].status == STATUS_READY) {
+    const hts_boolean cut_chunked = back_chunked_unterminated(&back[p]);
+    const hts_boolean short_body =
+        back[p].r.totalsize >= 0 && back[p].r.size != back[p].r.totalsize
+            ? HTS_TRUE
+            : HTS_FALSE;
+
+    if ((short_body || cut_chunked) && back[p].r.statuscode > 0 &&
+        !opt->tolerant) {
+      if (cut_chunked) {
+        hts_log_print(
+            opt, LOG_WARNING,
+            "truncated chunked transfer (terminating chunk missing, got " LLintP
+            " bytes): file not cached, will be retried on the next"
+            " update (use -%%B to cache anyway): %s%s",
+            back[p].r.size, back[p].url_adr, back[p].url_fil);
+      } else if (back[p].status == STATUS_READY) {
         hts_log_print(opt, LOG_WARNING,
                       "incomplete transfer (expected " LLintP
                       " bytes, got " LLintP
@@ -844,12 +931,21 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
                     hts_codec_parse(back[p].r.contentencoding);
                 /* Never decode over url_sav: a failed decode would destroy the
                    copy an --update re-fetch is supposed to refresh (#557). */
-                char BIGSTK unpacked[HTS_URLMAXSIZE * 2 + 4]; // room for ".u"
+                char BIGSTK unpacked[HTS_URLMAXSIZE * 2];
                 LLint size;
 
-                snprintf(unpacked, sizeof(unpacked), "%s.u", back[p].url_sav);
-                if ((size = hts_codec_unpack(codec, back[p].tmpfile,
-                                             unpacked)) >= 0) {
+                /* fits whenever the .z temp it decodes from did */
+                if (!back_tmpname(unpacked, sizeof(unpacked), back[p].url_sav,
+                                  "u")) {
+                  back[p].r.statuscode = STATUSCODE_INVALID;
+                  strcpybuff(back[p].r.msg, "Error when decompressing (the "
+                                            "temporary filename is too long)");
+                  /* as the decode-failure branch below: never let the coded
+                     bytes be committed as the page */
+                  if (!back[p].r.is_write)
+                    deleteaddr(&back[p].r);
+                } else if ((size = hts_codec_unpack(codec, back[p].tmpfile,
+                                                    unpacked)) >= 0) {
                   back[p].r.size = back[p].r.totalsize = size;
                   if (back[p].r.is_write) {
                     /* Sample the previous copy now: the rename below replaces
@@ -921,6 +1017,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
               /* ensure that no remaining temporary file exists */
               if (back[p].tmpfile != NULL) {
                 unlink(back[p].tmpfile);
+                back_tmpdir_drop(back[p].tmpfile); /* the .u went with it */
                 back[p].tmpfile = NULL;
               }
             }
@@ -933,7 +1030,14 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
         }
         /* Body fully received: keep the freshly written url_sav, drop the
            backup of the previous copy. */
-        back_finalize_backup(opt, &back[p], HTS_TRUE);
+        if (!back_finalize_backup(opt, &back[p], HTS_TRUE)) {
+          /* The previous copy is back because the new one was never created;
+             caching this response's validators against it would pin the stale
+             body on every later --update. */
+          if (fexist_utf8(back[p].url_sav))
+            filenote(&opt->state.strc, back[p].url_sav, NULL);
+          return -1;
+        }
         /* Write mode to disk */
         if (back[p].r.is_write && back[p].r.adr != NULL) {
           freet(back[p].r.adr);
@@ -993,16 +1097,15 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
           char flags[32];
           char s[256];
           time_t tt;
-          struct tm *A;
+          struct tm tmv;
 
           tt = time(NULL);
-          A = localtime(&tt);
-          if (A == NULL) {
+          if (!hts_localtime(tt, &tmv)) {
             int localtime_returned_null = 0;
 
             assertf(localtime_returned_null);
           }
-          strftime(s, 250, "%H:%M:%S", A);
+          strftime(s, 250, "%H:%M:%S", &tmv);
 
           flags[0] = '\0';
           /* input flags */
@@ -1221,6 +1324,7 @@ void back_connxfr(htsblk * src, htsblk * dst) {
   src->keep_alive_t = 0;
   dst->debugid = src->debugid;
   src->debugid = 0;
+  dst->address = src->address; // peer IP survives the cache-entry swap (#838)
 }
 
 /* Release the buffers a response owns. The connection members are left alone:
@@ -1776,6 +1880,7 @@ int back_clear_entry(lien_back * back) {
     // only for security
     if (back->tmpfile && back->tmpfile[0] != '\0') {
       (void) unlink(back->tmpfile);
+      back_tmpdir_drop(back->tmpfile);
       back->tmpfile = NULL;
     }
     // Tout nettoyer
@@ -2075,6 +2180,8 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
                             back[p].url_adr, back[p].url_fil);
             }
             back[p].r.notmodified = 1;  // fichier non modifié
+            // no request was sent at all, so this is never a server 304 (#839)
+            back[p].r.warc_forced_notmodified = HTS_TRUE;
             back[p].status = STATUS_READY; // OK prêt
             back_set_finished(sback, p);
 
@@ -3470,6 +3577,26 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
               back_finalize(opt, cache, sback, i);
             }
 
+            /* Same treatment for an unterminated chunked stream: the byte count
+               agrees with the chunks that arrived, the framing does not. */
+            if (back[i].r.statuscode > 0 &&
+                back_chunked_unterminated(&back[i])) {
+              if (!opt->tolerant) {
+                deleteaddr(&back[i].r);
+                back[i].r.statuscode = STATUSCODE_CONNERROR; // recatch
+                htsblk_failf(&back[i].r,
+                             "Truncated chunked transfer (" LLintP
+                             " Bytes, terminating chunk missing)",
+                             (LLint) back[i].r.size);
+              } else {
+                hts_log_print(opt, LOG_WARNING,
+                              "Truncated chunked transfer (" LLintP
+                              " Bytes, terminating chunk missing) for %s%s",
+                              (LLint) back[i].r.size, back[i].url_adr,
+                              back[i].url_fil);
+              }
+            }
+
             if (back[i].r.totalsize >= 0) { // tester totalsize
               if (back[i].r.totalsize != back[i].r.size) {      // pas la même!
                 if (!opt->tolerant) {
@@ -3529,10 +3656,22 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                   if (back[i].r.totalsize < 0)
                     back[i].r.totalsize = 0;    // initialiser à 0 (-1 == unknown)
                   if (back[i].status == STATUS_CHUNK_WAIT) {    // "real" chunk
-                    if (sscanf(chunk_data, "%x", &chunk_size) == 1) {
+                    /* The chunk-size line is hostile input, so parse it wide
+                       and unsigned and drop anything an int cannot hold: sscanf
+                       "%x" lands 80000000 on INT_MIN, which sign-extends into a
+                       16EB realloc below and drives totalsize negative. */
+                    char *chunk_end = NULL;
+                    const unsigned long long chunk_value =
+                        strtoull(chunk_data, &chunk_end, 16);
+
+                    if (chunk_end != chunk_data && *chunk_end == '\0' &&
+                        chunk_value <= (unsigned long long) INT32_MAX) {
+                      chunk_size = (int) chunk_value;
                       if (chunk_size > 0)
                         back[i].chunk_blocksize = chunk_size;   /* the data block chunk size */
-                      else
+                      /* only a real 0 ends the stream; the bound above keeps a
+                         negative from ever claiming the sentinel (#840) */
+                      else if (chunk_size == 0)
                         back[i].chunk_blocksize = -1;   /* ending */
                       back[i].r.totalsize += chunk_size;        // noter taille
                       if (back[i].r.adr != NULL || !back[i].r.is_write) {       // Not to disk
@@ -4082,6 +4221,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
 
                       memset(&tmp, 0, sizeof(tmp));
                       back_connxfr(&back[i].r, &tmp);
+                      /* a real 304's headers belong to the revisit record, so
+                         they must survive the swap (#826); a forced one has
+                         none */
+                      if (server_sent_304)
+                        warc_move_request(&back[i].r, &tmp);
                       /* the cache entry overwrites the whole struct, so drop
                          what the 304 response still owns first (#782) */
                       back_free_response(&back[i].r);
@@ -4090,6 +4234,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                                    back[i].url_sav, back[i].location_buffer);
                       back[i].r.location = back[i].location_buffer;
                       back_connxfr(&tmp, &back[i].r);
+                      warc_move_request(&tmp, &back[i].r);
                     }
 
                     // hack:
@@ -4115,6 +4260,9 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                       back[i].status = STATUS_READY;    // OK prêt
                       back_set_finished(sback, i);
                       back[i].r.notmodified = 1;        // NON modifié!
+                      // WARC must not claim a 304 the server never sent (#839)
+                      back[i].r.warc_forced_notmodified =
+                          server_sent_304 ? HTS_FALSE : HTS_TRUE;
                       hts_log_print(opt, LOG_DEBUG,
                                     "File loaded after test from cache: %s%s",
                                     back[i].url_adr, back[i].url_fil);

src/htsback.h

@@ -143,6 +143,18 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
    can put it back when the re-fetch fails (#77 follow-up). Call right before
    truncating url_sav; tmpfile stays NULL when there is nothing to save. */
 void back_refetch_backup(httrackp *opt, lien_back *const back);
+/* Commit or restore a re-fetch backup (#77 follow-up): a re-fetch over an
+   existing file moved the good copy to back->tmpfile before truncating url_sav.
+   commit keeps the new file and drops the backup, unless url_sav was never
+   created; else restore it so an aborted transfer leaves the previous copy
+   intact. Skips the zlib .z temp. HTS_FALSE when a requested commit had to
+   restore instead: the caller then holds the OLD body and must not cache this
+   response's validators against it. */
+hts_boolean back_finalize_backup(httrackp *opt, lien_back *const back,
+                                 hts_boolean commit);
+/* Remove the reserved directory a temporary sat in, once the last slot sharing
+   it is done; a non-empty one just refuses. No-op outside that directory. */
+void back_tmpdir_drop(const char *tmp);
 /* -#test=backswap: slots eligible for the on-disk ready table. */
 int back_selftest_slot_swap(void);
 void back_info(struct_back * sback, int i, int j, FILE * fp);

src/htsbauth.c

@@ -491,7 +491,7 @@ char *bauth_prefix(char *prefix, const char *adr, const char *fil) {
   if (a)
     *a = '\0';
   if (strchr(prefix, '/')) {
-    a = prefix + strlen(prefix) - 1;
+    a = hts_lastcharptr(prefix);
     while(*a != '/')
       a--;
     *(a + 1) = '\0';

src/htscache.c

@@ -424,6 +424,11 @@ htsblk cache_read_including_broken(httrackp *opt, cache_back *cache,
 
     if (back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
       r = itemback->r;
+      /* header fields only, like cache_readex(): the entry torn down below
+         owns these (#826) */
+      r.adr = NULL;
+      r.headers = NULL;
+      r.location = NULL;
       if (return_save != NULL)
         strlcpybuff(return_save, itemback->url_sav, HTS_URLMAXSIZE * 2);
       /* cleanup */
@@ -957,6 +962,35 @@ htsblk *cache_header(httrackp * opt, cache_back * cache, const char *adr,
     return NULL;
 }
 
+const char *cache_repair(httrackp *opt, const char *name,
+                         unsigned long *entries, unsigned long *bytes) {
+  char BIGSTK repairname[HTS_URLMAXSIZE * 2];
+  unzFile zip;
+
+  *entries = 0;
+  *bytes = 0;
+  if (!slprintfbuff(repairname, sizeof(repairname), "%s%s",
+                    StringBuff(opt->path_log), "hts-cache/repair.zip"))
+    return "the repair path is too long";
+  if (unzRepair(name, repairname,
+                fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                        StringBuff(opt->path_log), "hts-cache/repair.tmp"),
+                entries, bytes) != Z_OK)
+    return "could not repair the cache";
+  /* unzRepair writes an end-of-central-directory record whatever it found, so
+     an input holding no local file header at all yields a valid empty archive
+     and a short write yields a truncated one. Only an archive that holds
+     something and opens may replace the cache (#824). */
+  if (*entries == 0)
+    return "the repaired cache holds no entry, keeping the damaged one";
+  if ((zip = hts_unzOpen_utf8(repairname)) == NULL)
+    return "the repaired cache does not open, keeping the damaged one";
+  unzClose(zip);
+  if (!hts_rename_over(opt, repairname, name))
+    return "could not put the repaired cache in place";
+  return NULL;
+}
+
 // Initialisation du cache: créer nouveau, renomer ancien, charger..
 void cache_init(cache_back * cache, httrackp * opt) {
   // ---
@@ -1026,8 +1060,9 @@ void cache_init(cache_back * cache, httrackp * opt) {
       // Corrupted ZIP file ? Try to repair!
       if (cache->zipInput == NULL && !cache->ro) {
         char *name;
-        uLong repaired = 0;
-        uLong repairedBytes = 0;
+        const char *why;
+        unsigned long repaired = 0;
+        unsigned long repairedBytes = 0;
 
         if (!cache->ro) {
           name =
@@ -1040,25 +1075,16 @@ void cache_init(cache_back * cache, httrackp * opt) {
         }
         hts_log_print(opt, LOG_WARNING,
                       "Cache: damaged cache, trying to repair");
-        /* mztools has no UTF-8 hook, so repairing a corrupt cache under a
-           non-ASCII path_log fails cleanly (re-crawl), never forks a twin. */
-        if (unzRepair
-            (name,
-             fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                     "hts-cache/repair.zip"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
-                                                      StringBuff(opt->path_log),
-                                                      "hts-cache/repair.tmp"),
-             &repaired, &repairedBytes) == Z_OK) {
-          UNLINK(name);
-          RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
-                         StringBuff(opt->path_log), "hts-cache/repair.zip"),
-                 name);
-          cache->zipInput = hts_unzOpen_utf8(name);
+        why = cache_repair(opt, name, &repaired, &repairedBytes);
+        if (why != NULL) {
+          hts_log_print(opt, LOG_WARNING | LOG_ERRNO, "Cache: %s", why);
+        } else if ((cache->zipInput = hts_unzOpen_utf8(name)) != NULL) {
           hts_log_print(opt, LOG_WARNING,
                         "Cache: %d bytes successfully recovered in %d entries",
                         (int) repairedBytes, (int) repaired);
         } else {
-          hts_log_print(opt, LOG_WARNING, "Cache: could not repair the cache");
+          hts_log_print(opt, LOG_WARNING,
+                        "Cache: the repaired cache could not be reopened");
         }
       }
       // Opened ?

src/htscache.h

@@ -67,7 +67,9 @@ htsblk cache_read(httrackp * opt, cache_back * cache, const char *adr,
 htsblk cache_read_ro(httrackp * opt, cache_back * cache, const char *adr,
                      const char *fil, const char *save, char *location);
 /* Like cache_read, but also yields entries whose transfer broke; return_save
-   (optional, HTS_URLMAXSIZE*2) receives the entry's recorded save name. */
+   (optional, HTS_URLMAXSIZE*2) receives the entry's recorded save name.
+   Header fields only: adr, headers and location come back NULL, so the caller
+   owns and frees nothing. */
 htsblk cache_read_including_broken(httrackp *opt, cache_back *cache,
                                    const char *adr, const char *fil,
                                    char *return_save);
@@ -78,6 +80,14 @@ htsblk *cache_header(httrackp * opt, cache_back * cache, const char *adr,
                      const char *fil, htsblk * r);
 void cache_init(cache_back * cache, httrackp * opt);
 
+/* Recover the damaged cache at name into hts-cache/repair.zip and move it over
+   name, storing what was recovered in *entries and *bytes. Returns NULL on
+   success, else a reason the caller reports: a recovery that is empty or does
+   not open never replaces the cache, and neither does one that cannot be moved
+   into place (#786, #824). Note: utf-8. */
+const char *cache_repair(httrackp *opt, const char *name,
+                         unsigned long *entries, unsigned long *bytes);
+
 /* Which hts-cache/ generation (new.* vs old.*) is authoritative. */
 typedef enum {
   CACHE_RECONCILE_PROMOTE,     /* no new cache: promote the old generation */

src/htscache_selftest.c

@@ -309,6 +309,72 @@ static int disk_fallback_selftest(httrackp *opt) {
   return fail;
 }
 
+/* A cache miss sends cache_read_including_broken() to the serialized reference
+   of an interrupted transfer, whose entry it then frees: what it returns must
+   not point into that entry (#826). */
+static int broken_ref_selftest(httrackp *opt) {
+  int fail = 0;
+  cache_back cache;
+  lien_back *entry;
+  htsblk r;
+  char save[HTS_URLMAXSIZE * 2];
+  const char *const adr = "example.com";
+  const char *const fil = "/interrupted.html";
+  static const char body[] = "<html><body>half a p";
+  static const char headers[] =
+      "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n";
+
+  entry = calloct(1, sizeof(lien_back));
+  strcpybuff(entry->url_adr, adr);
+  strcpybuff(entry->url_fil, fil);
+  concat(entry->url_sav, sizeof(entry->url_sav),
+         StringBuff(opt->path_html_utf8), "example.com/interrupted.html");
+  hts_init_htsblk(&entry->r);
+  entry->r.statuscode = 200;
+  entry->r.size = (LLint) (sizeof(body) - 1);
+  strcpybuff(entry->r.msg, "OK");
+  strcpybuff(entry->r.contenttype, "text/html");
+  entry->r.location = entry->location_buffer;
+  entry->r.adr = strdupt(body);
+  entry->r.headers = strdupt(headers);
+  if (back_serialize_ref(opt, entry) != 0) {
+    fprintf(stderr, "%s: broken-ref: cannot write the reference\n",
+            selftest_tag);
+    fail++;
+  }
+  freet(entry->r.adr);
+  freet(entry->r.headers);
+  freet(entry);
+
+  selftest_open_for_read(&cache, opt);
+  save[0] = '\0';
+  r = cache_read_including_broken(opt, &cache, adr, fil, save);
+  selftest_close(&cache);
+
+  if (r.statuscode != 200 || strcmp(r.contenttype, "text/html") != 0) {
+    fprintf(stderr,
+            "%s: broken-ref: statuscode %d type '%s', want 200/text/html"
+            " (reference not read back)\n",
+            selftest_tag, r.statuscode, r.contenttype);
+    fail++;
+  }
+  if (strstr(save, "interrupted.html") == NULL) {
+    fprintf(stderr, "%s: broken-ref: save name '%s' lost\n", selftest_tag,
+            save);
+    fail++;
+  }
+  /* the entry owned all three and is gone, so a non-NULL one is dangling */
+  if (r.adr != NULL || r.headers != NULL || r.location != NULL) {
+    fprintf(stderr,
+            "%s: broken-ref: returned freed adr/headers/location"
+            " (%p/%p/%p)\n",
+            selftest_tag, (void *) r.adr, (void *) r.headers,
+            (void *) r.location);
+    fail++;
+  }
+  return fail;
+}
+
 typedef struct {
   size_t budget;  /**< bytes allowed through before writes start failing */
   int fail_errno; /**< errno set on the failing write (ENOSPC, EIO, ...) */
@@ -800,6 +866,9 @@ int cache_selftests(httrackp *opt, const char *dir) {
   /* pass 5: the disk-fallback read path (X-In-Cache: 0, body on disk) */
   failures += disk_fallback_selftest(opt);
 
+  /* pass 6: the broken-transfer reference fallback */
+  failures += broken_ref_selftest(opt);
+
   for (i = 0; i < large_count; i++) {
     freet(large_body[i]);
   }

src/htscharset.c

@@ -296,16 +296,41 @@ LPWSTR hts_convertUTF8StringToUCS2(const char *s, int size, int *pwsize) {
   return hts_convertStringToUCS2(s, size, CP_UTF8, pwsize);
 }
 
-char *hts_convertUCS2StringToCP(LPWSTR woutput, int wsize, UINT cp) {
+/* WideCharToMultiByte rejects lpUsedDefaultChar on the Unicode, ISO-2022, HZ,
+   GB18030 and ISCII codepages, where a substitution stays invisible. */
+static hts_boolean cp_reports_default_char(UINT cp) {
+  if (cp == 42 /* CP_SYMBOL */ || cp == CP_UTF7 || cp == CP_UTF8 ||
+      cp == 52936 || cp == 54936 || (cp >= 50220 && cp <= 50229) ||
+      (cp >= 57002 && cp <= 57011)) {
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
+/* When plossy is non-NULL, *plossy reports that the codepage lacked a character
+   and a substitute was emitted for it. */
+static char *hts_convertUCS2StringToCPEx(LPWSTR woutput, int wsize, UINT cp,
+                                         hts_boolean *plossy) {
   const int usize =
-    WideCharToMultiByte(cp, 0, woutput, wsize, NULL, 0, NULL, FALSE);
+      WideCharToMultiByte(cp, 0, woutput, wsize, NULL, 0, NULL, NULL);
+
+  if (plossy != NULL) {
+    *plossy = HTS_FALSE;
+  }
   if (usize > 0) {
     char *const uoutput = malloc((usize + 1) * sizeof(char));
 
     if (uoutput != NULL) {
-      if (WideCharToMultiByte
-          (cp, 0, woutput, wsize, uoutput, usize, NULL, FALSE) == usize) {
+      BOOL usedDefault = FALSE;
+      LPBOOL const pUsedDefault =
+          plossy != NULL && cp_reports_default_char(cp) ? &usedDefault : NULL;
+
+      if (WideCharToMultiByte(cp, 0, woutput, wsize, uoutput, usize, NULL,
+                              pUsedDefault) == usize) {
         uoutput[usize] = '\0';
+        if (plossy != NULL && usedDefault) {
+          *plossy = HTS_TRUE;
+        }
         return uoutput;
       } else {
         free(uoutput);
@@ -315,6 +340,10 @@ char *hts_convertUCS2StringToCP(LPWSTR woutput, int wsize, UINT cp) {
   return NULL;
 }
 
+char *hts_convertUCS2StringToCP(LPWSTR woutput, int wsize, UINT cp) {
+  return hts_convertUCS2StringToCPEx(woutput, wsize, cp, NULL);
+}
+
 char *hts_convertUCS2StringToUTF8(LPWSTR woutput, int wsize) {
   return hts_convertUCS2StringToCP(woutput, wsize, CP_UTF8);
 }
@@ -346,7 +375,11 @@ char *hts_convertStringCPToUTF8(const char *s, size_t size, UINT cp) {
   return NULL;
 }
 
-char *hts_convertStringCPFromUTF8(const char *s, size_t size, UINT cp) {
+static char *hts_convertStringCPFromUTF8Ex(const char *s, size_t size, UINT cp,
+                                           hts_boolean *plossy) {
+  if (plossy != NULL) {
+    *plossy = HTS_FALSE;
+  }
   /* Empty string ? */
   if (size == 0) {
     return hts_stringMemCopy(s, size);
@@ -362,7 +395,8 @@ char *hts_convertStringCPFromUTF8(const char *s, size_t size, UINT cp) {
     LPWSTR woutput = hts_convertStringToUCS2(s, (int) size, CP_UTF8, &wsize);
 
     if (woutput != NULL) {
-      char *const uoutput = hts_convertUCS2StringToCP(woutput, wsize, cp);
+      char *const uoutput =
+          hts_convertUCS2StringToCPEx(woutput, wsize, cp, plossy);
 
       free(woutput);
       return uoutput;
@@ -373,6 +407,10 @@ char *hts_convertStringCPFromUTF8(const char *s, size_t size, UINT cp) {
   return NULL;
 }
 
+char *hts_convertStringCPFromUTF8(const char *s, size_t size, UINT cp) {
+  return hts_convertStringCPFromUTF8Ex(s, size, cp, NULL);
+}
+
 HTSEXT_API char *hts_convertStringToUTF8(const char *s, size_t size,
                                          const char *charset) {
   const UINT cp = hts_getCodepage(charset);
@@ -386,6 +424,19 @@ char *hts_convertStringFromUTF8(const char *s, size_t size, const char *charset)
   return hts_convertStringCPFromUTF8(s, size, cp);
 }
 
+char *hts_convertStringFromUTF8Strict(const char *s, size_t size,
+                                      const char *charset) {
+  hts_boolean lossy = HTS_FALSE;
+  char *const out =
+      hts_convertStringCPFromUTF8Ex(s, size, hts_getCodepage(charset), &lossy);
+
+  if (lossy) {
+    free(out);
+    return NULL;
+  }
+  return out;
+}
+
 HTSEXT_API char *hts_convertStringSystemToUTF8(const char *s, size_t size) {
   return hts_convertStringCPToUTF8(s, size, GetACP());
 }
@@ -590,6 +641,13 @@ char *hts_convertStringFromUTF8(const char *s, size_t size, const char *charset)
   }
 }
 
+char *hts_convertStringFromUTF8Strict(const char *s, size_t size,
+                                      const char *charset) {
+  /* No transliteration is requested of iconv, so an unrepresentable code point
+     already fails the conversion outright. */
+  return hts_convertStringFromUTF8(s, size, charset);
+}
+
 #endif
 
 #ifdef _WIN32

src/htscharset.h

@@ -61,6 +61,15 @@ HTSEXT_API char *hts_convertStringToUTF8(const char *s, size_t size,
 extern char *hts_convertStringFromUTF8(const char *s, size_t size,
                                        const char *charset);
 
+/**
+ * Same, but refusing to lose a code point the charset can not represent:
+ * NULL is returned then, where hts_convertStringFromUTF8() may hand back a
+ * substituted string (Windows substitutes, iconv fails).
+ * Return NULL upon error.
+ **/
+extern char *hts_convertStringFromUTF8Strict(const char *s, size_t size,
+                                             const char *charset);
+
 /**
  * Convert an UTF-8 string to an IDNA (RFC 3492) string.
  **/

src/htscore.c

@@ -1045,13 +1045,14 @@ int httpmirror(char *url1, httrackp * opt) {
     {
       TStamp tl = 0;
       time_t tt;
-      struct tm *A;
+      struct tm tmv;
 
       tt = time(NULL);
-      A = localtime(&tt);
-      tl += A->tm_sec;
-      tl += A->tm_min * 60;
-      tl += A->tm_hour * 60 * 60;
+      if (hts_localtime(tt, &tmv)) {
+        tl += tmv.tm_sec;
+        tl += tmv.tm_min * 60;
+        tl += tmv.tm_hour * 60 * 60;
+      }
       if (tl > opt->waittime)   // attendre minuit
         rollover = 1;
     }
@@ -1061,13 +1062,14 @@ int httpmirror(char *url1, httrackp * opt) {
     do {
       TStamp tl = 0;
       time_t tt;
-      struct tm *A;
+      struct tm tmv;
 
       tt = time(NULL);
-      A = localtime(&tt);
-      tl += A->tm_sec;
-      tl += A->tm_min * 60;
-      tl += A->tm_hour * 60 * 60;
+      if (hts_localtime(tt, &tmv)) {
+        tl += tmv.tm_sec;
+        tl += tmv.tm_min * 60;
+        tl += tmv.tm_hour * 60 * 60;
+      }
 
       if (rollover) {
         if (tl <= opt->waittime)
@@ -1728,16 +1730,12 @@ int httpmirror(char *url1, httrackp * opt) {
               if (charset != NULL)
                 free(charset);
             }
-            /* Could not detect charset: could it be UTF-8 ? */
-            /* No, we can not do that: browsers do not do it
-               (and it would break links). */
-            /* Could not detect charset */
+            /* Left empty when the document declared none: guessing UTF-8 here
+               would break links, and the parser picks its own fallback */
             if (page_charset[0] == '\0') {
               hts_log_print(opt, LOG_INFO,
                             "Warning: could not detect encoding for: %s%s",
                             urladr(), urlfil());
-              /* Fallback to ISO-8859-1 (~== identity) ; accents will look weird */
-              strcpy(page_charset, "iso-8859-1");
             }
           }
 
@@ -1770,7 +1768,9 @@ int httpmirror(char *url1, httrackp * opt) {
             /* */
             str.ptr_ = &ptr;
             /* */
-            str.page_charset_ = page_charset[0] != '\0' ? page_charset : NULL;
+            /* NULL when conversion is off, empty when the document declared no
+               charset; the parser tells the two apart */
+            str.page_charset_ = opt->convert_utf8 ? page_charset : NULL;
             /* */
             /* */
             stre.r_ = &r;
@@ -3148,16 +3148,15 @@ int fspc(httrackp * opt, FILE * fp, const char *type) {
   if (fp != NULL) {
     char s[256];
     time_t tt;
-    struct tm *A;
+    struct tm tmv;
 
     tt = time(NULL);
-    A = localtime(&tt);
-    if (A == NULL) {
+    if (!hts_localtime(tt, &tmv)) {
       int localtime_returned_null = 0;
 
       assertf(localtime_returned_null);
     }
-    strftime(s, 250, "%H:%M:%S", A);
+    strftime(s, 250, "%H:%M:%S", &tmv);
     if (strnotempty(type))
       fprintf(fp, "%s\t%c%s: \t", s, hichar(*type), type + 1);
     else
@@ -3774,8 +3773,7 @@ int htsAddLink(htsmoduleStruct * str, char *link) {
         strcpybuff(codebase, heap(ptr)->fil);
       else
         strcpybuff(codebase, heap(heap(ptr)->precedent)->fil);
-      // empty codebase has no last char; codebase-1 would underflow
-      a = codebase[0] != '\0' ? codebase + strlen(codebase) - 1 : codebase;
+      a = hts_lastcharptr(codebase);
       while((*a) && (*a != '/') && (a > codebase))
         a--;
       if (*a == '/')

src/htscoremain.c

@@ -2160,8 +2160,9 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
               case 'R':
                 {
                   char *name;
-                  uLong repaired = 0;
-                  uLong repairedBytes = 0;
+                  const char *why;
+                  unsigned long repaired = 0;
+                  unsigned long repairedBytes = 0;
 
                   if (fexist_utf8(fconcat(
                           OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
@@ -2184,24 +2185,15 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                     return 1;
                   }
                   fprintf(stderr, "Cache: trying to repair %s\n", name);
-                  if (unzRepair
-                      (name,
-                       fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                               "hts-cache/repair.zip"),
-                       fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
-                               "hts-cache/repair.tmp"), &repaired,
-                       &repairedBytes) == Z_OK) {
-                    UNLINK(name);
-                    RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
-                                   StringBuff(opt->path_log),
-                                   "hts-cache/repair.zip"),
-                           name);
-                    fprintf(stderr,
-                            "Cache: %d bytes successfully recovered in %d entries\n",
-                            (int) repairedBytes, (int) repaired);
-                  } else {
-                    fprintf(stderr, "Cache: could not repair the cache\n");
+                  why = cache_repair(opt, name, &repaired, &repairedBytes);
+                  if (why != NULL) {
+                    fprintf(stderr, "Cache: %s\n", why);
+                    return 1;
                   }
+                  fprintf(
+                      stderr,
+                      "Cache: %d bytes successfully recovered in %d entries\n",
+                      (int) repairedBytes, (int) repaired);
                 }
                 return 0;
                 break;

src/htsencoding.c

@@ -63,6 +63,13 @@ static int get_hex_value(char c) {
   } while (0)
 
 int hts_unescapeEntitiesWithCharset(const char *src, char *dest, const size_t max, const char *charset) {
+  return hts_unescapeEntitiesWithCharsetSpecial(src, dest, max, charset, 0);
+}
+
+int hts_unescapeEntitiesWithCharsetSpecial(const char *src, char *dest,
+                                           const size_t max,
+                                           const char *charset,
+                                           const int flags) {
   size_t i, j, ampStart, ampStartDest;
   int uc;
   int hex;
@@ -120,9 +127,18 @@ int hts_unescapeEntitiesWithCharset(const char *src, char *dest, const size_t ma
             char buffer[32];
             len = 0;
             if ( ( ulen = hts_writeUTF8(uc, buffer, sizeof(buffer)) ) != 0) {
+              const hts_boolean urlQuery =
+                  (flags & UNESCAPE_ENTITIES_URL_QUERY) != 0;
               char *s;
               buffer[ulen] = '\0';
-              s = hts_convertStringFromUTF8(buffer, strlen(buffer), charset);
+              /* Strict for a query only: a substituted '?' must not pass for
+                 the code point the document wrote. */
+              if (urlQuery) {
+                s = hts_convertStringFromUTF8Strict(buffer, strlen(buffer),
+                                                    charset);
+              } else {
+                s = hts_convertStringFromUTF8(buffer, strlen(buffer), charset);
+              }
               if (s != NULL) {
                 const size_t sLen = strlen(s);
                 if (sLen < maxOut) {
@@ -130,7 +146,18 @@ int hts_unescapeEntitiesWithCharset(const char *src, char *dest, const size_t ma
                   memcpy(&dest[ampStartDest], s, sLen);
                   len = sLen;
                 }
-                free(s);
+                freet(s);
+              } else if (urlQuery) {
+                /* URL Standard: an unrepresentable code point is written
+                   %26%23<decimal>%3B rather than left as source text. */
+                char esc[32];
+                const int escLen =
+                    snprintf(esc, sizeof(esc), "%%26%%23%d%%3B", uc);
+
+                if (escLen > 0 && (size_t) escLen < maxOut) {
+                  memcpy(&dest[ampStartDest], esc, (size_t) escLen);
+                  len = (size_t) escLen;
+                }
               }
             }
           }

src/htsencoding.h

@@ -69,6 +69,29 @@ extern int hts_unescapeEntitiesWithCharset(const char *src,
                                            char *dest, const size_t max,
                                            const char *charset);
 
+/**
+ * Flags for hts_unescapeEntitiesWithCharsetSpecial(). Values stay distinct from
+ * unescapeFlags above: both reach their function as a plain int.
+ **/
+typedef enum unescapeEntitiesFlags {
+  /** The destination is a URL query string: write a reference the charset can
+      not represent as %26%23<decimal>%3B (URL Standard), instead of source text
+      whose '&' and '#' would re-split the query. **/
+  UNESCAPE_ENTITIES_URL_QUERY = 2
+} unescapeEntitiesFlags;
+
+/**
+ * Unescape HTML entities into their charset equivalents, "flags" being a mask
+ * of UNESCAPE_ENTITIES_XXX constants.
+ * Note: source and destination MUST NOT be the same with a flag that may grow
+ * the string (UNESCAPE_ENTITIES_URL_QUERY).
+ * Returns 0 upon success, -1 upon overflow or error.
+ **/
+extern int hts_unescapeEntitiesWithCharsetSpecial(const char *src, char *dest,
+                                                  const size_t max,
+                                                  const char *charset,
+                                                  const int flags);
+
 /**
  * Unescape an URL-encoded string. The implicit charset is UTF-8.
  * In case of UTF-8 decoding error inside URL-encoded characters, 

src/htsftp.c

@@ -149,6 +149,61 @@ void ftp_split_userpass(const char *src, const char *end, char *user,
   }
 }
 
+/* Build "<verb> <path>", quoting a path the server could not parse bare. */
+static void ftp_command(char *line, size_t line_size, const char *verb,
+                        const char *path) {
+  if (strchr(path, ' ') != NULL || strchr(path, '\"') != NULL ||
+      strchr(path, '\'') != NULL)
+    snprintf(line, line_size, "%s \"%s\"", verb, path);
+  else
+    snprintf(line, line_size, "%s %s", verb, path);
+}
+
+/* MDTM reply "213 YYYYMMDDHHMMSS[.frac]" (RFC 3659, UTC) into tm_time. */
+static hts_boolean ftp_parse_mdtm(const char *line, struct tm *tm_time) {
+  int year, mon, mday, hour, min, sec;
+
+  if (sscanf(line, "213 %4d%2d%2d%2d%2d%2d", &year, &mon, &mday, &hour, &min,
+             &sec) != 6)
+    return HTS_FALSE;
+  if (year < 1900 || mon < 1 || mon > 12 || mday < 1 || mday > 31 || hour < 0 ||
+      hour > 23 || min < 0 || min > 59 || sec < 0 || sec > 60)
+    return HTS_FALSE;
+  memset(tm_time, 0, sizeof(*tm_time));
+  tm_time->tm_year = year - 1900;
+  tm_time->tm_mon = mon - 1;
+  tm_time->tm_mday = mday;
+  tm_time->tm_hour = hour;
+  tm_time->tm_min = min;
+  tm_time->tm_sec = sec;
+  tm_time->tm_isdst = 0;
+  return HTS_TRUE;
+}
+
+/* Whether the local copy can still be a prefix of the remote file. FTP has no
+   conditional fetch, so an unprovable answer has to be no (#823). */
+static hts_boolean ftp_may_resume(httrackp *opt, const lien_back *back,
+                                  time_t remote_mtime) {
+  const time_t local_mtime = get_filetime(back->url_sav);
+
+  if (back->r.totalsize <= back->range_req_size) {
+    hts_log_print(opt, LOG_DEBUG,
+                  "FTP: not resuming %s%s, the local copy is not shorter",
+                  back->url_adr, back->url_fil);
+    return HTS_FALSE;
+  }
+  /* Equality, not ordering: a copy this code did not stamp carries a local
+     clock time newer than any MDTM, which "not newer" would wave through. */
+  if (remote_mtime == (time_t) -1 || local_mtime == (time_t) -1 ||
+      remote_mtime != local_mtime) {
+    hts_log_print(opt, LOG_DEBUG,
+                  "FTP: not resuming %s%s, the copy is not dated as the remote",
+                  back->url_adr, back->url_fil);
+    return HTS_FALSE;
+  }
+  return HTS_TRUE;
+}
+
 // la véritable fonction une fois lancées les routines thread/fork
 int run_launch_ftp(FTPDownloadStruct * pStruct) {
   lien_back *back = pStruct->pBack;
@@ -184,6 +239,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
   strcpybuff(back->r.msg, "");
   back->r.statuscode = 0;
   back->r.size = 0;
+  back->r.lastmodified[0] = '\0'; // a retry must not stamp the previous MDTM
 
   // récupérer user et pass si présents, et sauter user:id@ dans adr
   real_adr = strchr(back->url_adr, ':');
@@ -458,14 +514,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
               char catbuff[CATBUFF_SIZE];
               char *ua = unescape_http(catbuff, sizeof(catbuff), ftp_filename);
 
-              if ((strchr(ua, ' '))
-                  || (strchr(ua, '\"'))
-                  || (strchr(ua, '\''))
-                ) {
-                snprintf(line, sizeof(line), "SIZE \"%s\"", ua);
-              } else {
-                snprintf(line, sizeof(line), "SIZE %s", ua);
-              }
+              ftp_command(line, sizeof(line), "SIZE", ua);
 
               // SIZE?
               strcpybuff(back->info, "size");
@@ -474,6 +523,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
               _CHECK_HALT_FTP;
               if (line[0] == '2') {     // SIZE compris, ALORS tester REST (sinon pas tester: cf probleme des txt.gz decompresses a la volee)
                 char *szstr = strchr(line, ' ');
+                time_t remote_mtime = (time_t) -1;
+                struct tm remote_tm;
 
                 if (szstr) {
                   LLint size = 0;
@@ -483,10 +534,31 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
                     back->r.totalsize = size;
                   }
                 }
+
+                // MDTM?
+                ftp_command(line, sizeof(line), "MDTM", ua);
+                strcpybuff(back->info, "mdtm");
+                send_line(soc_ctl, line);
+                get_ftp_line(soc_ctl, line, sizeof(line), timeout);
+                _CHECK_HALT_FTP;
+                if (ftp_parse_mdtm(line, &remote_tm)) {
+                  char date[256];
+
+                  time_rfc822(date, &remote_tm);
+                  /* Stamp the mirror as the HTTP path does, so a later pass
+                     compares server-clock times instead of crossing clocks. */
+                  back->r.lastmodified[0] = '\0';
+                  strlncatbuff(back->r.lastmodified, date,
+                               sizeof(back->r.lastmodified),
+                               sizeof(back->r.lastmodified) - 1);
+                  remote_mtime = timegm(&remote_tm);
+                }
+
                 /* Only over a copy back_add() judged partial: on --update every
                    mirrored file exists, and resuming a complete one splices the
                    old body into the new (#798). */
-                if (back->range_req_size > 0 && (transfer_list == 0)) {
+                if (back->range_req_size > 0 && (transfer_list == 0) &&
+                    ftp_may_resume(opt, back, remote_mtime)) {
                   strcpybuff(back->info, "rest");
                   snprintf(line, sizeof(line), "REST " LLintP,
                            (LLint) back->range_req_size);
@@ -695,6 +767,10 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
               fclose(back->r.fp);
               back->r.fp = NULL;
             }
+            /* back_flush_output() stamps only r.is_write transfers, which FTP
+               never is; a partial needs the remote's date too (#823). */
+            if (strnotempty(back->r.lastmodified))
+              set_filetime_rfc822(back->url_sav, back->r.lastmodified);
           } else {
             strcpybuff(back->r.msg, "Unable to write file");
             back->r.statuscode = STATUSCODE_INVALID;

src/htslib.c

@@ -1322,7 +1322,7 @@ void treathead(t_cookie * cookie, const char *adr, const char *fil, htsblk * ret
       p++;                      // sauter espaces
     if ((int) strlen(rcvd + p) < 250) { // pas trop long?
       char tmp[256];
-      char *a = NULL, *b = NULL;
+      char *a = NULL;
 
       strcpybuff(tmp, rcvd + p);
       a = strstr(tmp, "filename=");
@@ -1335,15 +1335,9 @@ void treathead(t_cookie * cookie, const char *adr, const char *fil, htsblk * ret
 
           while((c = strchr(a, '/')))   /* skip all / (see RFC2616) */
             a = c + 1;
-          b = a + strlen(a) - 1;
-          while(is_space(*b))
-            b--;
-          b++;
-          if (b) {
-            *b = '\0';
-            if ((int) strlen(a) < 200) {        // pas trop long?
-              strcpybuff(retour->cdispo, a);
-            }
+          hts_rtrim(a, HTS_SPACES);
+          if ((int) strlen(a) < 200) { // pas trop long?
+            strcpybuff(retour->cdispo, a);
           }
         }
       }
@@ -2690,13 +2684,15 @@ HTSEXT_API void qsec2str(char *st, TStamp t) {
 // heure actuelle, GMT, format rfc (taille buffer 256o)
 void time_gmt_rfc822(char *s) {
   time_t tt;
-  struct tm *A;
+  struct tm tmv;
 
   tt = time(NULL);
-  A = gmtime(&tt);
-  if (A == NULL)
-    A = localtime(&tt);
-  time_rfc822(s, A);
+  /* no local-time fallback: it would format local time and still label it GMT
+   * (#806) */
+  if (hts_gmtime(tt, &tmv))
+    time_rfc822(s, &tmv);
+  else
+    s[0] = '\0';
 }
 
 void hts_now_iso8601(char out[32]) {
@@ -2711,11 +2707,13 @@ void hts_now_iso8601(char out[32]) {
 // heure actuelle, format rfc (taille buffer 256o)
 void time_local_rfc822(char *s) {
   time_t tt;
-  struct tm *A;
+  struct tm tmv;
 
   tt = time(NULL);
-  A = localtime(&tt);
-  time_rfc822_local(s, A);
+  if (hts_localtime(tt, &tmv))
+    time_rfc822_local(s, &tmv);
+  else
+    s[0] = '\0';
 }
 
 /* convertir une chaine en temps */
@@ -2848,20 +2846,28 @@ int set_filetime_rfc822(const char *file, const char *date) {
     return -1;
 }
 
+/* Note: utf-8 */
+time_t get_filetime(const char *file) {
+  STRUCT_STAT buf;
+
+  if (STAT(file, &buf) != 0)
+    return (time_t) -1;
+  return (time_t) buf.st_mtime;
+}
+
 /* Note: utf-8 */
 int get_filetime_rfc822(const char *file, char *date) {
   STRUCT_STAT buf;
 
   date[0] = '\0';
   if (STAT(file, &buf) == 0) {
-    struct tm *A;
+    struct tm tmv;
     time_t tt = buf.st_mtime;
 
-    A = gmtime(&tt);
-    if (A == NULL)
-      A = localtime(&tt);
-    if (A != NULL) {
-      time_rfc822(date, A);
+    /* no local-time fallback: it would format local time and still label it GMT
+     * (#806) */
+    if (hts_gmtime(tt, &tmv)) {
+      time_rfc822(date, &tmv);
       return 1;
     }
   }
@@ -3306,8 +3312,9 @@ int ishtml(httrackp * opt, const char *fil) {
   }
 
   /* Search for known ext */
-  for(a = fil_noquery + strlen(fil_noquery) - 1;
-      *a != '.' && *a != '/' && a > fil_noquery; a--) ;
+  for (a = hts_lastcharptr(fil_noquery);
+       *a != '.' && *a != '/' && a > fil_noquery; a--)
+    ;
   if (*a == '.') {              // a une extension
     char BIGSTK fil_noquery[HTS_URLMAXSIZE * 2];
     char *b;
@@ -4244,9 +4251,8 @@ HTSEXT_API hts_boolean get_httptype_sized(httrackp *opt, char *s, size_t ssize,
     return 1;
   } else {
     /* Check html -> text/html */
-    const char *a = fil + strlen(fil) - 1;
+    const char *a = hts_lastcharptr(fil);
 
-    /* a < fil when fil is empty: bound before dereferencing */
     while ((a > fil) && (*a != '.') && (*a != '/'))
       a--;
     if (a >= fil && *a == '.' && strlen(a) < 32) {

src/htslib.h

@@ -170,6 +170,17 @@ static HTS_INLINE HTS_UNUSED hts_boolean hts_gmtime(time_t t,
 #endif
 }
 
+/* Break t down as local time into the caller's buffer, HTS_FALSE if that
+   failed. localtime()'s static is shared, same rationale as hts_gmtime(). */
+static HTS_INLINE HTS_UNUSED hts_boolean hts_localtime(time_t t,
+                                                       struct tm *tmbuf) {
+#ifdef _WIN32
+  return localtime_s(tmbuf, &t) == 0 ? HTS_TRUE : HTS_FALSE;
+#else
+  return localtime_r(&t, tmbuf) != NULL ? HTS_TRUE : HTS_FALSE;
+#endif
+}
+
 /* Library internal definictions */
 #ifdef HTS_INTERNAL_BYTECODE
 
@@ -278,6 +289,8 @@ void hts_now_iso8601(char out[32]);
 struct tm *convert_time_rfc822(struct tm *buffer, const char *s);
 int set_filetime(const char *file, struct tm *tm_time);
 int set_filetime_rfc822(const char *file, const char *date);
+/* File mtime as a UTC time_t, or (time_t) -1 if it can not be read. */
+time_t get_filetime(const char *file);
 int get_filetime_rfc822(const char *file, char *date);
 HTS_INLINE void time_rfc822(char *s, struct tm *A);
 HTS_INLINE void time_rfc822_local(char *s, struct tm *A);

src/htsname.c

@@ -65,14 +65,76 @@ Please visit our Website: http://www.httrack.com
 
 /* Avoid stupid DOS system folders/file such as 'nul' */
 /* Based on linux/fs/umsdos/mangle.c */
+/* Hand-grouped rows; clang-format reflows the table into a blob. */
+/* clang-format off */
 static const char *hts_tbdev[] = {
-  "/prn", "/con", "/aux", "/nul",
-  "/lpt1", "/lpt2", "/lpt3", "/lpt4",
-  "/com1", "/com2", "/com3", "/com4",
-  "/clock$",
-  "/emmxxxx0", "/xmsxxxx0", "/setverxx",
+  "prn", "con", "aux", "nul",
+  "lpt1", "lpt2", "lpt3", "lpt4",
+  "com1", "com2", "com3", "com4",
+  "clock$",
+  "emmxxxx0", "xmsxxxx0", "setverxx",
   ""
 };
+/* clang-format on */
+
+/* Directories the engine owns inside the mirror: a URL naming one lands on the
+   cache and destroys it (#774). Defence in depth; the temporaries themselves
+   live where no savename can spell them (HTS_TMPDIR in htsback.c). */
+static const char *hts_tbreserved[] = {"hts-cache", "hts-tmp", ""};
+
+/* True once the component holds only what cleanEndingSpaceOrDot() strips. */
+static hts_boolean strippedToComponentEnd(const char *s) {
+  while (*s == ' ' || *s == '.')
+    s++;
+  return *s == '\0' || *s == '/' ? HTS_TRUE : HTS_FALSE;
+}
+
+/* Replace foo/<reserved>/bar by foo/<reserved>_/bar, matching a whole path
+   component only (case-insensitively: the filesystem may be too). */
+static void escapeReservedNames(char *save, size_t size,
+                                const char *const *names) {
+  int i;
+
+  for (i = 0; names[i][0] != '\0'; i++) {
+    const char *a = save;
+    const size_t len = strlen(names[i]);
+
+    while ((a = strstrcase(a, names[i]))) {
+      hts_boolean reserved = HTS_FALSE;
+
+      if (a == save) {
+        /* save has had its leading '/' stripped above, so the table's anchor
+           never reached the first component (#842). It usually holds the
+           hostname, so only a trailing run may end it here: '.' would rename
+           aux.example.com. */
+        reserved = strippedToComponentEnd(a + len);
+      } else if (a[-1] == '/') {
+        switch ((int) a[len]) {
+        case '\0':
+        case '/':
+        case '.':
+          reserved = HTS_TRUE;
+          break;
+        case ' ':
+          /* cleanEndingSpaceOrDot() runs after us and hands the name back */
+          reserved = strippedToComponentEnd(a + len);
+          break;
+        }
+      }
+      if (reserved) {
+        char BIGSTK tempo[HTS_URLMAXSIZE * 2];
+
+        tempo[0] = '\0';
+        strncatbuff(tempo, save, (int) (a - save) + (int) len);
+        strcatbuff(tempo, "_");
+        strcatbuff(tempo, a + len);
+        /* clip rather than abort: the name comes from the wire */
+        (void) strclipbuff(save, size, tempo);
+      }
+      a += len;
+    }
+  }
+}
 
 /* Strip all // */
 static void cleanDoubleSlash(char *s) {
@@ -823,7 +885,7 @@ int url_savename(lien_adrfilsave *const afs,
   // Change the extension? e.g. php3 saved as html, cgi as html or gif/xbm
   // depending on the resolved type.
   if (ext_chg && !opt->no_type_change) {
-    char *a = fil + strlen(fil) - 1;
+    char *a = hts_lastcharptr(fil);
 
     if ((opt->debug > 1) && (opt->log != NULL)) {
       if (ext_chg == 1)
@@ -858,7 +920,7 @@ int url_savename(lien_adrfilsave *const afs,
   }
   // Rechercher premier / et dernier .
   {
-    const char *a = fil + strlen(fil) - 1;
+    const char *a = hts_lastcharptr(fil);
 
     // passer structures
     start_pos = fil;
@@ -1203,29 +1265,29 @@ int url_savename(lien_adrfilsave *const afs,
     switch (opt->savename_type % 100) {
     case 4:
     case 5:{                   // séparer par types
-        const char *a = fil + strlen(fil) - 1;
+      const char *a = hts_lastcharptr(fil);
 
-        // passer structures
-        while((a > fil) && (*a != '/') && (*a != '\\'))
+      // passer structures
+      while ((a > fil) && (*a != '/') && (*a != '\\'))
+        a--;
+      if ((*a == '/') || (*a == '\\'))
+        a++;
+
+      // html?
+      if ((ext_chg != 0) ? (ishtml_ext(ext) == 1) : (ishtml(opt, fil) == 1)) {
+        if (opt->savename_type % 100 == 5)
+          strcatbuff(afs->save, "html/");
+      } else {
+        const char *a = hts_lastcharptr(fil);
+
+        while ((a > fil) && (*a != '/') && (*a != '.'))
           a--;
-        if ((*a == '/') || (*a == '\\'))
-          a++;
-
-        // html?
-        if ((ext_chg != 0) ? (ishtml_ext(ext) == 1) : (ishtml(opt, fil) == 1)) {
-          if (opt->savename_type % 100 == 5)
-            strcatbuff(afs->save, "html/");
-        } else {
-          const char *a = fil + strlen(fil) - 1;
-
-          while((a > fil) && (*a != '/') && (*a != '.'))
-            a--;
-          if (*a != '.')
-            strcatbuff(afs->save, "other");
-          else
-            strcatbuff(afs->save, a + 1);
-          strcatbuff(afs->save, "/");
-        }
+        if (*a != '.')
+          strcatbuff(afs->save, "other");
+        else
+          strcatbuff(afs->save, a + 1);
+        strcatbuff(afs->save, "/");
+      }
         /*strcatbuff(save,a); */
         /* add name */
         ADD_STANDARD_NAME(0);
@@ -1258,7 +1320,7 @@ int url_savename(lien_adrfilsave *const afs,
         }
         afs->save[i + j] = '\0';
         // ajouter extension
-        a = fil + strlen(fil) - 1;
+        a = hts_lastcharptr(fil);
         while((a > fil) && (*a != '/') && (*a != '.'))
           a--;
         if (*a == '.') {
@@ -1303,7 +1365,7 @@ int url_savename(lien_adrfilsave *const afs,
   // cela évite les /chez/toto et les /chez/toto/index.html incompatibles
   if (opt->savename_type != -1 &&
       opt->savename_delayed != HTS_SAVENAME_DELAYED_HARD) {
-    char *a = afs->save + strlen(afs->save) - 1;
+    char *a = hts_lastcharptr(afs->save);
 
     while((a > afs->save) && (*a != '.') && (*a != '/'))
       a--;
@@ -1369,35 +1431,12 @@ int url_savename(lien_adrfilsave *const afs,
   // éliminer les // (comme ftp://)
   cleanDoubleSlash(afs->save);
 
+  /* Runs on every platform, and before path_html is prepended below, so the
+     user's own output directory is never renamed. */
+  escapeReservedNames(afs->save, sizeof(afs->save), hts_tbreserved);
+
 #if HTS_OVERRIDE_DOS_FOLDERS
-  /* Replace /foo/nul/bar by /foo/nul_/bar */
-  {
-    int i = 0;
-
-    while(hts_tbdev[i][0]) {
-      const char *a = afs->save;
-
-      while((a = strstrcase(a, hts_tbdev[i]))) {
-        switch ((int) a[strlen(hts_tbdev[i])]) {
-        case '\0':
-        case '/':
-        case '.':
-          {
-            char BIGSTK tempo[HTS_URLMAXSIZE * 2];
-
-            tempo[0] = '\0';
-            strncatbuff(tempo, afs->save, (int) (a - afs->save) + strlen(hts_tbdev[i]));
-            strcatbuff(tempo, "_");
-            strcatbuff(tempo, a + strlen(hts_tbdev[i]));
-            strcpybuff(afs->save, tempo);
-          }
-          break;
-        }
-        a += strlen(hts_tbdev[i]);
-      }
-      i++;
-    }
-  }
+  escapeReservedNames(afs->save, sizeof(afs->save), hts_tbdev);
 
   /* Strip ending . or ' ' forbidden on windoz */
   cleanEndingSpaceOrDot(afs->save);
@@ -1420,8 +1459,10 @@ int url_savename(lien_adrfilsave *const afs,
   if (opt->savename_83 > 0) {
     char *a, *last;
 
-    for(last = afs->save + strlen(afs->save) - 1;
-        last != afs->save && *last != '/' && *last != '\\' && *last != '.'; last--) ;
+    for (last = hts_lastcharptr(afs->save);
+         last != afs->save && *last != '/' && *last != '\\' && *last != '.';
+         last--)
+      ;
     if (*last != '.') {
       last = NULL;
     }
@@ -1504,6 +1545,7 @@ int url_savename(lien_adrfilsave *const afs,
 #endif
 #define MIN_LAST_SEG_RESERVE 12
 #define MAX_LAST_SEG_RESERVE 24
+#define MAX_EXT_LEN 12 /* longest tail kept across a cut, sans the dot */
   if (hts_stringLengthUTF8(afs->save) +
       hts_stringLengthUTF8(StringBuff(opt->path_html_utf8)) >=
       HTS_MAX_PATH_LEN) {
@@ -1572,9 +1614,26 @@ int url_savename(lien_adrfilsave *const afs,
           markStart = (p > lastSeg && p < extDot && wsave[p - 1] == '.')
                           ? p - 1
                           : extDot;
+        } else {
+          // #852: reserve a plain extension too, or the cut costs the page
+          // the ".html" the mirror is browsed by.
+          size_t p = wsaveLen;
+
+          while (p > lastSeg && wsave[p - 1] != '.')
+            p--;
+          if (p > lastSeg + 1 && wsaveLen - p <= MAX_EXT_LEN) {
+            markStart = p - 1;
+          }
         }
-        // head, bounded so the marker still fits, then the marker itself
-        for (i = lastSeg; i < markStart && j + (wsaveLen - markStart) < maxLen;
+        // #852: clamp the name like any directory segment; it was bounded by
+        // the whole path alone, so one component could run to maxLen.
+        const size_t tailLen = wsaveLen - markStart;
+        const size_t headLen =
+            tailLen < MAX_SEG_LEN ? MAX_SEG_LEN - tailLen : 0;
+
+        // head, bounded so the reserved tail still fits, then the tail itself
+        for (i = lastSeg; i < markStart && i - lastSeg < headLen &&
+                          j < maxLen && tailLen < maxLen - j;
              i++)
           wsave[j++] = wsave[i];
         for (i = markStart; i < wsaveLen && j < maxLen; i++)
@@ -1627,6 +1686,7 @@ int url_savename(lien_adrfilsave *const afs,
 #undef MAX_UTF8_SEQ_CHARS
 #undef MIN_LAST_SEG_RESERVE
 #undef MAX_LAST_SEG_RESERVE
+#undef MAX_EXT_LEN
 #undef MAX_SEG_LEN
 #undef HTS_MAX_PATH_LEN
 #undef HTS_PATH_TAIL_RESERVE
@@ -1676,8 +1736,8 @@ int url_savename(lien_adrfilsave *const afs,
 #endif
         } else {                // utilisé par un AUTRE, changer de nom
           char BIGSTK tempo[HTS_URLMAXSIZE * 2];
-          char *a = afs->save + strlen(afs->save) - 1;
-          char *b;
+          char *a = hts_lastcharptr(afs->save);
+          size_t stem;
           int n = 2;
           char collisionSeparator =
               ((opt->savename_83 != HTS_SAVENAME_83_ISO9660) ? '-' : '_');
@@ -1699,18 +1759,16 @@ int url_savename(lien_adrfilsave *const afs,
             strcatbuff(tempo, afs->save);
 
           // tester la présence d'un -xx (ex: index-2.html -> index-3.html)
-          b = tempo + strlen(tempo) - 1;
-          while(isdigit((unsigned char) *b))
-            b--;
-          if (*b == collisionSeparator) {
-            sscanf(b + 1, "%d", &n);
-            *b = '\0';          // couper
+          stem = hts_rtrimlen(tempo, "0123456789");
+          if (stem != 0 && tempo[stem - 1] == collisionSeparator) {
+            sscanf(tempo + stem, "%d", &n);
+            tempo[stem - 1] = '\0'; // couper
             n++;                // plus un
           }
           // en plus il faut gérer le 8-3 .. pas facile le client
           if (opt->savename_83) {
             int max;
-            char *a = tempo + strlen(tempo) - 1;
+            char *a = hts_lastcharptr(tempo);
 
             while((a > tempo) && (*a != '/'))
               a--;

src/htsopt.h

@@ -695,6 +695,9 @@ struct htsblk {
   char *warc_rawpath; /**< verbatim WARC: spooled compressed body path, or NULL
                            (owns the file; unlinked on free). Tail: ABI */
   LLint warc_rawsize; /**< byte length of warc_rawpath. Tail: ABI */
+  /** notmodified came from an engine hack, not a server 304 (#839).
+      Tail: ABI */
+  hts_boolean warc_forced_notmodified;
   /*char digest[32+2];   // md5 digest generated by the engine ("" if none) */
 };
 

src/htsparse.c

@@ -279,6 +279,20 @@ static void url_drop_fragment(char *const url) {
     *frag = '\0';
 }
 
+/* Charset to decode the page with, or NULL when conversion is off. *declared
+   tells a charset the document named from the iso-8859-1 guess standing in. */
+static const char *page_charset(const htsmoduleStruct *str,
+                                hts_boolean *declared) {
+  const char *const charset = str->page_charset_;
+
+  if (charset == NULL) {
+    *declared = HTS_FALSE;
+    return NULL;
+  }
+  *declared = *charset != '\0' ? HTS_TRUE : HTS_FALSE;
+  return *declared ? charset : "iso-8859-1";
+}
+
 /* True if [s, s+len) is exactly an HTTP method token (XHR.open's first
    argument is a method, not a URL: #218). Case-insensitive. */
 static int is_http_method(const char *s, size_t len) {
@@ -685,14 +699,22 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                         }
 
                         // Decode title with encoding
-                        if (str->page_charset_ != NULL &&
-                            *str->page_charset_ != '\0') {
-                          char *sUtf = hts_convertStringToUTF8(
-                              s, strlen(s), str->page_charset_);
-                          if (sUtf != NULL) {
-                            /* UTF-8 can expand past s[]; truncate to fit */
-                            snprintf(s, sizeof(s), "%s", sUtf);
-                            freet(sUtf);
+                        {
+                          hts_boolean declared;
+                          const char *const charset =
+                              page_charset(str, &declared);
+
+                          // Guessed charset: never re-encode valid UTF-8 (#833)
+                          if (charset != NULL && !hts_isCharsetUTF8(charset) &&
+                              (declared || !hts_isStringUTF8(s, strlen(s)))) {
+                            char *sUtf =
+                                hts_convertStringToUTF8(s, strlen(s), charset);
+
+                            if (sUtf != NULL) {
+                              /* UTF-8 can expand past s[]; truncate to fit */
+                              snprintf(s, sizeof(s), "%s", sUtf);
+                              freet(sUtf);
+                            }
                           }
                         }
 
@@ -2057,7 +2079,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                   }
                   q = strchr(a, '?');   // ne pas traiter après '?'
                   if (!q)
-                    q = a + strlen(a) - 1;
+                    q = hts_lastcharptr(a);
                   while((p = strstr(a, "//")) && (!done)) {     // remplacer // par /
                     if (p > q) {    // après le ? (toto.cgi?param=1//2.3)
                       done = 1; // stopper
@@ -2110,9 +2132,9 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                 /* Unescape/escape %20 and other   */
                 {
                   // NULL when UTF-8 conversion is off (-%T0)
-                  const char *const charset = str->page_charset_;
-                  const int hasCharset = charset != NULL 
-                    && *charset != '\0';
+                  hts_boolean declared;
+                  const char *const charset = page_charset(str, &declared);
+                  const int hasCharset = charset != NULL;
                   char BIGSTK query[HTS_URLMAXSIZE * 2];
 
                   // cut query string
@@ -2154,13 +2176,20 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                       "could not decode URI '%s' with charset '%s'", lien, charset);
                   }
 
-                  // decode query string entities with page charset
+                  // decode query entities with the page charset, out-of-place
+                  // because the escape grows it (#854)
                   if (hasCharset) {
-                    if (hts_unescapeEntitiesWithCharset(query, 
-                                                        query, strlen(query) + 1,
-                                                        charset) != 0) {
-                        hts_log_print(opt, LOG_WARNING,
-                          "could not decode query string '%s' with charset '%s'", query, charset);
+                    char BIGSTK decoded[sizeof(query)];
+
+                    if (hts_unescapeEntitiesWithCharsetSpecial(
+                            query, decoded, sizeof(decoded), charset,
+                            UNESCAPE_ENTITIES_URL_QUERY) == 0) {
+                      strcpybuff(query, decoded);
+                    } else {
+                      hts_log_print(opt, LOG_WARNING,
+                                    "could not decode query string '%s' with "
+                                    "charset '%s'",
+                                    query, charset);
                     }
                   }
 
@@ -2208,7 +2237,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                 // OUI!!
 #if HTS_TILDE_SLASH
                 if (hts_lastchar(lien) != '/') {
-                  char *a = lien + strlen(lien) - 1;
+                  char *a = hts_lastcharptr(lien);
 
                   // éviter aussi index~1.html
                   while(a > lien && (*a != '~') && (*a != '/')
@@ -2254,7 +2283,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                 // vérifier que l'on ne doit pas ajouter de .class
                 if (!error) {
                   if (add_class) {
-                    char *a = lien + strlen(lien) - 1;
+                    char *a = hts_lastcharptr(lien);
 
                     while((a > lien) && (*a != '/') && (*a != '.'))
                       a--;
@@ -2309,7 +2338,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                         {
                           char *a;
 
-                          a = lien + strlen(lien) - 1;
+                          a = hts_lastcharptr(lien);
                           while((*a) && (*a != '/') && (a > lien))
                             a--;
                           if (*a == '/') {
@@ -2320,8 +2349,8 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
                     }
 
                     if (!error) {       // pas d'erreur?
-                      if (p_type == 2) {        // code ET PAS codebase      
-                        char *a = lien + strlen(lien) - 1;
+                      if (p_type == 2) { // code ET PAS codebase
+                        char *a = hts_lastcharptr(lien);
                         char *start_of_filename = jump_identification(lien);
 
                         if (start_of_filename != NULL

src/htssafe.h

@@ -569,6 +569,38 @@ static HTS_INLINE HTS_UNUSED hts_boolean hts_choplastchar(char *s) {
   return HTS_FALSE;
 }
 
+/* htslib.h's is_space() and is_realspace() as hts_rtrim() sets; -#test=rtrim
+   keeps them in sync. */
+#define HTS_SPACES " \"\n\r\t\f\v'"
+#define HTS_REALSPACES " \n\r\t\f\v"
+
+/* Length of s once its trailing bytes from set are dropped; 0 if they all are.
+   Counts down from the end, so it stops at s rather than below the buffer. */
+static HTS_INLINE HTS_UNUSED size_t hts_rtrimlen(const char *s,
+                                                 const char *set) {
+  size_t len = strlen(s);
+
+  while (len != 0 && strchr(set, s[len - 1]) != NULL)
+    len--;
+  return len;
+}
+
+/* Drop the trailing bytes of s that occur in set. */
+static HTS_INLINE HTS_UNUSED void hts_rtrim(char *s, const char *set) {
+  s[hts_rtrimlen(s, set)] = '\0';
+}
+
+/* Offset of the last character of s, or 0 when s is empty. */
+static HTS_INLINE HTS_UNUSED size_t hts_lastcharoffset(const char *s) {
+  const size_t len = strlen(s);
+
+  return len != 0 ? len - 1 : 0;
+}
+
+/* Address of the last character of S, or of its terminating NUL when S is
+   empty. S is evaluated twice, so pass an lvalue. */
+#define hts_lastcharptr(S) ((S) + hts_lastcharoffset(S))
+
 /* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
    added bounds checking. freet() also NULLs the freed pointer and tolerates
    NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

src/htsselftest.c

@@ -4442,14 +4442,14 @@ static int st_warc(httrackp *opt, int argc, char **argv) {
       "GET /a.html HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n"
       "Transfer-Encoding : chunked\r\nContent-Length: 999\r\n\r\n",
-      a_body, sizeof(a_body) - 1, NULL, 200, 0, 0);
+      a_body, sizeof(a_body) - 1, NULL, NULL, 200, 0, 0);
 
   /* 302 redirect: header-only, no body. */
   warc_write_transaction(
       w, "http://test.local/r", "127.0.0.1",
       "GET /r HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 302 Found\r\nLocation: http://test.local/a.html\r\n\r\n", NULL,
-      0, NULL, 302, 0, 0);
+      0, NULL, NULL, 302, 0, 0);
 
   /* 200 binary, chunked coding on the wire (already de-chunked here). */
   warc_write_transaction(
@@ -4457,7 +4457,7 @@ static int st_warc(httrackp *opt, int argc, char **argv) {
       "GET /b.bin HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n"
       "Transfer-Encoding: chunked\r\n\r\n",
-      "\x00\x01\x02\x03\x04", 5, NULL, 200, 0, 0);
+      "\x00\x01\x02\x03\x04", 5, NULL, NULL, 200, 0, 0);
 
   /* 200 with a body shorter than the declared Content-Length (rewritten). */
   warc_write_transaction(
@@ -4465,20 +4465,20 @@ static int st_warc(httrackp *opt, int argc, char **argv) {
       "GET /trunc HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: "
       "100\r\n\r\n",
-      "short", 5, NULL, 200, 0, 0);
+      "short", 5, NULL, NULL, 200, 0, 0);
 
   /* Same payload as a.html at a new URL: identical-payload-digest revisit
      (OpenSSL builds only; a plain build writes a second full response). */
   warc_write_transaction(w, "http://test.local/a2.html", "127.0.0.1",
                          "GET /a2.html HTTP/1.1\r\nHost: test.local\r\n\r\n",
                          "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n",
-                         a_body, sizeof(a_body) - 1, NULL, 200, 0, 0);
+                         a_body, sizeof(a_body) - 1, NULL, NULL, 200, 0, 0);
 
   /* 304 revisit with an EMPTY response-header block: the block is just the
      2-byte separator, so declared Content-Length must be exactly 2 (F3). */
   warc_write_transaction(w, "http://test.local/nm", "127.0.0.1",
                          "GET /nm HTTP/1.1\r\nHost: test.local\r\n\r\n", "",
-                         NULL, 0, NULL, 304, 1, 0);
+                         NULL, 0, NULL, NULL, 304, 1, 0);
 
   warc_close(w);
 
@@ -4649,13 +4649,13 @@ static int st_warc_trunc(httrackp *opt, int argc, char **argv) {
       w, "http://test.local/big.bin", "127.0.0.1",
       "GET /big.bin HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n\r\n", body,
-      sizeof(body) - 1, NULL, 200, 0, WARC_TRUNC_LENGTH);
+      sizeof(body) - 1, NULL, NULL, 200, 0, WARC_TRUNC_LENGTH);
   warc_write_transaction(
       w, "http://test.local/big.gz", "127.0.0.1",
       "GET /big.gz HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Encoding: "
       "gzip\r\n\r\n",
-      (const char *) gz, sizeof(gz), NULL, 200, 0, WARC_TRUNC_TIME);
+      (const char *) gz, sizeof(gz), NULL, NULL, 200, 0, WARC_TRUNC_TIME);
   warc_close(w);
 
   data = warc_slurp(path, &data_len);
@@ -4809,7 +4809,7 @@ static int st_warc_rotate(httrackp *opt, int argc, char **argv) {
     warc_write_transaction(
         w, uri, "127.0.0.1", "GET / HTTP/1.1\r\nHost: test.local\r\n\r\n",
         "HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n\r\n",
-        (const char *) body, sizeof(body), NULL, 200, 0, 0);
+        (const char *) body, sizeof(body), NULL, NULL, 200, 0, 0);
   }
   warc_close(w);
   opt->warc_max_size = saved_max;
@@ -4891,7 +4891,7 @@ static int st_warc_verbatim(httrackp *opt, int argc, char **argv) {
       "GET /z.html HTTP/1.1\r\nHost: test.local\r\n\r\n",
       "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Encoding: "
       "gzip\r\nTransfer-Encoding: chunked\r\nContent-Length: 999\r\n\r\n",
-      (const char *) a_gz, sizeof(a_gz), NULL, 200, 0, 0);
+      (const char *) a_gz, sizeof(a_gz), NULL, NULL, 200, 0, 0);
   warc_close(w);
 
   data = warc_slurp(path, &data_len);
@@ -5057,7 +5057,7 @@ static int st_warc_longurl(httrackp *opt, int argc, char **argv) {
     /* Distinct payloads: identical ones dedupe into revisit records. */
     snprintf(body, sizeof(body), "<html><body>%04d</body></html>\n", (int) len);
     if (warc_write_transaction(w, uri, NULL, NULL, resp_hdr, body, strlen(body),
-                               NULL, 200, 0, 0) != 0) {
+                               NULL, NULL, 200, 0, 0) != 0) {
       fprintf(stderr, "warc-longurl: write failed at length %d\n", (int) len);
       err = 1;
     }
@@ -5117,7 +5117,7 @@ static int st_warc_cdx(httrackp *opt, int argc, char **argv) {
   unsigned char *warc = NULL, *cdx = NULL;
   size_t warc_len = 0, cdx_len = 0;
   hts_boolean saved_cdx;
-  int err = 0, nlines = 0;
+  int err = 0, nlines = 0, nm_lines = 0;
   const char *lp, *cend;
   char prev[2048];
 
@@ -5134,20 +5134,27 @@ static int st_warc_cdx(httrackp *opt, int argc, char **argv) {
   warc_write_transaction(w, "http://www.example.com/one", "127.0.0.1",
                          "GET /one HTTP/1.1\r\nHost: www.example.com\r\n\r\n",
                          "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n",
-                         "one body\n", 9, NULL, 200, 0, 0);
+                         "one body\n", 9, NULL, NULL, 200, 0, 0);
   warc_write_resource(w, "ftp://files.example.com/data.bin", "127.0.0.1",
                       "application/octet-stream", "\x00\x01\x02\x03", 4, NULL,
                       0);
   warc_write_transaction(w, "http://alpha.example.com/two", "127.0.0.1",
                          "GET /two HTTP/1.1\r\nHost: alpha.example.com\r\n\r\n",
                          "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n",
-                         "two body\n", 9, NULL, 200, 0, 0);
+                         "two body\n", 9, NULL, NULL, 200, 0, 0);
   /* Same payload as /one at a new URL: identical-payload-digest revisit under
      OpenSSL, a full response otherwise; either way one index line. */
   warc_write_transaction(w, "http://zeta.example.com/dup", "127.0.0.1",
                          "GET /dup HTTP/1.1\r\nHost: zeta.example.com\r\n\r\n",
                          "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n",
-                         "one body\n", 9, NULL, 200, 0, 0);
+                         "one body\n", 9, NULL, NULL, 200, 0, 0);
+  /* A 304 declares no type, so the index line takes the caller's (#826). */
+  warc_write_transaction(w, "http://nm.example.com/kept", "127.0.0.1",
+                         "GET /kept HTTP/1.1\r\nHost: nm.example.com\r\n"
+                         "If-Modified-Since: Mon, 01 Jan 2024 00:00:00 GMT\r\n"
+                         "\r\n",
+                         "HTTP/1.1 304 Not Modified\r\n\r\n", NULL, 0, NULL,
+                         "text/html", 200, 1, 0);
   warc_close(w);
   opt->warc_cdx = saved_cdx;
 
@@ -5200,6 +5207,11 @@ static int st_warc_cdx(httrackp *opt, int argc, char **argv) {
     }
     memcpy(url, us, urllen);
     url[urllen] = '\0';
+    if (strstr(url, "nm.example.com") != NULL) {
+      nm_lines++;
+      if (strstr(line, "\"mime\": \"text/html\"") == NULL)
+        err = 1;
+    }
     if (len == 0 || off > warc_len || len > warc_len - off) {
       err = 1;
       goto nextline;
@@ -5223,8 +5235,8 @@ static int st_warc_cdx(httrackp *opt, int argc, char **argv) {
   }
   freet(warc);
   freet(cdx);
-  if (nlines != 4)
-    err = 1; /* 3 responses/revisits + 1 resource; no warcinfo/request */
+  if (nlines != 5 || nm_lines != 1)
+    err = 1; /* 4 responses/revisits + 1 resource; no warcinfo/request */
   printf("warc-cdx: %d index lines: %s\n", nlines, err ? "FAIL" : "OK");
   return err;
 }
@@ -5291,13 +5303,13 @@ static int st_warc_wacz(httrackp *opt, int argc, char **argv) {
   warc_write_transaction(w, "http://www.example.com/", "127.0.0.1",
                          "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n",
                          "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n",
-                         "<html>home</html>\n", 18, NULL, 200, 0, 0);
+                         "<html>home</html>\n", 18, NULL, NULL, 200, 0, 0);
   warc_write_transaction(
       w, "http://www.example.com/data.bin", "127.0.0.1",
       "GET /data.bin HTTP/1.1\r\nHost: www.example.com\r\n\r\n",
       "HTTP/1.1 200 OK\r\n"
       "Content-Type: application/octet-stream\r\n\r\n",
-      "\x00\x01\x02\x03\x04", 5, NULL, 200, 0, 0);
+      "\x00\x01\x02\x03\x04", 5, NULL, NULL, 200, 0, 0);
   warc_close(w);
 
   /* Unzip every member in-process. */
@@ -5601,12 +5613,20 @@ static int sf_nesting(const char *hay, const char *mime) {
 static const char sf_png[] = "\x89PNG\r\n\x1a\n\x00\x01\x02\xff";
 #define SF_PNG_LEN 12
 
+static const char sf_svg[] = "<svg><g id=\"icon-a\"/></svg>";
+
 static const char sf_page[] =
     "<html><head>\n"
     "<link rel=\"stylesheet\" href=\"css/main.css\">\n"
     "<link rel=\"canonical\" href=\"other.html\">\n"
     "<title>t</title>\n"
-    "<style>body { background: url(\"img/a%20b.png\"); }</style>\n"
+    "<style>body { background: url(\"img/a%20b.png\"); }\n"
+    /* Everything the escape set has to neutralise before a fragment reaches an
+       unquoted url() token: whitespace, '<'/'>', then a quote, a paren, a
+       backslash and a high byte together. */
+    "b { background: url('img/sprite.svg#w x'); }\n"
+    "i { background: url('img/sprite.svg#lt<s>gt'); }\n"
+    "u { background: url(\"img/sprite.svg#z'(\\\xC3\xA9\"); }</style>\n"
     "</head><body>\n"
     "<img src=\"img/a%20b.png\" srcset=\"img/a%20b.png 1x, img/big.png 2x\">\n"
     "<link rel=\"icon\" href=\"icon.png\">\n"
@@ -5638,6 +5658,19 @@ static const char sf_page[] =
     "<script src=\"js/app.js\"></script>\n"
     "<script>var s = \"</scripting>\"; var t = \"<img src='img/a%20b.png'>\";"
     "</script>\n"
+    /* A fragment selects inside the asset, so it has to survive onto the
+       data: URI; the query named the remote resource and must not. */
+    "<img src=\"img/sprite.svg#icon-a\">\n"
+    "<img srcset=\"img/sprite.svg#icon-b 2x\">\n"
+    "<svg><image xlink:href=\"img/sprite.svg#icon-c\"/></svg>\n"
+    "<div style=\"background:url(img/sprite.svg#icon-d)\"></div>\n"
+    "<div style=\"background:url('img/sprite.svg#i)e')\"></div>\n"
+    "<img src=\"img/sprite.svg?v=1#icon-f\">\n"
+    /* Already carries the document's own escapes: re-encoding either would
+       make the browser look for a different id. */
+    "<img src=\"img/sprite.svg#g&amp;h%2Di\">\n"
+    /* A '"' left raw here would end the attribute the rewriter re-quotes. */
+    "<img src='img/sprite.svg#q\"z'>\n"
     "<img src=\"missing.png\" >\n"
     "<!--><img src=\"img/a%20b.png\">\n"
     "<div style=\"background:url(img/a%20b.png)\"></div>\n"
@@ -5656,6 +5689,10 @@ static void sf_fixture(const char *root) {
       "@font-face { font-family: f; src: url(../font/f.woff2); }\n"
       "body { background: url(../img/a b.png); }\n"
       "div { background: url(../img/big.png); }\n"
+      "div.s { background: url(../img/big-sprite.svg#icon-g); }\n"
+      /* A name whose escapes the rebase has to put back, unlike a fragment's;
+         the '#' has to come back encoded or it reads as one. */
+      "div.h { background: url(../img/b&amp;c%25d%23e.png); }\n"
       "/* url(../img/never.png) */\n";
   static const char nested[] = "div { background: url(../../img/a b.png); }\n";
   static const char two[] = "p { background: url(../../img/a b.png); }\n";
@@ -5675,6 +5712,9 @@ static void sf_fixture(const char *root) {
   sf_put(root, "js/app.js", js, sizeof(js) - 1);
   sf_put(root, "img/a b.png", sf_png, SF_PNG_LEN);
   sf_put(root, "img/big.png", big, sizeof(big));
+  sf_put(root, "img/sprite.svg", sf_svg, sizeof(sf_svg) - 1);
+  sf_put(root, "img/big-sprite.svg", big, sizeof(big));
+  sf_put(root, "img/b&c%d#e.png", big, sizeof(big));
   sf_put(root, "img/in.png", sf_png, SF_PNG_LEN);
   sf_put(root, "img/po.png", sf_png, SF_PNG_LEN);
   sf_put(root, "img/sv.png", sf_png, SF_PNG_LEN);
@@ -5811,6 +5851,29 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
                NULL,
            "quote inside a rewritten style attribute not escaped");
 
+  /* Fragments: kept on the replacement, escaped where they would close the
+     url() token, and never joined by the query the mirrored name dropped. */
+  sf_check(strstr(out, "img/sprite.svg") == NULL,
+           "a fragment-bearing reference was left a link");
+  sf_check(sf_count(out, "#icon-a\"") == 1, "img src fragment dropped");
+  sf_check(sf_count(out, "#icon-b 2x\"") == 1, "srcset fragment dropped");
+  sf_check(sf_count(out, "#icon-c\"") == 1, "xlink:href fragment dropped");
+  sf_check(sf_count(out, "#icon-d)") == 1, "style url() fragment dropped");
+  sf_check(sf_count(out, "#i%29e)") == 1,
+           "a fragment closing the url() token was not escaped");
+  sf_check(sf_count(out, "#icon-f\"") == 1, "fragment after a query dropped");
+  sf_check(strstr(out, "?v=1") == NULL, "query carried onto the data: URI");
+  sf_check(sf_count(out, "#g&amp;h%2Di\"") == 1,
+           "an escape the document already carried was encoded again");
+  sf_check(sf_count(out, "#q%22z\"") == 1,
+           "a quote in a fragment was not escaped");
+  sf_check(sf_count(out, "#w%20x)") == 1,
+           "whitespace in a fragment was not escaped");
+  sf_check(sf_count(out, "#lt%3Cs%3Egt)") == 1,
+           "'<'/'>' in a fragment were not escaped");
+  sf_check(sf_count(out, "#z%27%28%5C%C3%A9)") == 1,
+           "a quote, paren, backslash or high byte was left raw");
+
   sf_check(strstr(out, "img/big.png 2x") != NULL, "over-cap asset inlined");
   sf_check(strstr(out, " 1x") != NULL, "srcset descriptor lost");
   sf_check(sf_count(out, "img/a%20b.png") ==
@@ -5837,6 +5900,10 @@ static int st_singlefile(httrackp *opt, int argc, char **argv) {
        root, so it has to come back out as img/big.png or it dangles. */
     sf_check(strstr(css, "url(img/big.png)") != NULL,
              "over-cap url() not rebased onto the page's directory");
+    sf_check(strstr(css, "url(img/big-sprite.svg#icon-g)") != NULL,
+             "a rebased url() lost its fragment");
+    sf_check(strstr(css, "url(img/b%26c%25d%23e.png)") != NULL,
+             "a rebased name came back unescaped");
     nested = sf_decode(css, "text/css", NULL);
     sf_check(nested != NULL &&
                  strstr(nested, "url(data:image/png;base64,") != NULL,
@@ -6318,6 +6385,89 @@ static int st_renameover(httrackp *opt, int argc, char **argv) {
   return err;
 }
 
+// -#test=refetchbackup <dir>: the #77 re-fetch backup must build its temporary
+// inside the ~hts-tmp directory, which no save name can spell (#774), and must
+// never leave the resource without a copy (#775).
+static int st_refetchbackup(httrackp *opt, int argc, char **argv) {
+  lien_back *back;
+  char want[HTS_URLMAXSIZE * 2 + 32];
+  int err = 0;
+
+  if (argc < 1) {
+    fprintf(stderr, "refetchbackup: needs a writable base dir\n");
+    return 1;
+  }
+  back = calloct(1, sizeof(lien_back));
+  if (back == NULL) {
+    fprintf(stderr, "refetchbackup: out of memory\n");
+    return 1;
+  }
+  /* explicit separator: fconcat() joins without one, which would put the
+     temporary in the parent of the directory under test */
+  snprintf(back->url_sav, sizeof(back->url_sav), "%s/refetch.bin", argv[0]);
+  snprintf(want, sizeof(want), "%s/~hts-tmp/refetch.bin.bak", argv[0]);
+
+  /* #774: pin the name, so moving the temporary back into the mirror namespace
+     cannot pass unnoticed. */
+  ro_put(back->url_sav, "old");
+  back_refetch_backup(opt, back);
+  if (back->tmpfile == NULL || fexist_utf8(back->url_sav)) {
+    fprintf(stderr, "refetchbackup: the previous copy was not moved aside\n");
+    err++;
+  } else if (strcmp(back->tmpfile, want) != 0) {
+    fprintf(stderr, "refetchbackup: temporary is %s, want %s\n", back->tmpfile,
+            want);
+    err++;
+  }
+  ro_put(back->url_sav, "new"); /* what filecreate() + the transfer produce */
+  back_finalize_backup(opt, back, HTS_TRUE);
+  if (!ro_is(back->url_sav, "new")) {
+    fprintf(stderr, "refetchbackup: the committed copy is not the new one\n");
+    err++;
+  }
+
+  /* #758: only a killed run can leave something there, and it must be replaced
+     rather than disable the backup for good. */
+  if (structcheck(want) != 0) {
+    fprintf(stderr, "refetchbackup: cannot create %s\n", want);
+    freet(back);
+    return 1;
+  }
+  ro_put(want, "leftover");
+  back_refetch_backup(opt, back);
+  if (back->tmpfile == NULL || !ro_is(want, "new")) {
+    fprintf(stderr, "refetchbackup: a leftover temporary blocked the backup\n");
+    err++;
+  }
+
+  /* #775: filecreate() failed, so there is nothing to commit to. Saying so is
+     load-bearing: the caller must not cache this response against the old
+     body, or the next --update gets a 304 pinning it. */
+  (void) UNLINK(back->url_sav);
+  if (back_finalize_backup(opt, back, HTS_TRUE)) {
+    fprintf(stderr, "refetchbackup: a commit that restored reported success\n");
+    err++;
+  }
+  if (!ro_is(back->url_sav, "new")) {
+    fprintf(stderr, "refetchbackup: a commit with no new copy lost both\n");
+    err++;
+  }
+
+  /* An aborted transfer restores, as before. */
+  back_refetch_backup(opt, back);
+  ro_put(back->url_sav, "partial");
+  back_finalize_backup(opt, back, HTS_FALSE);
+  if (!ro_is(back->url_sav, "new")) {
+    fprintf(stderr, "refetchbackup: an aborted re-fetch kept the partial\n");
+    err++;
+  }
+
+  (void) UNLINK(back->url_sav);
+  freet(back);
+  printf("refetchbackup: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
 // -#test=direnum <dir>: enumerate a long+non-ASCII directory via the
 // opendir/readdir wrappers; children must round-trip as UTF-8 (#133,#630).
 static int st_direnum(httrackp *opt, int argc, char **argv) {
@@ -6797,6 +6947,149 @@ static int st_gmtime(httrackp *opt, int argc, char **argv) {
   return err;
 }
 
+/* #806: hts_localtime() must own its output too, same rationale as
+   hts_gmtime() (#794). Reference table computed under TZ=XXX5 (fixed
+   UTC-5, no DST), which the driving .test script sets. */
+#define LOCALTIME_THREADS 8
+#define LOCALTIME_ROUNDS 50000
+
+static const struct {
+  time_t t;
+  int year, mon, mday, hour, min, sec, wday, yday;
+} localtime_refs[] = {
+    {(time_t) 0, 69, 11, 31, 19, 0, 0, 3, 364},
+    {(time_t) 951782400, 100, 1, 28, 19, 0, 0, 1,
+     58}, /* a leap day, GMT side */
+    {(time_t) 1000000000, 101, 8, 8, 20, 46, 40, 6, 250},
+    {(time_t) 2147483647, 138, 0, 18, 22, 14, 7, 1, 17},
+};
+
+#define LOCALTIME_REFS                                                         \
+  ((int) (sizeof(localtime_refs) / sizeof(localtime_refs[0])))
+
+static hts_boolean localtime_ref_matches(int i, const struct tm *tm) {
+  if (tm->tm_year != localtime_refs[i].year ||
+      tm->tm_mon != localtime_refs[i].mon ||
+      tm->tm_mday != localtime_refs[i].mday ||
+      tm->tm_hour != localtime_refs[i].hour ||
+      tm->tm_min != localtime_refs[i].min ||
+      tm->tm_sec != localtime_refs[i].sec ||
+      tm->tm_wday != localtime_refs[i].wday ||
+      tm->tm_yday != localtime_refs[i].yday)
+    return HTS_FALSE;
+  return HTS_TRUE;
+}
+
+static htsmutex localtime_lock = HTSMUTEX_INIT;
+static int localtime_bad = 0;
+
+static void localtime_thread(void *arg) {
+  const int i = *(const int *) arg;
+  int bad = 0, round;
+
+  for (round = 0; round < LOCALTIME_ROUNDS; round++) {
+    struct tm tmv;
+
+    if (!hts_localtime(localtime_refs[i].t, &tmv) ||
+        !localtime_ref_matches(i, &tmv))
+      bad++;
+  }
+  hts_mutexlock(&localtime_lock);
+  localtime_bad += bad;
+  hts_mutexrelease(&localtime_lock);
+}
+
+static int st_localtime(httrackp *opt, int argc, char **argv) {
+  static int idx[LOCALTIME_THREADS];
+  int err = 0, i;
+
+  (void) opt;
+
+  if (argc < 1) {
+    fprintf(stderr, "usage: -#test=localtime <writable directory>\n");
+    return 1;
+  }
+
+  for (i = 0; i < LOCALTIME_REFS; i++) {
+    struct tm tmv;
+
+    if (!hts_localtime(localtime_refs[i].t, &tmv)) {
+      fprintf(stderr, "localtime: conversion #%d failed\n", i);
+      err = 1;
+    } else if (!localtime_ref_matches(i, &tmv)) {
+      fprintf(stderr,
+              "localtime: #%d gave %04d-%02d-%02d %02d:%02d:%02d (wday %d, "
+              "yday %d)\n",
+              i, tmv.tm_year + 1900, tmv.tm_mon + 1, tmv.tm_mday, tmv.tm_hour,
+              tmv.tm_min, tmv.tm_sec, tmv.tm_wday, tmv.tm_yday);
+      err = 1;
+    }
+  }
+
+  if (sizeof(time_t) >= 8) {
+    const time_t beyond = (time_t) INT64_MAX;
+    struct tm tmv;
+
+    if (hts_localtime(beyond, &tmv)) {
+      fprintf(stderr,
+              "localtime: an out-of-range time_t was reported converted\n");
+      err = 1;
+    }
+  }
+
+  for (i = 0; i < LOCALTIME_THREADS; i++) {
+    idx[i] = i % LOCALTIME_REFS;
+    if (hts_newthread(localtime_thread, &idx[i]) != 0) {
+      fprintf(stderr, "localtime: cannot spawn\n");
+      return 1;
+    }
+  }
+  htsthread_wait();
+  if (localtime_bad != 0) {
+    fprintf(stderr, "localtime: %d/%d concurrent conversions were corrupt\n",
+            localtime_bad, LOCALTIME_THREADS * LOCALTIME_ROUNDS);
+    err = 1;
+  }
+
+  /* get_filetime_rfc822() must report GMT, never the process's local zone,
+     and never a silent fallback to it on gmtime() failure (#806). */
+  {
+    char path[HTS_URLMAXSIZE];
+    char date[256];
+    struct tm parsed;
+
+    snprintf(path, sizeof(path), "%s/filetime.bin", argv[0]);
+    structcheck(path);
+    {
+      FILE *fp = FOPEN(path, "wb");
+
+      if (fp == NULL) {
+        fprintf(stderr, "localtime: cannot write %s\n", path);
+        return 1;
+      }
+      fputc('x', fp);
+      fclose(fp);
+    }
+    if (set_filetime_rfc822(path, "Tue, 29 Feb 2000 00:00:00 GMT") != 0) {
+      fprintf(stderr, "localtime: cannot set %s's mtime\n", path);
+      err = 1;
+    } else if (!get_filetime_rfc822(path, date)) {
+      fprintf(stderr, "localtime: get_filetime_rfc822 failed on %s\n", path);
+      err = 1;
+    } else if (convert_time_rfc822(&parsed, date) == NULL ||
+               parsed.tm_year != 100 || parsed.tm_mon != 1 ||
+               parsed.tm_mday != 29 || parsed.tm_hour != 0) {
+      fprintf(stderr,
+              "localtime: get_filetime_rfc822 reported \"%s\" (TZ=%s)\n", date,
+              getenv("TZ") ? getenv("TZ") : "");
+      err = 1;
+    }
+  }
+
+  printf("localtime self-test: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
 #define CHANGES_RACE_FILES 8
 #define CHANGES_RACE_ROUNDS 400
 
@@ -6918,9 +7211,10 @@ static int st_changes_race(httrackp *opt, int argc, char **argv) {
   return err;
 }
 
-/* The x[strlen(x) - 1] class (#770). The string starts mid-arena so the byte
-   it must not touch is a real neighbour; poisoned with '#', not 0, or a stray
-   NUL terminator would read as untouched. */
+/* The x[strlen(x) - 1] class (#770), its pointer spelling x + strlen(x) - 1
+   (#781) and its size_t index spelling (#821). The string starts mid-arena so
+   the byte it must not touch is a real neighbour; poisoned with '#', not 0, or
+   a stray NUL terminator would read as untouched. */
 static int st_lastchar(httrackp *opt, int argc, char **argv) {
   enum { off = 8 };
 
@@ -6987,11 +7281,82 @@ static int st_lastchar(httrackp *opt, int argc, char **argv) {
   CHECK(s[0] == '\0');
   CHECK(arena[guard] == '#');
 
+  /* the pointer spelling (#781): on an empty string the address must be the
+     terminating NUL, never the byte before it */
+  REPOISON("");
+  CHECK(hts_lastcharoffset(s) == 0);
+  CHECK(hts_lastcharptr(s) == s);
+  CHECK(*hts_lastcharptr(s) == '\0');
+  *hts_lastcharptr(s) = 'Z'; /* a write through it must stay inside s */
+  CHECK(arena[guard] == '#');
+  CHECK(s[0] == 'Z');
+
+  /* the neighbour must not be mistaken for the string's own last byte */
+  REPOISON("");
+  arena[guard] = '/';
+  CHECK(hts_lastcharptr(s) == s);
+  CHECK(*hts_lastcharptr(s) != '/');
+  CHECK(arena[guard] == '/');
+
+  /* the walk-back loops the sites use must stop at once on an empty string */
+  REPOISON("");
+  {
+    const char *p = hts_lastcharptr(s);
+    int steps = 0;
+
+    while (p > s && *p != '/')
+      p--, steps++;
+    CHECK(steps == 0);
+    CHECK(p == s);
+  }
+
+  REPOISON("ab/");
+  CHECK(hts_lastcharoffset(s) == 2);
+  CHECK(hts_lastcharptr(s) == s + 2);
+  CHECK(*hts_lastcharptr(s) == '/');
+  REPOISON("/");
+  CHECK(hts_lastcharptr(s) == s);
+  CHECK(*hts_lastcharptr(s) == '/');
+  CHECK(arena[guard] == '#');
+
+  /* the size_t index spelling (#821): (i > 0) cannot reject SIZE_MAX, so only
+     a safe seed stops the sites' walk-back */
+  REPOISON("");
+  {
+    const size_t vacuous = strlen(s) - 1;
+    size_t i = hts_lastcharoffset(s);
+    int steps = 0;
+
+    CHECK(vacuous > 0);
+    CHECK(i == 0);
+    /* step-capped so a bad seed fails the count instead of running off */
+    while ((i > 0) && (steps < 8) && (s[i] != '/'))
+      i--, steps++;
+    CHECK(steps == 0);
+    CHECK(s[i] != '/');
+    CHECK(arena[guard] == '#');
+  }
+
+  /* and the same loop must still find the real byte on a non-empty string */
+  REPOISON("a/b");
+  {
+    size_t i = hts_lastcharoffset(s);
+    int steps = 0;
+
+    while ((i > 0) && (steps < 8) && (s[i] != '/'))
+      i--, steps++;
+    CHECK(i == 1);
+    CHECK(arena[guard] == '#');
+  }
+
   /* control: the canary must be able to fail, or the checks above prove
      nothing. Clobber it exactly as the unguarded idiom would. */
   REPOISON("");
   s[-1] = '\0';
   CHECK(arena[guard] != '#');
+  REPOISON("");
+  *(s + strlen(s) - 1) = 'X';
+  CHECK(arena[guard] != '#');
 
 #undef REPOISON
 #undef CHECK
@@ -7000,6 +7365,91 @@ static int st_lastchar(httrackp *opt, int argc, char **argv) {
   return err;
 }
 
+/* hts_rtrim() and the sets it is called with. The string starts mid-arena, and
+   the byte below it is poisoned with '#' rather than 0, or the stray NUL the
+   old loop wrote there would read as untouched. */
+static int st_rtrim(httrackp *opt, int argc, char **argv) {
+  enum { off = 8 };
+
+  char arena[24];
+  char *const s = &arena[off];
+  const int guard = off - 1;
+  int err = 0;
+  int c;
+
+  (void) opt;
+  (void) argc;
+  (void) argv;
+
+#define REPOISON(str)                                                          \
+  do {                                                                         \
+    memset(arena, '#', sizeof(arena));                                         \
+    strlcpybuff(s, (str), sizeof(arena) - off);                                \
+  } while (0)
+#define CHECK(cond)                                                            \
+  do {                                                                         \
+    if (!(cond)) {                                                             \
+      printf("  FAIL line %d: %s\n", __LINE__, #cond);                         \
+      err = 1;                                                                 \
+    }                                                                          \
+  } while (0)
+
+  /* nothing but spaces: the case that ran the old loop off the front */
+  REPOISON("   ");
+  hts_rtrim(s, HTS_REALSPACES);
+  CHECK(s[0] == '\0');
+  CHECK(arena[guard] == '#');
+
+  /* a space sitting below the string must not be eaten as if it were part of
+     it, which is exactly what the old loop did */
+  REPOISON("   ");
+  arena[guard] = ' ';
+  hts_rtrim(s, HTS_REALSPACES);
+  CHECK(s[0] == '\0');
+  CHECK(arena[guard] == ' ');
+
+  REPOISON("");
+  hts_rtrim(s, HTS_REALSPACES);
+  CHECK(s[0] == '\0');
+  CHECK(arena[guard] == '#');
+
+  REPOISON("a b \t\r\n");
+  hts_rtrim(s, HTS_REALSPACES);
+  CHECK(strcmp(s, "a b") == 0);
+  CHECK(arena[guard] == '#');
+
+  REPOISON("ab");
+  hts_rtrim(s, HTS_REALSPACES);
+  CHECK(strcmp(s, "ab") == 0);
+
+  /* quotes count as space for is_space() but not for is_realspace() */
+  REPOISON("v\" ");
+  hts_rtrim(s, HTS_REALSPACES);
+  CHECK(strcmp(s, "v\"") == 0);
+  REPOISON("v\" ");
+  hts_rtrim(s, HTS_SPACES);
+  CHECK(strcmp(s, "v") == 0);
+
+  /* the sets must stay the macros they stand for */
+  for (c = 1; c < 256; c++) {
+    const char b = (char) c;
+
+    CHECK((strchr(HTS_SPACES, b) != NULL) == (is_space(b) != 0));
+    CHECK((strchr(HTS_REALSPACES, b) != NULL) == (is_realspace(b) != 0));
+  }
+
+  /* control: the canary must be able to fail */
+  REPOISON("   ");
+  s[-1] = '\0';
+  CHECK(arena[guard] != '#');
+
+#undef REPOISON
+#undef CHECK
+
+  printf("rtrim self-test: %s\n", err ? "FAIL" : "OK");
+  return err;
+}
+
 /* ------------------------------------------------------------ */
 /* Registry: name -> handler, with a usage hint and a one-line description. */
 /* ------------------------------------------------------------ */
@@ -7055,8 +7505,10 @@ static const struct selftest_entry {
     {"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
      "bounded string-op self-test", st_strsafe},
     {"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt},
-    {"lastchar", "", "last-char helpers never index before the buffer (#770)",
+    {"lastchar", "",
+     "last-char helpers never index before the buffer (#770, #781, #821)",
      st_lastchar},
+    {"rtrim", "", "hts_rtrim never walks below the buffer", st_rtrim},
     {"changes", "", "--changes bucket accounting and JSON escaping (#714)",
      st_changes},
     {"changes-race", "<dir>", "--changes under a late transfer thread (#714)",
@@ -7065,6 +7517,9 @@ static const struct selftest_entry {
      st_threadwait},
     {"gmtime", "",
      "hts_gmtime() fills the caller's buffer, not a static (#794)", st_gmtime},
+    {"localtime", "<dir>",
+     "hts_localtime() and get_filetime_rfc822()'s GMT labelling (#806)",
+     st_localtime},
     {"backswap", "", "which backlog slots may be swapped to the ready table",
      st_backswap},
     {"pause", "", "randomized inter-file pause target self-test", st_pause},
@@ -7173,6 +7628,9 @@ static const struct selftest_entry {
      "hts_rename_over(): replace dst, but never delete a dst it did not "
      "replace",
      st_renameover},
+    {"refetchbackup", "<dir>",
+     "the re-fetch backup always leaves a copy, and stays out of the mirror",
+     st_refetchbackup},
     {"direnum", "<dir>",
      "enumerate a long+non-ASCII directory through opendir/readdir",
      st_direnum},

src/htssinglefile.c

@@ -92,6 +92,30 @@ static int sf_is_space(int c) {
 
 static int sf_is_sep(int c) { return c == '/' || c == '\\'; }
 
+/* Drop the whitespace surrounding a reference. */
+static void sf_trim(const char **ref, size_t *reflen) {
+  while (*reflen > 0 && sf_is_space((unsigned char) **ref)) {
+    (*ref)++;
+    (*reflen)--;
+  }
+  while (*reflen > 0 && sf_is_space((unsigned char) (*ref)[*reflen - 1]))
+    (*reflen)--;
+}
+
+/* Text after a reference's first '#', or NULL when it carries no fragment. */
+static const char *sf_fragment(const char *ref, size_t reflen, size_t *len) {
+  size_t i;
+
+  sf_trim(&ref, &reflen);
+  for (i = 0; i < reflen; i++) {
+    if (ref[i] == '#') {
+      *len = reflen - i - 1;
+      return ref + i + 1;
+    }
+  }
+  return NULL;
+}
+
 /* Case-insensitive equality between the span [p,p+n) and a lowercase literal.
  */
 static hts_boolean sf_span_eq(const char *p, size_t n, const char *lit) {
@@ -202,12 +226,7 @@ static hts_boolean sf_resolve(const sf_ctx *ctx, const char *base_dir,
   size_t i, n;
   int sp = 0, part;
 
-  while (reflen > 0 && sf_is_space((unsigned char) *ref)) {
-    ref++;
-    reflen--;
-  }
-  while (reflen > 0 && sf_is_space((unsigned char) ref[reflen - 1]))
-    reflen--;
+  sf_trim(&ref, &reflen);
   /* A mirrored name carries no query or fragment. */
   for (n = 0; n < reflen; n++) {
     if (ref[n] == '#' || ref[n] == '?')
@@ -347,17 +366,20 @@ static hts_boolean sf_append_base64(String *out, char *data, size_t len) {
   return HTS_TRUE;
 }
 
-/* Append a mirror-relative path, percent-escaping everything an unquoted CSS
-   url() or an HTML attribute could choke on. */
-static void sf_append_escaped_path(String *out, const char *p) {
+/* Append [p,p+len), escaping what an unquoted CSS url(), an HTML attribute or
+   an appended fragment could choke on. preencoded text already carries the
+   document's own '%' and '&' escapes, and encoding those again changes it. */
+static void sf_append_escaped(String *out, const char *p, size_t len,
+                              hts_boolean preencoded) {
   static const char hex[] = "0123456789ABCDEF";
   size_t i;
 
-  for (i = 0; p[i] != '\0'; i++) {
+  for (i = 0; i < len; i++) {
     const unsigned char c = (unsigned char) p[i];
 
     if (c <= 32 || c >= 127 || c == '"' || c == '\'' || c == '(' || c == ')' ||
-        c == '\\' || c == '<' || c == '>' || c == '&' || c == '%') {
+        c == '\\' || c == '<' || c == '>' || c == '#' ||
+        (!preencoded && (c == '%' || c == '&'))) {
       StringAddchar(*out, '%');
       StringAddchar(*out, hex[c >> 4]);
       StringAddchar(*out, hex[c & 15]);
@@ -367,6 +389,13 @@ static void sf_append_escaped_path(String *out, const char *p) {
   }
 }
 
+/* Re-attach a fragment: it selects inside the asset (an SVG sprite id) rather
+   than naming it, so the replacement needs it too. */
+static void sf_append_fragment(String *out, const char *frag, size_t len) {
+  StringAddchar(*out, '#');
+  sf_append_escaped(out, frag, len, HTS_TRUE);
+}
+
 static void sf_warn_oversize(sf_ctx *ctx, const char *path, LLint size,
                              LLint cap) {
   if (*ctx->warn_budget <= 0)
@@ -399,7 +428,8 @@ static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
   String path = STRING_EMPTY;
   char mime[HTS_MIMETYPE_SIZE];
   char *body;
-  size_t body_len = 0;
+  size_t body_len = 0, frag_len = 0;
+  const char *const frag = sf_fragment(ref, reflen, &frag_len);
   LLint size, cap;
   int cls;
   hts_boolean done = HTS_FALSE;
@@ -457,6 +487,8 @@ static hts_boolean sf_inline(sf_ctx *ctx, const char *base_dir, const char *ref,
       StringCat(*out, mime);
       StringCat(*out, ";base64,");
       StringMemcat(*out, StringBuff(payload), StringLength(payload));
+      if (frag != NULL)
+        sf_append_fragment(out, frag, frag_len);
     }
     StringFree(payload);
   }
@@ -475,7 +507,9 @@ fallback:
     StringClear(rel);
     sf_relative_from(rebase_dir, StringBuff(path), &rel);
     if (StringLength(rel) > 0) {
-      sf_append_escaped_path(out, StringBuff(rel));
+      sf_append_escaped(out, StringBuff(rel), StringLength(rel), HTS_FALSE);
+      if (frag != NULL)
+        sf_append_fragment(out, frag, frag_len);
       done = HTS_TRUE;
     }
     StringFree(rel);

src/htsthread.c

@@ -148,20 +148,23 @@ HTSEXT_API int hts_newthread(void (*fun) (void *arg), void *arg) {
     const size_t stackSize = 1024 * 1024 * 8;
     pthread_attr_t attr;
     pthread_t handle = 0;
-    int retcode;
+    hts_boolean created;
 
-    if (pthread_attr_init(&attr) != 0
-        || pthread_attr_setstacksize(&attr, stackSize) != 0
-        || (retcode =
-            pthread_create(&handle, &attr, hts_entry_point, s_args)) != 0) {
+    /* init kept apart: destroying an uninitialised attr is undefined (#772) */
+    if (pthread_attr_init(&attr) == 0) {
+      created = pthread_attr_setstacksize(&attr, stackSize) == 0 &&
+                pthread_create(&handle, &attr, hts_entry_point, s_args) == 0;
+      pthread_attr_destroy(&attr); /* create() copied what it needed */
+    } else {
+      created = HTS_FALSE;
+    }
+    if (!created) {
       process_chain_add(-1);
       freet(s_args);
       return -1;
-    } else {
-      /* detach the thread from the main process so that is can be independent */
-      pthread_detach(handle);
-      pthread_attr_destroy(&attr);
     }
+    /* detach the thread from the main process so that it can be independent */
+    pthread_detach(handle);
   }
 #endif
   return 0;

src/htstools.c

@@ -324,8 +324,7 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
   curr = _curr;
   strlcpybuff(curr, curr_fil, sizeof(_curr));
   if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
-    // an empty path has no last character: curr-1 would read before the buffer
-    a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
+    a = hts_lastcharptr(curr);
   }
   while((*a != '/') && (a > curr))
     a--;                        // chercher dernier / du chemin courant

src/htswarc.c

@@ -38,6 +38,7 @@ Please visit our Website: http://www.httrack.com
 
 #include "htscore.h"
 #include "htslib.h"
+#include "htsback.h"
 #include "htstools.h"
 #include "htssafe.h"
 #include "htszlib.h"
@@ -93,7 +94,8 @@ struct warc_writer {
   hts_boolean protect_prev;   /* previous archive present: build in a temp */
   hts_boolean opened;         /* open completed; a failed one swaps nothing */
   hts_boolean failed;         /* a record or segment was lost: swap nothing */
-  uint64_t unbacked_revisits; /* revisits whose payload no file here holds */
+  uint64_t unbacked_revisits; /* URLs this pass didn't capture here: an
+                                  unbacked revisit, or nothing written at all */
   char **page_lines; /* one JSON page line per 200 text/html response, owned */
   size_t page_count;
   size_t page_cap;
@@ -1354,12 +1356,24 @@ void warc_free_request(htsblk *r) {
     freet(r->warc_resphdr);
     if (r->warc_rawpath != NULL) {
       (void) UNLINK(r->warc_rawpath); /* owns the verbatim spool file */
+      back_tmpdir_drop(r->warc_rawpath);
       freet(r->warc_rawpath);
       r->warc_rawpath = NULL;
     }
   }
 }
 
+void warc_move_request(htsblk *src, htsblk *dst) {
+  if (src == NULL || dst == NULL || src == dst)
+    return;
+  freet(dst->warc_reqhdr);
+  freet(dst->warc_resphdr);
+  dst->warc_reqhdr = src->warc_reqhdr;
+  dst->warc_resphdr = src->warc_resphdr;
+  src->warc_reqhdr = NULL;
+  src->warc_resphdr = NULL;
+}
+
 void warc_adopt_rawspool(htsblk *r, const char *tmpfile_path) {
   if (r != NULL) {
     LLint rawsize;
@@ -1599,7 +1613,7 @@ static hts_boolean warc_commit(warc_writer *w) {
     hts_log_print(
         w->opt, LOG_ERROR,
         "WARC: this pass revisited %llu URL(s) without re-downloading them, so "
-        "its archive would name bodies no file holds; kept the previous %s "
+        "this archive doesn't hold their current content; kept the previous %s "
         "(re-run with -C0, or --warc-file with a name of its own)",
         (unsigned long long) w->unbacked_revisits,
         warc_seg_path(w, 0, finalbuf, sizeof(finalbuf)));
@@ -1664,8 +1678,8 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
                            const char *ip, const char *req_hdr,
                            const char *resp_hdr, const char *body,
                            size_t body_len, const char *body_path,
-                           int statuscode, int is_update_unchanged,
-                           int truncated) {
+                           const char *content_type, int statuscode,
+                           int unchanged_kind, int truncated) {
   wbuf http;
   char resp_id[64];
   char pdig[33];
@@ -1688,16 +1702,19 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
    */
   snprintf(statusbuf, sizeof(statusbuf), "%d", statuscode);
   http_header_value(resp_hdr, "Content-Type", mimebuf, sizeof(mimebuf));
+  /* a 304 declares no type, so index the one only the caller knows */
+  if (mimebuf[0] == '\0' && content_type != NULL)
+    strlncatbuff(mimebuf, content_type, sizeof(mimebuf), sizeof(mimebuf) - 1);
 
   /* A payload exists (for digesting) unless this is a bodyless 304. */
   has_payload = (body_len > 0 && (body != NULL || body_path != NULL) &&
-                 !is_update_unchanged);
+                 unchanged_kind != WARC_UNCHANGED_SERVER_304);
 
   /* Payload digest drives identical-payload-digest dedup (OpenSSL only). */
   have_pdig =
       has_payload ? payload_digest_b32(body, body_len, body_path, pdig) : 0;
 
-  if (is_update_unchanged) {
+  if (unchanged_kind == WARC_UNCHANGED_SERVER_304) {
     is_revisit = 1;
     profile = "http://netpreserve.org/warc/1.1/revisit/server-not-modified";
     /* Replay resolves a revisit by this field alone, and a 304 stands in for
@@ -1706,6 +1723,22 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
     refers_uri = target_uri;
     /* Served from cache: the payload sits in the previous archive, not here. */
     w->unbacked_revisits++;
+  } else if (unchanged_kind == WARC_UNCHANGED_ENGINE_FORCED) {
+    /* has_payload requires body_len>0, so a genuinely empty body looks
+       digest-less too; it still has a well-defined digest (sha1 of nothing),
+       so compute it here rather than treat it as a missing-crypto case. */
+    if (!have_pdig && body_len == 0 && (body != NULL || body_path != NULL))
+      have_pdig = payload_digest_b32(body, body_len, body_path, pdig);
+    /* Served from cache with no exchange either way: still unbacked. */
+    w->unbacked_revisits++;
+    /* No digest (no OpenSSL) means nothing to point a revisit at, and there
+       was no real exchange to record as a response; write nothing (#839). */
+    if (!have_pdig)
+      return 0;
+    is_revisit = 1;
+    profile =
+        "http://netpreserve.org/warc/1.1/revisit/identical-payload-digest";
+    refers_uri = target_uri;
   } else if (have_pdig && w->seen != NULL) {
     void *prev = NULL;
     if (coucal_read_pvoid(w->seen, pdig, &prev) && prev != NULL) {
@@ -1815,7 +1848,7 @@ void warc_write_backtransaction(httrackp *opt, lien_back *back) {
   const char *body_path;
   const char *resp_hdr;
   char synth[512];
-  int is_unchanged;
+  int unchanged_kind;
   int is_ftp;
 
   if (opt->state.warc == WARC_DISABLED)
@@ -1876,7 +1909,12 @@ void warc_write_backtransaction(httrackp *opt, lien_back *back) {
     body_len = (size_t) back->r.warc_rawsize;
   }
 
-  is_unchanged = (back->r.notmodified && opt->is_update) ? 1 : 0;
+  if (!back->r.notmodified || !opt->is_update)
+    unchanged_kind = WARC_UNCHANGED_NONE;
+  else if (back->r.warc_forced_notmodified)
+    unchanged_kind = WARC_UNCHANGED_ENGINE_FORCED;
+  else
+    unchanged_kind = WARC_UNCHANGED_SERVER_304;
 
   /* Prefer the stashed raw headers; synthesize a minimal status line for the
      header-less (HTTP/0.9-style) responses that never carried a header block.
@@ -1891,6 +1929,7 @@ void warc_write_backtransaction(httrackp *opt, lien_back *back) {
   }
 
   warc_write_transaction(w, uri, ip, back->r.warc_reqhdr, resp_hdr, body,
-                         body_len, body_path, back->r.statuscode, is_unchanged,
+                         body_len, body_path, back->r.contenttype,
+                         back->r.statuscode, unchanged_kind,
                          back->r.warc_truncated);
 }

src/htswarc.h

@@ -68,6 +68,10 @@ void warc_stash_response(htsblk *r, const char *resphdr);
 /* Free both stashed header blocks (idempotent, NULL-safe). */
 void warc_free_request(htsblk *r);
 
+/* Move both stashed header blocks from src to dst, freeing dst's own. For the
+   304 path, which replaces the whole htsblk with the cache entry. */
+void warc_move_request(htsblk *src, htsblk *dst);
+
 /* Adopt the de-chunked compressed spool at tmpfile_path onto
    r->warc_rawpath/warc_rawsize (strdupt; frees any prior) so the WARC record
    stores the body verbatim. No-op leaving warc_rawpath NULL when tmpfile_path
@@ -97,6 +101,12 @@ void warc_close(warc_writer *w);
    success, -1 on error or truncation. Exposed for the -#test=warc-surt test. */
 int warc_surt(const char *url, char *out, size_t outsz);
 
+/* unchanged_kind for warc_write_transaction. */
+#define WARC_UNCHANGED_NONE 0       /* a fresh exchange: a full response */
+#define WARC_UNCHANGED_SERVER_304 1 /* the server itself answered 304 */
+#define WARC_UNCHANGED_ENGINE_FORCED                                           \
+  2 /* the engine, not the server, decided nothing changed (#839) */
+
 /* Write one transaction's request + response (or revisit) records.
    target_uri:  absolute URL fetched.
    ip:          numeric peer IP, or NULL/"" to omit.
@@ -104,7 +114,11 @@ int warc_surt(const char *url, char *out, size_t outsz);
    resp_hdr:    raw received response header block (status line + headers).
    body/body_len: decoded in-memory body, or NULL when on disk.
    body_path:   file re-read for the body when body==NULL (may be NULL).
-   is_update_unchanged: nonzero for a 304 server-not-modified revisit.
+   content_type: CDXJ mime when resp_hdr declares none, as a 304 does; NULL
+                to leave the index line without one.
+   unchanged_kind: WARC_UNCHANGED_*; SERVER_304 writes a server-not-modified
+                revisit, ENGINE_FORCED a self-referencing identical-payload-
+                digest one (no 304 to claim), both unbacked (#839).
    truncated: a WARC_TRUNC_* reason to tag a cap-truncated body, else 0.
    The body is stored verbatim: Content-Encoding is kept and Content-Length set
    to body_len, so body/body_len must be the as-received (coded) bytes.
@@ -113,8 +127,8 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
                            const char *ip, const char *req_hdr,
                            const char *resp_hdr, const char *body,
                            size_t body_len, const char *body_path,
-                           int statuscode, int is_update_unchanged,
-                           int truncated);
+                           const char *content_type, int statuscode,
+                           int unchanged_kind, int truncated);
 
 /* Write one non-HTTP capture as a single WARC 'resource' record: the block is
    the raw payload (no HTTP envelope), Content-Type is the payload's own MIME.

src/htswizard.c

@@ -374,8 +374,8 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
         forbidden_url = 1;
       break;                    // interdicton de sortir au dela de l'adresse
     case HTS_TRAVEL_SAME_DOMAIN: {
-      size_t i = strlen(adr) - 1;
-      size_t j = strlen(urladr()) - 1;
+      size_t i = hts_lastcharoffset(adr);
+      size_t j = hts_lastcharoffset(urladr());
 
       if ((i > 0) && (j > 0)) {
         while ((i > 0) && (adr[i] != '.'))
@@ -411,8 +411,8 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
         forbidden_url = 1;
     } break;
     case HTS_TRAVEL_SAME_TLD: {
-      size_t i = strlen(adr) - 1;
-      size_t j = strlen(urladr()) - 1;
+      size_t i = hts_lastcharoffset(adr);
+      size_t j = hts_lastcharoffset(urladr());
 
       while ((i > 0) && (adr[i] != '.'))
         i--;
@@ -728,7 +728,7 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
       case 1: // forbid the whole directory and subdirs: adr/path/*
         forbidden_url = 1;
         {
-          size_t i = strlen(fil) - 1;
+          size_t i = hts_lastcharoffset(fil);
 
           while((fil[i] != '/') && (i > 0))
             i--;
@@ -794,7 +794,7 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
 
       case 5: // allow the whole directory and its children
         if ((opt->seeker & HTS_SEEKER_UP) == 0) { // not allowed to go up
-          size_t i = strlen(fil) - 1;
+          size_t i = hts_lastcharoffset(fil);
 
           while((fil[i] != '/') && (i > 0))
             i--;
@@ -836,7 +836,7 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
         //
       case 7: // allow this directory
       {
-        size_t i = strlen(fil) - 1;
+        size_t i = hts_lastcharoffset(fil);
 
         while ((fil[i] != '/') && (i > 0))
           i--;

src/proxy/proxytrack.c

@@ -748,7 +748,8 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
             const char *thisUrl = list[i];
             const char *mimeType = "application/octet-stream";
             unsigned int thisUrlLen = (unsigned int) strlen(thisUrl);
-            int thisIsDir = (thisUrl[thisUrlLen - 1] == '/') ? 1 : 0;
+            /* the folder's default document is enumerated as an empty name */
+            int thisIsDir = (hts_lastchar(thisUrl) == '/') ? 1 : 0;
 
             /* Item URL */
             StringRoom(itemUrl,
@@ -853,8 +854,7 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
     for(isDir = 1; isDir >= 0; isDir--) {
       for(i = 0; list[i] != NULL; i++) {
         char *thisUrl = list[i];
-        unsigned int thisUrlLen = (unsigned int) strlen(thisUrl);
-        int thisIsDir = (thisUrl[thisUrlLen - 1] == '/') ? 1 : 0;
+        int thisIsDir = (hts_lastchar(thisUrl) == '/') ? 1 : 0;
 
         if (thisIsDir == isDir) {
           if (isDir)

src/proxy/store.c

@@ -259,7 +259,6 @@ static int binput(char *buff, char *s, int max) {
   int count = 0;
   int destCount = 0;
 
-  // Note: \0 will return 1
   while(destCount < max && buff[count] != '\0' && buff[count] != '\n') {
     if (buff[count] != '\r') {
       s[destCount++] = buff[count];
@@ -268,8 +267,9 @@ static int binput(char *buff, char *s, int max) {
   }
   s[destCount] = '\0';
 
-  // then return the supplemental jump offset
-  return count + 1;
+  /* only step over a real separator: past the terminating NUL leaves the
+     caller's buffer, past a truncated field's next byte silently eats it */
+  return buff[count] == '\n' ? count + 1 : count;
 }
 
 static time_t file_timestamp(const char *file) {
@@ -888,6 +888,31 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
 	} \
 } while(0)
 
+/* Set path (capacity size) to filename's parent directory, separator included,
+   from an absolute filename. Empty when there is none, or when it would not
+   fit: a truncated prefix names a different directory. */
+static void index_base_path(char *path, size_t size, const char *filename) {
+  const char *abpath;
+  int slashes;
+
+  for (slashes = 2, abpath = hts_lastcharptr(filename);
+       abpath > filename &&
+       ((*abpath != '/' && *abpath != '\\') || --slashes > 0);
+       abpath--)
+    ;
+  path[0] = '\0';
+  if (slashes == 0 && *abpath != 0 && (size_t) (abpath - filename) < size - 1) {
+    int i;
+
+    strlncatbuff(path, filename, size, (size_t) (abpath - filename) + 1);
+    for (i = 0; path[i] != 0; i++) {
+      if (path[i] == '\\') {
+        path[i] = '/';
+      }
+    }
+  }
+}
+
 int PT_LoadCache__New(PT_Index index_, const char *filename) {
   if (index_ != NULL && filename != NULL) {
     PT_Index__New index = &index_->slots.formatNew;
@@ -898,26 +923,9 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
 
     // Opened ?
     if (zFile != NULL) {
-      const char *abpath;
-      int slashes;
       coucal hashtable = index->hash;
 
-      /* Compute base path for this index - the filename MUST be absolute! */
-      for(slashes = 2, abpath = filename + (int) strlen(filename) - 1;
-          abpath > filename && ((*abpath != '/' && *abpath != '\\')
-                                || --slashes > 0);
-          abpath--) ;
-      index->path[0] = '\0';
-      if (slashes == 0 && *abpath != 0) {
-        int i;
-
-        strncat(index->path, filename, (int) (abpath - filename) + 1);
-        for(i = 0; index->path[i] != 0; i++) {
-          if (index->path[i] == '\\') {
-            index->path[i] = '/';
-          }
-        }
-      }
+      index_base_path(index->path, sizeof(index->path), filename);
 
       /* Ready directory entries */
       if (unzGoToFirstFile(zFile) == Z_OK) {
@@ -1382,11 +1390,10 @@ static int cache_brstr(char *adr, char *s, size_t s_size) {
   char buff[256 + 4];
 
   off = binput(adr, buff, 256);
-  /* binput stops at adr's terminating NUL; a value only follows a real line
-     terminator, so never step past end-of-buffer. */
-  if (adr[off - 1] == '\0') {
+  /* no length-prefixed value follows a field the terminating NUL stopped */
+  if (adr[off] == '\0') {
     s[0] = '\0';
-    return off - 1;
+    return off;
   }
   /* an empty/non-numeric field leaves i unset: treat as length 0 */
   if (sscanf(buff, "%d", &i) != 1 || i < 0 || i > 32768)
@@ -1490,41 +1497,23 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
 
     cache->filenameDat[0] = '\0';
     cache->filenameNdx[0] = '\0';
-    cache->path[0] = '\0';
 
-    {
-      PT_Index__Old index = cache;
-      const char *abpath;
-      int slashes;
-
-      /* -------------------- COPY OF THE __New() CODE -------------------- */
-      /* Compute base path for this index - the filename MUST be absolute! */
-      for(slashes = 2, abpath = filename + (int) strlen(filename) - 1;
-          abpath > filename && ((*abpath != '/' && *abpath != '\\')
-                                || --slashes > 0);
-          abpath--) ;
-      index->path[0] = '\0';
-      if (slashes == 0 && *abpath != 0) {
-        int i;
-
-        strncat(index->path, filename, (int) (abpath - filename) + 1);
-        for(i = 0; index->path[i] != 0; i++) {
-          if (index->path[i] == '\\') {
-            index->path[i] = '/';
-          }
-        }
-      }
-      /* -------------------- END OF COPY OF THE __New() CODE -------------------- */
-    }
+    index_base_path(cache->path, sizeof(cache->path), filename);
 
     /* Index/data filenames */
     if (pos != NULL) {
-      int nLen = (int) (pos - filename);
+      const size_t nLen = (size_t) (pos - filename);
 
-      strncat(cache->filenameDat, filename, nLen);
-      strncat(cache->filenameNdx, filename, nLen);
-      strcat(cache->filenameDat, ".dat");
-      strcat(cache->filenameNdx, ".ndx");
+      /* a base clipped to fit would name a different pair of files */
+      if (nLen > sizeof(cache->filenameDat) - sizeof(".dat")) {
+        return 0;
+      }
+      strlncatbuff(cache->filenameDat, filename, sizeof(cache->filenameDat),
+                   nLen);
+      strlncatbuff(cache->filenameNdx, filename, sizeof(cache->filenameNdx),
+                   nLen);
+      strcatbuff(cache->filenameDat, ".dat");
+      strcatbuff(cache->filenameNdx, ".ndx");
     }
     ndxSize = filesize(cache->filenameNdx);
     cache->timestamp = file_timestamp(cache->filenameDat);
@@ -1587,7 +1576,10 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
               }
               /* read position */
               a += binput(a, linepos, 200);
-              sscanf(linepos, "%d", &pos);
+              /* an unparseable field must not carry the previous entry's
+                 offset over, nor read the stack on the first one */
+              if (sscanf(linepos, "%d", &pos) != 1)
+                pos = 0;
 
               /* Add entry */
               coucal_add(cache->hash, line, pos);
@@ -2022,7 +2014,12 @@ static time_t getGMT(struct tm *tm) {   /* hey, time_t is local! */
     /* BSD does not have static "timezone" declared */
 #if (defined(BSD) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__NetBSD__) || defined(__FreeBSD_kernel__))
     time_t now = time(NULL);
-    time_t timezone = -localtime(&now)->tm_gmtoff;
+    struct tm nowtm;
+    time_t timezone;
+
+    if (!hts_localtime(now, &nowtm))
+      return (time_t) -1;
+    timezone = -nowtm.tm_gmtoff;
 #elif defined(_MSC_VER)
     /* MSVC spells it _timezone */
     const time_t timezone = _timezone;
@@ -2372,11 +2369,28 @@ typedef struct PT_SaveCache__Arc_t {
   char md5[32 + 2];
 } PT_SaveCache__Arc_t;
 
+/* Append src to an .arc header block of capacity size, clipping what does not
+   fit: the values come from a cache entry, so shortening beats dropping it.
+   HTS_FALSE when it had to clip. */
+static hts_boolean arc_headers_cat(char *headers, size_t size,
+                                   const char *src) {
+  const size_t used = strlen(headers);
+  const size_t left = used < size - 1 ? size - used - 1 : 0;
+
+  if (left != 0) {
+    strlncatbuff(headers, src, size, left);
+  }
+  return strlen(src) <= left ? HTS_TRUE : HTS_FALSE;
+}
+
 static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element) {
   PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
   FILE *const fp = st->fp;
   struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
   struct tm unknown_date;
+  /* the two strcatbuff calls closing the block rely on these 4 bytes */
+  const size_t room = sizeof(st->headers) - 4;
+  hts_boolean fit;
   int size_headers;
 
   /* a cached entry with no parseable Last-Modified must not take the writer
@@ -2388,34 +2402,42 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
     tm = &unknown_date;
   }
 
-  sprintf(st->headers,
-          "HTTP/1.0 %d %s"
-          "\r\n"
-          "X-Server: ProxyTrack " PROXYTRACK_VERSION "\r\n"
-          "Content-type: %s%s%s%s"
-          "\r\n"
-          "Last-modified: %s"
-          "\r\n"
-          "Content-length: %d"
-          "\r\n",
-          element->statuscode, element->msg,
-          /**/ hts_effective_mime(element->contenttype),
-          (element->charset[0] ? "; charset=\"" : ""),
-          (element->charset[0] ? element->charset : ""),
-          (element->charset[0] ? "\"" : ""), /**/ element->lastmodified,
-          (int) element->size);
+  fit = slprintfbuff(st->headers, room,
+                     "HTTP/1.0 %d %s"
+                     "\r\n"
+                     "X-Server: ProxyTrack " PROXYTRACK_VERSION "\r\n"
+                     "Content-type: %s%s%s%s"
+                     "\r\n"
+                     "Last-modified: %s"
+                     "\r\n"
+                     "Content-length: %d"
+                     "\r\n",
+                     element->statuscode, element->msg,
+                     /**/ hts_effective_mime(element->contenttype),
+                     (element->charset[0] ? "; charset=\"" : ""),
+                     (element->charset[0] ? element->charset : ""),
+                     (element->charset[0] ? "\"" : ""),
+                     /**/ element->lastmodified, (int) element->size);
   if (element->location != NULL && element->location[0] != '\0') {
-    sprintf(st->headers + strlen(st->headers), "Location: %s" "\r\n",
-            element->location);
-  }
-  if (element->headers != NULL) {
-    if (strlen(element->headers) <
-        sizeof(st->headers) - strlen(element->headers) - 1) {
-      strcat(st->headers, element->headers);
+    if (!arc_headers_cat(st->headers, room, "Location: ") ||
+        !arc_headers_cat(st->headers, room, element->location) ||
+        !arc_headers_cat(st->headers, room, "\r\n")) {
+      fit = HTS_FALSE;
     }
   }
-  strcat(st->headers, "\r\n");
+  if (element->headers != NULL &&
+      !arc_headers_cat(st->headers, room, element->headers)) {
+    fit = HTS_FALSE;
+  }
+  /* a clip landing mid-line must still end it, or the body reads as a header */
+  if (hts_lastchar(st->headers) != '\n') {
+    strcatbuff(st->headers, "\r\n");
+  }
+  strcatbuff(st->headers, "\r\n");
   size_headers = (int) strlen(st->headers);
+  if (!fit) {
+    fprintf(stderr, "Headers of %s clipped to %d bytes" LF, url, size_headers);
+  }
 
   /* doc == <nl><URL-record><nl><network_doc> */
 

src/punycode.c

@@ -425,10 +425,11 @@ int main(int argc, char **argv) {
       fail(io_error);
     if (feof(stdin))
       fail(invalid_input);
-    input_length = strlen(input) - 1;
-    if (input[input_length] != '\n')
+    /* a leading NUL byte leaves strlen() at 0, so trim before subtracting */
+    input_length = strlen(input);
+    if (input_length == 0 || input[input_length - 1] != '\n')
       fail(too_big);
-    input[input_length] = 0;
+    input[--input_length] = 0;
 
     for(p = input; *p != 0; ++p) {
       pp = strchr(print_ascii, *p);

tests/01_engine-lastchar.test

@@ -1,8 +1,8 @@
 #!/bin/bash
 #
-# Runs the lastchar self-test (#770), then keeps the idiom it replaced from
-# coming back: x[strlen(x) - 1] indexes one byte before the buffer when x is
-# empty, and the guard is never where the index is.
+# Runs the lastchar self-test (#770, #781, #821), then keeps the idiom it
+# replaced from coming back: x[strlen(x) - 1], x + strlen(x) - 1 and an unsigned
+# index seeded from it all land before the buffer when x is empty.
 
 set -eu
 
@@ -33,3 +33,52 @@ test -z "$hits" || {
     echo "$hits" >&2
     exit 1
 }
+
+# Same for the pointer spelling. The three survivors sit in commented-out
+# blocks in htsname.c; the rest are this class being named, plus the self-test's
+# own deliberate underflow.
+hits=$(command grep -rnE '\+ *\(?(int|size_t)?\)? *strlen *\([^)]*\) *- *1' \
+    "$top_srcdir/src" --include='*.c' --include='*.h' --exclude-dir=coucal |
+    grep -v 'htsname\.c:.*char\* a=\(fil+strlen(fil)\|save+strlen(save)\)-1;' |
+    grep -v 'htsselftest\.c:.*pointer spelling x + strlen(x) - 1' |
+    grep -v 'htsselftest\.c:.*\*(s + strlen(s) - 1) =' || true)
+
+test -z "$hits" || {
+    echo "x + strlen(x) - 1 reintroduced; use hts_lastcharptr:" >&2
+    echo "$hits" >&2
+    exit 1
+}
+
+# Same for the unsigned index spelling, whose (i > 0) guard is vacuous since
+# SIZE_MAX passes it. The second pattern catches a bare reassignment, which
+# carries no type to match; an (int) cast is the safe form, so it must be uncast.
+hits=$(command grep -rnE \
+    -e '(size_t|unsigned +(int|long|short)) +[A-Za-z_][A-Za-z0-9_]* *= *(\([^)]*\) *)?strlen *\([^;]*\) *- *1' \
+    -e '[A-Za-z_][A-Za-z0-9_]* *= *strlen *\([^;]*\) *- *1' \
+    "$top_srcdir/src" --include='*.c' --include='*.h' --exclude-dir=coucal |
+    grep -v 'htsselftest\.c:.*size_t vacuous = strlen' |
+    grep -v 'htswizard\.c:.*int i=strlen(adr)-1;' || true)
+
+test -z "$hits" || {
+    echo "an unsigned index from strlen(x) - 1 reintroduced; use hts_lastcharoffset:" >&2
+    echo "$hits" >&2
+    exit 1
+}
+
+# url_savename() has nine of these sites and reaches all of them with an empty
+# "fil", which "?x=1" produces. Names are asserted loosely, since the point is
+# the run: on unfixed code the sanitized CI leg aborts inside htsname.c.
+for args in "?x=1 text/plain" "?x=1 text/html" "?x=1 text/html type=4"; do
+    # shellcheck disable=SC2086
+    name=$(httrack -#test=savename $args) || {
+        echo "httrack -#test=savename $args exited non-zero: $name" >&2
+        exit 1
+    }
+    case "$name" in
+    "savename: "?*) ;;
+    *)
+        echo "-#test=savename $args produced no name: $name" >&2
+        exit 1
+        ;;
+    esac
+done

tests/01_engine-savename.test

@@ -155,13 +155,31 @@ full '/%2e%2e/%2e%2e/etc/passwd' 'text/html' '/dev/null/www.example.com///etc/pa
 full '/x.php' 'application/pdf' '/dev/null/www.example.com///evil.exe' 'cdispo=../../evil.exe'
 name $'/evil\rname\t.php' 'text/html' 'evil name .html'
 
-# #133: oversized names are capped only on Windows (MAX_PATH, chopping the
-# extension); elsewhere the platform PATH_MAX leaves a 300-char name whole.
+# #133: oversized names are capped only on Windows, where the name takes the
+# same 48-char clamp as a directory segment and keeps its extension across the
+# cut (#852); elsewhere the platform PATH_MAX leaves a 300-char name whole.
 case "$(uname -s 2>/dev/null)" in
 MINGW* | MSYS* | CYGWIN*)
-    name "/$(printf 'a%.0s' {1..300}).php" 'text/html' "$(printf 'a%.0s' {1..210})"
+    name "/$(printf 'a%.0s' {1..300}).php" 'text/html' "$(printf 'a%.0s' {1..43}).html"
     ;;
 *)
     name "/$(printf 'a%.0s' {1..300}).php" 'text/html' "$(printf 'a%.0s' {1..300}).html"
     ;;
 esac
+
+# #774: the engine owns hts-cache/ and hts-tmp/ inside the mirror, so a URL must
+# never be able to name one; a site serving them gets the /nul -> nul_ treatment.
+full '/hts-tmp/a.bin' 'application/octet-stream' \
+    '/dev/null/www.example.com/hts-tmp_/a.bin'
+full '/hts-cache/new.zip' 'application/octet-stream' \
+    '/dev/null/www.example.com/hts-cache_/new.zip'
+full '/d/hts-tmp/a.bin.bak' 'application/octet-stream' \
+    '/dev/null/www.example.com/d/hts-tmp_/a.bin.bak'
+# A whole component only, and case-insensitively: the filesystem may be too.
+full '/hts-tmpfoo/a.bin' 'application/octet-stream' \
+    '/dev/null/www.example.com/hts-tmpfoo/a.bin'
+full '/HTS-TMP/a.bin' 'application/octet-stream' \
+    '/dev/null/www.example.com/HTS-TMP_/a.bin'
+# Control: the DOS device names this reuses must still be escaped.
+full '/nul/x.bin' 'application/octet-stream' \
+    '/dev/null/www.example.com/nul_/x.bin'

tests/101_local-update-stale-bak.test

@@ -12,8 +12,8 @@ set -euo pipefail
 : "${top_srcdir:=..}"
 
 bash "$top_srcdir/tests/local-crawl.sh" \
-    --plant-file bigtrunc/slow.bin.bak \
-    --plant-dir bigtrunc/fast.bin.bak \
+    --plant-file bigtrunc/~hts-tmp/slow.bin.bak \
+    --plant-dir bigtrunc/~hts-tmp/fast.bin.bak \
     --rerun-args '--update -M400000' \
     --log-found 'More than 400000 bytes have been transferred.. giving up' \
     --log-found 'replacing leftover backup .*slow\.bin\.bak' \

tests/106_engine-repair-rename.test

@@ -0,0 +1,173 @@
+#!/bin/bash
+#
+# #786: both cache-repair paths moved the recovered zip onto the cache without
+# checking the move, and announced success either way. #824: they also committed
+# a recovery holding no entry at all. Neither may replace the cache, and both
+# must be reported.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+tmpdir=$(mktemp -d) # honors the Windows-shaped TMPDIR the MSYS suite exports
+cleanup() { rm -rf "$tmpdir"; }
+trap 'set +e; cleanup' EXIT
+trap 'set +e; cleanup; exit 1' HUP INT QUIT PIPE TERM
+
+# One local file header, no central directory: unzOpen fails, unzRepair
+# recovers the single entry. Same vector as -#test=zip-repair-shift.
+plant_damaged_cache() {
+    rm -rf "$1"
+    mkdir -p "$1/hts-cache"
+    printf '\120\113\003\004\024\000\000\000\000\000\000\000\000\000\000\000\350\212\000\000\000\000\000\000\000\000\001\000\000\000\141' \
+        >"$1/hts-cache/new.zip"
+}
+
+dead=http://127.0.0.1:1/
+
+# --- the CLI path (-#R), repair succeeding -----------------------------------
+printf '[-#R repairs in place] ..\t'
+proj="${tmpdir}/cli"
+plant_damaged_cache "$proj"
+out=$(httrack -O "$proj" -#R "$dead" 2>&1)
+grep -q 'successfully recovered' <<<"$out" || {
+    echo "FAIL: no recovery reported: $out"
+    exit 1
+}
+test ! -e "${proj}/hts-cache/repair.zip" || {
+    echo "FAIL: the recovery was left in repair.zip"
+    exit 1
+}
+test "$(wc -c <"${proj}/hts-cache/new.zip")" -gt 31 || {
+    echo "FAIL: new.zip is still the damaged one"
+    exit 1
+}
+echo "OK"
+
+# --- #824: nothing was recoverable, so nothing may be committed --------------
+printf '[-#R refuses an empty recovery] ..\t'
+proj="${tmpdir}/empty"
+rm -rf "$proj"
+mkdir -p "$proj/hts-cache"
+printf 'not a zip at all, some bytes' >"$proj/hts-cache/new.zip"
+rc=0
+out=$(httrack -O "$proj" -#R "$dead" 2>&1) || rc=$?
+test "$rc" -ne 0 || {
+    echo "FAIL: an empty recovery exited 0: $out"
+    exit 1
+}
+if grep -q 'successfully recovered' <<<"$out"; then
+    echo "FAIL: an empty recovery claimed success: $out"
+    exit 1
+fi
+test "$(wc -c <"${proj}/hts-cache/new.zip")" -eq 28 || {
+    echo "FAIL: the damaged cache was replaced by the empty recovery"
+    exit 1
+}
+echo "OK"
+
+if [ "$(uname -s)" != "Linux" ]; then
+    echo "repair-rename: LD_PRELOAD interposition is Linux-only here, skipping"
+    exit 0
+fi
+# A --disable-shared build has nothing to preload; anything else missing is a
+# build problem, not a skip, or the failure legs would pass vacuously.
+if [ ! -r "${RENAMEFAIL_LA:-}" ]; then
+    echo "repair-rename: ${RENAMEFAIL_LA:-\$RENAMEFAIL_LA} was not built" >&2
+    exit 1
+fi
+if grep -q "^dlname=''" "$RENAMEFAIL_LA"; then
+    echo "repair-rename: static-only build, skipping the interposed legs"
+    exit 0
+fi
+if [ ! -r "${RENAMEFAIL_LIB:-}" ]; then
+    echo "repair-rename: ${RENAMEFAIL_LIB:-\$RENAMEFAIL_LIB} was not built" >&2
+    exit 1
+fi
+
+# An LD_PRELOAD library loads ahead of the executable's own libasan, which ASan
+# refuses by default. The shim allocates nothing, so the ordering is harmless.
+export ASAN_OPTIONS="${ASAN_OPTIONS:+$ASAN_OPTIONS:}verify_asan_link_order=0"
+export RENAMEFAIL_MODE=repair
+export LD_PRELOAD="$RENAMEFAIL_LIB"
+
+# --- the CLI path, move refused ----------------------------------------------
+printf '[-#R reports a refused move] ..\t'
+proj="${tmpdir}/cli-fail"
+plant_damaged_cache "$proj"
+rc=0
+out=$(httrack -O "$proj" -#R "$dead" 2>&1) || rc=$?
+test "$rc" -ne 0 || {
+    echo "FAIL: a refused move exited 0: $out"
+    exit 1
+}
+if grep -q 'successfully recovered' <<<"$out"; then
+    echo "FAIL: a refused move still claimed success: $out"
+    exit 1
+fi
+test -s "${proj}/hts-cache/repair.zip" || {
+    echo "FAIL: the recovery was not kept for a retry"
+    exit 1
+}
+test "$(wc -c <"${proj}/hts-cache/new.zip")" -eq 31 || {
+    echo "FAIL: the damaged cache was destroyed anyway"
+    exit 1
+}
+echo "OK"
+
+# --- the engine path (cache_init), move refused ------------------------------
+printf '[cache_init reports a refused move] ..\t'
+proj="${tmpdir}/engine"
+plant_damaged_cache "$proj"
+httrack -O "$proj" "$dead" --timeout=2 --retries=0 -q >/dev/null 2>&1 || true
+log="${proj}/hts-log.txt"
+test -r "$log" || {
+    echo "FAIL: no crawl log"
+    exit 1
+}
+grep -q 'damaged cache' "$log" || {
+    echo "FAIL: the repair path never ran"
+    exit 1
+}
+grep -q 'could not put the repaired cache in place' "$log" || {
+    echo "FAIL: a refused move was not reported: $(grep -i cache "$log")"
+    exit 1
+}
+if grep -q 'successfully recovered' "$log"; then
+    echo "FAIL: a refused move still claimed success"
+    exit 1
+fi
+test -s "${proj}/hts-cache/repair.zip" || {
+    echo "FAIL: the recovery was not kept for a retry"
+    exit 1
+}
+echo "OK"
+
+# --- the move is refused the way Windows refuses it --------------------------
+# The destination always exists there, so every repair takes hts_rename_over()'s
+# fallback. Before #790 that fallback removed the cache and then failed.
+printf '[a refused fallback keeps the cache] ..\t'
+proj="${tmpdir}/eexist"
+plant_damaged_cache "$proj"
+rc=0
+out=$(RENAMEFAIL_MODE=repair-eexist httrack -O "$proj" -#R "$dead" 2>&1) || rc=$?
+test "$rc" -ne 0 || {
+    echo "FAIL: a refused fallback exited 0: $out"
+    exit 1
+}
+test "$(wc -c <"${proj}/hts-cache/new.zip")" -eq 31 || {
+    echo "FAIL: the fallback destroyed the cache"
+    exit 1
+}
+test -s "${proj}/hts-cache/repair.zip" || {
+    echo "FAIL: the recovery was not kept for a retry"
+    exit 1
+}
+leftover=$(find "${proj}/hts-cache" -name '*.hts-old*' | head -1)
+test -z "$leftover" || {
+    echo "FAIL: the parked copy was left behind: $leftover"
+    exit 1
+}
+echo "OK"

tests/107_local-bak-collision.test

@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# #774: the site serves hts-tmp/, the reserved segment url_savename must escape
+# so a mirrored file cannot sit where the engine writes. Asserting the unescaped
+# path would pass with the bug live, which is how a first attempt missed it.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" \
+    --rerun-args '--update' \
+    --file-matches 'bakname/a.bin' 'BAKNAME-MAIN-V2' \
+    --file-matches 'bakname/hts-tmp_/a.bin.bak' 'BAKNAME-SIBLING' \
+    --file-min-bytes 'bakname/hts-tmp_/a.bin.bak' 2048 \
+    --not-found 'bakname/hts-tmp/a.bin.bak' \
+    --log-not-found 'could not back up' \
+    --log-not-found 'could not restore' \
+    httrack 'BASEURL/bakname/index.html'

tests/108_engine-refetch-backup.test

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Drives -#test=refetchbackup: the #77 re-fetch backup must leave a copy of the
+# resource whatever happens, and must not build its temporary name inside the
+# mirror namespace (#774, #775).
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+trap 'set +e; rm -rf "$dir"' EXIT
+trap 'set +e; rm -rf "$dir"; exit 1' HUP INT QUIT PIPE TERM
+
+rc=0
+out=$(httrack -O /dev/null -#test=refetchbackup "$dir" 2>&1) || rc=$?
+if test "$rc" -ne 0 || ! grep -q "refetchbackup: OK" <<<"$out"; then
+    echo "FAIL (exit $rc): $out" >&2
+    exit 1
+fi
+
+# The temporaries live in ~hts-tmp and the last user removes it.
+leftovers=$(find "$dir" -mindepth 1 | head -5)
+test -z "$leftovers" || {
+    echo "FAIL: leftover temporaries: $leftovers" >&2
+    exit 1
+}

tests/112_local-single-file-fragment.test

@@ -0,0 +1,108 @@
+#!/bin/bash
+# Issue #766: --single-file substituted the whole reference, dropping the '#'
+# fragment an SVG sprite selector needs to pick one icon out of the sheet.
+set -e
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+server=$(nativepath "${testdir}/local-server.py")
+
+python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_766.XXXXXX") || exit 1
+serverpid=
+cleanup() {
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+}
+trap 'set +e; cleanup' EXIT
+trap cleanup HUP INT QUIT PIPE TERM
+
+fail() {
+    echo "FAIL: $1"
+    exit 1
+}
+
+# --- the site ---------------------------------------------------------------
+doc="${tmpdir}/doc"
+mkdir -p "$doc"
+cat >"${doc}/index.html" <<'EOF'
+<html><head><link rel="stylesheet" href="style.css"></head><body>
+<img src="sprite.svg#icon-foo" alt="f">
+<img srcset="sprite.svg#icon-set 2x" alt="s">
+</body></html>
+EOF
+cat >"${doc}/style.css" <<'EOF'
+body { background: url(sprite.svg#icon-css); }
+div { background: url(big.svg#icon-big); }
+EOF
+printf '<svg xmlns="http://www.w3.org/2000/svg"><g id="icon-foo"/></svg>\n' \
+    >"${doc}/sprite.svg"
+{
+    printf '<svg xmlns="http://www.w3.org/2000/svg"><!--'
+    head -c 20000 /dev/zero | tr '\0' 'x'
+    printf '%s\n' '--><g id="icon-big"/></svg>'
+} >"${doc}/big.svg"
+
+# --- server -----------------------------------------------------------------
+serverlog="${tmpdir}/server.log"
+: >"$serverlog"
+"$python" "$server" --root "$(nativepath "$doc")" >"$serverlog" 2>&1 &
+serverpid=$!
+port=
+for _ in $(seq 1 50); do
+    line=$(head -n1 "$serverlog" 2>/dev/null)
+    if test "${line%% *}" == "PORT"; then
+        port="${line#PORT }"
+        break
+    fi
+    kill -0 "$serverpid" 2>/dev/null || fail "server exited: $(cat "$serverlog")"
+    sleep 0.1
+done
+test -n "$port" || fail "could not discover server port"
+base="http://127.0.0.1:${port}"
+
+which httrack >/dev/null || fail "could not find httrack"
+
+# --- crawl ------------------------------------------------------------------
+# The cap sits between sprite.svg and big.svg, so one inlines and one is left
+# for the rebase path.
+out="${tmpdir}/crawl"
+mkdir "$out"
+httrack -O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 \
+    --retries=0 --single-file --single-file-max-size 4096 \
+    "${base}/index.html" >"${tmpdir}/log" 2>&1
+
+root="${out}/127.0.0.1_${port}"
+page="${root}/index.html"
+test -f "$page" || fail "no mirrored page at $page"
+test -f "${root}/big.svg" || fail "the over-cap asset never reached the mirror"
+
+b64='[A-Za-z0-9+/=]+'
+
+printf '[fragment survives onto the data: URI] ..\t'
+# Matching the data: prefix as well is what makes this more than "#icon-foo is
+# somewhere in the page": an un-inlined reference would carry it too.
+grep -Eq "src=\"data:image/svg\\+xml;base64,${b64}#icon-foo\"" "$page" ||
+    fail "img src lost its fragment: $(grep -o 'src="[^"]*"' "$page" | head -n2)"
+grep -Eq "srcset=\"data:image/svg\\+xml;base64,${b64}#icon-set 2x\"" "$page" ||
+    fail "srcset lost its fragment or its descriptor"
+echo OK
+
+printf '[fragment survives inside an inlined stylesheet] ..\t'
+"$python" - "$page" "${tmpdir}/got.css" <<'PY' || fail "no inlined stylesheet"
+import base64, re, sys
+html = open(sys.argv[1], "rb").read().decode("utf-8", "replace")
+m = re.search(r'data:text/css;base64,([A-Za-z0-9+/=]*)', html)
+if m is None:
+    sys.exit(1)
+open(sys.argv[2], "wb").write(base64.b64decode(m.group(1)))
+PY
+grep -Eq "url\\(data:image/svg\\+xml;base64,${b64}#icon-css\\)" "${tmpdir}/got.css" ||
+    fail "url() lost its fragment: $(cat "${tmpdir}/got.css")"
+# Over the cap, so it stays a link, rebased onto the page's directory: it still
+# names the sprite sheet, so it still needs the selector.
+grep -q 'url(big.svg#icon-big)' "${tmpdir}/got.css" ||
+    fail "the rebased over-cap url() lost its fragment"
+echo OK

tests/113_engine-threadattr-leak.test

@@ -0,0 +1,62 @@
+#!/bin/bash
+#
+# Issue #772: hts_newthread() destroyed the pthread attributes object only when
+# the thread started, so a failed pthread_create() left it initialised forever.
+# glibc allocates nothing for a default attr, so no leak checker sees it: the
+# interposer counts the init/destroy imbalance instead.
+
+set -euo pipefail
+
+if [ "$(uname -s)" != "Linux" ]; then
+    echo "threadattr: LD_PRELOAD interposition is Linux-only here, skipping"
+    exit 0
+fi
+# A --disable-shared build has nothing to preload; anything else missing is a
+# build problem, not a skip, or the leg would pass vacuously.
+if [ ! -r "${THREADATTRFAIL_LA:-}" ]; then
+    echo "threadattr: ${THREADATTRFAIL_LA:-\$THREADATTRFAIL_LA} was not built" >&2
+    exit 1
+fi
+if grep -q "^dlname=''" "$THREADATTRFAIL_LA"; then
+    echo "threadattr: static-only build, skipping"
+    exit 0
+fi
+if [ ! -r "${THREADATTRFAIL_LIB:-}" ]; then
+    echo "threadattr: ${THREADATTRFAIL_LIB:-\$THREADATTRFAIL_LIB} was not built" >&2
+    exit 1
+fi
+
+dir=$(mktemp -d)
+trap 'set +e; rm -rf "$dir"' EXIT
+trap 'rm -rf "$dir"' HUP INT QUIT PIPE TERM
+
+# An LD_PRELOAD library loads ahead of the executable's own libasan, which ASan
+# refuses by default. The shim allocates nothing, so the ordering is harmless.
+export ASAN_OPTIONS="${ASAN_OPTIONS:+$ASAN_OPTIONS:}verify_asan_link_order=0"
+
+# The self-test spawns threads and gives up on the first refusal; it is the
+# driver here, so its failure is expected and not the verdict.
+log="$dir/counts"
+THREADATTRFAIL=1 THREADATTRFAIL_LOG="$log" LD_PRELOAD="$THREADATTRFAIL_LIB" \
+    httrack -O "$dir/o" -#test=threadwait >"$dir/out" 2>&1 || true
+
+test -r "$log" || {
+    echo "threadattr: the interposer wrote no counts" >&2
+    cat "$dir/out" >&2
+    exit 1
+}
+sabotaged=$(sed -n 's/^sabotaged //p' "$log")
+undestroyed=$(sed -n 's/^undestroyed //p' "$log")
+
+# Teeth: with no refused create there is nothing to leak, and the check below
+# would pass on any build.
+test "${sabotaged:-0}" -ge 1 || {
+    echo "threadattr: no pthread_create() was refused, nothing was tested" >&2
+    cat "$dir/out" >&2
+    exit 1
+}
+test "${undestroyed:-1}" -eq 0 || {
+    echo "threadattr: $undestroyed of $sabotaged attributes objects survived a refused pthread_create()" >&2
+    exit 1
+}
+echo "OK: a refused pthread_create() destroys the attributes object ($sabotaged refused)"

tests/116_engine-rtrim.test

@@ -0,0 +1,107 @@
+#!/bin/bash
+#
+# Backward scans that start at s + strlen(s) - 1 and decrement with no lower
+# bound: the config-file right trim and the collision-rename digit scan both
+# walked off the front of a stack buffer. Neither ASan nor _FORTIFY_SOURCE sees
+# the self-test's overrun (it lands in the same frame), hence the poisoned byte.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+
+# Resolve httrack before any cd: PATH may carry a build-relative entry.
+bin=$(command -v httrack) || {
+    echo "FAIL: httrack not found on PATH" >&2
+    exit 1
+}
+case "$bin" in
+/*) ;;
+*) bin="$(cd "$(dirname "$bin")" && pwd)/$(basename "$bin")" ;;
+esac
+
+tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_rtrim.XXXXXX") || exit 1
+trap 'set +e; rm -rf "$tmp"' EXIT
+trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
+
+fail() {
+    echo "FAIL: $1" >&2
+    exit 1
+}
+
+# --- 1. the helper itself ---------------------------------------------------
+out=$("$bin" -#test=rtrim) || fail "httrack -#test=rtrim exited non-zero: $out"
+test "$out" = "rtrim self-test: OK" || fail "expected 'rtrim self-test: OK', got: $out"
+
+# --- 2. collision rename over an all-digit name -----------------------------
+# Two pages with the same all-digit name save to 2024.html, so the second goes
+# through the -<n> rename and the digit scan has nothing but digits to walk
+# over. Driven twice, because the two routes differ in who picks the URLs.
+site="$tmp/site"
+mkdir -p "$site/a" "$site/b"
+printf '%s\n' '<html><body><a href="a/2024.html">a</a> <a href="b/2024.html">b</a></body></html>' \
+    >"$site/index.html"
+echo '<html><body>A</body></html>' >"$site/a/2024.html"
+echo '<html><body>B</body></html>' >"$site/b/2024.html"
+
+# -g flattens the tree, and pins depth to 0: the operator supplies both URLs.
+out1="$tmp/out1"
+mkdir -p "$out1"
+(cd "$out1" && "$bin" -g "file://$site/a/2024.html" "file://$site/b/2024.html" \
+    --quiet -n >.log 2>&1) || fail "collision crawl exited non-zero: $(tail -3 "$out1/.log")"
+
+test -f "$out1/2024.html" || fail "2024.html not mirrored"
+test -f "$out1/2024-2.html" || fail "colliding name was not renamed to 2024-2.html"
+
+# -N "%n.%t" collides the same way with depth unrestricted, so one starting URL
+# is enough and the crawled page picks both colliding names itself.
+out3="$tmp/out3"
+mkdir -p "$out3"
+(cd "$out3" && "$bin" "file://$site/index.html" -N "%n.%t" --quiet -n >.log 2>&1) ||
+    fail "userdef-savename crawl exited non-zero: $(tail -3 "$out3/.log")"
+test -f "$out3/2024.html" || fail "2024.html not mirrored under -N '%n.%t'"
+test -f "$out3/2024-2.html" ||
+    fail "colliding name was not renamed under -N '%n.%t' (links followed, not argv)"
+
+# The rename must keep counting, not restart: a third collision is -3.
+mkdir -p "$site/c"
+echo '<html><body>C</body></html>' >"$site/c/2024.html"
+out2="$tmp/out2"
+mkdir -p "$out2"
+(cd "$out2" && "$bin" -g "file://$site/a/2024.html" "file://$site/b/2024.html" \
+    "file://$site/c/2024.html" --quiet -n >.log 2>&1) ||
+    fail "three-way collision crawl exited non-zero: $(tail -3 "$out2/.log")"
+test -f "$out2/2024-3.html" || fail "third colliding name was not renamed to 2024-3.html"
+
+# --- 3. a blank config line is skipped, not parsed ---------------------------
+# HTS_HTTRACKRC is ".httrackrc" on POSIX and "httrackrc" on Windows.
+rcdir="$tmp/rc"
+mkdir -p "$rcdir"
+echo '<html><body>hi</body></html>' >"$rcdir/index.html"
+printf '   \nzzz-not-an-option\n' >"$rcdir/.httrackrc"
+cp "$rcdir/.httrackrc" "$rcdir/httrackrc"
+
+rc=0
+log=$( (cd "$rcdir" && "$bin" "file://$rcdir/index.html" --quiet -n 2>&1)) || rc=$?
+test "$rc" -eq 0 || fail "rc-file crawl exited $rc: $log"
+log=${log//$'\r'/} # the empty-option check below must not miss a CRLF line
+
+# The line after the blank one must still be read, or this proves nothing.
+case "$log" in
+*"Unknown option: zzz-not-an-option"*) ;;
+*) fail "config file was not parsed past the blank line: $log" ;;
+esac
+# The blank line itself must not become an empty option.
+case "$log" in
+*"Unknown option: "$'\n'*) fail "blank config line was parsed as an empty option: $log" ;;
+esac
+
+# --- 4. the call sites stay bounded -----------------------------------------
+# Sections 2 and 3 only go red under a sanitizer: on a plain build the byte
+# below the buffer is usually not a digit, so a reverted scan stops anyway and
+# produces the same names. Pin the two sites at the source instead.
+command grep -q 'hts_rtrim(line, HTS_REALSPACES)' "$top_srcdir/src/htsalias.c" ||
+    fail "htsalias.c right trim no longer goes through hts_rtrim"
+command grep -q 'hts_rtrimlen(tempo, "0123456789")' "$top_srcdir/src/htsname.c" ||
+    fail "htsname.c collision-suffix scan no longer goes through hts_rtrimlen"
+
+exit 0

tests/117_local-proxytrack-ndx-cursor.test

@@ -0,0 +1,107 @@
+#!/bin/bash
+# Issue #793: binput() stepped its caller's cursor over a byte it never
+# consumed, so any .ndx whose walk reaches the buffer's terminating NUL drifts
+# past it. The sanitizer CI build turns that read into a heap-buffer-overflow.
+
+set -euo pipefail
+
+dir=$(mktemp -d)
+cleanup() { rm -rf "$dir"; }
+trap 'set +e; cleanup' EXIT
+trap 'cleanup' HUP INT QUIT PIPE TERM
+
+fail() {
+    echo "FAIL: $1" >&2
+    exit 1
+}
+
+pad() { printf "%${1}s" '' | tr ' ' "$2"; }
+
+hdr() { printf 'CACHE-1.4\nWed, 01 Jan 2025 00:00:00 GMT\n'; }
+
+# Number of cache keys proxytrack built from index $1.
+items() {
+    local out n
+    : >"${1%.ndx}.dat" # the reader only proceeds when the sibling .dat opens
+    out=$(proxytrack --convert "$dir/out.arc" "$1" 2>&1) ||
+        fail "proxytrack died on $(basename "$1")"
+    n=$(sed -n 's/.*(\([0-9][0-9]*\) items added).*/\1/p' <<<"$out")
+    if [ -z "$n" ]; then
+        grep -q '(no items added)' <<<"$out" ||
+            fail "proxytrack reported no item count for $(basename "$1")"
+        n=0
+    fi
+    echo "$n"
+}
+
+# The URL is read in two halves into one buffer, bounded at 1024 then at what
+# the first half left. Two URLs differing only on that first bound must stay
+# two distinct keys; eating the byte on it collapses them into one.
+for at in 1023 1024; do
+    {
+        hdr
+        printf '%sA\n\n10\n' "$(pad "$at" h)"
+        printf '%sB\n\n20\n' "$(pad "$at" h)"
+    } >"$dir/first-$at.ndx"
+done
+got=$(items "$dir/first-1023.ndx")
+test "$got" = 2 || fail "control: distinct URLs gave $got keys, expected 2"
+got=$(items "$dir/first-1024.ndx")
+test "$got" = 2 || fail "URLs differing on the 1024 bound gave $got keys, expected 2"
+
+# The header field is read through the same helper, bounded at 256. One byte
+# past that bound must parse like any other over-long header, not lose an entry.
+for at in 257 258; do
+    {
+        pad "$at" C
+        printf '\na/1\nx\n1\nb/2\ny\n2\nc/3\nz\n3\n'
+    } >"$dir/header-$at.ndx"
+done
+got=$(items "$dir/header-258.ndx")
+test "$got" = 3 || fail "header 2 past its bound gave $got keys, expected 3"
+got=$(items "$dir/header-257.ndx")
+test "$got" = 3 || fail "header 1 past its bound gave $got keys, expected 3"
+
+# Second half, bounded at 2048 - 10 - 1: it fills that bound with the index
+# ending right there, so the byte stepped over is the terminating NUL.
+{
+    hdr
+    printf '%s\n' "$(pad 10 h)"
+    pad 2037 f
+} >"$dir/second.ndx"
+got=$(items "$dir/second.ndx")
+test "$got" = 1 || fail "second half at its bound gave $got keys, expected 1"
+
+# Third field, bounded at 200, likewise ending the index.
+{
+    hdr
+    printf 'h/1\nx\n'
+    pad 200 9
+} >"$dir/position.ndx"
+got=$(items "$dir/position.ndx")
+test "$got" = 1 || fail "position field at its bound gave $got keys, expected 1"
+
+# Nothing has to fill a bound: a field of any length ending the index without a
+# newline stops on the same terminating NUL.
+{
+    printf '0\nSKIP\n'
+    pad 1023 h
+} >"$dir/short.ndx"
+got=$(items "$dir/short.ndx")
+test "$got" = 1 || fail "index ending mid-field gave $got keys, expected 1"
+
+# The reported reproducer: both halves full, and the walk reaches the last line.
+{
+    hdr
+    printf '%s\n%s\n0\n' "$(pad 1024 h)" "$(pad 1024 f)"
+} >"$dir/tail.ndx"
+got=$(items "$dir/tail.ndx")
+test "$got" = 1 || fail "index ending on a full-length entry gave $got keys, expected 1"
+
+# A NUL at offset 0 stops the header field with nothing behind it to inspect.
+# Not named nul.ndx: Windows resolves NUL.anything to the null device.
+printf '\000' >"$dir/zerobyte.ndx"
+got=$(items "$dir/zerobyte.ndx")
+test "$got" = 0 || fail "NUL-only index gave $got keys, expected 0"
+
+echo "OK: an .ndx field ending on its bound or on the buffer end keeps the walk in place"

tests/118_local-proxytrack-arcwrite.test

@@ -0,0 +1,123 @@
+#!/bin/bash
+# The .arc writer builds each entry's headers in a fixed 8192-byte block, and
+# its one bound ignored what the block already held: a long cached header
+# string wrapped the subtraction and smashed it.
+
+set -euo pipefail
+export LC_ALL=C
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+dir=$(mktemp -d)
+cleanup() { rm -rf "$dir"; }
+trap 'set +e; cleanup' EXIT
+trap 'set +e; cleanup; exit 1' HUP INT TERM
+
+# sizeof(PT_SaveCache__Arc_t.headers)
+HDRMAX=8192
+BODY=BODYMARKER
+
+fail() {
+    echo "FAIL: $1" >&2
+    exit 1
+}
+
+pad() { printf "%${1}s" '' | tr ' ' "$2"; }
+
+P1000=$(pad 1000 P)
+
+# Header lines contributing exactly $1 bytes to the entry's stored headers: the
+# reader rebuilds each as "name: value\r\n", and its line buffer holds 2048.
+gen_headers() {
+    local want=$1 c
+    while [ "$want" -gt 0 ]; do
+        c=$want
+        if [ "$c" -gt 1008 ]; then
+            c=1008
+            test "$((want - c))" -ge 9 || c=$((c - 9))
+        fi
+        printf 'X-P0: %s\r\n' "${P1000:0:$((c - 8))}"
+        want=$((want - c))
+    done
+}
+
+# $1 reason phrase, $2 Location value, $3 bytes of passthrough headers.
+build_arc() {
+    {
+        printf 'HTTP/1.1 200 %s\r\n' "$1"
+        printf 'Content-Type: text/html\r\n'
+        printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
+        printf 'Content-Length: %d\r\n' "${#BODY}"
+        test -z "$2" || printf 'Location: %s\r\n' "$2"
+        test "$3" -eq 0 || gen_headers "$3"
+        printf '\r\n'
+    } >"$dir/hdr"
+    printf '%s' "$BODY" >"$dir/body"
+    local alen
+    alen=$(($(wc -c <"$dir/hdr") + ${#BODY}))
+    {
+        printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+        printf '2 0 test\n'
+        printf '\n\n'
+        printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+        cat "$dir/hdr" "$dir/body"
+    } >"$dir/in.arc"
+}
+
+# Convert $dir/in.arc and check the emitted record, $1 naming the case. Sets
+# last_alen to the record's declared length.
+convert_check() {
+    local rec off alen total dataoff
+    proxytrack --convert "$dir/out.arc" "$dir/in.arc" >"$dir/log" 2>&1 ||
+        fail "$1: proxytrack died writing the .arc"
+    rec=$(grep -a -m1 '^http://example.com/p.html ' "$dir/out.arc") ||
+        fail "$1: entry dropped from the .arc"
+    off=$(grep -abo -m1 '^http://example.com/p.html ' "$dir/out.arc" | cut -d: -f1)
+    alen=${rec##* }
+    total=$(wc -c <"$dir/out.arc")
+    dataoff=$((off + ${#rec} + 1))
+    test "$((total - dataoff))" -eq "$alen" ||
+        fail "$1: record declares $alen bytes, wrote $((total - dataoff))"
+    test "$((alen - ${#BODY}))" -lt "$HDRMAX" ||
+        fail "$1: header block is $((alen - ${#BODY})) bytes, cap is $((HDRMAX - 1))"
+    # a clip landing mid-line must still close it, or the body reads as headers
+    tail -c $((${#BODY} + 4)) "$dir/out.arc" >"$dir/got.tail"
+    printf '\r\n\r\n%s' "$BODY" >"$dir/want.tail"
+    cmp -s "$dir/got.tail" "$dir/want.tail" ||
+        fail "$1: header block does not end on a blank line"
+    last_alen=$alen
+}
+
+# Control: nothing overshoots, so a broken probe cannot pass as a clean bound.
+last_alen=0
+build_arc OK "" 0
+convert_check "short entry"
+# what the writer adds on its own, so the sweep below can bracket the cap
+fixed=$((last_alen - ${#BODY}))
+
+# The reported bug: 12000 bytes wrap "sizeof - strlen - 1", so the guard passes.
+build_arc OK "" 12000
+convert_check "over-long cached headers"
+grep -q 'clipped to' "$dir/log" ||
+    fail "a clipped header block was not reported on stderr"
+
+# Three sources at once, each with its own limit: the reason phrase is cut at
+# msg[1024], the shorter location not at all, the passthrough takes what is
+# left. One all-fields bound gets at least one of them wrong.
+build_arc "$(pad 1200 M)" "http://example.com/$(pad 1500 L)" 5000
+convert_check "long reason, location and headers"
+got=$(runlen "$dir/out.arc" M)
+test "${got:-0}" = 1023 || fail "reason phrase kept ${got:-0} bytes, expected 1023"
+got=$(runlen "$dir/out.arc" L)
+test "${got:-0}" = 1500 || fail "location kept ${got:-0} bytes, expected 1500"
+
+# An off-by-one in the room held back for the closing blank line shows up only
+# on the exact boundary, so walk the block across its cap byte by byte.
+for n in $(seq $((HDRMAX - fixed - 32)) $((HDRMAX - fixed + 32))); do
+    build_arc OK "" "$n"
+    convert_check "passthrough of $n bytes"
+done
+
+echo "OK: the .arc writer clips its header block instead of overflowing it"

tests/119_local-proxytrack-ndx-fields2.test

@@ -0,0 +1,155 @@
+#!/bin/bash
+# Two bounds in the .ndx/.dat reader: a position field that did not parse left
+# "pos" holding the previous entry's offset, or the stack on the first entry,
+# and the base path and index filenames were copied with strncat calls bounded
+# by the source path instead of by their 1024-byte destinations.
+
+set -euo pipefail
+export LC_ALL=C
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+dir=$(mktemp -d)
+cleanup() { rm -rf "$dir"; }
+trap 'set +e; cleanup' EXIT
+trap 'set +e; cleanup; exit 1' HUP INT TERM
+
+fail() {
+    echo "FAIL: $1" >&2
+    exit 1
+}
+
+enc() { printf '%d\n%s' "${#1}" "$1"; }
+
+# One version-1.4 .dat record in cache_rstr() framing, field by field:
+# $1 content-type, $2 status message, $3 body.
+record() {
+    enc 200
+    enc "${#3}"
+    enc "$2"
+    enc "$1"
+    enc ""
+    enc ""
+    enc ""
+    enc ""
+    enc ""
+    enc ""
+    enc ""
+    enc ""
+    enc HTS
+    enc "${#3}"
+    printf '%s' "$3"
+}
+
+# Two records in $1.dat, and in $1.ndx two entries with positions $2 and $3.
+# The parser walks the index a line at a time and skips one line per entry, so
+# each block leads with a filler line.
+build_cache() {
+    record audio/midi ZERO ZEROBODY >"$dir/r0"
+    record image/gif ALPHA ALPHABODY >"$dir/ra"
+    cat "$dir/r0" "$dir/ra" >"$1.dat"
+    {
+        enc CACHE-1.4
+        enc "Wed, 01 Jan 2025 00:00:00 GMT"
+        printf 'X\nexample.com/a.html\n\n%s\n' "$2"
+        printf 'X\nexample.com/b.html\n\n%s\n' "$3"
+    } >"$1.ndx"
+}
+
+# Content-type proxytrack wrote for URL $2 of the .arc $1, or "none".
+mime_of() {
+    local rec
+    rec=$(grep -a -m1 "^http://$2 " "$1") || {
+        echo none
+        return
+    }
+    awk '{ print $4 }' <<<"$rec"
+}
+
+# Offset of the second record: the position an entry must not inherit.
+record audio/midi ZERO ZEROBODY >"$dir/r0"
+posA=$(wc -c <"$dir/r0")
+
+convert() {
+    proxytrack --convert "$dir/out.arc" "$1" >"$dir/log" 2>&1 ||
+        fail "$2: proxytrack died on the cache"
+}
+
+# Control: both positions parse, so each URL resolves to its own record and a
+# broken probe cannot pass as a fixed reader.
+build_cache "$dir/ctl" 0 "$posA"
+convert "$dir/ctl.ndx" "control"
+test "$(mime_of "$dir/out.arc" example.com/a.html)" = audio/midi ||
+    fail "control: a.html did not resolve to the record at offset 0"
+test "$(mime_of "$dir/out.arc" example.com/b.html)" = image/gif ||
+    fail "control: b.html did not resolve to the record at offset $posA"
+
+# A field that does not parse must read as offset 0, not as the offset the
+# entry before it left behind.
+build_cache "$dir/carry" "$posA" Z
+convert "$dir/carry.ndx" "carry-over"
+test "$(mime_of "$dir/out.arc" example.com/b.html)" = audio/midi ||
+    fail "an unparseable position inherited the previous entry's offset"
+
+# Same on the first entry, where there is no previous value and the read came
+# off the stack.
+build_cache "$dir/first" Z "$posA"
+convert "$dir/first.ndx" "first entry"
+test "$(mime_of "$dir/out.arc" example.com/a.html)" = audio/midi ||
+    fail "an unparseable position on the first entry did not read as 0"
+
+# The .ndx name reaches the copy before anything opens it, so no file needed.
+long=$(printf '%3000s' '' | tr ' ' 'x')
+convert "$dir/$long.ndx" "over-long .ndx name"
+
+if is_windows; then
+    echo "deep paths need a POSIX filesystem; index name half only" >&2
+    exit 0
+fi
+
+comp=$(printf '%200s' '' | tr ' ' 'd')
+
+# A path just short of the 1024-byte destinations must still load: refusing
+# these is as wrong as overflowing them.
+fits="$dir/$comp/$comp/$comp"
+mkdir -p "$fits"
+build_cache "$fits/c" 0 "$posA"
+convert "$fits/c.ndx" "path that fits"
+test "$(mime_of "$dir/out.arc" example.com/b.html)" = image/gif ||
+    fail "a cache at a $((${#fits} + 6))-byte path was dropped"
+
+# Past them, the base path and both index filenames each overshoot.
+deep="$fits/$comp/$comp/$comp"
+mkdir -p "$deep" 2>/dev/null || {
+    echo "filesystem will not hold a ${#deep}-byte path; deep half skipped" >&2
+    exit 0
+}
+build_cache "$deep/c" 0 "$posA"
+convert "$deep/c.ndx" "path past the destinations"
+
+# The zip reader keeps its own copy of that base-path code, and only reaches it
+# once the archive opens, so this one needs a real zip at the deep path.
+python=$(find_python) || {
+    echo "python3 not found; zip half skipped" >&2
+    exit 0
+}
+"$python" - "$deep/c.zip" <<'EOF'
+import sys, zipfile
+
+zi = zipfile.ZipInfo("example.com/page.html")
+zi.compress_type = zipfile.ZIP_STORED
+zi.extra = (
+    "X-In-Cache: 1\r\nX-StatusCode: 200\r\nX-StatusMessage: OK\r\n"
+    "X-Size: 5\r\nContent-Type: text/html\r\n"
+    "Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\nX-Save: out/page.html\r\n"
+).encode()
+with zipfile.ZipFile(sys.argv[1], "w") as z:
+    z.writestr(zi, b"hello")
+EOF
+convert "$deep/c.zip" "zip at a path past the base buffer"
+test "$(mime_of "$dir/out.arc" example.com/page.html)" = text/html ||
+    fail "the zip entry was dropped instead of keeping an empty base path"
+
+echo "OK: an unparseable .ndx position reads as 0, and an over-long cache path is refused"

tests/120_local-proxytrack-webdav-default.test

@@ -0,0 +1,121 @@
+#!/bin/bash
+#
+# A PROPFIND whose path is an exact cache entry enumerates that entry's default
+# document under an empty name; proxytrack used to index one byte before it and
+# die on the whole listener.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+# shellcheck source=tests/testlib.sh
+. "$top_srcdir/tests/testlib.sh"
+
+python=$(find_python) || {
+    echo "python3 missing, skipping"
+    exit 77
+}
+command -v curl >/dev/null 2>&1 || {
+    echo "curl missing, skipping"
+    exit 77
+}
+# MSYS cannot reap a native listener, and the orphan wedges the suite (#595).
+if is_windows; then
+    echo "windows: cannot reap a backgrounded proxytrack, skipping"
+    exit 77
+fi
+
+dir=$(mktemp -d)
+ptpid=
+cleanup() {
+    stop_server "$ptpid"
+    rm -rf "$dir"
+}
+trap 'set +e; cleanup' EXIT
+trap 'set +e; cleanup; exit 1' INT TERM HUP
+
+printf 'HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 5\r\n\r\n' >"$dir/hdr"
+printf 'hello' >"$dir/body"
+alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
+{
+    printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
+    printf '2 0 test\n'
+    printf '\n\n'
+    printf 'http://example.com/page.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
+    cat "$dir/hdr" "$dir/body"
+} >"$dir/in.arc"
+
+freeport() {
+    "$python" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'
+}
+proxyport=$(freeport)
+icpport=$(freeport)
+
+proxytrack "127.0.0.1:$proxyport" "127.0.0.1:$icpport" "$dir/in.arc" >"$dir/pt.log" 2>&1 &
+ptpid=$!
+waited=0
+until grep -qE "HTTP Proxy installed on|Unable to (initialize a temporary server|create the server)" "$dir/pt.log"; do
+    kill -0 "$ptpid" 2>/dev/null || {
+        echo "FAIL: proxytrack exited before listening"
+        cat "$dir/pt.log"
+        exit 1
+    }
+    test "$waited" -lt 50 || {
+        echo "FAIL: proxytrack never announced its listen port"
+        exit 1
+    }
+    sleep 0.1
+    waited=$((waited + 1))
+done
+grep -q "HTTP Proxy installed on" "$dir/pt.log" || {
+    echo "FAIL: proxytrack failed to bind"
+    cat "$dir/pt.log"
+    exit 1
+}
+
+propfind() {
+    curl -s -i --max-time 30 -X PROPFIND -H "Depth: 1" \
+        "http://127.0.0.1:$proxyport/webdav/example.com/$1"
+}
+
+resp=$(propfind page.html) || {
+    echo "FAIL: PROPFIND on an exact cache entry did not answer"
+    cat "$dir/pt.log"
+    exit 1
+}
+grep -q '^HTTP/1\.[01] 207 ' <<<"$resp" || {
+    echo "FAIL: expected 207 Multi-Status"
+    printf '%s\n' "$resp"
+    exit 1
+}
+grep -q '<displayname>Default Document for the Folder</displayname>' <<<"$resp" || {
+    echo "FAIL: the default document of the entry was not listed"
+    printf '%s\n' "$resp"
+    exit 1
+}
+# The name alone would still pass on a truncated or garbage URL, so pin the
+# href: the default document is the collection path plus an empty name.
+grep -q '<href>/webdav/example\.com/page\.html/</href>' <<<"$resp" || {
+    echo "FAIL: the default document was listed under the wrong href"
+    printf '%s\n' "$resp"
+    exit 1
+}
+responses=$(grep -c '<response ' <<<"$resp" || true)
+test "$responses" -eq 2 || {
+    echo "FAIL: expected the root and its default document, got $responses responses"
+    printf '%s\n' "$resp"
+    exit 1
+}
+
+# The crash killed the process, so a second request is the liveness proof.
+resp=$(propfind "") || {
+    echo "FAIL: proxytrack died serving the previous PROPFIND"
+    cat "$dir/pt.log"
+    exit 1
+}
+grep -q '^HTTP/1\.[01] 207 ' <<<"$resp" || {
+    echo "FAIL: expected 207 Multi-Status on the follow-up listing"
+    printf '%s\n' "$resp"
+    exit 1
+}
+
+echo "OK: PROPFIND on an exact cache entry lists its default document and survives"

tests/121_local-ftp-nocache-splice.test

@@ -0,0 +1,188 @@
+#!/bin/bash
+#
+# With no cache entry an FTP transfer resumed on the strength of the local
+# file's mere existence, so a changed remote got its tail spliced onto the old
+# head (#823). Resuming now needs SIZE and MDTM to agree that the remote can
+# still extend the local copy; pass 2 proves a genuine partial still resumes.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
+command -v httrack >/dev/null || {
+    echo "could not find httrack" >&2
+    exit 1
+}
+
+server=$(nativepath "${testdir}/ftp-server.py")
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_ftp.XXXXXX")
+serverpid=
+cleanup() {
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+}
+trap 'set +e; cleanup' EXIT
+trap cleanup HUP INT QUIT PIPE TERM
+
+root="${tmpdir}/root"
+out="${tmpdir}/crawl"
+mode="${tmpdir}/mode"
+cmds="${tmpdir}/cmds"
+mkdir -p "$root" "$out"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+ok() { echo "OK: $*"; }
+size_of() { wc -c <"$1" | tr -d '[:space:]'; }
+# hts-log.txt is rewritten per pass, so this only ever sees the pass just run.
+no_errors() {
+    local errors
+    errors=$(grep -a 'Error:' "${out}/hts-log.txt" || true)
+    test -z "$errors" || fail "$1 reported errors: ${errors}"
+}
+# Commands the server saw since the last pass; each pass starts by emptying it.
+sent() { cat "$cmds"; }
+
+# Generations differ in their first bytes, so a splice shows wherever it lands;
+# the mtime is explicit because the fix decides on it, and a race would not do.
+write_body() {
+    "$python" -c 'import os, sys
+path, tag, fill, stamp = sys.argv[1:5]
+with open(path, "wb") as fp:
+    fp.write(("%s-BODY " % tag).encode() + tag[:1].encode() * int(fill))
+os.utime(path, (int(stamp), int(stamp)))' "${root}/a.bin" "$1" "$2" "$3"
+}
+# A copy this engine did not stamp: an archive restore, a cp without -p, or any
+# mirror predating the fix. Content is stale, so a splice fails cmp.
+plant_stale() {
+    "$python" -c 'import os, sys
+path, fill, stamp = sys.argv[1:4]
+with open(path, "wb") as fp:
+    fp.write(b"STALE-BODY " + b"S" * int(fill))
+os.utime(path, (int(stamp), int(stamp)))' "$mirror" "$1" "$2"
+}
+old_stamp=1767225600    # 2026-01-01Z
+new_stamp=1769904000    # 2026-02-01Z
+future_stamp=1772582400 # 2026-03-04Z, later than any remote below
+
+write_body OLD 8000 "$old_stamp"
+echo '/a.bin truncate' >"$mode"
+
+serverlog="${tmpdir}/server.out"
+"$python" "$server" --root "$(nativepath "$root")" \
+    --mode-file "$(nativepath "$mode")" --log "$(nativepath "$cmds")" \
+    >"$serverlog" 2>&1 &
+serverpid=$!
+port=
+for _ in $(seq 1 300); do
+    line=$(grep -m1 '^PORT ' "$serverlog" 2>/dev/null) && port="${line#PORT }" && break
+    kill -0 "$serverpid" 2>/dev/null || {
+        echo "ftp server exited early: $(cat "$serverlog")" >&2
+        exit 1
+    }
+    sleep 0.1
+done
+test -n "$port" || {
+    echo "could not discover ftp server port: $(cat "$serverlog")" >&2
+    exit 1
+}
+host="127.0.0.1_${port}"
+url="ftp://127.0.0.1:${port}/a.bin"
+mirror="${out}/${host}/a.bin"
+# --cache=0 is the whole point: no cache entry to tell a partial from a stale
+# copy. Bounded like every crawl pass, so a wedge fails instead of hanging.
+crawl() { run_with_timeout 300 httrack "$url" -O "$out" --quiet \
+    --disable-security-limits --robots=0 --timeout=20 --max-time=120 --retries=1 \
+    --cache=0; }
+
+# --- pass 1: the transfer dies mid-body and leaves a partial -----------------
+crawl >"${tmpdir}/log1" 2>&1 || true
+test -f "$mirror" || fail "pass 1 mirrored nothing to resume"
+partial=$(size_of "$mirror")
+whole=$(size_of "${root}/a.bin")
+if test "$partial" -le 0 || test "$partial" -ge "$whole"; then
+    fail "pass 1 left ${partial} bytes, wanted a partial of ${whole}"
+fi
+ok "pass 1 left a ${partial}-byte partial copy"
+
+# --- pass 2: an untouched remote is still resumed ----------------------------
+: >"$mode"
+: >"$cmds"
+crawl >"${tmpdir}/log2" 2>&1
+case "$(sent)" in
+*"REST ${partial}"*) ok "pass 2 resumed at the partial's ${partial} bytes" ;;
+*) fail "pass 2 sent no REST ${partial}; commands were: $(sent)" ;;
+esac
+cmp -s "$mirror" "${root}/a.bin" ||
+    fail "the resumed copy is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
+no_errors "pass 2"
+ok "the resumed transfer produced the served body and no error"
+
+# --- pass 3: a longer, newer remote over that complete copy ------------------
+write_body NEW 12000 "$new_stamp"
+: >"$cmds"
+crawl >"${tmpdir}/log3" 2>&1
+case "$(sent)" in
+*REST*) fail "resumed over a remote that had changed: $(sent)" ;;
+*RETR*) ok "the changed remote was re-fetched whole" ;;
+*) fail "pass 3 never reached RETR; commands were: $(sent)" ;;
+esac
+cmp -s "$mirror" "${root}/a.bin" ||
+    fail "the mirror is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
+no_errors "pass 3"
+ok "the mirror holds the new body, not a splice of both"
+
+# --- pass 4: a remote that changed without changing length or mtime ----------
+# MDTM cannot see this one: only the size test refuses it.
+write_body ALT 12000 "$new_stamp"
+: >"$cmds"
+crawl >"${tmpdir}/log4" 2>&1
+case "$(sent)" in
+*REST*) fail "resumed over a copy the remote can no longer extend: $(sent)" ;;
+*RETR*) ok "the same-length remote was re-fetched whole" ;;
+*) fail "pass 4 never reached RETR; commands were: $(sent)" ;;
+esac
+cmp -s "$mirror" "${root}/a.bin" ||
+    fail "the mirror is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
+no_errors "pass 4"
+ok "the same-length change replaced the mirror"
+
+# --- pass 5: a server with no MDTM cannot prove anything, so no resume -------
+# The copy is dated exactly as the remote, so only the missing MDTM refuses it.
+write_body NEW 12000 "$new_stamp"
+plant_stale 4000 "$new_stamp"
+echo '/a.bin nomdtm' >"$mode"
+: >"$cmds"
+crawl >"${tmpdir}/log5" 2>&1
+case "$(sent)" in
+*REST*) fail "resumed against a server that answers no MDTM: $(sent)" ;;
+*RETR*) ok "no MDTM, no resume" ;;
+*) fail "pass 5 never reached RETR; commands were: $(sent)" ;;
+esac
+cmp -s "$mirror" "${root}/a.bin" ||
+    fail "the mirror is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
+no_errors "pass 5"
+ok "the unprovable resume was re-fetched whole"
+
+# --- pass 6: a copy this engine never stamped, so dated by the local clock ---
+# Every mirror made before the fix looks like this. "Not newer than the remote"
+# waves it through, which is why the date has to match rather than merely lead.
+: >"$mode"
+plant_stale 4000 "$future_stamp"
+: >"$cmds"
+crawl >"${tmpdir}/log6" 2>&1
+case "$(sent)" in
+*REST*) fail "resumed over a copy dated by the local clock: $(sent)" ;;
+*RETR*) ok "the unstamped copy was re-fetched whole" ;;
+*) fail "pass 6 never reached RETR; commands were: $(sent)" ;;
+esac
+cmp -s "$mirror" "${root}/a.bin" ||
+    fail "the mirror is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
+no_errors "pass 6"
+ok "a newer local mtime does not license a resume"

tests/122_local-warc-revisit-headers.test

@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# A server-not-modified revisit stands for a 304 exchange, so its record has to
+# carry it: the 304 response block plus the concurrent request record holding the
+# conditional header that drew it (#826). The update pass archives under a name
+# of its own, or a revisit-only pass keeps the previous file instead (#759).
+
+set -eu
+
+: "${top_srcdir:=..}"
+
+export WARC_VALIDATE_UPDATE="warc-revisit.warc.gz"
+export WARC_VALIDATE_NORESP="index.html page.html"
+export WARC_VALIDATE_EXCHANGE=1
+
+bash "$top_srcdir/tests/local-crawl.sh" --errors 0 --warc-validate \
+    --rerun-args '--warc-file warc-revisit' \
+    --log-found 'no files updated' \
+    --found 'mini304/index.html' --found 'mini304/page.html' \
+    httrack 'BASEURL/mini304/index.html' --warc-file warc-out

tests/130_local-chunked-truncated.test

@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# A chunked response cut before its terminating zero-length chunk must be refused
+# like a short Content-Length body (#840): never cached, and never allowed to
+# replace a good copy on an --update. As with a short Content-Length, a first
+# crawl still leaves the partial bytes of a direct-to-disk resource in the mirror
+# (always.bin below); only the in-memory HTML class never reaches the disk.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+
+# First crawl: always.* are cut on every pass, the other three are the control.
+# reset.bin dies on an RST instead of a clean EOF, so the read error must keep
+# its own diagnostic rather than be relabelled by the framing check.
+bash "$top_srcdir/tests/local-crawl.sh" \
+    --log-found 'truncated chunked transfer.*chunktrunc/always\.html' \
+    --log-found 'Truncated chunked transfer.*after 2 retries.*chunktrunc/always\.html' \
+    --log-found 'Interrupted transfer.*chunktrunc/reset\.bin' \
+    --log-not-found 'runcated chunked transfer.*chunktrunc/(page|stay|file|reset|hostile)' \
+    --not-found 'chunktrunc/always.html' \
+    --found 'chunktrunc/always.bin' \
+    --cache-not-found '/chunktrunc/always.html' \
+    --cache-not-found '/chunktrunc/always.bin' \
+    --cache-not-found '/chunktrunc/reset.bin' \
+    --cache-found '/chunktrunc/page.html' \
+    --cache-found '/chunktrunc/file.bin' \
+    --cache-found '/chunktrunc/hostile.html' \
+    --file-matches 'chunktrunc/page.html' 'CHUNKTRUNC-PAGE-V1' \
+    --file-matches 'chunktrunc/file.bin' 'CHUNKTRUNC-BIN-V1' \
+    httrack 'BASEURL/chunktrunc/index.html'
+
+# --update: page.html and file.bin come back cut, hostile.html comes back with a
+# 0x80000000 chunk-size line (rejected outright, since as an int it is INT_MIN
+# and would otherwise reach realloc sign-extended), stay.html is the control.
+bash "$top_srcdir/tests/local-crawl.sh" \
+    --rerun-args '--update' \
+    --log-found 'truncated chunked transfer.*chunktrunc/page\.html' \
+    --log-found 'truncated chunked transfer.*chunktrunc/file\.bin' \
+    --log-found 'truncated chunked transfer.*chunktrunc/hostile\.html' \
+    --log-found 'Illegal chunk \(80000000\) for .*chunktrunc/hostile\.html' \
+    --log-not-found 'not enough memory \(-' \
+    --file-matches 'chunktrunc/page.html' 'CHUNKTRUNC-PAGE-V1' \
+    --file-not-matches 'chunktrunc/page.html' 'CHUNKTRUNC-PAGE-V2' \
+    --file-matches 'chunktrunc/file.bin' 'CHUNKTRUNC-BIN-V1' \
+    --file-not-matches 'chunktrunc/file.bin' 'CHUNKTRUNC-BIN-V2' \
+    --file-min-bytes 'chunktrunc/file.bin' 32768 \
+    --file-matches 'chunktrunc/hostile.html' 'CHUNKHOSTILE-V1' \
+    --cache-not-found '/chunktrunc/hostile.html' \
+    --file-matches 'chunktrunc/stay.html' 'CHUNKSTAY-V2' \
+    httrack 'BASEURL/chunktrunc/index.html'
+
+# -%B (tolerant) takes the truncated body, as it does a short Content-Length.
+bash "$top_srcdir/tests/local-crawl.sh" \
+    --log-not-found 'truncated chunked transfer \(terminating chunk missing' \
+    --found 'chunktrunc/always.html' \
+    --cache-found '/chunktrunc/always.html' \
+    --cache-found '/chunktrunc/always.bin' \
+    httrack 'BASEURL/chunktrunc/index.html' '-%B'

tests/131_local-savename-reserved.test

@@ -0,0 +1,62 @@
+#!/bin/bash
+#
+# url_savename must not name a URL into a directory the engine owns (#842).
+# Asserted on the tail only, so the -O root's rendering stays out of it.
+
+set -euo pipefail
+
+# resolve httrack before cd: make check puts a RELATIVE ../src on PATH
+httrack_bin=$(cd "$(dirname "$(command -v httrack)")" && pwd)/httrack
+
+scratch=$(mktemp -d)
+trap 'set +e; rm -rf "$scratch"' EXIT
+cd "$scratch"
+
+tail_is() {
+    local fil="$1" want="$2"
+    shift 2
+    local out
+    out="$("$httrack_bin" -O "$scratch/out" -#test=savename "$fil" \
+        application/octet-stream "$@" | sed -n 's/^savename: //p')"
+    case "${out//\\//}" in
+    */"$want") ;;
+    *)
+        echo "FAIL: '$fil' $* -> '$out' (want tail '$want')"
+        exit 1
+        ;;
+    esac
+}
+
+# A trailing space: cleanEndingSpaceOrDot() strips it again after the check.
+tail_is '/d/hts-tmp%20/a.bin.bak' 'd/hts-tmp_/a.bin.bak'
+tail_is '/d/hts-cache%20/new.zip' 'd/hts-cache_/new.zip'
+tail_is '/d/hts-cache%20%20./new.zip' 'd/hts-cache_/new.zip'
+
+# The first component, which the table's leading '/' never reached.
+tail_is '/hts-cache/new.zip' 'hts-cache_/new.zip' type=100
+tail_is '/hts-tmp/a.bin' 'hts-tmp_/a.bin' type=100
+# Wider than the engine directories: the DOS devices share the table.
+tail_is '/nul/x.bin' 'nul_/x.bin' type=100
+
+# A single-label hostname sits in that same first component.
+tail_is '/a.bin.bak' 'hts-tmp_/a.bin.bak' adr=hts-tmp
+tail_is '/x.bin' 'nul_/x.bin' adr=nul
+
+# '~' never survives the forbidden-character pass, which is what puts the
+# temporaries out of reach whatever the escape above does.
+tail_is '/~hts-tmp/a.bin' '_hts-tmp/a.bin'
+tail_is '/~hts-tmp/a.bin' '_hts-tmp/a.bin' type=100
+
+# Controls: a whole component only, and a space that does not end one.
+tail_is '/hts-tmpfoo/a.bin' 'hts-tmpfoo/a.bin'
+tail_is '/xhts-cache/a.bin' 'xhts-cache/a.bin'
+tail_is '/con%20ference/a.bin' 'con ference/a.bin'
+tail_is '/d/a.bin' 'www.example.com/d/a.bin'
+
+# A hostname label is not a reserved name, and renaming one would move an
+# existing mirror out from under --update.
+tail_is '/d/a.bin' 'aux.example.com/d/a.bin' adr=aux.example.com
+tail_is '/d/a.bin' 'nul.example.com/d/a.bin' adr=nul.example.com
+tail_is '/d/a.bin' 'hts-cache.example.com/d/a.bin' adr=hts-cache.example.com
+# But a trailing run still resolves back to the bare name.
+tail_is '/a.bin' 'hts-tmp_/a.bin' adr='hts-tmp.'

tests/132_local-bak-space-collision.test

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# #842/#774 end to end: the site serves a.bin and a sibling at "hts-tmp /",
+# which used to resolve to the very path the re-fetch backup of a.bin is
+# renamed to, so the update destroyed the sibling and reported success.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" \
+    --rerun-args '--update' \
+    --plant-file 'tmpspace/~hts-tmp/a.bin.bak' \
+    --log-found 'replacing leftover backup .*/~hts-tmp/a\.bin\.bak' \
+    --file-matches 'tmpspace/a.bin' 'BAKNAME-MAIN-V2' \
+    --file-matches 'tmpspace/hts-tmp_/a.bin.bak' 'BAKNAME-SIBLING' \
+    --file-min-bytes 'tmpspace/hts-tmp_/a.bin.bak' 2048 \
+    --not-found 'tmpspace/hts-tmp/a.bin.bak' \
+    --log-not-found 'could not back up' \
+    --log-not-found 'could not restore' \
+    httrack 'BASEURL/tmpspace/index.html'

tests/133_engine-reentrant-time.test

@@ -0,0 +1,40 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+: "${top_srcdir:=..}"
+
+# A non-UTC zone, so a helper that reached for gmtime instead of localtime (or
+# vice-versa) gives a different answer: CI runners are UTC, where they agree.
+TZ=XXX5
+export TZ
+LC_ALL=C
+export LC_ALL
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_localtime_st.XXXXXX") || exit 1
+trap 'set +e; rm -rf "$tmpdir"' EXIT
+trap 'rm -rf "$tmpdir"' HUP INT QUIT PIPE TERM
+
+# hts_localtime() reentrancy, and get_filetime_rfc822()'s GMT labelling (#806).
+expect_ok "localtime self-test" httrack -O "${tmpdir}/o1" \
+    -#test=localtime "${tmpdir}/data"
+
+# time_gmt_rfc822()/get_filetime_rfc822() must never format local time and
+# still label it GMT: on a gmtime() failure they must fail, not fall back to
+# (hts_)localtime() (#806). gmtime() failure is not reproducible at runtime
+# (it needs an out-of-range time_t, and neither function's input can be
+# pushed out of range from the outside), so this is a source guard against
+# the fallback creeping back in, not a dynamic probe.
+for fn in time_gmt_rfc822 get_filetime_rfc822; do
+    body=$(awk "/^[a-z ]*${fn}\\(/,/^}/" "$top_srcdir/src/htslib.c")
+    case "$body" in
+    *localtime*)
+        echo "FAIL: ${fn}() still falls back to (hts_)localtime()"
+        exit 1
+        ;;
+    esac
+done

tests/134_local-warc-revisit-ip.test

@@ -0,0 +1,15 @@
+#!/bin/bash
+# A 304 revisit record must name the peer it revisited (#838).
+set -eu
+
+: "${top_srcdir:=..}"
+
+export WARC_VALIDATE_UPDATE="warc-revisit.warc.gz"
+export WARC_VALIDATE_NORESP="mini304/index.html mini304/page.html"
+export WARC_VALIDATE_IP="mini304/index.html=127.0.0.1 mini304/page.html=127.0.0.1"
+
+bash "$top_srcdir/tests/local-crawl.sh" --errors 0 --warc-validate \
+    --rerun-args '--warc-file warc-revisit' \
+    --log-found 'no files updated' \
+    --found 'mini304/index.html' --found 'mini304/page.html' \
+    httrack 'BASEURL/mini304/index.html' --warc-file warc-out

tests/135_local-warc-forced-notmodified.test

@@ -0,0 +1,37 @@
+#!/bin/bash
+# An engine-forced not-modified is not a 304 the server sent (#839).
+set -eu
+
+: "${top_srcdir:=..}"
+
+# Scenario 1: real 304 (mini304) and engine-forced (errmask) side by side.
+export WARC_VALIDATE_UPDATE="warc-forced.warc.gz"
+export WARC_VALIDATE_EXCHANGE=1
+if test "${HTTPS_SUPPORT:-}" = "no"; then
+    # no OpenSSL: no payload digest to point a revisit at, so no record.
+    export WARC_VALIDATE_NORESP="mini304/index.html mini304/page.html"
+    export WARC_VALIDATE_NORECORD="errmask/keep.dat errmask/empty.dat"
+    export WARC_VALIDATE_PROFILE="mini304/index.html=server-not-modified mini304/page.html=server-not-modified"
+else
+    # empty.dat is a genuinely zero-length body: it still has a well-defined
+    # digest and must get its own revisit, not fall into the no-crypto case.
+    export WARC_VALIDATE_NORESP="mini304/index.html mini304/page.html errmask/keep.dat errmask/empty.dat"
+    export WARC_VALIDATE_PROFILE="mini304/index.html=server-not-modified mini304/page.html=server-not-modified errmask/keep.dat=identical-payload-digest errmask/empty.dat=identical-payload-digest"
+fi
+
+bash "$top_srcdir/tests/local-crawl.sh" --warc-validate \
+    --rerun-args '--warc-file warc-forced' \
+    --found 'mini304/index.html' --found 'mini304/page.html' \
+    --found 'errmask/keep.dat' --found 'errmask/empty.dat' \
+    --file-matches 'errmask/keep.dat' '^KEEP' \
+    --file-not-matches 'errmask/keep.dat' 'error 403' \
+    httrack 'BASEURL/mini304/index.html' 'BASEURL/errmask/index.html' --warc-file warc-out
+
+# Scenario 2: an all-forced pass, same archive name as pass 1, must not
+# replace the previous archive with an (almost) empty one (#777's shape).
+# No --wacz: that needs OpenSSL and would skip this scenario on a build
+# without it, hiding the one thing it exists to catch.
+bash "$top_srcdir/tests/local-crawl.sh" --archive-kept-on-rerun \
+    --archive-min-files 2 --rerun \
+    --found 'errmask/keep.dat' \
+    httrack 'BASEURL/errmask/index.html' --warc-file warc-out --warc-cdx

tests/136_local-warc-cache-priority.test

@@ -0,0 +1,26 @@
+#!/bin/bash
+# A cache-priority resume (-C1) never sent a request either: it must not be
+# archived as a server-not-modified revisit (#839).
+set -eu
+
+: "${top_srcdir:=..}"
+
+export WARC_VALIDATE_UPDATE="warc-cprio.warc.gz"
+export WARC_VALIDATE_EXCHANGE=1
+# Positive control on the fresh (pass-1) archive: a write-nothing regression
+# in warc_write_transaction would otherwise slip past the update-pass checks
+# below, which only assert the ABSENCE of certain records.
+export WARC_VALIDATE_BODY="mini304/index.html=3c68746d6c3e3c626f64793e0a093c6120687265663d22706167652e68746d6c223e706167653c2f613e0a3c2f626f64793e3c2f68746d6c3e0a"
+if test "${HTTPS_SUPPORT:-}" = "no"; then
+    # no OpenSSL: no payload digest to point a revisit at, so no record.
+    export WARC_VALIDATE_NORECORD="mini304/index.html mini304/page.html"
+    export WARC_VALIDATE_NO_REVISIT=1
+else
+    export WARC_VALIDATE_NORESP="mini304/index.html mini304/page.html"
+    export WARC_VALIDATE_PROFILE="mini304/index.html=identical-payload-digest mini304/page.html=identical-payload-digest"
+fi
+
+bash "$top_srcdir/tests/local-crawl.sh" --warc-validate \
+    --rerun-args '-C1 --warc-file warc-cprio' \
+    --found 'mini304/index.html' --found 'mini304/page.html' \
+    httrack 'BASEURL/mini304/index.html' --warc-file warc-out

tests/137_local-charsetless-encoding.test

@@ -0,0 +1,116 @@
+#!/bin/bash
+#
+# #833: a document declaring no charset falls back to iso-8859-1, so anything
+# the engine lifts out of it that already is UTF-8 must be left alone.
+
+set -e
+: "${top_srcdir:=..}"
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+server=$(nativepath "${testdir}/local-server.py")
+
+python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_833.XXXXXX") || exit 1
+serverpid=
+cleanup() {
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+}
+trap 'set +e; cleanup' EXIT
+trap cleanup HUP INT QUIT PIPE TERM
+
+serverlog="${tmpdir}/server.log"
+: >"$serverlog"
+mkdir "${tmpdir}/docroot"
+"$python" "$server" --root "$(nativepath "${tmpdir}/docroot")" \
+    >"$serverlog" 2>&1 &
+serverpid=$!
+port=
+for _ in $(seq 1 50); do
+    line=$(head -n1 "$serverlog" 2>/dev/null)
+    if test "${line%% *}" == "PORT"; then
+        port="${line#PORT }"
+        break
+    fi
+    kill -0 "$serverpid" 2>/dev/null || {
+        echo "server exited early: $(cat "$serverlog")"
+        exit 1
+    }
+    sleep 0.1
+done
+test -n "$port" || {
+    echo "could not discover server port"
+    exit 1
+}
+base="http://127.0.0.1:${port}"
+
+which httrack >/dev/null || {
+    echo "could not find httrack"
+    exit 1
+}
+
+# One crawl per variant: the generated top index carries the seed page's title.
+for variant in u8 l1 declared gb2312 l1decl; do
+    mkdir "${tmpdir}/${variant}"
+    httrack -O "${tmpdir}/${variant}" --quiet --disable-security-limits \
+        --robots=0 --timeout=30 --retries=0 \
+        "${base}/titleenc/${variant}.html" >"${tmpdir}/log-${variant}" 2>&1
+done
+
+# Byte-level assertions in python: portable, and the shell cannot mangle them.
+"$python" - "$tmpdir" "$port" <<'PY'
+import os
+import sys
+
+tmpdir, port = sys.argv[1], sys.argv[2]
+host = "127.0.0.1_%s" % port
+# What a second Latin-1 -> UTF-8 pass turns the two bytes of "é" into.
+DOUBLE = "é".encode().decode("latin-1").encode()
+failed = []
+
+
+def check(label, cond):
+    print("[%s] ..\t%s" % (label, "OK" if cond else "FAIL"))
+    if not cond:
+        failed.append(label)
+
+
+def rd(*parts):
+    with open(os.path.join(tmpdir, *parts), "rb") as f:
+        return f.read()
+
+
+index = rd("u8", "index.html")
+check("charset-less title kept its UTF-8 bytes", "Café-u8".encode() in index)
+check("charset-less title not double-encoded", DOUBLE not in index)
+
+page = rd("u8", host, "titleenc", "u8.html")
+check("fragment kept its UTF-8 bytes", "#café".encode() in page)
+check("fragment not double-encoded", DOUBLE not in page and b"%C3%83" not in page.upper())
+
+names = []
+for root, dirs, files in os.walk(os.path.join(tmpdir, "u8", host)):
+    names += dirs + files
+check("the accented leaf was mirrored", any("é" in n for n in names))
+check("no mirrored name was double-encoded", not any("Ã" in n for n in names))
+
+# Control: a real Latin-1 title still has to be converted, or the fix above is
+# just the conversion deleted.
+check("Latin-1 title converted", "Café-l1".encode() in rd("l1", "index.html"))
+check("declared UTF-8 title untouched",
+      "Café-declared".encode() in rd("declared", "index.html"))
+
+# Both titles below are valid UTF-8 already, and must still be decoded from the
+# charset the page declares.
+gb = rd("gb2312", "index.html")
+check("declared gb2312 title decoded", "图片-gb".encode() in gb)
+check("declared gb2312 title not left raw", "图片".encode("gb2312") + b"-gb" not in gb)
+l1d = rd("l1decl", "index.html")
+check("declared Latin-1 title decoded", "é-l1d".encode() in l1d)
+check("declared Latin-1 title not left raw", "é".encode("latin-1") + b"-l1d" not in l1d)
+
+if failed:
+    sys.exit("FAILED: %s" % ", ".join(failed))
+PY

tests/138_local-spool-name-overflow.test

@@ -0,0 +1,80 @@
+#!/bin/bash
+#
+# A frozen backlog slot spools to disk under a path_html-derived name when -p0
+# is in force, into a buffer sized from the save name instead; -N '%n' shortens
+# the save name enough to overrun it. -Z pins that a slot really was spooled,
+# so a build with neither fortify nor ASan still tests something.
+
+set -e
+: "${top_srcdir:=..}"
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+server=$(nativepath "${testdir}/local-server.py")
+
+python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_spool.XXXXXX") || exit 1
+serverpid=
+cleanup() {
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+}
+trap 'set +e; cleanup' EXIT
+trap cleanup HUP INT QUIT PIPE TERM
+
+serverlog="${tmpdir}/server.log"
+: >"$serverlog"
+"$python" "$server" --root "$(nativepath "${testdir}/server-root")" \
+    >"$serverlog" 2>&1 &
+serverpid=$!
+port=
+for _ in $(seq 1 50); do
+    line=$(head -n1 "$serverlog" 2>/dev/null)
+    if test "${line%% *}" == "PORT"; then
+        port="${line#PORT }"
+        break
+    fi
+    kill -0 "$serverpid" 2>/dev/null || {
+        echo "server exited early: $(cat "$serverlog")"
+        exit 1
+    }
+    sleep 0.1
+done
+test -n "$port" || {
+    echo "could not discover server port"
+    exit 1
+}
+
+which httrack >/dev/null || {
+    echo "could not find httrack"
+    exit 1
+}
+
+rc=0
+httrack -O "${tmpdir}/out" --quiet --disable-security-limits --robots=0 \
+    --timeout=30 --retries=0 -Z -p0 -N '%n' -c4 \
+    "http://127.0.0.1:${port}/bakname/index.html" \
+    >"${tmpdir}/log" 2>&1 || rc=$?
+test "$rc" -eq 0 || {
+    echo "httrack exited $rc"
+    cat "${tmpdir}/log"
+    exit 1
+}
+
+enginelog="${tmpdir}/out/hts-log.txt"
+test -r "$enginelog" || {
+    echo "no engine log at $enginelog"
+    exit 1
+}
+moved=$(grep -c 'slots ready moved to background' "$enginelog" || true)
+test "$moved" -gt 0 || {
+    echo "no slot was ever spooled, so the name was never built"
+    exit 1
+}
+serr=$(grep -c 'serialize error' "$enginelog" || true)
+test "$serr" -eq 0 || {
+    echo "the spool refused to write:"
+    grep 'serialize error' "$enginelog"
+    exit 1
+}

tests/139_local-query-charref.test

@@ -0,0 +1,127 @@
+#!/bin/bash
+# Issue #854: a character reference the page charset can not represent must be
+# percent-escaped in a query string (%26%23<decimal>%3B), not left as source
+# text whose '&' and '#' split the query into extra parameters. A reference the
+# charset does represent must still go out as its encoded bytes.
+set -eu
+
+: "${top_srcdir:=..}"
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+server=$(nativepath "${testdir}/local-server.py")
+root=$(nativepath "${testdir}/server-root")
+
+python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
+httrack=$(command -v httrack) || ! echo "could not find httrack" >&2 || exit 1
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_854.XXXXXX") || exit 1
+serverpid=
+cleanup() {
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+}
+trap 'cleanup || true' EXIT HUP INT QUIT PIPE TERM
+
+serverlog="${tmpdir}/server.log"
+: >"$serverlog"
+reqlog="${tmpdir}/requests.txt"
+: >"$reqlog"
+CHARREF_LOG="$reqlog" "$python" "$server" --root "$root" >"$serverlog" 2>&1 &
+serverpid=$!
+port=
+for _ in $(seq 1 50); do
+    line=$(head -n1 "$serverlog" 2>/dev/null) || line=
+    if test "${line%% *}" == "PORT"; then
+        port="${line#PORT }"
+        break
+    fi
+    kill -0 "$serverpid" 2>/dev/null || {
+        echo "server exited early: $(cat "$serverlog")"
+        exit 1
+    }
+    sleep 0.1
+done
+test -n "$port" || {
+    echo "could not discover server port"
+    exit 1
+}
+
+"$httrack" -O "${tmpdir}/crawl" --quiet --disable-security-limits --robots=0 \
+    --timeout=30 --retries=0 "http://127.0.0.1:${port}/charref/index.html" \
+    >"${tmpdir}/crawl.log" 2>&1 || true
+
+query_of() {
+    grep -F "/charref/$1.html?" "$reqlog" | cut -f1 |
+        sed "s|^/charref/$1.html?||" | sort -u
+}
+
+# check <label> <page> <want-parameter-count> <acceptable-query>...
+# The count is the origin's own parse of the wire bytes, independent of the
+# expected string. More than one query is acceptable where iconv and Windows
+# encode the same code point differently.
+check() {
+    local label="$1" page="$2" wantcount="$3" got count want ok=
+    shift 3
+    printf '[%s] ..\t' "$label"
+    got=$(query_of "$page") || got=
+    count=$(grep -F "/charref/${page}.html?" "$reqlog" | cut -f2 | sort -u) || count=
+    for want in "$@"; do
+        if test "$got" == "$want"; then
+            ok=1
+        fi
+    done
+    if test -z "$ok"; then
+        echo "FAIL: expected query '$1', got '${got}'"
+        exit 1
+    fi
+    if test "$count" != "$wantcount"; then
+        echo "FAIL: origin parsed '${count}' parameters, expected ${wantcount}"
+        exit 1
+    fi
+    echo "OK (${count} parameters)"
+}
+
+# iso-8859-1 fallback: € is unrepresentable (named and numeric alike), é is not.
+check "charset-less page" qnone 4 \
+    'a=%26%238364%3B&b=%26%238364%3B&c=%e9&d=x'
+# Control: utf-8 represents both, so they go out as their encoded bytes.
+check "declared utf-8 page" qutf8 4 \
+    'a=%e2%82%ac&b=%e2%82%ac&c=%c3%a9&d=x'
+# Multi-byte: あ and 〜 must survive as their encoded bytes, only € gets escaped.
+# The two cp932 tables disagree about 〜 (U+301C): iconv encodes it, Windows maps
+# 0x8160 to U+FF5E instead and reports the substitution, so both are correct.
+check "declared shift_jis page" qsjis 4 \
+    'a=%82%a0&c=%81`&b=%26%238364%3B&d=x' \
+    'a=%82%a0&c=%26%2312316%3B&b=%26%238364%3B&d=x'
+
+# iso-2022-jp encodes both references 7-bit clean, where a round-trip check saw
+# every one of them as unrepresentable. Asserted per field: iconv and Windows
+# differ on the trailing shift back to ASCII, so the bytes are not one string.
+printf '[declared iso-2022-jp page] ..\t'
+jis=$(query_of qjis) || jis=
+jiscount=$(grep -F "/charref/qjis.html?" "$reqlog" | cut -f2 | sort -u) || jiscount=
+for field in ${jis//&/ }; do
+    case "$field" in
+    a=%1b\$B* | c=%1b\$B*)
+        case "$field" in
+        *%26%23*)
+            echo "FAIL: '$field' escaped a reference iso-2022-jp represents"
+            exit 1
+            ;;
+        esac
+        ;;
+    # WideCharToMultiByte refuses lpUsedDefaultChar on a stateful codepage, so
+    # Windows substitutes € invisibly; harmless, '?' is no query delimiter.
+    b=%26%238364%3B | b=%3f | 'b=?' | d=x) ;;
+    *)
+        echo "FAIL: unexpected field '$field' in '$jis'"
+        exit 1
+        ;;
+    esac
+done
+if test "$jiscount" != 4; then
+    echo "FAIL: origin parsed '${jiscount}' parameters, expected 4"
+    exit 1
+fi
+echo "OK (${jiscount} parameters)"

tests/75_engine-longpath-posix.test

@@ -61,4 +61,19 @@ MINGW* | MSYS* | CYGWIN*)
     ;;
 esac
 
+# #852: with the directories eating the budget, the final segment was bounded
+# by the whole path alone, so the cut took its extension with it. Vacuous on
+# Linux, whose ceiling is out of the save buffer's reach; it bites on Windows
+# and macOS.
+dirs="/$(printf 'p%.0s' {1..200})/$(printf 'q%.0s' {1..200})/$(printf 'r%.0s' {1..200})"
+cut="$("$httrack_bin" -O "/$(printf 'o%.0s' {1..100})" -#test=savename \
+    "$dirs/$(printf 'Y%.0s' {1..300}).html" text/html | sed -n 's/^savename: //p')"
+case "$cut" in
+*.html) ;;
+*)
+    echo "FAIL: the cut dropped the extension: '$cut'"
+    exit 1
+    ;;
+esac
+
 echo "longpath-posix OK"

tests/86_local-proxytrack-cache-longfields.test

@@ -13,11 +13,6 @@ trap 'set +e; rm -rf "$dir"' EXIT
 
 pad() { printf "%${1}s" '' | tr ' ' "$2"; }
 
-# Longest surviving run of char $2 in file $1, or 0.
-runlen() {
-    grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
-}
-
 # --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
 # Each header overshoots its own destination, so a per-field clip is the only
 # way to get every expected length right at once.

tests/Makefile.am

@@ -1,7 +1,10 @@
+# One entry per line so parallel branches union-merge instead of colliding (#844).
+include $(srcdir)/tests-list.mk
+
 # Committed binary fixture read by 01_zlib-cache-golden.test. List it
 # explicitly: automake does not expand wildcards in EXTRA_DIST, so a glob would
 # silently drop it from the dist tarball and break "make distcheck".
-EXTRA_DIST = $(TESTS) renamefail.c crawl-test.sh run-all-tests.sh check-network.sh \
+EXTRA_DIST = $(TESTS) renamefail.c threadattrfail.c crawl-test.sh run-all-tests.sh check-network.sh \
 	proxy-https-server.py socks5-server.py proxy-connect-server.py \
 	proxytestlib.py tls-stall-server.py warc-validate.py wacz-validate.py \
 	pty-resize.py test-timeout.sh \
@@ -24,14 +27,21 @@ TESTS_ENVIRONMENT += V6_SUPPORT=$(V6_SUPPORT)
 TESTS_ENVIRONMENT += top_srcdir=$(top_srcdir)
 TESTS_ENVIRONMENT += RENAMEFAIL_LA=$(abs_builddir)/librenamefail.la
 TESTS_ENVIRONMENT += RENAMEFAIL_LIB=$(abs_builddir)/$(LT_CV_OBJDIR)/librenamefail.so
+TESTS_ENVIRONMENT += THREADATTRFAIL_LA=$(abs_builddir)/libthreadattrfail.la
+TESTS_ENVIRONMENT += THREADATTRFAIL_LIB=$(abs_builddir)/$(LT_CV_OBJDIR)/libthreadattrfail.so
 
 # rename() interposer for 01_engine-renameover.test; -rpath is what makes
 # libtool build a shared module rather than a static-only convenience library.
-check_LTLIBRARIES = librenamefail.la
+check_LTLIBRARIES = librenamefail.la libthreadattrfail.la
 librenamefail_la_SOURCES = renamefail.c
 librenamefail_la_LDFLAGS = -module -avoid-version -rpath $(abs_builddir)
 librenamefail_la_LIBADD = $(DL_LIBS)
 
+# pthread_create() interposer for 113_engine-threadattr-leak.test
+libthreadattrfail_la_SOURCES = threadattrfail.c
+libthreadattrfail_la_LDFLAGS = -module -avoid-version -rpath $(abs_builddir)
+libthreadattrfail_la_LIBADD = $(DL_LIBS)
+
 TEST_EXTENSIONS = .test
 # Run each .test through bash instead of execve()ing it. This lets "make check"
 # work when the source tree sits on a noexec filesystem (the driver would
@@ -43,193 +53,4 @@ TEST_EXTENSIONS = .test
 # cancelled step keeps neither its log nor its artifacts, so today a hang tells
 # us nothing at all. HTTRACK_TEST_TIMEOUT overrides the budget; 0 disables it.
 TEST_LOG_COMPILER = $(BASH) $(srcdir)/test-timeout.sh
-TESTS = \
-	00_runnable.test \
-	01_engine-addlink.test \
-	01_engine-changes.test \
-	01_engine-charset.test \
-	01_engine-cmdline.test \
-	01_engine-cmdline-split.test \
-	01_engine-cookies.test \
-	01_engine-copyopt.test \
-	01_engine-crange.test \
-	01_engine-dns.test \
-	01_engine-dnstimeout.test \
-	01_engine-doitlog.test \
-	01_engine-entities.test \
-	01_engine-footerfmt.test \
-	01_engine-footer-overflow.test \
-	01_engine-filelist.test \
-	01_engine-filter.test \
-	01_engine-filterdual.test \
-	01_engine-ftp-line.test \
-	01_engine-ftp-userpass.test \
-	01_engine-gmtime.test \
-	01_engine-backswap.test \
-	01_engine-cacheindex.test \
-	01_engine-hashtable.test \
-	01_engine-header.test \
-	01_engine-idna.test \
-	01_engine-identurl.test \
-	01_engine-proxyurl.test \
-	01_engine-socks5.test \
-	01_engine-identabs.test \
-	01_engine-escape-room.test \
-	01_engine-inplace-escape.test \
-	01_engine-lastchar.test \
-	01_engine-logcallback.test \
-	01_engine-makeindex.test \
-	01_engine-mime.test \
-	01_engine-parse.test \
-	01_engine-pause.test \
-	01_engine-rcfile.test \
-	01_engine-reconcile.test \
-	01_engine-expandhome.test \
-	01_engine-fsize.test \
-	01_engine-growsize.test \
-	01_engine-redirect.test \
-	01_engine-longpath-io.test \
-	01_engine-mirror-io.test \
-	01_engine-direnum.test \
-	01_engine-cookieimport.test \
-	01_engine-relative.test \
-	01_engine-renameover.test \
-	01_engine-robots.test \
-	01_engine-savename.test \
-	01_engine-selftest-dispatch.test \
-	01_engine-simplify.test \
-	01_engine-sniff.test \
-	01_engine-status.test \
-	01_engine-stripquery.test \
-	01_engine-strsafe.test \
-	01_engine-structcheck.test \
-	01_engine-syscharset.test \
-	01_engine-threadwait.test \
-	01_engine-topindex.test \
-	01_engine-urlhack.test \
-	01_engine-unescape-bounds.test \
-	01_engine-useragent.test \
-	01_engine-version-macros.test \
-	01_engine-warc-surt.test \
-	01_engine-xfread.test \
-	01_zlib-acceptencoding.test \
-	01_zlib-warc.test \
-	01_zlib-sitemap.test \
-	01_zlib-warc-cdx.test \
-	01_zlib-warc-wacz.test \
-	01_zlib-contentcodings.test \
-	01_zlib-cache.test \
-	01_zlib-cache-corrupt.test \
-	01_zlib-cache-legacy.test \
-	01_zlib-cache-golden.test \
-	01_zlib-cache-writefail.test \
-	01_zlib-repair-shift.test \
-	01_zlib-savename-cached.test \
-	02_manpage-regen.test \
-	02_update-cache.test \
-	10_crawl-simple.test \
-	11_crawl-cookies.test \
-	11_crawl-idna.test \
-	11_crawl-international.test \
-	11_crawl-longurl.test \
-	11_crawl-parsing.test \
-	12_crawl_https.test \
-	13_crawl_proxy_https.test \
-	13_local-cookies.test \
-	14_local-https.test \
-	15_local-types.test \
-	16_local-assume.test \
-	17_local-empty-ct.test \
-	18_local-update.test \
-	19_local-connect-fallback.test \
-	20_local-resume-loop.test \
-	21_local-intl-update.test \
-	22_local-broken-size.test \
-	23_local-errpage.test \
-	24_local-resume-overlap.test \
-	25_local-mime-exclude.test \
-	26_local-strip-query.test \
-	27_local-cookies-file.test \
-	28_local-pause.test \
-	29_local-redirect-fragment.test \
-	30_local-fragment-link.test \
-	32_local-cdispo.test \
-	33_local-delayed.test \
-	34_local-maxtime.test \
-	35_local-maxsize.test \
-	36_local-bigcrawl.test \
-	37_local-cache-outage.test \
-	38_local-update-304.test \
-	39_local-delayed-cancel.test \
-	40_local-why.test \
-	41_local-utf8-link.test \
-	42_local-maxsize-slow.test \
-	43_local-update-truncate.test \
-	44_local-update-errormask.test \
-	45_local-maxsize-recv.test \
-	46_local-update-304-resume.test \
-	47_local-crange-overflow.test \
-	48_local-crange-memresume.test \
-	49_local-cookiewall.test \
-	50_local-contentcodings.test \
-	51_local-update-codec.test \
-	52_local-socks5.test \
-	53_local-proxytrack-arc-reason.test \
-	53_local-proxytrack-cache-corrupt.test \
-	54_local-update-truncate-purge.test \
-	55_local-chunked.test \
-	56_local-proxy-noleak.test \
-	57_local-proxy-connect.test \
-	58_watchdog.test \
-	59_local-tls-stall.test \
-	60_crawl-log-salvage.test \
-	61_webhttrack-locale.test \
-	62_lang-integrity.test \
-	63_webhttrack-home.test \
-	64_local-intl-outdir.test \
-	65_port-siblings.test \
-	66_engine-port80-strip.test \
-	67_engine-delayed-truncate.test \
-	68_webhttrack-outdir-charset.test \
-	69_local-intl-logdir.test \
-	71_local-crange-repaircache.test \
-	72_watchdog-crawl.test \
-	73_local-warc.test \
-	74_local-warc-wacz.test \
-	74_local-warc-verbatim.test \
-	75_engine-longpath-posix.test \
-	76_cli-resize.test \
-	77_webhttrack-redirect.test \
-	78_webhttrack-sid.test \
-	79_local-proxytrack-webdav-mime.test \
-	80_engine-crash-symbolize.test \
-	81_webhttrack-maxsize.test \
-	82_webhttrack-browse-links.test \
-	83_webhttrack-argescape.test \
-	84_webhttrack-mirror-verbatim.test \
-	85_webhttrack-projpath.test \
-	86_local-proxytrack-cache-longfields.test \
-	87_local-proxytrack-nodate.test \
-	88_local-proxytrack-badmtime.test \
-	89_webhttrack-error-overflow.test \
-	90_webhttrack-checkbox-clear.test \
-	91_webhttrack-directory.test \
-	92_local-proxytrack-ndx-fields.test \
-	93_local-changes.test \
-	94_local-single-file.test \
-	95_local-sitemap.test \
-	96_local-refetch-keep.test \
-	97_local-warc-update-keep.test \
-	98_local-warc-segments.test \
-	99_local-robots-error.test \
-	100_local-purge-longpath.test \
-	101_local-update-stale-bak.test \
-	102_local-ftp-refetch.test \
-	103_teardown-status.test \
-	104_engine-warc-longurl.test \
-	105_suite-timeout.test \
-	110_local-ftp-parallel.test \
-	111_local-ftp-update-rest.test \
-	114_local-update-304-leak.test
-
 CLEANFILES = check-network_sh.cache

tests/ftp-server.py

@@ -9,6 +9,7 @@ directory). Each line is "<path> <mode>...", path "*" matching everything:
   truncate  send a prefix of the body, then drop the data connection
   empty     open the data connection and send nothing
   norest    answer REST with 500, so the client re-fetches from scratch
+  nomdtm    answer MDTM with 500, like a server predating RFC 3659
 """
 
 import argparse
@@ -16,6 +17,7 @@ import os
 import socket
 import sys
 import threading
+import time
 
 
 def reply(conn, text):
@@ -148,6 +150,15 @@ class Session(threading.Thread):
                 reply(conn, "213 %d" % os.path.getsize(path))
             else:
                 reply(conn, "550 no such file")
+        elif verb == "MDTM":
+            path = self.resolve(arg)
+            if "nomdtm" in self.modes():
+                reply(conn, "500 MDTM not understood")
+            elif path and os.path.isfile(path):
+                stamp = time.gmtime(os.path.getmtime(path))
+                reply(conn, "213 " + time.strftime("%Y%m%d%H%M%S", stamp))
+            else:
+                reply(conn, "550 no such file")
         elif verb == "REST":
             if "norest" in self.modes():
                 reply(conn, "500 REST not understood")

tests/local-crawl.sh

@@ -17,6 +17,7 @@
 #       --errors N --errors-content N --files N --found PATH ... --directory PATH ... \
 #       --log-found REGEX ... --log-not-found REGEX ... \
 #       --file-matches PATH REGEX ... --file-not-matches PATH REGEX ... \
+#       --cache-found URLTAIL ... --cache-not-found URLTAIL ... \
 #       --file-min-bytes PATH N --file-mode PATH OCTAL --max-mirror-bytes N \
 #       httrack BASEURL/some/path [httrack-args...]
 # --errors counts every "Error:" log line; --errors-content drops transient
@@ -25,6 +26,9 @@
 # --max/--min-mirror-bytes bound the mirrored content bytes (host root).
 # --file-matches/--file-not-matches grep (ERE) a mirrored file (PATH under the
 # host root), to assert rewritten link/content survived the crawl.
+# --cache-found/--cache-not-found assert whether hts-cache/new.zip holds an
+# entry whose URL ends with URLTAIL, e.g. /dir/page.html; being mirrored and
+# being cached are separate outcomes (#840).
 # --file-min-bytes asserts a mirrored file (PATH) is at least N bytes.
 # --file-mode asserts its octal permissions (e.g. 644); POSIX hosts only.
 # --rerun-args runs a second pass (same server and mirror dir) with the given
@@ -112,6 +116,25 @@ function find_hostroot {
     die "could not find host root under $out"
 }
 
+# Does the cache hold an entry whose URL ends with $1? An unreadable index is a
+# hard failure, else --cache-not-found would pass on a cache that never existed.
+# Suffix match over the whole key, so a URL with a query string needs the query
+# spelled out; a bare path would silently match nothing.
+function cache_has {
+    local rc
+    "$python" -c '
+import sys, zipfile
+try:
+    names = zipfile.ZipFile(sys.argv[1]).namelist()
+except Exception:
+    sys.exit(2)
+sys.exit(0 if any(n.endswith(sys.argv[2]) for n in names) else 1)
+' "${logroot}/hts-cache/new.zip" "$1"
+    rc=$?
+    test "$rc" -le 1 || die "cannot read cache index ${logroot}/hts-cache/new.zip"
+    return "$rc"
+}
+
 function assert_equals {
     info "$1"
     if test ! "$2" == "$3"; then
@@ -200,7 +223,7 @@ while test "$pos" -lt "$nargs"; do
         audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
         pos=$((pos + 1))
         ;;
-    --found | --not-found | --directory | --log-found | --log-not-found | --max-mirror-bytes | --min-mirror-bytes)
+    --found | --not-found | --directory | --log-found | --log-not-found | --max-mirror-bytes | --min-mirror-bytes | --cache-found | --cache-not-found)
         audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
         pos=$((pos + 1))
         ;;
@@ -345,6 +368,7 @@ if test "${#plants[@]}" -gt 0; then
         if test "${plants[$i]}" = "--plant-dir"; then
             mkdir -p "$path" || die "could not create $path"
         else
+            mkdir -p "$(dirname "$path")" || die "could not create ${path%/*}"
             echo "$plant_poison" >"$path" || die "could not write $path"
         fi
         result "OK"
@@ -461,7 +485,13 @@ debug "host root: $hostroot"
 
 # --- optional WARC validation (stdlib validator, no warcio) ------------------
 # WARC_VALIDATE_BODY="URLSUB=HEX" byte-checks a fresh-crawl response body;
-# WARC_VALIDATE_NORESP="URLSUB..." asserts those assets are revisits post-update.
+# WARC_VALIDATE_NORESP="URLSUB..." asserts those assets are revisits post-update;
+# WARC_VALIDATE_NORECORD="URLSUB..." asserts those assets have no record at all;
+# WARC_VALIDATE_IP="URLSUB=IP..." asserts the exact WARC-IP-Address on the record;
+# WARC_VALIDATE_PROFILE="URLSUB=SUBSTR..." asserts a revisit's WARC-Profile;
+# WARC_VALIDATE_NO_REVISIT=1 skips the "at least one revisit" requirement (a
+# no-OpenSSL leg where the only unchanged assets end up with no record at all);
+# WARC_VALIDATE_EXCHANGE=1 asserts each revisit carries its 304 request/response.
 if test -n "$warc_validate"; then
     validator=$(nativepath "${testdir}/warc-validate.py")
     warc=$(find "$mirrorroot" -maxdepth 2 \( -name '*.warc.gz' -o -name '*.warc' \) 2>/dev/null | sort | tail -n1)
@@ -479,7 +509,10 @@ if test -n "$warc_validate"; then
     # body and keeps Content-Encoding, instead of expecting a decoded body.
     test -n "${WARC_VALIDATE_VERBATIM:-}" && bodyargs+=(--verbatim)
     info "validating fresh WARC (response bodies)"
-    "$python" "$validator" "$(nativepath "$fresh")" "${bodyargs[@]}" >&2 ||
+    # macOS bash 3.2 calls an empty array unbound under set -u, and a caller
+    # asking only for the revisit checks leaves this one empty.
+    "$python" "$validator" "$(nativepath "$fresh")" \
+        ${bodyargs[@]+"${bodyargs[@]}"} >&2 ||
         die "fresh WARC validation failed"
     result "OK"
 
@@ -489,10 +522,21 @@ if test -n "$warc_validate"; then
     if test -n "${WARC_VALIDATE_UPDATE:-}"; then
         upd=$(find "$mirrorroot" -maxdepth 2 -name "$WARC_VALIDATE_UPDATE" 2>/dev/null | head -n1)
         test -n "$upd" || die "no $WARC_VALIDATE_UPDATE produced under $mirrorroot"
-        declare -a revargs=(--expect-revisit)
+        declare -a revargs=()
+        test -z "${WARC_VALIDATE_NO_REVISIT:-}" && revargs+=(--expect-revisit)
         for sub in ${WARC_VALIDATE_NORESP:-}; do
             revargs+=(--no-response-for "$sub")
         done
+        for sub in ${WARC_VALIDATE_NORECORD:-}; do
+            revargs+=(--no-record-for "$sub")
+        done
+        for spec in ${WARC_VALIDATE_IP:-}; do
+            revargs+=(--expect-ip "$spec")
+        done
+        for spec in ${WARC_VALIDATE_PROFILE:-}; do
+            revargs+=(--expect-revisit-profile "$spec")
+        done
+        test -n "${WARC_VALIDATE_EXCHANGE:-}" && revargs+=(--revisit-exchange)
         info "validating update WARC (revisits)"
         "$python" "$validator" "$(nativepath "$upd")" "${revargs[@]}" >&2 ||
             die "update WARC validation failed"
@@ -529,9 +573,25 @@ if test -n "$wacz_validate"; then
 fi
 
 # No crawl, even a cancelled one, may leave engine temporaries: .delayed (#107,
-# #483), or the .z/.u content-coding temps (#557).
+# #483), the .z/.u content-coding temps (#557), or the ~hts-tmp directory those
+# and the re-fetch backup live in (#774). Only a test that planted something in
+# ~hts-tmp itself owns what is left there, so only that case skips the scan.
 info "checking for leftover engine temporaries"
-leftovers=$(find "$out" \( -name '*.delayed' -o -name '*.z' -o -name '*.u' \) 2>/dev/null | head -5)
+scan_tmpdir=1
+i=0
+while test "$i" -lt "${#plants[@]}"; do
+    case "${plants[$((i + 1))]}" in
+    */~hts-tmp/*) scan_tmpdir=0 ;;
+    esac
+    i=$((i + 2))
+done
+if test "$scan_tmpdir" -eq 1; then
+    leftovers=$(find "$out" \( -name '*.delayed' -o -name '*.z' -o -name '*.u' \
+        -o -name '~hts-tmp' \) 2>/dev/null | head -5)
+else
+    leftovers=$(find "$out" \( -name '*.delayed' -o -name '*.z' -o -name '*.u' \) \
+        2>/dev/null | head -5)
+fi
 if test -z "$leftovers"; then result "OK"; else
     result "leftover: $leftovers"
     exit 1
@@ -569,6 +629,22 @@ while test "$i" -lt "${#audit[@]}"; do
             exit 1
         fi
         ;;
+    --cache-found)
+        i=$((i + 1))
+        info "checking cache holds ${audit[$i]}"
+        if cache_has "${audit[$i]}"; then result "OK"; else
+            result "not cached"
+            exit 1
+        fi
+        ;;
+    --cache-not-found)
+        i=$((i + 1))
+        info "checking cache lacks ${audit[$i]}"
+        if cache_has "${audit[$i]}"; then
+            result "cached"
+            exit 1
+        else result "OK"; fi
+        ;;
     --found)
         i=$((i + 1))
         info "checking for ${audit[$i]}"

tests/local-server.py

@@ -19,6 +19,8 @@ import gzip
 import hashlib
 import os
 import re
+import socket
+import struct
 import sys
 import time
 from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
@@ -1196,6 +1198,98 @@ class Handler(SimpleHTTPRequestHandler):
         self.send_header("Content-Length", "0")
         self.end_headers()
 
+    # Character references in a query string (#854). CHARREF_LOG collects the
+    # request-target the origin saw, plus the parameter count it parses out of
+    # it. Per page: a reference the declared charset represents (é, あ) and one
+    # it does not (€).
+    CHARREF_QUERY = "?a=€&b=€&c=é&d=x"
+    CHARREF_MB_QUERY = "?a=あ&c=〜&b=€&d=x"
+    CHARREF_PAGES = {
+        "none": ("text/html", CHARREF_QUERY),
+        "utf8": ("text/html; charset=utf-8", CHARREF_QUERY),
+        "sjis": ("text/html; charset=shift_jis", CHARREF_MB_QUERY),
+        "jis": ("text/html; charset=iso-2022-jp", CHARREF_MB_QUERY),
+    }
+
+    def record_charref(self):
+        log = os.environ.get("CHARREF_LOG")
+        if log:
+            fields = [f for f in urlsplit(self.path).query.split("&") if f]
+            with open(log, "a") as fp:
+                fp.write("%s\t%d\n" % (self.path, len(fields)))
+
+    def route_charref(self):
+        name = urlsplit(self.path).path[len("/charref/") :]
+        variant = name[: -len(".html")] if name.endswith(".html") else ""
+        if variant == "index":
+            body = "".join(
+                '<a href="%s.html">%s</a>' % (v, v) for v in self.CHARREF_PAGES
+            )
+            self.send_raw(("<html><body>%s</body></html>" % body).encode(), "text/html")
+        elif variant in self.CHARREF_PAGES:
+            ctype, query = self.CHARREF_PAGES[variant]
+            body = '<a href="q%s.html%s">q</a>' % (variant, query)
+            self.send_raw(("<html><body>%s</body></html>" % body).encode(), ctype)
+        elif variant[1:] in self.CHARREF_PAGES:
+            self.send_raw(b"<html><body>q</body></html>", "text/html")
+        else:
+            self.send_response(404)
+            self.send_header("Content-Length", "0")
+            self.end_headers()
+
+    # variant -> (Content-Type, title bytes, body bytes); the gb2312 and l1decl
+    # titles are valid UTF-8 as bytes, so only the declared charset decodes them
+    TITLEENC_PAGES = {
+        "u8": (
+            "text/html",
+            "Café-u8".encode(),
+            '<img src="s.svg#café"><a href="né.txt">leaf</a>'.encode(),
+        ),
+        # genuine latin-1: the guess must still be applied
+        "l1": (
+            "text/html",
+            "Café-l1".encode("latin-1"),
+            b'<a href="l1.txt">leaf</a>',
+        ),
+        "declared": ("text/html; charset=utf-8", "Café-declared".encode(), b"x"),
+        "gb2312": ("text/html; charset=gb2312", "图片".encode("gb2312") + b"-gb", b"x"),
+        "l1decl": (
+            "text/html; charset=iso-8859-1",
+            "é".encode("latin-1") + b"-l1d",
+            b"x",
+        ),
+    }
+
+    def route_titleenc(self):
+        name = unquote(urlsplit(self.path).path)[len("/titleenc/") :]
+        variant = name[: -len(".html")] if name.endswith(".html") else None
+        if variant in self.TITLEENC_PAGES:
+            ctype, title, body = self.TITLEENC_PAGES[variant]
+            self.send_raw(
+                b"<html><head><title>"
+                + title
+                + b"</title></head><body>"
+                + body
+                + b"</body></html>",
+                ctype,
+            )
+            return
+        if name == "s.svg":
+            self.send_raw(
+                (
+                    '<svg xmlns="http://www.w3.org/2000/svg">'
+                    '<symbol id="café"><rect/></symbol></svg>'
+                ).encode(),
+                "image/svg+xml",
+            )
+            return
+        if name in ("né.txt", "l1.txt"):
+            self.send_raw(b"leaf\n", "text/plain")
+            return
+        self.send_response(404)
+        self.send_header("Content-Length", "0")
+        self.end_headers()
+
     # resume / 416 loop (#206): the first GET stalls after a prefix so the crawl
     # can be interrupted (partial + temp-ref); every later request is 416.
     RESUME_PREFIX = b"PARTIAL-" + b"x" * 4096  # flushed before the stall
@@ -1507,6 +1601,120 @@ class Handler(SimpleHTTPRequestHandler):
             self.wfile.write(b"%X\r\n" % len(piece) + piece + b"\r\n")
         self.wfile.write(b"0\r\n\r\n")
 
+    # #840: a chunked stream cut before its terminating zero-length chunk.
+    CHUNKTRUNC_V1 = b"<html><body><p>CHUNKTRUNC-PAGE-V1</p></body></html>"
+    CHUNKTRUNC_V2 = b"<html><body><p>CHUNKTRUNC-PAGE-V2</p></body></html>"
+
+    CHUNKTRUNC_BIN_V1 = b"CHUNKTRUNC-BIN-V1\n" + b"\x07\x08\x09\xfe" * 8192
+    CHUNKTRUNC_BIN_V2 = b"CHUNKTRUNC-BIN-V2\n" + b"\x17\x18\x19\xee" * 8192
+
+    def route_chunktrunc_index(self):
+        self.send_html(
+            '\t<a href="page.html">page</a>\n'
+            '\t<a href="always.html">always</a>\n'
+            '\t<a href="stay.html">stay</a>\n'
+            '\t<a href="file.bin">file</a>\n'
+            '\t<a href="always.bin">alwaysbin</a>\n'
+            '\t<a href="hostile.html">hostile</a>\n'
+            '\t<a href="reset.bin">reset</a>\n'
+        )
+
+    def send_chunked(self, body, terminate, ctype="text/html; charset=utf-8"):
+        self.protocol_version = "HTTP/1.1"
+        self.send_response(200)
+        self.send_header("Content-Type", ctype)
+        self.send_header("Transfer-Encoding", "chunked")
+        self.send_header("Connection", "close")
+        self.end_headers()
+        if self.command == "HEAD":
+            return
+        try:
+            self.wfile.write(b"%X\r\n" % len(body) + body + b"\r\n")
+            if terminate:
+                self.wfile.write(b"0\r\n\r\n")
+            self.wfile.flush()
+        except OSError:
+            pass
+        self.close_connection = True
+
+    def route_chunktrunc_page(self):
+        if self.refetch_pass() == 1:
+            self.send_chunked(self.CHUNKTRUNC_V1, True)
+        else:
+            self.send_chunked(self.CHUNKTRUNC_V2, False)
+
+    # Same over the direct-to-disk path: the update pass delivers half the new
+    # body and never terminates it.
+    def route_chunktrunc_file(self):
+        octet = "application/octet-stream"
+        if self.refetch_pass() == 1:
+            self.send_chunked(self.CHUNKTRUNC_BIN_V1, True, octet)
+        else:
+            half = self.CHUNKTRUNC_BIN_V2[: len(self.CHUNKTRUNC_BIN_V2) // 2]
+            self.send_chunked(half, False, octet)
+
+    # Truncated on every pass, so a first crawl has nothing good to fall back on.
+    def route_chunktrunc_always(self):
+        self.send_chunked(b"<html><body><p>CHUNKTRUNC-ALWAYS</p></body></html>", False)
+
+    def route_chunktrunc_alwaysbin(self):
+        self.send_chunked(
+            b"CHUNKTRUNC-ALWAYSBIN\n" + b"\x27\x28\x29\xde" * 1000,
+            False,
+            "application/octet-stream",
+        )
+
+    # Control: terminated on both passes, so a normal --update still lands.
+    def route_chunktrunc_stay(self):
+        v = 1 if self.refetch_pass() == 1 else 2
+        self.send_chunked(b"<html><body><p>CHUNKSTAY-V%d</p></body></html>" % v, True)
+
+    # The update pass declares a chunk of 0x80000000, which sscanf("%x") lands in
+    # an int as INT_MIN: it must not read as the terminating chunk, and the sum
+    # it drives negative must not read as a complete body either.
+    def route_chunktrunc_hostile(self):
+        if self.refetch_pass() == 1:
+            self.send_chunked(b"<html><body><p>CHUNKHOSTILE-V1</p></body></html>", True)
+            return
+        self.protocol_version = "HTTP/1.1"
+        self.send_response(200)
+        self.send_header("Content-Type", "text/html; charset=utf-8")
+        self.send_header("Transfer-Encoding", "chunked")
+        self.send_header("Connection", "close")
+        self.end_headers()
+        if self.command == "HEAD":
+            return
+        try:
+            self.wfile.write(b"80000000\r\n")
+            self.wfile.flush()
+        except OSError:
+            pass
+        self.close_connection = True
+
+    # Aborts the chunked body with an RST, so the read fails rather than seeing a
+    # clean EOF and the transfer is already in error before the framing check.
+    def route_chunktrunc_reset(self):
+        self.protocol_version = "HTTP/1.1"
+        self.send_response(200)
+        self.send_header("Content-Type", "application/octet-stream")
+        self.send_header("Transfer-Encoding", "chunked")
+        self.send_header("Connection", "close")
+        self.end_headers()
+        if self.command == "HEAD":
+            return
+        try:
+            body = b"CHUNKRESET\n" + b"\x31\x32\x33\xcd" * 4096
+            self.wfile.write(b"%X\r\n" % len(body) + body + b"\r\n")
+            self.wfile.flush()
+            time.sleep(0.5)  # let the client consume the chunk first
+            self.connection.setsockopt(
+                socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0)
+            )
+            self.connection.close()
+        except OSError:
+            pass
+        self.close_connection = True
+
     # Content-Disposition naming: the attachment filename replaces the
     # URL-derived name; path components in it are stripped (RFC 2616).
     CDISPO_NAMES = {
@@ -1553,13 +1761,57 @@ class Handler(SimpleHTTPRequestHandler):
     def route_mini304_page(self):
         self.big_send(b"<html><body>tiny cacheable page</body></html>\n", "text/html")
 
+    # --- /bakname/: #774 — a mirrored file named like a re-fetch backup ----
+    # a.bin gets a new body every pass, so the update re-fetches it and takes a
+    # backup. The sibling carries a validator, so it only revalidates and must
+    # still be there afterwards. It is served under hts-tmp/, the directory the
+    # backup lives in, so the crawl asks for the exact path the engine writes.
+    BAKNAME_SIB = b"BAKNAME-SIBLING\n" + b"\x41\x42\x43\x44" * 512
+
+    def route_bakname_index(self):
+        self.send_html(
+            '\t<a href="a.bin">a</a>\n' '\t<a href="hts-tmp/a.bin.bak">bak</a>\n'
+        )
+
+    def route_bakname_main(self):
+        v = 1 if self.refetch_pass() == 1 else 2
+        self.send_raw(
+            b"BAKNAME-MAIN-V%d\n" % v + b"\x31\x32\x33\x34" * 512,
+            "application/octet-stream",
+        )
+
+    def route_bakname_sibling(self):
+        if self.headers.get("If-Modified-Since") or self.headers.get("If-None-Match"):
+            self.send_response(304)
+            self.send_header("Content-Length", "0")
+            self.end_headers()
+            return
+        self.send_response(200)
+        self.send_header("Content-Type", "application/octet-stream")
+        self.send_header("Last-Modified", BIG_LASTMOD)
+        self.send_header("Content-Length", str(len(self.BAKNAME_SIB)))
+        self.end_headers()
+        if self.command != "HEAD":
+            self.wfile.write(self.BAKNAME_SIB)
+
+    # --- /tmpspace/: #842 — same collision, reached through the trailing-space
+    # bypass of the reserved-segment escape. "hts-tmp%20" was escaped by nothing
+    # and then had its space stripped, so the sibling landed on the backup's own
+    # name on shipped defaults.
+    def route_tmpspace_index(self):
+        self.send_html(
+            '\t<a href="a.bin">a</a>\n' '\t<a href="hts-tmp%20/a.bin.bak">bak</a>\n'
+        )
+
     # --- /errmask/: issue #176 — a page that 200'd on the first crawl but 403s
     # on the update fetch must keep its good copy, not be overwritten nor purged.
     ERRMASK_GOOD = b"KEEP" + b"." * 1020  # 1024 B distinctive non-HTML body
     ERRMASK_ERR = b"<html><body>error 403</body></html>\n"
 
     def route_errmask_index(self):
-        self.send_html('\t<a href="keep.dat">keep</a>\n')
+        self.send_html(
+            '\t<a href="keep.dat">keep</a>\n\t<a href="empty.dat">empty</a>\n'
+        )
 
     def route_errmask_keep(self):
         # First crawl (no validator) gets the 1024 B body + Last-Modified; the
@@ -1580,6 +1832,24 @@ class Handler(SimpleHTTPRequestHandler):
         if self.command != "HEAD":
             self.wfile.write(self.ERRMASK_GOOD)
 
+    def route_errmask_empty(self):
+        # Same masking shape as keep.dat, but a genuinely zero-length body: an
+        # engine-forced revisit must still digest and archive it, not treat
+        # it as a missing-crypto case (#839).
+        if self.headers.get("If-Modified-Since") or self.headers.get("If-None-Match"):
+            self.send_response(403, "Forbidden")
+            self.send_header("Content-Type", "text/html")
+            self.send_header("Content-Length", str(len(self.ERRMASK_ERR)))
+            self.end_headers()
+            if self.command != "HEAD":
+                self.wfile.write(self.ERRMASK_ERR)
+            return
+        self.send_response(200)
+        self.send_header("Content-Type", "application/octet-stream")
+        self.send_header("Last-Modified", BIG_LASTMOD)
+        self.send_header("Content-Length", "0")
+        self.end_headers()
+
     # --- delayed-type degenerate paths (issues #5/#107) --------------------
     def route_delayed_index(self):
         self.send_html(
@@ -2059,6 +2329,14 @@ class Handler(SimpleHTTPRequestHandler):
         "/size/oversize.bin": route_size_oversize,
         "/chunked/index.html": route_chunked_index,
         "/chunked/page.html": route_chunked_page,
+        "/chunktrunc/index.html": route_chunktrunc_index,
+        "/chunktrunc/page.html": route_chunktrunc_page,
+        "/chunktrunc/always.html": route_chunktrunc_always,
+        "/chunktrunc/file.bin": route_chunktrunc_file,
+        "/chunktrunc/always.bin": route_chunktrunc_alwaysbin,
+        "/chunktrunc/stay.html": route_chunktrunc_stay,
+        "/chunktrunc/hostile.html": route_chunktrunc_hostile,
+        "/chunktrunc/reset.bin": route_chunktrunc_reset,
         "/errpage/index.html": route_errpage_index,
         "/errpage/good.html": route_errpage_good,
         "/errpage/missing.html": route_errpage_missing,
@@ -2134,10 +2412,17 @@ class Handler(SimpleHTTPRequestHandler):
         "/redir/index.html": route_redir_index,
         "/redir/go.php": route_redir_go,
         "/redir/target.html": route_redir_target,
+        "/bakname/index.html": route_bakname_index,
+        "/bakname/a.bin": route_bakname_main,
+        "/bakname/hts-tmp/a.bin.bak": route_bakname_sibling,
+        "/tmpspace/index.html": route_tmpspace_index,
+        "/tmpspace/a.bin": route_bakname_main,
+        "/tmpspace/hts-tmp /a.bin.bak": route_bakname_sibling,
         "/mini304/index.html": route_mini304_index,
         "/mini304/page.html": route_mini304_page,
         "/errmask/index.html": route_errmask_index,
         "/errmask/keep.dat": route_errmask_keep,
+        "/errmask/empty.dat": route_errmask_empty,
         "/maxrecv/index.html": route_maxrecv_index,
         "/maxrecv/r0.bin": route_maxrecv_404,
         "/maxrecv/r1.bin": route_maxrecv_404,
@@ -2325,6 +2610,12 @@ class Handler(SimpleHTTPRequestHandler):
         if path.startswith("/charset/"):
             self.route_charset()
             return True
+        if path.startswith("/charref/"):
+            self.route_charref()
+            return True
+        if path.startswith("/titleenc/"):
+            self.route_titleenc()
+            return True
         # Match percent-encoded paths (accented #157 route) by their decoded form.
         handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path))
         if handler is None:
@@ -2340,6 +2631,9 @@ class Handler(SimpleHTTPRequestHandler):
         return False
 
     def do_GET(self):
+        # Before reject_fragment(), so an unescaped '#' still shows up recorded.
+        if self.path.startswith("/charref/"):
+            self.record_charref()
         if self.reject_fragment():
             return
         if not self.dispatch():

tests/renamefail.c

@@ -1,9 +1,10 @@
-/* Borrows Windows' rename() for 01_engine-renameover.test, since POSIX cannot
-   reach hts_rename_over()'s aside fallback: an existing target is refused with
+/* Borrows Windows' rename() for the tests POSIX cannot otherwise reach:
+   hts_rename_over()'s aside fallback needs an existing target refused with
    EEXIST, and RENAMEFAIL_MODE=locked reports EACCES instead, as the CRT does
-   for a source another process holds. RENAMEFAIL_ASIDE_FAILS=N refuses the
-   first N moves back out of the parked ".hts-old" name, which is the only way
-   to reach hts_rename_over()'s restore retry (#790). */
+   for a source another process holds. =repair and =repair-eexist narrow the
+   refusal to the cache-repair move (#786), the latter through the fallback so
+   the retry inside it is the failing one. RENAMEFAIL_ASIDE_FAILS=N refuses the
+   first N moves back out of a parked ".hts-old" name (#790). */
 
 #define _GNU_SOURCE
 #include <dlfcn.h>
@@ -23,20 +24,29 @@ SHIM_EXPORT int rename(const char *oldpath, const char *newpath) {
   static int aside_failures = 0;
   const char *const mode = getenv("RENAMEFAIL_MODE");
   const char *const aside_fails = getenv("RENAMEFAIL_ASIDE_FAILS");
-  const int locked = mode != NULL && strcmp(mode, "locked") == 0;
+  const char *const slash = strrchr(oldpath, '/');
+  const char *const base = slash != NULL ? slash + 1 : oldpath;
+  const int repaired = strcmp(base, "repair.zip") == 0;
   struct stat st;
 
-  if (locked) {
+  if (mode != NULL && (strcmp(mode, "locked") == 0 ||
+                       (repaired && strcmp(mode, "repair") == 0))) {
     errno = EACCES;
     return -1;
   }
-  if (aside_fails != NULL && strstr(oldpath, ".hts-old") != NULL &&
+  if (repaired && mode != NULL && strcmp(mode, "repair-eexist") == 0) {
+    errno = stat(newpath, &st) == 0 ? EEXIST : EACCES;
+    return -1;
+  }
+  if (aside_fails != NULL && strstr(base, ".hts-old") != NULL &&
       aside_failures < atoi(aside_fails)) {
     aside_failures++;
     errno = EACCES;
     return -1;
   }
-  if (aside_fails == NULL && stat(newpath, &st) == 0) {
+  /* Only the bare shim emulates Windows for every rename; a narrowed mode
+     leaves the rest of the run on plain POSIX semantics. */
+  if (mode == NULL && aside_fails == NULL && stat(newpath, &st) == 0) {
     errno = EEXIST;
     return -1;
   }

tests/testlib.sh

@@ -24,6 +24,12 @@ nativepath() {
     fi
 }
 
+# Longest surviving run of char $2 in file $1, or 0: the length a field was
+# clipped to, read back out of a binary artifact.
+runlen() {
+    grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
+}
+
 # Run an engine self-test and require its "<label>: OK" line. No pipe into grep:
 # SIGPIPE would mask a failing exit status.
 expect_ok() {
@@ -97,7 +103,11 @@ kill_tree() {
             taskkill /F /IM httrack.exe >/dev/null 2>&1 || true
             taskkill /F /IM python.exe >/dev/null 2>&1 || true
         fi
+        return 0
     fi
+    # No caller puts $pid in its own group on Windows (set -m is skipped there),
+    # so -"$pid" here would target whatever real group $pid's number collides
+    # with -- possibly the harness's own -- and taskkill above already reaped it.
     kill -9 -"$pid" 2>/dev/null || kill -9 "$pid" 2>/dev/null || true
 }
 

tests/tests-list.mk

@@ -0,0 +1,213 @@
+# One entry per line, never a backslash continuation: unioning a continued list
+# drops an entry silently (#844). A deletion merged alongside another branch's
+# append is restored by the union -- remove a test in a commit of its own.
+TESTS =
+TESTS += 00_runnable.test
+TESTS += 01_engine-addlink.test
+TESTS += 01_engine-backswap.test
+TESTS += 01_engine-cacheindex.test
+TESTS += 01_engine-changes.test
+TESTS += 01_engine-charset.test
+TESTS += 01_engine-cmdline-split.test
+TESTS += 01_engine-cmdline.test
+TESTS += 01_engine-cookieimport.test
+TESTS += 01_engine-cookies.test
+TESTS += 01_engine-copyopt.test
+TESTS += 01_engine-crange.test
+TESTS += 01_engine-direnum.test
+TESTS += 01_engine-dns.test
+TESTS += 01_engine-dnstimeout.test
+TESTS += 01_engine-doitlog.test
+TESTS += 01_engine-entities.test
+TESTS += 01_engine-escape-room.test
+TESTS += 01_engine-expandhome.test
+TESTS += 01_engine-filelist.test
+TESTS += 01_engine-filterdual.test
+TESTS += 01_engine-filter.test
+TESTS += 01_engine-footerfmt.test
+TESTS += 01_engine-footer-overflow.test
+TESTS += 01_engine-fsize.test
+TESTS += 01_engine-ftp-line.test
+TESTS += 01_engine-ftp-userpass.test
+TESTS += 01_engine-gmtime.test
+TESTS += 01_engine-growsize.test
+TESTS += 01_engine-hashtable.test
+TESTS += 01_engine-header.test
+TESTS += 01_engine-identabs.test
+TESTS += 01_engine-identurl.test
+TESTS += 01_engine-idna.test
+TESTS += 01_engine-inplace-escape.test
+TESTS += 01_engine-lastchar.test
+TESTS += 01_engine-logcallback.test
+TESTS += 01_engine-longpath-io.test
+TESTS += 01_engine-makeindex.test
+TESTS += 01_engine-mime.test
+TESTS += 01_engine-mirror-io.test
+TESTS += 01_engine-parse.test
+TESTS += 01_engine-pause.test
+TESTS += 01_engine-proxyurl.test
+TESTS += 01_engine-rcfile.test
+TESTS += 01_engine-reconcile.test
+TESTS += 01_engine-redirect.test
+TESTS += 01_engine-relative.test
+TESTS += 01_engine-renameover.test
+TESTS += 01_engine-robots.test
+TESTS += 01_engine-savename.test
+TESTS += 01_engine-selftest-dispatch.test
+TESTS += 01_engine-simplify.test
+TESTS += 01_engine-sniff.test
+TESTS += 01_engine-socks5.test
+TESTS += 01_engine-status.test
+TESTS += 01_engine-stripquery.test
+TESTS += 01_engine-strsafe.test
+TESTS += 01_engine-structcheck.test
+TESTS += 01_engine-syscharset.test
+TESTS += 01_engine-threadwait.test
+TESTS += 01_engine-topindex.test
+TESTS += 01_engine-unescape-bounds.test
+TESTS += 01_engine-urlhack.test
+TESTS += 01_engine-useragent.test
+TESTS += 01_engine-version-macros.test
+TESTS += 01_engine-warc-surt.test
+TESTS += 01_engine-xfread.test
+TESTS += 01_zlib-acceptencoding.test
+TESTS += 01_zlib-cache-corrupt.test
+TESTS += 01_zlib-cache-golden.test
+TESTS += 01_zlib-cache-legacy.test
+TESTS += 01_zlib-cache.test
+TESTS += 01_zlib-cache-writefail.test
+TESTS += 01_zlib-contentcodings.test
+TESTS += 01_zlib-repair-shift.test
+TESTS += 01_zlib-savename-cached.test
+TESTS += 01_zlib-sitemap.test
+TESTS += 01_zlib-warc-cdx.test
+TESTS += 01_zlib-warc.test
+TESTS += 01_zlib-warc-wacz.test
+TESTS += 02_manpage-regen.test
+TESTS += 02_update-cache.test
+TESTS += 100_local-purge-longpath.test
+TESTS += 101_local-update-stale-bak.test
+TESTS += 102_local-ftp-refetch.test
+TESTS += 103_teardown-status.test
+TESTS += 104_engine-warc-longurl.test
+TESTS += 105_suite-timeout.test
+TESTS += 106_engine-repair-rename.test
+TESTS += 10_crawl-simple.test
+TESTS += 110_local-ftp-parallel.test
+TESTS += 111_local-ftp-update-rest.test
+TESTS += 113_engine-threadattr-leak.test
+TESTS += 112_local-single-file-fragment.test
+TESTS += 114_local-update-304-leak.test
+TESTS += 116_engine-rtrim.test
+TESTS += 117_local-proxytrack-ndx-cursor.test
+TESTS += 121_local-ftp-nocache-splice.test
+TESTS += 11_crawl-cookies.test
+TESTS += 11_crawl-idna.test
+TESTS += 11_crawl-international.test
+TESTS += 11_crawl-longurl.test
+TESTS += 11_crawl-parsing.test
+TESTS += 12_crawl_https.test
+TESTS += 13_crawl_proxy_https.test
+TESTS += 13_local-cookies.test
+TESTS += 14_local-https.test
+TESTS += 15_local-types.test
+TESTS += 16_local-assume.test
+TESTS += 17_local-empty-ct.test
+TESTS += 18_local-update.test
+TESTS += 19_local-connect-fallback.test
+TESTS += 20_local-resume-loop.test
+TESTS += 21_local-intl-update.test
+TESTS += 22_local-broken-size.test
+TESTS += 23_local-errpage.test
+TESTS += 24_local-resume-overlap.test
+TESTS += 25_local-mime-exclude.test
+TESTS += 26_local-strip-query.test
+TESTS += 27_local-cookies-file.test
+TESTS += 28_local-pause.test
+TESTS += 29_local-redirect-fragment.test
+TESTS += 30_local-fragment-link.test
+TESTS += 32_local-cdispo.test
+TESTS += 33_local-delayed.test
+TESTS += 34_local-maxtime.test
+TESTS += 35_local-maxsize.test
+TESTS += 36_local-bigcrawl.test
+TESTS += 37_local-cache-outage.test
+TESTS += 38_local-update-304.test
+TESTS += 39_local-delayed-cancel.test
+TESTS += 40_local-why.test
+TESTS += 41_local-utf8-link.test
+TESTS += 42_local-maxsize-slow.test
+TESTS += 43_local-update-truncate.test
+TESTS += 44_local-update-errormask.test
+TESTS += 45_local-maxsize-recv.test
+TESTS += 46_local-update-304-resume.test
+TESTS += 47_local-crange-overflow.test
+TESTS += 48_local-crange-memresume.test
+TESTS += 49_local-cookiewall.test
+TESTS += 50_local-contentcodings.test
+TESTS += 51_local-update-codec.test
+TESTS += 52_local-socks5.test
+TESTS += 53_local-proxytrack-arc-reason.test
+TESTS += 53_local-proxytrack-cache-corrupt.test
+TESTS += 54_local-update-truncate-purge.test
+TESTS += 55_local-chunked.test
+TESTS += 56_local-proxy-noleak.test
+TESTS += 57_local-proxy-connect.test
+TESTS += 58_watchdog.test
+TESTS += 59_local-tls-stall.test
+TESTS += 60_crawl-log-salvage.test
+TESTS += 61_webhttrack-locale.test
+TESTS += 62_lang-integrity.test
+TESTS += 63_webhttrack-home.test
+TESTS += 64_local-intl-outdir.test
+TESTS += 65_port-siblings.test
+TESTS += 66_engine-port80-strip.test
+TESTS += 67_engine-delayed-truncate.test
+TESTS += 68_webhttrack-outdir-charset.test
+TESTS += 69_local-intl-logdir.test
+TESTS += 71_local-crange-repaircache.test
+TESTS += 72_watchdog-crawl.test
+TESTS += 73_local-warc.test
+TESTS += 74_local-warc-verbatim.test
+TESTS += 74_local-warc-wacz.test
+TESTS += 75_engine-longpath-posix.test
+TESTS += 76_cli-resize.test
+TESTS += 77_webhttrack-redirect.test
+TESTS += 78_webhttrack-sid.test
+TESTS += 79_local-proxytrack-webdav-mime.test
+TESTS += 80_engine-crash-symbolize.test
+TESTS += 81_webhttrack-maxsize.test
+TESTS += 82_webhttrack-browse-links.test
+TESTS += 83_webhttrack-argescape.test
+TESTS += 84_webhttrack-mirror-verbatim.test
+TESTS += 85_webhttrack-projpath.test
+TESTS += 86_local-proxytrack-cache-longfields.test
+TESTS += 87_local-proxytrack-nodate.test
+TESTS += 88_local-proxytrack-badmtime.test
+TESTS += 89_webhttrack-error-overflow.test
+TESTS += 90_webhttrack-checkbox-clear.test
+TESTS += 91_webhttrack-directory.test
+TESTS += 92_local-proxytrack-ndx-fields.test
+TESTS += 93_local-changes.test
+TESTS += 94_local-single-file.test
+TESTS += 95_local-sitemap.test
+TESTS += 96_local-refetch-keep.test
+TESTS += 97_local-warc-update-keep.test
+TESTS += 98_local-warc-segments.test
+TESTS += 99_local-robots-error.test
+TESTS += 120_local-proxytrack-webdav-default.test
+TESTS += 107_local-bak-collision.test
+TESTS += 108_engine-refetch-backup.test
+TESTS += 122_local-warc-revisit-headers.test
+TESTS += 131_local-savename-reserved.test
+TESTS += 132_local-bak-space-collision.test
+TESTS += 119_local-proxytrack-ndx-fields2.test
+TESTS += 118_local-proxytrack-arcwrite.test
+TESTS += 130_local-chunked-truncated.test
+TESTS += 134_local-warc-revisit-ip.test
+TESTS += 135_local-warc-forced-notmodified.test
+TESTS += 136_local-warc-cache-priority.test
+TESTS += 137_local-charsetless-encoding.test
+TESTS += 138_local-spool-name-overflow.test
+TESTS += 133_engine-reentrant-time.test
+TESTS += 139_local-query-charref.test

tests/threadattrfail.c

@@ -0,0 +1,83 @@
+/* Forces pthread_create() to fail for 113_engine-threadattr-leak.test, and
+   tracks which attributes objects were left undestroyed after that failure
+   (#772). glibc allocates nothing for a default attr, so the imbalance is
+   invisible to a leak checker and has to be counted here. */
+
+#define _GNU_SOURCE
+#include <dlfcn.h>
+#include <errno.h>
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+/* The tree builds with -fvisibility=hidden, which would hide the interposer. */
+#define SHIM_EXPORT __attribute__((visibility("default")))
+
+/* Every create is refused in the only mode this shim runs in, so the process
+   stays single-threaded: unlocked counters and a fixed 64 slots are enough. */
+#define PENDING_MAX 64
+
+static const void *pending[PENDING_MAX];
+static unsigned int pending_n;
+static unsigned int sabotaged;
+
+SHIM_EXPORT int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
+                               void *(*start)(void *), void *arg);
+SHIM_EXPORT int pthread_attr_destroy(pthread_attr_t *attr);
+
+SHIM_EXPORT int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
+                               void *(*start)(void *), void *arg) {
+  static int (*real_create)(pthread_t *, const pthread_attr_t *,
+                            void *(*) (void *), void *) = NULL;
+
+  if (getenv("THREADATTRFAIL") != NULL) {
+    /* only an attr-bearing spawn can leak one, so only those count as tested */
+    if (attr != NULL) {
+      sabotaged++;
+      if (pending_n < PENDING_MAX)
+        pending[pending_n++] = attr;
+    }
+    return EAGAIN;
+  }
+  if (real_create == NULL) {
+    *(void **) &real_create = dlsym(RTLD_NEXT, "pthread_create");
+    if (real_create == NULL)
+      return ENOSYS;
+  }
+  return real_create(thread, attr, start, arg);
+}
+
+SHIM_EXPORT int pthread_attr_destroy(pthread_attr_t *attr) {
+  static int (*real_destroy)(pthread_attr_t *) = NULL;
+  unsigned int i;
+
+  /* drop one entry, so an address reused by a later init still counts once */
+  for (i = pending_n; i > 0; i--) {
+    if (pending[i - 1] == attr) {
+      memmove(&pending[i - 1], &pending[i],
+              (pending_n - i) * sizeof(pending[0]));
+      pending_n--;
+      break;
+    }
+  }
+  if (real_destroy == NULL) {
+    *(void **) &real_destroy = dlsym(RTLD_NEXT, "pthread_attr_destroy");
+    if (real_destroy == NULL)
+      return ENOSYS;
+  }
+  return real_destroy(attr);
+}
+
+__attribute__((destructor)) static void threadattrfail_report(void) {
+  const char *const path = getenv("THREADATTRFAIL_LOG");
+  FILE *fp;
+
+  if (path == NULL)
+    return;
+  fp = fopen(path, "wb");
+  if (fp == NULL)
+    return;
+  fprintf(fp, "sabotaged %u\nundestroyed %u\n", sabotaged, pending_n);
+  fclose(fp);
+}

tests/warc-validate.py

@@ -11,6 +11,16 @@
 #                             WARC-Payload-Digest matching sha1(body) when present
 #   --no-response-for SUB     the asset containing SUB must be a revisit: no
 #                             response may target it, and a revisit must
+#   --no-record-for SUB       no record of any type (response/revisit/resource)
+#                             may target an asset containing SUB
+#   --expect-ip SUB=IP        a response or revisit targeting SUB must carry
+#                             WARC-IP-Address: IP exactly
+#   --expect-revisit-profile SUB=SUBSTR  a revisit targeting SUB must carry a
+#                             WARC-Profile containing SUBSTR
+#   --revisit-exchange        every server-not-modified revisit carries the 304
+#                             exchange it stands for: a 304 status line in its
+#                             block, and a concurrent request record holding the
+#                             conditional header that drew it
 #   --verbatim                compressed asset: --expect-body-hex instead keeps
 #                             Content-Encoding, checks the HTTP Content-Length is
 #                             the stored (compressed) length, asserts the stored
@@ -101,9 +111,15 @@ def check_body_verbatim(rec, http_hdr, body, sub, want, digests_emitted):
 def main():
     argv = sys.argv[1:]
     expect_revisit = "--expect-revisit" in argv
+    revisit_exchange = "--revisit-exchange" in argv
     verbatim = "--verbatim" in argv
     body_specs = [s.split("=", 1) for s in opt_values(argv, "--expect-body-hex")]
     no_resp = opt_values(argv, "--no-response-for")
+    no_record = opt_values(argv, "--no-record-for")
+    ip_specs = [s.split("=", 1) for s in opt_values(argv, "--expect-ip")]
+    profile_specs = [
+        s.split("=", 1) for s in opt_values(argv, "--expect-revisit-profile")
+    ]
     path = [a for a in argv if not a.startswith("--") and "=" not in a][0]
     data = open(path, "rb").read()
 
@@ -116,6 +132,10 @@ def main():
     total = revisits = responses = infos = 0
     body_hits = {sub: False for sub, _ in body_specs}
     revisit_hits = {sub: False for sub in no_resp}
+    ip_hits = {sub: False for sub, _ in ip_specs}
+    profile_hits = {sub: False for sub, _ in profile_specs}
+    requests = {}  # WARC-Concurrent-To -> request block
+    exchanges = []  # (shown uri, record id) of the server-not-modified revisits
     for rec in records(data):
         total += 1
         if not rec.startswith(b"WARC/1."):
@@ -138,6 +158,23 @@ def main():
             sys.exit("record %d: missing \\r\\n\\r\\n trailer" % total)
         wtype = field(header, b"WARC-Type")
         uri = field(header, b"WARC-Target-URI") or b""
+        if wtype in (b"response", b"revisit", b"resource"):
+            for sub in no_record:
+                if sub.encode() in uri:
+                    sys.exit(
+                        "unexpected %s record for %s (want no record)"
+                        % (wtype.decode(), sub)
+                    )
+        if wtype in (b"response", b"revisit"):
+            for sub, want_ip in ip_specs:
+                if sub.encode() in uri:
+                    got = field(header, b"WARC-IP-Address") or b""
+                    if got.decode("ascii", "replace") != want_ip:
+                        sys.exit(
+                            "%s for %s: WARC-IP-Address %r != %r"
+                            % (wtype.decode(), sub, got, want_ip)
+                        )
+                    ip_hits[sub] = True
         if wtype == b"warcinfo":
             infos += 1
         elif wtype == b"response":
@@ -176,6 +213,34 @@ def main():
             for sub in no_resp:
                 if sub.encode() in uri:
                     revisit_hits[sub] = True
+            for sub, want_sub in profile_specs:
+                if sub.encode() in uri:
+                    if want_sub.encode() not in profile:
+                        sys.exit(
+                            "revisit for %s: WARC-Profile %r lacks %r"
+                            % (shown, profile, want_sub)
+                        )
+                    profile_hits[sub] = True
+            if revisit_exchange and b"server-not-modified" in profile:
+                block = rec[hdr_end : hdr_end + block_len]
+                status = block.split(b"\r\n", 1)[0].split()
+                if len(status) < 2 or status[1] != b"304":
+                    sys.exit(
+                        "revisit for %s does not carry the 304 it stands for: %r"
+                        % (shown, block.split(b"\r\n", 1)[0])
+                    )
+                exchanges.append((shown, field(header, b"WARC-Record-ID")))
+        elif wtype == b"request":
+            conc = field(header, b"WARC-Concurrent-To")
+            if conc is not None:
+                requests[conc] = rec[hdr_end : hdr_end + block_len].lower()
+
+    for shown, rec_id in exchanges:
+        req = requests.get(rec_id)
+        if req is None:
+            sys.exit("revisit for %s has no request record" % shown)
+        if b"if-modified-since:" not in req and b"if-none-match:" not in req:
+            sys.exit("request record for %s carries no conditional header" % shown)
 
     if total < 1:
         sys.exit("no records found")
@@ -189,6 +254,12 @@ def main():
     for sub, hit in revisit_hits.items():
         if not hit:
             sys.exit("no revisit record found for unchanged asset %s" % sub)
+    for sub, hit in ip_hits.items():
+        if not hit:
+            sys.exit("no response/revisit record found for --expect-ip %s" % sub)
+    for sub, hit in profile_hits.items():
+        if not hit:
+            sys.exit("no revisit record found for --expect-revisit-profile %s" % sub)
     print(
         "warc-validate: %d records OK (%d response, %d revisit)"
         % (total, responses, revisits)