Correct stale facts in the HTML help pages

- faq.html: the SOCKS question answered "Not yet!" — SOCKS5 and HTTP
  CONNECT proxies have shipped (-P socks5://|connect://); rewrite the
  answer and fix the "If there"/"Is there" typo.
- cache.html: minizip emits a ZIP64 central directory automatically for
  archives over 4GiB or 65535 entries, so drop the "(not currently
  supported)" claim and keep the real limit (a single entry of 4GiB+).
- filters.html: two -mime:video/* example rows described "application/"
  instead of the video rule; fix them, and "priority is it is declared
  later" -> "if it is declared later".
- plug.html: compile-breaking fprintf(stder, ...) in the sample module.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

.github/dependabot.yml

@@ -0,0 +1,8 @@
+version: 2
+updates:
+  # Bumps builtin-baseline in src/vcpkg.json so the pinned OpenSSL/zlib can't rot.
+  # windows-build.yml validates each bump; the pin makes what ships reproducible.
+  - package-ecosystem: vcpkg
+    directory: /src
+    schedule:
+      interval: weekly

.github/workflows/windows-build.yml

@@ -44,6 +44,20 @@ jobs:
         shell: pwsh
         run: vcpkg integrate install
 
+      # The runner image's vcpkg checkout is pinned to some commit; our manifest's
+      # builtin-baseline is usually newer, so `git show <baseline>:versions/...`
+      # fails until that commit is local. Fetch exactly the pinned baseline (read
+      # from the manifest, so Dependabot bumps need no workflow edit).
+      - name: Fetch the pinned vcpkg baseline
+        shell: pwsh
+        run: |
+          $ErrorActionPreference = "Stop"
+          $root = if ($env:VCPKG_INSTALLATION_ROOT) { $env:VCPKG_INSTALLATION_ROOT } else { "C:\vcpkg" }
+          $baseline = (Get-Content src/vcpkg.json -Raw | ConvertFrom-Json).'builtin-baseline'
+          if (-not $baseline) { throw "no builtin-baseline in src/vcpkg.json" }
+          git -C $root fetch --no-tags origin $baseline
+          if ($LASTEXITCODE -ne 0) { throw "could not fetch vcpkg baseline $baseline" }
+
       # httrack and webhttrack carry a ProjectReference to libhttrack, so building
       # them builds it first; proxytrack is standalone.
       - name: Build
@@ -188,7 +202,7 @@ jobs:
 
           # Every gate here exits 77, so an all-skipped suite would report green having
           # tested nothing: pin the skips, and floor the passes in case the glob empties.
-          expected_skips=" 48_local-crange-memresume.test"  # pending #581
+          expected_skips=" 48_local-crange-memresume.test 71_local-crange-repaircache.test"  # pending #581
           [ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
           [ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
           [ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

AGENTS.md

@@ -19,6 +19,13 @@ the operational checklist: toolchain, invariants, and how to ship a change.
   (`request_queue_size`) so macOS/BSD don't drop connections under a parallel
   `-c16` bigcrawl the way Python's default backlog of 5 did.
   Or run `sh build.sh` to do bootstrap + configure + make in one shot.
+- A `tests/NN_*.test` runs only if listed in `tests/Makefile.am`'s `TESTS`; an
+  unregistered file is silently skipped.
+- `make check` prepends the build's `src/` to `PATH`, but a hand-run `.test` does
+  not — an installed `/usr/bin/httrack` then shadows your build. Run via `make
+  check`, or `PATH="<bld>/src:$PATH"` for a manual run.
+- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
+  `local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
 
 ## Hard invariants
 - **Generated autotools files are NOT in git.** `configure`, every
@@ -31,15 +38,32 @@ the operational checklist: toolchain, invariants, and how to ship a change.
 - **Format only changed lines** with `git clang-format` (clang-format 19). Never
   reformat untouched code: the engine was formatted by an old tool and won't
   round-trip.
-- **Byte-safe edits.** Files with raw high bytes are ISO-8859-1 (French
-  comments). Edit them byte-wise (`perl -0pi`, `sed`), not through a tool that
-  re-encodes to UTF-8 and corrupts them.
+- **Byte-safe edits.** A few tracked files carry raw ISO-8859-1 high bytes
+  (French comments): `src/htsconcat.c`, `lang/*.txt`, `html/contact.html`, and
+  the `fuzz/corpus/*` vectors. Edit those byte-wise (`perl -0pi`, `sed`), not
+  through a tool that re-encodes to UTF-8 and corrupts them. The rest of the tree
+  is UTF-8 and safe to edit normally.
 
 ## Security (HTTrack parses hostile input off the network)
 - Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
   `untrusted < limit - controlled` — never `controlled + untrusted < limit`,
   which can wrap and pass.
 
+## C conventions
+- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
+  `malloct`/`calloct`/`realloct`/`freet`/`strdupt`, in test and selftest code
+  too. `freet` NULLs its (lvalue) argument and tolerates NULL; `calloct(n, sz)`
+  keeps calloc's arg order. Only exception: storing or calling a libc symbol
+  itself (e.g. a resolver-backend function pointer).
+- **Exported API is `HTSEXT_API`.** Everything else is hidden by
+  `-fvisibility=hidden` and free to change (check with `nm -D --defined-only
+  libhttrack.so`). Touching an installed-header struct (see `DevIncludes_DATA` in
+  `src/Makefile.am`) or an exported signature is an ABI break — flag and discuss,
+  bump the soname, and prefer keeping the old entry point beside a new one.
+- **Windows ABI is free to break, POSIX is not.** The Windows DLL ships next to
+  the exe with no soname contract, so a `_WIN32`-only ABI change needs no
+  deprecation dance; POSIX/ELF keeps the flag-discuss-bump rules.
+
 ## Code & prose
 - Be terse. Comment the why, in English; translate French comments you touch.
 - Strip AI tells from prose (em-dash overuse, rule-of-three, filler, vague

html/cache.html

@@ -173,7 +173,7 @@ There are also specific issues regarding this format:
 <ul>
 <li>The data in the central directory (such as CD extra field, and CD comments) are not used</li>
 <li>The ZIP archive is allowed to contains more than 2^16 files (65535) ; in such case the total number of entries in the 32-bit central directory is 65536 (0xffff), but the presence of the 64-bit central directory is not mandatory</li>
-<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory must be present <b>(not currently supported)</b></li>
+<li>The ZIP archive is allowed to contains more than 2^32 bytes (4GiB) ; in such case the 64-bit central directory is emitted automatically (a single stored entry of 4GiB or more is not supported)</li>
 </ul>
 
 <br />

html/faq.html

@@ -263,7 +263,7 @@ clear language!
 
 <li><a href="#QM8">Can HTTrack generates HP-UX or ISO9660 compatible files?</a><br></li>
 
-<li><a href="#QM9">If there any SOCKS support?</a><br></li>
+<li><a href="#QM9">Is there any SOCKS support?</a><br></li>
 
 <li><a href="#QM10">What's this hts-cache directory? Can I remove it?</a><br></li>
 
@@ -835,8 +835,8 @@ A: <em>Yes. Use user:password@your_proxy_name as your proxy name (example: <tt>s
 <br><br><a NAME="QM8">Q: <strong>Can HTTrack generates HP-UX or ISO9660 compatible files?</strong></a><br>
 A: <em>Yes. See the build options (-N, or see the WinHTTrack options)</em>
 
-<br><br><a NAME="QM9">Q: <strong>If there any SOCKS support?</strong></a><br>
-A: <em>Not yet!</em>
+<br><br><a NAME="QM9">Q: <strong>Is there any SOCKS support?</strong></a><br>
+A: <em>Yes. HTTrack supports SOCKS5 and HTTP CONNECT proxies: give the proxy with a scheme prefix, e.g. <tt>-P socks5://host:port</tt> or <tt>-P connect://host:port</tt> (prefix <tt>user:pass@</tt> before the host for authenticated proxies).</em>
 
 <br><br><a NAME="QM10">Q: <strong>What's this hts-cache directory? Can I remove it?</strong></a><br>
 A: <em>NO if you want to update the site, because this directory is used by HTTrack for this purpose. 

html/filters.html

@@ -189,7 +189,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     <br>
     Filters are analyzed by HTTrack from the first filter to the last one. The complete URL
     name is compared to filters defined by the user or added automatically by HTTrack. <br><br>
-    A scan rule has an higher priority is it is declared later - hierarchy is important: <br>
+    A scan rule has a higher priority if it is declared later - hierarchy is important: <br>
 
     <br>
     <table BORDER="1" CELLPADDING="2">
@@ -307,7 +307,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     <br>
     Filters are analyzed by HTTrack from the first filter to the last one. The sizes
     are compared against scan rules defined by the user.<br><br>
-    A scan rule has an higher priority is it is declared later - hierarchy is important.<br>
+    A scan rule has a higher priority if it is declared later - hierarchy is important.<br>
     
     Note: scan rules based on size can be mixed with regular URL patterns<br>
 
@@ -361,7 +361,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
     <br>
     Filters are analyzed by HTTrack from the first filter to the last one. The complete MIME
     type is compared against scan rules defined by the user.<br><br>
-    A scan rule has an higher priority is it is declared later - hierarchy is important<br>
+    A scan rule has a higher priority if it is declared later - hierarchy is important<br>
 
     Note: scan rules based on MIME types can <b>NOT</b> be mixed with regular URL patterns or size patterns within the same rule, but you can use both of them in distinct ones<br>
 
@@ -387,12 +387,12 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
       <tr>
         <td nowrap><tt>-mime:video/*</tt></td>
         <td>This will refuse all video links that were already scheduled for download
-        (i.e. all other 'application/' link download will be aborted)</td>
+        (i.e. the download will be aborted)</td>
       </tr>
       <tr>
         <td nowrap><tt>-mime:video/* -mime:audio/*</tt></td>
         <td>This will refuse all audio and video links that were already scheduled for download
-        (i.e. all other 'application/' link download will be aborted)</td>
+        (i.e. the download will be aborted)</td>
       </tr>
       <tr>
         <td nowrap><tt>-mime:*/* +mime:text/html +mime:image/*</tt></td>

html/plug.html

@@ -286,7 +286,7 @@ module exit point
 the function name and prototype MUST match this prototype
 */
 EXTERNAL_FUNCTION int hts_unplug(httrackp *opt) {
-  fprintf(stder, "Module unplugged");
+  fprintf(stderr, "Module unplugged");
 
   return 1;  /* success */
 }

src/htscharset.c

@@ -373,7 +373,8 @@ char *hts_convertStringCPFromUTF8(const char *s, size_t size, UINT cp) {
   return NULL;
 }
 
-char *hts_convertStringToUTF8(const char *s, size_t size, const char *charset) {
+HTSEXT_API char *hts_convertStringToUTF8(const char *s, size_t size,
+                                         const char *charset) {
   const UINT cp = hts_getCodepage(charset);
 
   return hts_convertStringCPToUTF8(s, size, cp);
@@ -554,7 +555,8 @@ static char *hts_convertStringCharset(const char *s, size_t size,
   return NULL;
 }
 
-char *hts_convertStringToUTF8(const char *s, size_t size, const char *charset) {
+HTSEXT_API char *hts_convertStringToUTF8(const char *s, size_t size,
+                                         const char *charset) {
   /* Empty string ? */
   if (size == 0) {
     return strdup("");

src/htscharset.h

@@ -51,8 +51,8 @@ typedef unsigned int hts_UCS4;
  * Convert the string "s" from charset "charset" to UTF-8.
  * Return NULL upon error.
  **/
-extern char *hts_convertStringToUTF8(const char *s, size_t size,
-                                     const char *charset);
+HTSEXT_API char *hts_convertStringToUTF8(const char *s, size_t size,
+                                         const char *charset);
 
 /**
  * Convert the string "s" from UTF-8 to charset "charset".

src/htscoremain.c

@@ -2368,30 +2368,28 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
       opt->errlog = stderr;
     } else if (httrack_logmode >= 2) {
       // deux fichiers log
-      structcheck(StringBuff(opt->path_log));
-      if (fexist
-          (fconcat
-           (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-log.txt")))
-        remove(fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-log.txt"));
-      if (fexist
-          (fconcat
-           (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-err.txt")))
-        remove(fconcat
-               (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-err.txt"));
+      // path_log holds UTF-8 bytes (argv is UTF-8): the ANSI file calls would
+      // read them as the codepage and drop the logs into a mangled twin (#630).
+      structcheck_utf8(StringBuff(opt->path_log));
+      if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                              StringBuff(opt->path_log), "hts-log.txt")))
+        UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                       StringBuff(opt->path_log), "hts-log.txt"));
+      if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                              StringBuff(opt->path_log), "hts-err.txt")))
+        UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                       StringBuff(opt->path_log), "hts-err.txt"));
 
       /* Check FS directory structure created */
-      structcheck(StringBuff(opt->path_log));
+      structcheck_utf8(StringBuff(opt->path_log));
 
-      opt->log =
-        fopen(fconcat
-              (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-log.txt"),
-              "w");
+      opt->log = FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                               StringBuff(opt->path_log), "hts-log.txt"),
+                       "w");
       if (httrack_logmode == 2)
-        opt->errlog =
-          fopen(fconcat
-                (OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-err.txt"),
-                "w");
+        opt->errlog = FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                                    StringBuff(opt->path_log), "hts-err.txt"),
+                            "w");
       else
         opt->errlog = opt->log;
       if (opt->log == NULL) {

src/htsselftest.c

@@ -2146,6 +2146,52 @@ static int st_cache_corrupt(httrackp *opt, int argc, char **argv) {
   return err;
 }
 
+/* Drives unzRepair over a damaged local file header whose CRC field's high
+   16-bit word has bit 15 set. Before the READ_32 fix that shifted an int and
+   overflowed, so UBSan aborts here; after it, repair recovers the one entry. */
+static int st_zip_repair_shift(httrackp *opt, int argc, char **argv) {
+  static const unsigned char zip[] = {
+      0x50, 0x4b, 0x03, 0x04, /* local file header signature */
+      0x14, 0x00,             /* version needed */
+      0x00, 0x00,             /* general purpose flag */
+      0x00, 0x00,             /* method */
+      0x00, 0x00,             /* time */
+      0x00, 0x00,             /* date */
+      0x00, 0x00, 0xe8, 0x8a, /* crc: high word 0x8ae8, bit 15 set */
+      0x00, 0x00, 0x00, 0x00, /* compressed size */
+      0x00, 0x00, 0x00, 0x00, /* uncompressed size */
+      0x01, 0x00,             /* filename length */
+      0x00, 0x00,             /* extra field length */
+      0x61                    /* filename "a" */
+  };
+  char in[HTS_URLMAXSIZE], out[HTS_URLMAXSIZE], tmp[HTS_URLMAXSIZE];
+  uLong nrec = 0, bytes = 0;
+  FILE *fp;
+  int err;
+
+  (void) opt;
+  if (argc < 1) {
+    fprintf(stderr, "zip-repair-shift: needs a directory\n");
+    return 1;
+  }
+  snprintf(in, sizeof(in), "%s/damaged.zip", argv[0]);
+  snprintf(out, sizeof(out), "%s/repair.zip", argv[0]);
+  snprintf(tmp, sizeof(tmp), "%s/repair.tmp", argv[0]);
+  fp = fopen(in, "wb");
+  if (fp == NULL || fwrite(zip, 1, sizeof(zip), fp) != sizeof(zip)) {
+    if (fp != NULL)
+      fclose(fp);
+    fprintf(stderr, "zip-repair-shift: cannot write %s\n", in);
+    return 1;
+  }
+  fclose(fp);
+  err = unzRepair(in, out, tmp, &nrec, &bytes);
+  printf("zip-repair-shift: %s (recovered %lu entr%s)\n",
+         (err == Z_OK && nrec == 1) ? "OK" : "FAIL", (unsigned long) nrec,
+         nrec == 1 ? "y" : "ies");
+  return (err == Z_OK && nrec == 1) ? 0 : 1;
+}
+
 static int st_cache_legacy(httrackp *opt, int argc, char **argv) {
   int err;
 
@@ -3253,6 +3299,9 @@ static const struct selftest_entry {
      st_cache_legacy},
     {"cache-corrupt", "<dir>", "cache read-side corruption self-test",
      st_cache_corrupt},
+    {"zip-repair-shift", "<dir>",
+     "cache zip-repair header read must not overflow a signed shift",
+     st_zip_repair_shift},
     {"dns", "", "DNS resolver/cache self-test", st_dns},
     {"dnstimeout", "", "a slow DNS resolve is bounded and holds no lock",
      st_dnstimeout},

src/htsserver.c

@@ -40,6 +40,7 @@ Please visit our Website: http://www.httrack.com
 
 #include "htsnet.h"
 #include "htslib.h"
+#include "htscharset.h"
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -749,7 +750,14 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
                            RUN THE SERVER
                          */
                         if (strcmp((char *) adrcd, "start") == 0) {
-                          webhttrack_main((char *) adr + p);
+                          /* POST body is in the form's charset, not the
+                             UTF-8 argv the engine now assumes (#629). */
+                          char *const cmdl = (char *) adr + p;
+                          char *cmdlUtf8 = hts_convertStringToUTF8(
+                              cmdl, strlen(cmdl), LANGSEL("LANGUAGE_CHARSET"));
+
+                          webhttrack_main(cmdlUtf8 != NULL ? cmdlUtf8 : cmdl);
+                          freet(cmdlUtf8);
                         } else {
                           commandRunning = 0;
                           commandEnd = 1;

src/htsweb.c

@@ -65,6 +65,7 @@ Please visit our Website: http://www.httrack.com
 #include "htsserver.h"
 #include "htsurlport.h"
 #include "htsweb.h"
+#include "htscharset.h"
 
 #if USE_BEGINTHREAD==0
 #error fatal: no threads support
@@ -156,6 +157,7 @@ int main(int argc, char *argv[]) {
   printf("Initializing the server..\n");
 
 #ifdef _WIN32
+  hts_argv_utf8(&argc, &argv);
   {
     WORD wVersionRequested;     // requested version WinSock API
     WSADATA wsadata;            // Windows Sockets API data

src/minizip/mztools.c

@@ -14,7 +14,7 @@
 
 #define READ_8(adr)  ((unsigned char)*(adr))
 #define READ_16(adr) ( READ_8(adr) | (READ_8(adr+1) << 8) )
-#define READ_32(adr) ( READ_16(adr) | (READ_16((adr)+2) << 16) )
+#define READ_32(adr) ((uLong) READ_16(adr) | ((uLong) READ_16((adr) + 2) << 16))
 
 #define WRITE_8(buff, n) do { \
   *((unsigned char*)(buff)) = (unsigned char) ((n) & 0xff); \

src/vcpkg.json

@@ -1,12 +1,12 @@
 {
-  "$schema":
-      "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
+  "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
   "name": "libhttrack",
   "version-string": "3.49",
+  "builtin-baseline": "cd61e1e26a038e82d6550a3ebbe0fbbfe7da78e3",
   "dependencies": [
     "brotli",
     "openssl",
     "zlib",
     "zstd"
   ]
-}
+}

tests/01_zlib-repair-shift.test

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Keep this POSIX-portable: the harness runs it via $(BASH), which is a plain
+# POSIX /bin/sh on some platforms (e.g. macOS), so avoid bashisms and GNU-only
+# tool flags despite the #!/bin/bash above.
+
+# unzRepair header read must not overflow a signed shift (-#test=zip-repair-shift
+# <dir>). A damaged local file header whose CRC high word has bit 15 set made
+# READ_32 shift an int past INT_MAX; UBSan aborts before the fix casts to uLong.
+
+set -eu
+
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+out=$(httrack -#test=zip-repair-shift "$dir")
+
+printf '%s\n' "$out" | grep -qx "zip-repair-shift: OK (recovered 1 entry)" || {
+    echo "expected 'zip-repair-shift: OK (recovered 1 entry)', got: $out" >&2
+    exit 1
+}

tests/68_webhttrack-outdir-charset.test

@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# webhttrack's POST body arrives in the form's declared charset, not UTF-8, yet
+# the engine now assumes UTF-8 argv. A non-ASCII -O output dir submitted through
+# the web UI must land under the UTF-8 directory, not an ISO-8859-1 twin (#629).
+# Drives the real htsserver over HTTP; the default (English) form is served
+# ISO-8859-1, so 'café' travels as the single byte 0xE9. The crawl target is a
+# dead port: only the -O directory name is under test, and htsserver creates it
+# from the decoded path before any fetch.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+python=$(find_python) || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_charset.XXXXXX") || fail "no tmpdir"
+srvlog=$(mktemp)
+srv=
+cleanup() {
+    # htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+    wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
+    rm -rf "${work}" "${srvlog}"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
+# ~/.httrack.ini out of it.
+sport=$("${python}" -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()')
+(
+    trap '' TERM TTOU
+    export HOME="${work}"
+    exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
+) &
+srv=$!
+for _ in $(seq 1 40); do
+    url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
+    kill -0 "${srv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
+
+# Post a "start" whose -O dir is 'café' in the form's ISO-8859-1 charset.
+"${python}" - "${url}" "${work}" <<'PY'
+import sys, urllib.parse, urllib.request
+
+url, work = sys.argv[1], sys.argv[2]
+outdir = work + "/caf\xe9"  # 'café' as the single byte the browser would send
+# Port 1 refuses at once: only the decoded -O dir matters, not the fetch.
+cmd = "httrack --quiet --robots=0 http://127.0.0.1:1/x.html -O " + outdir
+fields = [("path", work), ("projname", "proj"), ("command_do", "start"),
+          ("winprofile", "x"), ("command", cmd)]
+body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="", encoding="latin-1"))
+                for k, v in fields)
+req = urllib.request.Request(url + "step4.html", data=body.encode("latin-1"),
+                             method="POST")
+urllib.request.urlopen(req, timeout=20).read()
+PY
+
+# The crawl runs inside htsserver; wait for the -O dir, then check it chose the
+# UTF-8 name and not the ISO-8859-1 twin.
+utf8dir="${work}/café"
+latin1dir="${work}/caf"$'\xe9'
+for _ in $(seq 1 80); do
+    test -e "${utf8dir}" && break
+    kill -0 "${srv}" 2>/dev/null || break
+    sleep 0.25
+done
+test -e "${utf8dir}" || fail "-O dir never landed as UTF-8 café/: $(find "${work}" -maxdepth 2 2>/dev/null)"
+test ! -e "${latin1dir}" || fail "-O dir created as the ISO-8859-1 twin instead"
+
+echo "PASS"

tests/69_local-intl-logdir.test

@@ -0,0 +1,15 @@
+#!/bin/bash
+#
+# A single non-ASCII -O sets both path_html and path_log to "café" (#630). The
+# logs (hts-log.txt/hts-err.txt) must land there, not in an ANSI-mangled twin:
+# on Windows path_log holds UTF-8 bytes the raw file calls read as the codepage.
+# The --errors/--files/--log-found audits all grep logroot=café/hts-log.txt, so
+# a log written to the twin fails them. POSIX has no twin, so this bites on the
+# Windows CI leg (like test 64). The cache twin is a separate, larger fix.
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" --outdir-intl 'café' --errors 0 --files 5 \
+    --found 'simple/basic.html' \
+    --log-found 'mirror complete in' \
+    httrack 'BASEURL/simple/basic.html'

tests/71_local-crange-repaircache.test

@@ -0,0 +1,134 @@
+#!/bin/bash
+# #581: an interrupted mirror can leave a truncated new.zip that pass 2 must
+# repair before resuming. The repaired-cache resume then hits the same hostile
+# 206 as test 48, so restart-whole must still drop the partial and refetch the
+# whole file. Pass 1 leaves a partial + temp-ref; we truncate new.zip past its
+# last local entry (dropping the central directory) so unzOpen fails and the
+# repair path runs; pass 2 resumes, rejects the range, and refetches whole.
+set -u
+
+: "${top_srcdir:=..}"
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+server=$(nativepath "${testdir}/local-server.py")
+root=$(nativepath "${testdir}/server-root")
+
+python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
+
+# On Windows the pass-1 interrupt is a hard kill (MSYS can't signal a native
+# exe) and the restart-whole path fails on the repaired cache (#581) -- the very
+# bug this exercises; skip until the engine fix lands.
+if is_windows; then
+    echo "Windows: restart-whole fails on a repaired cache (#581), skipping"
+    exit 77
+fi
+
+tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_crangerep.XXXXXX") || exit 1
+serverpid=
+crawlpid=
+cleanup() {
+    test -n "$crawlpid" && kill -9 "$crawlpid" 2>/dev/null
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+serverlog="${tmpdir}/server.log"
+: >"$serverlog"
+"$python" "$server" --root "$root" >"$serverlog" 2>&1 &
+serverpid=$!
+port=
+for _ in $(seq 1 50); do
+    line=$(head -n1 "$serverlog" 2>/dev/null)
+    if test "${line%% *}" == "PORT"; then
+        port="${line#PORT }"
+        break
+    fi
+    kill -0 "$serverpid" 2>/dev/null || {
+        echo "server exited early: $(cat "$serverlog")"
+        exit 1
+    }
+    sleep 0.1
+done
+test -n "$port" || {
+    echo "could not discover server port"
+    exit 1
+}
+base="http://127.0.0.1:${port}"
+
+which httrack >/dev/null || {
+    echo "could not find httrack"
+    exit 1
+}
+out="${tmpdir}/crawl"
+mkdir "$out"
+common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --retries=1 -c1)
+refdir="${out}/hts-cache/ref"
+newzip="${out}/hts-cache/new.zip"
+
+# --- pass 1: crawl, interrupt once the blob download is underway -------------
+printf '[pass 1: interrupt mid-download] ..\t'
+httrack "${common[@]}" "${base}/crange206mem/index.html" >"${tmpdir}/log1" 2>&1 &
+crawlpid=$!
+for _ in $(seq 1 300); do
+    test -n "$(find "$refdir" -name '*.ref' 2>/dev/null)" && break
+    kill -0 "$crawlpid" 2>/dev/null || break
+    sleep 0.1
+done
+sleep 0.3
+kill -TERM "$crawlpid" 2>/dev/null
+wait "$crawlpid" 2>/dev/null
+crawlpid=
+test -n "$(find "$refdir" -name '*.ref' 2>/dev/null)" || {
+    echo "FAIL: no temp-ref survived pass 1; cannot drive the resume"
+    exit 1
+}
+echo "OK (temp-ref present)"
+
+# --- damage the cache: drop the central directory a hard kill never wrote ----
+printf '[damage new.zip -> forces repair] ..\t'
+"$python" - "$newzip" <<'PY' || exit 1
+import os, sys
+path = sys.argv[1]
+data = open(path, "rb").read()
+cut = data.find(b"PK\x01\x02")  # first central-directory header
+if cut <= 0:
+    sys.exit("no central directory to drop in %s" % path)
+os.truncate(path, cut)
+PY
+echo "OK"
+
+# --- pass 2: --continue -> repair -> resume -> hostile 206 -> refetch whole ---
+printf '[pass 2: repair, reject range, refetch] ..\t'
+rc=0
+httrack "${common[@]}" --continue "${base}/crange206mem/index.html" >"${tmpdir}/log2" 2>&1 || rc=$?
+test "$rc" -eq 0 || {
+    echo "FAIL: httrack exited $rc"
+    cat "${tmpdir}/log2" >&2
+    exit 1
+}
+echo "OK (terminated)"
+
+# The repair path must actually have run, else this is just a copy of test 48.
+printf '[cache repair fired] ..\t'
+grep -q 'damaged cache' "${out}/hts-log.txt" 2>/dev/null || {
+    echo "FAIL: repair path did not run; damage was ineffective"
+    exit 1
+}
+echo "OK"
+
+blob=$(find "$out" -name blob.bin 2>/dev/null | head -1)
+full=6008 # len(CRANGE206_BODY) = len("CR206DAT") + 6000
+
+printf '[file recovered whole] ..\t'
+test -s "$blob" || {
+    echo "FAIL: blob.bin missing after the repaired-cache resume"
+    exit 1
+}
+got=$(wc -c <"$blob")
+test "$got" -eq "$full" || {
+    echo "FAIL: blob.bin is ${got} bytes, expected ${full}"
+    exit 1
+}
+echo "OK (${got} bytes)"

tests/Makefile.am

@@ -84,6 +84,7 @@ TESTS = \
 	01_zlib-cache-legacy.test \
 	01_zlib-cache-golden.test \
 	01_zlib-cache-writefail.test \
+	01_zlib-repair-shift.test \
 	01_zlib-savename-cached.test \
 	02_manpage-regen.test \
 	02_update-cache.test \
@@ -148,6 +149,9 @@ TESTS = \
 	64_local-intl-outdir.test \
 	65_port-siblings.test \
 	66_engine-port80-strip.test \
-	67_engine-delayed-truncate.test
+	67_engine-delayed-truncate.test \
+	68_webhttrack-outdir-charset.test \
+	69_local-intl-logdir.test \
+	71_local-crange-repaircache.test
 
 CLEANFILES = check-network_sh.cache

tests/local-crawl.sh

@@ -48,6 +48,7 @@ key="${testdir}/server.key"
 tls=
 verbose=
 html_subdir=
+outdir_intl=
 rerun=
 rerun_args=
 rerun_dead=
@@ -140,6 +141,13 @@ while test "$pos" -lt "$nargs"; do
         pos=$((pos + 1))
         html_subdir="${args[$pos]}"
         ;;
+    --outdir-intl)
+        # Single non-ASCII -O "$out/NAME": path_html AND path_log are NAME, so
+        # the logs (and the harness reads of them) go through the non-ASCII path
+        # (#630). Distinct from --html-subdir, which keeps path_log ASCII.
+        pos=$((pos + 1))
+        outdir_intl="${args[$pos]}"
+        ;;
     --errors | --errors-content | --files)
         audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
         pos=$((pos + 1))
@@ -216,13 +224,18 @@ test -n "$ver" || die "could not run httrack"
 out="${tmpdir}/crawl"
 mkdir "$out" || die "could not create $out"
 # path_html holds the mirror + index; path_log holds hts-cache/hts-log.txt.
-# Default: both are "$out". With --html-subdir, path_html becomes "$out/NAME"
-# (the mirror root the audits inspect) while path_log stays "$out".
+# Default: both are "$out". --html-subdir moves path_html to "$out/NAME" while
+# path_log (logroot) stays "$out"; --outdir-intl moves both to "$out/NAME".
 mirrorroot="$out"
+logroot="$out"
 odir="$out"
 if test -n "$html_subdir"; then
     mirrorroot="${out}/${html_subdir}"
     odir="${mirrorroot},${out}"
+elif test -n "$outdir_intl"; then
+    mirrorroot="${out}/${outdir_intl}"
+    logroot="$mirrorroot"
+    odir="$mirrorroot"
 fi
 # Localhost is fast; disable the rate/bandwidth safety limits but keep a
 # max-time backstop so a hang cannot wedge the suite.
@@ -241,7 +254,7 @@ test "$crawlres" -eq 0 || ! result "httrack exited $crawlres" || {
     exit 1
 }
 result "OK"
-grep -iE "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt" >&2
+grep -iE "^[0-9:]*[[:space:]]Error:" "${logroot}/hts-log.txt" >&2
 
 # --- optional second pass: re-mirror into the same dir (cache/update path) ----
 if test -n "$rerun"; then
@@ -260,7 +273,7 @@ if test -n "$rerun"; then
     # The update summary reports "files updated"; a fresh crawl never does. Assert
     # it so a regression that bypasses the cache (re-crawls fresh) can't pass.
     info "checking update used the cache"
-    if grep -aqE "mirror complete in .*files updated" "${out}/hts-log.txt"; then
+    if grep -aqE "mirror complete in .*files updated" "${logroot}/hts-log.txt"; then
         result "OK"
     else
         result "update pass did not report cache activity"
@@ -292,7 +305,7 @@ if test -n "$rerun_dead"; then
     zip="${out}/hts-cache/new.zip"
     test -s "$zip" || die "no cache was written by the first pass"
     cp "$zip" "${tmpdir}/cache-before.zip"
-    cp "${out}/hts-log.txt" "${tmpdir}/log-before.txt"
+    cp "${logroot}/hts-log.txt" "${tmpdir}/log-before.txt"
     stop_server "$serverpid"
     serverpid=
     info "re-running httrack against the stopped server"
@@ -305,7 +318,7 @@ if test -n "$rerun_dead"; then
     # The dead pass must have gone through the no-data rollback, not bailed out
     # before the mirror loop (which would leave the cache trivially untouched).
     info "checking the dead pass hit the rollback"
-    if grep -aq "No data seems to have been transferred" "${out}/hts-log.txt"; then
+    if grep -aq "No data seems to have been transferred" "${logroot}/hts-log.txt"; then
         result "OK"
     else
         result "rollback notice not found in hts-log.txt"
@@ -320,7 +333,7 @@ if test -n "$rerun_dead"; then
         exit 1
     fi
     # Audits below describe the healthy crawl, not the dead pass.
-    cp "${tmpdir}/log-before.txt" "${out}/hts-log.txt"
+    cp "${tmpdir}/log-before.txt" "${logroot}/hts-log.txt"
 fi
 
 # --- discover the single host root (127.0.0.1_<port> or 127.0.0.1) -----------
@@ -350,19 +363,19 @@ while test "$i" -lt "${#audit[@]}"; do
     --errors)
         i=$((i + 1))
         assert_equals "checking errors" "${audit[$i]}" \
-            "$(grep -iEc "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt")"
+            "$(grep -iEc "^[0-9:]*[[:space:]]Error:" "${logroot}/hts-log.txt")"
         ;;
     --errors-content)
         i=$((i + 1))
-        total=$(grep -icE "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt")
+        total=$(grep -icE "^[0-9:]*[[:space:]]Error:" "${logroot}/hts-log.txt")
         # transient network failures (statuscode -2..-6) flake on busy loopback;
         # the code parens are followed by " at link" or " after N retries at link"
-        transient=$(grep -cE '\(-[2-6]\) (at link|after )' "${out}/hts-log.txt" || true)
+        transient=$(grep -cE '\(-[2-6]\) (at link|after )' "${logroot}/hts-log.txt" || true)
         assert_equals "checking content errors" "${audit[$i]}" "$((total - transient))"
         ;;
     --files)
         i=$((i + 1))
-        nFiles=$(grep -E "^HTTrack Website Copier/[^ ]* mirror complete in " "${out}/hts-log.txt" |
+        nFiles=$(grep -E "^HTTrack Website Copier/[^ ]* mirror complete in " "${logroot}/hts-log.txt" |
             sed -e 's/.*[[:space:]]\([^ ]*\)[[:space:]]files written.*/\1/g')
         assert_equals "checking files" "${audit[$i]}" "$nFiles"
         ;;
@@ -393,7 +406,7 @@ while test "$i" -lt "${#audit[@]}"; do
     --log-found)
         i=$((i + 1))
         info "checking log matches ${audit[$i]}"
-        if grep -aqE "${audit[$i]}" "${out}/hts-log.txt"; then result "OK"; else
+        if grep -aqE "${audit[$i]}" "${logroot}/hts-log.txt"; then result "OK"; else
             result "not in log"
             exit 1
         fi
@@ -401,7 +414,7 @@ while test "$i" -lt "${#audit[@]}"; do
     --log-not-found)
         i=$((i + 1))
         info "checking log lacks ${audit[$i]}"
-        if grep -aqE "${audit[$i]}" "${out}/hts-log.txt"; then
+        if grep -aqE "${audit[$i]}" "${logroot}/hts-log.txt"; then
             result "present in log"
             exit 1
         else result "OK"; fi