Fix SIGPIPE flake in 01_engine-xfread under pipefail

The test matched engine output with `echo "$out" | grep -q PATTERN`. Under
`set -o pipefail`, grep -q exits the moment it matches, so echo takes a SIGPIPE
writing the rest and the pipeline reports failure even though the pattern was
found. The `|| { echo FAIL; exit 1; }` guard then fired on a passing case,
turning it into an intermittent, output-size-dependent failure (seen on the
Debian buildd leg of #632).

Match with here-strings instead, dropping the pipe and the race entirely.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

configure.ac

@@ -1,6 +1,6 @@
 AC_PREREQ([2.71])
 
-AC_INIT([httrack], [3.49.12], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
+AC_INIT([httrack], [3.49.13], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
 AC_COPYRIGHT([
 HTTrack Website Copier, Offline Browser for Windows and Unix
 Copyright (C) 1998-2015 Xavier Roche and other contributors
@@ -29,11 +29,13 @@ AC_CONFIG_SRCDIR(src/httrack.c)
 AC_CONFIG_MACRO_DIR([m4])
 AC_CONFIG_HEADERS(config.h)
 AM_INIT_AUTOMAKE([subdir-objects])
-# 3:4:0: 3.49.12 tail-appends one field (why_url) to httrackp and otherwise only
-# deletes dead comments; no struct layout or exported signature broke vs 3.49.11,
-# so it stays soname .so.3; bump revision.
+# 3:5:0: 3.49.13 leaves every installed struct layout and the exported symbol set
+# untouched vs 3.49.12; the 2 GB (_stat64) and UTF-8 export widenings are _WIN32-only,
+# and INTsys widened on POSIX (#600) but sits in no installed struct. Stays soname
+# .so.3; bump revision. x32 alone changes layout (LLint was 4 bytes there, #524), and
+# is not a release architecture.
 # (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
-VERSION_INFO="3:4:0"
+VERSION_INFO="3:5:0"
 AM_MAINTAINER_MODE
 AC_USE_SYSTEM_EXTENSIONS
 

debian/changelog

@@ -1,3 +1,18 @@
+httrack (3.49.13-1) unstable; urgency=medium
+
+  * New upstream release: SOCKS5 and CONNECT proxy support, brotli and zstd
+    content decoding, --update data-loss fixes, and hardening of several
+    network-facing parsers; full list in history.txt.
+  * Add libbrotli-dev and libzstd-dev to Build-Depends for the new content
+    codings.
+  * libhttrack3 and libhttrack-dev no longer ship libhtsjava.so.3*: the
+    obsolete Java-applet parser was removed upstream. It was a dlopen plugin
+    with no reverse dependencies, so this is a plain file drop on upgrade;
+    libhttrack.so.3 itself is unchanged and needs no rename.
+  * Drop the htsjava attribution from debian/copyright.
+
+ -- Xavier Roche <xavier@debian.org>  Fri, 17 Jul 2026 10:49:52 +0200
+
 httrack (3.49.12-1) unstable; urgency=medium
 
   * New upstream release: security and crawl-correctness fixes (remote stack

history.txt

@@ -4,6 +4,31 @@ HTTrack Website Copier release history:
 
 This file lists all changes and fixes that have been made for HTTrack
 
+3.49-13
++ New: SOCKS5 proxy support, with scheme-aware -P URLs (socks5://, socks5h://, connect://) and plain HTTP tunneled through a CONNECT-only proxy (#563, #564)
++ New: decode brotli and zstd content codings, advertised over TLS only as browsers do (#556)
++ New: webhttrack exposes the engine options added since 3.49-2, among them --cookies-file, --pause and --strip-query (#587)
++ Fixed: files of 2 GB or more were mishandled on Windows and on every 32-bit build (#569)
++ Fixed: --update destroyed a good local copy when the re-fetch returned an HTTP error, was aborted by -M/-E, failed to decode, or came in short (#176, #521, #557, #562)
++ Fixed: a self-redirect cookie wall was dropped instead of being re-fetched with the cookie (#15)
++ Fixed: a stalled TLS handshake ignored --timeout, and synchronous DNS resolution could wedge a crawl past --max-time (#607, #613)
++ Fixed: -M metered saved bytes rather than received volume, and overshot its cap under a slow server (#77, #520)
++ Fixed: several network-facing overflows and denial-of-service paths in the Content-Range, chunked-transfer, cookie, filter and ProxyTrack cache parsers
++ Fixed: a failed connect did not fall back to the next address on Windows (#579)
++ Fixed: -P took an out-of-range port as a garbage port, and scanned past an IPv6 literal's closing bracket (#598, #602)
++ Fixed: reject a port outside 1..65535 wherever one is parsed (a crawled URL, the htsserver and proxytrack listen arguments, an ftp:// URL), instead of letting a bare sscanf wrap a huge value into a plausible port and silently use it (#614, #626)
++ Fixed: a configured proxy still resolved and dialed the origin itself (#592)
++ Fixed: ~/ in the -O base path was never expanded (#270)
++ Fixed: a non-ASCII -O output path was double-encoded on Windows once argv became UTF-8 (#621)
++ Fixed: files under a non-ASCII project path were saved to a mangled directory on Windows (#217)
++ Fixed: --build-top-index (-%i) and --protocol (-@i) were taken for the -i continue flag, wiping the URL list and exiting on the usage screen (#615)
++ Fixed: webhttrack ignored LC_ALL/LC_MESSAGES and picked the wrong Chinese and Portuguese (#95)
++ Fixed: webhttrack wrote its base path and httrack.ini to the filesystem root when $HOME was empty (#625)
++ Fixed: crawls on a non-default port were slowed by a per-request pre-resolve (#181)
++ Changed: Windows builds moved to Visual Studio 2022 and OpenSSL 3.x, the VS2008 project files are retired, and the binaries carry a version resource
++ Changed: removed the obsolete Java-applet (.class) parser and the dead SWF module remnants
++ Changed: multiple internal hardening, test and CI improvements (Windows and macOS crawl suites, HTML-parser fuzzing, parallel make check)
+
 3.49-12
 + New: --why explains which filter rule accepts or rejects a given URL, then exits (#505)
 + Fixed: links carrying raw UTF-8 bytes were fetched double-encoded and 404'd (#516)

lang.indexes

@@ -23,7 +23,7 @@ pt:7
 ro:25
 ru:8
 sk:20
-si:24
+sl:24
 sv:17
 tr:10
 uk:22

lang/Finnish.txt

@@ -930,7 +930,6 @@ Server terminated
 Palvelin lopetettu
 A fatal error has occurred during this mirror
 Tällä peilillä tapahtui vakava virhe
-
 Proxy type:
 Välityspalvelimen tyyppi:
 Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.

lang/Italiano.txt

@@ -374,7 +374,7 @@ Create error logging and report files
 Crea file di log per segnalare errori e informazioni
 Generate DOS 8-3 filenames ONLY
 Genera solo nomi di file in formato 8.3
-Generate ISO 9660 filenames ONLY for CDROM medias
+Generate ISO9660 filenames ONLY for CDROM medias
 Genera nomi di file in formato ISO9660 per i CDROM
 Do not create HTML error pages
 Non generare pagine d'errore HTML

lang/Portugues-Brasil.txt

@@ -918,7 +918,7 @@ normal\nextended\ndebug
 normal\nextendido\ncorrigir
 Download web site(s)\nDownload web site(s) + questions\nGet individual files\nDownload all sites in pages (multiple mirror)\nTest links in pages (bookmark test)\n* Continue interrupted download\n* Update existing download
 Copiar site(s) da Web\nCopiar site(s) interativos da Web (perguntas)\nReceber arquivos específicos\nCopiar todas as páginas do site (alternação múltipla)\nTestar links nas páginas (testar indicador)\n* Retomar download interrompido\n* Atualizar download existente
-Relative URL / Absolute URL (default)\nAbsolute URL / Absolute URL\nAbsolute URI / Absolute URL\nOriginal URL / Original URL
+Relative URI / Absolute URL (default)\nAbsolute URL / Absolute URL\nAbsolute URI / Absolute URL\nOriginal URL / Original URL
 URI Relativa / URL Absoluta (padrão)\nURL Absoluta / URL Absoluta\nURI Absoluta / URL Absoluta\nURL Original / URL Original
 Open Source offline browser
 Abrir origem offline no navegador

lang/Slovenian.txt

@@ -3,7 +3,7 @@ Slovenian
 LANGUAGE_FILE
 Slovenian
 LANGUAGE_ISO
-si
+sl
 LANGUAGE_AUTHOR
 Jadran Rudec,iur.\r\njrudec@email.si \r\n
 LANGUAGE_CHARSET

src/Makefile.am

@@ -46,9 +46,11 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
 
 lib_LTLIBRARIES = libhttrack.la
 
-htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h
+htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
+	htsurlport.c htsurlport.h
 proxytrack_SOURCES = proxy/main.c \
 	proxy/proxytrack.c proxy/store.c \
+	htsurlport.c htsurlport.h \
 	coucal/coucal.c htsmd5.c md5.c \
 	minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c
 
@@ -58,7 +60,7 @@ whttrackrun_SCRIPTS = webhttrack
 libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htscache_selftest.c htsdns_selftest.c htsselftest.c \
 	htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
-	htshelp.c htslib.c htscoremain.c \
+	htshelp.c htslib.c htsurlport.c htscoremain.c \
 	htsname.c htsrobots.c htstools.c htswizard.c \
 	htsalias.c htsthread.c htsindex.c htsbauth.c \
 	htsmd5.c htscodec.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
@@ -69,7 +71,7 @@ libhttrack_la_SOURCES =  htscore.c htsparse.c htsback.c htscache.c \
 	htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h  \
 	htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h  \
 	htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
-	htshelp.h htsindex.h htslib.h htsmd5.h \
+	htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
 	htsmodules.h htsname.h htsnet.h htssniff.h \
 	htsopt.h htsrobots.h htsthread.h  \
 	htstools.h htswizard.h htswrap.h htscodec.h htsproxy.h htszlib.h  \

src/htsalias.c

@@ -567,24 +567,29 @@ int optinclude_file(const char *name, int *argc, char **argv, char *x_argvblk,
   return 0;
 }
 
-/* Get home directory, '.' if failed */
+/* Get home directory, '.' if unset or empty */
 /* example: /home/smith */
 const char *hts_gethome(void) {
   const char *home = getenv("HOME");
 
-  if (home)
-    return home;
-  else
-    return ".";
+  /* An empty $HOME would expand ~/foo into the absolute /foo */
+  return strnotempty(home) ? home : ".";
 }
 
-/* Convert ~/foo into /home/smith/foo */
+/* Convert ~/foo into /home/smith/foo (~user/ left alone: no getpwnam here) */
 void expand_home(String * str) {
-  if (StringSub(*str, 1) == '~') {
+  if (StringNotEmpty(*str) && StringSub(*str, 0) == '~' &&
+      (StringLength(*str) == 1 || StringSub(*str, 1) == '/')) {
     char BIGSTK tempo[HTS_URLMAXSIZE * 2];
+    const char *const home = hts_gethome();
+    const size_t homelen = strlen(home);
+    const size_t taillen = StringLength(*str) - 1;
 
-    strcpybuff(tempo, hts_gethome());
-    strcatbuff(tempo, StringBuff(*str) + 1);
-    StringCopy(*str, tempo);
+    /* Leave untouched rather than abort() in strcatbuff on a huge $HOME */
+    if (taillen < sizeof(tempo) && homelen < sizeof(tempo) - taillen) {
+      strcpybuff(tempo, home);
+      strcatbuff(tempo, StringBuff(*str) + 1);
+      StringCopy(*str, tempo);
+    }
   }
 }

src/htscoremain.c

@@ -425,25 +425,10 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
 
       }                         // for
 
-      // Convert path to UTF-8
-#ifdef _WIN32
-      {
-        char *const path =
-          hts_convertStringSystemToUTF8(StringBuff(opt->path_html),
-                                        (int) StringLength(opt->path_html));
-        if (path != NULL) {
-          StringCopy(opt->path_html_utf8, path);
-          free(path);
-        } else {
-          StringCopyN(opt->path_html_utf8, StringBuff(opt->path_html),
-                      StringLength(opt->path_html));
-        }
-      }
-#else
-      // Assume UTF-8 filesystem.
+      // path_html is already UTF-8 (argv is UTF-8 on Windows via
+      // hts_argv_utf8), so no re-encoding.
       StringCopyN(opt->path_html_utf8, StringBuff(opt->path_html),
                   StringLength(opt->path_html));
-#endif
 
       /* if doit.log exists, or if new URL(s) defined, 
          then DO NOT load standard config files */
@@ -2486,9 +2471,12 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
          }
          } */
 
-      // vérifier existence de la structure
-      structcheck(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_html), "/"));
-      structcheck(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "/"));
+      // vérifier existence de la structure (path_html/path_log are UTF-8, use
+      // the UTF-8 mkdir path)
+      structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                               StringBuff(opt->path_html), "/"));
+      structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
+                               StringBuff(opt->path_log), "/"));
 
       // reprise/update
       if (opt->cache) {

src/htsdns_selftest.c

@@ -402,6 +402,50 @@ int dns_selftests(httrackp *opt) {
       deletesoc(s);
   }
 
+  /* A URL port outside 1..65535 must refuse the link, not fold into range and
+     connect elsewhere (#614). *addr_count discriminates: 0 only if refused
+     before the resolve, still 2 for one merely truncated or defaulted. */
+  {
+    /* an empty "dual.test:" means the default port (WHATWG, curl): keep it */
+    static const char *const good[] = {"dual.test:1",    "dual.test:80",
+                                       "dual.test:8080", "dual.test:65535",
+                                       "dual.test:080",  "dual.test:"};
+    /* 65616 and 4294967376 are load-bearing: both wrap to a plausible 80 */
+    static const char *const bad[] = {
+        "dual.test:0",          "dual.test:65536",      "dual.test:65616",
+        "dual.test:99999",      "dual.test:2147483648", "dual.test:4294967296",
+        "dual.test:4294967376", "dual.test:-1",         "dual.test:-23437",
+        "dual.test:80x",        "dual.test:+80",        "dual.test: 80",
+        "dual.test:0x50"};
+    size_t k;
+
+    for (k = 0; k < sizeof(good) / sizeof(good[0]); k++) {
+      htsblk r;
+      int count = -1;
+      T_SOC s;
+
+      hts_init_htsblk(&r);
+      s = newhttp_addr(opt, good[k], &r, -1, 0, 0, &count);
+      CHECK(count == 2); /* accepted: reached the resolve */
+      if (s != INVALID_SOCKET)
+        deletesoc(s);
+    }
+
+    for (k = 0; k < sizeof(bad) / sizeof(bad[0]); k++) {
+      htsblk r;
+      int count = -1;
+      T_SOC s;
+
+      hts_init_htsblk(&r);
+      s = newhttp_addr(opt, bad[k], &r, -1, 0, 0, &count);
+      CHECK(s == INVALID_SOCKET);
+      CHECK(count == 0); /* refused before resolving, not a failed connect */
+      CHECK(strstr(r.msg, "Invalid port") != NULL);
+      if (s != INVALID_SOCKET)
+        deletesoc(s);
+    }
+  }
+
   /* Connect-fallback decision (consumer of the multi-address list): when a
      stuck connect should abandon the current address for the next one. */
   {

src/htsftp.c

@@ -248,7 +248,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
     // port
     a = strchr(adr, ':');       // port
     if (a) {
-      sscanf(a + 1, "%d", &port);
+      // folding a nonsense port into 1..65535 fetches one the link never named;
+      // an empty "host:" just means the default (#614)
+      if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
+        snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
+        back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
+        _HALT_FTP return 0;
+      }
       strncatbuff(_adr, adr, (int) (a - adr));
     } else
       strcpybuff(_adr, adr);

src/htsglobal.h

@@ -43,8 +43,8 @@ Please visit our Website: http://www.httrack.com
    configure.ac, decoupled from these). VERSION is the display form, VERSIONID
    the dotted numeric form, AFF_VERSION the short form shown in footers,
    LIB_VERSION the data/cache format generation. */
-#define HTTRACK_VERSION "3.49-12"
-#define HTTRACK_VERSIONID "3.49.12"
+#define HTTRACK_VERSION "3.49-13"
+#define HTTRACK_VERSIONID "3.49.13"
 #define HTTRACK_AFF_VERSION "3.x"
 #define HTTRACK_LIB_VERSION "2.0"
 

src/htslib.c

@@ -2167,15 +2167,18 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
 #endif
 
       if (a != NULL) {
-        int i = -1;
-
-        iadr2[0] = '\0';
-        sscanf(a + 1, "%d", &i);
-        if (i != -1) {
-          port = (unsigned short int) i;
+        // folding a nonsense port into 0..65535 crawls one neither the link nor
+        // a port filter named; an empty "host:" just means the default (#614)
+        if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
+          if (retour != NULL) {
+            snprintf(retour->msg, sizeof(retour->msg), "Invalid port: %s",
+                     a + 1);
+          }
+          return INVALID_SOCKET;
         }
 
-        // adresse véritable (sans :xx)
+        iadr2[0] = '\0';
+        // the address itself, without the ":port"
         strncatbuff(iadr2, iadr, (int) (a - iadr));
         resolve_host = iadr2;
       }
@@ -3751,15 +3754,11 @@ static int proxy_default_port(const char *arg) {
 // port "a" of -P argument "arg": digits fitting TCP's 1..65535, else the scheme
 // default. Not sscanf("%d"): past INT_MAX it wraps to a garbage port (#602)
 static int parse_proxy_port(const char *a, const char *arg) {
-  char *end;
-  long p;
+  int port;
 
-  if (!isdigit((unsigned char) *a)) // strtol would eat a sign or leading space
+  if (!hts_parse_url_port(a, &port))
     return proxy_default_port(arg);
-  p = strtol(a, &end, 10);
-  if (*end != '\0' || p < 1 || p > 65535) // ERANGE lands out of range too
-    return proxy_default_port(arg);
-  return (int) p;
+  return port;
 }
 
 void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port) {

src/htslib.h

@@ -61,6 +61,7 @@ typedef struct lien_adrfilsave lien_adrfilsave;
 
 /* définitions globales */
 #include "htsglobal.h"
+#include "htsurlport.h"
 
 /* basic net definitions */
 #include "htsbase.h"

src/htsproxy.c

@@ -491,14 +491,11 @@ static int socks5_handshake_stream(httrackp *opt, socks5_stream *st,
     if ((unsigned char) host[i] < ' ')
       return socks5_fail(msg, msgsize, "SOCKS5: invalid origin host");
   }
-  if (portsep != NULL) {
-    int p = -1;
-
-    sscanf(portsep + 1, "%d", &p);
-    if (p <= 0 || p > 65535)
-      return socks5_fail(msg, msgsize, "SOCKS5: invalid origin port");
-    port = p;
-  }
+  // the old range check ran after sscanf("%d") had wrapped a huge value into a
+  // plausible port (#614). An empty "host:" stays refused here, unlike the
+  // direct path, as it was before #614.
+  if (portsep != NULL && !hts_parse_url_port(portsep + 1, &port))
+    return socks5_fail(msg, msgsize, "SOCKS5: invalid origin port");
   if (link_has_authorization(proxy_name)) {
     if (!socks5_credentials(opt, proxy_name, user, sizeof(user), &userlen, pass,
                             sizeof(pass), &passlen, msg, msgsize))

src/htsselftest.c

@@ -45,6 +45,7 @@ Please visit our Website: http://www.httrack.com
 #include "htscore.h"
 #include "htsdefines.h"
 #include "htslib.h"
+#include "htsalias.h"
 #include "htsparse.h"
 #include "htscache.h"
 #include "htscache_selftest.h"
@@ -825,6 +826,21 @@ static int st_simplify(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+static int st_expandhome(httrackp *opt, int argc, char **argv) {
+  String path = STRING_EMPTY;
+
+  (void) opt;
+  if (argc < 1) {
+    fprintf(stderr, "expandhome: needs a path\n");
+    return 1;
+  }
+  StringCopy(path, argv[0]);
+  expand_home(&path);
+  printf("expanded=%s\n", StringBuff(path));
+  StringFree(path);
+  return 0;
+}
+
 static int st_mime(httrackp *opt, int argc, char **argv) {
   char mime[256];
 
@@ -1453,6 +1469,25 @@ static int st_socks5(httrackp *opt, int argc, char **argv) {
   assertf(socks5_handshake_scripted(opt, "origin.test:8443", proxy, &io) == 1);
   assertf(memcmp(io.sent + io.sent_len - 2, "\x20\xfb", 2) == 0);
 
+  /* a bad origin port is refused before any byte goes out (#614). 4294967376 is
+     the case the old range check could not see: it overflowed the sscanf("%d")
+     into a plausible 80 and passed. 65616 would not prove anything here, since
+     it fits an int and the old check already caught it. */
+  {
+    static const char *const bad[] = {"origin.test:4294967376",
+                                      "origin.test:80x", "origin.test:+80",
+                                      "origin.test: 80", "origin.test:8.0"};
+    size_t k;
+
+    for (k = 0; k < sizeof(bad) / sizeof(bad[0]); k++) {
+      len = socks5_reply(script, 0x01, v4, sizeof(v4));
+      io.reply = script;
+      io.reply_len = len;
+      assertf(socks5_handshake_scripted(opt, bad[k], proxy, &io) == 0);
+      assertf(io.sent_len == 0);
+    }
+  }
+
   /* credentials: split on the first colon of the escaped userinfo, so %3a stays
      inside the username and a colon in the password is not a delimiter */
   {
@@ -2442,6 +2477,45 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+// hts_buildtopindex takes a system-charset path but verif_backblue() below it
+// expects utf-8, so on Windows a non-ASCII project dir gets the gifs written to
+// a mangled twin (issues #216/#217). argv[0] is a writable dir.
+static int st_topindex(httrackp *opt, int argc, char **argv) {
+  char projdir[HTS_URLMAXSIZE];
+  char path[HTS_URLMAXSIZE + 16]; /* projdir plus a basename */
+#ifdef _WIN32
+  /* the GUI hands hts_buildtopindex an ANSI path; mimic it. CP1252 'cafe' */
+  static const char *const projName = "caf\xE9";
+#else
+  /* POSIX system charset is UTF-8 */
+  static const char *const projName = "caf\xC3\xA9";
+#endif
+
+  assertf(argc >= 1);
+  snprintf(projdir, sizeof(projdir), "%s/%s", argv[0], projName);
+
+  /* structcheck(), not the utf-8 MKDIR family: same charset as buildtopindex */
+  snprintf(path, sizeof(path), "%s/", projdir);
+  assertf(structcheck(path) == 0);
+
+  /* returns 0 here: the dir holds no sub-project, only the gifs matter */
+  (void) hts_buildtopindex(opt, projdir, "");
+
+  /* the gifs must land in the project dir itself, not in a mangled sibling */
+  snprintf(path, sizeof(path), "%s/backblue.gif", projdir);
+  assertf(fexist(path));
+
+  /* raw unlink/rmdir: UNLINK is utf-8 on Windows, these paths aren't */
+  unlink(path);
+  snprintf(path, sizeof(path), "%s/fade.gif", projdir);
+  unlink(path);
+  snprintf(path, sizeof(path), "%s/index.html", projdir);
+  unlink(path);
+  rmdir(projdir);
+  printf("topindex self-test OK\n");
+  return 0;
+}
+
 /* Each inplace_escape_*() must equal escape_*() on a copy. */
 static int st_inplace_escape(httrackp *opt, int argc, char **argv) {
   /* >255 bytes forces the helper's malloct path, not the stack buffer */
@@ -3079,6 +3153,7 @@ static const struct selftest_entry {
     {"filterbounds", "", "matcher length/work caps reject hostile patterns",
      st_filterbounds},
     {"simplify", "<path>", "collapse ./ and ../ in a path", st_simplify},
+    {"expandhome", "<path>", "expand a leading ~/ into $HOME", st_expandhome},
     {"stripquery", "", "--strip-query pattern/key stripping self-test",
      st_stripquery},
     {"urlhack", "", "-%u url-hack sub-flag (www/slash/query) self-test",
@@ -3148,6 +3223,9 @@ static const struct selftest_entry {
     {"useragent", "", "default User-Agent self-test", st_useragent},
     {"makeindex", "[dir]", "hts_finish_makeindex footer/refresh self-test",
      st_makeindex},
+    {"topindex", "[dir]",
+     "hts_buildtopindex charset handling of a non-ASCII project dir",
+     st_topindex},
     {"inplace-escape", "", "inplace_escape_* vs escape_* equivalence self-test",
      st_inplace_escape},
     {"escape-room", "", "HT_ADD_HTMLESCAPED* reservation-factor self-test",

src/htsserver.h

@@ -164,13 +164,12 @@ HTS_UNUSED static int linputsoc_t(T_SOC soc, char *s, int max, int timeout) {
   return -1;
 }
 
+/* Same contract as hts_gethome(), which is hidden and out of reach from here */
 static const char *gethomedir(void) {
   const char *home = getenv("HOME");
 
-  if (home)
-    return home;
-  else
-    return ".";
+  /* An empty $HOME would put the base path and httrack.ini at the root */
+  return strnotempty(home) ? home : ".";
 }
 static int linput_cpp(FILE * fp, char *s, int max) {
   int rlen = 0;

src/htstools.c

@@ -896,7 +896,20 @@ HTSEXT_API int hts_buildtopindex(httrackp * opt, const char *path,
     if (fpo) {
       find_handle h;
 
-      verif_backblue(opt, concat(catbuff, sizeof(catbuff), rpath, "/")); // générer gif
+      // générer gif. verif_backblue() is utf-8, but our path is the system
+      // charset, so on Windows convert it or the gifs land in a mangled twin
+      // dir (#217). Elsewhere the system charset is already utf-8.
+#ifdef _WIN32
+      {
+        const char *const base = concat(catbuff, sizeof(catbuff), rpath, "/");
+        char *const base_utf8 =
+            hts_convertStringSystemToUTF8(base, strlen(base));
+        verif_backblue(opt, base_utf8 != NULL ? base_utf8 : base);
+        free(base_utf8);
+      }
+#else
+      verif_backblue(opt, concat(catbuff, sizeof(catbuff), rpath, "/"));
+#endif
       // Header
       hts_template_format(fpo, toptemplate_header,
               "<!-- Mirror and index made by HTTrack Website Copier/"

src/htsurlport.c

@@ -0,0 +1,50 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: TCP port parser, shared by the engine, htsserver and    */
+/*       proxytrack                                              */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#include "htsurlport.h"
+
+#include <ctype.h>
+#include <stdlib.h>
+
+hts_boolean hts_parse_url_port(const char *a, int *port) {
+  char *end;
+  long p;
+
+  if (!isdigit((unsigned char) *a)) // strtol would eat a sign or leading space
+    return HTS_FALSE;
+  p = strtol(a, &end, 10);
+  if (*end != '\0' || p < 1 || p > 65535) // ERANGE lands out of range too
+    return HTS_FALSE;
+  *port = (int) p;
+  return HTS_TRUE;
+}

src/htsurlport.h

@@ -0,0 +1,45 @@
+/* ------------------------------------------------------------ */
+/*
+HTTrack Website Copier, Offline Browser for Windows and Unix
+Copyright (C) 1998 Xavier Roche and other contributors
+
+SPDX-License-Identifier: GPL-3.0-or-later
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+Ethical use: we kindly ask that you NOT use this software to harvest email
+addresses or to collect any other private information about people. Doing so
+would dishonor our work and waste the many hours we have spent on it.
+
+Please visit our Website: http://www.httrack.com
+*/
+
+/* ------------------------------------------------------------ */
+/* File: TCP port parser, shared by the engine, htsserver and    */
+/*       proxytrack                                              */
+/* Author: Xavier Roche                                          */
+/* ------------------------------------------------------------ */
+
+#ifndef HTSURLPORT_DEFH
+#define HTSURLPORT_DEFH
+
+#include "htsglobal.h"
+
+/* Parse the port text "a" (after the ':', up to the end of the string): TRUE
+   and *port set for a bare decimal in 1..65535, else FALSE and *port left
+   alone. Not sscanf("%d"), which range-checks nothing and wraps past INT_MAX.
+   Its own file so proxytrack, which does not link the library, can share it. */
+hts_boolean hts_parse_url_port(const char *a, int *port);
+
+#endif

src/htsweb.c

@@ -63,6 +63,7 @@ Please visit our Website: http://www.httrack.com
 #include "md5.c"
 
 #include "htsserver.h"
+#include "htsurlport.h"
 #include "htsweb.h"
 
 #if USE_BEGINTHREAD==0
@@ -257,8 +258,10 @@ int main(int argc, char *argv[]) {
   /* set commandline keys */
   for(i = 2; i < argc; i += 2) {
     if (strcmp(argv[i], "--port") == 0 && i + 1 < argc) {
-      if (sscanf(argv[i + 1], "%d", &defaultPort) != 1 || defaultPort < 0
-          || defaultPort >= 65535) {
+      // the range check ran after sscanf("%d") had wrapped a huge value into a
+      // plausible port, and listened there (#614). 0 (was the auto-pick) and
+      // 65535 (was refused, off by one) now both mean what they say.
+      if (!hts_parse_url_port(argv[i + 1], &defaultPort)) {
         fprintf(stderr, "couldn't set the port number to %s\n", argv[i + 1]);
         return -1;
       }

src/libhttrack.vcxproj

@@ -121,6 +121,7 @@
     <ClCompile Include="htshelp.c" />
     <ClCompile Include="htsindex.c" />
     <ClCompile Include="htslib.c" />
+    <ClCompile Include="htsurlport.c" />
     <ClCompile Include="htsmd5.c" />
     <ClCompile Include="htsmodules.c" />
     <ClCompile Include="htsname.c" />

src/proxy/main.c

@@ -47,7 +47,9 @@ static void sig_brpipe(int code) {
 }
 #endif
 
-static int scanHostPort(const char *str, char *host, int *port) {
+// split a "host:port" listen argument; FALSE sends the caller to the usage
+// screen. The port was unchecked, so a huge one wrapped into range (#614).
+static hts_boolean scanHostPort(const char *str, char *host, int *port) {
   char *pos = strrchr(str, ':');
 
   if (pos != NULL) {
@@ -56,12 +58,10 @@ static int scanHostPort(const char *str, char *host, int *port) {
     if (n < 256) {
       host[0] = '\0';
       strncat(host, str, n);
-      if (sscanf(pos + 1, "%d", port) == 1) {
-        return 1;
-      }
+      return hts_parse_url_port(pos + 1, port);
     }
   }
-  return 0;
+  return HTS_FALSE;
 }
 
 int main(int argc, char *argv[]) {

src/proxy/proxytrack.h

@@ -107,13 +107,12 @@ HTS_UNUSED static void proxytrack_print_log(const char *severity, const char *fo
 	"<!-- _-._.--._._-._.--._._-._.--._._-._.--._._-._.--._. -->\r\n" \
 	"<!-- End Disable IE Friendly HTTP Error Messages -->\r\n"
 
+/* Same contract as hts_gethome(): proxytrack does not link libhttrack */
 HTS_UNUSED static const char *gethomedir(void) {
   const char *home = getenv("HOME");
 
-  if (home)
-    return home;
-  else
-    return ".";
+  /* An empty $HOME would resolve a relative path against the root */
+  return strnotempty(home) ? home : ".";
 }
 
 HTS_UNUSED static int linput(FILE * fp, char *s, int max) {

src/proxytrack.vcxproj

@@ -99,6 +99,7 @@
     <ClCompile Include="proxy\main.c" />
     <ClCompile Include="proxy\proxytrack.c" />
     <ClCompile Include="proxy\store.c" />
+    <ClCompile Include="htsurlport.c" />
     <ClCompile Include="coucal\coucal.c" />
     <ClCompile Include="htsmd5.c" />
     <ClCompile Include="md5.c" />

src/version.rc

@@ -17,8 +17,8 @@
 #endif
 
 VS_VERSION_INFO VERSIONINFO
-  FILEVERSION 3, 49, 12, 0
-  PRODUCTVERSION 3, 49, 12, 0
+  FILEVERSION 3, 49, 13, 0
+  PRODUCTVERSION 3, 49, 13, 0
   FILEFLAGSMASK VS_FFI_FILEFLAGSMASK
 #ifdef _DEBUG
   FILEFLAGS VS_FF_DEBUG
@@ -35,12 +35,12 @@ BEGIN
     BEGIN
       VALUE "CompanyName", "Xavier Roche"
       VALUE "FileDescription", VER_FILE_DESCRIPTION
-      VALUE "FileVersion", "3.49.12"
+      VALUE "FileVersion", "3.49.13"
       VALUE "InternalName", VER_ORIGINAL_FILENAME
       VALUE "LegalCopyright", "Copyright (C) 1998-2026 Xavier Roche and other contributors. GNU GPL v3 or later."
       VALUE "OriginalFilename", VER_ORIGINAL_FILENAME
       VALUE "ProductName", "HTTrack Website Copier"
-      VALUE "ProductVersion", "3.49-12"
+      VALUE "ProductVersion", "3.49-13"
     END
   END
   BLOCK "VarFileInfo"

src/webhttrack

@@ -33,6 +33,19 @@ function log {
     return 0
 }
 
+# Map a POSIX locale ("zh_TW.UTF-8@euro") to its lang.indexes number, English (1) if unknown
+function lang_index {
+    local locale=$1 indexes=$2 tag n t
+    tag=$(echo "${locale}" | cut -f1 -d'.' | cut -f1 -d'@' | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z_')
+    # a few languages carry a region-specific entry (zh_tw, pt_br); else use the bare language
+    for t in "${tag}" "${tag%%_*}"; do
+        n=$(grep -E "^${t}:" "${indexes}" | cut -f2 -d':')
+        test -n "${n}" && break
+    done
+    test -n "${n}" || n=1
+    echo "${n}"
+}
+
 function launch_browser {
     log "Launching $1"
     browser=$1
@@ -59,13 +72,11 @@ test -f "${DISTPATH}/lang.indexes" || ! log "Could not find ${DISTPATH}/lang.ind
 test -d "${DISTPATH}/lang" || ! log "Could not find ${DISTPATH}/lang" || exit 1
 test -d "${DISTPATH}/html" || ! log "Could not find ${DISTPATH}/html" || exit 1
 
-# Locale
-HTSLANG="${LC_MESSAGES}"
-! test -n "${HTSLANG}" && HTSLANG="${LC_ALL}"
+# Locale: POSIX precedence, LC_ALL overrides LC_MESSAGES overrides LANG
+HTSLANG="${LC_ALL}"
+! test -n "${HTSLANG}" && HTSLANG="${LC_MESSAGES}"
 ! test -n "${HTSLANG}" && HTSLANG="${LANG}"
-HTSLANG="$(echo "$LANG" | cut -f1 -d'.' | cut -f1 -d'_')"
-LANGN=$(grep -E "^${HTSLANG}:" "${DISTPATH}/lang.indexes" | cut -f2 -d':')
-! test -n "${LANGN}" && LANGN=1
+LANGN=$(lang_index "${HTSLANG}" "${DISTPATH}/lang.indexes")
 
 # Find the browser
 # note: not all systems have sensible-browser or www-browser alternative

src/webhttrack.vcxproj

@@ -97,6 +97,7 @@
   <ItemGroup>
     <ClCompile Include="htsserver.c" />
     <ClCompile Include="htsweb.c" />
+    <ClCompile Include="htsurlport.c" />
   </ItemGroup>
 
   <ItemGroup>

tests/01_engine-expandhome.test

@@ -0,0 +1,53 @@
+#!/bin/bash
+#
+
+# SC2088: the literal, unexpanded '~' is the input under test.
+# shellcheck disable=SC2088
+
+set -euo pipefail
+
+# ~ expansion for the -O base path (expand_home). $HOME is pinned so the cases
+# cannot depend on the caller's, but MSYS rewrites it into a Windows path
+# before the native httrack.exe reads it: ask the engine what it saw rather
+# than hardcoding the value.
+ask() {
+    HOME=HTSHOME httrack -O /dev/null -#test=expandhome "$1"
+}
+exp() {
+    test "$(ask "$1")" == "expanded=$2" || exit 1
+}
+
+home=$(ask '~')
+home=${home#expanded=}
+
+# or every case below is vacuous: '~' means expansion never fired, '.' means it
+# fell back to the default without ever reading $HOME
+test "$home" != '~' || exit 1
+test "$home" != '.' || exit 1
+
+exp '~' "$home"
+exp '~/foo' "$home/foo"
+exp '~/foo/bar/#' "$home/foo/bar/#"
+exp '~/' "$home/"
+exp '~/../x' "$home/../x"
+
+# ~user/ needs getpwnam: left alone rather than mangled into $HOME + "user/"
+exp '~smith/foo' '~smith/foo'
+exp '~root' '~root'
+
+# only a leading '~' expands; anything else is a literal path component
+exp 'a~/x' 'a~/x'
+exp './~/x' './~/x'
+exp 'foo~bar' 'foo~bar'
+
+# an unset or empty $HOME must not turn ~/foo into the absolute /foo
+test "$(env -u HOME httrack -O /dev/null -#test=expandhome '~/foo')" == "expanded=./foo" || exit 1
+test "$(HOME='' httrack -O /dev/null -#test=expandhome '~/foo')" == "expanded=./foo" || exit 1
+
+# A $HOME past the 2 * HTS_URLMAXSIZE buffer is left alone: strcatbuff aborts
+# rather than truncates. Only meaningful where the engine sees the value we set:
+# MSYS rewrites HOME into a path of its own choosing, long or not.
+if test "$home" = HTSHOME; then
+    long=$(printf '%04096d' 0 | tr '0' 'A')
+    test "$(HOME="$long" httrack -O /dev/null -#test=expandhome '~/foo')" == "expanded=~/foo" || exit 1
+fi

tests/01_engine-topindex.test

@@ -0,0 +1,12 @@
+#!/bin/bash
+#
+
+set -euo pipefail
+
+# hts_buildtopindex takes a system-charset path but verif_backblue below it
+# expects utf-8, mangling a non-ASCII project dir on Windows (#216, #217).
+dir=$(mktemp -d)
+trap 'rm -rf "$dir"' EXIT
+
+httrack -O /dev/null -#test=topindex "$dir" run |
+    grep -q "topindex self-test OK"

tests/01_engine-xfread.test

@@ -9,21 +9,23 @@ set -euo pipefail
 # drives both refusal paths and the accept path.
 
 out="$(httrack -O /dev/null -#test=xfread-limit)"
+# Match with here-strings, not `echo | grep -q`: under pipefail the early grep
+# exit SIGPIPEs echo and fails the pipeline even when the pattern matched.
 for case in bylen bygrow; do
-    echo "$out" | grep -q "${case}: refused=1 adr=null msg=In-memory content too large" || {
+    grep -q "${case}: refused=1 adr=null msg=In-memory content too large" <<<"$out" || {
         echo "FAIL ${case}: $out"
         exit 1
     }
 done
 
 # Exactly INT32_MAX must be refused too (the reallocs add 1).
-echo "$out" | grep -q 'boundary: msg=In-memory content too large' || {
+grep -q 'boundary: msg=In-memory content too large' <<<"$out" || {
     echo "FAIL boundary: $out"
     exit 1
 }
 
 # The guard must NOT fire for a legitimate small size.
-if echo "$out" | grep -q 'accept: msg=In-memory content too large'; then
+if grep -q 'accept: msg=In-memory content too large' <<<"$out"; then
     echo "FAIL accept (guard fired on a legit size): $out"
     exit 1
 fi

tests/52_local-socks5.test

@@ -165,6 +165,13 @@ auth="$tmpdir/auth"
 start_server "$auth" ok
 run_crawl "$auth/out" "http://${host}:${http_port}/" \
     "socks5://user:secret@127.0.0.1:${socks_port}"
+# name a rejected version byte here: the crawl check below only knows the
+# handshake failed, not why
+grep -q "^AUTHVER-BAD" "$auth/socks.log" && {
+    echo "FAIL: sub-negotiation version was not RFC 1929's 0x01" >&2
+    cat "$auth/socks.log" >&2
+    exit 1
+}
 grep -rq "ORIGIN-PAGE-563" "$auth/out" || {
     echo "FAIL: origin not downloaded through the authenticated socks proxy" >&2
     cat "$auth/out.log" "$auth/socks.log" >&2

tests/61_webhttrack-locale.test

@@ -0,0 +1,91 @@
+#!/bin/bash
+#
+# webhttrack must map the caller's locale to the right lang.indexes number.
+# Guards: LC_ALL/LC_MESSAGES were ignored, and zh_tw/pt_br were unreachable.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=$(cd "${testdir}/.." && pwd)
+script="${distdir}/src/webhttrack"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+test -r "${script}" || fail "no ${script}"
+test -r "${distdir}/lang.indexes" || fail "no ${distdir}/lang.indexes"
+
+# Lift the locale code out of the launcher, so this tests the shipped logic itself.
+funcs=$(sed -n '/^function lang_index/,/^}/p' "${script}")
+# End on the next section, not a blank line: a blank line grown inside the block
+# would truncate the lift and silently stop testing everything below it.
+grep -q '^# Find the browser' "${script}" || fail "locale block terminator moved in ${script}"
+block=$(sed -n '/^# Locale/,/^# Find the browser/p' "${script}" | sed '$d')
+echo "${block}" | grep -q 'LANGN=.*lang_index' || fail "could not lift the whole locale block from ${script}"
+
+# Run the lifted block against whatever locale vars the caller exported.
+run_block() {
+    # shellcheck disable=SC2034 # read by the eval'd locale block
+    DISTPATH="${distdir}"
+    # the launcher runs under neither -e ("! test -n x && y" returns 1) nor -u
+    set +eu
+    eval "${funcs}"
+    eval "${block}"
+    # shellcheck disable=SC2153 # LANGN is set by the eval'd block, not a LANG typo
+    echo "${LANGN}"
+}
+
+# Resolve LANG/LC_ALL/LC_MESSAGES (empty = unset) to the language number.
+lang_of() {
+    (
+        unset LANG LC_ALL LC_MESSAGES
+        # only the value matters here; hide bash's setlocale gripe when the locale is not installed
+        # shellcheck disable=SC2030 # confining the locale to this subshell is the point
+        {
+            if test -n "$1"; then export LANG="$1"; fi
+            if test -n "$2"; then export LC_ALL="$2"; fi
+            if test -n "$3"; then export LC_MESSAGES="$3"; fi
+        } 2>/dev/null
+        run_block
+    )
+}
+
+check() {
+    local want=$1 got desc
+    shift
+    desc="LANG='$1' LC_ALL='$2' LC_MESSAGES='$3'"
+    got=$(lang_of "$1" "$2" "$3")
+    test "${got}" = "${want}" || fail "${desc}: want ${want}, got ${got}"
+    echo "ok: ${desc} -> ${got}"
+}
+
+#      want  LANG            LC_ALL          LC_MESSAGES
+check 2 "fr_FR.UTF-8" "" ""
+check 14 "zh_TW.UTF-8" "" ""                      # region-qualified: Chinese-BIG5, not Simplified (13)
+check 12 "pt_BR.UTF-8" "" ""                      # region-qualified: Portugues-Brasil, not Portugues (7)
+check 13 "zh_CN.UTF-8" "" ""                      # unlisted region falls back to the bare language
+check 24 "sl_SI.UTF-8" "" ""                      # Slovenian is ISO 639-1 "sl"
+check 4 "" "de_DE.UTF-8" ""                       # LC_ALL alone
+check 4 "" "" "de_DE.UTF-8"                       # LC_MESSAGES alone
+check 4 "fr_FR.UTF-8" "de_DE.UTF-8" ""            # LC_ALL wins over LANG
+check 4 "fr_FR.UTF-8" "" "de_DE.UTF-8"            # LC_MESSAGES wins over LANG
+check 2 "de_DE.UTF-8" "fr_FR.UTF-8" "de_DE.UTF-8" # LC_ALL wins over LC_MESSAGES
+check 2 "fr_FR@euro" "" ""                        # @modifier without a charset
+check 1 "C" "" ""                                 # C, unknown and unset all mean English
+check 1 "xx_YY.UTF-8" "" ""
+check 1 "" "" ""
+
+# Exported but empty is unset (POSIX), so it must not mask LANG. lang_of cannot
+# express this: it exports nothing for an empty argument.
+got=$(
+    unset LANG LC_ALL LC_MESSAGES
+    # shellcheck disable=SC2031 # confining the locale to this subshell is the point
+    { export LANG="fr_FR.UTF-8" LC_ALL="" LC_MESSAGES=""; } 2>/dev/null
+    run_block
+)
+test "${got}" = 2 || fail "empty LC_ALL/LC_MESSAGES must not mask LANG: want 2, got ${got}"
+echo "ok: LANG='fr_FR.UTF-8' with LC_ALL='' exported -> ${got}"
+
+echo "PASS"

tests/62_lang-integrity.test

@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# lang/*.txt are line pairs: an English msgid, then its translation. A stray
+# blank or a drifted msgid silently unhooks every translation that follows,
+# rather than failing, so assert the pairing and the join against English.txt.
+# Pairs are physical lines here; a msgid ending in \ continues onto the next one
+# for the engine (linput_cpp) but not for us. None do today.
+
+set -euo pipefail
+
+langdir="${top_srcdir:-..}/lang"
+def="${top_srcdir:-..}/lang.def"
+eng="$langdir/English.txt"
+for f in "$eng" "$def"; do
+    [ -f "$f" ] || {
+        echo "cannot find $f"
+        exit 1
+    }
+done
+
+tmp=$(mktemp -d)
+trap 'rm -rf "$tmp"' EXIT
+keys="$tmp/english.keys"
+
+# Byte-wise: each file is in its own declared legacy charset, lang.def is CRLF.
+LC_ALL=C awk 'NR%2==1 { sub(/\r$/, ""); print }' "$eng" >"$keys"
+
+# An empty parse would make every check below pass vacuously.
+nkeys=$(wc -l <"$keys")
+if [ "$nkeys" -lt 400 ]; then
+    echo "only $nkeys msgids parsed from $eng; the parse is broken"
+    exit 1
+fi
+
+nfiles=0
+fail=0
+for f in "$langdir"/*.txt; do
+    nfiles=$((nfiles + 1))
+    LC_ALL=C awk -v keys="$keys" -v name="${f##*/}" '
+        BEGIN { while ((getline k < keys) > 0) known[k] = 1 }
+        { sub(/\r$/, "") }
+        NR % 2 == 1 {
+            if ($0 == "")
+                printf "%s:%d: blank line where an English msgid belongs\n", name, NR
+            else if (!($0 in known))
+                printf "%s:%d: msgid absent from English.txt: %s\n", name, NR, $0
+            else
+                next
+            bad++
+        }
+        END {
+            if (NR % 2) {
+                printf "%s: odd line count (%d): msgid/translation pairing is broken\n", name, NR
+                bad++
+            }
+            exit(bad > 0)
+        }
+    ' "$f" || fail=1
+done
+
+if [ "$nfiles" -lt 25 ]; then
+    echo "only $nfiles language files found in $langdir"
+    exit 1
+fi
+
+# lang.def maps each LANG_* macro to the English string the GUI compiles in,
+# which is the lookup key into every lang/*.txt: no msgid, no translations.
+LC_ALL=C awk -v keys="$keys" '
+    BEGIN { while ((getline k < keys) > 0) known[k] = 1 }
+    { sub(/\r$/, "") }
+    NR % 2 == 1 { macro = $0; next }
+    macro ~ /^(LANG_|LISTDEF_)/ {
+        n++
+        if (!($0 in known)) {
+            printf "lang.def:%d: %s maps to \"%s\", which is not a msgid in English.txt\n", NR, macro, $0
+            bad++
+        }
+    }
+    END {
+        if (n < 400) {
+            printf "lang.def: only %d LANG_ entries parsed; the parse is broken\n", n
+            bad++
+        }
+        exit(bad > 0)
+    }
+' "$def" || fail=1
+
+[ "$fail" -eq 0 ] || exit 1
+
+echo "$nfiles language files consistent with $nkeys English msgids"

tests/63_webhttrack-home.test

@@ -0,0 +1,88 @@
+#!/bin/bash
+#
+# htsserver's $HOME lookup, through the web UI it actually renders. An exported
+# but empty $HOME must not resolve the default base path against the root.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
+distdir=$(cd "${distdir}" && pwd)
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+command -v htsserver >/dev/null || fail "no htsserver in PATH"
+# Mirror local-crawl.sh's skip-with-77: the Debian buildd chroot has no python3.
+command -v python3 >/dev/null || {
+    echo "python3 not found; skipping" >&2
+    exit 77
+}
+
+srv=
+cleanup() {
+    test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
+}
+trap cleanup EXIT HUP INT QUIT PIPE TERM
+
+log=$(mktemp)
+trap 'cleanup; rm -f "${log}"' EXIT
+
+# Ask htsserver for its rendered default base path, under the given $HOME
+# ("-" = leave $HOME unset).
+pathfor() {
+    local homeval=$1 port url got
+    port=$(python3 -c 'import socket
+s = socket.socket()
+s.bind(("127.0.0.1", 0))
+print(s.getsockname()[1])
+s.close()')
+
+    : >"${log}"
+    (
+        # htsserver ignores SIGTERM/SIGTTOU handling from the harness shell
+        trap '' TERM TTOU
+        if test "${homeval}" = -; then unset HOME; else export HOME="${homeval}"; fi
+        exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
+    ) &
+    srv=$!
+
+    for _ in $(seq 1 40); do
+        url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
+        kill -0 "${srv}" 2>/dev/null || break
+        sleep 0.25
+    done
+    test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
+
+    # The UI is served ISO-8859-1, so keep python out of text mode.
+    got=$(python3 -c 'import re, sys, urllib.request
+body = urllib.request.urlopen(sys.argv[1] + "server/step2.html", timeout=20).read()
+m = re.search(br"name=\"path\" value=\"([^\"]*)\"", body)
+print(m.group(1).decode("latin-1") if m else "")' "${url}")
+
+    kill -9 "${srv}" 2>/dev/null || true
+    wait "${srv}" 2>/dev/null || true
+    srv=
+    test -n "${got}" || fail "no path field in the rendered step2.html"
+    echo "${got}"
+}
+
+check() {
+    local desc=$1 homeval=$2 want=$3 got
+    got=$(pathfor "${homeval}")
+    test "${got}" = "${want}" || fail "${desc}: want ${want}, got ${got}"
+    echo "ok: ${desc} -> ${got}"
+}
+
+# A real HOME must reach the page, or the two cases below are vacuous: they
+# also pass when gethomedir() ignores the environment and always answers ".".
+check 'a set HOME is used' /nonexistent/hts-home /nonexistent/hts-home/websites
+
+# Exported but empty is the regression: getenv() returns "", not NULL, so a
+# bare NULL check let the base path collapse onto /websites.
+check 'an empty HOME' '' ./websites
+check 'an unset HOME' - ./websites
+
+echo "PASS"

tests/64_local-intl-outdir.test

@@ -0,0 +1,13 @@
+#!/bin/bash
+#
+# A non-ASCII -O output dir must receive the mirror, not a double-encoded twin
+# (#621). argv is UTF-8 on Windows, so path_html was re-encoded as if it were
+# the ANSI codepage, prefixing every saved file with a mangled directory. Only
+# path_html is the non-ASCII "café"; path_log stays ASCII so the harness reads
+# hts-log.txt/hts-cache normally and this isolates the -O encoding path.
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" --html-subdir 'café' --errors 0 --files 5 \
+    --found 'simple/basic.html' \
+    httrack 'BASEURL/simple/basic.html'

tests/65_port-siblings.test

@@ -0,0 +1,127 @@
+#!/bin/bash
+#
+
+# Port parsing outside the crawled-URL path (#614): the htsserver --port listen
+# port, proxytrack's host:port listen arguments, and an ftp:// URL port. All
+# three took the port from sscanf("%d"), which range-checks nothing and wraps
+# past INT_MAX, so a value naming no real port silently became a plausible one.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# run_with_timeout: timeout(1) is absent on macOS, and the listen servers block.
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_portsib.XXXXXX") || exit 1
+trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
+
+# A port the old code accepted only by wrapping: 4295009395 = 2^32 + 42099 and
+# 4294967376 = 2^32 + 80 both fold to a plausible listen port. They are the
+# load-bearing cases, since a value that merely exceeds 65535 while still
+# fitting an int (65616) was already caught by the htsserver range check.
+overflowing="4295009395"
+
+# Values sscanf("%d") took as a port that no operator meant: a trailing
+# character, a sign, leading whitespace, a decimal point. Each silently listened
+# on the prefix it managed to scan.
+malformed=("80x" "+80" " 80" "8.0" "")
+
+# --- htsserver --port ------------------------------------------------------
+# Its argv[1] is the html root; the option pairs follow. It blocks serving once
+# a port is accepted, so bound it; we assert on the output, not the exit status.
+
+websrv() {
+    run_with_timeout 3 htsserver "$tmp" --port "$1" >"$tmp/web.log" 2>&1 || true
+}
+
+web_refused() {
+    websrv "$1"
+    grep -q "couldn't set the port number" "$tmp/web.log" ||
+        ! echo "FAIL: #614: htsserver --port '$1' not refused" || exit 1
+    # it must not have reached the listen socket on some other port
+    ! grep -q "^URL=" "$tmp/web.log" ||
+        ! echo "FAIL: #614: htsserver --port '$1' listened anyway" || exit 1
+}
+
+web_accepted() {
+    local port=$1
+    : >"$tmp/web.log"
+    # A bound server blocks, so stop it once it announces, not on a fixed
+    # deadline: the announce trails a local-hostname resolve that stalls on macOS,
+    # where a 3s kill landed mid-resolve, before the URL= line ever printed.
+    local had_m=
+    case "$-" in *m*) had_m=1 ;; esac
+    is_windows || set -m
+    htsserver "$tmp" --port "$port" >"$tmp/web.log" 2>&1 &
+    local pid=$!
+    test -n "$had_m" || is_windows || set +m
+    local waited=0
+    until grep -qE "^URL=http://.*:$port/|couldn't set the port number|Unable to initialize a temporary server" "$tmp/web.log"; do
+        test "$waited" -lt 20 || break
+        sleep 1
+        waited=$((waited + 1))
+    done
+    kill_tree "$pid"
+    wait "$pid" 2>/dev/null || true
+    ! grep -q "couldn't set the port number" "$tmp/web.log" ||
+        ! echo "FAIL: #614: htsserver --port '$port' wrongly refused" || exit 1
+    grep -qE "^URL=http://.*:$port/|Unable to initialize a temporary server" "$tmp/web.log" ||
+        ! echo "FAIL: #614: htsserver --port '$port' never reached the listener" || exit 1
+}
+
+# this used to listen on 42099
+web_refused "$overflowing"
+for p in "${malformed[@]}" 0 65536 99999 -1; do
+    web_refused "$p"
+done
+
+# 65535 is a valid port the old "< 65535" bound refused
+web_accepted 65535
+
+# --- proxytrack <proxy-addr:port> <ICP-addr:port> --------------------------
+# A bad argument falls through to the usage screen; it had no range check at
+# all, so 65616 quietly listened on port 80. A valid one binds and blocks.
+
+proxy_refused() {
+    run_with_timeout 3 proxytrack "127.0.0.1:$1" 127.0.0.1:3130 >"$tmp/pt.log" 2>&1 || true
+    grep -q "^usage:" "$tmp/pt.log" ||
+        ! echo "FAIL: #614: proxytrack port '$1' not refused" || exit 1
+}
+
+proxy_accepted() {
+    run_with_timeout 3 proxytrack "127.0.0.1:$1" 127.0.0.1:3130 >"$tmp/pt.log" 2>&1 || true
+    ! grep -q "^usage:" "$tmp/pt.log" ||
+        ! echo "FAIL: #614: proxytrack port '$1' wrongly refused" || exit 1
+}
+
+for p in "${malformed[@]}" 65616 4294967376 "$overflowing" 0 -1; do
+    proxy_refused "$p"
+done
+proxy_accepted 45678
+
+# --- ftp:// URL port -------------------------------------------------------
+# A URL port, so it follows the crawled-URL policy: refuse the link rather than
+# fold it into range. An ftp:// link in scanned HTML is not followed, so the
+# seed has to come from the command line.
+
+ftp_port() {
+    local out="$tmp/ftp"
+
+    rm -rf "$out"
+    run_with_timeout 30 httrack "ftp://127.0.0.1:$1/x" -O "$out" --quiet -n \
+        >/dev/null 2>&1 || true
+    grep -c "Invalid port: $1" "$out/hts-log.txt" 2>/dev/null || true
+}
+
+# 65616 truncated to 80 and really did connect there
+for p in 65616 99999; do
+    test "$(ftp_port "$p")" -gt 0 ||
+        ! echo "FAIL: #614: ftp:// port $p not refused" || exit 1
+done
+
+# a valid port still reaches the connect attempt
+test "$(ftp_port 65535)" -eq 0 ||
+    ! echo "FAIL: #614: ftp:// port 65535 wrongly refused" || exit 1
+
+exit 0

tests/Makefile.am

@@ -59,6 +59,7 @@ TESTS = \
 	01_engine-pause.test \
 	01_engine-rcfile.test \
 	01_engine-reconcile.test \
+	01_engine-expandhome.test \
 	01_engine-fsize.test \
 	01_engine-redirect.test \
 	01_engine-relative.test \
@@ -70,6 +71,7 @@ TESTS = \
 	01_engine-status.test \
 	01_engine-stripquery.test \
 	01_engine-strsafe.test \
+	01_engine-topindex.test \
 	01_engine-urlhack.test \
 	01_engine-unescape-bounds.test \
 	01_engine-useragent.test \
@@ -139,6 +141,11 @@ TESTS = \
 	57_local-proxy-connect.test \
 	58_watchdog.test \
 	59_local-tls-stall.test \
-	60_crawl-log-salvage.test
+	60_crawl-log-salvage.test \
+	61_webhttrack-locale.test \
+	62_lang-integrity.test \
+	63_webhttrack-home.test \
+	64_local-intl-outdir.test \
+	65_port-siblings.test
 
 CLEANFILES = check-network_sh.cache

tests/local-crawl.sh

@@ -47,6 +47,7 @@ key="${testdir}/server.key"
 
 tls=
 verbose=
+html_subdir=
 rerun=
 rerun_args=
 rerun_dead=
@@ -132,6 +133,13 @@ while test "$pos" -lt "$nargs"; do
         pos=$((pos + 1))
         rerun_args="${args[$pos]}"
         ;;
+    --html-subdir)
+        # Mirror into "$out/NAME" (path_html) but keep logs/cache in "$out"
+        # (path_log): a non-ASCII NAME exercises the -O path encoding (#621)
+        # without routing the harness-read hts-log.txt through a non-ASCII path.
+        pos=$((pos + 1))
+        html_subdir="${args[$pos]}"
+        ;;
     --errors | --errors-content | --files)
         audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
         pos=$((pos + 1))
@@ -207,12 +215,21 @@ test -n "$ver" || die "could not run httrack"
 
 out="${tmpdir}/crawl"
 mkdir "$out" || die "could not create $out"
+# path_html holds the mirror + index; path_log holds hts-cache/hts-log.txt.
+# Default: both are "$out". With --html-subdir, path_html becomes "$out/NAME"
+# (the mirror root the audits inspect) while path_log stays "$out".
+mirrorroot="$out"
+odir="$out"
+if test -n "$html_subdir"; then
+    mirrorroot="${out}/${html_subdir}"
+    odir="${mirrorroot},${out}"
+fi
 # Localhost is fast; disable the rate/bandwidth safety limits but keep a
 # max-time backstop so a hang cannot wedge the suite.
 declare -a moreargs=(--quiet --max-time=120 --timeout=30 --disable-security-limits --robots=0)
 log="${tmpdir}/log"
 info "running httrack ${hts[*]}"
-httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" "${moreargs[@]}" "${hts[@]}" >"$log" 2>&1 &
+httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" "${moreargs[@]}" "${hts[@]}" >"$log" 2>&1 &
 crawlpid=$!
 wait "$crawlpid"
 crawlres=$?
@@ -229,7 +246,7 @@ grep -iE "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt" >&2
 # --- optional second pass: re-mirror into the same dir (cache/update path) ----
 if test -n "$rerun"; then
     info "re-running httrack (update pass)"
-    httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" \
+    httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" \
         "${moreargs[@]}" "${hts[@]}" >"${log}.2" 2>&1 &
     crawlpid=$!
     wait "$crawlpid"
@@ -257,7 +274,7 @@ fi
 if test -n "$rerun_args"; then
     read -ra extra <<<"$rerun_args"
     info "re-running httrack with ${rerun_args}"
-    httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" \
+    httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" \
         "${moreargs[@]}" "${hts[@]}" "${extra[@]}" >"${log}.2" 2>&1 &
     crawlpid=$!
     wait "$crawlpid"
@@ -279,7 +296,7 @@ if test -n "$rerun_dead"; then
     stop_server "$serverpid"
     serverpid=
     info "re-running httrack against the stopped server"
-    httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" \
+    httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" \
         "${moreargs[@]}" "${hts[@]}" >"${log}.dead" 2>&1 &
     crawlpid=$!
     wait "$crawlpid" || true
@@ -308,7 +325,7 @@ fi
 
 # --- discover the single host root (127.0.0.1_<port> or 127.0.0.1) -----------
 hostroot=
-for cand in "${out}/127.0.0.1_${port}" "${out}/127.0.0.1"; do
+for cand in "${mirrorroot}/127.0.0.1_${port}" "${mirrorroot}/127.0.0.1"; do
     if test -d "$cand"; then
         hostroot="$cand"
         break

tests/socks5-server.py

@@ -24,6 +24,7 @@ from proxytestlib import bind_ephemeral, pipe  # noqa: E402
 # The one name the proxy answers for; a .invalid TLD never resolves (RFC 6761),
 # so a locally-resolving client could not reach us -- success proves remote DNS.
 REMOTE_HOST = b"socks-origin.invalid"
+AUTH_VERSION = 0x01  # RFC 1929 sub-negotiation version
 # index links the subpages so an -r3 crawl reuses one keep-alive socket
 LINKS = "".join('<a href="/p%d.html">%d</a>' % (i, i) for i in range(1, 6))
 ORIGIN_BODY = ("<html><body>ORIGIN-PAGE-563 " + LINKS + "</body></html>").encode()
@@ -104,6 +105,12 @@ def negotiate_auth(conn, logdir):
     if 0x02 in methods:  # prefer user/pass so the auth test exercises RFC 1929
         conn.sendall(b"\x05\x02")
         (subver,) = recvn(conn, 1)
+        # RFC 1929's version byte is 0x01, not SOCKS5's 0x05: a real proxy
+        # rejects a mismatch instead of tunnelling anyway
+        if subver != AUTH_VERSION:
+            log(logdir, "AUTHVER-BAD %d" % subver)
+            conn.sendall(bytes([AUTH_VERSION, 0x01]))  # sub-negotiation failure
+            return False
         (ulen,) = recvn(conn, 1)
         uname = recvn(conn, ulen)
         (plen,) = recvn(conn, 1)