~/ in the -O base path has never been expanded (#619)

* Restore ~/ expansion in the -O base path

expand_home() has never fired since 660b569 (3.41.2) refactored it from char*
to String: the correct str[0] == '~' became StringSub(*str, 1) == '~', but the
body still skips a single leading character. "~/foo"[1] is '/', so the guard
never matches, while any path whose *second* char is '~' silently lost its
first character and gained $HOME ("a~/x" -> "/home/smith~/x").

Index from 0 and gate on a following '/' so ~user/ is left alone rather than
mangled into $HOME + "user/" -- resolving it needs getpwnam, which this does
not do. An empty $HOME now falls back to '.' like an unset one, so ~/foo can
no longer expand to the absolute /foo, and an oversized $HOME leaves the path
untouched instead of aborting inside strcatbuff.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the expandhome test to a $HOME that MSYS leaves alone

The Win32 leg spawns a native httrack.exe, and MSYS rewrites a POSIX absolute
$HOME into a Windows path on the way in, so the engine read
C:\Program Files\Git\home\smith and the pinned /home/smith never matched. A
$HOME with no leading '/' is passed through verbatim; expand_home only
concatenates, so the sentinel exercises the same code.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Restore the clang-format run over the gate condition

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Ask the engine for $HOME rather than pinning it in the expandhome test

MSYS rewrites $HOME into a cwd-relative Windows path before the native
httrack.exe reads it (HTSHOME became D:\a\httrack\httrack\tests\HTSHOME), so
no pinned value survives the Win32 leg. Take the expansion of a bare '~' as
the reference instead, and assert it moved off '~' so the derived cases cannot
go vacuous. The ~user/ and a~/ negatives are literals and stay exact.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin what the expandhome test could not distinguish

A hts_gethome() that ignored $HOME and always returned the fallback passed
the whole suite: the empty-HOME cases expect ./foo, which a constant "."
satisfies. That is the same silent no-op the fix exists to remove. Also cover
the oversized-$HOME bail, which had no coverage at all.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the oversized-$HOME case inside Windows' environment limit

70000 chars cannot be set as an environment variable on Windows, so the case
failed the MSVC legs, and $(seq 1 70000) flooded the -x log. The buffer is
2 * HTS_URLMAXSIZE, so 4096 clears it and still kills the no-guard mutant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Run the oversized-$HOME case only where HOME survives the shell

MSYS rewrites HOME into a path of its own, so the engine never saw the long
value: it expanded ~/foo against the build directory instead of bailing. Gate
the case on the engine reporting the HOME we set; it still runs, and still
fails without the guard, on every platform that passes HOME through.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/workflows/windows-build.yml

@@ -158,13 +158,21 @@ jobs:
 
           pass=0 fail=0 skip=0 failed="" skipped=""
           for t in 00_runnable.test 01_engine-*.test 01_zlib-*.test \
-                   *_local-*.test 13_crawl_proxy_https.test 58_watchdog.test; do
+                   *_local-*.test 13_crawl_proxy_https.test 58_watchdog.test \
+                   60_crawl-log-salvage.test; do
             rc=0
             run_with_timeout 600 bash "$t" >"$t.log" 2>&1 || rc=$?
             case "$rc" in
               0) pass=$((pass + 1)); echo "PASS $t" ;;
               77) skip=$((skip + 1)) skipped="$skipped $t"; echo "SKIP $t" ;;
-              124) fail=$((fail + 1)) failed="$failed $t"; echo "FAIL $t (timed out, tree killed)" ;;
+              124)
+                fail=$((fail + 1)) failed="$failed $t"
+                echo "FAIL $t (timed out, tree killed)"
+                # Re-running a wedge traced would just hang again: salvage the
+                # killed crawl's own logs into the artifact instead.
+                dump_crawl_logs >>"$t.log"
+                tail -n 25 "$t.log" | sed 's/^/      /'
+                ;;
               *)
                 fail=$((fail + 1)) failed="$failed $t"
                 echo "FAIL $t (exit $rc)"

lang.indexes

@@ -23,7 +23,7 @@ pt:7
 ro:25
 ru:8
 sk:20
-si:24
+sl:24
 sv:17
 tr:10
 uk:22

lang/Finnish.txt

@@ -930,7 +930,6 @@ Server terminated
 Palvelin lopetettu
 A fatal error has occurred during this mirror
 Tällä peilillä tapahtui vakava virhe
-
 Proxy type:
 Välityspalvelimen tyyppi:
 Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.

lang/Italiano.txt

@@ -374,7 +374,7 @@ Create error logging and report files
 Crea file di log per segnalare errori e informazioni
 Generate DOS 8-3 filenames ONLY
 Genera solo nomi di file in formato 8.3
-Generate ISO 9660 filenames ONLY for CDROM medias
+Generate ISO9660 filenames ONLY for CDROM medias
 Genera nomi di file in formato ISO9660 per i CDROM
 Do not create HTML error pages
 Non generare pagine d'errore HTML

lang/Portugues-Brasil.txt

@@ -918,7 +918,7 @@ normal\nextended\ndebug
 normal\nextendido\ncorrigir
 Download web site(s)\nDownload web site(s) + questions\nGet individual files\nDownload all sites in pages (multiple mirror)\nTest links in pages (bookmark test)\n* Continue interrupted download\n* Update existing download
 Copiar site(s) da Web\nCopiar site(s) interativos da Web (perguntas)\nReceber arquivos específicos\nCopiar todas as páginas do site (alternação múltipla)\nTestar links nas páginas (testar indicador)\n* Retomar download interrompido\n* Atualizar download existente
-Relative URL / Absolute URL (default)\nAbsolute URL / Absolute URL\nAbsolute URI / Absolute URL\nOriginal URL / Original URL
+Relative URI / Absolute URL (default)\nAbsolute URL / Absolute URL\nAbsolute URI / Absolute URL\nOriginal URL / Original URL
 URI Relativa / URL Absoluta (padrão)\nURL Absoluta / URL Absoluta\nURI Absoluta / URL Absoluta\nURL Original / URL Original
 Open Source offline browser
 Abrir origem offline no navegador

lang/Slovenian.txt

@@ -3,7 +3,7 @@ Slovenian
 LANGUAGE_FILE
 Slovenian
 LANGUAGE_ISO
-si
+sl
 LANGUAGE_AUTHOR
 Jadran Rudec,iur.\r\njrudec@email.si \r\n
 LANGUAGE_CHARSET

src/htsalias.c

@@ -567,24 +567,29 @@ int optinclude_file(const char *name, int *argc, char **argv, char *x_argvblk,
   return 0;
 }
 
-/* Get home directory, '.' if failed */
+/* Get home directory, '.' if unset or empty */
 /* example: /home/smith */
 const char *hts_gethome(void) {
   const char *home = getenv("HOME");
 
-  if (home)
-    return home;
-  else
-    return ".";
+  /* An empty $HOME would expand ~/foo into the absolute /foo */
+  return strnotempty(home) ? home : ".";
 }
 
-/* Convert ~/foo into /home/smith/foo */
+/* Convert ~/foo into /home/smith/foo (~user/ left alone: no getpwnam here) */
 void expand_home(String * str) {
-  if (StringSub(*str, 1) == '~') {
+  if (StringNotEmpty(*str) && StringSub(*str, 0) == '~' &&
+      (StringLength(*str) == 1 || StringSub(*str, 1) == '/')) {
     char BIGSTK tempo[HTS_URLMAXSIZE * 2];
+    const char *const home = hts_gethome();
+    const size_t homelen = strlen(home);
+    const size_t taillen = StringLength(*str) - 1;
 
-    strcpybuff(tempo, hts_gethome());
-    strcatbuff(tempo, StringBuff(*str) + 1);
-    StringCopy(*str, tempo);
+    /* Leave untouched rather than abort() in strcatbuff on a huge $HOME */
+    if (taillen < sizeof(tempo) && homelen < sizeof(tempo) - taillen) {
+      strcpybuff(tempo, home);
+      strcatbuff(tempo, StringBuff(*str) + 1);
+      StringCopy(*str, tempo);
+    }
   }
 }

src/htsback.c

@@ -2816,6 +2816,9 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                 if (SSL_set_fd(back[i].r.ssl_con, (int) back[i].r.soc) == 1) {
                   SSL_set_connect_state(back[i].r.ssl_con);
                   back[i].status = STATUS_SSL_WAIT_HANDSHAKE;   /* handshake wait */
+                  // the handshake gets its own timeout window, as connect does
+                  if (back[i].timeout > 0)
+                    back[i].timeout_refresh = time_local();
                 } else
                   back[i].r.statuscode = STATUSCODE_SSL_HANDSHAKE;
               } else
@@ -2875,6 +2878,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
       }
 #if HTS_USEOPENSSL
       else if (back[i].status == STATUS_SSL_WAIT_HANDSHAKE) {       // wait for SSL handshake
+        // a peer that never speaks TLS must be reaped by --timeout too (#607)
+        if (!gestion_timeout)
+          if (back[i].timeout > 0)
+            gestion_timeout = 1;
+
         /* SSL mode */
         if (back[i].r.ssl) {
           int conn_code;
@@ -4265,6 +4273,8 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
                 strcpybuff(back[i].r.msg, "Connect Time Out");
               else if (back[i].status == STATUS_WAIT_DNS)
                 strcpybuff(back[i].r.msg, "DNS Time Out");
+              else if (back[i].status == STATUS_SSL_WAIT_HANDSHAKE)
+                strcpybuff(back[i].r.msg, "SSL/TLS Handshake Time Out");
               else
                 strcpybuff(back[i].r.msg, "Receive Time Out");
               back[i].status = STATUS_READY;    // terminé

src/htscoremain.c

@@ -113,6 +113,7 @@ HTSEXT_API int hts_main(int argc, char **argv) {
 }
 
 static int hts_main_internal(int argc, char **argv, httrackp * opt);
+static hts_boolean cmdl_shortopt_has(const char *s, char c);
 
 // Main, récupère les paramètres et appelle le robot
 HTSEXT_API int hts_main2(int argc, char **argv, httrackp * opt) {
@@ -304,12 +305,12 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
                      hts_get_version_info(opt));
               return 0;
             } else {
-              if (strncmp(tmp_argv[0], "--", 2)) {      /* pas */
-                if ((strchr(tmp_argv[0], 'q') != NULL))
-                  opt->quiet = 1;       // ne pas poser de questions! (nohup par exemple)
-                if ((strchr(tmp_argv[0], 'i') != NULL)) {       // doit.log!
-                  argv_url = -1;        /* forcer */
-                  opt->quiet = 1;
+              if (strncmp(tmp_argv[0], "--", 2)) { /* not a long option */
+                if (cmdl_shortopt_has(tmp_argv[0], 'q'))
+                  opt->quiet = HTS_TRUE; // never ask questions (nohup)
+                if (cmdl_shortopt_has(tmp_argv[0], 'i')) { // doit.log!
+                  argv_url = -1;
+                  opt->quiet = HTS_TRUE;
                 }
               } else if (strcmp(tmp_argv[0] + 2, "quiet") == 0) {
                 opt->quiet = 1; // ne pas poser de questions! (nohup par exemple)
@@ -2807,6 +2808,32 @@ int check_path(String * s, char *defaultname) {
   return return_value;
 }
 
+/* Does the short-option cluster s carry c from the main option set (-i, -iC2,
+   -%Mi)? Walked as the parser does below: %, &, @ and # each take the letter
+   after them into another set, so the i of -%i is not the main-set -i. */
+static hts_boolean cmdl_shortopt_has(const char *s, char c) {
+  const char *com;
+
+  if (s[0] != '-' || s[1] == '-')
+    return HTS_FALSE;
+  for (com = s + 1; *com != '\0'; com++) {
+    switch (*com) {
+    case '%':
+    case '&':
+    case '@':
+    case '#':
+      if (*(com + 1) != '\0')
+        com++; /* skip the other set's letter */
+      break;
+    default:
+      if (*com == c)
+        return HTS_TRUE;
+      break;
+    }
+  }
+  return HTS_FALSE;
+}
+
 // détermine si l'argument est une option
 int cmdl_opt(char *s) {
   if (s[0] == '-') {            // c'est peut être une option

src/htslib.c

@@ -65,6 +65,7 @@ Please visit our Website: http://www.httrack.com
 #endif /* _WIN32 */
 #include <stdarg.h>
 
+#include <ctype.h>
 #include <string.h>
 #include <time.h>
 #include <stdarg.h>
@@ -3747,6 +3748,20 @@ static int proxy_default_port(const char *arg) {
   return hts_proxy_is_socks(arg) ? 1080 : 8080;
 }
 
+// port "a" of -P argument "arg": digits fitting TCP's 1..65535, else the scheme
+// default. Not sscanf("%d"): past INT_MAX it wraps to a garbage port (#602)
+static int parse_proxy_port(const char *a, const char *arg) {
+  char *end;
+  long p;
+
+  if (!isdigit((unsigned char) *a)) // strtol would eat a sign or leading space
+    return proxy_default_port(arg);
+  p = strtol(a, &end, 10);
+  if (*end != '\0' || p < 1 || p > 65535) // ERANGE lands out of range too
+    return proxy_default_port(arg);
+  return (int) p;
+}
+
 void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port) {
   const char *authority = strstr(arg, "://");
   const char *a;
@@ -3762,10 +3777,7 @@ void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port) {
   while (a > authority && a[-1] != ':' && a[-1] != '@' && a[-1] != ']')
     a--;
   if (a > authority && a[-1] == ':') {
-    int p = -1;
-
-    sscanf(a, "%d", &p);
-    *port = (p > 0) ? p : proxy_default_port(arg);
+    *port = parse_proxy_port(a, arg);
     namelen = (size_t) (a - 1 - arg);
   } else {
     *port = proxy_default_port(arg);

src/htsselftest.c

@@ -45,6 +45,7 @@ Please visit our Website: http://www.httrack.com
 #include "htscore.h"
 #include "htsdefines.h"
 #include "htslib.h"
+#include "htsalias.h"
 #include "htsparse.h"
 #include "htscache.h"
 #include "htscache_selftest.h"
@@ -825,6 +826,21 @@ static int st_simplify(httrackp *opt, int argc, char **argv) {
   return 0;
 }
 
+static int st_expandhome(httrackp *opt, int argc, char **argv) {
+  String path = STRING_EMPTY;
+
+  (void) opt;
+  if (argc < 1) {
+    fprintf(stderr, "expandhome: needs a path\n");
+    return 1;
+  }
+  StringCopy(path, argv[0]);
+  expand_home(&path);
+  printf("expanded=%s\n", StringBuff(path));
+  StringFree(path);
+  return 0;
+}
+
 static int st_mime(httrackp *opt, int argc, char **argv) {
   char mime[256];
 
@@ -3079,6 +3095,7 @@ static const struct selftest_entry {
     {"filterbounds", "", "matcher length/work caps reject hostile patterns",
      st_filterbounds},
     {"simplify", "<path>", "collapse ./ and ../ in a path", st_simplify},
+    {"expandhome", "<path>", "expand a leading ~/ into $HOME", st_expandhome},
     {"stripquery", "", "--strip-query pattern/key stripping self-test",
      st_stripquery},
     {"urlhack", "", "-%u url-hack sub-flag (www/slash/query) self-test",

src/webhttrack

@@ -33,6 +33,19 @@ function log {
     return 0
 }
 
+# Map a POSIX locale ("zh_TW.UTF-8@euro") to its lang.indexes number, English (1) if unknown
+function lang_index {
+    local locale=$1 indexes=$2 tag n t
+    tag=$(echo "${locale}" | cut -f1 -d'.' | cut -f1 -d'@' | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z_')
+    # a few languages carry a region-specific entry (zh_tw, pt_br); else use the bare language
+    for t in "${tag}" "${tag%%_*}"; do
+        n=$(grep -E "^${t}:" "${indexes}" | cut -f2 -d':')
+        test -n "${n}" && break
+    done
+    test -n "${n}" || n=1
+    echo "${n}"
+}
+
 function launch_browser {
     log "Launching $1"
     browser=$1
@@ -59,13 +72,11 @@ test -f "${DISTPATH}/lang.indexes" || ! log "Could not find ${DISTPATH}/lang.ind
 test -d "${DISTPATH}/lang" || ! log "Could not find ${DISTPATH}/lang" || exit 1
 test -d "${DISTPATH}/html" || ! log "Could not find ${DISTPATH}/html" || exit 1
 
-# Locale
-HTSLANG="${LC_MESSAGES}"
-! test -n "${HTSLANG}" && HTSLANG="${LC_ALL}"
+# Locale: POSIX precedence, LC_ALL overrides LC_MESSAGES overrides LANG
+HTSLANG="${LC_ALL}"
+! test -n "${HTSLANG}" && HTSLANG="${LC_MESSAGES}"
 ! test -n "${HTSLANG}" && HTSLANG="${LANG}"
-HTSLANG="$(echo "$LANG" | cut -f1 -d'.' | cut -f1 -d'_')"
-LANGN=$(grep -E "^${HTSLANG}:" "${DISTPATH}/lang.indexes" | cut -f2 -d':')
-! test -n "${LANGN}" && LANGN=1
+LANGN=$(lang_index "${HTSLANG}" "${DISTPATH}/lang.indexes")
 
 # Find the browser
 # note: not all systems have sensible-browser or www-browser alternative

tests/01_engine-cmdline.test

@@ -54,6 +54,13 @@ refused() {
         ! echo "FAIL: $1 (exit $RC)" || exit 1
 }
 
+# assert continue mode was entered: it drops the URL list, so with no cache to
+# resume the run ends on the usage screen rather than on any other error
+continued() {
+    { test "$RC" -ne 0 && grep -q 'usage:' "$1/.log"; } ||
+        ! echo "FAIL: $2 (exit $RC)" || exit 1
+}
+
 # a value past the old 126/256 caps but within the cap is accepted, on both the
 # short and long form of each option
 long=$(nchars 900)
@@ -102,4 +109,50 @@ for bad in nan nan:5 5:nan inf 10:5 99999; do
     refused "#185: invalid --pause '$bad' not refused cleanly"
 done
 
+# An option is not -i (continue) merely because its name contains an 'i' (#615).
+# These used to wipe the URL given before them and exit on the usage screen.
+run "$tmp/ord-bti" --build-top-index
+accepted "$tmp/ord-bti" "#615: --build-top-index after the URL wiped the URL list"
+run "$tmp/ord-bti-s" "-%i"
+accepted "$tmp/ord-bti-s" "#615: -%i after the URL wiped the URL list"
+run "$tmp/ord-proto" --protocol 2
+accepted "$tmp/ord-proto" "#615: --protocol after the URL wiped the URL list"
+run "$tmp/ord-proto-s" "-@i2"
+accepted "$tmp/ord-proto-s" "#615: -@i2 after the URL wiped the URL list"
+
+# %, &, @ and # take the letter after them into another option set, wherever
+# they sit in the cluster, so the i of -q%i is not the main-set -i either.
+run "$tmp/ord-qpi" "-q%i"
+accepted "$tmp/ord-qpi" "#615: -q%i after the URL wiped the URL list"
+run "$tmp/ord-qai" "-q@i"
+accepted "$tmp/ord-qai" "#615: -q@i after the URL wiped the URL list"
+
+# -%q only forces quiet mode, which nothing here can see: a run whose stdout is
+# not a tty is quiet from the start (htscoremain.c:174). Just check it crawls.
+run "$tmp/ord-iqs" "-%q"
+accepted "$tmp/ord-iqs" "#615: -%q after the URL broke the crawl"
+
+# The real -i still forces continue mode and drops the URL list, in every form:
+# a fix that merely stopped looking for 'i' would pass the checks above.
+run "$tmp/cont-s" -i
+continued "$tmp/cont-s" "#615: -i after the URL no longer forces continue mode"
+run "$tmp/cont-c" -iC2
+continued "$tmp/cont-c" "#615: -iC2 after the URL no longer forces continue mode"
+run "$tmp/cont-l" --continue
+continued "$tmp/cont-l" "#615: --continue after the URL no longer forces continue mode"
+run "$tmp/cont-u" --update
+continued "$tmp/cont-u" "#615: --update after the URL no longer forces continue mode"
+
+# ...including where another set's letter precedes it in the cluster.
+run "$tmp/cont-pq" "-%qi"
+continued "$tmp/cont-pq" "#615: -%qi after the URL no longer forces continue mode"
+run "$tmp/cont-pm" "-%Mi"
+continued "$tmp/cont-pm" "#615: -%Mi after the URL no longer forces continue mode"
+
+# Placed before the URL these always worked; they must keep working.
+run_only "$tmp/pre-bti" "-%i" "file://$tmp/index.html"
+accepted "$tmp/pre-bti" "#615: -%i before the URL broke the crawl"
+run_only "$tmp/pre-proto" "-@i2" "file://$tmp/index.html"
+accepted "$tmp/pre-proto" "#615: -@i2 before the URL broke the crawl"
+
 exit 0

tests/01_engine-expandhome.test

@@ -0,0 +1,53 @@
+#!/bin/bash
+#
+
+# SC2088: the literal, unexpanded '~' is the input under test.
+# shellcheck disable=SC2088
+
+set -euo pipefail
+
+# ~ expansion for the -O base path (expand_home). $HOME is pinned so the cases
+# cannot depend on the caller's, but MSYS rewrites it into a Windows path
+# before the native httrack.exe reads it: ask the engine what it saw rather
+# than hardcoding the value.
+ask() {
+    HOME=HTSHOME httrack -O /dev/null -#test=expandhome "$1"
+}
+exp() {
+    test "$(ask "$1")" == "expanded=$2" || exit 1
+}
+
+home=$(ask '~')
+home=${home#expanded=}
+
+# or every case below is vacuous: '~' means expansion never fired, '.' means it
+# fell back to the default without ever reading $HOME
+test "$home" != '~' || exit 1
+test "$home" != '.' || exit 1
+
+exp '~' "$home"
+exp '~/foo' "$home/foo"
+exp '~/foo/bar/#' "$home/foo/bar/#"
+exp '~/' "$home/"
+exp '~/../x' "$home/../x"
+
+# ~user/ needs getpwnam: left alone rather than mangled into $HOME + "user/"
+exp '~smith/foo' '~smith/foo'
+exp '~root' '~root'
+
+# only a leading '~' expands; anything else is a literal path component
+exp 'a~/x' 'a~/x'
+exp './~/x' './~/x'
+exp 'foo~bar' 'foo~bar'
+
+# an unset or empty $HOME must not turn ~/foo into the absolute /foo
+test "$(env -u HOME httrack -O /dev/null -#test=expandhome '~/foo')" == "expanded=./foo" || exit 1
+test "$(HOME='' httrack -O /dev/null -#test=expandhome '~/foo')" == "expanded=./foo" || exit 1
+
+# A $HOME past the 2 * HTS_URLMAXSIZE buffer is left alone: strcatbuff aborts
+# rather than truncates. Only meaningful where the engine sees the value we set:
+# MSYS rewrites HOME into a path of its own choosing, long or not.
+if test "$home" = HTSHOME; then
+    long=$(printf '%04096d' 0 | tr '0' 'A')
+    test "$(HOME="$long" httrack -O /dev/null -#test=expandhome '~/foo')" == "expanded=~/foo" || exit 1
+fi

tests/01_engine-proxyurl.test

@@ -71,3 +71,23 @@ p 'http://a]b:c@host' 'name=http://a]b:c@host port=8080 host=host kind=http'
 
 # a lone ':' must not underflow the backward scan
 p ':' 'name= port=8080 host= kind=http'
+
+# a port is *DIGIT in 1..65535, else the scheme default: sscanf("%d") used to
+# wrap past INT_MAX and hand back a garbage port (#602)
+p 'http://host:65535' 'name=http://host port=65535 host=host kind=http'
+p 'http://host:1' 'name=http://host port=1 host=host kind=http'
+p 'http://host:65536' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:2147483648' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:4294967296' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:99999999999999' 'name=http://host port=8080 host=host kind=http'
+# discriminating: this one wrapped to a plausible 80, so range-checking the
+# sscanf result instead of rejecting the overflow still passes everything above
+p 'http://host:4294967376' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:999999999999999999999999999999' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:0' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:-1' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:80x' 'name=http://host port=8080 host=host kind=http'
+p 'http://host: 80' 'name=http://host port=8080 host=host kind=http'
+p 'http://host:' 'name=http://host port=8080 host=host kind=http'
+# the fallback is the scheme's own default, not a hardcoded 8080
+p 'socks5://host:99999999999999' 'name=socks5://host port=1080 host=host kind=socks'

tests/59_local-tls-stall.test

@@ -0,0 +1,95 @@
+#!/bin/bash
+#
+# Issue #607: a peer that accepts the TCP connect but never speaks TLS leaves the
+# slot stuck in the handshake. The per-slot --timeout must reap it; before the
+# fix only --max-time did, so these crawls ran until the kill guard.
+
+set -euo pipefail
+
+: "${top_srcdir:=..}"
+
+# shellcheck source=tests/testlib.sh
+. "$top_srcdir/tests/testlib.sh"
+
+if test "${HTTPS_SUPPORT:-}" == "no"; then
+    echo "no https support compiled, skipping"
+    exit 77
+fi
+python=$(find_python) || {
+    echo "python3 missing, skipping"
+    exit 77
+}
+
+server=$(nativepath "$top_srcdir/tests/tls-stall-server.py")
+tmpdir=$(mktemp -d)
+serverpid=
+
+cleanup() {
+    stop_server "$serverpid"
+    rm -rf "$tmpdir"
+    return 0
+}
+trap cleanup EXIT
+
+# start_stall_server <tag> <mode-args...>: sets $port from the announced one.
+start_stall_server() {
+    local tag="$1"
+    shift
+    "$python" "$server" "$@" >"$tmpdir/$tag.out" 2>"$tmpdir/$tag.err" &
+    serverpid=$!
+    port=
+    for _ in $(seq 1 50); do
+        line=$(head -n1 "$tmpdir/$tag.out" 2>/dev/null || true)
+        if test "${line%% *}" == "PORT"; then
+            port="${line#PORT }"
+            return 0
+        fi
+        kill -0 "$serverpid" 2>/dev/null || {
+            echo "server exited early: $(cat "$tmpdir/$tag.err")"
+            exit 1
+        }
+        sleep 0.1
+    done
+    echo "could not discover server port"
+    exit 1
+}
+
+# no -E/--max-time on purpose: --timeout is the only thing that can end these.
+# The kill guard stands in for the hang, so a wall time near it means no reap.
+crawl_wall() {
+    local out="$tmpdir/$1"
+    shift
+    local start
+    start=$(date +%s)
+    run_with_timeout 60 httrack -O "$out" -c1 --robots=0 --retries=0 --quiet -Z \
+        "$@" >>"$tmpdir/log" 2>&1 || true
+    echo $(($(date +%s) - start))
+}
+
+# 1. handshake stalled from the first byte: reaped at --timeout, not before.
+start_stall_server direct direct
+wall=$(crawl_wall crawl1 "https://127.0.0.1:$port/" --timeout=5)
+if test "$wall" -ge 30 || test "$wall" -lt 3; then
+    echo "FAIL: stalled handshake reaped after ${wall}s, expected about 5s" >&2
+    cat "$tmpdir/log" >&2
+    exit 1
+fi
+grep -q 'Handshake Time Out' "$tmpdir/crawl1/hts-log.txt" || {
+    echo "FAIL: crawl ended in ${wall}s but not on a handshake timeout" >&2
+    cat "$tmpdir/crawl1/hts-log.txt" >&2
+    exit 1
+}
+echo "OK: stalled TLS handshake reaped by --timeout after ${wall}s"
+
+# 2. the handshake window is its own, not what the connect left over: a proxy
+# that takes 4s to answer CONNECT must still leave the full --timeout=5 for the
+# handshake (~9s total). Sharing the connect's clock would reap at ~5s.
+stop_server "$serverpid"
+start_stall_server proxy proxy 4
+wall=$(crawl_wall crawl2 "https://127.0.0.1:443/" -P "127.0.0.1:$port" --timeout=5)
+if test "$wall" -ge 20 || test "$wall" -lt 7; then
+    echo "FAIL: handshake after a slow connect reaped at ${wall}s, expected about 9s" >&2
+    cat "$tmpdir/log" >&2
+    exit 1
+fi
+echo "OK: handshake keeps its own timeout window after a slow connect (${wall}s)"

tests/60_crawl-log-salvage.test

@@ -0,0 +1,46 @@
+#!/bin/bash
+#
+# Unit-tests dump_crawl_logs (testlib.sh): the killed test's hts-log.txt must
+# reach the uploaded logs, or a Windows wedge past --max-time stays
+# undiagnosable (#605).
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+# shellcheck source=tests/testlib.sh
+. "${testdir}/testlib.sh"
+
+TMPDIR=$(mktemp -d "${TMPDIR:-/tmp}/httrack_salv.XXXXXX")
+export TMPDIR
+trap 'rm -rf "$TMPDIR"' EXIT
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+verdict="More than 120 seconds passed.. giving up"
+
+# Vacuity control: with nothing left behind the assertions below cannot pass on
+# ambient output.
+test -z "$(dump_crawl_logs)" || fail "dumped something with no crawl tmpdir"
+
+# A killed local-crawl.sh leaves exactly this shape.
+d="${TMPDIR}/httrack_local.probe"
+mkdir -p "$d/crawl"
+echo "$verdict" >"$d/crawl/hts-log.txt"
+echo "httrack stdout marker" >"$d/log"
+
+# Append to an unterminated line, as a killed test's own log always ends.
+art="${TMPDIR}/artifact.log"
+printf '[running httrack ...] ..\t' >"$art"
+dump_crawl_logs >>"$art"
+
+grep -qF "$verdict" "$art" || fail "hts-log.txt verdict not salvaged"
+grep -qF "httrack stdout marker" "$art" || fail "httrack stdout not salvaged"
+grep -q '^--- .*hts-log.txt' "$art" || fail "dump glued its header onto the unterminated line"
+
+# Cleared, so the next timing-out test cannot re-report this crawl as its own.
+test ! -d "$d" || fail "salvaged tmpdir left behind"
+test -z "$(dump_crawl_logs)" || fail "salvaged crawl reported twice"
+
+echo "crawl log salvage OK"

tests/61_webhttrack-locale.test

@@ -0,0 +1,91 @@
+#!/bin/bash
+#
+# webhttrack must map the caller's locale to the right lang.indexes number.
+# Guards: LC_ALL/LC_MESSAGES were ignored, and zh_tw/pt_br were unreachable.
+
+set -euo pipefail
+
+testdir=$(cd "$(dirname "$0")" && pwd)
+distdir=$(cd "${testdir}/.." && pwd)
+script="${distdir}/src/webhttrack"
+
+fail() {
+    echo "FAIL: $*" >&2
+    exit 1
+}
+
+test -r "${script}" || fail "no ${script}"
+test -r "${distdir}/lang.indexes" || fail "no ${distdir}/lang.indexes"
+
+# Lift the locale code out of the launcher, so this tests the shipped logic itself.
+funcs=$(sed -n '/^function lang_index/,/^}/p' "${script}")
+# End on the next section, not a blank line: a blank line grown inside the block
+# would truncate the lift and silently stop testing everything below it.
+grep -q '^# Find the browser' "${script}" || fail "locale block terminator moved in ${script}"
+block=$(sed -n '/^# Locale/,/^# Find the browser/p' "${script}" | sed '$d')
+echo "${block}" | grep -q 'LANGN=.*lang_index' || fail "could not lift the whole locale block from ${script}"
+
+# Run the lifted block against whatever locale vars the caller exported.
+run_block() {
+    # shellcheck disable=SC2034 # read by the eval'd locale block
+    DISTPATH="${distdir}"
+    # the launcher runs under neither -e ("! test -n x && y" returns 1) nor -u
+    set +eu
+    eval "${funcs}"
+    eval "${block}"
+    # shellcheck disable=SC2153 # LANGN is set by the eval'd block, not a LANG typo
+    echo "${LANGN}"
+}
+
+# Resolve LANG/LC_ALL/LC_MESSAGES (empty = unset) to the language number.
+lang_of() {
+    (
+        unset LANG LC_ALL LC_MESSAGES
+        # only the value matters here; hide bash's setlocale gripe when the locale is not installed
+        # shellcheck disable=SC2030 # confining the locale to this subshell is the point
+        {
+            if test -n "$1"; then export LANG="$1"; fi
+            if test -n "$2"; then export LC_ALL="$2"; fi
+            if test -n "$3"; then export LC_MESSAGES="$3"; fi
+        } 2>/dev/null
+        run_block
+    )
+}
+
+check() {
+    local want=$1 got desc
+    shift
+    desc="LANG='$1' LC_ALL='$2' LC_MESSAGES='$3'"
+    got=$(lang_of "$1" "$2" "$3")
+    test "${got}" = "${want}" || fail "${desc}: want ${want}, got ${got}"
+    echo "ok: ${desc} -> ${got}"
+}
+
+#      want  LANG            LC_ALL          LC_MESSAGES
+check 2 "fr_FR.UTF-8" "" ""
+check 14 "zh_TW.UTF-8" "" ""                      # region-qualified: Chinese-BIG5, not Simplified (13)
+check 12 "pt_BR.UTF-8" "" ""                      # region-qualified: Portugues-Brasil, not Portugues (7)
+check 13 "zh_CN.UTF-8" "" ""                      # unlisted region falls back to the bare language
+check 24 "sl_SI.UTF-8" "" ""                      # Slovenian is ISO 639-1 "sl"
+check 4 "" "de_DE.UTF-8" ""                       # LC_ALL alone
+check 4 "" "" "de_DE.UTF-8"                       # LC_MESSAGES alone
+check 4 "fr_FR.UTF-8" "de_DE.UTF-8" ""            # LC_ALL wins over LANG
+check 4 "fr_FR.UTF-8" "" "de_DE.UTF-8"            # LC_MESSAGES wins over LANG
+check 2 "de_DE.UTF-8" "fr_FR.UTF-8" "de_DE.UTF-8" # LC_ALL wins over LC_MESSAGES
+check 2 "fr_FR@euro" "" ""                        # @modifier without a charset
+check 1 "C" "" ""                                 # C, unknown and unset all mean English
+check 1 "xx_YY.UTF-8" "" ""
+check 1 "" "" ""
+
+# Exported but empty is unset (POSIX), so it must not mask LANG. lang_of cannot
+# express this: it exports nothing for an empty argument.
+got=$(
+    unset LANG LC_ALL LC_MESSAGES
+    # shellcheck disable=SC2031 # confining the locale to this subshell is the point
+    { export LANG="fr_FR.UTF-8" LC_ALL="" LC_MESSAGES=""; } 2>/dev/null
+    run_block
+)
+test "${got}" = 2 || fail "empty LC_ALL/LC_MESSAGES must not mask LANG: want 2, got ${got}"
+echo "ok: LANG='fr_FR.UTF-8' with LC_ALL='' exported -> ${got}"
+
+echo "PASS"

tests/62_lang-integrity.test

@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# lang/*.txt are line pairs: an English msgid, then its translation. A stray
+# blank or a drifted msgid silently unhooks every translation that follows,
+# rather than failing, so assert the pairing and the join against English.txt.
+# Pairs are physical lines here; a msgid ending in \ continues onto the next one
+# for the engine (linput_cpp) but not for us. None do today.
+
+set -euo pipefail
+
+langdir="${top_srcdir:-..}/lang"
+def="${top_srcdir:-..}/lang.def"
+eng="$langdir/English.txt"
+for f in "$eng" "$def"; do
+    [ -f "$f" ] || {
+        echo "cannot find $f"
+        exit 1
+    }
+done
+
+tmp=$(mktemp -d)
+trap 'rm -rf "$tmp"' EXIT
+keys="$tmp/english.keys"
+
+# Byte-wise: each file is in its own declared legacy charset, lang.def is CRLF.
+LC_ALL=C awk 'NR%2==1 { sub(/\r$/, ""); print }' "$eng" >"$keys"
+
+# An empty parse would make every check below pass vacuously.
+nkeys=$(wc -l <"$keys")
+if [ "$nkeys" -lt 400 ]; then
+    echo "only $nkeys msgids parsed from $eng; the parse is broken"
+    exit 1
+fi
+
+nfiles=0
+fail=0
+for f in "$langdir"/*.txt; do
+    nfiles=$((nfiles + 1))
+    LC_ALL=C awk -v keys="$keys" -v name="${f##*/}" '
+        BEGIN { while ((getline k < keys) > 0) known[k] = 1 }
+        { sub(/\r$/, "") }
+        NR % 2 == 1 {
+            if ($0 == "")
+                printf "%s:%d: blank line where an English msgid belongs\n", name, NR
+            else if (!($0 in known))
+                printf "%s:%d: msgid absent from English.txt: %s\n", name, NR, $0
+            else
+                next
+            bad++
+        }
+        END {
+            if (NR % 2) {
+                printf "%s: odd line count (%d): msgid/translation pairing is broken\n", name, NR
+                bad++
+            }
+            exit(bad > 0)
+        }
+    ' "$f" || fail=1
+done
+
+if [ "$nfiles" -lt 25 ]; then
+    echo "only $nfiles language files found in $langdir"
+    exit 1
+fi
+
+# lang.def maps each LANG_* macro to the English string the GUI compiles in,
+# which is the lookup key into every lang/*.txt: no msgid, no translations.
+LC_ALL=C awk -v keys="$keys" '
+    BEGIN { while ((getline k < keys) > 0) known[k] = 1 }
+    { sub(/\r$/, "") }
+    NR % 2 == 1 { macro = $0; next }
+    macro ~ /^(LANG_|LISTDEF_)/ {
+        n++
+        if (!($0 in known)) {
+            printf "lang.def:%d: %s maps to \"%s\", which is not a msgid in English.txt\n", NR, macro, $0
+            bad++
+        }
+    }
+    END {
+        if (n < 400) {
+            printf "lang.def: only %d LANG_ entries parsed; the parse is broken\n", n
+            bad++
+        }
+        exit(bad > 0)
+    }
+' "$def" || fail=1
+
+[ "$fail" -eq 0 ] || exit 1
+
+echo "$nfiles language files consistent with $nkeys English msgids"

tests/Makefile.am

@@ -3,7 +3,7 @@
 # silently drop it from the dist tarball and break "make distcheck".
 EXTRA_DIST = $(TESTS) crawl-test.sh run-all-tests.sh check-network.sh \
 	proxy-https-server.py socks5-server.py proxy-connect-server.py \
-	proxytestlib.py \
+	proxytestlib.py tls-stall-server.py \
 	local-crawl.sh local-server.py testlib.sh server.crt server.key \
 	server-root/simple/basic.html server-root/simple/link.html \
 	server-root/stripquery/index.html server-root/stripquery/a.html \
@@ -59,6 +59,7 @@ TESTS = \
 	01_engine-pause.test \
 	01_engine-rcfile.test \
 	01_engine-reconcile.test \
+	01_engine-expandhome.test \
 	01_engine-fsize.test \
 	01_engine-redirect.test \
 	01_engine-relative.test \
@@ -137,6 +138,10 @@ TESTS = \
 	55_local-chunked.test \
 	56_local-proxy-noleak.test \
 	57_local-proxy-connect.test \
-	58_watchdog.test
+	58_watchdog.test \
+	59_local-tls-stall.test \
+	60_crawl-log-salvage.test \
+	61_webhttrack-locale.test \
+	62_lang-integrity.test
 
 CLEANFILES = check-network_sh.cache

tests/proxytestlib.py

@@ -18,6 +18,15 @@ PROXY_LOG = "proxy.log"
 ORIGIN_LOG = "origin-headers.log"
 
 
+def bind_ephemeral():
+    """Listening socket on a free loopback port, and that port."""
+    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+    srv.bind(("127.0.0.1", 0))
+    srv.listen(16)
+    return srv, srv.getsockname()[1]
+
+
 def pipe(src, dst):
     """Relay bytes one way until EOF, then tear both ends down."""
     try:
@@ -113,11 +122,7 @@ def handle_client(conn, logdir, mode, default_port):
 
 
 def start_proxy(logdir, mode, default_port):
-    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-    srv.bind(("127.0.0.1", 0))
-    srv.listen(16)
-    port = srv.getsockname()[1]
+    srv, port = bind_ephemeral()
 
     def accept_loop():
         while True:

tests/socks5-server.py

@@ -19,7 +19,7 @@ import threading
 # python3 -P (PYTHONSAFEPATH) drops the script's own directory from sys.path
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 
-from proxytestlib import pipe  # noqa: E402
+from proxytestlib import bind_ephemeral, pipe  # noqa: E402
 
 # The one name the proxy answers for; a .invalid TLD never resolves (RFC 6761),
 # so a locally-resolving client could not reach us -- success proves remote DNS.
@@ -180,11 +180,7 @@ def handle_socks(conn, logdir, mode):
 
 
 def start_socks(logdir, mode):
-    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-    srv.bind(("127.0.0.1", 0))
-    srv.listen(16)
-    port = srv.getsockname()[1]
+    srv, port = bind_ephemeral()
 
     def serve():
         while True:

tests/testlib.sh

@@ -44,6 +44,25 @@ stop_server() {
     return 0
 }
 
+# Dump and clear the crawl logs a hard-killed test leaves in TMPDIR (its cleanup
+# trap never ran): hts-log.txt alone records "More than N seconds passed.. giving
+# up", so a wedge past --max-time is undiagnosable without it (#605).
+dump_crawl_logs() {
+    local d f
+    for d in "${TMPDIR:-/tmp}"/httrack_local.*; do
+        test -d "$d" || continue
+        for f in "$d/crawl/hts-log.txt" "$d/log" "$d/log.2"; do
+            test -f "$f" || continue
+            # Leading newline: the killed test's last line has no terminator.
+            printf '\n--- %s (last 200 lines)\n' "$f"
+            tail -n 200 "$f"
+        done
+        # so a later test's dump cannot re-report this one; never fatal, the
+        # caller is already handling a failure and Windows may still hold a file
+        rm -rf "$d" || true
+    done
+}
+
 # Kill a backgrounded job and its whole descendant tree. POSIX: the caller must
 # have put the job in its own process group (run_with_timeout does) so we signal
 # the group; a bare kill would orphan the grandchildren. Windows: the tree is

tests/tls-stall-server.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Peers that accept a connection and never speak TLS (#607).
+
+Modes: "direct" stalls the handshake straight away; "proxy <secs>" answers a
+CONNECT after <secs> before stalling, so the handshake starts on a clock the
+connect has already eaten into. Prints "PORT <n>" once listening.
+Usage: tls-stall-server.py [direct | proxy <secs>]
+"""
+import os
+import sys
+import threading
+import time
+
+# python3 -P (PYTHONSAFEPATH) drops the script's own directory from sys.path
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+from proxytestlib import bind_ephemeral  # noqa: E402
+
+held = []  # keep every socket open: a close would fail the handshake outright
+
+
+def stall(conn, delay):
+    if delay is not None:
+        rfile = conn.makefile("rb")
+        while rfile.readline() not in (b"\r\n", b"\n", b""):
+            pass
+        time.sleep(delay)
+        conn.sendall(b"HTTP/1.0 200 Connection established\r\n\r\n")
+    held.append(conn)
+
+
+def main():
+    mode = sys.argv[1] if len(sys.argv) > 1 else "direct"
+    delay = float(sys.argv[2]) if mode == "proxy" else None
+    srv, port = bind_ephemeral()
+    sys.stdout.reconfigure(newline="\n")  # Windows would emit \r\n
+    print("PORT %d" % port, flush=True)
+    while True:
+        conn, _ = srv.accept()
+        threading.Thread(target=stall, args=(conn, delay), daemon=True).start()
+
+
+main()