tests: lock special-char URL naming across an update (#157 )

#157 reported accented URLs (pt-BR MediaWiki) losing their .html extension on an update pass, observed with 3.49-2 on Windows. It does not reproduce on current master: the update resolves the cached content-type and re-applies .html consistently, for UTF-8 and ISO-8859-1 sources, raw Latin-1 href bytes, either percent-encoding case, and dotted tails. The original symptom was a Windows codepage vs UTF-8 X-Save filename mismatch that cannot occur on a UTF-8 filesystem. Add a regression test that locks the invariant: a dotless, accented basename served as text/html, crawled then updated, must keep its .html name and not leave an extensionless sibling. Also assert in the --rerun harness that the update pass reported "files updated" (a fresh crawl never does), so a regression that bypasses the cache and silently re-crawls fresh can no longer pass the update tests. Closes #157 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Stop the 412/416 partial-reget loop on continue/update (#206 ) (#422 )
2026-06-25 11:37:28 +03:00 · 2026-06-24 22:16:30 +02:00 · 2026-06-24 21:12:40 +02:00
4 changed files with 37 additions and 2 deletions
--- a/tests/21_local-intl-update.test
+++ b/tests/21_local-intl-update.test
@@ -0,0 +1,11 @@
+#!/bin/bash
+#
+# #157: a dotless, accented URL named .html on the first crawl must keep .html
+# across an update -- not revert to the extensionless name.
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" --errors 0 --rerun \
+    --found 'intl/Instalação_CVS_no_Ubuntu.html' \
+    --not-found 'intl/Instalação_CVS_no_Ubuntu' \
+    httrack 'BASEURL/intl/index.html'
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -60,6 +60,7 @@ TESTS = \
 	17_local-empty-ct.test \
 	18_local-update.test \
 	19_local-connect-fallback.test \
-	20_local-resume-loop.test
+	20_local-resume-loop.test \
+	21_local-intl-update.test

 CLEANFILES = check-network_sh.cache
--- a/tests/local-crawl.sh
+++ b/tests/local-crawl.sh
@@ -196,6 +196,15 @@ if test -n "$rerun"; then
        exit 1
    }
    result "OK (update)"
+    # The update summary reports "files updated"; a fresh crawl never does. Assert
+    # it so a regression that bypasses the cache (re-crawls fresh) can't pass.
+    info "checking update used the cache"
+    if grep -aqE "mirror complete in .*files updated" "${out}/hts-log.txt"; then
+        result "OK"
+    else
+        result "update pass did not report cache activity"
+        exit 1
+    fi
 fi

 # --- discover the single host root (127.0.0.1_<port> or 127.0.0.1) -----------
--- a/tests/local-server.py
+++ b/tests/local-server.py
@@ -177,6 +177,17 @@ class Handler(SimpleHTTPRequestHandler):
        body, ctype = self.TYPE_MATRIX[path]
        self.send_raw(body, ctype)

+    # --- special chars in URLs across an update (issue #157) ---------------
+    # A dotless, accented basename served as text/html (MediaWiki style). The
+    # name the first crawl picks (.html) must survive the update pass.
+    INTL_NAME = "Instalação_CVS_no_Ubuntu"
+
+    def route_intl_index(self):
+        self.send_html('\t<a href="%s">accented</a>\n' % self.INTL_NAME)
+
+    def route_intl_page(self):
+        self.send_raw(b"<html><body>accented page</body></html>\n", "text/html")
+
    # resume / 416 loop (#206): the first GET stalls after a prefix so the crawl
    # can be interrupted (partial + temp-ref); every later request is 416.
    RESUME_PREFIX = b"PARTIAL-" + b"x" * 4096  # flushed before the stall
@@ -233,6 +244,8 @@ class Handler(SimpleHTTPRequestHandler):
        "/types/style.css": route_types,
        "/types/data.json": route_types,
        "/types/gen.php": route_types,
+        "/intl/index.html": route_intl_index,
+        "/intl/" + INTL_NAME: route_intl_page,
        "/resume/index.html": route_resume_index,
        "/resume/blob.txt": route_resume,
    }
@@ -242,7 +255,8 @@ class Handler(SimpleHTTPRequestHandler):
    def dispatch(self):
        self._set_cookies = []
        path = urlsplit(self.path).path
-        handler = self.ROUTES.get(path)
+        # Match percent-encoded paths (accented #157 route) by their decoded form.
+        handler = self.ROUTES.get(path) or self.ROUTES.get(unquote(path))
        if handler is not None:
            handler(self)
            return True