Compare commits

..

7 Commits

Author SHA1 Message Date
Xavier Roche
d9d25196bb Fix two more tests the Win32 CI leg exposed
Both are test bugs surfaced by running on 32-bit Windows for the first
time, not engine bugs:

- strsafe drove its overflow through a char[4]. On a 32-bit build that
  equals sizeof(char*), so htssafe's array-vs-pointer heuristic
  (sizeof(A) != sizeof(char*), the MSVC path) reads the array as a
  pointer and skips the bound: the copy then genuinely overflows and
  crashes instead of aborting cleanly. Size the buffer off the pointer
  width so the checked path runs everywhere.

- idna's malformed-UTF-8 rejection case passed the bad bytes through
  argv. They cannot survive a Windows command line: the UTF-16 round-trip
  turns them into valid U+FFFD, which the encoder then accepts. Feed them
  as hex so the exact bytes reach the bundled punycode encoder, which
  rejects them identically on every platform.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 19:08:34 +02:00
Xavier Roche
d6153cbc86 Fix the Win32 CLI build: include htscharset.h after winsock2.h
htscharset.h pulls in <windows.h>, which drags in the legacy <winsock.h>
unless <winsock2.h> was included first. In httrack.c it sat ahead of the
net headers, so the x86 build redefined every sockaddr/winsock symbol
(C2011). Move it below htslib.h, where winsock2.h is already in.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 18:52:25 +02:00
Xavier Roche
2438f55327 Log the launch banner argv without re-encoding it
The banner ran hts_convertStringSystemToUTF8() over each argv element on
Windows, which was right when argv came in the ANSI codepage. argv is
UTF-8 on every platform now, so that pass double-encoded a non-ASCII
argument into mojibake in hts-log.txt. Log the bytes as they are.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 18:41:10 +02:00
Xavier Roche
0e5c2d6ee9 Run the engine test suite on Windows CI
The ~90 tests only ever ran on Linux and macOS, so a Windows-only
regression was invisible until a user hit it - and the first run of the
offline ones found a third of them failing.

Drive them from Git Bash against the native MSVC httrack.exe. Two things
that silently ruin the run: MSYS rewrites any argument shaped like a POSIX
path, and a URL path is shaped exactly like one, so "-#test=mime /a/b.html"
reached the engine as "C:/Program Files/Git/a/b.html"; and a suite that
degrades to all-skipped would report green having tested nothing, hence
the floor on tests actually passed.

This subsumes the codec and cache self-tests the workflow ran inline. The
.gitattributes pins the test scripts to LF: a converting checkout rewrites
them to CRLF and bash then dies on $'\r' on every line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 18:24:54 +02:00
Xavier Roche
ae0298aac6 Make the engine tests pass on Windows
Three of them could never pass there, and none of the three was an engine
bug:

- rcfile wrote .httrackrc, but HTS_HTTRACKRC is "httrackrc" on Windows,
  so the engine never found it. Write both names.
- filelist made a file unreadable with chmod 000, which Windows ignores.
  Probe whether the mode is enforced instead of assuming, which also
  subsumes the root special-case.
- ftp-line was #ifndef _WIN32 for its socketpair(), though get_ftp_line()
  itself is portable. Pair over loopback TCP and register it everywhere.

The charset self-tests fed raw non-UTF-8 bytes through argv to probe the
decoders. Those bytes cannot survive any Windows command line, whatever
argv does, so take them as hex - the convention the sniff self-test
already uses - and keep the coverage on every platform.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 18:24:54 +02:00
Xavier Roche
8a4ca8357a Take argv as UTF-8 on Windows, not the ANSI codepage (#573)
httrack.exe is an MBCS build, so the CRT transcodes the UTF-16 command
line down to the machine's ANSI codepage before handing us char **argv.
Non-ASCII arguments are lossy, and anything outside that codepage is
destroyed: a Cyrillic or CJK URL on a Western-codepage box becomes '?'.

Everything else in the engine already treats char* as UTF-8 on Windows -
FOPEN, STAT, UNLINK are hts_*_utf8 wrappers converting to UTF-16 at the
syscall boundary. argv is the one thing that never got the memo.

Decode the real UTF-16 command line at the entry point instead. The
manifest's activeCodePage would fix it in one line, but it needs Windows
10 1903+ and is silently ignored below that, which is precisely the
population running the codepages that mangle non-Latin URLs today.

Windows-only, so no POSIX symbol and no soname change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 18:24:37 +02:00
Xavier Roche
1184839060 Cap the wildcard matcher's recursion depth (#574)
strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.

Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.

The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-14 18:15:26 +02:00
4 changed files with 12 additions and 23 deletions

View File

@@ -140,9 +140,8 @@ int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
/* Pending-connect result for a non-blocking socket reported ready by select():
0 = connected, >0 = the connect errno (refused, unreachable, ...), -1 if the
probe itself failed. A failed connect is reported ready as well (writable on
posix, exception set on winsock), so this is how success is told from failure
without blocking. */
probe itself failed. A failed connect is reported writable too, so this is
how success is told from failure without blocking. */
static int connect_socket_error(T_SOC soc) {
int soerr = 0;
socklen_t len = (socklen_t) sizeof(soerr);
@@ -2659,9 +2658,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
for (i_mod = 0; i_mod < (unsigned int) back_max; i_mod++) {
unsigned int i = (i_mod + mod_random) % (back_max);
// winsock flags a failed connect in the exception set only: leave a
// connecting slot to the connect handler, which can still fall back
if (back[i].status > 0 && back[i].status != STATUS_CONNECTING) {
if (back[i].status > 0) {
if (!back[i].r.is_file) { // not file..
if (back[i].r.soc != INVALID_SOCKET) { // hey, you never know..
int err = FD_ISSET(back[i].r.soc, &fds_e);
@@ -2675,7 +2672,10 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
}
back[i].r.soc = INVALID_SOCKET;
back[i].r.statuscode = STATUSCODE_CONNERROR;
strcpybuff(back[i].r.msg, "Receive Error");
if (back[i].status == STATUS_CONNECTING)
strcpybuff(back[i].r.msg, "Connect Error");
else
strcpybuff(back[i].r.msg, "Receive Error");
if (back[i].status == STATUS_ALIVE) { /* Keep-alive socket */
back_delete(opt, cache, sback, i);
} else {
@@ -2708,11 +2708,10 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
gestion_timeout = 1;
// connecté?
dispo = back[i].r.soc != INVALID_SOCKET &&
(FD_ISSET(back[i].r.soc, &fds_c) ||
FD_ISSET(back[i].r.soc, &fds_e));
dispo = FD_ISSET(back[i].r.soc, &fds_c);
if (dispo) { // socket ready: connect() finished (ok or failed)
// probe SO_ERROR and, on failure, fall back to the next address
// a refused/failed connect is reported writable too; probe SO_ERROR
// and, on failure, fall back to the next address (or fail the slot)
if (connect_socket_error(back[i].r.soc) != 0) {
if (!back_connect_next(opt, sback, i)) {
deletehttp(&back[i].r);

View File

@@ -385,7 +385,7 @@ char *hts_convertStringFromUTF8(const char *s, size_t size, const char *charset)
return hts_convertStringCPFromUTF8(s, size, cp);
}
HTSEXT_API char *hts_convertStringSystemToUTF8(const char *s, size_t size) {
char *hts_convertStringSystemToUTF8(const char *s, size_t size) {
return hts_convertStringCPToUTF8(s, size, GetACP());
}

View File

@@ -175,7 +175,7 @@ extern char *hts_convertUCS2StringToUTF8(LPWSTR woutput, int wsize);
* Convert current system codepage to UTF-8.
* This function is WIN32 specific.
**/
HTSEXT_API char *hts_convertStringSystemToUTF8(const char *s, size_t size);
extern char *hts_convertStringSystemToUTF8(const char *s, size_t size);
/**
* Replace the CRT's ANSI argv by a UTF-8 one decoded from the real UTF-16

View File

@@ -784,16 +784,6 @@ static int st_filterbounds(httrackp *opt, int argc, char **argv) {
/* Depth caps the stack: uncapped this recurses 2046 frames, ~900KB (#574). */
assertf(depth == maxdepth);
assertf(strjokerfind(subj, pat) == NULL);
/* Pin the cap from below: 32 segments must still match, so a cap set so low
it would break real multi-segment filters (which use far fewer) fails. */
for (i = 0; i < 32; i++) {
pat[2 * i] = '*';
pat[2 * i + 1] = 'a';
}
pat[64] = '\0';
memset(subj, 'a', 32);
subj[32] = '\0';
assertf(strjoker(subj, pat, NULL, NULL) != NULL);
freet(pat);
freet(subj);
printf("filterbounds: OK\n");