Compare commits

..

4 Commits

Author SHA1 Message Date
Xavier Roche
45f56e6194 Export hts_convertStringSystemToUTF8 from the DLL
Mark the CP_ACP->UTF-8 converter HTSEXT_API, matching hts_argv_utf8 next
to it, so WinHTTrack can call the engine's converter instead of keeping a
private copy. Windows-only (inside the _WIN32 block), so no POSIX ABI
change and no soname bump.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-15 07:32:09 +02:00
Xavier Roche
d9ff780d0f Let a failed connect fall back to the next address on Windows (#580)
Winsock reports a failed non-blocking connect in select()'s exception set,
not as writable. The exception loop in back_wait runs before the connect
handler and finished the slot with STATUSCODE_CONNERROR, so back_connect_next
never got a chance and a multi-homed host with one dead address failed the
whole fetch.

Skip connecting slots in that loop and treat their presence in the exception
set as "the connect finished": SO_ERROR is probed as on POSIX, and the next
resolved address is dialed.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 07:20:44 +02:00
Xavier Roche
b026c80bb6 Run the engine test suite on Windows (#577)
* Cap the wildcard matcher's recursion depth (#574)

strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.

Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.

The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Take argv as UTF-8 on Windows, not the ANSI codepage (#573)

httrack.exe is an MBCS build, so the CRT transcodes the UTF-16 command
line down to the machine's ANSI codepage before handing us char **argv.
Non-ASCII arguments are lossy, and anything outside that codepage is
destroyed: a Cyrillic or CJK URL on a Western-codepage box becomes '?'.

Everything else in the engine already treats char* as UTF-8 on Windows -
FOPEN, STAT, UNLINK are hts_*_utf8 wrappers converting to UTF-16 at the
syscall boundary. argv is the one thing that never got the memo.

Decode the real UTF-16 command line at the entry point instead. The
manifest's activeCodePage would fix it in one line, but it needs Windows
10 1903+ and is silently ignored below that, which is precisely the
population running the codepages that mangle non-Latin URLs today.

Windows-only, so no POSIX symbol and no soname change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Make the engine tests pass on Windows

Three of them could never pass there, and none of the three was an engine
bug:

- rcfile wrote .httrackrc, but HTS_HTTRACKRC is "httrackrc" on Windows,
  so the engine never found it. Write both names.
- filelist made a file unreadable with chmod 000, which Windows ignores.
  Probe whether the mode is enforced instead of assuming, which also
  subsumes the root special-case.
- ftp-line was #ifndef _WIN32 for its socketpair(), though get_ftp_line()
  itself is portable. Pair over loopback TCP and register it everywhere.

The charset self-tests fed raw non-UTF-8 bytes through argv to probe the
decoders. Those bytes cannot survive any Windows command line, whatever
argv does, so take them as hex - the convention the sniff self-test
already uses - and keep the coverage on every platform.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Run the engine test suite on Windows CI

The ~90 tests only ever ran on Linux and macOS, so a Windows-only
regression was invisible until a user hit it - and the first run of the
offline ones found a third of them failing.

Drive them from Git Bash against the native MSVC httrack.exe. Two things
that silently ruin the run: MSYS rewrites any argument shaped like a POSIX
path, and a URL path is shaped exactly like one, so "-#test=mime /a/b.html"
reached the engine as "C:/Program Files/Git/a/b.html"; and a suite that
degrades to all-skipped would report green having tested nothing, hence
the floor on tests actually passed.

This subsumes the codec and cache self-tests the workflow ran inline. The
.gitattributes pins the test scripts to LF: a converting checkout rewrites
them to CRLF and bash then dies on $'\r' on every line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Log the launch banner argv without re-encoding it

The banner ran hts_convertStringSystemToUTF8() over each argv element on
Windows, which was right when argv came in the ANSI codepage. argv is
UTF-8 on every platform now, so that pass double-encoded a non-ASCII
argument into mojibake in hts-log.txt. Log the bytes as they are.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix the Win32 CLI build: include htscharset.h after winsock2.h

htscharset.h pulls in <windows.h>, which drags in the legacy <winsock.h>
unless <winsock2.h> was included first. In httrack.c it sat ahead of the
net headers, so the x86 build redefined every sockaddr/winsock symbol
(C2011). Move it below htslib.h, where winsock2.h is already in.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix two more tests the Win32 CI leg exposed

Both are test bugs surfaced by running on 32-bit Windows for the first
time, not engine bugs:

- strsafe drove its overflow through a char[4]. On a 32-bit build that
  equals sizeof(char*), so htssafe's array-vs-pointer heuristic
  (sizeof(A) != sizeof(char*), the MSVC path) reads the array as a
  pointer and skips the bound: the copy then genuinely overflows and
  crashes instead of aborting cleanly. Size the buffer off the pointer
  width so the checked path runs everywhere.

- idna's malformed-UTF-8 rejection case passed the bad bytes through
  argv. They cannot survive a Windows command line: the UTF-16 round-trip
  turns them into valid U+FFFD, which the encoder then accepts. Feed them
  as hex so the exact bytes reach the bundled punycode encoder, which
  rejects them identically on every platform.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 20:03:40 +02:00
Xavier Roche
e0648beff9 Cap the wildcard matcher's recursion depth (#575)
* Cap the wildcard matcher's recursion depth (#574)

strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.

Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.

The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Pin the matcher depth cap from below in the self-test

The self-test asserted the cap is reached, which a cap set absurdly low
would also satisfy: real filters use fewer than ten segments, so a cap of
5 would break them and still ship green. Assert a 32-segment pattern still
matches, so the cap cannot be lowered past real use unnoticed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 19:49:39 +02:00
4 changed files with 23 additions and 12 deletions

View File

@@ -140,8 +140,9 @@ int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
/* Pending-connect result for a non-blocking socket reported ready by select():
0 = connected, >0 = the connect errno (refused, unreachable, ...), -1 if the
probe itself failed. A failed connect is reported writable too, so this is
how success is told from failure without blocking. */
probe itself failed. A failed connect is reported ready as well (writable on
posix, exception set on winsock), so this is how success is told from failure
without blocking. */
static int connect_socket_error(T_SOC soc) {
int soerr = 0;
socklen_t len = (socklen_t) sizeof(soerr);
@@ -2658,7 +2659,9 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
for (i_mod = 0; i_mod < (unsigned int) back_max; i_mod++) {
unsigned int i = (i_mod + mod_random) % (back_max);
if (back[i].status > 0) {
// winsock flags a failed connect in the exception set only: leave a
// connecting slot to the connect handler, which can still fall back
if (back[i].status > 0 && back[i].status != STATUS_CONNECTING) {
if (!back[i].r.is_file) { // not file..
if (back[i].r.soc != INVALID_SOCKET) { // hey, you never know..
int err = FD_ISSET(back[i].r.soc, &fds_e);
@@ -2672,10 +2675,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
}
back[i].r.soc = INVALID_SOCKET;
back[i].r.statuscode = STATUSCODE_CONNERROR;
if (back[i].status == STATUS_CONNECTING)
strcpybuff(back[i].r.msg, "Connect Error");
else
strcpybuff(back[i].r.msg, "Receive Error");
strcpybuff(back[i].r.msg, "Receive Error");
if (back[i].status == STATUS_ALIVE) { /* Keep-alive socket */
back_delete(opt, cache, sback, i);
} else {
@@ -2708,10 +2708,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
gestion_timeout = 1;
// connecté?
dispo = FD_ISSET(back[i].r.soc, &fds_c);
dispo = back[i].r.soc != INVALID_SOCKET &&
(FD_ISSET(back[i].r.soc, &fds_c) ||
FD_ISSET(back[i].r.soc, &fds_e));
if (dispo) { // socket ready: connect() finished (ok or failed)
// a refused/failed connect is reported writable too; probe SO_ERROR
// and, on failure, fall back to the next address (or fail the slot)
// probe SO_ERROR and, on failure, fall back to the next address
if (connect_socket_error(back[i].r.soc) != 0) {
if (!back_connect_next(opt, sback, i)) {
deletehttp(&back[i].r);

View File

@@ -385,7 +385,7 @@ char *hts_convertStringFromUTF8(const char *s, size_t size, const char *charset)
return hts_convertStringCPFromUTF8(s, size, cp);
}
char *hts_convertStringSystemToUTF8(const char *s, size_t size) {
HTSEXT_API char *hts_convertStringSystemToUTF8(const char *s, size_t size) {
return hts_convertStringCPToUTF8(s, size, GetACP());
}

View File

@@ -175,7 +175,7 @@ extern char *hts_convertUCS2StringToUTF8(LPWSTR woutput, int wsize);
* Convert current system codepage to UTF-8.
* This function is WIN32 specific.
**/
extern char *hts_convertStringSystemToUTF8(const char *s, size_t size);
HTSEXT_API char *hts_convertStringSystemToUTF8(const char *s, size_t size);
/**
* Replace the CRT's ANSI argv by a UTF-8 one decoded from the real UTF-16

View File

@@ -784,6 +784,16 @@ static int st_filterbounds(httrackp *opt, int argc, char **argv) {
/* Depth caps the stack: uncapped this recurses 2046 frames, ~900KB (#574). */
assertf(depth == maxdepth);
assertf(strjokerfind(subj, pat) == NULL);
/* Pin the cap from below: 32 segments must still match, so a cap set so low
it would break real multi-segment filters (which use far fewer) fails. */
for (i = 0; i < 32; i++) {
pat[2 * i] = '*';
pat[2 * i + 1] = 'a';
}
pat[64] = '\0';
memset(subj, 'a', 32);
subj[32] = '\0';
assertf(strjoker(subj, pat, NULL, NULL) != NULL);
freet(pat);
freet(subj);
printf("filterbounds: OK\n");