Compare commits

..

5 Commits

Author SHA1 Message Date
Xavier Roche
9de0878622 Never let the log salvage's cleanup fail the test
dump_crawl_logs runs on the timeout path, where the caller is already
handling a failure: rm -rf can fail on Windows while the killed httrack
still holds a file, and under set -e that would abort the branch whose
whole job is to salvage the evidence.

Renumber the test to 60: #611 adds a 59 too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-17 06:47:04 +02:00
Xavier Roche
fecb67e139 Salvage a killed crawl's logs into the Windows test artifact
A test the suite watchdog kills at 600s left nothing to diagnose it with.
local-crawl.sh sends the crawl's output to its tmpdir and hts-log.txt lives
there too, but the artifact only uploads tests/*.log, so the killed test's
entry was a single truncated line. The hard kill also skips local-crawl.sh's
cleanup trap, so the tmpdir survives on the runner and is then discarded.

Dump those logs into the test's own .log on a timeout, and clear the tmpdir so
a later timeout cannot re-report it. hts-log.txt is where "More than N seconds
passed.. giving up" lands, which is the question #605 asks of a wedge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-17 06:47:04 +02:00
Xavier Roche
7be61d3111 Synchronous DNS resolution can wedge a crawl past --max-time (#613)
* Bound DNS resolution so a slow resolver cannot outlast --max-time

host_wait() is a stub returning 1, so a STATUS_WAIT_DNS slot resolves
synchronously through hts_dns_resolve_all() -> getaddrinfo() on the engine
thread. Nothing bounded that call, and it ran holding opt->state.lock, so a
black-hole resolver stalled the crawl past --max-time and --timeout and blocked
hts_request_stop()/hts_has_stopped() with it. resolv.conf caps getaddrinfo on
POSIX and hides this most of the time; Windows has no equivalent.

Resolve a cache miss on a worker thread and wait at most opt->timeout for it,
with the lock held only for the cache lookup and the store. A timed-out resolve
is abandoned rather than cancelled (getaddrinfo has no portable cancellation),
so the job is refcounted and the worker touches nothing but it. A timeout is
reported as "does not resolve" but never cached, so the host is retried instead
of written off for the rest of the crawl.

This bounds the wedge; it does not make resolution asynchronous. The engine
thread still stalls other slots for up to --timeout per resolve, which only
driving STATUS_WAIT_DNS from the poll loop would fix.

New -#test=dnstimeout self-test drives a mock resolver that blocks for 3s past
a 1s --timeout; all five of its checks fail on master.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Derive the DNS timeout assertion from the spec, and join the mock workers

The self-test's ceiling came from the mock's sleep (MOCK_SLOW_MS - 500), not
from opt->timeout: a resolve using a hardcoded 2400ms deadline that ignored
opt->timeout entirely still passed it. Derive the bound from opt->timeout so
the assertion tests the contract instead of the fixture.

The scaffolding also raced its own abandoned workers. mock_host.calls was read
while a worker was still writing it, and tearing the backend down at the end
raced the worker's freeaddrinfo read, ordered only by a sleep. Serialize the
counters under a mutex, wait for the backend calls to actually return instead
of guessing, and leave the backend installed, since an abandoned worker still
reads it. Drops the drain sleep: 6.0s -> 4.0s.

Also: --timeout now bounds resolution, so say so in the help text (the man page
is generated from it), and factor the three address-copy loops into a helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 06:43:47 +02:00
Xavier Roche
3cc49b79c3 Stalled TLS handshake ignores --timeout (#611)
* Reap a stalled TLS handshake with the per-slot --timeout

back_wait only runs its per-slot timeout check when the local gestion_timeout
flag is armed. The CONNECTING, WAIT_DNS and receiving handlers arm it, but the
STATUS_SSL_WAIT_HANDSHAKE handler did not, so a peer that completes the TCP
connect and never speaks TLS left SSL_connect returning WANT_READ until
--max-time fired, ignoring --timeout entirely.

Arm the flag in the handshake handler and start a fresh timeout window when the
slot enters the handshake, so it is measured from there rather than from the
connect. The generic check already reaps any status > 0 slot once armed; give it
a distinct message instead of the generic "Receive Time Out".

Test 59 crawls a server that accepts the connect and stays silent: it ends in
--timeout seconds with the fix, and hangs until the kill guard without it.

Closes #607

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Cover the handshake timeout window, and share the ephemeral-bind helper

Test 59 only bounded the crawl from above, so an engine that reaped every
handshake instantly passed it, and nothing exercised the timeout_refresh at the
handshake entry: on loopback the connect is instant, so the handshake's own
window is indistinguishable from the connect's.

Add a floor to the first case, and a second one behind a proxy that takes 4s to
answer CONNECT. The handshake must still get its full --timeout=5 from there
(~9s); sharing the connect's clock reaps at ~5s. Dropping the refresh now fails
the test, as does collapsing the window to zero.

The stall server grows a proxy mode for that, and takes its listening socket
from a new proxytestlib bind_ephemeral(), which replaces the same boilerplate
in the socks5 and proxy servers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 06:39:11 +02:00
Xavier Roche
959deb0afb An out-of-range -P proxy port parses to a garbage port instead of being rejected (#610)
* Reject an out-of-range -P proxy port instead of wrapping it

hts_parse_proxy read the port with sscanf(a, "%d", &p), which is signed
overflow UB past INT_MAX. glibc wraps rather than fails, so a garbage
port got through: -P 'http://host:99999999999999' parsed to port
276447231, while 'http://host:4294967296' happened to fail and fall back
to the default. 65536 was accepted as-is.

Parse with strtol and range-check to *DIGIT in 1..65535 (RFC 3986); an
out-of-range or malformed port now falls back to proxy_default_port,
exactly as an unparsable one already did.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Attribute the proxy port range to TCP, not RFC 3986

RFC 3986 defines port as *DIGIT with no range; the 1..65535 bound comes from
TCP's 16-bit port field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 06:37:58 +02:00
13 changed files with 437 additions and 59 deletions

View File

@@ -158,13 +158,21 @@ jobs:
pass=0 fail=0 skip=0 failed="" skipped=""
for t in 00_runnable.test 01_engine-*.test 01_zlib-*.test \
*_local-*.test 13_crawl_proxy_https.test 58_watchdog.test; do
*_local-*.test 13_crawl_proxy_https.test 58_watchdog.test \
60_crawl-log-salvage.test; do
rc=0
run_with_timeout 600 bash "$t" >"$t.log" 2>&1 || rc=$?
case "$rc" in
0) pass=$((pass + 1)); echo "PASS $t" ;;
77) skip=$((skip + 1)) skipped="$skipped $t"; echo "SKIP $t" ;;
124) fail=$((fail + 1)) failed="$failed $t"; echo "FAIL $t (timed out, tree killed)" ;;
124)
fail=$((fail + 1)) failed="$failed $t"
echo "FAIL $t (timed out, tree killed)"
# Re-running a wedge traced would just hang again: salvage the
# killed crawl's own logs into the artifact instead.
dump_crawl_logs >>"$t.log"
tail -n 25 "$t.log" | sed 's/^/ /'
;;
*)
fail=$((fail + 1)) failed="$failed $t"
echo "FAIL $t (exit $rc)"

View File

@@ -3,7 +3,7 @@
.\"
.\" This file is generated by man/makeman.sh; do not edit by hand.
.\" SPDX-License-Identifier: GPL-3.0-or-later
.TH httrack 1 "14 July 2026" "httrack website copier"
.TH httrack 1 "16 July 2026" "httrack website copier"
.SH NAME
httrack \- offline browser : copy websites to a local directory
.SH SYNOPSIS
@@ -165,7 +165,7 @@ pause transfer if N bytes reached, and wait until lock file is deleted (\-\-max\
.IP \-cN
number of multiple connections (*c8) (\-\-sockets[=N])
.IP \-TN
timeout, number of seconds after a non\-responding link is shutdown (\-\-timeout[=N])
timeout, number of seconds after a non\-responding link is shutdown; also bounds host name resolution (\-\-timeout[=N])
.IP \-RN
number of retries, in case of timeout or non\-fatal errors (*R1) (\-\-retries[=N])
.IP \-JN

View File

@@ -63,9 +63,14 @@ typedef struct mock_host {
int gai_err; /* non-zero: getaddrinfo returns this */
int naddr;
mock_addr addr[6];
int calls; /* times the backend resolved this host */
int calls; /* times the backend resolved this host */
int slow_ms; /* non-zero: block this long, as a black-hole resolver would */
} mock_host;
/* Long enough to outlast the 1s --timeout the bounded resolve is checked
against, short enough to keep the self-test quick. */
#define MOCK_SLOW_MS 3000
static mock_host mock_hosts[] = {
{"v4only.test", 0, 1, {{AF_INET, {1, 2, 3, 4}}}, 0},
{"v6only.test", 0, 1, {{AF_INET6, {0x20, 0x01, 0x0d, 0xb8, [15] = 1}}}, 0},
@@ -95,8 +100,15 @@ static mock_host mock_hosts[] = {
{AF_INET, {10, 0, 0, 6}}},
0},
{"nodns.test", EAI_NONAME, 0, {{0}}, 0},
/* resolves, but only well after --timeout: the #606 wedge */
{"slow.test", 0, 1, {{AF_INET, {127, 0, 0, 9}}}, 0, MOCK_SLOW_MS},
};
/* Serializes mock_host bookkeeping: a timed-out resolve is abandoned, so its
worker is still inside the backend while the test reads the counters. */
static htsmutex mock_lock = HTSMUTEX_INIT;
static int mock_finished = 0; /* backend calls that have returned */
static mock_host *mock_find(const char *name) {
for (size_t i = 0; i < sizeof(mock_hosts) / sizeof(mock_hosts[0]); i++) {
if (strcmp(mock_hosts[i].name, name) == 0)
@@ -106,8 +118,34 @@ static mock_host *mock_find(const char *name) {
}
static void mock_reset_calls(void) {
hts_mutexlock(&mock_lock);
for (size_t i = 0; i < sizeof(mock_hosts) / sizeof(mock_hosts[0]); i++)
mock_hosts[i].calls = 0;
mock_finished = 0;
hts_mutexrelease(&mock_lock);
}
static int mock_read_calls(const char *name) {
int calls;
hts_mutexlock(&mock_lock);
calls = mock_find(name)->calls;
hts_mutexrelease(&mock_lock);
return calls;
}
/* Wait for n backend calls to return, ordering their writes against ours. */
static void mock_wait_finished(int n) {
for (;;) {
hts_boolean done;
hts_mutexlock(&mock_lock);
done = (mock_finished >= n) ? HTS_TRUE : HTS_FALSE;
hts_mutexrelease(&mock_lock);
if (done)
break;
Sleep(10);
}
}
/* Build one addrinfo node owning its sockaddr (freed by mock_freeaddrinfo). */
@@ -133,10 +171,10 @@ static struct addrinfo *mock_mkai(const mock_addr *a) {
return ai;
}
static int HTS_RESOLVER_CALL mock_getaddrinfo(const char *node,
const char *service,
const struct addrinfo *hints,
struct addrinfo **res) {
static int HTS_RESOLVER_CALL mock_getaddrinfo_(const char *node,
const char *service,
const struct addrinfo *hints,
struct addrinfo **res) {
mock_host *const h = mock_find(node);
const int want = (hints != NULL) ? hints->ai_family : PF_UNSPEC;
struct addrinfo *head = NULL, *tail = NULL;
@@ -145,7 +183,11 @@ static int HTS_RESOLVER_CALL mock_getaddrinfo(const char *node,
*res = NULL;
if (h == NULL)
return EAI_NONAME;
hts_mutexlock(&mock_lock);
h->calls++; /* a real backend hit; a cached host skips this */
hts_mutexrelease(&mock_lock);
if (h->slow_ms != 0)
Sleep(h->slow_ms);
if (h->gai_err != 0)
return h->gai_err;
for (int i = 0; i < h->naddr; i++) {
@@ -165,6 +207,18 @@ static int HTS_RESOLVER_CALL mock_getaddrinfo(const char *node,
return 0;
}
static int HTS_RESOLVER_CALL mock_getaddrinfo(const char *node,
const char *service,
const struct addrinfo *hints,
struct addrinfo **res) {
const int ret = mock_getaddrinfo_(node, service, hints, res);
hts_mutexlock(&mock_lock);
mock_finished++;
hts_mutexrelease(&mock_lock);
return ret;
}
static void HTS_RESOLVER_CALL mock_freeaddrinfo(struct addrinfo *res) {
while (res != NULL) {
struct addrinfo *const next = res->ai_next;
@@ -370,6 +424,91 @@ int dns_selftests(httrackp *opt) {
return failures;
}
/* Probes how long acquiring opt->state.lock takes while a resolve is in
flight. hts_has_stopped() takes that same lock and mutates nothing, and the
API promises it stays callable from another thread during a mirror. */
typedef struct lock_probe {
httrackp *opt;
htsmutex lock;
hts_boolean done;
TStamp blocked_ms;
} lock_probe;
static void lock_probe_thread(void *arg) {
lock_probe *const p = (lock_probe *) arg;
TStamp start;
Sleep(MOCK_SLOW_MS / 10); /* let the resolve get under way first */
start = mtime_local();
(void) hts_has_stopped(p->opt);
hts_mutexlock(&p->lock);
p->blocked_ms = mtime_local() - start;
p->done = HTS_TRUE;
hts_mutexrelease(&p->lock);
}
int dns_timeout_selftests(httrackp *opt) {
SOCaddr addrs[HTS_MAXADDRNUM];
const char *err = NULL;
lock_probe probe;
TStamp start, elapsed;
int count;
failures = 0;
hts_dns_set_resolver_backend(&mock_backend);
IPV6_resolver = 0;
mock_reset_calls();
opt->timeout = 1; /* the bound under test */
memset(&probe, 0, sizeof(probe));
probe.opt = opt;
probe.lock = HTSMUTEX_INIT;
hts_mutexinit(&probe.lock);
CHECK(hts_newthread(lock_probe_thread, &probe) == 0);
start = mtime_local();
count = hts_dns_resolve_all(opt, "slow.test", addrs, HTS_MAXADDRNUM, &err);
elapsed = mtime_local() - start;
/* the resolve returns on opt->timeout, not when the resolver deigns to
answer: this is what lets --max-time and --timeout fire (#606). The bound
is derived from opt->timeout, never from the mock's sleep, or a resolve
that ignored opt->timeout would still pass under the mock. */
CHECK(elapsed < (TStamp) opt->timeout * 1000 + 500);
CHECK(count == 0); /* a timeout is reported as "does not resolve" */
/* state.lock is not held across the resolve; a concurrent stop query, which
the mirror API promises stays live, is not blocked behind it */
for (;;) {
hts_boolean done;
TStamp blocked;
hts_mutexlock(&probe.lock);
done = probe.done;
blocked = probe.blocked_ms;
hts_mutexrelease(&probe.lock);
if (done) {
CHECK(blocked < 500);
break;
}
Sleep(20);
}
hts_mutexfree(&probe.lock);
/* a timeout is not an answer, so it must not be negative-cached: the host is
resolved again rather than written off for the rest of the crawl */
CHECK(mock_read_calls("slow.test") == 1);
count = hts_dns_resolve_all(opt, "slow.test", addrs, HTS_MAXADDRNUM, &err);
CHECK(count == 0);
CHECK(mock_read_calls("slow.test") == 2); /* re-resolved, not cached */
/* Both resolves were abandoned mid-backend; wait for their workers to leave
it before returning. The backend stays installed: an abandoned worker
still reads it (to free its addrinfo) after the last call returns. */
mock_wait_finished(2);
return failures;
}
#else
int dns_selftests(httrackp *opt) {

View File

@@ -46,6 +46,12 @@ typedef struct httrackp httrackp;
Returns the number of failed checks (0 == success). */
int dns_selftests(httrackp *opt);
/* Drive a deliberately slow (mock) resolver, asserting that a resolve is
bounded by opt->timeout, does not hold opt->state.lock while it runs, and
does not cache a timeout as an answer (#606). Takes a few seconds.
Returns the number of failed checks (0 == success). */
int dns_timeout_selftests(httrackp *opt);
#endif
#endif

View File

@@ -496,8 +496,8 @@ void help(const char *app, int more) {
infomsg("");
infomsg("Flow control:");
infomsg(" cN number of multiple connections (*c8)");
infomsg
(" TN timeout, number of seconds after a non-responding link is shutdown");
infomsg(" TN timeout, number of seconds after a non-responding link is"
" shutdown; also bounds host name resolution");
infomsg
(" RN number of retries, in case of timeout or non-fatal errors (*R1)");
infomsg

View File

@@ -65,6 +65,7 @@ Please visit our Website: http://www.httrack.com
#endif /* _WIN32 */
#include <stdarg.h>
#include <ctype.h>
#include <string.h>
#include <time.h>
#include <stdarg.h>
@@ -3747,6 +3748,20 @@ static int proxy_default_port(const char *arg) {
return hts_proxy_is_socks(arg) ? 1080 : 8080;
}
// port "a" of -P argument "arg": digits fitting TCP's 1..65535, else the scheme
// default. Not sscanf("%d"): past INT_MAX it wraps to a garbage port (#602)
static int parse_proxy_port(const char *a, const char *arg) {
char *end;
long p;
if (!isdigit((unsigned char) *a)) // strtol would eat a sign or leading space
return proxy_default_port(arg);
p = strtol(a, &end, 10);
if (*end != '\0' || p < 1 || p > 65535) // ERANGE lands out of range too
return proxy_default_port(arg);
return (int) p;
}
void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port) {
const char *authority = strstr(arg, "://");
const char *a;
@@ -3762,10 +3777,7 @@ void hts_parse_proxy(const char *arg, char *name, size_t name_size, int *port) {
while (a > authority && a[-1] != ':' && a[-1] != '@' && a[-1] != ']')
a--;
if (a > authority && a[-1] == ':') {
int p = -1;
sscanf(a, "%d", &p);
*port = (p > 0) ? p : proxy_default_port(arg);
*port = parse_proxy_port(a, arg);
namelen = (size_t) (a - 1 - arg);
} else {
*port = proxy_default_port(arg);
@@ -5084,50 +5096,164 @@ HTSEXT_API int check_hostname_dns(const char *const hostname) {
return hts_dns_resolve_nocache(hostname, &buffer) != NULL;
}
// Needs locking
// Internal DNS cache. Fill out[0..count-1] with up to max addresses for _iadr,
// resolving (and caching the full list) on a miss. Returns the count.
static int hts_dns_resolve_list_(httrackp *opt, const char *_iadr,
SOCaddr *const out, const int max,
const char **error) {
char BIGSTK iadr[HTS_URLMAXSIZE * 2];
coucal cache = hts_cache(opt); // le cache dns
/* A resolve in flight. Refcounted: a timed-out resolve is abandoned, not
cancelled, so the last of caller/worker to leave frees the job. */
typedef struct dns_resolve_job {
htsmutex lock;
int refcount;
hts_boolean done;
char *hostname;
SOCaddr addr[HTS_MAXADDRNUM];
int count;
const char *error;
} dns_resolve_job;
/* Copy the first min(count, max) addresses of src into dest. */
static void dns_copy_addrs(SOCaddr *dest, SOCaddr *src, int count, int max) {
int i;
for (i = 0; i < count && i < max; i++)
SOCaddr_copy_SOCaddr(dest[i], src[i]);
}
static void dns_job_release(dns_resolve_job *job) {
hts_boolean last;
hts_mutexlock(&job->lock);
last = (--job->refcount == 0) ? HTS_TRUE : HTS_FALSE;
hts_mutexrelease(&job->lock);
if (last) {
hts_mutexfree(&job->lock);
freet(job->hostname);
freet(job);
}
}
/* Outlives a timed-out resolve, so it writes only the job: never opt (freed
before the thread wait at exit) nor the DNS cache. */
static void dns_resolve_thread(void *arg) {
dns_resolve_job *const job = (dns_resolve_job *) arg;
SOCaddr resolved[HTS_MAXADDRNUM];
const char *error = NULL;
const int count = hts_dns_resolve_nocache_list(job->hostname, resolved,
HTS_MAXADDRNUM, &error);
hts_mutexlock(&job->lock);
dns_copy_addrs(job->addr, resolved, count, HTS_MAXADDRNUM);
job->count = count;
job->error = error;
job->done = HTS_TRUE; /* published last: gates the caller's read of addr[] */
hts_mutexrelease(&job->lock);
dns_job_release(job);
}
/* Resolve hostname on a worker thread, giving up after timeout seconds.
Returns the address count, or -1 on timeout -- distinct from 0 ("does not
resolve"), which is a real answer and gets negative-cached. */
static int hts_dns_resolve_nocache_list_bounded(const char *hostname,
SOCaddr *const out,
const int max,
const int timeout,
const char **error) {
dns_resolve_job *job;
TStamp deadline;
int count = -1;
int poll_ms = 1;
if (timeout <= 0) /* no bound asked for (--timeout 0) */
return hts_dns_resolve_nocache_list(hostname, out, max, error);
job = calloct(1, sizeof(*job));
assertf(job != NULL);
hts_mutexinit(&job->lock);
job->hostname = strdupt(hostname);
job->refcount = 2; /* this caller + the worker */
if (hts_newthread(dns_resolve_thread, job) != 0) {
job->refcount = 1; /* no worker: fall back to resolving inline */
dns_job_release(job);
return hts_dns_resolve_nocache_list(hostname, out, max, error);
}
deadline = mtime_local() + (TStamp) timeout * 1000;
for (;;) {
hts_boolean done;
hts_mutexlock(&job->lock);
done = job->done;
if (done) {
count = job->count;
dns_copy_addrs(out, job->addr, count, max);
if (error != NULL)
*error = job->error;
}
hts_mutexrelease(&job->lock);
if (done || mtime_local() >= deadline)
break;
Sleep(poll_ms);
if (poll_ms < 50) /* short first polls keep a fast resolve fast */
poll_ms *= 2;
}
dns_job_release(job);
return count;
}
int hts_dns_resolve_all(httrackp *opt, const char *iadr, SOCaddr *out, int max,
const char **error) {
char BIGSTK host[HTS_URLMAXSIZE * 2];
SOCaddr resolved[HTS_MAXADDRNUM];
coucal cache;
int count, i;
assertf(opt != NULL);
assertf(_iadr != NULL);
assertf(out != NULL);
if (!strnotempty(iadr) || max <= 0) {
return 0;
}
strcpybuff(iadr, jump_identification_const(_iadr));
// couper éventuel :
/* cache key and resolver input: identification and any ":port" stripped */
strcpybuff(host, jump_identification_const(iadr));
{
char *a;
if ((a = jump_toport(iadr)))
if ((a = jump_toport(host)))
*a = '\0';
}
/* get IP from the dns cache */
count = hts_ghbn_all(cache, iadr, out, max);
hts_mutexlock(&opt->state.lock);
cache = hts_cache(opt);
#if HTS_INET6 != 0
hts_resolver_check_env(); /* settle the backend before a worker reads it */
#endif
count = hts_ghbn_all(cache, host, out, max);
hts_mutexrelease(&opt->state.lock);
if (count >= 0) { // cache hit (0 == negative-cached)
return count;
} else { // non présent dans le cache dns, tester
SOCaddr resolved[HTS_MAXADDRNUM];
t_dnscache *record;
int i;
}
#if DEBUGDNS
printf("resolving (not cached) %s\n", iadr);
printf("resolving (not cached) %s\n", host);
#endif
count = hts_dns_resolve_nocache_list(iadr, resolved, HTS_MAXADDRNUM, error);
/* Resolve with no lock held: getaddrinfo can block for a long time, and
state.lock also gates the stop request (#606). */
count = hts_dns_resolve_nocache_list_bounded(host, resolved, HTS_MAXADDRNUM,
opt->timeout, error);
#if HTS_WIDE_DEBUG
DEBUG_W("gethostbyname done\n");
DEBUG_W("gethostbyname done\n");
#endif
/* attempt to store new entry (coucal owns it and dups the host key) */
record = malloct(sizeof(t_dnscache));
if (count < 0) { /* timed out: no answer to cache, and none to report */
if (error != NULL)
*error = "host name resolution timed out";
return 0;
}
hts_mutexlock(&opt->state.lock);
{ /* store the full list (coucal owns the record and dups the host key; a
concurrent resolve of the same host replaces, and frees, this one) */
t_dnscache *const record = malloct(sizeof(t_dnscache));
if (record != NULL) {
memset(record, 0, sizeof(*record));
record->host_count = count;
@@ -5137,28 +5263,13 @@ static int hts_dns_resolve_list_(httrackp *opt, const char *_iadr,
memcpy(record->host_addr[i], &SOCaddr_sockaddr(resolved[i]),
record->host_length[i]);
}
coucal_add_pvoid(cache, iadr, record);
coucal_add_pvoid(cache, host, record);
}
/* copy result to caller (cache store may have failed; result still valid)
*/
for (i = 0; i < count && i < max; i++) {
SOCaddr_copy_SOCaddr(out[i], resolved[i]);
}
return count;
} // retour hp du cache
}
int hts_dns_resolve_all(httrackp *opt, const char *iadr, SOCaddr *out, int max,
const char **error) {
int count;
if (!strnotempty(iadr) || max <= 0) {
return 0;
}
hts_mutexlock(&opt->state.lock);
count = hts_dns_resolve_list_(opt, iadr, out, max, error);
hts_mutexrelease(&opt->state.lock);
/* copy result to caller (cache store may have failed; result still valid) */
dns_copy_addrs(out, resolved, count, max);
return count;
}

View File

@@ -222,7 +222,8 @@ LLint http_xfread1(htsblk * r, int bufl);
/* Cached resolver: fill out[0..count-1] with up to max addresses for iadr (in
resolver order), returning the count (0 = does not resolve, negative-cached).
Resolves once per host; later calls read the DNS cache. Must hold no lock
(brackets opt->state.lock itself). */
(brackets opt->state.lock itself, never across the resolve). A miss resolves
on a worker thread bounded by opt->timeout; a timeout reports 0, uncached. */
int hts_dns_resolve_all(httrackp *opt, const char *iadr, SOCaddr *out, int max,
const char **error);
HTS_INLINE SOCaddr *hts_dns_resolve2(httrackp *opt, const char *iadr,

View File

@@ -2109,6 +2109,15 @@ static int st_dns(httrackp *opt, int argc, char **argv) {
return err;
}
static int st_dnstimeout(httrackp *opt, int argc, char **argv) {
const int err = dns_timeout_selftests(opt);
(void) argc;
(void) argv;
printf("dns-timeout-selftest: %s\n", err ? "FAIL" : "OK");
return err;
}
static int st_cookies(httrackp *opt, int argc, char **argv) {
static t_cookie cookie;
char hdr[1024];
@@ -3133,6 +3142,8 @@ static const struct selftest_entry {
{"cache-corrupt", "<dir>", "cache read-side corruption self-test",
st_cache_corrupt},
{"dns", "", "DNS resolver/cache self-test", st_dns},
{"dnstimeout", "", "a slow DNS resolve is bounded and holds no lock",
st_dnstimeout},
{"cookies", "", "cookie request-header self-test", st_cookies},
{"useragent", "", "default User-Agent self-test", st_useragent},
{"makeindex", "[dir]", "hts_finish_makeindex footer/refresh self-test",

View File

@@ -0,0 +1,15 @@
#!/bin/bash
#
set -euo pipefail
# A resolve against a deliberately slow mock getaddrinfo (no network) must
# return on --timeout, must not hold opt->state.lock meanwhile, and must not
# cache the timeout as an answer (#606).
# 'run' is an ignored placeholder argument.
out=$(httrack -#test=dnstimeout run)
test "$out" = "dns-timeout-selftest: OK" || {
echo "expected 'dns-timeout-selftest: OK', got: $out" >&2
exit 1
}

View File

@@ -71,3 +71,23 @@ p 'http://a]b:c@host' 'name=http://a]b:c@host port=8080 host=host kind=http'
# a lone ':' must not underflow the backward scan
p ':' 'name= port=8080 host= kind=http'
# a port is *DIGIT in 1..65535, else the scheme default: sscanf("%d") used to
# wrap past INT_MAX and hand back a garbage port (#602)
p 'http://host:65535' 'name=http://host port=65535 host=host kind=http'
p 'http://host:1' 'name=http://host port=1 host=host kind=http'
p 'http://host:65536' 'name=http://host port=8080 host=host kind=http'
p 'http://host:2147483648' 'name=http://host port=8080 host=host kind=http'
p 'http://host:4294967296' 'name=http://host port=8080 host=host kind=http'
p 'http://host:99999999999999' 'name=http://host port=8080 host=host kind=http'
# discriminating: this one wrapped to a plausible 80, so range-checking the
# sscanf result instead of rejecting the overflow still passes everything above
p 'http://host:4294967376' 'name=http://host port=8080 host=host kind=http'
p 'http://host:999999999999999999999999999999' 'name=http://host port=8080 host=host kind=http'
p 'http://host:0' 'name=http://host port=8080 host=host kind=http'
p 'http://host:-1' 'name=http://host port=8080 host=host kind=http'
p 'http://host:80x' 'name=http://host port=8080 host=host kind=http'
p 'http://host: 80' 'name=http://host port=8080 host=host kind=http'
p 'http://host:' 'name=http://host port=8080 host=host kind=http'
# the fallback is the scheme's own default, not a hardcoded 8080
p 'socks5://host:99999999999999' 'name=socks5://host port=1080 host=host kind=socks'

View File

@@ -0,0 +1,46 @@
#!/bin/bash
#
# Unit-tests dump_crawl_logs (testlib.sh): the killed test's hts-log.txt must
# reach the uploaded logs, or a Windows wedge past --max-time stays
# undiagnosable (#605).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
TMPDIR=$(mktemp -d "${TMPDIR:-/tmp}/httrack_salv.XXXXXX")
export TMPDIR
trap 'rm -rf "$TMPDIR"' EXIT
fail() {
echo "FAIL: $*" >&2
exit 1
}
verdict="More than 120 seconds passed.. giving up"
# Vacuity control: with nothing left behind the assertions below cannot pass on
# ambient output.
test -z "$(dump_crawl_logs)" || fail "dumped something with no crawl tmpdir"
# A killed local-crawl.sh leaves exactly this shape.
d="${TMPDIR}/httrack_local.probe"
mkdir -p "$d/crawl"
echo "$verdict" >"$d/crawl/hts-log.txt"
echo "httrack stdout marker" >"$d/log"
# Append to an unterminated line, as a killed test's own log always ends.
art="${TMPDIR}/artifact.log"
printf '[running httrack ...] ..\t' >"$art"
dump_crawl_logs >>"$art"
grep -qF "$verdict" "$art" || fail "hts-log.txt verdict not salvaged"
grep -qF "httrack stdout marker" "$art" || fail "httrack stdout not salvaged"
grep -q '^--- .*hts-log.txt' "$art" || fail "dump glued its header onto the unterminated line"
# Cleared, so the next timing-out test cannot re-report this crawl as its own.
test ! -d "$d" || fail "salvaged tmpdir left behind"
test -z "$(dump_crawl_logs)" || fail "salvaged crawl reported twice"
echo "crawl log salvage OK"

View File

@@ -35,6 +35,7 @@ TESTS = \
01_engine-copyopt.test \
01_engine-crange.test \
01_engine-dns.test \
01_engine-dnstimeout.test \
01_engine-doitlog.test \
01_engine-entities.test \
01_engine-filelist.test \
@@ -137,6 +138,7 @@ TESTS = \
56_local-proxy-noleak.test \
57_local-proxy-connect.test \
58_watchdog.test \
59_local-tls-stall.test
59_local-tls-stall.test \
60_crawl-log-salvage.test
CLEANFILES = check-network_sh.cache

View File

@@ -44,6 +44,25 @@ stop_server() {
return 0
}
# Dump and clear the crawl logs a hard-killed test leaves in TMPDIR (its cleanup
# trap never ran): hts-log.txt alone records "More than N seconds passed.. giving
# up", so a wedge past --max-time is undiagnosable without it (#605).
dump_crawl_logs() {
local d f
for d in "${TMPDIR:-/tmp}"/httrack_local.*; do
test -d "$d" || continue
for f in "$d/crawl/hts-log.txt" "$d/log" "$d/log.2"; do
test -f "$f" || continue
# Leading newline: the killed test's last line has no terminator.
printf '\n--- %s (last 200 lines)\n' "$f"
tail -n 200 "$f"
done
# so a later test's dump cannot re-report this one; never fatal, the
# caller is already handling a failure and Windows may still hold a file
rm -rf "$d" || true
done
}
# Kill a backgrounded job and its whole descendant tree. POSIX: the caller must
# have put the job in its own process group (run_with_timeout does) so we signal
# the group; a bare kill would orphan the grandchildren. Windows: the tree is