mirror of
https://github.com/xroche/httrack.git
synced 2026-07-24 01:29:49 +03:00
d267aaf1bf
#581 reports a Windows-only data loss: an interrupted mirror that resumes into an unusable Content-Range 206 drops its partial and never refetches, where the same sequence recovers the file whole on Linux. The trigger is the damaged cache a hard TerminateProcess leaves behind (MSYS can't signal a native exe), which pass 2 has to repair before resuming. This does not fix the engine. I couldn't reproduce the failure on Linux, so an engine change would be guesswork. What it adds is a deterministic test that drives the damaged-cache regime, plus an analysis of where the two platforms part. Test 71 leaves a partial and a temp-ref in pass 1, truncates `new.zip` past its last local entry so the central directory a hard kill never wrote is gone and the repair path runs, then resumes into the hostile 206. On Linux that fires the repair, takes the "unusable range -> restart whole" branch, and recovers the file whole every time, across every damage severity I tried, including a repair that recovers zero entries or fails outright. So the cache repair and the restart-whole logic are not themselves where Linux and Windows differ. Root cause, as far as I can pin it from Linux: restart-whole doesn't refetch. It removes the partial and the temp-ref, flags `STATUSCODE_NON_FATAL`, and leans on the ordinary retry to requeue the URL. The requeued attempt rebuilds the request from the cache, then the temp-ref, then the on-disk partial. On Linux the removals leave none of those, so the retry is a clean whole-file GET and it succeeds. For the file to be lost, the retried attempt has to send a Range again: a second unusable 206, a second -5, and once the retry budget is spent the partial is already gone. That second Range can only come from a temp-ref or partial that outlived the restart-whole removal, which points at a Windows-specific removal or path effect I can't confirm from here. For the maintainer: the fragile hinge is that restart-whole depends on a budget-consuming retry that re-derives its Range state from disk. A sturdier fix would make the retried attempt refuse to resume, via a per-link "refetch whole, no Range" flag the request builder honors, so a leftover temp-ref or partial can't re-enter the 206 loop whatever the removal quirk turns out to be. Checking that `UNLINK` and `url_savename_refname_remove` actually succeed on Windows would confirm the mechanism first. Test 71 carries the same Windows skip as test 48. Lifting that skip should reproduce the failure, and turn the test into the fix's verification. Refs #581. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
158 lines
5.0 KiB
Makefile
158 lines
5.0 KiB
Makefile
# Committed binary fixture read by 01_zlib-cache-golden.test. List it
|
|
# explicitly: automake does not expand wildcards in EXTRA_DIST, so a glob would
|
|
# silently drop it from the dist tarball and break "make distcheck".
|
|
EXTRA_DIST = $(TESTS) crawl-test.sh run-all-tests.sh check-network.sh \
|
|
proxy-https-server.py socks5-server.py proxy-connect-server.py \
|
|
proxytestlib.py tls-stall-server.py \
|
|
local-crawl.sh local-server.py testlib.sh server.crt server.key \
|
|
server-root/simple/basic.html server-root/simple/link.html \
|
|
server-root/stripquery/index.html server-root/stripquery/a.html \
|
|
server-root/fraglink/index.html server-root/fraglink/target.html \
|
|
fixtures/cache-golden/hts-cache/new.zip
|
|
|
|
TESTS_ENVIRONMENT =
|
|
TESTS_ENVIRONMENT += PATH=$(top_builddir)/src$(PATH_SEPARATOR)$$PATH
|
|
# note: libtool should handle that
|
|
### TESTS_ENVIRONMENT += $(SHLIBPATH_VAR)="$(top_builddir)/src/$(LT_CV_OBJDIR)$${$(SHLIBPATH_VAR):+$(PATH_SEPARATOR)}$$$(SHLIBPATH_VAR)"
|
|
TESTS_ENVIRONMENT += ONLINE_UNIT_TESTS=$(ONLINE_UNIT_TESTS)
|
|
TESTS_ENVIRONMENT += HTTPS_SUPPORT=$(HTTPS_SUPPORT)
|
|
TESTS_ENVIRONMENT += BROTLI_ENABLED=$(BROTLI_ENABLED)
|
|
TESTS_ENVIRONMENT += ZSTD_ENABLED=$(ZSTD_ENABLED)
|
|
TESTS_ENVIRONMENT += V6_SUPPORT=$(V6_SUPPORT)
|
|
TESTS_ENVIRONMENT += top_srcdir=$(top_srcdir)
|
|
|
|
TEST_EXTENSIONS = .test
|
|
# Run each .test through bash instead of execve()ing it. This lets "make check"
|
|
# work when the source tree sits on a noexec filesystem (the driver would
|
|
# otherwise fail with "Permission denied"), and removes any reliance on the
|
|
# scripts' executable bit. The scripts are #!/bin/bash and use bash features.
|
|
TEST_LOG_COMPILER = $(BASH)
|
|
TESTS = \
|
|
00_runnable.test \
|
|
01_engine-charset.test \
|
|
01_engine-cmdline.test \
|
|
01_engine-cookies.test \
|
|
01_engine-copyopt.test \
|
|
01_engine-crange.test \
|
|
01_engine-dns.test \
|
|
01_engine-dnstimeout.test \
|
|
01_engine-doitlog.test \
|
|
01_engine-entities.test \
|
|
01_engine-filelist.test \
|
|
01_engine-filter.test \
|
|
01_engine-filterdual.test \
|
|
01_engine-ftp-line.test \
|
|
01_engine-ftp-userpass.test \
|
|
01_engine-cacheindex.test \
|
|
01_engine-hashtable.test \
|
|
01_engine-header.test \
|
|
01_engine-idna.test \
|
|
01_engine-identurl.test \
|
|
01_engine-proxyurl.test \
|
|
01_engine-socks5.test \
|
|
01_engine-identabs.test \
|
|
01_engine-escape-room.test \
|
|
01_engine-inplace-escape.test \
|
|
01_engine-makeindex.test \
|
|
01_engine-mime.test \
|
|
01_engine-parse.test \
|
|
01_engine-pause.test \
|
|
01_engine-rcfile.test \
|
|
01_engine-reconcile.test \
|
|
01_engine-expandhome.test \
|
|
01_engine-fsize.test \
|
|
01_engine-redirect.test \
|
|
01_engine-relative.test \
|
|
01_engine-robots.test \
|
|
01_engine-savename.test \
|
|
01_engine-selftest-dispatch.test \
|
|
01_engine-simplify.test \
|
|
01_engine-sniff.test \
|
|
01_engine-status.test \
|
|
01_engine-stripquery.test \
|
|
01_engine-strsafe.test \
|
|
01_engine-topindex.test \
|
|
01_engine-urlhack.test \
|
|
01_engine-unescape-bounds.test \
|
|
01_engine-useragent.test \
|
|
01_engine-version-macros.test \
|
|
01_engine-xfread.test \
|
|
01_zlib-acceptencoding.test \
|
|
01_zlib-contentcodings.test \
|
|
01_zlib-cache.test \
|
|
01_zlib-cache-corrupt.test \
|
|
01_zlib-cache-legacy.test \
|
|
01_zlib-cache-golden.test \
|
|
01_zlib-cache-writefail.test \
|
|
01_zlib-repair-shift.test \
|
|
01_zlib-savename-cached.test \
|
|
02_manpage-regen.test \
|
|
02_update-cache.test \
|
|
10_crawl-simple.test \
|
|
11_crawl-cookies.test \
|
|
11_crawl-idna.test \
|
|
11_crawl-international.test \
|
|
11_crawl-longurl.test \
|
|
11_crawl-parsing.test \
|
|
12_crawl_https.test \
|
|
13_crawl_proxy_https.test \
|
|
13_local-cookies.test \
|
|
14_local-https.test \
|
|
15_local-types.test \
|
|
16_local-assume.test \
|
|
17_local-empty-ct.test \
|
|
18_local-update.test \
|
|
19_local-connect-fallback.test \
|
|
20_local-resume-loop.test \
|
|
21_local-intl-update.test \
|
|
22_local-broken-size.test \
|
|
23_local-errpage.test \
|
|
24_local-resume-overlap.test \
|
|
25_local-mime-exclude.test \
|
|
26_local-strip-query.test \
|
|
27_local-cookies-file.test \
|
|
28_local-pause.test \
|
|
29_local-redirect-fragment.test \
|
|
30_local-fragment-link.test \
|
|
32_local-cdispo.test \
|
|
33_local-delayed.test \
|
|
34_local-maxtime.test \
|
|
35_local-maxsize.test \
|
|
36_local-bigcrawl.test \
|
|
37_local-cache-outage.test \
|
|
38_local-update-304.test \
|
|
39_local-delayed-cancel.test \
|
|
40_local-why.test \
|
|
41_local-utf8-link.test \
|
|
42_local-maxsize-slow.test \
|
|
43_local-update-truncate.test \
|
|
44_local-update-errormask.test \
|
|
45_local-maxsize-recv.test \
|
|
46_local-update-304-resume.test \
|
|
47_local-crange-overflow.test \
|
|
48_local-crange-memresume.test \
|
|
49_local-cookiewall.test \
|
|
50_local-contentcodings.test \
|
|
51_local-update-codec.test \
|
|
52_local-socks5.test \
|
|
53_local-proxytrack-cache-corrupt.test \
|
|
54_local-update-truncate-purge.test \
|
|
55_local-chunked.test \
|
|
56_local-proxy-noleak.test \
|
|
57_local-proxy-connect.test \
|
|
58_watchdog.test \
|
|
59_local-tls-stall.test \
|
|
60_crawl-log-salvage.test \
|
|
61_webhttrack-locale.test \
|
|
62_lang-integrity.test \
|
|
63_webhttrack-home.test \
|
|
64_local-intl-outdir.test \
|
|
65_port-siblings.test \
|
|
66_engine-port80-strip.test \
|
|
67_engine-delayed-truncate.test \
|
|
68_webhttrack-outdir-charset.test \
|
|
69_local-intl-logdir.test \
|
|
71_local-crange-repaircache.test
|
|
|
|
CLEANFILES = check-network_sh.cache
|