mirror of
https://github.com/xroche/httrack.git
synced 2026-07-28 19:43:02 +03:00
1f9728d816
* A failed re-fetch overwrote the mirrored file with the aborted read's debris A transfer that dies before a complete response has no body, yet the save path still consulted r.adr, which at that point holds whatever the aborted header read left behind: raw status-line bytes, or an empty buffer that truncated the file to zero on macOS. Require a successful transfer, as the empty-body half of the condition already did. Closes #748 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: assert the failed re-fetch keeps its bytes on every platform Test 93 filtered reset.bin out of its bucket lists because a connection killed before the status line surfaced differently per platform. It no longer does, so assert the resource like any other: unchanged, with the bytes pass 1 mirrored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: make the re-fetch assertions catch a crawl that never re-fetched The new test passed with no second pass at all, so a regression that stopped re-fetching would have looked green. Assert the failure the fixture provokes, give stay.bin a fresh pass-2 body so a fix that stopped overwriting anything fails, and compare reset.bin by checksum rather than by length. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * --purge-old deleted a file whose re-fetch never got a response A transfer that dies on the wire leaves the previously mirrored copy in place, but back_finalize() returned without noting it, so the URL fell out of new.lst and the end-of-update purge treated the file like a page that had vanished from the site. Note the surviving copy, the way the incomplete-transfer branch above already does, for the connection-level failures htsparse.c retries on. A deliberate skip (too big, MIME-excluded, cancelled) keeps its current fate. Closes #746 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Keep the copy for every failure, not just the five retryable codes A malformed status line, an oversized declared length or a mid-flight abort all land on STATUSCODE_INVALID, outside the retryable set, and the purge still ate the file. Invert the test: any failure keeps the copy except the codes that mean the engine passed the resource over on purpose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: assert the retry exhaustion, not the per-platform failure message The message a cut connection produces depends on whether any bytes arrived, so matching it would fail on a runner that sees none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
26 lines
1.0 KiB
Bash
26 lines
1.0 KiB
Bash
#!/bin/bash
|
|
#
|
|
# A re-fetch cut mid-header must leave the mirrored file alone (#748) and keep
|
|
# it out of the update purge (#746). err.bin is the control, an HTTP 500 on the
|
|
# same resource already surviving both; stay.bin answers normally with a new
|
|
# body, so a fix that stopped overwriting anything fails.
|
|
|
|
set -euo pipefail
|
|
|
|
: "${top_srcdir:=..}"
|
|
|
|
bash "$top_srcdir/tests/local-crawl.sh" \
|
|
--rerun-args '--update' \
|
|
--log-found 'after 2 retries at link .*keep/page\.html' \
|
|
--log-found 'after 2 retries at link .*keep/data\.bin' \
|
|
--file-matches 'keep/page.html' 'KEEP-PAGE-V1' \
|
|
--file-not-matches 'keep/page.html' 'HTTP/1\.0 200' \
|
|
--file-matches 'keep/data.bin' 'KEEP-BIN-V1' \
|
|
--file-not-matches 'keep/data.bin' 'HTTP/1\.0 200' \
|
|
--file-min-bytes 'keep/data.bin' 2048 \
|
|
--file-matches 'keep/err.bin' 'KEEP-ERR-V1' \
|
|
--file-min-bytes 'keep/err.bin' 2048 \
|
|
--file-matches 'keep/stay.bin' 'KEEP-STAY-V2' \
|
|
--file-min-bytes 'keep/stay.bin' 2048 \
|
|
httrack 'BASEURL/keep/index.html' --retries=2
|