Compare commits

..

2 Commits

Author SHA1 Message Date
Xavier Roche
9a1913ff55 htsserver never returns from main(), it blocks on its own exit wait
background_threads counted two unrelated things: the pinger, which only ever
leaves through exit(), and each mirror thread, which does return. main() then
waited on background_threads - 1, so with no pinger it asked for a negative
number of outstanding threads and the counter cannot go there, and with one it
waited on the pinger itself. Either way htsserver hung instead of returning.

Count only what never returns, and let each wait name what it must leave
running: the pinger for main(), the pinger plus itself for back_launch_cmd().
Waiting on background_threads alone would have swapped the hang for an early
return, letting main() tear down while a mirror was still crawling.

Closes #753

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-27 08:31:41 +02:00
Xavier Roche
1172eeca43 htsserver never returns from main(), it blocks on its own exit wait
htsthread_wait_n(background_threads - 1) subtracts one more than the count of
threads that must not be joined. Without --ppid that count is zero, so the wait
asks for a negative number of outstanding threads and the counter never gets
there; with --ppid it waits on the pinger, which by design never returns.

Wait for background_threads instead, which is what the sibling call inside
back_launch_cmd() already passes.

Closes #753

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
2026-07-27 08:11:22 +02:00
155 changed files with 678 additions and 3634 deletions

View File

@@ -79,10 +79,7 @@ jobs:
- name: Build
run: make -j"$(nproc)"
# A backstop only: tests/test-timeout.sh bounds each test, and a healthy run
# is a minute here, two on macOS. Without it a stall ran to the job's 6h default.
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -122,7 +119,6 @@ jobs:
run: make -j"$(nproc)"
- name: Test without python3
timeout-minutes: 20
run: |
set -euo pipefail
# Hide every python3* so `command -v python3` fails like it does in the
@@ -177,7 +173,6 @@ jobs:
sudo ifconfig lo0 alias 127.0.0.3 up
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(sysctl -n hw.ncpu) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -250,7 +245,6 @@ jobs:
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -308,7 +302,6 @@ jobs:
env:
ASAN_OPTIONS: detect_leaks=0:abort_on_error=1:halt_on_error=1:strict_string_checks=1:malloc_fill_byte=202:max_malloc_fill_size=2147483647:free_fill_byte=203:max_free_fill_size=2147483647
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -352,7 +345,6 @@ jobs:
- name: Test (offline self-tests under MSan)
env:
MSAN_OPTIONS: abort_on_error=1:halt_on_error=1
timeout-minutes: 20
run: |
set -euo pipefail
# 01_engine-* only; zlib-dependent self-tests are named 01_zlib-* and
@@ -434,7 +426,6 @@ jobs:
run: make -j"$(nproc)"
- name: Test
timeout-minutes: 20
run: |
jobs=$(( $(nproc) * 2 )); [ "$jobs" -le 16 ] || jobs=16
make check -j"$jobs"
@@ -627,12 +618,7 @@ jobs:
set -euo pipefail
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
# Merge base, not the branch tip: master moves during a run, and a
# tip-relative diff blames its new C on this PR.
if ! base="$(git merge-base "origin/${{ github.base_ref }}" HEAD)"; then
echo "::error::no merge base with origin/${{ github.base_ref }}; cannot scope the check."
exit 1
fi
base="origin/${{ github.base_ref }}"
set +e
diff="$(git clang-format --binary clang-format-19 --style=file \
--diff --extensions c,h "$base")"
@@ -674,8 +660,6 @@ jobs:
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
# Three dots: merge-base scoped, so commits landing on master mid-run
# are not counted as this PR's.
changed="$(git diff --name-only "$base"...HEAD)"
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
if has man/httrack.1 && ! has html/httrack.man.html; then

View File

@@ -189,53 +189,25 @@ jobs:
# A wedged crawl must not eat the job's timeout budget. timeout(1)'s
# signals can't reap a native httrack.exe (MSYS signals don't reach it),
# so a hang orphaned processes that starved the runner; run_with_timeout
# TerminateProcess-es the whole tree. 600s is unchanged: it clears the
# 540s a three-pass crawl may legitimately take under local-crawl.sh's
# own watchdogs, against a slowest healthy test here of 39s.
# TerminateProcess-es the whole tree. 600s clears the slowest multi-pass
# crawl (a few passes at --max-time=120 each).
. ./testlib.sh
per_test=600
# The whole suite must give up before the step timeout above. A cancelled
# step keeps neither its log nor the artifacts the later if:always()
# steps would upload, so an overrun that ends in a cancel tells us
# nothing; failing on our own terms keeps both. Healthy runs take 8-9
# min. The check sits between tests, so the step can still reach 25 min
# plus one per-test budget, and that worst case stays inside the 45.
suite_deadline=1500
started=$SECONDS
# Survives into the artifact even if the tail of the step log does not.
progress=suite-progress.log
: >"$progress"
pass=0 fail=0 skip=0 failed="" skipped="" deadline=0
pass=0 fail=0 skip=0 failed="" skipped=""
for t in 00_runnable.test 01_engine-*.test 01_zlib-*.test \
*_local-*.test 13_crawl_proxy_https.test 58_watchdog.test \
60_crawl-log-salvage.test; do
elapsed=$((SECONDS - started))
if [ "$elapsed" -ge "$suite_deadline" ]; then
echo "::error::suite deadline: ${elapsed}s elapsed, stopping before $t"
echo "DEADLINE before $t after ${elapsed}s" >>"$progress"
# Per-test start times, so the slow ones are named rather than guessed.
sed 's/^/ /' "$progress"
deadline=1
break
fi
echo "RUN $t at ${elapsed}s" >>"$progress"
rc=0
# Same guard "make check" uses on POSIX, so a wedge is diagnosed the
# same way on every platform. It dumps before it kills, which a bare
# run_with_timeout cannot: by the time that returns, the tree whose
# stack we wanted is already gone.
HTTRACK_TEST_TIMEOUT=$per_test bash ./test-timeout.sh "$t" >"$t.log" 2>&1 || rc=$?
run_with_timeout 600 bash "$t" >"$t.log" 2>&1 || rc=$?
case "$rc" in
0) pass=$((pass + 1)); echo "PASS $t" ;;
77) skip=$((skip + 1)) skipped="$skipped $t"; echo "SKIP $t" ;;
124)
fail=$((fail + 1)) failed="$failed $t"
# test-timeout.sh has already written the process list, the stacks
# and the killed crawl's own logs into $t.log.
echo "FAIL $t (timed out, tree killed)"
# Re-running a wedge traced would just hang again: salvage the
# killed crawl's own logs into the artifact instead.
dump_crawl_logs >>"$t.log"
tail -n 25 "$t.log" | sed 's/^/ /'
;;
*)
@@ -243,30 +215,21 @@ jobs:
echo "FAIL $t (exit $rc)"
# These assert with `test "$(...)" == "..." || exit 1`, which
# says nothing at all on failure. Re-run traced, still bounded.
run_with_timeout "$per_test" bash -x "$t" >>"$t.log" 2>&1 || true
run_with_timeout 600 bash -x "$t" >>"$t.log" 2>&1 || true
tail -n 25 "$t.log" | sed 's/^/ /'
;;
esac
echo "$rc $t" >>"$progress"
# An orphaned native httrack.exe spins and starves the runner, which
# is how this job dies with "lost communication" rather than a plain
# timeout. Clear them between tests and name whoever leaked them.
reap_leftover_processes "$t" | tee -a "$progress"
done
echo "ran=$((pass + fail + skip)) pass=$pass fail=$fail skip=$skip" |
tee -a "$GITHUB_STEP_SUMMARY"
# Every gate here exits 77, so an all-skipped suite would report green having
# tested nothing: pin the skips, and floor the passes in case the glob empties.
# footer-overflow and purge-longpath skip on Windows (need a path past MAX_PATH);
# crange pending #581;
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
# webdav-mime needs a reapable background listener, which MSYS cannot give it;
# badmtime needs a filesystem that stores an mtime past gmtime's range;
# single-file ends on a GUI half needing htsserver, which this job does not build;
# update-304-leak needs a LeakSanitizer build, which MSVC has no equivalent of.
expected_skips=" 01_engine-footer-overflow.test 100_local-purge-longpath.test 114_local-update-304-leak.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test 94_local-single-file.test"
# First, or the deadline reads as an unexplained shortfall in the gates below.
[ "$deadline" -eq 0 ] || { echo "::error::suite did not finish within ${suite_deadline}s"; exit 1; }
# single-file ends on a GUI half needing htsserver, which this job does not build.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test 94_local-single-file.test"
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
[ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

View File

@@ -26,18 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
- Run teardown with errexit off: `trap 'set +e; cleanup' EXIT`. Under `set -e` a
failing cleanup command becomes the test's exit status (#773). Keep the other
signals on their own `trap` line, or errexit stays off for the rest of the run.
The guard also resets `$?`, so save it first if teardown reads it.
- Never pipe into `grep -q`: it exits on the first match, so whatever the
producer had left to write takes SIGPIPE, and under `pipefail` that becomes
the pipeline's status. `cmd | grep -q M && fail` then never fires and a probe
that proved nothing reads as "marker absent"; `cmd | grep -q M || fail` fails
a test whose marker was present. bash issues one `write()` per line, so any
match that is not on the last line is exposed. Capture the reply, assert the
status line it must carry (an empty, truncated or redirected one is
marker-free too), then match with a here-string: `grep -q M <<<"$reply"`.
- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
as "marker absent". Capture the reply, assert the status line it must carry
(an empty, truncated or redirected one is marker-free too), then match with a
here-string.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every

View File

@@ -237,7 +237,7 @@ Build options:
x replace external html links by error pages (--replace-external)
%x do not include any password for external password protected websites (%x0 include) (--no-passwords)
%q *include query string for local files (useless, for information purpose only) (%q0 don't include) (--include-query-string)
o *save the server's error pages (404..) (o0 discard them) (--generate-errors)
o *generate output html file in case of error (404..) (o0 don't generate) (--generate-errors)
X *purge old files after update (X0 keep delete) (--purge-old[=N])
Spider options:
@@ -413,7 +413,7 @@ site. Specifically, the defauls are:
NN name conversion type (0 *original structure, 1+: see below)
LN long names (L1 *long names / L0 8-3 conversion)
K keep original links (e.g. http://www.adr/link) (K0 *relative link)
o *save the server's error pages (404..) (o0 discard them)
o *generate output html file in case of error (404..) (o0 don't generate)
X *purge old files after update (X0 keep delete)
bN accept cookies in cookies.txt (0=do not accept,* 1=accept)
u check document type if unknown (cgi,asp..) (u0 don't check, * u1 check but /, u2 check always)
@@ -473,11 +473,11 @@ store them with the same names used on the web site.
URLs within this web site are adjusted to point to the files in the
mirror.
<pre><b><i> o *save the server's error pages (404..) (o0 discard them) </i></b></pre>
<pre><b><i> o *generate output html file in case of error (404..) (o0 don't generate) </i></b></pre>
<p align=justify> IF a page cannot be downloaded, the error page the
server sent is saved in its place, so a broken link still lands on the
site's own 'not found' page. This makes browsing go a lot smoother.
<p align=justify> IF there are errors in downloading, create a file that
indicates that the URL was not found. This makes browsing go a lot
smoother.
<pre><b><i> X *purge old files after update (X0 keep delete) </i></b></pre>
@@ -1011,7 +1011,7 @@ Build options:
LN long names (L1 *long names / L0 8-3 conversion)
K keep original links (e.g. http://www.adr/link) (K0 *relative link)
x replace external html links by error pages
o *save the server's error pages (404..) (o0 discard them)
o *generate output html file in case of error (404..) (o0 don't generate)
X *purge old files after update (X0 keep delete)
%x do not include any password for external password protected websites (%x0 include) (--no-passwords)
%q *include query string for local files (information only) (%q0 don't include) (--include-query-string)
@@ -1118,9 +1118,9 @@ deactivated byt his process.
httrack http://www.shoesizes.com -O /tmp/shoesizes -x
</i></b></pre>
<p align=justify> This option keeps the server's '404' error pages out
of the mirror, even though there were URLs pointing to the missing
files. It is useful for saving space as well as eliminating
<p align=justify> This option prevents the generation of '404' error
files to replace files that were not found even though there were URLs
pointing to them. It is useful for saving space as well as eliminating
unnecessary files in operations where a working web site is not the
desired result.

View File

@@ -777,8 +777,8 @@ information purpose only) (%q0 don&rsquo;t include)
<td width="82%">
<p>*save the server&rsquo;s error pages (404..) (o0 discard
them) (--generate-errors)</p></td></tr>
<p>*generate output html file in case of error (404..) (o0
don&rsquo;t generate) (--generate-errors)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">

View File

@@ -122,7 +122,7 @@ h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
</small><br><br>
<!-- -->
<li>No error pages</li>
<br><small>Do not save the error pages sent by the server (if a 404 error occurred, for example)
<br><small>Do not generate error pages (if a 404 error occurred, for example)
<br>If a page is missing on the remote site, there will not be any warning on the local site
</small><br><br>
<!-- -->

View File

@@ -3,7 +3,7 @@
.\"
.\" This file is generated by man/makeman.sh; do not edit by hand.
.\" SPDX-License-Identifier: GPL-3.0-or-later
.TH httrack 1 "27 July 2026" "httrack website copier"
.TH httrack 1 "26 July 2026" "httrack website copier"
.SH NAME
httrack \- offline browser : copy websites to a local directory
.SH SYNOPSIS
@@ -220,7 +220,7 @@ do not include any password for external password protected websites (%x0 includ
.IP \-%g
strip query keys for dedup ([host/pattern=]key1,key2,...) (\-\-strip\-query <param>)
.IP \-o
*save the server's error pages (404..) (o0 discard them) (\-\-generate\-errors)
*generate output html file in case of error (404..) (o0 don't generate) (\-\-generate\-errors)
.IP \-X
*purge old files after update (X0 keep delete) (\-\-purge\-old[=N])
.IP \-%p

View File

@@ -487,6 +487,141 @@ regen:
#define HTS_DATA_UNKNOWN_HTML_LEN 0
#define HTS_DATA_ERROR_HTML "<html>"LF\
"<html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en\">"LF\
""LF\
"<head>"LF\
" <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />"LF\
" <meta name=\"description\" content=\"HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help\" />"LF\
" <meta name=\"keywords\" content=\"httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software\" />"LF\
" <title>Page not retrieved! - HTTrack Website Copier</title>"LF\
" <style type=\"text/css\">"LF\
" <!--"LF\
""LF\
"body {"LF\
" margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;"LF\
" background: #77b;"LF\
"}"LF\
"body, td {"LF\
" font: 14px \"Trebuchet MS\", Verdana, Arial, Helvetica, sans-serif;"LF\
" }"LF\
""LF\
"#subTitle {"LF\
" background: #000; color: #fff; padding: 4px; font-weight: bold; "LF\
" }"LF\
""LF\
"#siteNavigation a, #siteNavigation .current {"LF\
" font-weight: bold; color: #448;"LF\
" }"LF\
"#siteNavigation a:link { text-decoration: none; }"LF\
"#siteNavigation a:visited { text-decoration: none; }"LF\
""LF\
"#siteNavigation .current { background-color: #ccd; }"LF\
""LF\
"#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }"LF\
"#siteNavigation a:active { text-decoration: none; background-color: #ccc; }"LF\
""LF\
""LF\
"a:link { text-decoration: underline; color: #00f; }"LF\
"a:visited { text-decoration: underline; color: #000; }"LF\
"a:hover { text-decoration: underline; color: #c00; }"LF\
"a:active { text-decoration: underline; }"LF\
""LF\
"#pageContent {"LF\
" clear: both;"LF\
" border-bottom: 6px solid #000;"LF\
" padding: 10px; padding-top: 20px;"LF\
" line-height: 1.65em;"LF\
" background-image: url(backblue.gif);"LF\
" background-repeat: no-repeat;"LF\
" background-position: top right;"LF\
" }"LF\
""LF\
"#pageContent, #siteNavigation {"LF\
" background-color: #ccd;"LF\
" }"LF\
""LF\
""LF\
".imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }"LF\
".imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }"LF\
""LF\
"hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }"LF\
""LF\
"h1 { margin: 0; font-weight: bold; font-size: 2em; }"LF\
"h2 { margin: 0; font-weight: bold; font-size: 1.6em; }"LF\
"h3 { margin: 0; font-weight: bold; font-size: 1.3em; }"LF\
"h4 { margin: 0; font-weight: bold; font-size: 1.18em; }"LF\
""LF\
".blak { background-color: #000; }"LF\
".hide { display: none; }"LF\
".tableWidth { min-width: 400px; }"LF\
""LF\
".tblRegular { border-collapse: collapse; }"LF\
".tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }"LF\
".tblHeaderColor, .tblHeaderColor td { background: #99c; }"LF\
".tblNoBorder td { border: 0; }"LF\
""LF\
""LF\
"// -->"LF\
"</style>"LF\
""LF\
"</head>"LF\
""LF\
"<table width=\"76%%\" border=\"0\" align=\"center\" cellspacing=\"0\" cellpadding=\"3\" class=\"tableWidth\">"LF\
" <tr>"LF\
" <td id=\"subTitle\">HTTrack Website Copier - Open Source offline browser</td>"LF\
" </tr>"LF\
"</table>"LF\
"<table width=\"76%%\" border=\"0\" align=\"center\" cellspacing=\"0\" cellpadding=\"0\" class=\"tableWidth\">"LF\
"<tr class=\"blak\">"LF\
"<td>"LF\
" <table width=\"100%%\" border=\"0\" align=\"center\" cellspacing=\"1\" cellpadding=\"0\">"LF\
" <tr>"LF\
" <td colspan=\"6\"> "LF\
" <table width=\"100%%\" border=\"0\" align=\"center\" cellspacing=\"0\" cellpadding=\"10\">"LF\
" <tr> "LF\
" <td id=\"pageContent\"> "LF\
"<!-- ==================== End prologue ==================== -->"LF\
"<h1><strong><u>Oops!...</u></strong></h1>"LF\
"<h3>This page has <font color=\"red\"><em>not</em></font> been retrieved by HTTrack Website Copier (%s). </h3>"LF\
"<script language=\"Javascript\">"LF\
"<!--"LF\
" var loc=document.location.toString();"LF\
" if (loc) {"LF\
" var pos=loc.indexOf('link=');"LF\
" if (pos>0) {"LF\
" document.write('Clic to the link <b>below</b> to go to the online location!<br><a href=\"'+loc.substring(pos+5)+'\">'+loc.substring(pos+5)+'</a><br>');"LF\
" } else"LF\
" document.write('(no location defined)');"LF\
" }"LF\
"// -->"LF\
"</script>"LF\
"<h6 align=\"right\">Mirror by HTTrack Website Copier</h6>"LF\
"</body>"LF\
"</html>"LF\
"<!-- ==================== Start epilogue ==================== -->"LF\
" </td>"LF\
" </tr>"LF\
" </table>"LF\
" </td>"LF\
" </tr>"LF\
" </table>"LF\
"</td>"LF\
"</tr>"LF\
"</table>"LF\
""LF\
"<table width=\"76%%\" height=\"100%%\" border=\"0\" align=\"center\" valign=\"bottom\" cellspacing=\"0\" cellpadding=\"0\">"LF\
" <tr>"LF\
" <td id=\"footer\"><small>&copy; 2014 Xavier Roche & other contributors - Web Design: Kauler Leto.</small></td>"LF\
" </tr>"LF\
"</table>"LF\
""LF\
"</body>"LF\
""LF\
"</html>"LF\
""LF\
""LF
// image gif "unknown"
#define HTS_DATA_UNKNOWN_GIF \
"\x47\x49\x46\x38\x39\x61\x20\x0\x20\x0\xf7\xff\x0\xc0\xc0\xc0\xff\x0\x0\xfc\x3\x0\xf8\x6\x0\xf6\x9\x0\xf2\xc\x0\xf0\xf\x0\xf0\xe\x0\xed\x11\x0\xec\x13\x0\xeb\x14\x0\xe9\x15\x0\xe8\x18\x0\xe6\x18\x0\xe5\x1a\x0\xe3\x1c\x0\xe2\x1d\x0\xe1\x1e\x0\xdf\x20\x0\xdd\x23\x0\xdd\x22\x0\xdb\x23\x0\xda\x25\x0\xd9\x25\x0\xd8\x27\x0\xd6\x29\x0\xd5\x2a\x0\xd3\x2c\x0\xd2\x2d\x0"\

View File

@@ -340,111 +340,12 @@ static int back_index_ready(httrackp * opt, struct_back * sback, const char *adr
}
static int slot_can_be_cached_on_disk(const lien_back * back) {
/* A pending backup or spool means the slot is not finalized, and the swap
would unlink it through back_clear_entry() (#771). */
if (back->tmpfile != NULL && back->tmpfile[0] != '\0')
return 0;
return (back->status == STATUS_READY && back->locked == 0
&& back->url_sav[0] != '\0'
&& strcmp(back->url_sav, BACK_ADD_TEST) != 0);
/* Note: not checking !IS_DELAYED_EXT(back->url_sav) or it will quickly cause the slots to be filled! */
}
int back_selftest_slot_swap(void) {
lien_back back;
int err = 0;
#define CHECK(want, why) \
do { \
if (slot_can_be_cached_on_disk(&back) != (want)) { \
fprintf(stderr, "backswap: expected %d for %s\n", (want), (why)); \
err = 1; \
} \
} while (0)
memset(&back, 0, sizeof(back));
back.status = STATUS_READY;
strcpybuff(back.url_sav, "/tmp/httrack-selftest.bin");
CHECK(1, "a plain ready slot");
back.tmpfile = back.tmpfile_buffer;
strcpybuff(back.tmpfile_buffer, "/tmp/httrack-selftest.bin.bak");
CHECK(0, "a slot still holding a re-fetch backup");
/* Callers clear a spent temporary by emptying the name, not the pointer. */
back.tmpfile_buffer[0] = '\0';
CHECK(1, "a slot whose temporary was already dropped");
back.tmpfile = NULL;
back.locked = 1;
CHECK(0, "a locked slot");
back.locked = 0;
back.status = STATUS_TRANSFER;
CHECK(0, "a slot still transferring");
back.status = STATUS_READY;
back.url_sav[0] = '\0';
CHECK(0, "a slot with no save name");
strcpybuff(back.url_sav, BACK_ADD_TEST);
CHECK(0, "the dummy test slot");
#undef CHECK
/* The swap round-trip must not lose the size of a slot whose body is already
at url_sav, or the link writer blanks the file (#797). */
{
static const char body[] = "swapped body";
int c;
for (c = 0; c < 2; c++) {
const hts_boolean inmemory = c == 0 ? HTS_TRUE : HTS_FALSE;
FILE *const fp = tmpfile();
lien_back *copy = NULL;
memset(&back, 0, sizeof(back));
back.status = STATUS_READY;
strcpybuff(back.url_sav, "/tmp/httrack-selftest.bin");
back.r.size = (LLint) sizeof(body) - 1;
if (inmemory) {
back.r.adr = strdupt(body);
}
if (fp == NULL || back_serialize(fp, &back) != 0 ||
fseek(fp, 0, SEEK_SET) != 0 || back_unserialize(fp, &copy) != 0) {
fprintf(stderr, "backswap: round-trip failed for a %s slot\n",
inmemory ? "buffered" : "direct-to-disk");
err = 1;
} else {
if (copy->r.size != back.r.size) {
fprintf(stderr,
"backswap: %s slot came back with size " LLintP
", expected " LLintP "\n",
inmemory ? "buffered" : "direct-to-disk", copy->r.size,
back.r.size);
err = 1;
}
if (inmemory && (copy->r.adr == NULL ||
memcmp(copy->r.adr, body, sizeof(body) - 1) != 0)) {
fprintf(stderr, "backswap: buffered slot lost its body\n");
err = 1;
}
if (!inmemory && copy->r.adr != NULL) {
fprintf(stderr, "backswap: direct-to-disk slot gained a body\n");
err = 1;
}
back_clear_entry(copy);
freet(copy);
}
if (fp != NULL)
fclose(fp);
freet(back.r.adr);
}
}
printf("backswap self-test: %s\n", err ? "FAIL" : "OK");
return err;
}
/* Put all backing entries that are ready in the storage hashtable to spare space and CPU */
int back_cleanup_background(httrackp * opt, cache_back * cache,
struct_back * sback) {
@@ -684,40 +585,12 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
return 0;
}
/* Note: utf-8 */
void back_refetch_backup(httrackp *opt, lien_back *const back) {
back->tmpfile = NULL;
if (fexist_utf8(back->url_sav)) {
hts_boolean saved = HTS_FALSE;
if (create_back_tmpfile(opt, back, "bak") == 0) {
/* clobber a .bak a killed run left behind, or the guard stays off for
good (#758) */
if (fexist_utf8(back->tmpfile))
hts_log_print(opt, LOG_WARNING, "replacing leftover backup %s",
back->tmpfile);
saved = hts_rename_over(opt, back->url_sav, back->tmpfile);
}
if (!saved) {
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"could not back up %s; an aborted re-fetch will lose it",
back->url_sav);
back->tmpfile = NULL;
}
}
}
/* Did the fetch fail to produce a response, as opposed to the engine
deliberately passing the resource over? Only the latter may be purged. */
static hts_boolean back_transfer_failed(const int statuscode) {
switch (statuscode) {
case STATUSCODE_TOO_BIG:
case STATUSCODE_EXCLUDED:
case STATUSCODE_TEST_OK:
return HTS_FALSE;
default:
return statuscode <= 0 ? HTS_TRUE : HTS_FALSE;
}
/* Move src onto dst; RENAME does not clobber an existing target on Windows. */
static hts_boolean replace_file(const char *src, const char *dst) {
if (RENAME(src, dst) == 0)
return HTS_TRUE;
(void) UNLINK(dst);
return RENAME(src, dst) == 0 ? HTS_TRUE : HTS_FALSE;
}
/* Commit or restore a re-fetch backup (#77 follow-up): a re-fetch over an
@@ -737,7 +610,7 @@ static void back_finalize_backup(httrackp *opt, lien_back *const back,
}
/* On failure keep the backup: an orphaned temp beats losing the good copy.
*/
if (!hts_rename_over(opt, back->tmpfile, back->url_sav))
if (!replace_file(back->tmpfile, back->url_sav))
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"could not restore %s; previous copy kept as %s",
back->url_sav, back->tmpfile);
@@ -868,7 +741,7 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
"Read error when decompressing");
}
UNLINK(unpacked);
} else if (hts_rename_over(opt, unpacked, back[p].url_sav)) {
} else if (replace_file(unpacked, back[p].url_sav)) {
/* The temp bypassed filecreate(), which is what chmods. */
#ifndef _WIN32
chmod(back[p].url_sav, HTS_ACCESS_FILE);
@@ -1166,14 +1039,6 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
/* Aborted, error, or not ready: url_sav (if written) is broken; restore the
previous copy from the backup. */
back_finalize_backup(opt, &back[p], HTS_FALSE);
/* Note the surviving copy, or the end-of-update purge drops what this run
never managed to replace (#746). */
if (!back[p].testmode && back_transfer_failed(back[p].r.statuscode) &&
back[p].url_sav[0] != '\0' && fexist_utf8(back[p].url_sav)) {
filenote(&opt->state.strc, back[p].url_sav, NULL);
file_notify(opt, back[p].url_adr, back[p].url_fil, back[p].url_sav, 0, 0,
back[p].r.notmodified);
}
return -1;
}
@@ -1223,13 +1088,6 @@ void back_connxfr(htsblk * src, htsblk * dst) {
src->debugid = 0;
}
/* Release the buffers a response owns. The connection members are left alone:
back_connxfr() moves those, and the file handles are closed elsewhere. */
static void back_free_response(htsblk *r) {
deleteaddr(r);
warc_free_request(r);
}
void back_move(lien_back * src, lien_back * dst) {
memcpy(dst, src, sizeof(lien_back));
memset(src, 0, sizeof(lien_back));
@@ -1323,10 +1181,7 @@ int back_unserialize(FILE * fp, lien_back ** dst) {
(*dst)->r.ssl_con = NULL;
#endif
if (back_data_unserialize(fp, (void **) &(*dst)->r.adr, &size) == 0) {
/* A bodyless slot already wrote its bytes to url_sav (FTP, direct to
disk); zeroing r.size makes the writer blank that file (#797). */
if ((*dst)->r.adr != NULL)
(*dst)->r.size = size;
(*dst)->r.size = size;
(*dst)->r.headers = NULL;
if (back_string_unserialize(fp, &(*dst)->r.headers) == 0)
return 0; /* ok */
@@ -1765,7 +1620,10 @@ int back_clear_entry(lien_back * back) {
back->r.soc = INVALID_SOCKET;
}
back_free_response(&back->r);
if (back->r.adr != NULL) { // reste un bloc à désallouer
freet(back->r.adr);
back->r.adr = NULL;
}
if (back->chunk_adr != NULL) { // reste un bloc à désallouer
freet(back->chunk_adr);
back->chunk_adr = NULL;
@@ -1778,6 +1636,12 @@ int back_clear_entry(lien_back * back) {
(void) unlink(back->tmpfile);
back->tmpfile = NULL;
}
// headers
if (back->r.headers != NULL) {
freet(back->r.headers);
back->r.headers = NULL;
}
warc_free_request(&back->r);
// Tout nettoyer
memset(back, 0, sizeof(lien_back));
back->r.soc = INVALID_SOCKET;
@@ -3287,7 +3151,20 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
back[i].url_sav, 1, 1,
back[i].r.notmodified);
back[i].r.compressed = 0;
back_refetch_backup(opt, &back[i]);
/* Re-fetch over an existing file (#77 follow-up):
move the good copy aside before truncating it
so an aborted transfer can restore it. url_sav
is still written normally (file list intact).
*/
back[i].tmpfile = NULL;
if (fexist_utf8(back[i].url_sav)) {
if (create_back_tmpfile(opt, &back[i], "bak") !=
0 ||
RENAME(back[i].url_sav, back[i].tmpfile) !=
0) {
back[i].tmpfile = NULL;
}
}
if ((back[i].r.out =
filecreate(&opt->state.strc,
back[i].url_sav)) == NULL) {
@@ -4082,9 +3959,6 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
memset(&tmp, 0, sizeof(tmp));
back_connxfr(&back[i].r, &tmp);
/* the cache entry overwrites the whole struct, so drop
what the 304 response still owns first (#782) */
back_free_response(&back[i].r);
back[i].r =
cache_read(opt, cache, back[i].url_adr, back[i].url_fil,
back[i].url_sav, back[i].location_buffer);

View File

@@ -139,12 +139,6 @@ int back_trylive(httrackp * opt, cache_back * cache, struct_back * sback,
const int p);
int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
const int p);
/* Move the previous copy of back->url_sav to back->tmpfile so back_finalize()
can put it back when the re-fetch fails (#77 follow-up). Call right before
truncating url_sav; tmpfile stays NULL when there is nothing to save. */
void back_refetch_backup(httrackp *opt, lien_back *const back);
/* -#test=backswap: slots eligible for the on-disk ready table. */
int back_selftest_slot_swap(void);
void back_info(struct_back * sback, int i, int j, FILE * fp);
void back_infostr(struct_back *sback, int i, int j, char *s, size_t size);
LLint back_transferred(LLint add, struct_back * sback);

View File

@@ -855,6 +855,13 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
return r;
}
// lecture d'un fichier dans le cache
// si save==null alors test unqiquement
static int hts_rename(httrackp * opt, const char *a, const char *b) {
hts_log_print(opt, LOG_DEBUG, "Cache: rename %s -> %s (%p %p)", a, b, a, b);
return RENAME(a, b);
}
/* Open the cache ZIP via hts_fopen_utf8 so a non-ASCII path_log isn't mangled
to ANSI (#630); 64-bit funcs keep multi-GB caches whole on Windows LLP64. */
static voidpf ZCALLBACK hts_zip_fopen_utf8(voidpf opaque, const void *filename,
@@ -984,16 +991,29 @@ void cache_init(cache_back * cache, httrackp * opt) {
OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/new.zip")))) { // a previous cache exists.. rename it
if (!hts_rename_over(
opt,
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.zip"),
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.zip"))) {
/* Remove OLD cache */
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.zip"))) {
if (UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.zip")) !=
0) {
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"Cache: error while moving previous cache");
}
}
/* Rename */
if (hts_rename
(opt,
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.zip"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.zip")) != 0) {
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"Cache: error while moving previous cache");
} else {
hts_log_print(opt, LOG_DEBUG, "Cache: rotated new.zip to old.zip");
hts_log_print(opt, LOG_DEBUG, "Cache: successfully renamed");
}
}
} else {

View File

@@ -676,7 +676,7 @@ int cache_selftests(httrackp *opt, const char *dir) {
char base[HTS_URLMAXSIZE];
strcpybuff(base, dir);
if (base[0] != '\0' && hts_lastchar(base) != '/') {
if (base[0] != '\0' && base[strlen(base) - 1] != '/') {
strcatbuff(base, "/");
}
StringCopy(opt->path_log, base);
@@ -856,7 +856,7 @@ static void golden_setup(httrackp *opt, const char *dir) {
char base[HTS_URLMAXSIZE];
strcpybuff(base, dir);
if (base[0] != '\0' && hts_lastchar(base) != '/') {
if (base[0] != '\0' && base[strlen(base) - 1] != '/') {
strcatbuff(base, "/");
}
StringCopy(opt->path_log, base);

View File

@@ -72,7 +72,7 @@ hts_codec hts_codec_parse(const char *encoding) {
return HTS_CODEC_IDENTITY;
if (strfield2(encoding, "gzip") || strfield2(encoding, "x-gzip") ||
strfield2(encoding, "deflate") || strfield2(encoding, "x-deflate"))
return HTS_CODEC_DEFLATE;
return HTS_USEZLIB ? HTS_CODEC_DEFLATE : HTS_CODEC_UNSUPPORTED;
if (strfield2(encoding, "br"))
return HTS_USEBROTLI ? HTS_CODEC_BROTLI : HTS_CODEC_UNSUPPORTED;
if (strfield2(encoding, "zstd"))
@@ -98,11 +98,16 @@ hts_codec hts_codec_parse(const char *encoding) {
const char *hts_acceptencoding(hts_boolean compressible, hts_boolean secure) {
if (!compressible)
return "identity";
#if HTS_USEZLIB
/* br and zstd over TLS only, as browsers do: a cleartext intermediary that
rewrites a coding it can not read would corrupt the mirror. */
if (secure)
return "gzip, deflate" HTS_AE_BROTLI HTS_AE_ZSTD ", identity;q=0.9";
return "gzip, deflate, identity;q=0.9";
#else
(void) secure;
return "identity";
#endif
}
hts_boolean hts_codec_is_archive_ext(hts_codec codec, const char *ext) {
@@ -295,7 +300,11 @@ int hts_codec_unpack(hts_codec codec, const char *filename,
return -1;
switch (codec) {
case HTS_CODEC_DEFLATE:
#if HTS_USEZLIB
return hts_zunpack(filename, newfile);
#else
return -1;
#endif
case HTS_CODEC_BROTLI:
case HTS_CODEC_ZSTD:
break;
@@ -339,8 +348,10 @@ size_t hts_codec_head(hts_codec codec, const void *in, size_t in_len, void *out,
if (in == NULL || in_len == 0 || out == NULL || out_len == 0)
return 0;
switch (codec) {
#if HTS_USEZLIB
case HTS_CODEC_DEFLATE:
return hts_zhead(in, in_len, out, out_len);
#endif
#if HTS_USEBROTLI
case HTS_CODEC_BROTLI:
return codec_head_brotli(in, in_len, out, out_len);

View File

@@ -773,7 +773,7 @@ int httpmirror(char *url1, httrackp * opt) {
// sauter les + sans rien après..
if (strnotempty(tempo)) {
if ((plus == 0) && (type == 1)) { // implicite: *www.edf.fr par exemple
if (hts_lastchar(tempo) != '*') {
if (tempo[strlen(tempo) - 1] != '*') {
strcatbuff(tempo, "*"); // ajouter un *
}
}
@@ -1819,6 +1819,60 @@ int httpmirror(char *url1, httrackp * opt) {
// -- -- --
// sauver fichier
/* En cas d'erreur, vérifier que fichier d'erreur existe */
if (strnotempty(savename()) == 0) { // chemin de sauvegarde existant
if (strcmp(urlfil(), "/robots.txt") == 0) { // pas robots.txt
if (store_errpage) { // c'est une page d'erreur
int create_html_warning = 0;
int create_gif_warning = 0;
switch (ishtml(opt, urlfil())) { /* pas fichier html */
case 0: /* non html */
{
char buff[256];
guess_httptype_sized(opt, buff, sizeof(buff), urlfil());
if (strcmp(buff, "image/gif") == 0)
create_gif_warning = 1;
}
break;
case 1: /* html */
if (!r.adr) {
}
break;
default: /* don't know.. */
break;
}
/* Créer message d'erreur ? */
if (create_html_warning) {
char *adr =
(char *) malloct(strlen(HTS_DATA_ERROR_HTML) + 1100);
hts_log_print(opt, LOG_INFO, "Creating HTML warning file (%s)",
r.msg);
if (adr) {
if (r.adr) {
freet(r.adr);
r.adr = NULL;
}
sprintf(adr, HTS_DATA_ERROR_HTML, r.msg);
r.adr = adr;
}
} else if (create_gif_warning) {
char *adr = (char *) malloct(HTS_DATA_UNKNOWN_GIF_LEN);
hts_log_print(opt, LOG_INFO, "Creating GIF dummy file (%s)",
r.msg);
if (r.adr) {
freet(r.adr);
r.adr = NULL;
}
memcpy(adr, HTS_DATA_UNKNOWN_GIF, HTS_DATA_UNKNOWN_GIF_LEN);
r.adr = adr;
}
}
}
}
if (strnotempty(savename()) == 0) { // pas de chemin de sauvegarde
if (strcmp(urlfil(), "/robots.txt") == 0) { // robots.txt
char BIGSTK sitemaps[8192];
@@ -1926,9 +1980,10 @@ int httpmirror(char *url1, httrackp * opt) {
}
// ATTENTION C'EST ICI QU'ON SAUVE LE FICHIER!!
// A failed transfer has no body: r.adr holds debris from the aborted
// read, which would destroy the copy being re-fetched (#748).
if (r.statuscode > 0 && (r.adr != NULL || r.size == 0)) {
// An empty body must not overwrite the file when the transfer failed
// (statuscode <= 0, e.g. an -M hard-stop): it would truncate a good
// copy to 0 (#77 follow-up).
if (r.adr != NULL || (r.size == 0 && r.statuscode > 0)) {
file_notify(opt, urladr(), urlfil(), savename(), 1, 1, r.notmodified);
if (filesave(opt, r.adr, (int) r.size, savename(), urladr(), urlfil()) !=
0) {
@@ -2136,9 +2191,7 @@ int httpmirror(char *url1, httrackp * opt) {
continue;
strcpybuff(file, StringBuff(opt->path_html));
strcatbuff(file, line + 1);
/* strip filenote()'s ']', absent when linput() truncated the
line */
hts_striplastchar(file, ']');
file[strlen(file) - 1] = '\0';
hts_changes_previous(opt, file + StringLength(opt->path_html));
if (!strstr(adr, line)) { // not found in the new list?
if (fexist_utf8(file)) { // still on disk
@@ -2165,11 +2218,12 @@ int httpmirror(char *url1, httrackp * opt) {
fseek(old_lst, 0, SEEK_SET);
while(!feof(old_lst)) {
linput(old_lst, line, 1000);
while (strnotempty(line) && (hts_lastchar(line) != '/') &&
(hts_lastchar(line) != '\\')) {
hts_choplastchar(line);
while(strnotempty(line) && (line[strlen(line) - 1] != '/')
&& (line[strlen(line) - 1] != '\\')) {
line[strlen(line) - 1] = '\0';
}
hts_choplastchar(line);
if (strnotempty(line))
line[strlen(line) - 1] = '\0';
if (strnotempty(line))
if (!strstr(adr, line)) { // non trouvé?
char BIGSTK file[HTS_URLMAXSIZE * 2];
@@ -2182,12 +2236,13 @@ int httpmirror(char *url1, httrackp * opt) {
if (opt->log) {
hts_log_print(opt, LOG_INFO, "Purging directory %s/",
file);
while (strnotempty(file) &&
(hts_lastchar(file) != '/') &&
(hts_lastchar(file) != '\\')) {
hts_choplastchar(file);
while(strnotempty(file)
&& (file[strlen(file) - 1] != '/')
&& (file[strlen(file) - 1] != '\\')) {
file[strlen(file) - 1] = '\0';
}
hts_choplastchar(file);
if (strnotempty(file))
file[strlen(file) - 1] = '\0';
}
}
}
@@ -2638,11 +2693,7 @@ HTSEXT_API int structcheck(const char *path) {
if (!S_ISDIR(st.st_mode)) {
#if HTS_REMOVE_ANNOYING_INDEX
if (S_ISREG(st.st_mode)) { /* Regular file in place ; move it and create directory */
/* bounded here, not by the path-length guard far above */
if (!sprintfbuff(tmpbuf, "%s.txt", file)) {
errno = ENAMETOOLONG;
return -1;
}
sprintf(tmpbuf, "%s.txt", file);
if (rename(file, tmpbuf) != 0) { /* Can't rename regular file */
return -1;
}
@@ -2750,11 +2801,7 @@ HTSEXT_API int structcheck_utf8(const char *path) {
if (!S_ISDIR(st.st_mode)) {
#if HTS_REMOVE_ANNOYING_INDEX
if (S_ISREG(st.st_mode)) { /* Regular file in place ; move it and create directory */
/* bounded here, not by the path-length guard far above */
if (!sprintfbuff(tmpbuf, "%s.txt", file)) {
errno = ENAMETOOLONG;
return -1;
}
sprintf(tmpbuf, "%s.txt", file);
if (RENAME(file, tmpbuf) != 0) { /* Can't rename regular file */
return -1;
}
@@ -3774,12 +3821,11 @@ int htsAddLink(htsmoduleStruct * str, char *link) {
strcpybuff(codebase, heap(ptr)->fil);
else
strcpybuff(codebase, heap(heap(ptr)->precedent)->fil);
// empty codebase has no last char; codebase-1 would underflow
a = codebase[0] != '\0' ? codebase + strlen(codebase) - 1 : codebase;
a = codebase + strlen(codebase) - 1;
while((*a) && (*a != '/') && (a > codebase))
a--;
if (*a == '/')
*(a + 1) = '\0'; // cut
*(a + 1) = '\0'; // couper
} else { // couper http:// éventuel
if (strfield(codebase, "http://")) {
char BIGSTK tempo[HTS_URLMAXSIZE * 2];

View File

@@ -376,8 +376,8 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
const char *template_footer, const char *adr,
const char *fil);
// Flush ht_buff[0..ht_len] to save on disk; *fp closed+NULLed on write.
// Precondition: ht_len>0.
// Flush ht_buff[0..ht_len] to save on disk (skip if MD5 unchanged); *fp
// closed+NULLed on write. Precondition: ht_len>0.
void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r,
FILE **fp, const char *ht_buff, size_t ht_len,
const char *adr, const char *fil, const char *save);

View File

@@ -357,7 +357,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
char BIGSTK tempo[HTS_CDLMAXSIZE];
strcpybuff(tempo, argv[na] + 1);
if (hts_lastchar(tempo) != '"') {
if (tempo[0] == '\0' || tempo[strlen(tempo) - 1] != '"') {
char BIGSTK s[HTS_CDLMAXSIZE];
sprintf(s, "Missing quote in %s", argv[na]);
@@ -365,7 +365,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
htsmain_free();
return -1;
}
hts_choplastchar(tempo);
tempo[strlen(tempo) - 1] = '\0';
/* tempo is argv[na] minus its surrounding quotes, so it fits in place
*/
strlcpybuff(argv[na], tempo, strlen(argv[na]) + 1);
@@ -863,7 +863,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
char BIGSTK tempo[HTS_CDLMAXSIZE + 256];
strcpybuff(tempo, argv[na] + 1);
if (hts_lastchar(tempo) != '"') {
if (tempo[0] == '\0' || tempo[strlen(tempo) - 1] != '"') {
char s[HTS_CDLMAXSIZE + 256];
sprintf(s, "Missing quote in %s", argv[na]);
@@ -871,7 +871,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
htsmain_free();
return -1;
}
hts_choplastchar(tempo);
tempo[strlen(tempo) - 1] = '\0';
/* tempo is argv[na] minus its surrounding quotes, so it fits in place
*/
strlcpybuff(argv[na], tempo, strlen(argv[na]) + 1);
@@ -2724,7 +2724,10 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
char *a;
strcpybuff(rpath, StringBuff(opt->path_html));
hts_striplastchar(rpath, '/');
if (rpath[0]) {
if (rpath[strlen(rpath) - 1] == '/')
rpath[strlen(rpath) - 1] = '\0';
}
a = strrchr(rpath, '/');
if (a) {
*a = '\0';

View File

@@ -483,19 +483,16 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.totalsize = size;
}
}
/* Only over a copy back_add() judged partial: on --update every
mirrored file exists, and resuming a complete one splices the
old body into the new (#798). */
if (back->range_req_size > 0 && (transfer_list == 0)) {
// REST?
if (fexist(back->url_sav) && (transfer_list == 0)) {
strcpybuff(back->info, "rest");
snprintf(line, sizeof(line), "REST " LLintP,
(LLint) back->range_req_size);
snprintf(line, sizeof(line), "REST " LLintP, (LLint) fsize(back->url_sav));
send_line(soc_ctl, line);
get_ftp_line(soc_ctl, line, sizeof(line), timeout);
_CHECK_HALT_FTP;
if ((line[0] == '3') || (line[0] == '2')) { // ok
rest_understood = 1;
} // else never mind
} // sinon tant pis
}
} // sinon tant pis
}
@@ -620,19 +617,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// Ok, connexion initiée
//
if (soc_dat != INVALID_SOCKET) {
if (rest_understood) { // REST sent and understood
if (rest_understood) { // REST envoyée et comprise
file_notify(opt, back->url_adr, back->url_fil, back->url_sav, 0, 1,
0);
/* The bytes already on disk count too, or the completeness check
below rejects every resumed transfer (#798). */
back->r.size = back->range_req_size;
back->r.fp = fileappend(&opt->state.strc, back->url_sav);
} else {
file_notify(opt, back->url_adr, back->url_fil, back->url_sav, 1, 1,
0);
/* Every failure exit below would else leave the mirror truncated
(#771); the resume branch appends and needs no backup. */
back_refetch_backup(opt, back);
back->r.fp = filecreate(&opt->state.strc, back->url_sav);
}
strcpybuff(back->info, "receiving");

View File

@@ -83,8 +83,9 @@ void infomsg(const char *msg) {
/* try the flag as-is, then strip a trailing N as the numeric-arg
placeholder (cN -> c); this order keeps -%N from becoming -% */
p = optreal_find(cmd);
if (p < 0 && (int) strlen(cmd) > 2 && hts_lastchar(cmd) == 'N') {
hts_striplastchar(cmd, 'N');
if (p < 0 && (int) strlen(cmd) > 2 &&
cmd[strlen(cmd) - 1] == 'N') {
cmd[strlen(cmd) - 1] = '\0';
p = optreal_find(cmd);
}
if (p >= 0) {
@@ -210,7 +211,7 @@ void help_wizard(httrackp * opt) {
strcatbuff(str, "/websites/");
}
if (strnotempty(str))
if ((hts_lastchar(str) != '/') && (hts_lastchar(str) != '\\'))
if ((str[strlen(str) - 1] != '/') && (str[strlen(str) - 1] != '\\'))
strcatbuff(str, "/");
strcatbuff(stropt2, "-O \"");
strcatbuff(stropt2, str);
@@ -558,7 +559,8 @@ void help(const char *app, int more) {
infomsg
(" %q *include query string for local files (useless, for information purpose only) (%q0 don't include)");
infomsg(" %g strip query keys for dedup ([host/pattern=]key1,key2,...)");
infomsg(" o *save the server's error pages (404..) (o0 discard them)");
infomsg
(" o *generate output html file in case of error (404..) (o0 don't generate)");
infomsg(" X *purge old files after update (X0 keep delete)");
infomsg(" %p preserve html files 'as is' (identical to '-K4 -%F \"\"')");
infomsg(" %T links conversion to UTF-8");

View File

@@ -1131,10 +1131,12 @@ int http_sendhead(httrackp * opt, t_cookie * cookie, int mode,
// Compression accepted ?
if (retour->req.http11) {
hts_boolean compressible =
(!retour->req.range_used && !retour->req.nocompression);
hts_boolean compressible = HTS_FALSE;
hts_boolean secure = HTS_FALSE;
#if HTS_USEZLIB
compressible = (!retour->req.range_used && !retour->req.nocompression);
#endif
#if HTS_USEOPENSSL
secure = retour->ssl ? HTS_TRUE : HTS_FALSE;
#endif
@@ -2703,8 +2705,17 @@ void hts_now_iso8601(char out[32]) {
time_t t = time(NULL);
struct tm tmv;
if (!hts_gmtime(t, &tmv))
#if defined(_WIN32)
struct tm *g = gmtime(&t);
if (g != NULL)
tmv = *g;
else
memset(&tmv, 0, sizeof(tmv));
#else
if (gmtime_r(&t, &tmv) == NULL)
memset(&tmv, 0, sizeof(tmv));
#endif
strftime(out, 32, "%Y-%m-%dT%H:%M:%SZ", &tmv);
}
@@ -5069,7 +5080,7 @@ static int hts_dns_resolve_nocache_list(const char *const hostname,
if (!strnotempty(hostname) || max <= 0) {
return 0;
}
if ((hostname[0] == '[') && (hts_lastchar(hostname) == ']')) {
if ((hostname[0] == '[') && (hostname[strlen(hostname) - 1] == ']')) {
size_t size = strlen(hostname);
char *copy = malloct(size + 1);
int count;
@@ -5482,8 +5493,9 @@ void cut_path(char *fullpath, char *path, size_t path_size, char *pname,
size_t pname_size) {
path[0] = pname[0] = '\0';
if (strnotempty(fullpath)) {
if (!hts_striplastchar(fullpath, '/'))
hts_striplastchar(fullpath, '\\');
if ((fullpath[strlen(fullpath) - 1] == '/')
|| (fullpath[strlen(fullpath) - 1] == '\\'))
fullpath[strlen(fullpath) - 1] = '\0';
if (strlen(fullpath) > 1) {
char *a;
@@ -5781,8 +5793,7 @@ HTSEXT_API void hts_log_vprint(httrackp * opt, int type, const char *format, va_
if (hts_log_print_callback != NULL) {
va_list args_copy;
va_copy(args_copy, args);
/* the copy, so the vfprintf() below still has an unread list */
hts_log_print_callback(opt, type, format, args_copy);
hts_log_print_callback(opt, type, format, args);
va_end(args_copy);
}
if (opt != NULL && opt->log != NULL) {
@@ -6639,12 +6650,9 @@ int hts_rename_utf8(const char *oldpath, const char *newpath) {
LPWSTR wnewpath = hts_pathToUCS2(newpath);
if (woldpath != NULL && wnewpath != NULL) {
const int result = _wrename(woldpath, wnewpath);
/* Save errno: callers key off it (#779) and free() may clobber it. */
const int err = errno;
free(woldpath);
free(wnewpath);
errno = err;
return result;
} else {
if (woldpath != NULL)

View File

@@ -157,19 +157,6 @@ struct t_dnscache {
char host_addr[HTS_MAXADDRNUM][HTS_MAXADDRLEN];
};
/* Break t down as UTC into the caller's buffer, HTS_FALSE if that failed.
gmtime()'s static is shared, and both the engine and ProxyTrack convert on
worker threads. */
static HTS_INLINE HTS_UNUSED hts_boolean hts_gmtime(time_t t,
struct tm *tmbuf) {
#ifdef _WIN32
/* Microsoft's gmtime_s takes the destination first, unlike C11 Annex K. */
return gmtime_s(tmbuf, &t) == 0 ? HTS_TRUE : HTS_FALSE;
#else
return gmtime_r(&t, tmbuf) != NULL ? HTS_TRUE : HTS_FALSE;
#endif
}
/* Library internal definictions */
#ifdef HTS_INTERNAL_BYTECODE

View File

@@ -43,7 +43,9 @@ Please visit our Website: http://www.httrack.com
#include "htsencoding.h"
#include "htssniff.h"
#include "htscodec.h"
#if HTS_USEZLIB
#include "htszlib.h"
#endif
#include <ctype.h>
#include <limits.h>
@@ -437,7 +439,9 @@ int url_savename(lien_adrfilsave *const afs,
strcpybuff(fil_complete_patche, normfil);
// Version avec ou sans /
if (!hts_striplastchar(fil_complete_patche, '/'))
if (fil_complete_patche[strlen(fil_complete_patche) - 1] == '/')
fil_complete_patche[strlen(fil_complete_patche) - 1] = '\0';
else
strcatbuff(fil_complete_patche, "/");
i = hash_read(hash, normadr, fil_complete_patche, HASH_STRUCT_ORIGINAL_ADR_PATH); // recherche table 2 (former->adr+former->fil)
if (i >= 0) {
@@ -513,8 +517,7 @@ int url_savename(lien_adrfilsave *const afs,
&& protocol != PROTOCOL_FTP
) {
// tester type avec requète HEAD si on ne connait pas le type du fichier
if (!((opt->check_type == 1) &&
(hts_lastchar(fil) == '/'))) // slash doit être html?
if (!((opt->check_type == 1) && (fil[strlen(fil) - 1] == '/'))) // slash doit être html?
if (opt->savename_delayed == HTS_SAVENAME_DELAYED_HARD ||
ishtml(opt, fil) < 0) { // unsure whether it's html or a file
// lire dans le cache
@@ -809,7 +812,7 @@ int url_savename(lien_adrfilsave *const afs,
// - - - DEBUT NOMMAGE - - -
// Donner nom par défaut?
if (hts_lastchar(fil) == '/') {
if (fil[strlen(fil) - 1] == '/') {
if (!strfield(adr_complete, "ftp://")
) {
strcatbuff(fil, DEFAULT_HTML); // nommer page par défaut!!
@@ -1283,7 +1286,7 @@ int url_savename(lien_adrfilsave *const afs,
hts_lowcase(afs->save);
if (hts_lastchar(afs->save) == '/')
if (afs->save[strlen(afs->save) - 1] == '/')
strcatbuff(afs->save, DEFAULT_HTML); // nommer page par défaut!!
}

View File

@@ -1701,8 +1701,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
#endif
) // ok pas de problème
url_ok = 1;
else if (hts_lastchar(tempo) ==
'/') { // un slash: ok..
else if (tempo[strlen(tempo) - 1] == '/') { // un slash: ok..
if (inscript) // sinon si pas javascript, méfiance (répertoire style base?)
url_ok = 1;
}
@@ -2006,8 +2005,9 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
if (eadr - html - 1 < HTS_URLMAXSIZE) { // pas trop long?
strncpy(lien, html, eadr - html - 1);
lien[eadr - html - 1] = '\0';
while (hts_striplastchar(lien, ' ')) {
}
// supprimer les espaces
while((lien[strlen(lien) - 1] == ' ') && (strnotempty(lien)))
lien[strlen(lien) - 1] = '\0';
} else
lien[0] = '\0'; // erreur
@@ -2207,7 +2207,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
// supposition dangereuse?
// OUI!!
#if HTS_TILDE_SLASH
if (hts_lastchar(lien) != '/') {
if (lien[strlen(lien) - 1] != '/') {
char *a = lien + strlen(lien) - 1;
// éviter aussi index~1.html
@@ -2272,7 +2272,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
// Vérifier les codebase=applet (au lieu de applet/)
if (p_type == -2) { // codebase
if (strnotempty(lien)) {
if (hts_lastchar(lien) != '/') { // pas répertoire
if (lien[strlen(lien) - 1] != '/') { // pas répertoire
strcatbuff(lien, "/");
}
}
@@ -2688,11 +2688,9 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
int cat_data_len = 0;
// ajouter lien external
switch ((link_has_authority(afs.af.adr))
? 1
: ((hts_lastchar(afs.af.fil) == '/')
? 1
: (ishtml(opt, afs.af.fil)))) {
switch ((link_has_authority(afs.af.adr)) ? 1
: ((afs.af.fil[strlen(afs.af.fil) - 1] ==
'/') ? 1 : (ishtml(opt, afs.af.fil)))) {
case 1:
case -2: // html ou répertoire
if (opt->getmode & HTS_GETMODE_HTML) {
@@ -2735,7 +2733,7 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
cat_data_len = HTS_DATA_UNKNOWN_HTML_LEN;
}
break;
} // html,gif
} // html,gif
if (patch_it) {
char BIGSTK save[HTS_URLMAXSIZE * 2];

View File

@@ -539,36 +539,6 @@ static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
#endif
/* Last character of s, or '\0' when s is empty. Replaces s[strlen(s) - 1],
which indexes one byte before the buffer on an empty string. */
static HTS_INLINE HTS_UNUSED char hts_lastchar(const char *s) {
const size_t len = strlen(s);
return len != 0 ? s[len - 1] : '\0';
}
/* Drop a trailing c from s if present; HTS_TRUE if one was dropped. */
static HTS_INLINE HTS_UNUSED hts_boolean hts_striplastchar(char *s, char c) {
const size_t len = strlen(s);
if (len != 0 && s[len - 1] == c) {
s[len - 1] = '\0';
return HTS_TRUE;
}
return HTS_FALSE;
}
/* Drop the last character of s whatever it is; HTS_TRUE if s was not empty. */
static HTS_INLINE HTS_UNUSED hts_boolean hts_choplastchar(char *s) {
const size_t len = strlen(s);
if (len != 0) {
s[len - 1] = '\0';
return HTS_TRUE;
}
return HTS_FALSE;
}
/* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
added bounds checking. freet() also NULLs the freed pointer and tolerates
NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

File diff suppressed because it is too large Load Diff

View File

@@ -1050,9 +1050,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
if (!linput(fp, line, sizeof(line) - 2)) {
*str = '\0';
}
if (hts_lastchar(str) == '\\') {
if (*str && str[strlen(str) - 1] == '\\') {
nocr = 1;
hts_striplastchar(str, '\\');
str[strlen(str) - 1] = '\0';
}
while(*str) {
char *pos;
@@ -1169,8 +1169,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
char *rpath = (char *) adr;
//find_handle h;
/* note: patching stored (inhash) value */
hts_striplastchar(rpath, '/');
if (rpath[0]) {
if (rpath[strlen(rpath) - 1] == '/') {
rpath[strlen(rpath) - 1] = '\0'; /* note: patching stored (inhash) value */
}
}
{
const char *profiles = hts_getcategories(rpath, 0);
const char *categ = hts_getcategories(rpath, 1);

View File

@@ -1108,8 +1108,13 @@ hts_boolean singlefile_rewrite_file(httrackp *opt, const char *root,
(void) chmod(fconv(catbuff, sizeof(catbuff), StringBuff(tmp)),
HTS_ACCESS_FILE);
#endif
if (ok)
ok = hts_rename_over(opt, StringBuff(tmp), page_path);
if (ok) {
/* RENAME does not clobber an existing target on Windows. */
if (RENAME(StringBuff(tmp), page_path) != 0) {
(void) UNLINK(page_path);
ok = RENAME(StringBuff(tmp), page_path) == 0 ? HTS_TRUE : HTS_FALSE;
}
}
if (!ok) {
hts_log_print(opt, LOG_ERROR, "single-file: could not rewrite %s",
page_path);

View File

@@ -985,7 +985,10 @@ HTSEXT_API int hts_buildtopindex(httrackp * opt, const char *path,
&& toptemplate_bodycat) {
strcpybuff(rpath, path);
hts_striplastchar(rpath, '/');
if (rpath[0]) {
if (rpath[strlen(rpath) - 1] == '/')
rpath[strlen(rpath) - 1] = '\0';
}
fpo = fopen(fconcat(catbuff, sizeof(catbuff), rpath, "/index.html"), "wb");
if (fpo) {
@@ -1198,8 +1201,11 @@ HTSEXT_API char *hts_getcategories(char *path, int type) {
find_handle h;
coucal hashCateg = NULL;
/* note: patching stored (inhash) value */
hts_striplastchar(rpath, '/');
if (rpath[0]) {
if (rpath[strlen(rpath) - 1] == '/') {
rpath[strlen(rpath) - 1] = '\0'; /* note: patching stored (inhash) value */
}
}
h = hts_findfirst(rpath);
if (h) {
String iname = STRING_EMPTY;
@@ -1296,7 +1302,7 @@ HTSEXT_API find_handle hts_findfirst(char *path) {
strcpybuff(rpath, path);
if (rpath[0]) {
if (hts_lastchar(rpath) != '\\')
if (rpath[strlen(rpath) - 1] != '\\')
strcatbuff(rpath, "\\");
}
strcatbuff(rpath, "*.*");
@@ -1308,7 +1314,7 @@ HTSEXT_API find_handle hts_findfirst(char *path) {
strcpybuff(find->path, path);
{
if (find->path[0]) {
if (hts_lastchar(find->path) != '/')
if (find->path[strlen(find->path) - 1] != '/')
strcatbuff(find->path, "/");
}
}
@@ -1437,76 +1443,3 @@ HTSEXT_API hts_boolean hts_findissystem(find_handle find) {
}
return 0;
}
/* Park cdst under a free sibling name; caside receives it. */
static hts_boolean rename_park_aside(char *caside, size_t size,
const char *cdst) {
int i;
for (i = 0; i < 16; i++) {
if (!slprintfbuff(caside, size, "%s.hts-old%d", cdst, i))
return HTS_FALSE;
/* Skip a name the mirror already holds: POSIX rename() would clobber it
(#774). A non-regular entry reads as free and the rename refuses it. */
if (fexist_utf8(caside))
continue;
if (RENAME(cdst, caside) == 0)
return HTS_TRUE;
}
return HTS_FALSE;
}
/* cdst is in the way of the move: park it, retry, and put it back if the retry
fails too. Unlinking it instead would leave nothing at all (#790). */
static hts_boolean rename_over_aside(httrackp *opt, const char *csrc,
const char *cdst) {
char caside[CATBUFF_SIZE];
int err;
/* Only a regular file may be parked: a directory in the way is not what the
caller asked to replace, and parking it orphans it (UNLINK cannot drop). */
if (!fexist_utf8(cdst))
return HTS_FALSE;
if (!rename_park_aside(caside, sizeof(caside), cdst))
return HTS_FALSE;
if (RENAME(csrc, cdst) == 0) {
(void) UNLINK(caside);
return HTS_TRUE;
}
err = errno;
/* Retry once, then name the parked copy: nothing else on disk or in the log
points at it, and an --update purge would delete it unnoticed. */
if (RENAME(caside, cdst) != 0 && RENAME(caside, cdst) != 0)
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"could not put %s back; its previous content is now %s", cdst,
caside);
errno = err;
return HTS_FALSE;
}
hts_boolean hts_rename_over(httrackp *opt, const char *src, const char *dst) {
char csrc[CATBUFF_SIZE], cdst[CATBUFF_SIZE];
fconv(csrc, sizeof(csrc), src);
fconv(cdst, sizeof(cdst), dst);
if (RENAME(csrc, cdst) == 0)
return HTS_TRUE;
/* Only a dst in the way is something the fallback can clear, and the CRT maps
that to EEXIST; it keeps EACCES for a src another process holds, where the
retry would fail the same way. The src check covers a CRT that reports
neither. */
const int err = errno;
if (err != EEXIST || !fexist_utf8(src))
return HTS_FALSE;
return rename_over_aside(opt, csrc, cdst);
}
hts_boolean hts_rename_over_aside_selftest(httrackp *opt, const char *src,
const char *dst) {
char csrc[CATBUFF_SIZE], cdst[CATBUFF_SIZE];
fconv(csrc, sizeof(csrc), src);
fconv(cdst, sizeof(cdst), dst);
return rename_over_aside(opt, csrc, cdst);
}

View File

@@ -137,18 +137,6 @@ HTSEXT_API hts_boolean hts_findisdir(find_handle find);
HTSEXT_API hts_boolean hts_findisfile(find_handle find);
HTSEXT_API hts_boolean hts_findissystem(find_handle find);
/* Move src onto dst, replacing an existing dst; HTS_TRUE on success. Both paths
are fconv()'d. A dst in the way is parked under a sibling name rather than
removed, so the old content survives a failure: back at dst, or under that
sibling (named in the log) when the move back failed too. Not atomic: a crash
between the two renames leaves dst absent and its content beside it. */
hts_boolean hts_rename_over(httrackp *opt, const char *src, const char *dst);
/* Selftest hook: run the aside fallback directly, on a platform whose rename()
never reaches it. Both paths are fconv()'d. */
hts_boolean hts_rename_over_aside_selftest(httrackp *opt, const char *src,
const char *dst);
#endif
#endif

View File

@@ -57,9 +57,6 @@ Please visit our Website: http://www.httrack.com
/* opt->state.warc value meaning "open failed once, do not retry". */
#define WARC_DISABLED ((void *) ~(uintptr_t) 0)
/* Suffix of the in-progress archive when a previous one must survive it. */
#define WARC_TMP_SUFFIX ".tmp"
struct warc_writer {
FILE *f;
httrackp *opt; /* kept for close-time logging (warc_wacz_package) */
@@ -89,11 +86,6 @@ struct warc_writer {
char *base_path; /* resolved archive path minus .warc[.gz] suffix */
const char *base_ext; /* ".warc.gz" or ".warc" (static) */
char *arc_path; /* full single-file archive path (NULL under rotation) */
/* A re-run must not destroy an archive it cannot replace (#759). */
hts_boolean protect_prev; /* previous archive present: build in a temp */
hts_boolean opened; /* open completed; a failed one swaps nothing */
hts_boolean failed; /* a record or segment was lost: swap nothing */
uint64_t unbacked_revisits; /* revisits whose payload no file here holds */
char **page_lines; /* one JSON page line per 200 text/html response, owned */
size_t page_count;
size_t page_cap;
@@ -127,32 +119,24 @@ static void wbuf_free(wbuf *b) {
b->len = b->cap = 0;
}
/* Make room for n more bytes; returns 0 on success, -1 on OOM/overflow. */
static int wbuf_reserve(wbuf *b, size_t n) {
size_t ncap;
char *nd;
if (n > (size_t) -1 - b->len)
return -1;
if (b->len + n <= b->cap)
return 0;
ncap = b->cap ? b->cap : 256;
while (ncap < b->len + n) {
if (ncap > (size_t) -1 / 2)
return -1;
ncap *= 2;
}
nd = realloct(b->data, ncap);
if (nd == NULL)
return -1;
b->data = nd;
b->cap = ncap;
return 0;
}
/* Append n bytes; returns 0 on success, -1 on OOM/overflow. */
static int wbuf_add(wbuf *b, const void *p, size_t n) {
if (wbuf_reserve(b, n) != 0)
if (n > (size_t) -1 - b->len)
return -1;
if (b->len + n > b->cap) {
size_t ncap = b->cap ? b->cap : 256;
char *nd;
while (ncap < b->len + n) {
if (ncap > (size_t) -1 / 2)
return -1;
ncap *= 2;
}
nd = realloct(b->data, ncap);
if (nd == NULL)
return -1;
b->data = nd;
b->cap = ncap;
}
memcpy(b->data + b->len, p, n);
b->len += n;
return 0;
@@ -164,32 +148,16 @@ static int wbuf_puts(wbuf *b, const char *s) {
static int wbuf_printf(wbuf *b, const char *fmt, ...) HTS_PRINTF_FUN(2, 3);
/* A header line carrying a URL has no useful bound, and giving up here loses
the whole record, so anything past the stack buffer is formatted into the
wbuf itself rather than rejected (#785). */
static int wbuf_printf(wbuf *b, const char *fmt, ...) {
char tmp[1024];
size_t need;
int n;
va_list ap;
va_start(ap, fmt);
n = vsnprintf(tmp, sizeof(tmp), fmt, ap);
va_end(ap);
if (n < 0)
if (n < 0 || (size_t) n >= sizeof(tmp))
return -1;
if ((size_t) n < sizeof(tmp))
return wbuf_add(b, tmp, (size_t) n);
need = (size_t) n + 1; /* +1: vsnprintf always writes the NUL */
if (wbuf_reserve(b, need) != 0)
return -1;
va_start(ap, fmt);
n = vsnprintf(b->data + b->len, need, fmt, ap);
va_end(ap);
/* Never advance past what was reserved, whatever the second pass returns. */
if (n < 0 || (size_t) n >= need)
return -1;
b->len += (size_t) n; /* the NUL is scratch, overwritten by the next append */
return 0;
return wbuf_add(b, tmp, (size_t) n);
}
/* ---- gzip-per-record member writer (mirrors ae_write_packed) ---- */
@@ -996,6 +964,16 @@ static zipFile wacz_zip_open(const char *path) {
return zipOpen2_64(path, 0 /*create*/, NULL, &ff);
}
/* Move src onto dst (UTF-8/Windows-safe); RENAME won't clobber on Windows, so
fall back to unlink+rename. Returns 0 on success. */
static int wacz_rename_over(const char *src, const char *dst) {
char cs[CATBUFF_SIZE], cd[CATBUFF_SIZE];
if (RENAME(fconv(cs, sizeof(cs), src), fconv(cd, sizeof(cd), dst)) == 0)
return 0;
(void) UNLINK(fconv(cd, sizeof(cd), dst));
return RENAME(fconv(cs, sizeof(cs), src), fconv(cd, sizeof(cd), dst));
}
/* Package the segment(s) + .cdx + a generated pages.jsonl into <base>.wacz at
crawl end (the archive file(s) and .cdx are already closed on disk). */
static void warc_wacz_package(warc_writer *w) {
@@ -1114,7 +1092,7 @@ static void warc_wacz_package(warc_writer *w) {
hts_log_print(w->opt, LOG_WARNING,
"WACZ: packaging failed, kept existing %s untouched",
waczpath);
} else if (!hts_rename_over(w->opt, tmppath, waczpath)) {
} else if (wacz_rename_over(tmppath, waczpath) != 0) {
(void) UNLINK(fconv(catbuff, sizeof(catbuff), tmppath));
hts_log_print(w->opt, LOG_WARNING | LOG_ERRNO,
"WACZ: could not finalize %s", waczpath);
@@ -1161,17 +1139,13 @@ static int warc_emit(warc_writer *w, const char *type, const char *content_type,
never split a record, and never rotate a warcinfo (it opens a segment). */
if (w->max_size > 0 && w->seg_base != NULL && w->offset >= w->max_size &&
strcmp(type, "warcinfo") != 0) {
if (warc_rotate(w) != 0) {
w->failed = HTS_TRUE;
if (warc_rotate(w) != 0)
return -1;
}
}
/* F4: overflow-safe block length; http_hdr_len+sep is provably small. */
if (payload > (size_t) -1 - http_hdr_len - sep) {
w->failed = HTS_TRUE;
if (payload > (size_t) -1 - http_hdr_len - sep)
return -1;
}
block_len = http_hdr_len + sep + payload;
memset(&hdr, 0, sizeof(hdr));
@@ -1277,23 +1251,11 @@ static int warc_emit(warc_writer *w, const char *type, const char *content_type,
rc = 0;
done:
wbuf_free(&hdr);
if (rc != 0)
w->failed = HTS_TRUE; /* a truncated run must not replace a whole one */
return rc;
}
/* ---- segment rotation (--warc-max-size) ---- */
/* Path to open for the segment whose final path is `final`: that path itself,
or a sibling temp while a previous archive must survive until close. */
static const char *warc_open_path(warc_writer *w, const char *final, char *buf,
size_t bufsz) {
if (!w->protect_prev)
return final;
snprintf(buf, bufsz, "%s" WARC_TMP_SUFFIX, final);
return buf;
}
/* Emit the warcinfo that heads a segment; sets w->info_id for its records. */
static int warc_write_warcinfo_record(warc_writer *w) {
w->info_id[0] = '\0'; /* warcinfo itself carries no WARC-Warcinfo-ID */
@@ -1305,23 +1267,17 @@ static int warc_write_warcinfo_record(warc_writer *w) {
static int warc_rotate(warc_writer *w) {
char namebuf[HTS_URLMAXSIZE * 2];
char openbuf[HTS_URLMAXSIZE * 2 + sizeof(WARC_TMP_SUFFIX)];
char catbuff[CATBUFF_SIZE];
const unsigned next = w->seg + 1;
FILE *f;
snprintf(namebuf, sizeof(namebuf), "%s-%05u%s", w->seg_base, next,
w->seg_ext);
/* Open before advancing: w->seg must only ever name a segment that exists,
or close-time packaging and swapping would work on a missing file. */
f = FOPEN(fconv(catbuff, sizeof(catbuff),
warc_open_path(w, namebuf, openbuf, sizeof(openbuf))),
"wb");
if (f == NULL)
return -1;
if (w->f != NULL)
if (w->f != NULL) {
fclose(w->f);
w->f = f;
w->seg = next;
w->f = NULL;
}
w->seg++;
snprintf(namebuf, sizeof(namebuf), "%s-%05u%s", w->seg_base, w->seg,
w->seg_ext);
w->f = FOPEN(fconv(catbuff, sizeof(catbuff), namebuf), "wb");
if (w->f == NULL)
return -1;
w->offset = 0;
if (w->cdx_on) {
freet(w->cur_seg);
@@ -1378,7 +1334,6 @@ void warc_adopt_rawspool(htsblk *r, const char *tmpfile_path) {
warc_writer *warc_open(httrackp *opt, const char *path) {
warc_writer *w;
char namebuf[HTS_URLMAXSIZE * 2];
char openbuf[HTS_URLMAXSIZE * 2 + sizeof(WARC_TMP_SUFFIX)];
char catbuff[CATBUFF_SIZE];
wbuf info;
const char *robots;
@@ -1392,8 +1347,16 @@ warc_writer *warc_open(httrackp *opt, const char *path) {
char ts[32];
time_t t = time(NULL);
struct tm tmv;
if (!hts_gmtime(t, &tmv))
#if defined(_WIN32)
struct tm *g = gmtime(&t);
if (g != NULL)
tmv = *g;
else
memset(&tmv, 0, sizeof(tmv));
#else
if (gmtime_r(&t, &tmv) == NULL)
memset(&tmv, 0, sizeof(tmv));
#endif
strftime(ts, sizeof(ts), "%Y%m%d%H%M%S", &tmv);
snprintf(catbuff, sizeof(catbuff), "httrack-%s.warc.gz", ts);
path =
@@ -1521,105 +1484,35 @@ warc_writer *warc_open(httrackp *opt, const char *path) {
path = namebuf;
}
if (w->max_size == 0 && (w->arc_path = strdupt(path)) == NULL) {
warc_close(w);
return NULL;
}
/* Set only once arc_path is recorded, so a half-built writer never swaps. */
w->protect_prev = fsize_utf8(path) > 0 ? HTS_TRUE : HTS_FALSE;
w->f = FOPEN(fconv(catbuff, sizeof(catbuff),
warc_open_path(w, path, openbuf, sizeof(openbuf))),
"wb");
w->f = FOPEN(fconv(catbuff, sizeof(catbuff), path), "wb");
if (w->f == NULL) {
warc_close(w);
return NULL;
}
if (w->cdx_on)
w->cur_seg = path_basename_dup(path);
if (w->wacz_on && w->max_size == 0)
w->arc_path = strdupt(path); /* single-file: package this exact path */
if (warc_write_warcinfo_record(w) != 0) {
warc_close(w);
return NULL;
}
w->opened = HTS_TRUE;
return w;
}
/* Final path of segment s (the run's only archive when rotation is off). */
static const char *warc_seg_path(warc_writer *w, unsigned s, char *buf,
size_t bufsz) {
if (w->max_size == 0)
return w->arc_path;
snprintf(buf, bufsz, "%s-%05u%s", w->seg_base, s, w->seg_ext);
return buf;
}
/* Swap this run's archive into place, unless it only holds revisits naming
bodies the previous archive still has and this one does not (#759).
HTS_FALSE: the previous archive was kept, so leave its .cdx and .wacz too. */
static hts_boolean warc_commit(warc_writer *w) {
char finalbuf[HTS_URLMAXSIZE * 2];
char tmpbuf[HTS_URLMAXSIZE * 2 + sizeof(WARC_TMP_SUFFIX)];
char catbuff[CATBUFF_SIZE];
const unsigned nseg = (w->max_size > 0) ? w->seg + 1 : 1;
hts_boolean swap;
unsigned s;
if (!w->protect_prev)
return HTS_TRUE; /* nothing was there to lose: written in place */
/* All or nothing: a swap stopping halfway would mix this run's segments with
the previous one's, so require every temp before renaming any. */
swap = w->opened && !w->failed && w->unbacked_revisits == 0;
for (s = 0; s < nseg && swap; s++) {
snprintf(tmpbuf, sizeof(tmpbuf), "%s" WARC_TMP_SUFFIX,
warc_seg_path(w, s, finalbuf, sizeof(finalbuf)));
swap = fsize_utf8(tmpbuf) > 0;
}
if (swap) {
for (s = 0; s < nseg; s++) {
const char *final = warc_seg_path(w, s, finalbuf, sizeof(finalbuf));
snprintf(tmpbuf, sizeof(tmpbuf), "%s" WARC_TMP_SUFFIX, final);
if (!hts_rename_over(w->opt, tmpbuf, final)) {
hts_log_print(w->opt, LOG_ERROR | LOG_ERRNO,
"WARC: could not replace %s", final);
return HTS_FALSE;
}
}
return HTS_TRUE;
}
for (s = 0; s < nseg; s++) {
snprintf(tmpbuf, sizeof(tmpbuf), "%s" WARC_TMP_SUFFIX,
warc_seg_path(w, s, finalbuf, sizeof(finalbuf)));
(void) UNLINK(fconv(catbuff, sizeof(catbuff), tmpbuf));
}
if (w->unbacked_revisits > 0)
hts_log_print(
w->opt, LOG_ERROR,
"WARC: this pass revisited %llu URL(s) without re-downloading them, so "
"its archive would name bodies no file holds; kept the previous %s "
"(re-run with -C0, or --warc-file with a name of its own)",
(unsigned long long) w->unbacked_revisits,
warc_seg_path(w, 0, finalbuf, sizeof(finalbuf)));
return HTS_FALSE;
}
void warc_close(warc_writer *w) {
size_t i;
if (w == NULL)
return;
warc_cdx_flush(w); /* sort + write <base>.cdx before tearing down */
if (w->f != NULL)
fclose(w->f);
w->f = NULL;
if (warc_commit(w)) {
warc_cdx_flush(w); /* sort + write <base>.cdx beside the archive */
#if HTS_USEOPENSSL
if (w->wacz_on) /* package once the segment(s) + .cdx are closed on disk */
warc_wacz_package(w);
if (w->wacz_on) /* package once the segment(s) + .cdx are closed on disk */
warc_wacz_package(w);
#endif
}
if (w->seen != NULL)
coucal_delete(&w->seen);
for (i = 0; i < w->cdx_count; i++)
@@ -1700,12 +1593,6 @@ int warc_write_transaction(warc_writer *w, const char *target_uri,
if (is_update_unchanged) {
is_revisit = 1;
profile = "http://netpreserve.org/warc/1.1/revisit/server-not-modified";
/* Replay resolves a revisit by this field alone, and a 304 stands in for
the same URL. No WARC-Refers-To-Date to go with it: the cache keeps the
document's Last-Modified, never the previous capture time. */
refers_uri = target_uri;
/* Served from cache: the payload sits in the previous archive, not here. */
w->unbacked_revisits++;
} else if (have_pdig && w->seen != NULL) {
void *prev = NULL;
if (coucal_read_pvoid(w->seen, pdig, &prev) && prev != NULL) {

View File

@@ -224,7 +224,9 @@ int main(int argc, char *argv[]) {
#ifdef HTS_USESWF
smallserver_setkey("USESWF", "1");
#endif
#ifdef HTS_USEZLIB
smallserver_setkey("USEZLIB", "1");
#endif
#ifdef _WIN32
smallserver_setkey("WIN32", "1");
#endif

View File

@@ -40,6 +40,7 @@ Please visit our Website: http://www.httrack.com
#include "htscodec.h"
#include "htszlib.h"
#if HTS_USEZLIB
/* zlib */
/*
#include <zlib.h>
@@ -273,3 +274,4 @@ const char *hts_get_zerror(int err) {
break;
}
}
#endif

View File

@@ -520,13 +520,12 @@ static void proxytrack_add_DAV_Item(String * item, String * buff,
const char *filename, size_t size,
time_t timestamp, const char *mime,
int isDir, int isRoot, int isDefault) {
struct tm timetmbuf;
struct tm *timetm = &timetmbuf;
struct tm *timetm;
if (timestamp == (time_t) 0 || timestamp == (time_t) - 1) {
timestamp = time(NULL);
}
if (hts_gmtime(timestamp, timetm)) {
if ((timetm = gmtime(&timestamp)) != NULL) {
char tms[256 + 1];
const char *name;

View File

@@ -360,14 +360,21 @@ HTS_UNUSED static struct tm *convert_time_rfc822(struct tm *result, const char *
HTS_UNUSED static struct tm PT_GetTime(time_t t) {
struct tm tmbuf;
if (!hts_gmtime(t, &tmbuf)) {
#ifdef _WIN32
struct tm *tm = gmtime(&t);
#else
struct tm *tm = gmtime_r(&t, &tmbuf);
#endif
if (tm != NULL)
return *tm;
else {
/* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
day of "00"; the epoch is the conventional "date unknown" */
memset(&tmbuf, 0, sizeof(tmbuf));
tmbuf.tm_year = 70;
tmbuf.tm_mday = 1;
return tmbuf;
}
return tmbuf;
}
HTS_UNUSED static int set_filetime(const char *file, struct tm *tm_time) {
struct utimbuf tim;

View File

@@ -1,8 +0,0 @@
#!/bin/bash
#
# an empty current path underflowed htsAddLink's codebase walk (#730).
set -euo pipefail
out=$(httrack -O /dev/null -#test=addlink)
grep -q "addlink self-test OK" <<<"$out"

View File

@@ -1,9 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# A ready slot still owning a temporary must stay in memory: swapping it out
# clears the entry, which unlinks the re-fetch backup (#771).
out=$(httrack -O /dev/null -#test=backswap)
grep -q "backswap self-test: OK" <<<"$out"

View File

@@ -3,13 +3,25 @@
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_changes_st.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmpdir"' EXIT
trap 'rm -rf "$tmpdir"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmpdir"' EXIT HUP INT QUIT PIPE TERM
# No pipe into grep: SIGPIPE would mask a failing exit status.
expect_ok() {
local label="$1" out
shift
out=$("$@" 2>&1) || {
echo "FAIL: ${label} exited non-zero: ${out}"
exit 1
}
case "$out" in
*"${label}: OK"*) ;;
*)
echo "FAIL: ${out}"
exit 1
;;
esac
}
# --changes bucket accounting and JSON escaping (#714).
expect_ok "changes self-test" httrack -O "${tmpdir}/o1" -#test=changes run

View File

@@ -5,5 +5,4 @@ set -euo pipefail
# webhttrack posts its command line as one string: the argv split must grow past
# 1024 arguments and keep a quote inside a value out of the option parser.
out=$(httrack -O /dev/null -#test=cmdline-split run)
grep -q "cmdline-split self-test OK" <<<"$out"
httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

View File

@@ -11,8 +11,7 @@
set -euo pipefail
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_cmdline.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
echo '<html><body>hello</body></html>' >"$tmp/index.html"

View File

@@ -7,7 +7,6 @@ set -euo pipefail
# cookies *@*.txt) from a long, non-ASCII folder through the UTF-8/long-path
# file wrappers (#133,#630).
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -O /dev/null -#test=cookieimport "$dir")
grep -q "cookieimport:.*OK" <<<"$out"
httrack -O /dev/null -#test=cookieimport "$dir" | grep -q "cookieimport:.*OK"

View File

@@ -6,7 +6,6 @@ set -euo pipefail
# Drives -#test=direnum: enumerate a long+non-ASCII directory through the
# opendir/readdir wrappers, checking each child round-trips as UTF-8 (#133,#630).
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -O /dev/null -#test=direnum "$dir")
grep -q "direnum:.*OK" <<<"$out"
httrack -O /dev/null -#test=direnum "$dir" | grep -q "direnum:.*OK"

View File

@@ -28,8 +28,7 @@ case "$bin" in
esac
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_doitlog.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
site="$tmp/site"
out="$tmp/out"

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# HT_ADD_HTMLESCAPED* must reserve the escaper's worst case (6 for _full).
out=$(httrack -O /dev/null -#test=escape-room run)
grep -q "escape-room self-test OK" <<<"$out"
httrack -O /dev/null -#test=escape-room run | grep -q "escape-room self-test OK"

View File

@@ -8,8 +8,7 @@
set -euo pipefail
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_filelist.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
echo '<html><body>hi</body></html>' >"$tmp/index.html"

View File

@@ -18,7 +18,7 @@ MINGW* | MSYS* | CYGWIN*) exit 77 ;;
esac
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
# A few-hundred-char file path (kept well under the URL length limit) makes the
# {path} field long.
@@ -50,7 +50,7 @@ httrack "file://$deep/index.html" -O "$mir" -%F "$footer" -q -s0 -%v0 \
# The crawled page must exist (proves the URL wasn't rejected for length, so the
# footer path ran). Look under file/, not $mir, to skip the makeindex top index.
test -n "$(find "$mir/file" -name index.html)" || {
find "$mir/file" -name index.html | grep -q . || {
echo "page not mirrored; the oversized-footer path was not exercised" >&2
exit 1
}

View File

@@ -6,7 +6,7 @@
set -euo pipefail
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
rc=0
out=$(httrack -#test=fsize "$dir") || rc=$?

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# get_ftp_line bounds a hostile CRLF-less FTP reply into its 1024-byte buffer.
out=$(httrack -O /dev/null -#test=ftp-line run)
grep -q "ftp-line self-test OK" <<<"$out"
httrack -O /dev/null -#test=ftp-line run | grep -q "ftp-line self-test OK"

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# ftp_split_userpass bounds an over-long user:pass@ from a hostile ftp:// URL.
out=$(httrack -O /dev/null -#test=ftp-userpass run)
grep -q "ftp-userpass self-test OK" <<<"$out"
httrack -O /dev/null -#test=ftp-userpass run | grep -q "ftp-userpass self-test OK"

View File

@@ -1,21 +0,0 @@
#!/bin/bash
#
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
# A non-UTC zone, so a helper that reached for localtime instead of gmtime gives
# a different answer: CI runners are UTC, where the two agree.
TZ=XXX5
export TZ
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_gmtime_st.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmpdir"' EXIT
trap 'rm -rf "$tmpdir"' HUP INT QUIT PIPE TERM
# Concurrent conversions do not tread on each other the way gmtime()'s shared
# static does (#794).
expect_ok "gmtime self-test" httrack -O "${tmpdir}/o1" -#test=gmtime

View File

@@ -10,8 +10,7 @@ echo "$out"
test "$out" == "growsize self-test OK"
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_growsize.XXXXXX")
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
echo '<html><body>hi</body></html>' >"$tmp/index.html"
printf -- '-*/zzmarker*\n' >"$tmp/rules.txt"
@@ -19,7 +18,7 @@ printf -- '-*/zzmarker*\n' >"$tmp/rules.txt"
# the rules file lands in the URL/filter string, echoed back by the banner
run=$(httrack -O "$tmp/out" --quiet -n "-%S" "$tmp/rules.txt" \
"file://$tmp/index.html" 2>&1) || true
grep -q 'zzmarker' <<<"$run" || {
printf '%s\n' "$run" | grep -q 'zzmarker' || {
echo "FAIL: -%S rules file was not loaded"
printf '%s\n' "$run"
exit 1

View File

@@ -6,4 +6,4 @@ set -euo pipefail
# httrack internal hashtable autotest on 100K keys. Assert the success line (on
# stderr) so a misrouted registry entry can't pass on exit code alone.
out=$(httrack -#test=hashtable 100000 2>&1)
grep -q "all hashtable tests were successful!" <<<"$out" || exit 1
printf '%s\n' "$out" | grep -q "all hashtable tests were successful!" || exit 1

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# inplace_escape_*() must match escape_*() on a copy: guards the shared helper.
out=$(httrack -O /dev/null -#test=inplace-escape run)
grep -q "inplace-escape self-test OK" <<<"$out"
httrack -O /dev/null -#test=inplace-escape run | grep -q "inplace-escape self-test OK"

View File

@@ -1,35 +0,0 @@
#!/bin/bash
#
# Runs the lastchar self-test (#770), then keeps the idiom it replaced from
# coming back: x[strlen(x) - 1] indexes one byte before the buffer when x is
# empty, and the guard is never where the index is.
set -eu
: "${top_srcdir:=..}"
out=$(httrack -#test=lastchar) || {
echo "httrack -#test=lastchar exited non-zero: $out" >&2
exit 1
}
# Exact-match the success line so a renamed/removed test can't pass silently.
test "$out" = "lastchar self-test: OK" || {
echo "expected 'lastchar self-test: OK', got: $out" >&2
exit 1
}
# The self-test only covers the helpers, so guard the class at the source: the
# lone survivor is inside a comment block in htsname.c. coucal is vendored, so
# its idioms are not ours to police.
hits=$(command grep -rn 'strlen([^)]*) *- *1\]' "$top_srcdir/src" \
--include='*.c' --include='*.h' --exclude-dir=coucal |
grep -v 'htsname\.c:.*save\[strlen(save)-1\]' |
grep -v 'Replaces s\[strlen(s) - 1\]' |
grep -v 'The x\[strlen(x) - 1\] class' || true)
test -z "$hits" || {
echo "x[strlen(x) - 1] reintroduced; use hts_lastchar/hts_striplastchar/hts_choplastchar:" >&2
echo "$hits" >&2
exit 1
}

View File

@@ -1,8 +0,0 @@
#!/bin/bash
#
# a log callback consumed the va_list the log file's vfprintf() still needed (#801).
set -euo pipefail
out=$(httrack -O /dev/null -#test=logcallback)
grep -q "logcallback self-test OK" <<<"$out"

View File

@@ -6,7 +6,6 @@ set -euo pipefail
# Drives -#test=longpath: a >MAX_PATH round trip exercising hts_pathToUCS2's
# \\?\ prefixing on Windows (#133); a positive control on POSIX.
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -O /dev/null -#test=longpath "$dir")
grep -q "longpath:.*OK" <<<"$out"
httrack -O /dev/null -#test=longpath "$dir" | grep -q "longpath:.*OK"

View File

@@ -6,7 +6,7 @@ set -euo pipefail
# hts_finish_makeindex writes the footer and gates the refresh meta on a single
# first link (guards the macro->function extraction).
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=makeindex "$dir" run |
grep -q "makeindex self-test OK"

View File

@@ -7,7 +7,6 @@ set -euo pipefail
# (>MAX_PATH) and non-ASCII, exercising the mirror I/O wrappers the engine's
# raw file ops now route to on Windows (#133, #630). Positive control on POSIX.
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -O /dev/null -#test=mirrorio "$dir")
grep -q "mirrorio:.*OK" <<<"$out"
httrack -O /dev/null -#test=mirrorio "$dir" | grep -q "mirrorio:.*OK"

View File

@@ -7,8 +7,7 @@
set -euo pipefail
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_parse.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
# a minimal valid 1x1 GIF, reused for every referenced asset
gif() {

View File

@@ -30,8 +30,7 @@ case "$bin" in
esac
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_rcfile.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
# HTS_HTTRACKRC is ".httrackrc" on POSIX but "httrackrc" on Windows: write both,
# each platform reads the one it knows.

View File

@@ -7,7 +7,7 @@
set -eu
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -#test=reconcile "$dir")

View File

@@ -6,5 +6,4 @@ set -euo pipefail
# #159: a redirect to a same-file alias (http<->https, user@host, ..) must be
# followed through, not turned into a self-pointing "moved" stub. The decision
# helper is exercised by the engine self-test.
out=$(httrack -O /dev/null -#test=redirect-samefile run)
grep -q "redirect-samefile self-test OK" <<<"$out"
httrack -O /dev/null -#test=redirect-samefile run | grep -q "redirect-samefile self-test OK"

View File

@@ -1,71 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# Drives -#test=renameover: hts_rename_over() must replace an existing dst, and
# must leave dst alone when the rename failed for a reason moving dst aside
# cannot fix (#779, #790). The selftest prints the regime and the restore
# outcome it took; pin both per leg so none can pass having tested another.
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
case "$(uname -s)" in
MINGW* | MSYS_NT*) want=fallback ;; # native rename() refuses an existing target
*) want=clobber ;;
esac
out=$(httrack -O /dev/null -#test=renameover "$dir")
grep -q "renameover: OK" <<<"$out"
grep -q "renameover: regime $want" <<<"$out"
grep -q "renameover: restore back" <<<"$out"
if [ "$(uname -s)" != "Linux" ]; then
echo "renameover: LD_PRELOAD interposition is Linux-only here, skipping"
exit 0
fi
# A --disable-shared build has nothing to preload; anything else missing is a
# build problem, not a skip, or the interposed legs would pass vacuously.
if [ ! -r "${RENAMEFAIL_LA:-}" ]; then
echo "renameover: ${RENAMEFAIL_LA:-\$RENAMEFAIL_LA} was not built" >&2
exit 1
fi
if grep -q "^dlname=''" "$RENAMEFAIL_LA"; then
echo "renameover: static-only build, skipping the interposed legs"
exit 0
fi
if [ ! -r "${RENAMEFAIL_LIB:-}" ]; then
echo "renameover: ${RENAMEFAIL_LIB:-\$RENAMEFAIL_LIB} was not built" >&2
exit 1
fi
# An LD_PRELOAD library loads ahead of the executable's own libasan, which ASan
# refuses by default. The shim allocates nothing, so the ordering is harmless.
export ASAN_OPTIONS="${ASAN_OPTIONS:+$ASAN_OPTIONS:}verify_asan_link_order=0"
# The aside fallback is dead code on POSIX, so borrow Windows' rename().
out=$(LD_PRELOAD="$RENAMEFAIL_LIB" httrack -O /dev/null \
-#test=renameover "$dir")
grep -q "renameover: OK" <<<"$out"
grep -q "renameover: regime fallback" <<<"$out"
grep -q "renameover: restore back" <<<"$out"
# #790: the move back out of the parked name fails once. Without the retry the
# old copy stays parked and dst is left absent.
out=$(RENAMEFAIL_ASIDE_FAILS=1 LD_PRELOAD="$RENAMEFAIL_LIB" httrack -O /dev/null \
-#test=renameover "$dir")
grep -q "renameover: OK" <<<"$out"
grep -q "renameover: restore back" <<<"$out"
# It keeps failing: the old copy must survive under the parked name, never be
# deleted, and the call must still report failure.
out=$(RENAMEFAIL_ASIDE_FAILS=9 LD_PRELOAD="$RENAMEFAIL_LIB" httrack -O /dev/null \
-#test=renameover "$dir")
grep -q "renameover: OK" <<<"$out"
grep -q "renameover: restore parked" <<<"$out"
# A source another process holds fails with EACCES, which dst had no part in.
out=$(RENAMEFAIL_MODE=locked LD_PRELOAD="$RENAMEFAIL_LIB" httrack -O /dev/null \
-#test=renameover "$dir")
grep -q "renameover: OK" <<<"$out"
grep -q "renameover: regime refused" <<<"$out"

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# robots.txt RFC 9309 Allow/Disallow precedence (#452): longest match wins.
out=$(httrack -O /dev/null -#test=robots run)
grep -q "robots self-test OK" <<<"$out"
httrack -O /dev/null -#test=robots run | grep -q "robots self-test OK"

View File

@@ -12,7 +12,7 @@ httrack_bin=$(cd "$(dirname "$(command -v httrack)")" && pwd)/httrack
# scratch dir: body= and cached= write temp files (st-savename-body.tmp, hts-cache/)
scratch=$(mktemp -d)
trap 'set +e; rm -rf "$scratch"' EXIT
trap 'rm -rf "$scratch"' EXIT
cd "$scratch"
run() {

View File

@@ -7,11 +7,11 @@ set -eu
# Bare -#test lists known tests (printed to stderr).
list=$(httrack -#test 2>&1)
grep -q "filter" <<<"$list" || exit 1
grep -q "cache-writefail" <<<"$list" || exit 1
printf '%s\n' "$list" | grep -q "filter" || exit 1
printf '%s\n' "$list" | grep -q "cache-writefail" || exit 1
# Unknown name: non-zero exit + diagnostic, and no test result line.
rc=0
err=$(httrack -#test=bogus 2>&1) || rc=$?
test "$rc" -ne 0 || exit 1
grep -q "Unknown self-test" <<<"$err" || exit 1
printf '%s\n' "$err" | grep -q "Unknown self-test" || exit 1

View File

@@ -6,5 +6,4 @@ set -euo pipefail
# The SOCKS5 handshake framing and credential split, driven against scripted
# server replies (frame draining, oversize rejects, RFC 1929 fields).
out=$(httrack -O /dev/null '-#test=socks5')
grep -q "socks5 self-test OK" <<<"$out"
httrack -O /dev/null '-#test=socks5' | grep -q "socks5 self-test OK"

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# HTTP status -> reason phrase, including the modern 429/451 (#453).
out=$(httrack -O /dev/null -#test=status run)
grep -q "status self-test OK" <<<"$out"
httrack -O /dev/null -#test=status run | grep -q "status self-test OK"

View File

@@ -5,5 +5,4 @@ set -euo pipefail
# --strip-query: pattern-scoped query-key stripping for dedup. All assertions
# live in the engine self-test (hts_query_strip_keys + fil_normalized_filtered).
out=$(httrack -O /dev/null -#test=stripquery)
grep -q "strip-query self-test OK" <<<"$out"
httrack -O /dev/null -#test=stripquery | grep -q "strip-query self-test OK"

View File

@@ -1,12 +0,0 @@
#!/bin/bash
#
# structcheck(): the path-length guard, and the <name>.txt rename that once
# built its target with an unbounded sprintf (#745).
set -euo pipefail
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=structcheck "$dir" |
grep -q "structcheck self-test OK"

View File

@@ -1,16 +0,0 @@
#!/bin/bash
#
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_threadwait_st.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmpdir"' EXIT
trap 'rm -rf "$tmpdir"' HUP INT QUIT PIPE TERM
# A thread is outstanding from the moment hts_newthread() returns, so a wait
# that follows the spawn joins it, and wait_n(n) still leaves n behind (#747).
expect_ok "threadwait self-test" httrack -O "${tmpdir}/o1" -#test=threadwait

View File

@@ -6,7 +6,7 @@ set -euo pipefail
# hts_buildtopindex takes a system-charset path but verif_backblue below it
# expects utf-8, mangling a non-ASCII project dir on Windows (#216, #217).
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=topindex "$dir" run |
grep -q "topindex self-test OK"

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# Entity/URL unescapers reserve one byte for the trailing NUL (no 1-byte OOB).
out=$(httrack -O /dev/null -#test=unescape-bounds run)
grep -q "unescape-bounds self-test OK" <<<"$out"
httrack -O /dev/null -#test=unescape-bounds run | grep -q "unescape-bounds self-test OK"

View File

@@ -5,5 +5,4 @@ set -euo pipefail
# -%u url-hack split (#271): www / // / query-order dedup toggle independently.
# All assertions live in the engine self-test (hash compare flag resolution).
out=$(httrack -O /dev/null -#test=urlhack run)
grep -q "urlhack self-test OK" <<<"$out"
httrack -O /dev/null -#test=urlhack run | grep -q "urlhack self-test OK"

View File

@@ -4,5 +4,4 @@
set -euo pipefail
# Default User-Agent (#449): honest HTTrack token, no Windows 98 relic.
out=$(httrack -O /dev/null -#test=useragent run)
grep -q "useragent self-test OK" <<<"$out"
httrack -O /dev/null -#test=useragent run | grep -q "useragent self-test OK"

View File

@@ -5,5 +5,4 @@ set -euo pipefail
# SURT canonicalization of the CDXJ sort key (--warc-cdx). Pure string work,
# so it runs under the MSan-instrumented 01_engine glob.
out=$(httrack -O /dev/null -#test=warc-surt)
grep -q "warc-surt: OK" <<<"$out"
httrack -O /dev/null -#test=warc-surt | grep -q "warc-surt: OK"

View File

@@ -5,7 +5,7 @@ set -euo pipefail
# Accept-Encoding (#450): advertise gzip+deflate; decode gzip/zlib/raw-deflate.
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=acceptencoding "$dir" run |
grep -q "acceptencoding self-test OK"

View File

@@ -7,7 +7,7 @@
set -eu
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
# the smashed-header case logs expected "Corrupted cache entry" warnings on
# stdout; the verdict is the last line

View File

@@ -27,7 +27,7 @@ test -e "$fixture/hts-cache/new.zip" || {
}
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
# Read against a private copy so the source tree is never touched (a read
# session does not write, but copying keeps the test hermetic). Create the dir

View File

@@ -6,7 +6,7 @@
set -eu
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
# the refusal errors land on stdout (no log file); pin them and the verdict
out=$(httrack -#test=cache-legacy "$dir" 2>/dev/null)

View File

@@ -1,6 +1,8 @@
#!/bin/bash
#
# Stick to POSIX tool flags: macOS ships BSD grep/sed, not the GNU ones.
# Keep this POSIX-portable: the harness runs it via $(BASH), which is a plain
# POSIX /bin/sh on some platforms (e.g. macOS), so avoid bashisms and GNU-only
# tool flags despite the #!/bin/bash above.
# Cache write-failure policy (-#test=cache-writefail <dir>). #174/#219: disk
# full or a failure streak aborts cleanly; an isolated failure or an oversized
@@ -9,19 +11,19 @@
set -eu
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -#test=cache-writefail "$dir")
# Match the exact success line (error logs also go to stdout); a renamed/removed
# test prints the registry to stderr, which exits non-zero but never prints this.
grep -qx "cache-writefail: OK" <<<"$out" || {
printf '%s\n' "$out" | grep -qx "cache-writefail: OK" || {
echo "expected 'cache-writefail: OK', got: $out" >&2
exit 1
}
# A skipped entry must be warned about with its URL.
grep -q "entry not cached: example.com/" <<<"$out" || {
printf '%s\n' "$out" | grep -q "entry not cached: example.com/" || {
echo "expected a URL-bearing skip warning" >&2
exit 1
}

View File

@@ -18,7 +18,7 @@
set -eu
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
# The working directory is a required argument; without it the test prints a
# usage line to stderr and returns non-zero.

View File

@@ -5,7 +5,7 @@ set -euo pipefail
# brotli/zstd decode, unknown codings, and the decoded-size budget.
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=contentcodings "$dir" run |
grep -q "contentcodings self-test OK"

View File

@@ -1,6 +1,8 @@
#!/bin/bash
#
# Stick to POSIX tool flags: macOS ships BSD grep/sed, not the GNU ones.
# Keep this POSIX-portable: the harness runs it via $(BASH), which is a plain
# POSIX /bin/sh on some platforms (e.g. macOS), so avoid bashisms and GNU-only
# tool flags despite the #!/bin/bash above.
# unzRepair header read must not overflow a signed shift (-#test=zip-repair-shift
# <dir>). A damaged local file header whose CRC high word has bit 15 set made
@@ -9,11 +11,11 @@
set -eu
dir=$(mktemp -d)
trap 'set +e; rm -rf "$dir"' EXIT
trap 'rm -rf "$dir"' EXIT
out=$(httrack -#test=zip-repair-shift "$dir")
grep -qx "zip-repair-shift: OK (recovered 1 entry)" <<<"$out" || {
printf '%s\n' "$out" | grep -qx "zip-repair-shift: OK (recovered 1 entry)" || {
echo "expected 'zip-repair-shift: OK (recovered 1 entry)', got: $out" >&2
exit 1
}

View File

@@ -10,7 +10,7 @@ set -euo pipefail
httrack_bin=$(cd "$(dirname "$(command -v httrack)")" && pwd)/httrack
scratch=$(mktemp -d)
trap 'set +e; rm -rf "$scratch"' EXIT
trap 'rm -rf "$scratch"' EXIT
cd "$scratch"
name() {

View File

@@ -9,7 +9,7 @@ set -euo pipefail
httrack_bin=$(cd "$(dirname "$(command -v httrack)")" && pwd)/httrack
scratch=$(mktemp -d)
trap 'set +e; rm -rf "$scratch"' EXIT
trap 'rm -rf "$scratch"' EXIT
out=$("$httrack_bin" -O /dev/null -#test=warc-cdx "$scratch/")
echo "$out"

View File

@@ -10,14 +10,13 @@ set -euo pipefail
httrack_bin=$(cd "$(dirname "$(command -v httrack)")" && pwd)/httrack
registry=$("$httrack_bin" -#test 2>&1 || true)
if ! grep -q '^ warc-wacz' <<<"$registry"; then
if ! "$httrack_bin" -#test 2>&1 | grep -q '^ warc-wacz'; then
echo "warc-wacz self-test unavailable (build without OpenSSL); skipping"
exit 77
fi
scratch=$(mktemp -d)
trap 'set +e; rm -rf "$scratch"' EXIT
trap 'rm -rf "$scratch"' EXIT
out=$("$httrack_bin" -O /dev/null -#test=warc-wacz "$scratch/")
echo "$out"

View File

@@ -10,7 +10,7 @@ set -euo pipefail
httrack_bin=$(cd "$(dirname "$(command -v httrack)")" && pwd)/httrack
scratch=$(mktemp -d)
trap 'set +e; rm -rf "$scratch"' EXIT
trap 'rm -rf "$scratch"' EXIT
for t in warc warc-trunc warc-ftp warc-rotate warc-verbatim; do
out=$("$httrack_bin" -O /dev/null "-#test=$t" "$scratch/")

View File

@@ -29,7 +29,7 @@ command -v httrack >/dev/null 2>&1 || {
tmp=$(mktemp) || exit 1
committed_clean=$(mktemp) || exit 1
generated_clean=$(mktemp) || exit 1
trap 'set +e; rm -f "$tmp" "$committed_clean" "$generated_clean"' EXIT
trap 'rm -f "$tmp" "$committed_clean" "$generated_clean"' EXIT
README="$top_srcdir/README" bash "$gen" httrack >"$tmp" 2>/dev/null || {
echo "makeman.sh failed" >&2

View File

@@ -14,7 +14,7 @@ set -euo pipefail
site=$(mktemp -d)
out=$(mktemp -d)
trap 'set +e; rm -rf "$site" "$out"' EXIT
trap 'rm -rf "$site" "$out"' EXIT
cat >"$site/index.html" <<EOF
<a href="a.html">a</a> <a href="sub/b.html">b</a>

View File

@@ -1,111 +0,0 @@
#!/bin/bash
#
# The end-of-mirror purge read old.lst with linput(fp, line, 1000), which caps a
# chunk at 999 bytes. filenote() writes "[<save path>]", so a 998-byte save path
# makes a 1000-byte line that comes back as 999 bytes plus a one-byte "]" tail.
# For that tail `line + 1` is empty, and with no -O so is path_html, so the purge
# indexed file[-1] (#770). Needs no -O, hence not local-crawl.sh.
#
# The OOB write lands in stack padding, so only the sanitized CI leg fails on
# unfixed code; the length assertion below is what keeps this from passing
# vacuously if the save path ever stops landing on the chunk boundary.
set -euo pipefail
: "${top_srcdir:=..}"
# shellcheck source=tests/testlib.sh
. "$top_srcdir/tests/testlib.sh"
# python3 is required; mirror check-network.sh's skip-with-77 convention.
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
# Resolve httrack now: the harness prepends a RELATIVE dir to PATH, so the cd
# below would otherwise silently fall through to an installed copy.
httrack_bin=$(command -v httrack)
test -n "$httrack_bin" || {
echo "httrack not on PATH" >&2
exit 1
}
httrack_bin=$(cd "$(dirname "$httrack_bin")" && pwd)/$(basename "$httrack_bin")
work=$(mktemp -d)
trap 'set +e; kill -9 "${spid:-}" 2>/dev/null || true; rm -rf "$work"' EXIT
mkdir -p "$work/root" "$work/proj"
"$python" "$top_srcdir/tests/local-server.py" --root "$work/root" \
>"$work/server.out" 2>"$work/server.err" &
spid=$!
port=
for _ in $(seq 1 100); do
port=$(sed -n 's/^PORT \([0-9]*\)$/\1/p' "$work/server.out" 2>/dev/null || true)
test -n "$port" && break
sleep 0.1
done
test -n "$port" || {
echo "local-server.py did not announce a port" >&2
exit 1
}
# Save path is "127.0.0.1_<port>/" + the URL path; target 998 bytes total.
prefix="127.0.0.1_${port}/"
need=$((998 - ${#prefix}))
# macOS caps PATH_MAX at 1024, so neither the docroot nor the mirror can hold a
# 998-byte path, and httrack's own HTS_MAX_PATH_LEN clamps below it too. The
# trigger is simply not reachable there.
pathmax=$(getconf PATH_MAX / 2>/dev/null || echo 4096)
test "$pathmax" -ge $((need + ${#work} + 128)) || {
echo "PATH_MAX $pathmax too small for a $need-byte path; skipping" >&2
exit 77
}
# Same on Win32, where MAX_PATH clamps the save path; getconf answers for the
# MSYS emulation, not the Win32 API httrack calls, so it cannot see this.
case "$(uname -s)" in
MINGW* | MSYS* | CYGWIN*)
echo "Win32 MAX_PATH cannot hold a $need-byte save path; skipping" >&2
exit 77
;;
esac
urlpath=""
while test $((need - ${#urlpath})) -gt 60; do
urlpath="${urlpath}$(printf 'd%02d' ${#urlpath})$(printf 'a%.0s' $(seq 1 46))/"
done
urlpath="${urlpath}$(printf 'f%.0s' $(seq 1 $((need - ${#urlpath} - 5)))).html"
test ${#urlpath} -eq "$need"
mkdir -p "$work/root/$(dirname "$urlpath")"
printf '<html><body>leaf</body></html>\n' >"$work/root/$urlpath"
printf '<html><body><a href="/%s">x</a></body></html>\n' "$urlpath" \
>"$work/root/index.html"
cd "$work/proj" # no -O: opt->path_html stays empty
"$httrack_bin" "http://127.0.0.1:$port/" -q -%v0 -s0 -r3 >"$work/crawl1.log" 2>&1 || true
lst=$work/proj/hts-cache/new.lst
test -f "$lst" || {
echo "pass 1 produced no new.lst" >&2
exit 1
}
# Arms the test: without a line of exactly 1000 bytes linput() never splits and
# the purge never sees a one-byte tail.
awk 'length == 1000 { found = 1 } END { exit !found }' "$lst" || {
echo "no 1000-byte line in new.lst; the chunk split is not exercised" >&2
awk '{ print length }' "$lst" | sort -n | tail -3 >&2
echo "--- urlpath len ${#urlpath}, prefix $prefix" >&2
tail -20 "$work/crawl1.log" >&2
exit 1
}
# Pass 2 renames new.lst to old.lst and runs it through the purge loop.
"$httrack_bin" "http://127.0.0.1:$port/" -q -%v0 -s0 -r3 >"$work/crawl2.log" 2>&1
test -s "$work/proj/$prefix$urlpath" || {
echo "the mirrored leaf did not survive the purge" >&2
exit 1
}

View File

@@ -1,27 +0,0 @@
#!/bin/bash
#
# A .bak left behind by a killed run must not disable the #77 re-fetch backup
# (#758): a leftover file (slow.bin.bak) has to be clobbered so the -M abort can
# still restore the pass-1 copy, and a leftover the engine cannot clobber
# (fast.bin.bak, planted as a directory) has to be reported instead of silently
# truncating with no safety net. Only the Windows leg arms the first half:
# POSIX rename() clobbers on its own.
set -euo pipefail
: "${top_srcdir:=..}"
bash "$top_srcdir/tests/local-crawl.sh" \
--plant-file bigtrunc/slow.bin.bak \
--plant-dir bigtrunc/fast.bin.bak \
--rerun-args '--update -M400000' \
--log-found 'More than 400000 bytes have been transferred.. giving up' \
--log-found 'replacing leftover backup .*slow\.bin\.bak' \
--log-found 'could not back up .*fast\.bin' \
--log-not-found 'could not back up .*slow\.bin' \
--log-not-found 'could not restore' \
--found bigtrunc/slow.bin \
--found bigtrunc/fast.bin \
--file-min-bytes bigtrunc/slow.bin 655360 \
--file-not-matches bigtrunc/slow.bin 'stale-leftover' \
httrack 'BASEURL/bigtrunc/index.html' -c2

View File

@@ -1,166 +0,0 @@
#!/bin/bash
#
# An FTP re-fetch used to truncate the mirror before the transfer, so a failed
# one destroyed the previous copy (#771). keep.bin is cut short and empty.bin
# gets nothing; both must keep their old bytes. stay.bin takes the same path and
# completes, so a fix that merely stopped writing would fail here.
set -euo pipefail
: "${top_srcdir:=..}"
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
command -v httrack >/dev/null || {
echo "could not find httrack" >&2
exit 1
}
server=$(nativepath "${testdir}/ftp-server.py")
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_ftp.XXXXXX")
serverpid=
cleanup() {
stop_server "$serverpid"
rm -rf "$tmpdir"
}
trap 'set +e; cleanup' EXIT
trap cleanup HUP INT QUIT PIPE TERM
root="${tmpdir}/root"
out="${tmpdir}/crawl"
mode="${tmpdir}/mode"
report="${out}/hts-changes.json"
mkdir -p "$root" "$out"
# The two generations differ in length, or a re-fetch that resumed at EOF
# instead of truncating would leave the same bytes and pass for the wrong reason.
write_bodies() {
local gen="$1" fill="$2" name
for name in keep empty stay; do
"$python" -c 'import sys; sys.stdout.buffer.write(
("%s-FTP-%s " % (sys.argv[1].upper(), sys.argv[2])).encode()
+ sys.argv[2][-1].encode() * int(sys.argv[3]))' "$name" "$gen" "$fill" \
>"${root}/${name}.bin"
done
}
write_bodies V1 4096
: >"$mode"
serverlog="${tmpdir}/server.out"
"$python" "$server" --root "$(nativepath "$root")" \
--mode-file "$(nativepath "$mode")" >"$serverlog" 2>&1 &
serverpid=$!
port=
for _ in $(seq 1 300); do
line=$(grep -m1 '^PORT ' "$serverlog" 2>/dev/null) && port="${line#PORT }" && break
kill -0 "$serverpid" 2>/dev/null || {
echo "ftp server exited early: $(cat "$serverlog")" >&2
exit 1
}
sleep 0.1
done
test -n "$port" || {
echo "could not discover ftp server port: $(cat "$serverlog")" >&2
exit 1
}
host="127.0.0.1_${port}"
urls=()
for name in keep empty stay; do
urls+=("ftp://127.0.0.1:${port}/${name}.bin")
done
# -c1: a parallel FTP crawl loses whole transfers to a separate, older bug in
# the backlog slot swap (#797), which would flake this test on a loaded runner.
common=(--quiet --disable-security-limits --robots=0 --timeout=20 --max-time=120
--retries=1 -c1 --changes)
fail() {
echo "FAIL: $*" >&2
test ! -f "$report" || cat "$report" >&2
exit 1
}
ok() { echo "OK: $*"; }
size_of() { wc -c <"$1" | tr -d '[:space:]'; }
# Files listed under a report bucket, one "file:size" per line.
listed() {
"$python" - "$report" "$1" <<'EOF' | tr -d '\r'
import json
import sys
with open(sys.argv[1], encoding="utf-8") as fp:
report = json.load(fp)
for entry in report[sys.argv[2]]:
print("%s:%s" % (entry["file"], entry.get("size", "none")))
EOF
}
# --- pass 1: a healthy mirror ------------------------------------------------
# Bounded like every local-crawl.sh pass: a wedged crawl must fail the test, not
# hang the job until CI cancels it and discards the log (#796).
run_with_timeout 300 httrack "${urls[@]}" -O "$out" "${common[@]}" \
>"${tmpdir}/log1" 2>&1
for name in keep empty stay; do
test -s "${out}/${host}/${name}.bin" || fail "pass 1 did not mirror ${name}.bin"
done
mkdir "${tmpdir}/snap"
cp "${out}/${host}"/*.bin "${tmpdir}/snap/"
ok "pass 1 mirrored three files"
# --- pass 2: the same three re-fetched, two of them failing -------------------
# norest throughout: an accepted REST sends the client down the append path,
# which is not the one that truncates.
write_bodies V2 6000
cat >"$mode" <<'EOF'
/keep.bin truncate norest
/empty.bin empty norest
/stay.bin norest
EOF
run_with_timeout 300 httrack "${urls[@]}" -O "$out" "${common[@]}" --update \
>"${tmpdir}/log2" 2>&1
# Both failures have to be the transfer dying mid-body: a connect or lookup
# failure never opens the file, and everything below would pass vacuously.
for name in keep empty; do
grep -aqE "Error:.*\"FTP file incomplete\".*${name}\.bin" "${out}/hts-log.txt" ||
fail "${name}.bin did not fail during the transfer"
done
ok "both re-fetches reached the body and failed there"
for name in keep empty; do
cmp -s "${out}/${host}/${name}.bin" "${tmpdir}/snap/${name}.bin" ||
fail "${name}.bin differs from the copy pass 1 left ($(size_of \
"${out}/${host}/${name}.bin") bytes now)"
done
ok "a failed FTP transfer left the previously mirrored bytes alone"
grep -aq "FTP-V2 " "${out}/${host}/stay.bin" ||
fail "stay.bin was not refreshed; the re-fetch writes nothing at all now"
ok "a successful FTP re-fetch still replaces the file"
leftover=$(find "${out}/${host}" -name '*.bak')
test -z "$leftover" || fail "backup left behind: ${leftover}"
ok "no backup temporary survived the pass"
# --purge-old is on by default: a kept file the run never replaced must still be
# in new.lst, or it is deleted as gone (#746).
for name in keep empty stay; do
test -f "${out}/${host}/${name}.bin" || fail "${name}.bin was purged"
done
ok "the kept copies survived the update purge"
# --- the change report is the mirror's own account of the pass ----------------
test -s "$report" || fail "pass 2 wrote no ${report}"
test -z "$(listed gone)" || fail "the report calls something gone: $(listed gone)"
unchanged=$(listed unchanged)
changed=$(listed changed)
for name in keep empty; do
grep -qx "${host}/${name}.bin:$(size_of "${tmpdir}/snap/${name}.bin")" <<<"$unchanged" ||
fail "${name}.bin is not reported unchanged at its previous size"
if grep -q "^${host}/${name}.bin:" <<<"$changed"; then
fail "${name}.bin is reported changed as well"
fi
done
grep -qx "${host}/stay.bin:$(size_of "${out}/${host}/stay.bin")" <<<"$changed" ||
fail "stay.bin is not reported changed"
ok "the change report calls the kept files unchanged and the refreshed one changed"

View File

@@ -1,81 +0,0 @@
#!/bin/bash
#
# Issue #773: a test must fail only when an assertion fails. Under "set -e" a
# failing command in an EXIT trap becomes the script's exit status, so a
# teardown hiccup turned a test whose assertions all passed into a red CI leg.
set -euo pipefail
: "${top_srcdir:=..}"
tmp=$(mktemp -d)
trap 'set +e; rm -rf "$tmp"' EXIT
fail() {
echo "FAIL: $1" >&2
exit 1
}
# run <trap-body> <script-body>: exit status and stderr of a subject script.
run() {
{
echo "set -euo pipefail"
# printf, so the scanner below does not read this as a trap of our own
printf 'trap %s EXIT\n' "$1"
echo "$2"
} >"$tmp/subject"
rc=0
err=$(bash "$tmp/subject" 2>&1 >/dev/null) || rc=$?
}
# 1. the shape in the tree: teardown fails, every assertion passed -> pass
run "'set +e; false'" "true"
test "$rc" -eq 0 || fail "guarded teardown failure exited $rc, want 0"
# 2. teeth for case 1: unguarded, the same teardown is what went red
run "'false'" "true"
test "$rc" -eq 1 || fail "unguarded teardown failure exited $rc, want 1"
# 3. the guard must not swallow a real failure, nor a skip
run "'set +e; false'" "exit 1"
test "$rc" -eq 1 || fail "assertion failure exited $rc, want 1"
run "'set +e; false'" "exit 77"
test "$rc" -eq 77 || fail "skip exited $rc, want 77"
# 4. not failing must not mean going quiet: teardown still reports why
run "'set +e; rm /nonexistent-teardown-probe'" "true"
test "$rc" -eq 0 || fail "guarded teardown failure exited $rc, want 0"
case "$err" in
*nonexistent-teardown-probe*) ;;
*) fail "teardown error was silenced (stderr: '$err')" ;;
esac
echo "OK: teardown cannot decide the verdict, and still reports"
# 5. no test may reintroduce the shape case 2 pins. Two spellings neutralise
# errexit for teardown: "set +e" first, or "|| true" over the whole body.
scanned=0
for f in "$top_srcdir"/tests/*.test "$top_srcdir"/tests/*.sh; do
test -r "$f" || continue
scanned=$((scanned + 1))
done
# an unexpanded glob would report no offender having read nothing
test "$scanned" -ge 50 || fail "scanned $scanned scripts in $top_srcdir/tests, want the whole suite"
bad=$(awk '
FNR == 1 { errexit = 0 }
/^[[:space:]]*set[[:space:]]+(-[a-z]*e|-o[[:space:]]+errexit)/ { errexit = 1 }
!errexit { next }
!/^[[:space:]]*trap[[:space:]]+.*[[:space:]]EXIT([[:space:]]|;|$)/ { next }
{
guarded = /^[[:space:]]*trap[[:space:]]+.set \+e; / || /\|\|[[:space:]]+true.[[:space:]]+EXIT/
if (!guarded)
print FILENAME ":" FNR " unguarded"
# "set +e" on a trap shared with a signal leaves errexit off for the
# rest of the run, so a torn-down test could still report success
else if (/^[[:space:]]*trap[[:space:]]+.set \+e; / && /[[:space:]]EXIT[[:space:]]+[A-Z]/)
print FILENAME ":" FNR " set +e shared with a signal trap"
}
' "$top_srcdir"/tests/*.test "$top_srcdir"/tests/*.sh)
test -z "$bad" || fail "EXIT trap can decide the verdict:
$bad"
echo "OK: every EXIT trap under set -e runs teardown with errexit off"

View File

@@ -1,13 +0,0 @@
#!/bin/bash
#
# A URL past the header-format buffer must still produce a record: the archive
# used to lose it whole, with no error and an exit status of 0 (#785).
set -e
: "${top_srcdir:=..}"
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_longurl.XXXXXX")
trap 'set +e; rm -rf "$tmp"' EXIT
httrack -O "$tmp/out" "-#test=warc-longurl" "$tmp"

View File

@@ -1,141 +0,0 @@
#!/bin/bash
#
# The harness timeout (tests/test-timeout.sh) must name the test it killed, say
# what was still running, and exit 124 -- otherwise a wedge only ever shows up as
# a cancelled CI step, which keeps no log and no artifact.
set -eu
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
driver="${testdir}/test-timeout.sh"
test -r "$driver" || {
echo "no test-timeout.sh next to $0" >&2
exit 1
}
fail() {
echo "FAIL: $*" >&2
test -z "${out:-}" || sed 's/^/ | /' <"$out" >&2
exit 1
}
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_sutmo.XXXXXX") || exit 1
trap 'set +e; rm -rf "$tmp"' EXIT
trap 'rm -rf "$tmp"' HUP INT QUIT PIPE TERM
out="$tmp/out"
# --- a hanging test is killed, named, and reported 124 ----------------------
cat >"$tmp/90_wedged.test" <<'EOF'
echo "marker: the wedged test started"
sleep 300
EOF
start=$SECONDS
rc=0
HTTRACK_TEST_TIMEOUT=5 bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 || rc=$?
elapsed=$((SECONDS - start))
test "$rc" -eq 124 || fail "wedged test reported $rc, want 124"
# Never before the budget, or a slow-but-healthy test would be killed too.
test "$elapsed" -ge 5 || fail "the guard fired early (${elapsed}s of a 5s budget)"
test "$elapsed" -lt 30 || fail "the guard fired late (${elapsed}s)"
grep -q 'marker: the wedged test started' "$out" || fail "the test's own output was lost"
# The header, not a bare name: the process list quotes the test's path too, so a
# wrapper that named nothing would still match that.
grep -q '^===== TIMEOUT: 90_wedged.test exceeded' "$out" ||
fail "the diagnostics do not name the test"
grep -q "own process tree" "$out" || fail "no process list in the diagnostics"
# The budget is read, not hard-coded: well under it, the same shape survives.
printf 'sleep 3\necho "slow but healthy"\n' >"$tmp/92_slow.test"
rc=0
HTTRACK_TEST_TIMEOUT=45 bash "$driver" "$tmp/92_slow.test" >"$out" 2>&1 || rc=$?
test "$rc" -eq 0 || fail "a healthy 3s test under a 45s budget reported $rc"
grep -q 'slow but healthy' "$out" || fail "healthy test output lost"
! grep -q 'TIMEOUT' "$out" || fail "the guard fired on a healthy test"
# The killed tree must really be gone, or the next test inherits its ports.
sleep 1
! grep -q "$tmp/90_wedged.test" <<<"$(ps -A -o args 2>/dev/null)" ||
fail "the wedged test survived the guard"
# --- exit status and output of a healthy test pass straight through ----------
# 77 is automake's SKIP and 99 its hard error, so both have to survive the wrapper.
for want in 0 1 77 99; do
printf 'echo "ran with %s"\nexit %s\n' "$want" "$want" >"$tmp/91_plain.test"
rc=0
HTTRACK_TEST_TIMEOUT=60 bash "$driver" "$tmp/91_plain.test" >"$out" 2>&1 || rc=$?
test "$rc" -eq "$want" || fail "exit $want came back as $rc"
grep -q "ran with $want" "$out" || fail "output lost for exit $want"
! grep -q 'TIMEOUT' "$out" || fail "the guard fired on a test that exited at once"
done
# A budget of 0 disables the guard: the test still runs, and is not killed. Its
# own file, so the assertion cannot ride on whatever the loop above left behind.
printf 'echo "guard off"\nexit 3\n' >"$tmp/93_off.test"
rc=0
HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/93_off.test" >"$out" 2>&1 || rc=$?
test "$rc" -eq 3 || fail "disabled guard changed the exit status ($rc)"
grep -q 'guard off' "$out" || fail "output lost with the guard disabled"
# --- a wedged crawl yields a symbolized engine stack ------------------------
# The whole point of the dump: name the frame the engine is stuck in. Windows has
# neither half (MSYS signals do not reach a native httrack.exe, and that build has
# no backtrace()), so only assert it where the machinery exists.
if is_windows; then
echo "suite timeout OK (no stack capture on Windows)"
exit 0
fi
command -v httrack >/dev/null || {
echo "SKIP: no httrack on PATH for the stack half" >&2
exit 77
}
# /watchdog/stall never answers, so the crawl wedges for real. CRAWL_DEADLINE is
# raised past our budget so the guard under test is what fires, not local-crawl's.
cat >"$tmp/92_stalled.test" <<EOF
CRAWL_DEADLINE=600 bash "${testdir}/local-crawl.sh" --errors 0 \\
httrack 'BASEURL/watchdog/stall'
EOF
rc=0
HTTRACK_TEST_TIMEOUT=20 bash "$driver" "$tmp/92_stalled.test" >"$out" 2>&1 || rc=$?
if grep -qE 'python3 not found|could not discover server port' "$out"; then
echo "SKIP: local crawl prerequisites missing" >&2
exit 77
fi
test "$rc" -eq 124 || fail "stalled crawl reported $rc, want 124"
if grep -q 'No stack trace available on this OS' "$out"; then
echo "suite timeout OK (no backtrace() on this platform)"
exit 0
fi
# Each platform's mechanism has its own marker, and the marker is what proves the
# whole chain connected up rather than the dump merely running.
case "$(uname -s)" in
Linux)
# sig_fatal writes to the engine's own stderr, so this reaching our log also
# proves the crawl-log salvage worked.
grep -q '^Caught signal ' "$out" || fail "no engine backtrace was salvaged"
;;
Darwin)
grep -q '^Call graph:' "$out" || fail "sample(1) produced no call graph"
;;
*)
grep -q 'no stack mechanism known' "$out" ||
fail "unknown platform reported neither a stack nor its absence"
echo "suite timeout OK (no stack mechanism on $(uname -s))"
exit 0
;;
esac
# A stalled crawl is necessarily inside httpmirror, so these frames must appear.
# Matching "httrack" alone would pass on the process list above.
grep -qE '\b(back_wait|httpmirror|hts_main_internal)\b' "$out" ||
fail "the stack names no engine frame"
echo "suite timeout OK (named the test, dumped a stack, exited 124)"

View File

@@ -1,98 +0,0 @@
#!/bin/bash
#
# A completed FTP transfer used to end as a 0-byte file when its ready backlog
# slot was swapped to the on-disk table (#797): the FTP worker writes url_sav
# itself and leaves no in-memory body, and the swap round-trip zeroed the slot's
# size, so the link writer then created an empty file over the bytes.
set -euo pipefail
: "${top_srcdir:=..}"
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
command -v httrack >/dev/null || {
echo "could not find httrack" >&2
exit 1
}
server=$(nativepath "${testdir}/ftp-server.py")
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_ftp.XXXXXX")
serverpid=
cleanup() {
stop_server "$serverpid"
rm -rf "$tmpdir"
}
trap 'set +e; cleanup' EXIT
trap cleanup HUP INT QUIT PIPE TERM
root="${tmpdir}/root"
out="${tmpdir}/crawl"
mkdir -p "$root" "$out"
fail() {
echo "FAIL: $*" >&2
exit 1
}
ok() { echo "OK: $*"; }
size_of() { wc -c <"$1" | tr -d '[:space:]'; }
# Distinct bodies and lengths, so a mirror that mixed two of them up cannot
# compare equal.
names=""
for i in 1 2 3 4 5 6 7 8 9 10 11 12; do
names="${names} f${i}"
"$python" -c 'import sys; sys.stdout.buffer.write(
("BODY-%s " % sys.argv[1]).encode() + sys.argv[1].encode() * (400 * int(sys.argv[1])))' \
"$i" >"${root}/f${i}.bin"
done
serverlog="${tmpdir}/server.out"
"$python" "$server" --root "$(nativepath "$root")" >"$serverlog" 2>&1 &
serverpid=$!
port=
for _ in $(seq 1 300); do
line=$(grep -m1 '^PORT ' "$serverlog" 2>/dev/null) && port="${line#PORT }" && break
kill -0 "$serverpid" 2>/dev/null || {
echo "ftp server exited early: $(cat "$serverlog")" >&2
exit 1
}
sleep 0.1
done
test -n "$port" || {
echo "could not discover ftp server port: $(cat "$serverlog")" >&2
exit 1
}
host="127.0.0.1_${port}"
urls=()
for name in $names; do
urls+=("ftp://127.0.0.1:${port}/${name}.bin")
done
# -c8, never -c1: a slot is only swapped out while other transfers keep the
# parser from picking it up. -Z because the swap only logs at debug level, and
# the run below proves nothing unless it happened. Bounded like every crawl
# pass: a wedge must fail the test, not hang the job until CI discards the log.
run_with_timeout 300 httrack "${urls[@]}" -O "$out" --quiet -Z -c8 \
--disable-security-limits --robots=0 --timeout=20 --max-time=120 --retries=1 \
>"${tmpdir}/log" 2>&1
swapped=$(grep -ac 'slots ready moved to background' "${out}/hts-log.txt" || true)
test "${swapped:-0}" -gt 0 ||
fail "no ready slot was swapped to disk; this run never reached the bug"
ok "the crawl swapped ready slots to the on-disk table ${swapped} times"
for name in $names; do
test -f "${out}/${host}/${name}.bin" || fail "${name}.bin was not mirrored"
cmp -s "${out}/${host}/${name}.bin" "${root}/${name}.bin" ||
fail "${name}.bin has $(size_of "${out}/${host}/${name}.bin") of the" \
"$(size_of "${root}/${name}.bin") bytes served"
done
ok "every FTP transfer landed on disk intact"
errors=$(grep -a 'Error:' "${out}/hts-log.txt" || true)
test -z "$errors" || fail "the crawl reported errors: ${errors}"
ok "no transfer was reported failed"

View File

@@ -1,125 +0,0 @@
#!/bin/bash
#
# FTP resumed with REST whenever the mirrored file merely existed, so an
# --update over a complete copy spliced the old body into the new one and the
# result matched the remote length (#798). Pass 1 is cut short to leave a real
# partial, pass 2 must still resume it, pass 3 must not.
set -euo pipefail
: "${top_srcdir:=..}"
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
command -v httrack >/dev/null || {
echo "could not find httrack" >&2
exit 1
}
server=$(nativepath "${testdir}/ftp-server.py")
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_ftp.XXXXXX")
serverpid=
cleanup() {
stop_server "$serverpid"
rm -rf "$tmpdir"
}
trap 'set +e; cleanup' EXIT
trap cleanup HUP INT QUIT PIPE TERM
root="${tmpdir}/root"
out="${tmpdir}/crawl"
mode="${tmpdir}/mode"
cmds="${tmpdir}/cmds"
mkdir -p "$root" "$out"
fail() {
echo "FAIL: $*" >&2
exit 1
}
ok() { echo "OK: $*"; }
size_of() { wc -c <"$1" | tr -d '[:space:]'; }
# hts-log.txt is rewritten per pass, so this only ever sees the pass just run.
no_errors() {
local errors
errors=$(grep -a 'Error:' "${out}/hts-log.txt" || true)
test -z "$errors" || fail "$1 reported errors: ${errors}"
}
# Commands the server saw since the last pass; each pass starts by emptying it.
sent() { cat "$cmds"; }
# The two generations differ in length and in their first bytes, so a splice
# shows up whichever end of the file it lands in.
write_body() {
"$python" -c 'import sys; sys.stdout.buffer.write(
("%s-BODY " % sys.argv[1]).encode() + sys.argv[1][:1].encode() * int(sys.argv[2]))' \
"$1" "$2" >"${root}/a.bin"
}
write_body OLD 8000
echo '/a.bin truncate' >"$mode"
serverlog="${tmpdir}/server.out"
"$python" "$server" --root "$(nativepath "$root")" \
--mode-file "$(nativepath "$mode")" --log "$(nativepath "$cmds")" \
>"$serverlog" 2>&1 &
serverpid=$!
port=
for _ in $(seq 1 300); do
line=$(grep -m1 '^PORT ' "$serverlog" 2>/dev/null) && port="${line#PORT }" && break
kill -0 "$serverpid" 2>/dev/null || {
echo "ftp server exited early: $(cat "$serverlog")" >&2
exit 1
}
sleep 0.1
done
test -n "$port" || {
echo "could not discover ftp server port: $(cat "$serverlog")" >&2
exit 1
}
host="127.0.0.1_${port}"
url="ftp://127.0.0.1:${port}/a.bin"
mirror="${out}/${host}/a.bin"
# Bounded like every crawl pass: a wedge must fail the test, not hang the job.
crawl() { run_with_timeout 300 httrack "$url" -O "$out" --quiet \
--disable-security-limits --robots=0 --timeout=20 --max-time=120 --retries=1 \
-c1 "$@"; }
# --- pass 1: the transfer dies mid-body and leaves a partial -----------------
crawl >"${tmpdir}/log1" 2>&1 || true
test -f "$mirror" || fail "pass 1 mirrored nothing to resume"
partial=$(size_of "$mirror")
whole=$(size_of "${root}/a.bin")
if test "$partial" -le 0 || test "$partial" -ge "$whole"; then
fail "pass 1 left ${partial} bytes, wanted a partial of ${whole}"
fi
ok "pass 1 left a ${partial}-byte partial copy"
# --- pass 2: the partial is resumed ------------------------------------------
: >"$mode"
: >"$cmds"
crawl >"${tmpdir}/log2" 2>&1
case "$(sent)" in
*"REST ${partial}"*) ok "pass 2 resumed at the partial's ${partial} bytes" ;;
*) fail "pass 2 sent no REST ${partial}; commands were: $(sent)" ;;
esac
cmp -s "$mirror" "${root}/a.bin" ||
fail "the resumed copy is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
# A resume used to count only the appended tail against the remote size, so it
# was reported "FTP file incomplete" even when it produced the whole file.
no_errors "pass 2"
ok "the resumed transfer produced the served body and no error"
# --- pass 3: --update over that complete copy --------------------------------
write_body NEW 12000
: >"$cmds"
crawl --update >"${tmpdir}/log3" 2>&1
case "$(sent)" in
*REST*) fail "the update resumed a complete mirror: $(sent)" ;;
*RETR*) ok "the update re-fetched without REST" ;;
*) fail "pass 3 never reached RETR; commands were: $(sent)" ;;
esac
cmp -s "$mirror" "${root}/a.bin" ||
fail "the updated copy is $(size_of "$mirror") bytes, served $(size_of "${root}/a.bin")"
no_errors "pass 3"
ok "the update replaced the mirror with the new body"

View File

@@ -1,108 +0,0 @@
#!/bin/bash
#
# Issue #782: a 304 during --update overwrote the whole response struct with the
# cache entry, dropping the 8 KB header buffer it owned. Only a leak checker
# sees it and the sanitized CI leg runs with detect_leaks off, so this one turns
# it back on around a two-pass mini304 crawl; the fresh pass is the control.
set -eu
: "${top_srcdir:=..}"
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
server=$(nativepath "${testdir}/local-server.py")
root=$(nativepath "${testdir}/server-root")
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
probe=$(ASAN_OPTIONS=help=1 httrack -O /dev/null --version 2>&1 || true)
case "$probe" in
*AddressSanitizer*) ;;
*)
echo "not a sanitized build, skipping" >&2
exit 77
;;
esac
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_304leak.XXXXXX") || exit 1
serverpid=
cleanup() {
stop_server "$serverpid"
rm -rf "$tmpdir"
}
trap 'set +e; cleanup' EXIT
trap cleanup HUP INT QUIT PIPE TERM
serverlog="${tmpdir}/server.log"
: >"$serverlog"
"$python" "$server" --root "$root" >"$serverlog" 2>&1 &
serverpid=$!
port=
for _ in $(seq 1 50); do
line=$(head -n1 "$serverlog" 2>/dev/null)
if test "${line%% *}" == "PORT"; then
port="${line#PORT }"
break
fi
kill -0 "$serverpid" 2>/dev/null || {
echo "server exited early: $(cat "$serverlog")"
exit 1
}
sleep 0.1
done
test -n "$port" || {
echo "could not discover server port"
exit 1
}
out="${tmpdir}/crawl"
mkdir "$out"
url="http://127.0.0.1:${port}/mini304/index.html"
# Last value wins in the sanitizer flag parser, so this overrides the job's
# settings; the report file, not the exit status, is the verdict.
asan_base="${ASAN_OPTIONS:+$ASAN_OPTIONS:}detect_leaks=1:abort_on_error=0:exitcode=0"
# run_pass <label> <extra httrack args...>
run_pass() {
local label="$1"
shift
rm -f "${tmpdir}/asan".*
printf '[%s] ..\t' "$label"
ASAN_OPTIONS="${asan_base}:log_path=${tmpdir}/asan" \
httrack -O "$out" --quiet --disable-security-limits --robots=0 \
--timeout=30 --retries=1 -c1 "$@" "$url" >"${tmpdir}/log_${label}" 2>&1 || true
report=$(cat "${tmpdir}/asan".* 2>/dev/null || true)
test -z "$report" || {
echo "FAIL: sanitizer report on the ${label} pass"
echo "$report"
exit 1
}
echo "OK (clean)"
}
run_pass fresh
run_pass update --update
# only --warc-file stashes the two header blocks, so the archive limb of the fix
# runs on this pass alone
run_pass warc --update --warc-file leak
# Teeth: the update passes must have gone through the 304 path, or a run that
# re-downloaded everything would report clean having tested nothing.
printf '[update used the cache] ..\t'
if grep -aqE "no files updated" "${out}/hts-log.txt"; then
echo "OK"
else
echo "FAIL: the update pass did not report unchanged files"
exit 1
fi
# ... and the WARC pass must have written an archive, or it stashed nothing.
printf '[warc pass archived] ..\t'
if [ -n "$(find "$out" -name 'leak*.warc*' -size +0c 2>/dev/null)" ]; then
echo "OK"
else
echo "FAIL: the --warc-file pass produced no archive"
exit 1
fi

View File

@@ -31,7 +31,7 @@ cleanup() {
done
rm -rf "$tmpdir"
}
trap 'set +e; cleanup' EXIT
trap cleanup EXIT
# self-signed cert for the local TLS origin (httrack does not verify certs)
openssl req -x509 -newkey rsa:2048 -keyout "$tmpdir/key.pem" \

View File

@@ -40,8 +40,9 @@ serverpid=
cleanup() {
stop_server "$serverpid"
rm -rf "$tmpdir"
return 0
}
trap 'set +e; cleanup' EXIT
trap cleanup EXIT
# bind the live server to 127.0.0.1 only, so 127.0.0.2 refuses the connect
"$python" "$server" --root "$root" --bind 127.0.0.1 >"$tmpdir/srv.out" 2>"$tmpdir/srv.err" &

View File

@@ -13,10 +13,7 @@ bash "$top_srcdir/tests/local-crawl.sh" --errors 1 \
--not-found 'errpage/missing.html' \
httrack 'BASEURL/errpage/index.html' '-o0'
# Control -o1 (default): the 404 body is the server's verbatim, no HTTrack marker (#787); good.html proves the marker check can fire.
# Control -o1 (default): the 404 error page is written.
bash "$top_srcdir/tests/local-crawl.sh" --errors 1 \
--found 'errpage/missing.html' \
--file-matches 'errpage/missing.html' '404 error body' \
--file-not-matches 'errpage/missing.html' 'HTTrack' \
--file-matches 'errpage/good.html' 'HTTrack' \
httrack 'BASEURL/errpage/index.html' '-o1'

Some files were not shown because too many files have changed in this diff Show More