mirror of
https://github.com/xroche/httrack.git
synced 2026-07-27 02:52:42 +03:00
Compare commits
25 Commits
feat/crash
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9484b32ecd | ||
|
|
1d647bfecd | ||
|
|
b2dc012263 | ||
|
|
aa1131982b | ||
|
|
f72e7ebe96 | ||
|
|
3b53bf85f2 | ||
|
|
ce7dcfa9de | ||
|
|
069573edc3 | ||
|
|
a75f437df9 | ||
|
|
783f6ee1f5 | ||
|
|
913caf68be | ||
|
|
59660102d6 | ||
|
|
e96399910b | ||
|
|
52d0ab2356 | ||
|
|
ca533abefd | ||
|
|
dc6819b7b7 | ||
|
|
3247e5b183 | ||
|
|
bef7120423 | ||
|
|
7cf54af485 | ||
|
|
c32a47110e | ||
|
|
1027a9f392 | ||
|
|
d0a1573a04 | ||
|
|
4362ac34cb | ||
|
|
6579436607 | ||
|
|
cb98b1b197 |
5
.github/workflows/windows-build.yml
vendored
5
.github/workflows/windows-build.yml
vendored
@@ -226,8 +226,9 @@ jobs:
|
||||
# Every gate here exits 77, so an all-skipped suite would report green having
|
||||
# tested nothing: pin the skips, and floor the passes in case the glob empties.
|
||||
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
|
||||
# webdav-mime needs a reapable background listener, which MSYS cannot give it.
|
||||
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
|
||||
# webdav-mime needs a reapable background listener, which MSYS cannot give it;
|
||||
# badmtime needs a filesystem that stores an mtime past gmtime's range.
|
||||
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test"
|
||||
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
|
||||
[ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
|
||||
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }
|
||||
|
||||
27
AGENTS.md
27
AGENTS.md
@@ -26,6 +26,12 @@ the operational checklist: toolchain, invariants, and how to ship a change.
|
||||
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
|
||||
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
|
||||
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
|
||||
- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
|
||||
pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
|
||||
and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
|
||||
as "marker absent". Capture the reply, assert the status line it must carry
|
||||
(an empty, truncated or redirected one is marker-free too), then match with a
|
||||
here-string.
|
||||
|
||||
## Hard invariants
|
||||
- **Generated autotools files are NOT in git.** `configure`, every
|
||||
@@ -48,6 +54,16 @@ the operational checklist: toolchain, invariants, and how to ship a change.
|
||||
- Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
|
||||
`untrusted < limit - controlled` — never `controlled + untrusted < limit`,
|
||||
which can wrap and pass.
|
||||
- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
|
||||
(`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
|
||||
our own data, wrong for anything read back from a cache, a header or the
|
||||
wire, where it trades a memory smash for a crash on malformed input. Clip
|
||||
with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
|
||||
- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
|
||||
a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
|
||||
into the same buffer never appears on it. Before scoping a hardening pass off
|
||||
compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
|
||||
`\bstrcpy\s*\(`, `\bstrcat\s*\(`).
|
||||
|
||||
## C conventions
|
||||
- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
|
||||
@@ -84,6 +100,17 @@ Before pushing, and when reviewing others, don't skim for bugs:
|
||||
layout/ABI, cache/wire format, or a security path? A static or unit check
|
||||
isn't enough; exercise the wrong behavior at runtime. Claude Code:
|
||||
`/review-recipe`.
|
||||
- **Poison a canary, never compare it against zero.** Checking that a
|
||||
neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
|
||||
terminator writes — the exact bug the canary is there for. Fill it with a
|
||||
non-zero byte, and prove it by killing both the stray-`'X'` and the
|
||||
stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
|
||||
lands inside the same struct.
|
||||
- **Overshoot every destination, not one.** A bounds test that oversizes a
|
||||
single field cannot tell a per-field bound from a one-size-fits-all one, nor
|
||||
from a fix that bounds that field and leaves its neighbours raw. Exercise
|
||||
each destination the path touches, spanning at least two capacities, and
|
||||
check what the code actually emits before writing the expected values.
|
||||
|
||||
## Commits
|
||||
- **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:
|
||||
|
||||
@@ -97,9 +97,8 @@ ${do:end-if}
|
||||
</pre>
|
||||
|
||||
${LANG_G8} :
|
||||
${do:output-mode:html-urlescaped}
|
||||
<a href="file://${path}/${projname}/" target="_new">
|
||||
${do:output-mode:}
|
||||
${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
|
||||
<a href="/website/index.html" target="_new">
|
||||
${path}/${projname}
|
||||
</a></li>
|
||||
<ul>
|
||||
|
||||
@@ -98,6 +98,9 @@ ${do:end-if}
|
||||
<input type="hidden" name="redirect" value="">
|
||||
<input type="hidden" name="closeme" value="">
|
||||
|
||||
<!-- clear if not checked -->
|
||||
<input type="hidden" name="ftpprox" value="">
|
||||
|
||||
${LANG_PROXYTYPE}
|
||||
<select name="proxytype"
|
||||
title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info(' '); return true"
|
||||
|
||||
@@ -98,6 +98,13 @@ ${do:end-if}
|
||||
<input type="hidden" name="redirect" value="">
|
||||
<input type="hidden" name="closeme" value="">
|
||||
|
||||
<!-- clear if not checked -->
|
||||
<input type="hidden" name="errpage" value="">
|
||||
<input type="hidden" name="external" value="">
|
||||
<input type="hidden" name="hidepwd" value="">
|
||||
<input type="hidden" name="hidequery" value="">
|
||||
<input type="hidden" name="nopurge" value="">
|
||||
|
||||
${LANG_I33}
|
||||
<br>
|
||||
<select name="build"
|
||||
|
||||
@@ -103,7 +103,7 @@ ${LANG_Q3}
|
||||
<tr><td>
|
||||
<table width="100%">
|
||||
<tr><td align="left">
|
||||
<input type="submit" value="${LANG_OK]"
|
||||
<input type="submit" value="${LANG_OK}"
|
||||
${do:output-mode:html-urlescaped}
|
||||
onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
|
||||
${do:output-mode:}
|
||||
|
||||
@@ -98,6 +98,9 @@ ${do:end-if}
|
||||
<input type="hidden" name="redirect" value="">
|
||||
<input type="hidden" name="closeme" value="">
|
||||
|
||||
<!-- clear if not checked -->
|
||||
<input type="hidden" name="windebug" value="">
|
||||
|
||||
${LANG_I40c}
|
||||
<br>
|
||||
|
||||
|
||||
@@ -98,6 +98,11 @@ ${do:end-if}
|
||||
<input type="hidden" name="redirect" value="">
|
||||
<input type="hidden" name="closeme" value="">
|
||||
|
||||
<!-- clear if not checked -->
|
||||
<input type="hidden" name="ka" value="">
|
||||
<input type="hidden" name="remt" value="">
|
||||
<input type="hidden" name="rems" value="">
|
||||
|
||||
<table border="0" width="100%" cellspacing="0">
|
||||
|
||||
<tr><td>
|
||||
|
||||
@@ -98,6 +98,17 @@ ${do:end-if}
|
||||
<input type="hidden" name="redirect" value="">
|
||||
<input type="hidden" name="closeme" value="">
|
||||
|
||||
<!-- clear if not checked -->
|
||||
<input type="hidden" name="cookies" value="">
|
||||
<input type="hidden" name="parsejava" value="">
|
||||
<input type="hidden" name="updhack" value="">
|
||||
<input type="hidden" name="urlhack" value="">
|
||||
<input type="hidden" name="keepwww" value="">
|
||||
<input type="hidden" name="keepslashes" value="">
|
||||
<input type="hidden" name="keepqueryorder" value="">
|
||||
<input type="hidden" name="toler" value="">
|
||||
<input type="hidden" name="http10" value="">
|
||||
|
||||
<input type="checkbox" name="cookies" ${checked:cookies}
|
||||
title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info(' '); return true"
|
||||
> ${LANG_I58}
|
||||
|
||||
@@ -98,6 +98,13 @@ ${do:end-if}
|
||||
<input type="hidden" name="redirect" value="">
|
||||
<input type="hidden" name="closeme" value="">
|
||||
|
||||
<!-- clear if not checked -->
|
||||
<input type="hidden" name="warc" value="">
|
||||
<input type="hidden" name="norecatch" value="">
|
||||
<input type="hidden" name="logf" value="">
|
||||
<input type="hidden" name="index" value="">
|
||||
<input type="hidden" name="index2" value="">
|
||||
|
||||
<input type="checkbox" name="cache2" ${checked:cache2}
|
||||
title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info(' '); return true"
|
||||
> ${LANG_I61}
|
||||
|
||||
@@ -114,16 +114,16 @@ ${do:end-if}
|
||||
|
||||
${/* Real commands and ini file generated below */}
|
||||
|
||||
<!-- engine commandline -->
|
||||
<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
|
||||
${do:output-mode:html}
|
||||
<textarea name="command" cols="50" rows="4" style="visibility:hidden">
|
||||
httrack \
|
||||
--quiet \
|
||||
--build-top-index \
|
||||
${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
|
||||
${urls}
|
||||
${test:filelist:-%L "}${filelist}${test:filelist:"}
|
||||
--path "${html:path}/${html:projname}"
|
||||
${unquoted:urls}
|
||||
${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
|
||||
--path "${arg:path}/${arg:projname}"
|
||||
\
|
||||
${test:parseall:--near}
|
||||
${test:link:--test}
|
||||
@@ -131,7 +131,7 @@ httrack \
|
||||
${test:htmlfirst::--priority=7}
|
||||
\
|
||||
${do:if-not-empty:BuildString}
|
||||
--structure "${BuildString}"
|
||||
--structure "${arg:BuildString}"
|
||||
${do:end-if}
|
||||
${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
|
||||
\
|
||||
@@ -150,51 +150,52 @@ ${do:end-if}
|
||||
${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
|
||||
${test:windebug:--debug-headers}
|
||||
\
|
||||
${test:connexion:--sockets=}${connexion}
|
||||
${test:connexion:--sockets=}${unquoted:connexion}
|
||||
${test:ka:--keep-alive}
|
||||
${test:timeout:--timeout=}${timeout}
|
||||
${test:timeout:--timeout=}${unquoted:timeout}
|
||||
${test:remt:--host-control=1}
|
||||
${test:retry:--retries=}${retry}
|
||||
${test:rate:--min-rate=}${rate}
|
||||
${test:retry:--retries=}${unquoted:retry}
|
||||
${test:rate:--min-rate=}${unquoted:rate}
|
||||
${test:rems:--host-control=2}
|
||||
\
|
||||
${test:depth:--depth=}${depth}
|
||||
${test:depth2:--ext-depth=}${depth2}
|
||||
${test:maxhtml:--max-files=,}${maxhtml}
|
||||
${test:othermax:--max-files=}${othermax}
|
||||
${test:sizemax:--max-files=}${sizemax}
|
||||
${test:pausebytes:--max-pause=}${pausebytes}
|
||||
${test:maxtime:--max-time=}${maxtime}
|
||||
${test:maxrate:--max-rate=}${maxrate}
|
||||
${test:maxconn:--connection-per-second=}${maxconn}
|
||||
${test:maxlinks:--advanced-maxlinks=}${maxlinks}
|
||||
${test:depth:--depth=}${unquoted:depth}
|
||||
${test:depth2:--ext-depth=}${unquoted:depth2}
|
||||
${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
|
||||
${test:othermax:--max-files=}${unquoted:othermax}
|
||||
${test:maxhtml:--max-files=,}${unquoted:maxhtml}
|
||||
${test:sizemax:--max-size=}${unquoted:sizemax}
|
||||
${test:pausebytes:--max-pause=}${unquoted:pausebytes}
|
||||
${test:maxtime:--max-time=}${unquoted:maxtime}
|
||||
${test:maxrate:--max-rate=}${unquoted:maxrate}
|
||||
${test:maxconn:--connection-per-second=}${unquoted:maxconn}
|
||||
${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
|
||||
\
|
||||
--user-agent "${html:user}"
|
||||
--footer "${html:footer}"
|
||||
--user-agent "${arg:user}"
|
||||
--footer "${arg:footer}"
|
||||
\
|
||||
${url2}
|
||||
${unquoted:url2}
|
||||
\
|
||||
${test:cookies:--cookies=0:}
|
||||
${test:parsejava:--parse-java=0:}
|
||||
${ztest:cookies:--cookies=0:}
|
||||
${ztest:parsejava:--parse-java=0:}
|
||||
${test:updhack:--updatehack}
|
||||
${test:urlhack:--urlhack=0:--urlhack}
|
||||
${test:keepwww:--keep-www-prefix}
|
||||
${test:keepslashes:--keep-double-slashes}
|
||||
${test:keepqueryorder:--keep-query-order}
|
||||
${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
|
||||
${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
|
||||
${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
|
||||
${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
|
||||
${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
|
||||
${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
|
||||
${test:toler:--tolerant}
|
||||
${test:http10:--http-10}
|
||||
${test:cache2:--store-all-in-cache}
|
||||
${test:warc:--warc}
|
||||
${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
|
||||
${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
|
||||
${test:norecatch:--do-not-recatch}
|
||||
${test:logf:--single-log}
|
||||
${test:logtype:::--extra-log:--debug-log}
|
||||
${test:index:--index=0:}
|
||||
${test:index2:--search-index=0:--search-index}
|
||||
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
|
||||
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
|
||||
${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
|
||||
</textarea>
|
||||
|
||||
@@ -213,7 +214,7 @@ ParseAll=${ztest:parseall:0:1}
|
||||
HTMLFirst=${ztest:htmlfirst:0:1}
|
||||
Cache=${ztest:cache:0:1}
|
||||
NoRecatch=${ztest:norecatch:0:1}
|
||||
Dos=${dos
|
||||
Dos=${dos}
|
||||
Index=${ztest:index:0:1}
|
||||
WordIndex=${ztest:index2:0:1}
|
||||
Log=${ztest:logf:0:1:2}
|
||||
|
||||
@@ -71,7 +71,9 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
|
||||
for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
|
||||
if (iisBogus[i][j] == '\0'
|
||||
&& (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
|
||||
strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
|
||||
/* j bytes matched, so j fit: copying j cannot overrun whatever the
|
||||
table holds, and the tail must survive untouched */
|
||||
memcpy(a, iisBogusReplace[i], (size_t) j);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -48,7 +48,8 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
|
||||
|
||||
lib_LTLIBRARIES = libhttrack.la
|
||||
|
||||
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
|
||||
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
|
||||
htscmdline.c htscmdline.h \
|
||||
htsurlport.c htsurlport.h
|
||||
proxytrack_SOURCES = proxy/main.c \
|
||||
proxy/proxytrack.c proxy/store.c \
|
||||
@@ -62,7 +63,7 @@ whttrackrun_SCRIPTS = webhttrack
|
||||
libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
|
||||
htscache_selftest.c htsdns_selftest.c htsselftest.c \
|
||||
htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
|
||||
htshelp.c htslib.c htsurlport.c htscoremain.c \
|
||||
htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
|
||||
htsname.c htsrobots.c htstools.c htswizard.c \
|
||||
htsalias.c htsthread.c htsindex.c htsbauth.c \
|
||||
htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
|
||||
@@ -71,7 +72,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
|
||||
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
|
||||
hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
|
||||
htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h \
|
||||
htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
|
||||
htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
|
||||
htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
|
||||
htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
|
||||
htsmodules.h htsname.h htsnet.h htssniff.h \
|
||||
@@ -86,7 +87,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
|
||||
libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
|
||||
libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
|
||||
|
||||
EXTRA_DIST = httrack.h webhttrack \
|
||||
EXTRA_DIST = httrack.h htsstats.h webhttrack \
|
||||
version.rc \
|
||||
libhttrack.rc \
|
||||
httrack.rc \
|
||||
|
||||
@@ -125,6 +125,22 @@ void back_free(struct_back ** sback) {
|
||||
above a normal handshake. The last candidate still gets the full timeout. */
|
||||
#define HTS_CONNECT_FALLBACK_TIMEOUT 10
|
||||
|
||||
void back_read_ftp_result(FILE *fp, htsblk *r) {
|
||||
size_t j = 0;
|
||||
|
||||
if (fscanf(fp, "%d ", &r->statuscode) != 1)
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
// an external helper writes this file: stop at capacity, not at EOF
|
||||
while (j + 1 < sizeof(r->msg)) {
|
||||
const int c = fgetc(fp);
|
||||
|
||||
if (c == EOF)
|
||||
break;
|
||||
r->msg[j++] = (char) c;
|
||||
}
|
||||
r->msg[j] = '\0';
|
||||
}
|
||||
|
||||
int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
|
||||
int timeout) {
|
||||
int deadline;
|
||||
@@ -542,8 +558,14 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
|
||||
// do not use tempnam() but a regular filename
|
||||
back->tmpfile_buffer[0] = '\0';
|
||||
if (back->url_sav[0] != '\0') {
|
||||
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
|
||||
back->url_sav, ext);
|
||||
/* same capacity as url_sav, so truncation drops the extension and aliases
|
||||
the temp name onto the live file that back_finalize_backup() UNLINKs */
|
||||
if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
|
||||
hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
|
||||
back->url_sav);
|
||||
back->tmpfile_buffer[0] = '\0';
|
||||
return -1;
|
||||
}
|
||||
back->tmpfile = back->tmpfile_buffer;
|
||||
if (structcheck(back->tmpfile) != 0) {
|
||||
hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
|
||||
@@ -551,8 +573,15 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
|
||||
return -1;
|
||||
}
|
||||
} else {
|
||||
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
|
||||
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
|
||||
/* truncation here would collide distinct tmpnameid's onto one name */
|
||||
if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
|
||||
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
|
||||
ext)) {
|
||||
hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
|
||||
StringBuff(opt->path_html_utf8));
|
||||
back->tmpfile_buffer[0] = '\0';
|
||||
return -1;
|
||||
}
|
||||
back->tmpfile = back->tmpfile_buffer;
|
||||
}
|
||||
/* OK */
|
||||
@@ -2947,7 +2976,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
back[i].r.msg[0] = '\0';
|
||||
strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
|
||||
if (!strnotempty(back[i].r.msg)) {
|
||||
sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
|
||||
htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
|
||||
}
|
||||
deletehttp(&back[i].r);
|
||||
back[i].r.soc = INVALID_SOCKET;
|
||||
@@ -3020,16 +3049,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
|
||||
"rb");
|
||||
if (fp) {
|
||||
int j = 0;
|
||||
|
||||
fscanf(fp, "%d ", &(back[i].r.statuscode));
|
||||
while(!feof(fp)) {
|
||||
int c = fgetc(fp);
|
||||
|
||||
if (c != EOF)
|
||||
back[i].r.msg[j++] = c;
|
||||
}
|
||||
back[i].r.msg[j++] = '\0';
|
||||
back_read_ftp_result(fp, &back[i].r);
|
||||
fclose(fp);
|
||||
UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
|
||||
strcpybuff(fconcat
|
||||
@@ -3330,10 +3350,11 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
deleteaddr(&back[i].r);
|
||||
if (back[i].r.size < back[i].r.totalsize)
|
||||
back[i].r.statuscode = STATUSCODE_CONNERROR; // recatch
|
||||
sprintf(back[i].r.msg,
|
||||
"Incorrect length (" LLintP " Bytes, " LLintP
|
||||
" expected)", (LLint) back[i].r.size,
|
||||
(LLint) back[i].r.totalsize);
|
||||
htsblk_failf(&back[i].r,
|
||||
"Incorrect length (" LLintP " Bytes, " LLintP
|
||||
" expected)",
|
||||
(LLint) back[i].r.size,
|
||||
(LLint) back[i].r.totalsize);
|
||||
} else {
|
||||
// Un warning suffira..
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
|
||||
@@ -74,6 +74,10 @@ void back_free(struct_back ** sback);
|
||||
// backing
|
||||
#define BACK_ADD_TEST "(dummy)"
|
||||
#define BACK_ADD_TEST2 "(dummy2)"
|
||||
/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
|
||||
clipping the message to r->msg. */
|
||||
void back_read_ftp_result(FILE *fp, htsblk *r);
|
||||
|
||||
int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
|
||||
const char *sav);
|
||||
int back_available(const struct_back * sback);
|
||||
|
||||
@@ -142,10 +142,12 @@ struct cache_back_zip_entry {
|
||||
int compressionMethod;
|
||||
};
|
||||
|
||||
/* A corrupt cache can carry a field wider than ours; clipping it keeps the
|
||||
entry, where aborting would take the crawl down. */
|
||||
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
|
||||
do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
strlcpybuff(refvalue, value, refvalue_size); \
|
||||
(void) strclipbuff(refvalue, refvalue_size, value); \
|
||||
line[0] = '\0'; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
@@ -1195,6 +1195,12 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
|
||||
|
||||
/* --- read-side corruption injection --------------------------------------- */
|
||||
|
||||
/* 100 'A's: a placeholder header line long enough to be overwritten by a
|
||||
forged, over-long X-StatusMessage of the same byte length. */
|
||||
#define CORRUPT_LONG_ETAG \
|
||||
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" \
|
||||
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
|
||||
|
||||
/* canary read back intact after each corruption; victim gets the byte surgery
|
||||
*/
|
||||
#define CORRUPT_ADR "corrupt.example.com"
|
||||
@@ -1219,6 +1225,24 @@ static void corrupt_build(httrackp *opt) {
|
||||
selftest_close(&cache);
|
||||
}
|
||||
|
||||
/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
|
||||
static void corrupt_build_longetag(httrackp *opt) {
|
||||
cache_back cache;
|
||||
|
||||
memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
|
||||
memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
|
||||
remove(reconcile_st_path(opt, "hts-cache/new.zip"));
|
||||
remove(reconcile_st_path(opt, "hts-cache/old.zip"));
|
||||
selftest_open_for_write(&cache, opt);
|
||||
store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
|
||||
"OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
|
||||
strlen(corrupt_body_a));
|
||||
store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
|
||||
"OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
|
||||
corrupt_body_b, strlen(corrupt_body_b));
|
||||
selftest_close(&cache);
|
||||
}
|
||||
|
||||
/* Like corrupt_build, but the victim carries a 20-char Etag whose header line
|
||||
is later overwritten with a forged oversized X-Size (same byte length). */
|
||||
static void corrupt_build_etag(httrackp *opt) {
|
||||
@@ -1403,6 +1427,48 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
|
||||
return fail;
|
||||
}
|
||||
|
||||
/* An over-long field from a foreign cache must clip, not abort: the entry
|
||||
still reads, clipped to capacity, and the canary survives. */
|
||||
static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
|
||||
size_t wantlastmod, const char *what) {
|
||||
cache_back cache;
|
||||
htsblk v, c;
|
||||
char BIGSTK lv[HTS_URLMAXSIZE * 2];
|
||||
char BIGSTK lc[HTS_URLMAXSIZE * 2];
|
||||
int fail = 0;
|
||||
|
||||
selftest_open_for_read(&cache, opt);
|
||||
lv[0] = lc[0] = '\0';
|
||||
v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
|
||||
if (v.statuscode != 200) {
|
||||
fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
|
||||
v.statuscode);
|
||||
fail++;
|
||||
}
|
||||
if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
|
||||
fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
|
||||
(unsigned) strlen(v.msg), (unsigned) wantmsg);
|
||||
fail++;
|
||||
}
|
||||
if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
|
||||
fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
|
||||
what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
|
||||
fail++;
|
||||
}
|
||||
c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
|
||||
if (c.statuscode != 200) {
|
||||
fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
|
||||
c.statuscode);
|
||||
fail++;
|
||||
}
|
||||
if (v.adr != NULL)
|
||||
freet(v.adr);
|
||||
if (c.adr != NULL)
|
||||
freet(c.adr);
|
||||
selftest_close(&cache);
|
||||
return fail;
|
||||
}
|
||||
|
||||
/* One zip corruption case: build, patch, then check victim+canary in-session.
|
||||
*/
|
||||
static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
|
||||
@@ -1439,6 +1505,31 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
|
||||
failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
|
||||
"garbled deflate stream");
|
||||
|
||||
/* A corrupt cache can hold a field wider than ours. Clipping keeps the
|
||||
entry; aborting would take the crawl down. Overwrite the placeholder Etag
|
||||
line in place, same byte length, so the zip offsets stay intact. */
|
||||
corrupt_build_longetag(opt);
|
||||
corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
|
||||
"X-StatusMessage: " /* 17 + 89 = 106 */
|
||||
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
|
||||
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
|
||||
1, 1);
|
||||
failures +=
|
||||
corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
|
||||
(size_t) -1, "over-long X-StatusMessage");
|
||||
|
||||
/* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
|
||||
cannot satisfy both. */
|
||||
corrupt_build_longetag(opt);
|
||||
corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
|
||||
"Last-Modified: " /* 15 + 91 = 106 */
|
||||
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
|
||||
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
|
||||
1, 1);
|
||||
failures += corrupt_expect_victim_clipped(
|
||||
opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
|
||||
"over-long Last-Modified");
|
||||
|
||||
/* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
|
||||
truncates negative in the (int) cast the malloc uses: a wraparound alloc.
|
||||
cache_add asserts size fits an int, so such a value only reaches the reader
|
||||
|
||||
95
src/htscmdline.c
Normal file
95
src/htscmdline.c
Normal file
@@ -0,0 +1,95 @@
|
||||
/* ------------------------------------------------------------ */
|
||||
/*
|
||||
HTTrack Website Copier, Offline Browser for Windows and Unix
|
||||
Copyright (C) 1998 Xavier Roche and other contributors
|
||||
|
||||
SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License
|
||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
Ethical use: we kindly ask that you NOT use this software to harvest email
|
||||
addresses or to collect any other private information about people. Doing so
|
||||
would dishonor our work and waste the many hours we have spent on it.
|
||||
|
||||
Please visit our Website: http://www.httrack.com
|
||||
*/
|
||||
|
||||
/* ------------------------------------------------------------ */
|
||||
/* File: command line splitter, shared by the engine and */
|
||||
/* htsserver */
|
||||
/* Author: Xavier Roche */
|
||||
/* ------------------------------------------------------------ */
|
||||
|
||||
#include "htscmdline.h"
|
||||
|
||||
#include "htssafe.h"
|
||||
|
||||
#include <limits.h>
|
||||
#include <stdint.h>
|
||||
|
||||
char **hts_split_cmdline(char *cmd, int *nargs) {
|
||||
size_t nsep = 0;
|
||||
size_t capacity;
|
||||
size_t r;
|
||||
size_t w;
|
||||
int argc = 0;
|
||||
hts_boolean quoted = HTS_FALSE;
|
||||
char **argv;
|
||||
|
||||
*nargs = 0;
|
||||
|
||||
/* fold the other separators, so counting them sizes the vector exactly */
|
||||
for (r = 0; cmd[r] != '\0'; r++) {
|
||||
if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
|
||||
cmd[r] = ' ';
|
||||
}
|
||||
if (cmd[r] == ' ') {
|
||||
nsep++;
|
||||
}
|
||||
}
|
||||
|
||||
/* at most one argument per separator, plus the leading one and the NULL */
|
||||
if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
|
||||
return NULL;
|
||||
}
|
||||
capacity = nsep + 2;
|
||||
argv = (char **) malloct(capacity * sizeof(char *));
|
||||
if (argv == NULL) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
argv[argc++] = cmd;
|
||||
for (r = 0, w = 0; cmd[r] != '\0';) {
|
||||
if (quoted && cmd[r] == '\\' &&
|
||||
(cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
|
||||
r++;
|
||||
cmd[w++] = cmd[r++];
|
||||
} else if (cmd[r] == '\"') {
|
||||
quoted = !quoted;
|
||||
cmd[w++] = cmd[r++];
|
||||
} else if (cmd[r] == ' ' && !quoted) {
|
||||
cmd[w++] = '\0';
|
||||
assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
|
||||
argv[argc++] = cmd + w;
|
||||
r++;
|
||||
} else {
|
||||
cmd[w++] = cmd[r++];
|
||||
}
|
||||
}
|
||||
cmd[w] = '\0';
|
||||
argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
|
||||
|
||||
*nargs = argc;
|
||||
return argv;
|
||||
}
|
||||
45
src/htscmdline.h
Normal file
45
src/htscmdline.h
Normal file
@@ -0,0 +1,45 @@
|
||||
/* ------------------------------------------------------------ */
|
||||
/*
|
||||
HTTrack Website Copier, Offline Browser for Windows and Unix
|
||||
Copyright (C) 1998 Xavier Roche and other contributors
|
||||
|
||||
SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License
|
||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
Ethical use: we kindly ask that you NOT use this software to harvest email
|
||||
addresses or to collect any other private information about people. Doing so
|
||||
would dishonor our work and waste the many hours we have spent on it.
|
||||
|
||||
Please visit our Website: http://www.httrack.com
|
||||
*/
|
||||
|
||||
/* ------------------------------------------------------------ */
|
||||
/* File: command line splitter, shared by the engine and */
|
||||
/* htsserver */
|
||||
/* Author: Xavier Roche */
|
||||
/* ------------------------------------------------------------ */
|
||||
|
||||
#ifndef HTSCMDLINE_DEFH
|
||||
#define HTSCMDLINE_DEFH
|
||||
|
||||
#include "htsglobal.h"
|
||||
|
||||
/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
|
||||
argv[0] being the program name and quotes left for the engine to strip.
|
||||
Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
|
||||
entries), or NULL when it cannot be sized or allocated. */
|
||||
char **hts_split_cmdline(char *cmd, int *nargs);
|
||||
|
||||
#endif
|
||||
@@ -447,13 +447,22 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
|
||||
const char *fil) {
|
||||
if (!*makeindex_done) {
|
||||
if (*makeindex_fp) {
|
||||
char BIGSTK tempo[1024];
|
||||
/* sized off link_escaped below: at the old flat 1024 a long first link
|
||||
produced a redirect to a clipped URL */
|
||||
char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
|
||||
if (makeindex_links == 1) {
|
||||
char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
|
||||
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
|
||||
snprintf(tempo, sizeof(tempo),
|
||||
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
|
||||
link_escaped);
|
||||
/* no redirect beats one pointing at a clipped URL */
|
||||
if (!sprintfbuff(
|
||||
tempo,
|
||||
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
|
||||
link_escaped)) {
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
"index redirect omitted: first link too long (%s)",
|
||||
makeindex_firstlink);
|
||||
tempo[0] = '\0';
|
||||
}
|
||||
} else
|
||||
tempo[0] = '\0';
|
||||
hts_template_format(*makeindex_fp, template_footer,
|
||||
|
||||
46
src/htsftp.c
46
src/htsftp.c
@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
// folding a nonsense port into 1..65535 fetches one the link never named;
|
||||
// an empty "host:" just means the default (#614)
|
||||
if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
|
||||
htsblk_failf(&back->r, "Invalid port: %s", a + 1);
|
||||
back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
|
||||
_HALT_FTP return 0;
|
||||
}
|
||||
@@ -262,8 +262,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
// récupérer adresse résolue
|
||||
strcpybuff(back->info, "host name");
|
||||
if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg),
|
||||
"Unable to get server's address: %s", error);
|
||||
htsblk_failf(&back->r, "Unable to get server's address: %s", error);
|
||||
back->r.statuscode = STATUSCODE_NON_FATAL;
|
||||
_HALT_FTP return 0;
|
||||
}
|
||||
@@ -332,18 +331,15 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
}
|
||||
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
|
||||
@@ -410,8 +406,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
}
|
||||
// -- fin analyse de l'adresse IP et du port --
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
} // sinon on est prêts
|
||||
} else {
|
||||
@@ -442,13 +437,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
}
|
||||
}
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
} // sinon on est prêts
|
||||
}
|
||||
@@ -554,8 +547,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
deletesoc(soc_dat);
|
||||
soc_dat = INVALID_SOCKET;
|
||||
//
|
||||
snprintf(back->r.msg, sizeof(back->r.msg),
|
||||
"RETR command error: %s", linejmp(line));
|
||||
htsblk_failf(&back->r, "RETR command error: %s",
|
||||
linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
} // sinon on est prêts
|
||||
} else {
|
||||
@@ -573,13 +566,12 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
} // sinon on est prêts
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg),
|
||||
"Unable to resolve IP %s: %s", adr_ip, error);
|
||||
htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
|
||||
error);
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
} // sinon on est prêts
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
} // sinon on est prêts
|
||||
#else
|
||||
@@ -603,13 +595,11 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg),
|
||||
"RETR command error: %s", linejmp(line));
|
||||
htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
#ifdef _WIN32
|
||||
@@ -652,8 +642,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
len = 0; // fin
|
||||
break;
|
||||
case 0:
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
|
||||
timeout);
|
||||
htsblk_failf(&back->r, "Time out (%d)", timeout);
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
len = 0; // fin
|
||||
break;
|
||||
@@ -716,8 +705,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
|
||||
strcpybuff(back->r.msg, "OK");
|
||||
back->r.statuscode = HTTP_OK;
|
||||
} else {
|
||||
snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
|
||||
linejmp(line));
|
||||
htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
|
||||
back->r.statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
|
||||
@@ -72,7 +72,8 @@ Please visit our Website: http://www.httrack.com
|
||||
HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
|
||||
static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
|
||||
compiler type-checks the format string at argument index fmt against the
|
||||
varargs starting at arg. */
|
||||
varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
|
||||
error signal, so dropping it is a bug; a (void) cast does not silence it. */
|
||||
#ifndef HTS_UNUSED
|
||||
#ifdef __GNUC__
|
||||
#define HTS_UNUSED __attribute__((unused))
|
||||
@@ -80,10 +81,13 @@ Please visit our Website: http://www.httrack.com
|
||||
#define HTS_STATIC static __attribute__((unused))
|
||||
|
||||
#define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
|
||||
|
||||
#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
|
||||
#else
|
||||
#define HTS_UNUSED
|
||||
#define HTS_STATIC static
|
||||
#define HTS_PRINTF_FUN(fmt, arg)
|
||||
#define HTS_CHECK_RESULT
|
||||
#endif
|
||||
#endif
|
||||
|
||||
|
||||
@@ -124,7 +124,9 @@ typedef struct help_wizard_buffers {
|
||||
char stropt[2048]; // options
|
||||
char stropt2[2048]; // options longues
|
||||
char strwild[2048]; // wildcards
|
||||
char cmd[4096];
|
||||
/* holds all four of the above plus separators: at 4096 a long answer set
|
||||
clipped the filters off the command line */
|
||||
char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
|
||||
char str[256];
|
||||
char *argv[256];
|
||||
} help_wizard_buffers;
|
||||
@@ -308,7 +310,14 @@ void help_wizard(httrackp * opt) {
|
||||
printf("\n");
|
||||
if (strlen(stropt) == 1)
|
||||
stropt[0] = '\0'; // aucune
|
||||
snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
|
||||
/* the tail is the filter list, and cmd is split into the argv handed to
|
||||
hts_main() below: a clipped line would silently widen the crawl */
|
||||
if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
|
||||
printf("* command line too long (%d bytes max)\n",
|
||||
(int) sizeof(cmd) - 1);
|
||||
freet(buffers);
|
||||
return;
|
||||
}
|
||||
printf("---> Wizard command line: httrack %s\n\n", cmd);
|
||||
printf("Ready to launch the mirror? (Y/n) :");
|
||||
fflush(stdout);
|
||||
@@ -424,7 +433,8 @@ void help_catchurl(const char *dest_path) {
|
||||
}
|
||||
// former URL!
|
||||
{
|
||||
char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
|
||||
/* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
|
||||
char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
|
||||
|
||||
inplace_escape_check_url(dest, sizeof(dest));
|
||||
snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);
|
||||
|
||||
20
src/htslib.c
20
src/htslib.c
@@ -708,11 +708,11 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
|
||||
#ifdef _WIN32
|
||||
int last_errno = WSAGetLastError();
|
||||
|
||||
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
|
||||
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
|
||||
#else
|
||||
int last_errno = errno;
|
||||
|
||||
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
|
||||
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
|
||||
#endif
|
||||
}
|
||||
}
|
||||
@@ -2240,13 +2240,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
|
||||
#ifdef _WIN32
|
||||
int last_errno = WSAGetLastError();
|
||||
|
||||
sprintf(retour->msg, "Unable to create a socket: %s",
|
||||
strerror(last_errno));
|
||||
htsblk_failf(retour, "Unable to create a socket: %s",
|
||||
strerror(last_errno));
|
||||
#else
|
||||
int last_errno = errno;
|
||||
|
||||
sprintf(retour->msg, "Unable to create a socket: %s",
|
||||
strerror(last_errno));
|
||||
htsblk_failf(retour, "Unable to create a socket: %s",
|
||||
strerror(last_errno));
|
||||
#endif
|
||||
}
|
||||
return INVALID_SOCKET; // erreur création socket impossible
|
||||
@@ -2312,13 +2312,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
|
||||
#ifdef _WIN32
|
||||
const int last_errno = WSAGetLastError();
|
||||
|
||||
sprintf(retour->msg, "Unable to connect to the server: %s",
|
||||
strerror(last_errno));
|
||||
htsblk_failf(retour, "Unable to connect to the server: %s",
|
||||
strerror(last_errno));
|
||||
#else
|
||||
const int last_errno = errno;
|
||||
|
||||
sprintf(retour->msg, "Unable to connect to the server: %s",
|
||||
strerror(last_errno));
|
||||
htsblk_failf(retour, "Unable to connect to the server: %s",
|
||||
strerror(last_errno));
|
||||
#endif
|
||||
}
|
||||
/* Close the socket and notify the error!!! */
|
||||
|
||||
@@ -199,6 +199,11 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
|
||||
etc.). */
|
||||
T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
|
||||
int waitconnect, int addr_index, int *addr_count);
|
||||
/* Clips the formatted failure reason into r->msg, which also round-trips
|
||||
through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
|
||||
#define htsblk_failf(R, ...) \
|
||||
slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
|
||||
|
||||
HTS_INLINE void deletehttp(htsblk * r);
|
||||
HTS_INLINE int deleteaddr(htsblk * r);
|
||||
HTS_INLINE void deletesoc(T_SOC soc);
|
||||
|
||||
@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
|
||||
if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
|
||||
code = 0;
|
||||
if (code < 200 || code >= 300) {
|
||||
snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
|
||||
strnotempty(line) ? line : "(no status)");
|
||||
htsblk_failf(retour, "proxy CONNECT refused: %s",
|
||||
strnotempty(line) ? line : "(no status)");
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ Please visit our Website: http://www.httrack.com
|
||||
#ifndef HTSSAFE_DEFH
|
||||
#define HTSSAFE_DEFH
|
||||
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
@@ -459,6 +460,85 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
|
||||
return b->buf;
|
||||
}
|
||||
|
||||
/**
|
||||
* Copy src into dest (capacity size, NUL included), truncating to fit and
|
||||
* always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
|
||||
* value read back from a cache, a header or the wire, where refusing the whole
|
||||
* record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
|
||||
* callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
|
||||
*/
|
||||
static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
|
||||
const char *src) {
|
||||
size_t len, copy;
|
||||
|
||||
assertf(dest != NULL && src != NULL && size != 0);
|
||||
len = strlen(src);
|
||||
copy = len < size ? len : size - 1;
|
||||
memcpy(dest, src, copy);
|
||||
dest[copy] = '\0';
|
||||
return copy == len ? HTS_TRUE : HTS_FALSE;
|
||||
}
|
||||
|
||||
/**
|
||||
* Callers that deliberately ignore truncation use this instead of
|
||||
* slprintfbuff(), so it is not HTS_CHECK_RESULT.
|
||||
*/
|
||||
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
|
||||
vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
|
||||
int ret;
|
||||
|
||||
assertf(dest != NULL && size != 0);
|
||||
/* a vsnprintf failing outright may write nothing at all, leaving whatever
|
||||
the caller had on the stack for it to publish */
|
||||
dest[0] = '\0';
|
||||
ret = vsnprintf(dest, size, fmt, args);
|
||||
/* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
|
||||
dest[size - 1] = '\0';
|
||||
return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
|
||||
}
|
||||
|
||||
/**
|
||||
* Formatted print into dest (capacity size, NUL included), truncating to fit
|
||||
* and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
|
||||
* result is the only truncation signal, so it must be acted on. Unlike
|
||||
* strcpybuff() it never aborts, so it suits text built from remote input.
|
||||
*/
|
||||
static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
|
||||
slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
|
||||
va_list args;
|
||||
hts_boolean ret;
|
||||
|
||||
va_start(args, fmt);
|
||||
ret = vslprintfbuff(dest, size, fmt, args);
|
||||
va_end(args);
|
||||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* slprintfbuff() for diagnostics quoting remote or client text, which are
|
||||
* meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
|
||||
* cast on slprintfbuff() is no substitute, GCC warns through it.
|
||||
*/
|
||||
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
|
||||
char *dest, size_t size, const char *fmt, ...) {
|
||||
va_list args;
|
||||
|
||||
va_start(args, fmt);
|
||||
(void) vslprintfbuff(dest, size, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
/**
|
||||
* slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
|
||||
* On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
|
||||
*/
|
||||
#if (defined(__GNUC__) && !defined(__cplusplus))
|
||||
#define sprintfbuff(ARR, ...) \
|
||||
slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
|
||||
#else
|
||||
#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
|
||||
#endif
|
||||
|
||||
/* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
|
||||
added bounds checking. freet() also NULLs the freed pointer and tolerates
|
||||
NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the
|
||||
|
||||
@@ -43,6 +43,7 @@ Please visit our Website: http://www.httrack.com
|
||||
|
||||
#include "htsglobal.h"
|
||||
#include "htscore.h"
|
||||
#include "htsback.h"
|
||||
#include "htsdefines.h"
|
||||
#include "htslib.h"
|
||||
#include "htsalias.h"
|
||||
@@ -51,6 +52,7 @@ Please visit our Website: http://www.httrack.com
|
||||
#include "htscache_selftest.h"
|
||||
#include "htsdns_selftest.h"
|
||||
#include "htscharset.h"
|
||||
#include "htscmdline.h"
|
||||
#include "htsencoding.h"
|
||||
#include "htsftp.h"
|
||||
#include "htsmd5.h"
|
||||
@@ -455,6 +457,12 @@ static void basic_selftests(void) {
|
||||
// link one level up -> a "../" prefix
|
||||
assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
|
||||
assertf(strcmp(s, "../a.html") == 0);
|
||||
// an empty current path: the trim used to walk off the front of it, which
|
||||
// "?x" reaches too because the query pre-pass hands on the part before it
|
||||
assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
|
||||
assertf(strcmp(s, "dir/page.html") == 0);
|
||||
assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
|
||||
assertf(strcmp(s, "dir/page.html") == 0);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -568,6 +576,214 @@ static int string_safety_selftests(void) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* sprintfbuff: truncate-and-report. Must never abort (its callers format
|
||||
remote banners) nor write past the array, which the canary catches. */
|
||||
{
|
||||
struct {
|
||||
char dst[8];
|
||||
char canary[8];
|
||||
} s;
|
||||
|
||||
const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
|
||||
|
||||
/* repoison before every call, or an implementation that measures first and
|
||||
writes nothing "passes" the truncating cases on the previous content */
|
||||
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
|
||||
|
||||
memset(&s, '#', sizeof(s));
|
||||
if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
|
||||
return 1;
|
||||
|
||||
/* exact fit: 7 characters plus the NUL */
|
||||
POISON_DST();
|
||||
if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
|
||||
return 1;
|
||||
|
||||
/* one over, then far over: truncated to the prefix, terminated, reported */
|
||||
POISON_DST();
|
||||
if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
|
||||
return 1;
|
||||
POISON_DST();
|
||||
if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
|
||||
return 1;
|
||||
|
||||
/* explicit-capacity form, down to the degenerate size 1 */
|
||||
{
|
||||
char *const p = s.dst;
|
||||
|
||||
POISON_DST();
|
||||
if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
|
||||
return 1;
|
||||
POISON_DST();
|
||||
if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
|
||||
return 1;
|
||||
}
|
||||
#undef POISON_DST
|
||||
|
||||
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* strclipbuff: truncate-and-report, never abort. Same canary shape; the
|
||||
destination is poisoned before every call so a case cannot pass on the
|
||||
previous one's bytes. */
|
||||
{
|
||||
struct {
|
||||
char dst[8];
|
||||
char canary[8];
|
||||
} s;
|
||||
|
||||
memset(&s, '#', sizeof(s));
|
||||
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
|
||||
|
||||
/* well under capacity: no padding, nothing eaten off the end */
|
||||
POISON_DST();
|
||||
if (!strclipbuff(s.dst, sizeof(s.dst), "abc") || strcmp(s.dst, "abc") != 0)
|
||||
return 1;
|
||||
|
||||
/* exact fit: capacity - 1 characters plus the NUL */
|
||||
POISON_DST();
|
||||
if (!strclipbuff(s.dst, sizeof(s.dst), "1234567") ||
|
||||
strcmp(s.dst, "1234567") != 0)
|
||||
return 1;
|
||||
|
||||
/* one over, then far over: clipped, terminated, reported. The expected
|
||||
bytes differ from the case above, so a write-nothing implementation
|
||||
cannot pass on the leftovers. */
|
||||
POISON_DST();
|
||||
if (strclipbuff(s.dst, sizeof(s.dst), "abcdefgh") ||
|
||||
strcmp(s.dst, "abcdefg") != 0)
|
||||
return 1;
|
||||
POISON_DST();
|
||||
if (strclipbuff(s.dst, sizeof(s.dst), "0123456789abcdef") ||
|
||||
strcmp(s.dst, "0123456") != 0)
|
||||
return 1;
|
||||
|
||||
/* degenerate capacity 1: only the NUL fits. Capacity 2 pins the boundary
|
||||
between that and the sizes above: one character plus the NUL. */
|
||||
POISON_DST();
|
||||
if (strclipbuff(s.dst, 1, "x") || s.dst[0] != '\0')
|
||||
return 1;
|
||||
POISON_DST();
|
||||
if (strclipbuff(s.dst, 2, "yz") || strcmp(s.dst, "y") != 0)
|
||||
return 1;
|
||||
|
||||
/* the empty string fits any non-zero capacity */
|
||||
POISON_DST();
|
||||
if (!strclipbuff(s.dst, sizeof(s.dst), "") || s.dst[0] != '\0')
|
||||
return 1;
|
||||
|
||||
/* a byte over 0x7f must not end the copy early */
|
||||
POISON_DST();
|
||||
if (!strclipbuff(s.dst, sizeof(s.dst), "\xff\xfe") ||
|
||||
strcmp(s.dst, "\xff\xfe") != 0)
|
||||
return 1;
|
||||
#undef POISON_DST
|
||||
|
||||
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
|
||||
touches nothing else in the block */
|
||||
{
|
||||
htsblk r;
|
||||
char expect[sizeof(r.msg)];
|
||||
char big[4 * sizeof(r.msg)];
|
||||
|
||||
/* contenttype abuts msg, so a one-past-the-end store lands in it rather
|
||||
than in padding. Poison it: a stray NUL is invisible against zeroes,
|
||||
and a stray NUL is exactly what an off-by-one terminator writes. */
|
||||
#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
|
||||
|
||||
memset(&r, 0, sizeof(r));
|
||||
memset(r.contenttype, '#', sizeof(r.contenttype));
|
||||
r.statuscode = 1234;
|
||||
|
||||
memset(r.msg, '#', sizeof(r.msg));
|
||||
htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
|
||||
if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
|
||||
return 1;
|
||||
|
||||
/* exact fit: capacity - 1 characters plus the NUL */
|
||||
memset(expect, 'y', sizeof(expect) - 1);
|
||||
expect[sizeof(expect) - 1] = '\0';
|
||||
memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
|
||||
memset(r.msg, '#', sizeof(r.msg));
|
||||
htsblk_failf(&r, "%s", expect);
|
||||
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
|
||||
return 1;
|
||||
|
||||
/* far over: the expected bytes differ from the cases above, so writing
|
||||
nothing cannot pass on the leftovers */
|
||||
memset(big, 'z', sizeof(big) - 1);
|
||||
big[sizeof(big) - 1] = '\0';
|
||||
memset(expect, 'z', sizeof(expect) - 1);
|
||||
expect[sizeof(expect) - 1] = '\0';
|
||||
memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
|
||||
|
||||
memset(r.msg, '#', sizeof(r.msg));
|
||||
htsblk_failf(&r, "Bad user name: %s", big);
|
||||
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
|
||||
return 1;
|
||||
#undef NEIGHBOURS_INTACT
|
||||
}
|
||||
|
||||
/* back_read_ftp_result: the helper's result file is external input, so an
|
||||
over-long message must stop at msg[]'s capacity */
|
||||
{
|
||||
htsblk r;
|
||||
size_t k;
|
||||
|
||||
/* poisoned so a short message cannot pass on leftovers, and so a stray
|
||||
NUL past msg[] is visible in the neighbour */
|
||||
#define FTP_RESULT_CASE(BODY) \
|
||||
do { \
|
||||
FILE *fp_ = tmpfile(); \
|
||||
\
|
||||
if (fp_ == NULL) \
|
||||
return 1; \
|
||||
BODY; \
|
||||
rewind(fp_); \
|
||||
memset(&r, 0, sizeof(r)); \
|
||||
memset(r.msg, '#', sizeof(r.msg)); \
|
||||
memset(r.contenttype, '#', sizeof(r.contenttype)); \
|
||||
back_read_ftp_result(fp_, &r); \
|
||||
fclose(fp_); \
|
||||
if (r.contenttype[0] != '#') \
|
||||
return 1; \
|
||||
} while (0)
|
||||
|
||||
/* over capacity: clipped to 79 payload bytes plus the NUL */
|
||||
FTP_RESULT_CASE({
|
||||
fprintf(fp_, "226 ");
|
||||
for (k = 0; k < 4 * sizeof(r.msg); k++)
|
||||
fputc('q', fp_);
|
||||
});
|
||||
if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
|
||||
return 1;
|
||||
for (k = 0; k < sizeof(r.msg) - 1; k++) {
|
||||
if (r.msg[k] != 'q')
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* well under capacity: nothing padded, nothing eaten off the end */
|
||||
FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
|
||||
if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
|
||||
return 1;
|
||||
|
||||
/* a byte over 0x7f must not read as EOF and cut the message short */
|
||||
FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
|
||||
if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
|
||||
return 1;
|
||||
|
||||
/* unparseable status: the message still loads, the code reports failure */
|
||||
FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
|
||||
if (r.statuscode != STATUSCODE_INVALID)
|
||||
return 1;
|
||||
#undef FTP_RESULT_CASE
|
||||
}
|
||||
|
||||
/* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
|
||||
n_eval == 2 on a double-eval macro. */
|
||||
{
|
||||
@@ -1126,6 +1342,128 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// hts_split_cmdline(): the vector must grow with the argument count, and a
|
||||
// quote inside a value must not end the argument and hand -V to the parser.
|
||||
static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
|
||||
char line[512];
|
||||
char **args;
|
||||
int nargs = 0;
|
||||
|
||||
(void) opt;
|
||||
(void) argc;
|
||||
(void) argv;
|
||||
|
||||
// control: every separator splits, and argv[0] is the program name
|
||||
strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(args != NULL && nargs == 6);
|
||||
assertf(args[nargs] == NULL); // callers may walk to the terminator
|
||||
assertf(strcmp(args[0], "httrack") == 0);
|
||||
assertf(strcmp(args[1], "http://x/") == 0);
|
||||
assertf(strcmp(args[2], "--quiet") == 0);
|
||||
assertf(strcmp(args[3], "-c8") == 0);
|
||||
assertf(strcmp(args[4], "-O") == 0);
|
||||
assertf(strcmp(args[5], "out") == 0);
|
||||
freet(args);
|
||||
|
||||
// the template pads with whitespace: empty arguments are kept (the engine
|
||||
// skips them), so the count is one per separator
|
||||
strcpybuff(line, "httrack --quiet");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 3 && args[1][0] == '\0');
|
||||
assertf(strcmp(args[2], "--quiet") == 0);
|
||||
freet(args);
|
||||
|
||||
// a quoted run keeps both its spaces and its quotes: the engine unquotes
|
||||
strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 4);
|
||||
assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
|
||||
assertf(strcmp(args[3], "-c8") == 0);
|
||||
freet(args);
|
||||
|
||||
// an escaped quote is a literal quote, not the end of the argument: the
|
||||
// engine strips only the outer pair
|
||||
strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 4);
|
||||
assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
|
||||
assertf(strcmp(args[3], "-c8") == 0);
|
||||
freet(args);
|
||||
|
||||
// \\ is a literal backslash, so a Windows path survives
|
||||
strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 3);
|
||||
assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
|
||||
freet(args);
|
||||
|
||||
// outside a quoted run a backslash is literal: the url and wildcard-filter
|
||||
// fields, which the wizard cannot quote, read as before
|
||||
strcpybuff(line, "httrack -*\\** +*.png");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 3);
|
||||
assertf(strcmp(args[1], "-*\\**") == 0);
|
||||
assertf(strcmp(args[2], "+*.png") == 0);
|
||||
freet(args);
|
||||
|
||||
// a quoted run leaves slots unused, so the terminator has to be written and
|
||||
// not inherited: size the vector from a full line first, so freeing it hands
|
||||
// the same chunk back with stale pointers in those slots
|
||||
strcpybuff(line, "httrack a b c d e");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 6);
|
||||
freet(args);
|
||||
strcpybuff(line, "httrack \"a b c d e\"");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 2);
|
||||
assertf(args[nargs] == NULL);
|
||||
freet(args);
|
||||
|
||||
// an unterminated quote protects the rest of the line, as one argument
|
||||
strcpybuff(line, "httrack --footer \"unbalanced -V x");
|
||||
args = hts_split_cmdline(line, &nargs);
|
||||
assertf(nargs == 3);
|
||||
assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
|
||||
freet(args);
|
||||
|
||||
// past the 1024 entries the vector used to hold: distinct arguments, so a
|
||||
// write beyond the allocation cannot read back as the expected parse
|
||||
{
|
||||
const int n = 2000;
|
||||
const size_t size = 16 * (size_t) n + 16;
|
||||
char *big = malloct(size);
|
||||
size_t pos = 0;
|
||||
int i;
|
||||
|
||||
assertf(big != NULL);
|
||||
pos = (size_t) snprintf(big, size, "httrack");
|
||||
assertf(pos < size);
|
||||
for (i = 0; i < n; i++) {
|
||||
// snprintf returns what it wanted to write, so accumulating it blind
|
||||
// would let the next size argument wrap
|
||||
const int len = snprintf(big + pos, size - pos, " a%d", i);
|
||||
|
||||
assertf(len > 0 && (size_t) len < size - pos);
|
||||
pos += (size_t) len;
|
||||
}
|
||||
args = hts_split_cmdline(big, &nargs);
|
||||
assertf(args != NULL && nargs == n + 1);
|
||||
assertf(args[nargs] == NULL);
|
||||
for (i = 0; i < n; i++) {
|
||||
char expect[16];
|
||||
|
||||
snprintf(expect, sizeof(expect), "a%d", i);
|
||||
assertf(strcmp(args[i + 1], expect) == 0);
|
||||
}
|
||||
freet(args);
|
||||
freet(big);
|
||||
}
|
||||
|
||||
printf("cmdline-split self-test OK\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int st_hashtable(httrackp *opt, int argc, char **argv) {
|
||||
char *snum;
|
||||
unsigned long count = 0;
|
||||
@@ -1247,7 +1585,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
|
||||
size_t i;
|
||||
for (i = bench[loop].offset; i < (size_t) count;
|
||||
i += bench[loop].modulus) {
|
||||
int result;
|
||||
int result = 0; /* no final else: an unknown type reports failure */
|
||||
FMT();
|
||||
if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
|
||||
size_t k;
|
||||
@@ -2762,6 +3100,32 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
|
||||
assertf(strstr(buf, "Refresh") != NULL);
|
||||
assertf(strstr(buf, "example.com") != NULL);
|
||||
|
||||
/* a first link whose escaped form overruns the old flat 1024-byte tempo: the
|
||||
redirect must carry the whole URL, not a clipped prefix */
|
||||
{
|
||||
char BIGSTK link[HTS_URLMAXSIZE * 2];
|
||||
char *p = link;
|
||||
|
||||
strcpybuff(link, "http://example.com/");
|
||||
p += strlen(link);
|
||||
memset(p, 'a', 1200);
|
||||
p += 1200;
|
||||
strcpy(p, "/end.html");
|
||||
|
||||
done = 0;
|
||||
fp = fopen(path, "wb");
|
||||
assertf(fp != NULL);
|
||||
hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
|
||||
assertf(fp == NULL);
|
||||
fp = fopen(path, "rb");
|
||||
assertf(fp != NULL);
|
||||
n = fread(buf, 1, sizeof(buf) - 1, fp);
|
||||
fclose(fp);
|
||||
buf[n] = '\0';
|
||||
/* the closing quote proves the URL was not clipped mid-way */
|
||||
assertf(strstr(buf, "/end.html\">") != NULL);
|
||||
}
|
||||
|
||||
/* no single link: footer only, no refresh meta */
|
||||
done = 0;
|
||||
fp = fopen(path, "wb");
|
||||
@@ -4801,10 +5165,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
|
||||
char fpath[HTS_URLMAXSIZE * 2];
|
||||
char file[HTS_URLMAXSIZE * 2];
|
||||
|
||||
snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
|
||||
assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
|
||||
|
||||
/* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
|
||||
snprintf(file, sizeof(file), "%scookies.txt", fpath);
|
||||
assertf(sprintfbuff(file, "%scookies.txt", fpath));
|
||||
{
|
||||
FILE *fp = FOPEN(file, "wb");
|
||||
|
||||
@@ -4814,7 +5178,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
|
||||
}
|
||||
|
||||
/* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
|
||||
snprintf(file, sizeof(file), "%su@v.txt", fpath);
|
||||
assertf(sprintfbuff(file, "%su@v.txt", fpath));
|
||||
{
|
||||
FILE *fp = FOPEN(file, "wb");
|
||||
|
||||
@@ -4836,7 +5200,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
|
||||
#endif
|
||||
|
||||
(void) UNLINK(file); /* u@v.txt (already gone on Windows) */
|
||||
snprintf(file, sizeof(file), "%scookies.txt", fpath);
|
||||
assertf(sprintfbuff(file, "%scookies.txt", fpath));
|
||||
(void) UNLINK(file);
|
||||
dir[dirlen] = '\0';
|
||||
while (strlen(dir) > base) {
|
||||
@@ -4910,6 +5274,9 @@ static const struct selftest_entry {
|
||||
st_footerfmt},
|
||||
{"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
|
||||
st_unescape_bounds},
|
||||
{"cmdline-split", "",
|
||||
"webhttrack command-line to argv split (bounds, quoting)",
|
||||
st_cmdlinesplit},
|
||||
{"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
|
||||
{"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
|
||||
"bounded string-op self-test", st_strsafe},
|
||||
|
||||
206
src/htsserver.c
206
src/htsserver.c
@@ -318,6 +318,18 @@ typedef struct {
|
||||
error_redirect = "/server/error.html"; \
|
||||
} while(0)
|
||||
|
||||
/* Longest error message shown on the error page; the rest is clipped. */
|
||||
#define ERROR_MESSAGE_MAX 1024
|
||||
|
||||
/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
|
||||
whose length the client picks. */
|
||||
#define SET_ERRORF(...) \
|
||||
do { \
|
||||
char errbuf[ERROR_MESSAGE_MAX]; \
|
||||
slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__); \
|
||||
SET_ERROR(errbuf); \
|
||||
} while (0)
|
||||
|
||||
/* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
|
||||
so anything near this is already invalid and is rejected unread. */
|
||||
#define SID_VALUE_MAX 64
|
||||
@@ -363,6 +375,59 @@ static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
|
||||
return seen;
|
||||
}
|
||||
|
||||
/** Append src to the NUL-terminated dst of capacity size (NUL included).
|
||||
False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
|
||||
never aborts, because every piece appended here is client-supplied. */
|
||||
static hts_boolean path_append(char *dst, size_t size, const char *src) {
|
||||
const size_t used = strlen(dst);
|
||||
const size_t len = strlen(src);
|
||||
|
||||
/* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
|
||||
len stays alone: "used + len < size" could wrap and pass. */
|
||||
if (len >= size - used) {
|
||||
return HTS_FALSE;
|
||||
}
|
||||
memcpy(dst + used, src, len + 1);
|
||||
return HTS_TRUE;
|
||||
}
|
||||
|
||||
/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
|
||||
static hts_boolean cat_html_escaped(String *dst, char c) {
|
||||
switch (c) {
|
||||
case '<':
|
||||
StringCat(*dst, "<");
|
||||
break;
|
||||
case '>':
|
||||
StringCat(*dst, ">");
|
||||
break;
|
||||
case '&':
|
||||
StringCat(*dst, "&");
|
||||
break;
|
||||
case '\'':
|
||||
StringCat(*dst, "'");
|
||||
break;
|
||||
default:
|
||||
return HTS_FALSE;
|
||||
}
|
||||
return HTS_TRUE;
|
||||
}
|
||||
|
||||
/* Append the value of a double-quoted command-line argument: escaped for HTML,
|
||||
which the browser undoes when it posts the command line back, and for the
|
||||
argv splitter, which does not. */
|
||||
static void cat_cmdline_arg(String *output, const char *value) {
|
||||
const char *a;
|
||||
|
||||
for (a = value; *a != '\0'; a++) {
|
||||
if (*a == '\\' || *a == '\"') {
|
||||
StringCat(*output, "\\");
|
||||
}
|
||||
if (!cat_html_escaped(output, *a)) {
|
||||
StringMemcat(*output, a, 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
int timeout = 30;
|
||||
int retour = 0;
|
||||
@@ -374,6 +439,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
String tmpbuff = STRING_EMPTY;
|
||||
String tmpbuff2 = STRING_EMPTY;
|
||||
String fspath = STRING_EMPTY;
|
||||
/* Project directory this server set up; the only root /website/ serves from,
|
||||
and deliberately not cleared between requests. */
|
||||
String website = STRING_EMPTY;
|
||||
char catbuff[CATBUFF_SIZE];
|
||||
|
||||
/* Load strings */
|
||||
@@ -669,7 +737,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
fp = fopen(StringBuff(fspath), "rb");
|
||||
if (fp) {
|
||||
/* Read file */
|
||||
while(!feof(fp)) {
|
||||
while (!feof(fp) && !ferror(fp)) {
|
||||
char *str = line;
|
||||
char *pos;
|
||||
|
||||
@@ -785,6 +853,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
if (!structcheck(StringBuff(tmpbuff))) {
|
||||
FILE *fp;
|
||||
|
||||
/* Both halves of fspath come from posted fields, so a ".."
|
||||
in them would escape the mirror once served. */
|
||||
if (strstr(StringBuff(fspath), "..") == NULL) {
|
||||
StringCopy(website, StringBuff(fspath));
|
||||
}
|
||||
StringCat(tmpbuff, "winprofile.ini");
|
||||
fp = fopen(StringBuff(tmpbuff), "wb");
|
||||
if (fp != NULL) {
|
||||
@@ -818,28 +891,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
commandEnd = 1;
|
||||
}
|
||||
} else {
|
||||
char tmp[1024];
|
||||
|
||||
sprintf(tmp,
|
||||
"Unable to write %d bytes in the the init file %s",
|
||||
count, StringBuff(fspath));
|
||||
SET_ERROR(tmp);
|
||||
SET_ERRORF(
|
||||
"Unable to write %d bytes in the the init file %s",
|
||||
count, StringBuff(fspath));
|
||||
}
|
||||
fclose(fp);
|
||||
} else {
|
||||
char tmp[1024];
|
||||
|
||||
sprintf(tmp, "Unable to create the init file %s",
|
||||
StringBuff(fspath));
|
||||
SET_ERROR(tmp);
|
||||
SET_ERRORF("Unable to create the init file %s",
|
||||
StringBuff(fspath));
|
||||
}
|
||||
} else {
|
||||
char tmp[1024];
|
||||
|
||||
sprintf(tmp,
|
||||
"Unable to create the directory structure in %s",
|
||||
StringBuff(fspath));
|
||||
SET_ERROR(tmp);
|
||||
SET_ERRORF("Unable to create the directory structure in %s",
|
||||
StringBuff(fspath));
|
||||
}
|
||||
|
||||
} else {
|
||||
@@ -857,7 +920,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
|
||||
/* Response */
|
||||
if (meth) {
|
||||
int virtualpath = 0;
|
||||
hts_boolean virtualpath = HTS_FALSE;
|
||||
char *pos;
|
||||
char *url = strchr(line1, ' ');
|
||||
|
||||
@@ -868,11 +931,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
char *qpos;
|
||||
|
||||
/* get the URL */
|
||||
fsfile[0] = '\0';
|
||||
if (error_redirect == NULL) {
|
||||
if ((qpos = strchr(url, '?'))) {
|
||||
*qpos = '\0';
|
||||
}
|
||||
fsfile[0] = '\0';
|
||||
if (strcmp(url, "/") == 0) {
|
||||
file = "/server/index.html";
|
||||
meth = 2;
|
||||
@@ -885,7 +948,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
}
|
||||
|
||||
if (strncmp(file, "/website/", 9) == 0) {
|
||||
virtualpath = 1;
|
||||
virtualpath = HTS_TRUE;
|
||||
}
|
||||
|
||||
/* override */
|
||||
@@ -899,20 +962,28 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
}
|
||||
}
|
||||
|
||||
if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
|
||||
if (strncmp(file, "/website/", 9) != 0) {
|
||||
sprintf(fsfile, "%shtml%s", path, file);
|
||||
} else {
|
||||
intptr_t adr = 0;
|
||||
/* the override above may have swapped a mirror path for a GUI page */
|
||||
virtualpath = strncmp(file, "/website/", 9) == 0;
|
||||
|
||||
if (coucal_readptr(NewLangList, "projpath", &adr)) {
|
||||
sprintf(fsfile, "%s%s", (char *) adr, file + 9);
|
||||
}
|
||||
if (!virtualpath) {
|
||||
if (!path_append(fsfile, sizeof(fsfile), path) ||
|
||||
!path_append(fsfile, sizeof(fsfile), "html") ||
|
||||
!path_append(fsfile, sizeof(fsfile), file)) {
|
||||
fsfile[0] = '\0';
|
||||
}
|
||||
} else if (StringNotEmpty(website)) {
|
||||
/* Never the posted "projpath": a client root reads any file. */
|
||||
if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
|
||||
!path_append(fsfile, sizeof(fsfile), "/") ||
|
||||
!path_append(fsfile, sizeof(fsfile), file + 9)) {
|
||||
fsfile[0] = '\0';
|
||||
}
|
||||
}
|
||||
|
||||
if (fsfile[0] && strstr(file, "..") == NULL
|
||||
&& (fp = fopen(fsfile, "rb"))) {
|
||||
/* Regular files only: reading a directory or FIFO never ends, and
|
||||
"path" may hold "..", so only the untrusted halves are checked. */
|
||||
if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
|
||||
(fp = fopen(fsfile, "rb"))) {
|
||||
char ok[] =
|
||||
"HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
|
||||
"Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
|
||||
@@ -965,11 +1036,13 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
StringCat(headers, "\r\n");
|
||||
}
|
||||
coucal_write(NewLangList, "redirect", (intptr_t) NULL);
|
||||
} else if (is_html(file)) {
|
||||
} else if (!virtualpath && is_html(file)) {
|
||||
/* GUI templates only: ${_sid} in a mirrored page would hand the
|
||||
crawled site the session id that authenticates commands */
|
||||
int outputmode = 0;
|
||||
|
||||
StringMemcat(headers, ok, sizeof(ok) - 1);
|
||||
while(!feof(fp)) {
|
||||
while (!feof(fp) && !ferror(fp)) {
|
||||
char *str = line;
|
||||
int prevlen = (int) StringLength(output);
|
||||
int nocr = 0;
|
||||
@@ -993,6 +1066,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
int p;
|
||||
int format = 0;
|
||||
int listDefault = 0;
|
||||
hts_boolean unquoted = HTS_FALSE;
|
||||
|
||||
name[0] = '\0';
|
||||
strlncatbuff(name, str, sizeof(name_), n);
|
||||
@@ -1002,6 +1076,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
} else if ((p = strfield(name, "html:"))) {
|
||||
name += p;
|
||||
format = 1;
|
||||
} else if ((p = strfield(name, "unquoted:"))) {
|
||||
name += p;
|
||||
unquoted = HTS_TRUE;
|
||||
} else if ((p = strfield(name, "arg:"))) {
|
||||
name += p;
|
||||
format = 5;
|
||||
} else if ((p = strfield(name, "list:"))) {
|
||||
name += p;
|
||||
format = 2;
|
||||
@@ -1138,8 +1218,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
test:<if ==0>:<if ==1>:<if == 2>..
|
||||
ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
|
||||
*/
|
||||
else if ((p = strfield(name, "test:"))
|
||||
|| (p = strfield(name, "ztest:"))) {
|
||||
else if ((p = strfield(name, "test:")) ||
|
||||
(p = strfield(name, "ztest:"))) {
|
||||
intptr_t adr = 0;
|
||||
char *pos2;
|
||||
int ztest = (name[0] == 'z');
|
||||
@@ -1242,6 +1322,12 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
}
|
||||
}
|
||||
}
|
||||
/* consumed here: it shares nothing with the list and
|
||||
option formats below */
|
||||
if (format == 5 && langstr != NULL && outputmode != -1) {
|
||||
cat_cmdline_arg(&output, langstr);
|
||||
langstr = NULL;
|
||||
}
|
||||
if (langstr && outputmode != -1) {
|
||||
switch (format) {
|
||||
case 0:
|
||||
@@ -1259,18 +1345,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
StringMemcat(output, &c, 1);
|
||||
}
|
||||
a += 2;
|
||||
} else if (outputmode && a[0] == '<') {
|
||||
StringCat(output, "<");
|
||||
} else if (outputmode && a[0] == '>') {
|
||||
StringCat(output, ">");
|
||||
} else if (outputmode && a[0] == '&') {
|
||||
StringCat(output, "&");
|
||||
} else if (outputmode && a[0] == '\'') {
|
||||
StringCat(output, "'");
|
||||
} else if (unquoted && a[0] == '\"') {
|
||||
/* the browser posts an entity back as a raw
|
||||
quote, which would open a quoted run in the
|
||||
argv splitter; a URI cannot hold one anyway */
|
||||
StringCat(output, "%22");
|
||||
} else if (outputmode &&
|
||||
cat_html_escaped(&output, a[0])) {
|
||||
/* appended as an entity */
|
||||
} else if (outputmode == 3 && a[0] == ' ') {
|
||||
StringCat(output, "%20");
|
||||
} else if (outputmode >= 2
|
||||
&& ((unsigned char) a[0]) < 32) {
|
||||
} else if (outputmode >= 2 &&
|
||||
((unsigned char) a[0]) < 32) {
|
||||
char tmp[32];
|
||||
|
||||
sprintf(tmp, "%%%02x", (unsigned char) a[0]);
|
||||
@@ -1331,20 +1417,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
}
|
||||
StringClear(tmpbuff);
|
||||
break;
|
||||
case '<':
|
||||
StringCat(tmpbuff, "<");
|
||||
break;
|
||||
case '>':
|
||||
StringCat(tmpbuff, ">");
|
||||
break;
|
||||
case '&':
|
||||
StringCat(tmpbuff, "&");
|
||||
break;
|
||||
case '\'':
|
||||
StringCat(tmpbuff, "'");
|
||||
break;
|
||||
default:
|
||||
StringMemcat(tmpbuff, fstr, 1);
|
||||
if (!cat_html_escaped(&tmpbuff, *fstr)) {
|
||||
StringMemcat(tmpbuff, fstr, 1);
|
||||
}
|
||||
break;
|
||||
}
|
||||
fstr++;
|
||||
@@ -1384,7 +1460,9 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
}
|
||||
#endif
|
||||
} else {
|
||||
if (is_text(file)) {
|
||||
if (is_html(file)) {
|
||||
StringMemcat(headers, ok, sizeof(ok) - 1);
|
||||
} else if (is_text(file)) {
|
||||
StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
|
||||
} else if (is_js(file)) {
|
||||
StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
|
||||
@@ -1398,14 +1476,15 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
while(!feof(fp)) {
|
||||
int n = (int) fread(line, 1, sizeof(line) - 2, fp);
|
||||
|
||||
if (n > 0) {
|
||||
StringMemcat(output, line, n);
|
||||
if (n <= 0) {
|
||||
break; /* short read: EOF or error, never a retry */
|
||||
}
|
||||
StringMemcat(output, line, n);
|
||||
}
|
||||
}
|
||||
fclose(fp);
|
||||
} else if (strcmp(file, "/ping") == 0
|
||||
|| strncmp(file, "/ping?", 6) == 0) {
|
||||
} else if (strcmp(file, "/ping") == 0 ||
|
||||
strncmp(file, "/ping?", 6) == 0) {
|
||||
char error_hdr[] =
|
||||
"HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
|
||||
"Content-type: text/html\r\n";
|
||||
@@ -1491,6 +1570,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
|
||||
StringFree(tmpbuff);
|
||||
StringFree(tmpbuff2);
|
||||
StringFree(fspath);
|
||||
StringFree(website);
|
||||
|
||||
if (buffer)
|
||||
free(buffer);
|
||||
|
||||
63
src/htsstats.h
Normal file
63
src/htsstats.h
Normal file
@@ -0,0 +1,63 @@
|
||||
/* ------------------------------------------------------------ */
|
||||
/*
|
||||
HTTrack Website Copier, Offline Browser for Windows and Unix
|
||||
Copyright (C) 1998 Xavier Roche and other contributors
|
||||
|
||||
SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License
|
||||
along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
Ethical use: we kindly ask that you NOT use this software to harvest email
|
||||
addresses or to collect any other private information about people. Doing so
|
||||
would dishonor our work and waste the many hours we have spent on it.
|
||||
|
||||
Please visit our Website: http://www.httrack.com
|
||||
*/
|
||||
|
||||
/* ------------------------------------------------------------ */
|
||||
/* File: in-progress display rows, shared by httrack and htsserver .h */
|
||||
/* Author: Xavier Roche */
|
||||
/* ------------------------------------------------------------ */
|
||||
|
||||
#ifndef HTS_STATS_DEFH
|
||||
#define HTS_STATS_DEFH
|
||||
|
||||
#include "htsglobal.h"
|
||||
|
||||
/* One row of the "in progress" display, shared so the CLI (httrack) and the
|
||||
web GUI (htsserver) cannot drift apart: each fills its own array. */
|
||||
|
||||
#define NStatsBuffer 14
|
||||
|
||||
#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
|
||||
#define HTS_DEF_FWSTRUCT_t_StatsBuffer
|
||||
typedef struct t_StatsBuffer t_StatsBuffer;
|
||||
#endif
|
||||
struct t_StatsBuffer {
|
||||
char name[1024];
|
||||
char file[1024];
|
||||
char state[288]; // a short label plus back->info[256]
|
||||
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
|
||||
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
|
||||
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
|
||||
LLint size;
|
||||
LLint sizetot;
|
||||
int offset;
|
||||
//
|
||||
int back;
|
||||
//
|
||||
int actived; // pour disabled
|
||||
};
|
||||
|
||||
#endif
|
||||
@@ -308,8 +308,10 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
|
||||
// copy only the current path
|
||||
curr = _curr;
|
||||
strlcpybuff(curr, curr_fil, sizeof(_curr));
|
||||
if ((a = strchr(curr, '?')) == NULL) // couper au ? (params)
|
||||
a = curr + strlen(curr) - 1; // pas de params: aller à la fin
|
||||
if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
|
||||
// an empty path has no last character: curr-1 would read before the buffer
|
||||
a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
|
||||
}
|
||||
while((*a != '/') && (a > curr))
|
||||
a--; // chercher dernier / du chemin courant
|
||||
if (*a == '/')
|
||||
|
||||
39
src/htsweb.c
39
src/htsweb.c
@@ -62,6 +62,7 @@ Please visit our Website: http://www.httrack.com
|
||||
#include "htsmd5.c"
|
||||
#include "md5.c"
|
||||
|
||||
#include "htscmdline.h"
|
||||
#include "htsserver.h"
|
||||
#include "htsurlport.h"
|
||||
#include "htsweb.h"
|
||||
@@ -319,10 +320,8 @@ int main(int argc, char *argv[]) {
|
||||
static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
|
||||
static void back_launch_cmd(void *pP) {
|
||||
char *cmd = (char *) pP;
|
||||
char **argv = (char **) malloct(1024 * sizeof(char *));
|
||||
char **argv;
|
||||
int argc = 0;
|
||||
int i = 0;
|
||||
int g = 0;
|
||||
|
||||
//
|
||||
httrackp *opt;
|
||||
@@ -333,28 +332,19 @@ static void back_launch_cmd(void *pP) {
|
||||
commandReturnCmdl = strdup(cmd);
|
||||
|
||||
/* split */
|
||||
argv[0] = strdup("webhttrack");
|
||||
argv[1] = cmd;
|
||||
argc++;
|
||||
i = 0;
|
||||
while(cmd[i]) {
|
||||
if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
|
||||
cmd[i] = ' ';
|
||||
}
|
||||
i++;
|
||||
}
|
||||
i = 0;
|
||||
while(cmd[i]) {
|
||||
if (cmd[i] == '\"')
|
||||
g = !g;
|
||||
if (cmd[i] == ' ') {
|
||||
if (!g) {
|
||||
cmd[i] = '\0';
|
||||
argv[argc++] = cmd + i + 1;
|
||||
}
|
||||
}
|
||||
i++;
|
||||
argv = hts_split_cmdline(cmd, &argc);
|
||||
if (argv == NULL) {
|
||||
if (commandReturnMsg)
|
||||
free(commandReturnMsg);
|
||||
commandReturnMsg = strdup("could not parse the command line");
|
||||
commandReturn = -1;
|
||||
commandRunning = 0;
|
||||
commandEnd = 1;
|
||||
free(cmd);
|
||||
return;
|
||||
}
|
||||
/* drop the program name the posted command line carries */
|
||||
argv[0] = strdupt("webhttrack");
|
||||
|
||||
/* init */
|
||||
hts_init();
|
||||
@@ -383,6 +373,7 @@ static void back_launch_cmd(void *pP) {
|
||||
|
||||
/* free */
|
||||
free(cmd);
|
||||
freet(argv[0]);
|
||||
freet(argv);
|
||||
return;
|
||||
}
|
||||
|
||||
18
src/htsweb.h
18
src/htsweb.h
@@ -35,26 +35,10 @@ Please visit our Website: http://www.httrack.com
|
||||
|
||||
#include "htsglobal.h"
|
||||
#include "htscore.h"
|
||||
#include "htsstats.h"
|
||||
|
||||
#define NStatsBuffer 14
|
||||
#define MAX_LEN_INPROGRESS 40
|
||||
|
||||
typedef struct t_StatsBuffer {
|
||||
char name[1024];
|
||||
char file[1024];
|
||||
char state[256];
|
||||
char url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
|
||||
char url_adr[HTS_URLMAXSIZE * 2];
|
||||
char url_fil[HTS_URLMAXSIZE * 2];
|
||||
LLint size;
|
||||
LLint sizetot;
|
||||
int offset;
|
||||
//
|
||||
int back;
|
||||
//
|
||||
int actived; // pour disabled
|
||||
} t_StatsBuffer;
|
||||
|
||||
typedef struct t_InpInfo {
|
||||
int ask_refresh;
|
||||
int refresh;
|
||||
|
||||
@@ -230,8 +230,7 @@ static hts_boolean vt_size_refresh(void) {
|
||||
*/
|
||||
#define STYLE_STATVALUES VT_BOLD
|
||||
#define STYLE_STATTEXT VT_UNBOLD
|
||||
#define STYLE_STATRESET VT_UNBOLD
|
||||
#define NStatsBuffer 14
|
||||
#define STYLE_STATRESET VT_UNBOLD
|
||||
/* Rows the stats block and "Current job" take above the in-progress list. */
|
||||
#define NStatsHeaderRows 7
|
||||
|
||||
|
||||
@@ -36,26 +36,7 @@ Please visit our Website: http://www.httrack.com
|
||||
#include "htsglobal.h"
|
||||
#include "htscore.h"
|
||||
#include "htssafe.h"
|
||||
|
||||
#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
|
||||
#define HTS_DEF_FWSTRUCT_t_StatsBuffer
|
||||
typedef struct t_StatsBuffer t_StatsBuffer;
|
||||
#endif
|
||||
struct t_StatsBuffer {
|
||||
char name[1024];
|
||||
char file[1024];
|
||||
char state[256];
|
||||
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
|
||||
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
|
||||
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
|
||||
LLint size;
|
||||
LLint sizetot;
|
||||
int offset;
|
||||
//
|
||||
int back;
|
||||
//
|
||||
int actived; // pour disabled
|
||||
};
|
||||
#include "htsstats.h"
|
||||
|
||||
#ifndef HTS_DEF_FWSTRUCT_t_InpInfo
|
||||
#define HTS_DEF_FWSTRUCT_t_InpInfo
|
||||
|
||||
@@ -122,6 +122,7 @@
|
||||
<ClCompile Include="htshelp.c" />
|
||||
<ClCompile Include="htsindex.c" />
|
||||
<ClCompile Include="htslib.c" />
|
||||
<ClCompile Include="htscmdline.c" />
|
||||
<ClCompile Include="htsurlport.c" />
|
||||
<ClCompile Include="htsmd5.c" />
|
||||
<ClCompile Include="htsmodules.c" />
|
||||
|
||||
@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
|
||||
PT_Element elt = PT_ElementNew();
|
||||
|
||||
elt->statuscode = 405;
|
||||
strcpy(elt->msg, "Method Not Allowed");
|
||||
strcpybuff(elt->msg, "Method Not Allowed");
|
||||
return elt;
|
||||
}
|
||||
|
||||
@@ -820,9 +820,9 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
|
||||
elt->size = StringLength(response);
|
||||
elt->adr = StringAcquire(&response);
|
||||
elt->statuscode = 207; /* Multi-Status */
|
||||
strcpy(elt->charset, "utf-8");
|
||||
strcpy(elt->contenttype, "text/xml");
|
||||
strcpy(elt->msg, "Multi-Status");
|
||||
strcpybuff(elt->charset, "utf-8");
|
||||
strcpybuff(elt->contenttype, "text/xml");
|
||||
strcpybuff(elt->msg, "Multi-Status");
|
||||
StringFree(response);
|
||||
|
||||
fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
|
||||
@@ -877,9 +877,9 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
|
||||
elt->size = StringLength(html);
|
||||
elt->adr = StringAcquire(&html);
|
||||
elt->statuscode = HTTP_OK;
|
||||
strcpy(elt->charset, "iso-8859-1");
|
||||
strcpy(elt->contenttype, "text/html");
|
||||
strcpy(elt->msg, "OK");
|
||||
strcpybuff(elt->charset, "iso-8859-1");
|
||||
strcpybuff(elt->contenttype, "text/html");
|
||||
strcpybuff(elt->msg, "OK");
|
||||
StringFree(html);
|
||||
return elt;
|
||||
}
|
||||
|
||||
@@ -368,7 +368,11 @@ HTS_UNUSED static struct tm PT_GetTime(time_t t) {
|
||||
if (tm != NULL)
|
||||
return *tm;
|
||||
else {
|
||||
/* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
|
||||
day of "00"; the epoch is the conventional "date unknown" */
|
||||
memset(&tmbuf, 0, sizeof(tmbuf));
|
||||
tmbuf.tm_year = 70;
|
||||
tmbuf.tm_mday = 1;
|
||||
return tmbuf;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -405,9 +405,9 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
|
||||
elt->size = StringLength(html);
|
||||
elt->adr = StringAcquire(&html);
|
||||
elt->statuscode = HTTP_OK;
|
||||
strcpy(elt->charset, "iso-8859-1");
|
||||
strcpy(elt->contenttype, "text/html");
|
||||
strcpy(elt->msg, "OK");
|
||||
strcpybuff(elt->charset, "iso-8859-1");
|
||||
strcpybuff(elt->contenttype, "text/html");
|
||||
strcpybuff(elt->msg, "OK");
|
||||
StringFree(html);
|
||||
return elt;
|
||||
}
|
||||
@@ -579,9 +579,10 @@ PT_Index PT_LoadCache(const char *filename) {
|
||||
if (chain != NULL) {
|
||||
const char *scheme = link_has_authority(chain->name) ? "" : "http://";
|
||||
|
||||
snprintf(index->slots.common.startUrl,
|
||||
sizeof(index->slots.common.startUrl), "%s%s", scheme,
|
||||
(const char *) chain->name);
|
||||
/* dropped rather than truncated: empty already reads as "unset" */
|
||||
if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
|
||||
(const char *) chain->name))
|
||||
index->slots.common.startUrl[0] = '\0';
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -827,6 +828,10 @@ PT_Element PT_ElementNew(void) {
|
||||
return r;
|
||||
}
|
||||
|
||||
/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
|
||||
#define PT_Element_failf(R, ...) \
|
||||
slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
|
||||
|
||||
PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
|
||||
if (index != NULL && SAFE_INDEX(index)) {
|
||||
return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
|
||||
@@ -865,12 +870,15 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
|
||||
sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
|
||||
(headersSize) += (int) strlen(headers + headersSize); \
|
||||
} while(0)
|
||||
#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
strcpy(refvalue, value); \
|
||||
line[0] = '\0'; \
|
||||
} \
|
||||
} while(0)
|
||||
/* refvalue_size is mandatory: the cache line is bounded only by the line
|
||||
buffer, not by the destination. */
|
||||
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
|
||||
do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
(void) strclipbuff(refvalue, refvalue_size, value); \
|
||||
line[0] = '\0'; \
|
||||
} \
|
||||
} while (0)
|
||||
#define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
int intval = 0; \
|
||||
@@ -972,9 +980,12 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
|
||||
const char *scheme =
|
||||
link_has_authority(filenameIndex) ? "" : "http://";
|
||||
|
||||
firstSeen = 1;
|
||||
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
|
||||
scheme, filenameIndex);
|
||||
/* dropped rather than truncated; try the next entry */
|
||||
if (sprintfbuff(index->startUrl, "%s%s", scheme,
|
||||
filenameIndex))
|
||||
firstSeen = 1;
|
||||
else
|
||||
index->startUrl[0] = '\0';
|
||||
}
|
||||
}
|
||||
} else {
|
||||
@@ -1021,7 +1032,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
previous_save[0] = previous_save_[0] = '\0';
|
||||
memset(r, 0, sizeof(_PT_Element));
|
||||
r->location = location_default;
|
||||
strcpy(r->location, "");
|
||||
r->location[0] = '\0';
|
||||
if (strncmp(url, "http://", 7) == 0)
|
||||
url += 7;
|
||||
hash_pos_return = coucal_read(index->hash, url, &hash_pos);
|
||||
@@ -1070,16 +1081,23 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
value++;
|
||||
ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
|
||||
ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
|
||||
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg); // msg
|
||||
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
|
||||
sizeof(r->msg));
|
||||
ZIP_READFIELD_INT(line, value, "X-Size", r->size); // size
|
||||
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
|
||||
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset); // contenttype
|
||||
ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
|
||||
ZIP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
|
||||
ZIP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
|
||||
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
|
||||
sizeof(r->contenttype));
|
||||
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
|
||||
sizeof(r->charset));
|
||||
ZIP_READFIELD_STRING(line, value, "Last-Modified",
|
||||
r->lastmodified, sizeof(r->lastmodified));
|
||||
ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
|
||||
sizeof(r->etag));
|
||||
ZIP_READFIELD_STRING(line, value, "Location", r->location,
|
||||
sizeof(location_default));
|
||||
ZIP_READFIELD_STRING(line, value, "Content-Disposition",
|
||||
r->cdispo); // Content-disposition
|
||||
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_); // Original save filename
|
||||
r->cdispo, sizeof(r->cdispo));
|
||||
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
|
||||
sizeof(previous_save_));
|
||||
if (line[0] != '\0') {
|
||||
int len = r->headers ? ((int) strlen(r->headers)) : 0;
|
||||
int nlen =
|
||||
@@ -1099,7 +1117,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
int pathLen = (int) strlen(index->path);
|
||||
|
||||
if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) { // old (<3.40) buggy format
|
||||
strcpy(previous_save, previous_save_);
|
||||
strcpybuff(previous_save, previous_save_);
|
||||
}
|
||||
// relative ? (hack)
|
||||
else if (index->safeCache || (previous_save_[0] != '/' // /home/foo/bar.gif
|
||||
@@ -1135,8 +1153,9 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
snprintf(previous_save, sizeof(previous_save), "%s%s",
|
||||
index->path, previous_save_ + index->fixedPath);
|
||||
} else {
|
||||
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
|
||||
previous_save_, (int) index->fixedPath);
|
||||
PT_Element_failf(
|
||||
r, "Bogus fixePath prefix for %s (prefixLen=%d)",
|
||||
previous_save_, (int) index->fixedPath);
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
@@ -1155,7 +1174,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
// Peut-on stocker le fichier directement sur disque?
|
||||
if (ok) {
|
||||
if (r->msg[0] == '\0') {
|
||||
strcpy(r->msg, "Cache Read Error : Unexpected error");
|
||||
strcpybuff(r->msg, "Cache Read Error : Unexpected error");
|
||||
}
|
||||
} else { // lire en mémoire
|
||||
|
||||
@@ -1173,24 +1192,27 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
int last_errno = errno;
|
||||
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
sprintf(r->msg, "Read error in cache disk data: %s",
|
||||
strerror(last_errno));
|
||||
PT_Element_failf(r,
|
||||
"Read error in cache disk data: %s",
|
||||
strerror(last_errno));
|
||||
}
|
||||
r->adr[r->size] = '\0';
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg,
|
||||
"Read error (memory exhausted) from cache");
|
||||
strcpybuff(r->msg,
|
||||
"Read error (memory exhausted) from cache");
|
||||
}
|
||||
fclose(fp);
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
|
||||
file_convert(catbuff, sizeof(catbuff), previous_save));
|
||||
PT_Element_failf(
|
||||
r, "Read error (can't open '%s') from cache",
|
||||
file_convert(catbuff, sizeof(catbuff),
|
||||
previous_save));
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cached file name is invalid");
|
||||
strcpybuff(r->msg, "Cached file name is invalid");
|
||||
}
|
||||
}
|
||||
} else {
|
||||
@@ -1202,12 +1224,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
free(r->adr);
|
||||
r->adr = NULL;
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Read Data");
|
||||
strcpybuff(r->msg, "Cache Read Error : Read Data");
|
||||
} else
|
||||
*(r->adr + r->size) = '\0';
|
||||
} else { // erreur
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Memory Error");
|
||||
strcpybuff(r->msg, "Cache Memory Error");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1215,21 +1237,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
|
||||
} // si save==null, ne rien charger (juste en tête)
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Read Header Data");
|
||||
strcpybuff(r->msg, "Cache Read Error : Read Header Data");
|
||||
}
|
||||
unzCloseCurrentFile(index->zFile);
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Open File");
|
||||
strcpybuff(r->msg, "Cache Read Error : Open File");
|
||||
}
|
||||
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Bad Offset");
|
||||
strcpybuff(r->msg, "Cache Read Error : Bad Offset");
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "File Cache Entry Not Found");
|
||||
strcpybuff(r->msg, "File Cache Entry Not Found");
|
||||
}
|
||||
if (r->location[0] != '\0') {
|
||||
r->location = strdup(r->location);
|
||||
@@ -1522,7 +1544,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
|
||||
cache->version = (int) (firstline[8] - '0'); // cache 1.x
|
||||
if (cache->version <= 5) {
|
||||
a += cache_brstr(a, firstline, sizeof(firstline));
|
||||
strcpy(cache->lastmodified, firstline);
|
||||
strcpybuff(cache->lastmodified, firstline);
|
||||
} else {
|
||||
fclose(cache->dat);
|
||||
cache->dat = NULL;
|
||||
@@ -1539,7 +1561,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
|
||||
} else { // Vieille version du cache
|
||||
/* */
|
||||
cache->version = 0; // cache 1.0
|
||||
strcpy(cache->lastmodified, firstline);
|
||||
strcpybuff(cache->lastmodified, firstline);
|
||||
}
|
||||
|
||||
/* Create hash table for the cache (MUCH FASTER!) */
|
||||
@@ -1555,7 +1577,14 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
|
||||
a++;
|
||||
/* read "host/file" */
|
||||
a += binput(a, line, HTS_URLMAXSIZE);
|
||||
a += binput(a, line + strlen(line), HTS_URLMAXSIZE);
|
||||
{
|
||||
/* binput writes its NUL at s[max], so the second field must be
|
||||
bounded by what is left of line[], not by the same constant
|
||||
*/
|
||||
const size_t used = strlen(line);
|
||||
|
||||
a += binput(a, line + used, (int) (sizeof(line) - used - 1));
|
||||
}
|
||||
/* read position */
|
||||
a += binput(a, linepos, 200);
|
||||
sscanf(linepos, "%d", &pos);
|
||||
@@ -1570,9 +1599,11 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
|
||||
const char *scheme =
|
||||
link_has_authority(line) ? "" : "http://";
|
||||
|
||||
firstSeen = 1;
|
||||
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
|
||||
scheme, line);
|
||||
/* dropped rather than truncated; try the next entry */
|
||||
if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
|
||||
firstSeen = 1;
|
||||
else
|
||||
index->startUrl[0] = '\0';
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1650,7 +1681,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
|
||||
previous_save[0] = previous_save_[0] = '\0';
|
||||
memset(r, 0, sizeof(_PT_Element));
|
||||
r->location = location_default;
|
||||
strcpy(r->location, "");
|
||||
r->location[0] = '\0';
|
||||
if (strncmp(url, "http://", 7) == 0)
|
||||
url += 7;
|
||||
hash_pos_return = coucal_read(cache->hash, url, &hash_pos);
|
||||
@@ -1757,7 +1788,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
|
||||
int pathLen = (int) strlen(index->path);
|
||||
|
||||
if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) { // old (<3.40) buggy format
|
||||
strcpy(previous_save, previous_save_);
|
||||
strcpybuff(previous_save, previous_save_);
|
||||
}
|
||||
// relative ? (hack)
|
||||
else if (index->safeCache || (previous_save_[0] != '/' // /home/foo/bar.gif
|
||||
@@ -1793,8 +1824,9 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
|
||||
snprintf(previous_save, sizeof(previous_save), "%s%s",
|
||||
index->path, previous_save_ + index->fixedPath);
|
||||
} else {
|
||||
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
|
||||
previous_save_, (int) index->fixedPath);
|
||||
PT_Element_failf(r,
|
||||
"Bogus fixePath prefix for %s (prefixLen=%d)",
|
||||
previous_save_, (int) index->fixedPath);
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
}
|
||||
} else {
|
||||
@@ -1820,17 +1852,18 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
|
||||
if (r->adr != NULL) {
|
||||
if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Read error in cache disk data");
|
||||
strcpybuff(r->msg, "Read error in cache disk data");
|
||||
}
|
||||
r->adr[r->size] = '\0';
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Read error (memory exhausted) from cache");
|
||||
strcpybuff(r->msg,
|
||||
"Read error (memory exhausted) from cache");
|
||||
}
|
||||
fclose(fp);
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Previous cache file not found (2)");
|
||||
strcpybuff(r->msg, "Previous cache file not found (2)");
|
||||
}
|
||||
}
|
||||
} else {
|
||||
@@ -1842,30 +1875,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
|
||||
free(r->adr);
|
||||
r->adr = NULL;
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Read Data");
|
||||
strcpybuff(r->msg, "Cache Read Error : Read Data");
|
||||
} else
|
||||
r->adr[r->size] = '\0';
|
||||
} else { // erreur
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Memory Error");
|
||||
strcpybuff(r->msg, "Cache Memory Error");
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Bad Data");
|
||||
strcpybuff(r->msg, "Cache Read Error : Bad Data");
|
||||
}
|
||||
} else { // erreur
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Read Header");
|
||||
strcpybuff(r->msg, "Cache Read Error : Read Header");
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Seek Failed");
|
||||
strcpybuff(r->msg, "Cache Read Error : Seek Failed");
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "File Cache Entry Not Found");
|
||||
strcpybuff(r->msg, "File Cache Entry Not Found");
|
||||
}
|
||||
if (r->location[0] != '\0') {
|
||||
r->location = strdup(r->location);
|
||||
@@ -2128,12 +2161,14 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
strcpy(refvalue, value); \
|
||||
line[0] = '\0'; \
|
||||
} \
|
||||
} while(0)
|
||||
/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
|
||||
#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
|
||||
do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
(void) strclipbuff(refvalue, refvalue_size, value); \
|
||||
line[0] = '\0'; \
|
||||
} \
|
||||
} while (0)
|
||||
#define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
|
||||
if (line[0] != '\0' && strfield2(line, refline)) { \
|
||||
int intval = 0; \
|
||||
@@ -2169,7 +2204,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
|
||||
location_default[0] = '\0';
|
||||
memset(r, 0, sizeof(_PT_Element));
|
||||
r->location = location_default;
|
||||
strcpy(r->location, "");
|
||||
r->location[0] = '\0';
|
||||
if (strncmp(url, "http://", 7) == 0)
|
||||
url += 7;
|
||||
hash_pos_return = coucal_read(index->hash, url, &hash_pos);
|
||||
@@ -2202,11 +2237,16 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
|
||||
*value = '\0';
|
||||
for(value++; *value == ' ' || *value == '\t'; value++) ;
|
||||
HTTP_READFIELD_INT(line, value, "Content-Length", r->size); // size
|
||||
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
|
||||
HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
|
||||
HTTP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
|
||||
HTTP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
|
||||
HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo); // Content-disposition
|
||||
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
|
||||
sizeof(r->contenttype));
|
||||
HTTP_READFIELD_STRING(line, value, "Last-Modified",
|
||||
r->lastmodified, sizeof(r->lastmodified));
|
||||
HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
|
||||
sizeof(r->etag));
|
||||
HTTP_READFIELD_STRING(line, value, "Location", r->location,
|
||||
sizeof(location_default));
|
||||
HTTP_READFIELD_STRING(line, value, "Content-Disposition",
|
||||
r->cdispo, sizeof(r->cdispo));
|
||||
if (line[0] != '\0') {
|
||||
int len = r->headers ? ((int) strlen(r->headers)) : 0;
|
||||
int nlen =
|
||||
@@ -2246,7 +2286,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
|
||||
fetchSize = dataLength - metaSize;
|
||||
} else if (fetchSize > dataLength - metaSize) {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Truncated Data");
|
||||
strcpybuff(r->msg, "Cache Read Error : Truncated Data");
|
||||
}
|
||||
r->size = 0;
|
||||
if (r->statuscode != STATUSCODE_INVALID) {
|
||||
@@ -2259,12 +2299,13 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
|
||||
int last_errno = errno;
|
||||
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
sprintf(r->msg, "Read error in cache disk data: %s",
|
||||
strerror(last_errno));
|
||||
PT_Element_failf(r, "Read error in cache disk data: %s",
|
||||
strerror(last_errno));
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Read error (memory exhausted) from cache");
|
||||
strcpybuff(r->msg,
|
||||
"Read error (memory exhausted) from cache");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2272,21 +2313,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
|
||||
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Read Header Error");
|
||||
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
|
||||
}
|
||||
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Read Header Error");
|
||||
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
|
||||
}
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "Cache Read Error : Seek Error");
|
||||
strcpybuff(r->msg, "Cache Read Error : Seek Error");
|
||||
}
|
||||
|
||||
} else {
|
||||
r->statuscode = STATUSCODE_INVALID;
|
||||
strcpy(r->msg, "File Cache Entry Not Found");
|
||||
strcpybuff(r->msg, "File Cache Entry Not Found");
|
||||
}
|
||||
if (r->location[0] != '\0') {
|
||||
r->location = strdup(r->location);
|
||||
@@ -2335,8 +2376,18 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
|
||||
PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
|
||||
FILE *const fp = st->fp;
|
||||
struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
|
||||
struct tm unknown_date;
|
||||
int size_headers;
|
||||
|
||||
/* a cached entry with no parseable Last-Modified must not take the writer
|
||||
down; the epoch is the conventional "date unknown" */
|
||||
if (tm == NULL) {
|
||||
memset(&unknown_date, 0, sizeof(unknown_date));
|
||||
unknown_date.tm_year = 70;
|
||||
unknown_date.tm_mday = 1;
|
||||
tm = &unknown_date;
|
||||
}
|
||||
|
||||
sprintf(st->headers,
|
||||
"HTTP/1.0 %d %s"
|
||||
"\r\n"
|
||||
@@ -2372,7 +2423,7 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
|
||||
if (element->adr != NULL) {
|
||||
domd5mem(element->adr, element->size, st->md5, 1);
|
||||
} else {
|
||||
strcpy(st->md5, "-");
|
||||
strcpybuff(st->md5, "-");
|
||||
}
|
||||
fprintf(fp,
|
||||
/* nl */
|
||||
|
||||
@@ -97,6 +97,7 @@
|
||||
<ItemGroup>
|
||||
<ClCompile Include="htsserver.c" />
|
||||
<ClCompile Include="htsweb.c" />
|
||||
<ClCompile Include="htscmdline.c" />
|
||||
<ClCompile Include="htsurlport.c" />
|
||||
</ItemGroup>
|
||||
|
||||
|
||||
8
tests/01_engine-cmdline-split.test
Executable file
8
tests/01_engine-cmdline-split.test
Executable file
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# webhttrack posts its command line as one string: the argv split must grow past
|
||||
# 1024 arguments and keep a quote inside a value out of the option parser.
|
||||
httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"
|
||||
@@ -146,11 +146,13 @@ sid=$(scrape_sid "${port}")
|
||||
resp=$(post_redirect '/foo
|
||||
X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
|
||||
stop
|
||||
printf '%s' "${resp}" | grep -qi 'X-Injected' &&
|
||||
# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
|
||||
# pipefail then turns the hit into a silent pass.
|
||||
grep -qi 'X-Injected' <<<"${resp}" &&
|
||||
fail "CRLF in the redirect value reached the response"
|
||||
test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
|
||||
test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
|
||||
fail "CRLF redirect still emitted a Location header"
|
||||
test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
|
||||
test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
|
||||
fail "CRLF redirect did not yield exactly one status line"
|
||||
|
||||
# Loopback unless asked otherwise. Assert the socket, not the announcement.
|
||||
|
||||
@@ -103,9 +103,10 @@ sid=$(scrape_sid "${port}")
|
||||
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
|
||||
|
||||
# Accept: the legitimate flow still works. "redirect" is the cheapest field
|
||||
# with a reply that is visible in the headers.
|
||||
# with a reply that is visible in the headers. Matched from a here-string, not a
|
||||
# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
|
||||
resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
|
||||
printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
|
||||
grep -q '^Location: /accepted' <<<"${resp}" ||
|
||||
fail "a body carrying the correct sid was refused"
|
||||
|
||||
# Refuse: missing and wrong. A missing one is the regression under test — the
|
||||
@@ -114,12 +115,12 @@ printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
|
||||
for bad in "redirect=/nosid" "sid=&redirect=/empty" \
|
||||
"sid=00000000000000000000000000000000&redirect=/wrong"; do
|
||||
resp=$(request "${port}" "${bad}")
|
||||
printf '%s' "${resp}" | grep -q '^Location:' &&
|
||||
grep -q '^Location:' <<<"${resp}" &&
|
||||
fail "body accepted without a valid sid: ${bad}"
|
||||
# A refusal has to be a well-formed reply, not a headerless fragment: the
|
||||
# release build used to emit only Content-length, which reads as a protocol
|
||||
# error to any client and hides the reason.
|
||||
printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
|
||||
grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
|
||||
fail "refusal was not a 403: ${bad}"
|
||||
done
|
||||
|
||||
@@ -127,16 +128,23 @@ done
|
||||
# applied to one global key store that later requests render from, and the
|
||||
# command dispatcher reads it from there. Probe the store itself — step3
|
||||
# interpolates ${projname} into its title — rather than the refused reply.
|
||||
title() { request "$1" "" step3; }
|
||||
store_page() {
|
||||
# a dead probe or a non-page reply must not read as "the store was not written"
|
||||
page=$(request "$1" "" step3) || fail "the key store probe failed"
|
||||
grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
|
||||
fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
|
||||
}
|
||||
|
||||
request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
|
||||
title "${port}" | grep -q 'UNAUTHWRITE' &&
|
||||
store_page "${port}"
|
||||
grep -q 'UNAUTHWRITE' <<<"${page}" &&
|
||||
fail "a body without a valid sid was written to the key store"
|
||||
|
||||
# Paired accept case: without it the assertion above passes even if the server
|
||||
# simply ignores every body, which would prove nothing.
|
||||
request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
|
||||
title "${port}" | grep -q 'AUTHWRITE' ||
|
||||
store_page "${port}"
|
||||
grep -q 'AUTHWRITE' <<<"${page}" ||
|
||||
fail "a body carrying the correct sid was not written to the key store"
|
||||
|
||||
echo "PASS"
|
||||
|
||||
142
tests/81_webhttrack-maxsize.test
Normal file
142
tests/81_webhttrack-maxsize.test
Normal file
@@ -0,0 +1,142 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# The wizard's size fields must render as distinct caps: sizemax is -M, othermax
|
||||
# then maxhtml share -m.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "${testdir}/testlib.sh"
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
python=$(find_python) || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_maxsize.XXXXXX") || fail "no tmpdir"
|
||||
srvlog=$(mktemp)
|
||||
srv=
|
||||
cleanup() {
|
||||
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
|
||||
srv=
|
||||
rm -rf "${work}" "${srvlog}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
|
||||
# ~/.httrack.ini out of it.
|
||||
sport=$("${python}" -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()')
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
export HOME="${work}"
|
||||
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
|
||||
) &
|
||||
srv=$!
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
|
||||
kill -0 "${srv}" 2>/dev/null || break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
|
||||
|
||||
# Audit what step4.html renders back; without command_do nothing is launched.
|
||||
"${python}" - "${url}" <<'PY' || fail "wizard rendered the wrong options (see above)"
|
||||
import re, sys, urllib.parse, urllib.request
|
||||
|
||||
url = sys.argv[1]
|
||||
html, other, site = "111111", "222222", "333333"
|
||||
|
||||
|
||||
def post(fields):
|
||||
# The body is refused without the session id the server puts in each form.
|
||||
form = urllib.request.urlopen(url + "server/index.html", timeout=20).read()
|
||||
m = re.search(rb'name="sid" value="([0-9a-f]+)"', form)
|
||||
if not m:
|
||||
raise SystemExit("no session id in server/index.html")
|
||||
fields = [("sid", m.group(1).decode())] + fields
|
||||
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in fields)
|
||||
req = urllib.request.Request(url + "server/step4.html",
|
||||
data=body.encode("latin-1"), method="POST")
|
||||
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
|
||||
|
||||
|
||||
def textarea(page, name):
|
||||
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
|
||||
if m is None:
|
||||
sys.exit("no %s textarea in the rendered step4.html" % name)
|
||||
return m.group(1)
|
||||
|
||||
|
||||
def want(ok, msg, ctx):
|
||||
if not ok:
|
||||
sys.exit("%s\nrendered:%s" % (msg, ctx))
|
||||
|
||||
|
||||
page = post([("maxhtml", html), ("othermax", other), ("sizemax", site),
|
||||
("dos", "2")])
|
||||
cmd = textarea(page, "command")
|
||||
want("--max-size=" + site in cmd, "site cap not emitted as --max-size", cmd)
|
||||
want("--max-files=" + site not in cmd, "site cap still an --max-files", cmd)
|
||||
# Positive controls: both per-file caps still ride -m, one option each.
|
||||
want("--max-files=" + other in cmd, "non-HTML cap not emitted as --max-files",
|
||||
cmd)
|
||||
want("--max-files=," + html in cmd, "HTML cap not emitted as --max-files=,",
|
||||
cmd)
|
||||
want(cmd.count("--max-files=") == 2, "unexpected --max-files count", cmd)
|
||||
want(cmd.index("--max-files=" + other) < cmd.index("--max-files=," + html),
|
||||
"the bare --max-files must precede the --max-files=, form", cmd)
|
||||
|
||||
# winprofile.ini feeds the same three caps to the Windows GUI, one key each.
|
||||
ini = textarea(page, "winprofile").replace("\r\n", "\n") # the ini is CRLF
|
||||
want("\nDos=2" in ini, "Dos not written from the dos field", ini)
|
||||
want("\nMaxHtml=" + html + "\n" in ini, "MaxHtml not written from maxhtml", ini)
|
||||
want("\nMaxOther=" + other + "\n" in ini, "MaxOther not written from othermax",
|
||||
ini)
|
||||
want("\nMaxAll=" + site + "\n" in ini, "MaxAll not written from sizemax", ini)
|
||||
|
||||
# A bare --max-files= (an unguarded empty field) parses as -m with no digits,
|
||||
# silently clearing the html cap.
|
||||
page = post([("maxhtml", ""), ("othermax", ""), ("sizemax", ""), ("dos", "2")])
|
||||
cmd = textarea(page, "command")
|
||||
want("--max-rate=" in cmd, "empty size fields rendered no command line", cmd)
|
||||
want("--max-files" not in cmd and "--max-size" not in cmd,
|
||||
"empty size fields still rendered a size option", cmd)
|
||||
|
||||
# A mistyped ${LANG_OK] key renders the OK button's label empty.
|
||||
opts = urllib.request.urlopen(url + "server/option2b.html",
|
||||
timeout=20).read().decode("latin-1")
|
||||
want("LANG_OK" not in opts, "unexpanded LANG_OK in option2b.html", "")
|
||||
want('<input type="submit" value="OK"' in opts, "no OK label in option2b.html",
|
||||
"")
|
||||
PY
|
||||
|
||||
cleanup
|
||||
|
||||
# A bare -m<n> resets the html limit, so only the bare-then-comma order keeps
|
||||
# both caps live. basic.html is 487 bytes, well past the 10-byte html cap.
|
||||
crawl() {
|
||||
bash "${distdir}/tests/local-crawl.sh" "$@"
|
||||
}
|
||||
crawl --errors 1 --log-found 'File too big' \
|
||||
httrack 'BASEURL/simple/basic.html' '--max-files=,10'
|
||||
crawl --errors 1 --log-found 'File too big' \
|
||||
httrack 'BASEURL/simple/basic.html' '--max-files=500000' '--max-files=,10'
|
||||
crawl --errors 0 --found 'simple/basic.html' --log-not-found 'File too big' \
|
||||
httrack 'BASEURL/simple/basic.html' '--max-files=,10' '--max-files=500000'
|
||||
|
||||
echo "PASS"
|
||||
135
tests/82_webhttrack-browse-links.test
Normal file
135
tests/82_webhttrack-browse-links.test
Normal file
@@ -0,0 +1,135 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# A browser will not follow an http: page to a file: URL, so the GUI's "browse
|
||||
# the mirror" link has to go through the server's own /website/ route.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "${testdir}/testlib.sh"
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
python=$(find_python) || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
# Control: on a path that does not resolve, grep "passes" without reading anything.
|
||||
test -f "${distdir}/html/server/finished.html" || fail "no GUI pages under ${distdir}"
|
||||
bad=$(grep -l 'file://' "${distdir}"/html/server/finished.html || true)
|
||||
test -z "${bad}" || fail "file: link left in: ${bad}"
|
||||
|
||||
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_browse.XXXXXX") || fail "no tmpdir"
|
||||
srvlog=$(mktemp)
|
||||
srv=
|
||||
srvpid=
|
||||
cleanup() {
|
||||
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
|
||||
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
|
||||
rm -rf "${work}" "${srvlog}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
# Stand in for a finished mirror: the crawl itself is not under test.
|
||||
proj="${work}/websites/proj"
|
||||
mkdir -p "${proj}"
|
||||
printf 'MIRROR-PROBE-OK\n' >"${proj}/probe.txt"
|
||||
printf '<html><body>MIRROR-INDEX-OK</body></html>\n' >"${proj}/index.html"
|
||||
|
||||
# An isolated HOME keeps a stray ~/.httrack.ini out of the server's settings.
|
||||
sport=$("${python}" -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()')
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
export HOME="${work}"
|
||||
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
|
||||
) &
|
||||
srv=$!
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
|
||||
kill -0 "${srv}" 2>/dev/null || break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
|
||||
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
|
||||
|
||||
# htsserver resolves the posted paths itself, so hand it native ones.
|
||||
"${python}" - "${url}" "$(nativepath "${work}/websites")" <<'PY' || fail "browse-link checks failed"
|
||||
import re, sys, urllib.error, urllib.parse, urllib.request
|
||||
|
||||
url, base = sys.argv[1].rstrip("/"), sys.argv[2]
|
||||
rc = 0
|
||||
|
||||
|
||||
def check(ok, what):
|
||||
global rc
|
||||
print(("ok: " if ok else "FAIL: ") + what)
|
||||
if not ok:
|
||||
rc = 1
|
||||
|
||||
|
||||
def get(path):
|
||||
# The UI is served ISO-8859-1, so stay in bytes.
|
||||
return urllib.request.urlopen(url + path, timeout=20).read()
|
||||
|
||||
|
||||
# No crawl has run, so no commandRunning/commandEnd override swaps the page out.
|
||||
finished = get("/server/finished.html").decode("latin-1")
|
||||
# Control: an empty or unrendered page would pass "no file:" on its own.
|
||||
check("HTTrack Website Copier" in finished, "finished.html rendered")
|
||||
check("file://" not in finished, "finished.html has no file: link")
|
||||
|
||||
# Control: /website/ 404s until a project is set, so the fetches below are real.
|
||||
try:
|
||||
get("/website/probe.txt")
|
||||
check(False, "/website/ served with no project set")
|
||||
except urllib.error.HTTPError as e:
|
||||
check(e.code == 404, "/website/ is bound to a project (got %d)" % e.code)
|
||||
|
||||
# Point the server at the mirror the way step4.html does; a body without the
|
||||
# session id is refused outright. Only a profile save records the served root,
|
||||
# so post one; command_do=save runs no crawl.
|
||||
sid = re.search(r'name="sid" value="([0-9a-f]+)"', finished)
|
||||
if sid is None:
|
||||
print("FAIL: no sid in the rendered form")
|
||||
sys.exit(1)
|
||||
fields = [("sid", sid.group(1)), ("path", base), ("projname", "proj"),
|
||||
("command", "httrack"), ("command_do", "save"), ("winprofile", "x")]
|
||||
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="")) for k, v in fields)
|
||||
urllib.request.urlopen(urllib.request.Request(
|
||||
url + "/server/step4.html", data=body.encode("latin-1"), method="POST"),
|
||||
timeout=20).read()
|
||||
|
||||
check(get("/website/probe.txt") == b"MIRROR-PROBE-OK\n", "mirror file over http")
|
||||
check("MIRROR-INDEX-OK" in get("/website/index.html").decode("latin-1"),
|
||||
"mirror index over http")
|
||||
|
||||
body = get("/server/finished.html").decode("latin-1")
|
||||
# Match the anchor by its mirror-path label: the list below it links /website/
|
||||
# too, so a bare "is the route mentioned" check passes on the unfixed page.
|
||||
check(re.search(r'href="/website/index\.html"[^>]*>\s*' +
|
||||
re.escape(base + "/proj"), body) is not None,
|
||||
"the mirror-path link points at the served mirror")
|
||||
check("file://" not in body, "finished.html has no file: link")
|
||||
|
||||
sys.exit(rc)
|
||||
PY
|
||||
|
||||
# A leaked htsserver wedges the parallel harness behind a green log.
|
||||
cleanup
|
||||
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
|
||||
|
||||
echo "PASS"
|
||||
148
tests/83_webhttrack-argescape.test
Executable file
148
tests/83_webhttrack-argescape.test
Executable file
@@ -0,0 +1,148 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# The wizard renders its httrack command line as one string, which the engine
|
||||
# splits back into argv: a double quote in a field must reach the split escaped,
|
||||
# or the rest of the value is parsed as fresh options (-V runs a shell command).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
command -v python3 >/dev/null || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
srv=
|
||||
log=$(mktemp)
|
||||
cleanup() {
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
rm -f "${log}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
freeport() {
|
||||
python3 -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()'
|
||||
}
|
||||
|
||||
# Echo the announced URL. Runs in a command substitution, so it is a
|
||||
# subshell and cannot export the pid: the caller reads it back with srvpid.
|
||||
start() {
|
||||
local port url
|
||||
port=$(freeport)
|
||||
: >"${log}"
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
|
||||
) &
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
|
||||
echo "${url}"
|
||||
}
|
||||
|
||||
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
|
||||
|
||||
portof() { echo "${1##*:}" | tr -d /; }
|
||||
|
||||
# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
|
||||
# default index). Prints the reply.
|
||||
request() {
|
||||
python3 -c 'import socket, sys
|
||||
port, body = int(sys.argv[1]), sys.argv[2]
|
||||
page = sys.argv[3] if len(sys.argv) > 3 else "index"
|
||||
if body:
|
||||
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
|
||||
"Content-type: application/x-www-form-urlencoded\r\n"
|
||||
"Content-length: %d\r\n\r\n%s" % (len(body), body))
|
||||
else:
|
||||
req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
|
||||
s = socket.create_connection(("127.0.0.1", port), 10)
|
||||
s.settimeout(30)
|
||||
s.sendall(req.encode())
|
||||
out = b""
|
||||
while True:
|
||||
b = s.recv(65536)
|
||||
if not b:
|
||||
break
|
||||
out += b
|
||||
s.close()
|
||||
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
|
||||
}
|
||||
|
||||
scrape_sid() {
|
||||
request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
|
||||
}
|
||||
|
||||
# urlencode the key=value pairs given as arguments
|
||||
formencode() {
|
||||
python3 -c 'import sys, urllib.parse
|
||||
print(urllib.parse.urlencode([tuple(a.split("=", 1)) for a in sys.argv[1:]]))' "$@"
|
||||
}
|
||||
|
||||
url=$(start)
|
||||
port=$(portof "${url}")
|
||||
srv=$(srvpid)
|
||||
test -n "${srv}" || fail "htsserver did not report its pid"
|
||||
|
||||
sid=$(scrape_sid "${port}")
|
||||
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
|
||||
|
||||
# Fill the wizard fields the command line quotes, then read back the generated
|
||||
# command line: the user-agent carries a break-out attempt, the footer a
|
||||
# backslash (which must survive the escape round trip), the project name a plain
|
||||
# value.
|
||||
body=$(formencode "sid=${sid}" 'user=Moz" -V "touch /tmp/pwn' 'footer=a\b"c' \
|
||||
"path=/tmp/p" "projname=plain proj" 'urls=http://x/a"b' 'url2=+*.png"')
|
||||
request "${port}" "${body}" >/dev/null
|
||||
cmdline=$(request "${port}" "" step4 |
|
||||
sed -n '/<textarea name="command"/,/<\/textarea>/p')
|
||||
|
||||
# Control: without the fields in the page every assertion below is vacuous.
|
||||
grep -q -- '--user-agent' <<<"${cmdline}" ||
|
||||
fail "no --user-agent in the generated command line (probe blind)"
|
||||
|
||||
# A plain value is passed through untouched: the escaping must not mangle the
|
||||
# ordinary case.
|
||||
grep -qF -- '--path "/tmp/p/plain proj"' <<<"${cmdline}" ||
|
||||
fail "a plain quoted value was not passed through: ${cmdline}"
|
||||
|
||||
# The quote is escaped, so the split keeps it inside the value...
|
||||
grep -qF -- '--user-agent "Moz\" -V \"touch /tmp/pwn"' <<<"${cmdline}" ||
|
||||
fail "the quote in the user-agent was not escaped: ${cmdline}"
|
||||
|
||||
# ...and the raw form, which would end the argument and hand -V to the option
|
||||
# parser, is gone.
|
||||
grep -qF -- '--user-agent "Moz" -V "touch' <<<"${cmdline}" &&
|
||||
fail "the user-agent still closes its argument early: ${cmdline}"
|
||||
|
||||
# A backslash is escaped too, or the split would eat it along with the quote
|
||||
# that follows.
|
||||
grep -qF -- '--footer "a\\b\"c"' <<<"${cmdline}" ||
|
||||
fail "the backslash in the footer was not escaped: ${cmdline}"
|
||||
|
||||
# The url and filter fields sit outside quotes, where a backslash cannot escape
|
||||
# anything: one quote there flips the parity of every quote after it, so the
|
||||
# escaping above would protect nothing. They must not emit a raw quote at all.
|
||||
grep -qF -- 'http://x/a%22b' <<<"${cmdline}" ||
|
||||
fail "the quote in the url field was not neutralised: ${cmdline}"
|
||||
grep -qF -- '+*.png%22' <<<"${cmdline}" ||
|
||||
fail "the quote in the filter field was not neutralised: ${cmdline}"
|
||||
grep -qF -- 'http://x/a"b' <<<"${cmdline}" &&
|
||||
fail "the url field still emits a raw quote: ${cmdline}"
|
||||
|
||||
echo "PASS"
|
||||
167
tests/84_webhttrack-mirror-verbatim.test
Normal file
167
tests/84_webhttrack-mirror-verbatim.test
Normal file
@@ -0,0 +1,167 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# htsserver serves the crawled mirror under /website/ alongside its own GUI:
|
||||
# mirrored pages must skip the ${...} expander, or ${_sid} leaks the session id.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
command -v python3 >/dev/null || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
log=$(mktemp)
|
||||
work=$(mktemp -d)
|
||||
csrv=
|
||||
# start() runs in a command substitution, so its $! never reaches this shell. A
|
||||
# missed kill leaves an orphan holding the CI job open long after a green suite.
|
||||
srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
|
||||
cleanup() {
|
||||
local pid
|
||||
pid=$(srvpid)
|
||||
test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
|
||||
test -z "${csrv}" || kill -9 "${csrv}" 2>/dev/null || true
|
||||
wait "${csrv}" 2>/dev/null || true # absorb bash's async "Killed" notice
|
||||
rm -rf "${log}" "${work}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
freeport() {
|
||||
python3 -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()'
|
||||
}
|
||||
|
||||
# Echo the announced URL.
|
||||
start() {
|
||||
local port url
|
||||
port=$(freeport)
|
||||
: >"${log}"
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
|
||||
) &
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
|
||||
echo "${url}"
|
||||
}
|
||||
|
||||
portof() { echo "${1##*:}" | tr -d /; }
|
||||
|
||||
# GET $2 from 127.0.0.1:$1, headers into $3 and the body, byte for byte, into $4.
|
||||
fetch() {
|
||||
python3 -c 'import socket, sys
|
||||
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
|
||||
s.settimeout(20)
|
||||
s.sendall(("GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % sys.argv[2]).encode())
|
||||
out = b""
|
||||
while True:
|
||||
b = s.recv(65536)
|
||||
if not b:
|
||||
break
|
||||
out += b
|
||||
s.close()
|
||||
head, _, body = out.partition(b"\r\n\r\n")
|
||||
open(sys.argv[3], "wb").write(head)
|
||||
open(sys.argv[4], "wb").write(body)' "$1" "$2" "$3" "$4"
|
||||
}
|
||||
|
||||
# POST the remaining args as urlencoded key=value fields to 127.0.0.1:$1.
|
||||
post() {
|
||||
local port=$1
|
||||
shift
|
||||
python3 -c 'import socket, sys, urllib.parse
|
||||
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe=""))
|
||||
for k, v in (a.split("=", 1) for a in sys.argv[2:])).encode()
|
||||
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
|
||||
s.settimeout(30)
|
||||
s.sendall(b"POST /step4.html HTTP/1.0\r\nHost: 127.0.0.1\r\n"
|
||||
b"Content-type: application/x-www-form-urlencoded\r\n"
|
||||
b"Content-length: %d\r\n\r\n" % len(body) + body)
|
||||
while s.recv(65536):
|
||||
pass
|
||||
s.close()' "${port}" "$@" >/dev/null
|
||||
}
|
||||
|
||||
# LF-only, and a line ending in a backslash: the expander rewrites both, so a
|
||||
# mangled reply fails the byte comparison even where no directive is present.
|
||||
mirror="${work}/proj/hostile.html"
|
||||
hdr="${work}/hdr"
|
||||
body="${work}/body"
|
||||
|
||||
# The server merges $HOME/.httrack.ini into the same store on the first request;
|
||||
# point it somewhere empty so a developer's own file cannot shadow the fields.
|
||||
export HOME="${work}"
|
||||
|
||||
url=$(start)
|
||||
port=$(portof "${url}")
|
||||
|
||||
# Positive control: the GUI's own templates must still expand. This is also
|
||||
# where the real token comes from, so its absence below can be asserted.
|
||||
fetch "${port}" /server/index.html "${hdr}" "${body}"
|
||||
sid=$(sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' "${body}" | head -1)
|
||||
test "${#sid}" -eq 32 || fail "GUI page did not expand \${sid} (got '${sid}')"
|
||||
|
||||
# /website/ serves only the root the server itself recorded, so save a profile
|
||||
# (no command_do=start, so nothing crawls) to create it, then plant the file.
|
||||
post "${port}" "sid=${sid}" command=httrack command_do=save winprofile=x \
|
||||
"path=${work}" projname=proj
|
||||
test -f "${work}/proj/hts-cache/winprofile.ini" ||
|
||||
fail "profile save did not create the project: $(cat "${log}")"
|
||||
mkdir -p "$(dirname "${mirror}")"
|
||||
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
|
||||
printf 'Hostile mirrored page.\nsid=${_sid} copy=${sid}\ntrailing backslash: \\\n' \
|
||||
>"${mirror}"
|
||||
|
||||
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
|
||||
grep -q '^HTTP/1\.0 200 ' "${hdr}" || fail "mirrored page not served: $(head -1 "${hdr}")"
|
||||
grep -qF "${sid}" "${body}" &&
|
||||
fail "the session id was expanded into mirrored content"
|
||||
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
|
||||
grep -qF '${_sid}' "${body}" || fail "\${_sid} did not survive verbatim"
|
||||
# shellcheck disable=SC2016
|
||||
grep -qF '${sid}' "${body}" || fail "\${sid} did not survive verbatim"
|
||||
cmp -s "${mirror}" "${body}" || fail "mirrored file not served byte for byte"
|
||||
# The mirror stays browsable: verbatim must not mean served as a download.
|
||||
grep -qi '^Content-type: text/html' "${hdr}" ||
|
||||
fail "mirrored page lost its text/html type: $(cat "${hdr}")"
|
||||
|
||||
# The other direction: while a crawl runs, every .html request is overridden to
|
||||
# the GUI's own refresh page, so a /website/ URL stops naming mirrored content.
|
||||
# /trickle/ dribbles for a minute, which holds the crawl open for the probe.
|
||||
clog="${work}/content.log"
|
||||
python3 "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
|
||||
csrv=$!
|
||||
for _ in $(seq 1 40); do
|
||||
cport=$(sed -n 's/^PORT //p' "${clog}") && test -n "${cport}" && break
|
||||
kill -0 "${csrv}" 2>/dev/null || break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${cport:-}" || fail "content server did not come up: $(cat "${clog}")"
|
||||
|
||||
post "${port}" "sid=${sid}" "path=${work}" projname=crawl winprofile=x \
|
||||
command_do=start \
|
||||
"command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/trickle/ -O ${work}/crawl"
|
||||
|
||||
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
|
||||
grep -q '^HTTP/1\.0 200 ' "${hdr}" ||
|
||||
fail "running crawl: /website/ was not overridden to the GUI page: $(head -1 "${hdr}")"
|
||||
grep -qF "'crawl' - HTTrack Website Copier" "${body}" ||
|
||||
fail "the overridden GUI page was not the expanded refresh page"
|
||||
|
||||
echo "PASS"
|
||||
183
tests/85_webhttrack-projpath.test
Normal file
183
tests/85_webhttrack-projpath.test
Normal file
@@ -0,0 +1,183 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# /website/ is served from the project directory htsserver set up itself, never
|
||||
# from a root the request body names, and composing that path must stay bounded.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
command -v python3 >/dev/null || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
srv=
|
||||
log=$(mktemp)
|
||||
base=$(mktemp -d)
|
||||
cleanup() {
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
rm -f "${log}"
|
||||
rm -rf "${base}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
freeport() {
|
||||
python3 -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()'
|
||||
}
|
||||
|
||||
# Echo the announced URL. Runs in a command substitution, so it is a
|
||||
# subshell and cannot export the pid: the caller reads it back with srvpid.
|
||||
start() {
|
||||
local port url
|
||||
port=$(freeport)
|
||||
: >"${log}"
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
|
||||
) &
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
|
||||
echo "${url}"
|
||||
}
|
||||
|
||||
# The server reports its own pid; the aliveness assertion below hangs off it.
|
||||
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
|
||||
|
||||
alive() { kill -0 "$1" 2>/dev/null; }
|
||||
|
||||
portof() { echo "${1##*:}" | tr -d /; }
|
||||
|
||||
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
|
||||
# $2 is empty. Prints the reply.
|
||||
request() {
|
||||
python3 -c 'import socket, sys
|
||||
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
||||
if path:
|
||||
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
|
||||
else:
|
||||
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
|
||||
"Content-type: application/x-www-form-urlencoded\r\n"
|
||||
"Content-length: %d\r\n\r\n%s" % (len(body), body))
|
||||
s = socket.create_connection(("127.0.0.1", port), 10)
|
||||
s.settimeout(30)
|
||||
s.sendall(req.encode())
|
||||
out = b""
|
||||
while True:
|
||||
b = s.recv(65536)
|
||||
if not b:
|
||||
break
|
||||
out += b
|
||||
s.close()
|
||||
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
|
||||
}
|
||||
|
||||
get() { request "$1" "$2" ""; }
|
||||
post() { request "$1" "" "$2"; }
|
||||
|
||||
# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
|
||||
# reads as marker-absent, and so do a truncated body and a 302 to the file.
|
||||
reply=
|
||||
fetch() {
|
||||
reply=$(get "$1" "$2") || fail "the request for $2 failed"
|
||||
grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
|
||||
fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
|
||||
}
|
||||
|
||||
url=$(start)
|
||||
port=$(portof "${url}")
|
||||
srv=$(srvpid)
|
||||
test -n "${srv}" || fail "htsserver did not report its pid"
|
||||
|
||||
# Every request body is gated by the session id (78_webhttrack-sid.test).
|
||||
sid=$(get "${port}" /server/index.html |
|
||||
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
|
||||
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
|
||||
|
||||
# A file the mirror must never expose, next to the project that may.
|
||||
echo "SECRETMARKER" >"${base}/secret.txt"
|
||||
mkdir -p "${base}/proj"
|
||||
echo "LOGMARKER" >"${base}/proj/hts-log.txt"
|
||||
|
||||
# error_redirect is the branch that skipped the fsfile clearing, so fail the save
|
||||
# with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
|
||||
# any successful save: commandEnd then swaps the error page for the finished one.
|
||||
get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
|
||||
saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
|
||||
grep -q '^Location: /server/error.html' <<<"${saved}" ||
|
||||
fail "the refused save did not redirect to the error page"
|
||||
|
||||
# No project yet, so no root to serve from.
|
||||
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
|
||||
fetch "${port}" /website/secret.txt 404
|
||||
grep -q SECRETMARKER <<<"${reply}" &&
|
||||
fail "a posted projpath served a file outside any project"
|
||||
|
||||
# Positive control: step4's "save settings" flow registers the project without
|
||||
# crawling, and browsing its mirror keeps working. Without it the assertions
|
||||
# around it would pass on a server that never serves /website/ at all.
|
||||
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
|
||||
body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
|
||||
post "${port}" "${body}" >/dev/null
|
||||
test -f "${base}/proj/hts-cache/winprofile.ini" ||
|
||||
fail "the project was not registered: $(cat "${log}")"
|
||||
fetch "${port}" /website/hts-log.txt 200
|
||||
grep -q LOGMARKER <<<"${reply}" ||
|
||||
fail "the registered project's mirror is not served"
|
||||
|
||||
# Same request with the root repointed: the project is legitimate, projpath is
|
||||
# not what decides where the bytes come from.
|
||||
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
|
||||
fetch "${port}" /website/secret.txt 404
|
||||
grep -q SECRETMARKER <<<"${reply}" &&
|
||||
fail "a posted projpath repointed the served root"
|
||||
post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
|
||||
fetch "${port}" /website/passwd 404
|
||||
grep -q '^root:' <<<"${reply}" &&
|
||||
fail "a posted projpath read an arbitrary system file"
|
||||
|
||||
# A ".." anywhere in the recorded root would escape the mirror on every later
|
||||
# request, so the save must be refused and the previous root kept.
|
||||
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
|
||||
fetch "${port}" /website/secret.txt 404
|
||||
grep -q SECRETMARKER <<<"${reply}" &&
|
||||
fail "a '..' in the saved project path escaped the mirror root"
|
||||
fetch "${port}" /website/hts-log.txt 200
|
||||
grep -q LOGMARKER <<<"${reply}" ||
|
||||
fail "rejecting the '..' root also lost the previous one"
|
||||
|
||||
# The root is now the server's own, but it is still built from two posted
|
||||
# fields: 800-odd bytes of them used to be sprintf'd into a 1024-byte buffer.
|
||||
seg=$(printf '%0200d' 0)
|
||||
longpath="${base}/${seg}/${seg}/${seg}/${seg}"
|
||||
fspath="${longpath}/proj"
|
||||
# structcheck() refuses a root over HTS_URLMAXSIZE, so the URL carries the rest.
|
||||
test "$((${#fspath} + 11))" -le 1024 ||
|
||||
fail "the long project path (${#fspath}) would not pass structcheck"
|
||||
longurl=$(printf '%0800d' 0)
|
||||
test "$((${#fspath} + 1 + ${#longurl}))" -gt 1024 ||
|
||||
fail "the composed path (${#fspath} + ${#longurl}) would not overflow"
|
||||
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${longpath}&projname=proj" >/dev/null
|
||||
test -f "${fspath}/hts-cache/winprofile.ini" ||
|
||||
fail "the long-path project was not registered: $(cat "${log}")"
|
||||
get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
|
||||
alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
|
||||
# Not just alive: still answering.
|
||||
fetch "${port}" /server/index.html 200
|
||||
|
||||
echo "PASS"
|
||||
111
tests/86_local-proxytrack-cache-longfields.test
Executable file
111
tests/86_local-proxytrack-cache-longfields.test
Executable file
@@ -0,0 +1,111 @@
|
||||
#!/bin/bash
|
||||
# A cached header longer than proxytrack's field must be clipped to that field,
|
||||
# not overflow into the next one (contenttype[64] abuts the location pointer).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "${testdir}/testlib.sh"
|
||||
|
||||
dir=$(mktemp -d)
|
||||
trap 'rm -rf "$dir"' EXIT
|
||||
|
||||
pad() { printf "%${1}s" '' | tr ' ' "$2"; }
|
||||
|
||||
# Longest surviving run of char $2 in file $1, or 0.
|
||||
runlen() {
|
||||
grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
|
||||
}
|
||||
|
||||
# --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
|
||||
# Each header overshoots its own destination, so a per-field clip is the only
|
||||
# way to get every expected length right at once.
|
||||
{
|
||||
printf 'HTTP/1.1 200 OK\r\n'
|
||||
printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)"
|
||||
printf 'Etag: %s\r\n' "$(pad 400 E)"
|
||||
printf 'Content-Disposition: %s\r\n' "$(pad 400 D)"
|
||||
printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
|
||||
printf 'Content-Length: 5\r\n\r\n'
|
||||
} >"$dir/hdr"
|
||||
printf 'hello' >"$dir/body"
|
||||
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
|
||||
{
|
||||
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
|
||||
printf '2 0 test\n'
|
||||
printf '\n\n'
|
||||
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
|
||||
cat "$dir/hdr" "$dir/body"
|
||||
} >"$dir/in.arc"
|
||||
|
||||
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
|
||||
echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2
|
||||
exit 1
|
||||
}
|
||||
test -s "$dir/out.arc" || {
|
||||
echo "FAIL: ARC entry dropped instead of clipped" >&2
|
||||
exit 1
|
||||
}
|
||||
# Only contenttype survives into the ARC output; the over-long Etag and
|
||||
# Content-Disposition above are still load-bearing, since bounding just one
|
||||
# field leaves the others smashing the element (the run above would crash).
|
||||
got=$(runlen "$dir/out.arc" A)
|
||||
test "${got:-0}" = 54 || {
|
||||
echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# --- ZIP reader (ZIP_READFIELD_STRING) --------------------------------------
|
||||
python=$(find_python) || {
|
||||
echo "python3 missing; ARC half only" >&2
|
||||
exit 0
|
||||
}
|
||||
|
||||
"$python" - "$dir/in.zip" <<'EOF'
|
||||
import sys, zipfile
|
||||
|
||||
body = b"hello world"
|
||||
meta = (
|
||||
"X-In-Cache: 1\r\n"
|
||||
"X-StatusCode: 200\r\n"
|
||||
"X-StatusMessage: ok%s\r\n"
|
||||
"X-Size: %d\r\n"
|
||||
"Content-Type: text/html%s\r\n"
|
||||
"X-Charset: %s\r\n"
|
||||
"Etag: %s\r\n"
|
||||
"Content-Disposition: %s\r\n"
|
||||
# a parseable date is required: the ARC writer dereferences it unchecked
|
||||
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
|
||||
"X-Save: out/page.html\r\n"
|
||||
% ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400)
|
||||
).encode()
|
||||
|
||||
zi = zipfile.ZipInfo("example.com/page.html")
|
||||
zi.compress_type = zipfile.ZIP_STORED
|
||||
zi.extra = meta
|
||||
with zipfile.ZipFile(sys.argv[1], "w") as z:
|
||||
z.writestr(zi, body)
|
||||
EOF
|
||||
|
||||
proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || {
|
||||
echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2
|
||||
exit 1
|
||||
}
|
||||
test -s "$dir/out2.arc" || {
|
||||
echo "FAIL: zip entry dropped instead of clipped" >&2
|
||||
exit 1
|
||||
}
|
||||
# three destinations, two capacities: msg[1024] keeps all 400 M's where a
|
||||
# one-size-fits-all clip would cut it to 63 like the others
|
||||
while read -r ch want; do
|
||||
got=$(runlen "$dir/out2.arc" "$ch")
|
||||
test "${got:-0}" = "$want" || {
|
||||
echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2
|
||||
exit 1
|
||||
}
|
||||
done <<'EOF'
|
||||
A 54
|
||||
C 63
|
||||
M 400
|
||||
EOF
|
||||
51
tests/87_local-proxytrack-nodate.test
Executable file
51
tests/87_local-proxytrack-nodate.test
Executable file
@@ -0,0 +1,51 @@
|
||||
#!/bin/bash
|
||||
# A cached entry whose Last-Modified is missing or unparseable must still
|
||||
# convert: the ARC writer used to dereference the parsed date unchecked.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
dir=$(mktemp -d)
|
||||
trap 'rm -rf "$dir"' EXIT
|
||||
|
||||
# $1 Last-Modified value (empty = omit the header), $2 expected archive date,
|
||||
# $3 label. The date is asserted exactly: a guard that fires unconditionally
|
||||
# would clobber the valid case, and one that skips the year/day would emit a
|
||||
# month and day of 00.
|
||||
run_case() {
|
||||
lastmod=$1
|
||||
wantdate=$2
|
||||
what=$3
|
||||
|
||||
{
|
||||
printf 'HTTP/1.1 200 OK\r\n'
|
||||
printf 'Content-Type: text/html\r\n'
|
||||
test -z "$lastmod" || printf 'Last-Modified: %s\r\n' "$lastmod"
|
||||
printf 'Content-Length: 5\r\n\r\n'
|
||||
} >"$dir/hdr"
|
||||
printf 'hello' >"$dir/body"
|
||||
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
|
||||
{
|
||||
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
|
||||
printf '2 0 test\n'
|
||||
printf '\n\n'
|
||||
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
|
||||
cat "$dir/hdr" "$dir/body"
|
||||
} >"$dir/in.arc"
|
||||
|
||||
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
|
||||
echo "FAIL: proxytrack crashed on $what" >&2
|
||||
exit 1
|
||||
}
|
||||
grep -aq "^http://example.com/p.html 0.0.0.0 ${wantdate} " "$dir/out.arc" || {
|
||||
echo "FAIL: $what: expected archive date ${wantdate}, got:" >&2
|
||||
grep -a '^http://example.com/' "$dir/out.arc" >&2 || echo "(entry dropped)" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
# the epoch stands in for "date unknown"
|
||||
run_case '' 19700101000000 'a missing Last-Modified'
|
||||
run_case 'not a date at all' 19700101000000 'an unparseable Last-Modified'
|
||||
run_case '0' 19700101000000 'a bare 0 Last-Modified'
|
||||
# a parseable date must still come through untouched
|
||||
run_case 'Sun, 06 Nov 1994 08:49:37 GMT' 19941106084937 'a valid Last-Modified'
|
||||
69
tests/88_local-proxytrack-badmtime.test
Executable file
69
tests/88_local-proxytrack-badmtime.test
Executable file
@@ -0,0 +1,69 @@
|
||||
#!/bin/bash
|
||||
# A cache file whose mtime is beyond what gmtime can represent must not put a
|
||||
# day of "00" in the ARC filedesc date: PT_GetTime fell back to an all-zero tm.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "${testdir}/testlib.sh"
|
||||
|
||||
python=$(find_python) || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
dir=$(mktemp -d)
|
||||
trap 'rm -rf "$dir"' EXIT
|
||||
|
||||
rc=0
|
||||
"$python" - "$dir/in.zip" <<'EOF' || rc=$?
|
||||
import os, sys, zipfile
|
||||
|
||||
body = b"hello world"
|
||||
meta = (
|
||||
"X-In-Cache: 1\r\n"
|
||||
"X-StatusCode: 200\r\n"
|
||||
"X-StatusMessage: OK\r\n"
|
||||
"X-Size: %d\r\n"
|
||||
"Content-Type: text/html\r\n"
|
||||
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
|
||||
"X-Save: out/page.html\r\n" % len(body)
|
||||
).encode()
|
||||
|
||||
zi = zipfile.ZipInfo("example.com/page.html")
|
||||
zi.compress_type = zipfile.ZIP_STORED
|
||||
zi.extra = meta
|
||||
with zipfile.ZipFile(sys.argv[1], "w") as z:
|
||||
z.writestr(zi, body)
|
||||
|
||||
# past gmtime's range; the index timestamp is this file's mtime
|
||||
huge = 4611686018427387903
|
||||
try:
|
||||
os.utime(sys.argv[1], (huge, huge))
|
||||
except OSError:
|
||||
sys.exit(77)
|
||||
if int(os.stat(sys.argv[1]).st_mtime) < huge:
|
||||
sys.exit(77) # filesystem clamped it, nothing to test
|
||||
EOF
|
||||
test "$rc" -eq 0 || {
|
||||
test "$rc" -eq 77 || {
|
||||
echo "FAIL: could not build the cache (python exit $rc)" >&2
|
||||
exit 1
|
||||
}
|
||||
echo "filesystem will not hold an out-of-range mtime; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
proxytrack --convert "$dir/out.arc" "$dir/in.zip" >/dev/null 2>&1 || {
|
||||
echo "FAIL: proxytrack failed on a cache with an out-of-range mtime" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# field 3 of the filedesc line is YYYYMMDDhhmmss; the epoch stands in for
|
||||
# "unknown", and 14 digits with a real day is the point (00 is what broke)
|
||||
date=$(head -n1 "$dir/out.arc" | awk '{ print $3 }')
|
||||
test "$date" = 19700101000000 || {
|
||||
echo "FAIL: filedesc date is '$date', expected 19700101000000" >&2
|
||||
exit 1
|
||||
}
|
||||
174
tests/89_webhttrack-error-overflow.test
Normal file
174
tests/89_webhttrack-error-overflow.test
Normal file
@@ -0,0 +1,174 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# The "save settings" failure messages quote a project path the request body
|
||||
# supplies, so composing them must clip instead of running off a fixed buffer.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
command -v python3 >/dev/null || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
# ERROR_MESSAGE_MAX - 1 in htsserver.c.
|
||||
msgmax=1023
|
||||
|
||||
srv=
|
||||
log=$(mktemp)
|
||||
# Physical path: macOS resolves TMPDIR through /var -> private/var, and the
|
||||
# paths built below sit within a few bytes of its 1024-byte PATH_MAX.
|
||||
base=$(cd "$(mktemp -d)" && pwd -P)
|
||||
cleanup() {
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
rm -f "${log}"
|
||||
rm -rf "${base}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
freeport() {
|
||||
python3 -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()'
|
||||
}
|
||||
|
||||
# Echo the announced URL. Runs in a command substitution, so it is a
|
||||
# subshell and cannot export the pid: the caller reads it back with srvpid.
|
||||
start() {
|
||||
local port url
|
||||
port=$(freeport)
|
||||
: >"${log}"
|
||||
(
|
||||
trap '' TERM TTOU XFSZ
|
||||
# Caps regular-file writes, which is how the failing-write branch below
|
||||
# is reached without /dev/full; the announce rides a pipe, exempt.
|
||||
ulimit -f 8
|
||||
exec htsserver "${distdir}/" --port "${port}" 2>&1
|
||||
) | cat >"${log}" &
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
|
||||
echo "${url}"
|
||||
}
|
||||
|
||||
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
|
||||
|
||||
alive() { kill -0 "$1" 2>/dev/null; }
|
||||
|
||||
portof() { echo "${1##*:}" | tr -d /; }
|
||||
|
||||
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
|
||||
# $2 is empty. Prints the reply.
|
||||
request() {
|
||||
python3 -c 'import socket, sys
|
||||
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
||||
if path:
|
||||
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
|
||||
else:
|
||||
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
|
||||
"Content-type: application/x-www-form-urlencoded\r\n"
|
||||
"Content-length: %d\r\n\r\n%s" % (len(body), body))
|
||||
s = socket.create_connection(("127.0.0.1", port), 10)
|
||||
s.settimeout(30)
|
||||
s.sendall(req.encode())
|
||||
out = b""
|
||||
while True:
|
||||
b = s.recv(65536)
|
||||
if not b:
|
||||
break
|
||||
out += b
|
||||
s.close()
|
||||
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
|
||||
}
|
||||
|
||||
get() { request "$1" "$2" ""; }
|
||||
post() { request "$1" "" "$2"; }
|
||||
|
||||
# Echo a path under the root $1 exactly $2 bytes long, in NAME_MAX-sized parts.
|
||||
padpath() {
|
||||
local root=$1 want=$2 p=$1 last
|
||||
while test $((${#p} + 203)) -lt "${want}"; do
|
||||
p="${p}/$(printf '%0200d' 0)"
|
||||
done
|
||||
last=$((want - ${#p} - 1))
|
||||
test "${last}" -ge 1 || fail "cannot build a ${want}-byte path under ${root}"
|
||||
printf '%s/%0*d' "${p}" "${last}" 0
|
||||
}
|
||||
|
||||
# Fail unless the error page renders the message opening with $1 about project
|
||||
# path $2, clipped to msgmax. error.html puts ${error} alone on its own line.
|
||||
check_clipped() {
|
||||
local prefix=$1 full=$((${#1} + ${#2})) page line shown
|
||||
test "${full}" -gt "${msgmax}" ||
|
||||
fail "a ${full}-byte message fits in ${msgmax}: it cannot show a clip"
|
||||
page=$(get "${port}" /server/error.html | tr -d '\r')
|
||||
line=$(printf '%s\n' "${page}" | grep "^${prefix}") || {
|
||||
shown=$(printf '%s\n' "${page}" | grep '^Unable to ' | cut -c1-80 || true)
|
||||
fail "the error page reports '${shown:-nothing}', want: ${prefix}"
|
||||
}
|
||||
test "${#line}" -eq "${msgmax}" ||
|
||||
fail "the message rendered ${#line} bytes, want ${full} clipped to ${msgmax}"
|
||||
}
|
||||
|
||||
url=$(start)
|
||||
port=$(portof "${url}")
|
||||
srv=$(srvpid)
|
||||
test -n "${srv}" || fail "htsserver did not report its pid"
|
||||
|
||||
# Every request body is gated by the session id (78_webhttrack-sid.test).
|
||||
sid=$(get "${port}" /server/index.html |
|
||||
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
|
||||
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
|
||||
|
||||
save() {
|
||||
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=$2&path=$1&projname=$3"
|
||||
}
|
||||
|
||||
# A project path too long to be created at all.
|
||||
path=$(padpath "${base}/nodir" 1098)
|
||||
save "${path}" x p >/dev/null
|
||||
alive "${srv}" ||
|
||||
fail "an over-long project path crashed the server: $(cat "${log}")"
|
||||
check_clipped "Unable to create the directory structure in " "${path}/p"
|
||||
|
||||
# A creatable project path whose hts-cache is not a directory, so the init file
|
||||
# under it cannot be opened.
|
||||
path=$(padpath "${base}/isdir" 993)
|
||||
mkdir -p "${path}/p"
|
||||
ln -s /dev/null "${path}/p/hts-cache"
|
||||
save "${path}" x p >/dev/null
|
||||
alive "${srv}" || fail "an unwritable init file crashed the server: $(cat "${log}")"
|
||||
check_clipped "Unable to create the init file " "${path}/p"
|
||||
|
||||
# The init file opens, then the write fails: the profile is past both stdio's
|
||||
# buffer, so fwrite() reaches the descriptor, and the server's file-size limit.
|
||||
path=$(padpath "${base}/full" 993)
|
||||
mkdir -p "${path}/p/hts-cache"
|
||||
profile=$(printf '%01024d' 0)
|
||||
profile=${profile}${profile}${profile}${profile}
|
||||
profile=${profile}${profile}${profile}${profile}
|
||||
test "${#profile}" -eq 16384 || fail "built a ${#profile}-byte profile, want 16384"
|
||||
save "${path}" "${profile}" p >/dev/null
|
||||
alive "${srv}" || fail "a short init file write crashed the server: $(cat "${log}")"
|
||||
written=$(wc -c <"${path}/p/hts-cache/winprofile.ini" | tr -d ' ')
|
||||
test "${written}" -lt "${#profile}" ||
|
||||
fail "the file-size limit did not stop the write (${written} bytes landed)"
|
||||
check_clipped "Unable to write ${#profile} bytes in the the init file " "${path}/p"
|
||||
|
||||
get "${port}" /server/index.html | grep -q '200 OK' ||
|
||||
fail "the server stopped answering after the refused saves"
|
||||
|
||||
echo "PASS"
|
||||
195
tests/90_webhttrack-checkbox-clear.test
Executable file
195
tests/90_webhttrack-checkbox-clear.test
Executable file
@@ -0,0 +1,195 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# An unchecked box posts nothing and the stored "1" survives: hence a hidden
|
||||
# companion per box, plus a disabling flag for the default-on ones.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "${testdir}/testlib.sh"
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
python=$(find_python) || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
|
||||
srvlog=$(mktemp)
|
||||
srv=
|
||||
srvpid=
|
||||
cleanup() {
|
||||
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
|
||||
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
|
||||
rm -rf "${work}" "${srvlog}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
|
||||
sport=$("${python}" -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()')
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
export HOME="${work}"
|
||||
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
|
||||
) &
|
||||
srv=$!
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
|
||||
kill -0 "${srv}" 2>/dev/null || break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
|
||||
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
|
||||
|
||||
"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
|
||||
import glob, os, re, sys, urllib.parse, urllib.request
|
||||
|
||||
url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
|
||||
rc = 0
|
||||
|
||||
|
||||
def check(ok, what):
|
||||
global rc
|
||||
print(("ok: " if ok else "FAIL: ") + what)
|
||||
if not ok:
|
||||
rc = 1
|
||||
|
||||
|
||||
# cache/cache2 are left to the separate rework of the dead "Cache" seed.
|
||||
skip = {"cache", "cache2"}
|
||||
page_of = {}
|
||||
boxes = 0
|
||||
for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
|
||||
page = open(path, "rb").read().decode("latin-1")
|
||||
cleared = dict((m.group(1), m.start()) for m in
|
||||
re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
|
||||
for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
|
||||
boxes += 1
|
||||
if m.group(1) in skip:
|
||||
continue
|
||||
page_of[m.group(1)] = os.path.basename(path)
|
||||
at = cleared.get(m.group(1))
|
||||
check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
|
||||
% (os.path.basename(path), m.group(1)))
|
||||
# Control: a regex that stopped matching would leave nothing to assert on.
|
||||
check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
|
||||
|
||||
sid = None
|
||||
|
||||
|
||||
def get(path):
|
||||
# The UI is served ISO-8859-1, so decode, do not assume UTF-8.
|
||||
return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
|
||||
|
||||
|
||||
def post(fields):
|
||||
global sid
|
||||
if sid is None:
|
||||
m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
|
||||
if m is None:
|
||||
sys.exit("no session id in server/index.html")
|
||||
sid = m.group(1)
|
||||
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
|
||||
[("sid", sid)] + fields)
|
||||
req = urllib.request.Request(url + "/server/step4.html",
|
||||
data=body.encode("latin-1"), method="POST")
|
||||
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
|
||||
|
||||
|
||||
def textarea(page, name):
|
||||
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
|
||||
if m is None:
|
||||
sys.exit("no %s textarea in the rendered step4.html" % name)
|
||||
return m.group(1)
|
||||
|
||||
|
||||
def checked(page, name):
|
||||
return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
|
||||
|
||||
|
||||
# What each state puts on the command line (None: nothing) and its profile key.
|
||||
# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
|
||||
# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
|
||||
BOXES = [
|
||||
# field profile key set cleared
|
||||
("parseall", "ParseAll", "--near", None),
|
||||
("link", "Near", "--test", None),
|
||||
("testall", "Test", "--extended-parsing", None),
|
||||
# htmlfirst emits --priority=7, which the scan-priority list also emits.
|
||||
("htmlfirst", "HTMLFirst", None, None),
|
||||
("errpage", "NoErrorPages", "--generate-errors=0", None),
|
||||
("external", "NoExternalPages", "--replace-external", None),
|
||||
("hidepwd", "NoPwdInPages", "--disable-passwords", None),
|
||||
("hidequery", "NoQueryStrings", "--include-query-string=0", None),
|
||||
("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
|
||||
("windebug", None, "--debug-headers", None),
|
||||
("ka", "KeepAlive", "--keep-alive", None),
|
||||
("remt", "RemoveTimeout", "--host-control=1", None),
|
||||
("rems", "RemoveRateout", "--host-control=2", None),
|
||||
("cookies", "Cookies", None, "--cookies=0"),
|
||||
("parsejava", "ParseJava", None, "--parse-java=0"),
|
||||
("updhack", "UpdateHack", "--updatehack", None),
|
||||
("urlhack", "URLHack", "--urlhack", None),
|
||||
("keepwww", "KeepWww", "--keep-www-prefix", None),
|
||||
("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
|
||||
("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
|
||||
("toler", "TolerantRequests", "--tolerant", None),
|
||||
("http10", "HTTP10", "--http-10", None),
|
||||
("warc", "Warc", "--warc", None),
|
||||
("norecatch", "NoRecatch", "--do-not-recatch", None),
|
||||
("logf", "Log", "--single-log", None),
|
||||
("index", "Index", None, None),
|
||||
("index2", "WordIndex", "--search-index", None),
|
||||
("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
|
||||
]
|
||||
|
||||
for field, key, when_set, when_clear in BOXES:
|
||||
for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
|
||||
("", when_clear, when_set, "0")):
|
||||
rendered = post([(field, value)])
|
||||
cmd = textarea(rendered, "command").split()
|
||||
label = "%s %s" % (field, "set" if value else "cleared")
|
||||
if want:
|
||||
check(want in cmd, "%s emits %s" % (label, want))
|
||||
if unwanted:
|
||||
check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
|
||||
if key:
|
||||
# The Windows GUI reads the same option out of the profile.
|
||||
check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
|
||||
"%s writes %s=%s" % (label, key, stored))
|
||||
check(checked(page_of[field], field) == bool(value),
|
||||
"%s draws %s box" % (label, "a ticked" if value else "an empty"))
|
||||
missing = sorted(set(page_of) - set(b[0] for b in BOXES))
|
||||
check(not missing, "every checkbox is exercised (missing %s)" % missing)
|
||||
|
||||
# A browser posts the companion and the ticked box under one name, so the fix
|
||||
# rests on the body loop keeping the last value; both orders pin that down.
|
||||
cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
|
||||
check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
|
||||
check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
|
||||
cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
|
||||
check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
|
||||
check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
|
||||
|
||||
sys.exit(rc)
|
||||
PY
|
||||
|
||||
# A leaked htsserver wedges the parallel harness behind a green log.
|
||||
cleanup
|
||||
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
|
||||
|
||||
echo "PASS"
|
||||
181
tests/91_webhttrack-directory.test
Normal file
181
tests/91_webhttrack-directory.test
Normal file
@@ -0,0 +1,181 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# fopen() succeeds on a directory on POSIX and reading one never reaches EOF; a
|
||||
# FIFO blocks in fopen() instead. Either used to wedge the single-threaded server.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
|
||||
distdir=$(cd "${distdir}" && pwd)
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
command -v htsserver >/dev/null || fail "no htsserver in PATH"
|
||||
command -v python3 >/dev/null || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
srv=
|
||||
log=$(mktemp)
|
||||
base=$(mktemp -d)
|
||||
cleanup() {
|
||||
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
|
||||
rm -f "${log}"
|
||||
rm -rf "${base}"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
# First line only. A "| head -1" would close the pipe early and, under pipefail,
|
||||
# SIGPIPE the producer into a spurious failure.
|
||||
firstline() { echo "${1%%$'\n'*}"; }
|
||||
|
||||
freeport() {
|
||||
python3 -c 'import socket
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
print(s.getsockname()[1])
|
||||
s.close()'
|
||||
}
|
||||
|
||||
# Echo the announced URL. Runs in a command substitution, so it is a
|
||||
# subshell and cannot export the pid: the caller reads it back with srvpid.
|
||||
start() {
|
||||
local port url
|
||||
port=$(freeport)
|
||||
: >"${log}"
|
||||
(
|
||||
trap '' TERM TTOU
|
||||
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
|
||||
) &
|
||||
for _ in $(seq 1 40); do
|
||||
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
|
||||
sleep 0.25
|
||||
done
|
||||
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
|
||||
echo "${url}"
|
||||
}
|
||||
|
||||
# The server reports its own pid; the aliveness assertion below hangs off it.
|
||||
srvpid() { firstline "$(sed -n 's/^PID=//p' "${log}")"; }
|
||||
|
||||
alive() { kill -0 "$1" 2>/dev/null; }
|
||||
|
||||
portof() { echo "${1##*:}" | tr -d /; }
|
||||
|
||||
# GET the path $2 from 127.0.0.1:$1, or POST the body $3 to / when $2 is empty.
|
||||
# The socket timeout is what makes an unfixed tree fail rather than wedge CI.
|
||||
request() {
|
||||
python3 -c 'import socket, sys
|
||||
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
||||
if path:
|
||||
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
|
||||
else:
|
||||
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
|
||||
"Content-type: application/x-www-form-urlencoded\r\n"
|
||||
"Content-length: %d\r\n\r\n%s" % (len(body), body))
|
||||
s = socket.create_connection(("127.0.0.1", port), 10)
|
||||
s.settimeout(15)
|
||||
s.sendall(req.encode())
|
||||
out = b""
|
||||
try:
|
||||
while True:
|
||||
b = s.recv(65536)
|
||||
if not b:
|
||||
break
|
||||
out += b
|
||||
except socket.timeout:
|
||||
sys.stderr.write("timed out after %d bytes\n" % len(out))
|
||||
sys.exit(9)
|
||||
s.close()
|
||||
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
|
||||
}
|
||||
|
||||
# GET $2, or fail with $3 when the reply never comes.
|
||||
get() {
|
||||
local out
|
||||
out=$(request "$1" "$2" "") || fail "$2 did not answer: $3"
|
||||
echo "${out}"
|
||||
}
|
||||
|
||||
post() { request "$1" "" "$2"; }
|
||||
|
||||
url=$(start)
|
||||
port=$(portof "${url}")
|
||||
srv=$(srvpid)
|
||||
test -n "${srv}" || fail "htsserver did not report its pid"
|
||||
|
||||
# Needs no session id and no project: html/server is a directory of the install.
|
||||
reply=$(get "${port}" /server/ "an unauthenticated request wedged the server")
|
||||
case "${reply}" in
|
||||
"HTTP/1.0 404 "*) ;;
|
||||
*) fail "a GUI directory did not answer 404: $(firstline "${reply}")" ;;
|
||||
esac
|
||||
|
||||
# Every request body is gated by the session id.
|
||||
reply=$(get "${port}" /server/index.html "the GUI is not served")
|
||||
sid=$(firstline "$(echo "${reply}" |
|
||||
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p')")
|
||||
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
|
||||
|
||||
mkdir -p "${base}/proj/sub"
|
||||
echo "PROBEMARKER" >"${base}/proj/probe.txt"
|
||||
|
||||
# mkfifo and ln -s may not work on a Windows checkout, so both stay optional.
|
||||
refuse=(/website/ /website/sub/)
|
||||
if mkfifo "${base}/proj/fifo.txt" 2>/dev/null; then
|
||||
refuse+=(/website/fifo.txt)
|
||||
fi
|
||||
ln -s probe.txt "${base}/proj/link.txt" 2>/dev/null || true
|
||||
|
||||
# step4's "save settings" flow registers the project without crawling, and that
|
||||
# is what arms the /website/ root.
|
||||
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
|
||||
body="${body}&path=${base}&projname=proj"
|
||||
post "${port}" "${body}" >/dev/null || fail "the save request did not answer"
|
||||
test -f "${base}/proj/hts-cache/winprofile.ini" ||
|
||||
fail "the project was not registered: $(cat "${log}")"
|
||||
|
||||
# Positive control: every assertion below would pass on a server serving nothing.
|
||||
served() {
|
||||
case "$(get "$1" "$2" "$3")" in
|
||||
*PROBEMARKER*) ;;
|
||||
*) fail "$3" ;;
|
||||
esac
|
||||
}
|
||||
served "${port}" /website/probe.txt "the registered project's mirror is not served"
|
||||
if test -L "${base}/proj/link.txt"; then
|
||||
# The guard stats rather than lstats, so a symlinked mirror file still serves.
|
||||
served "${port}" /website/link.txt "a symlink to a mirror file is not served"
|
||||
fi
|
||||
|
||||
# /website/ is where the GUI's own "browse mirrored site" link points.
|
||||
for path in "${refuse[@]}"; do
|
||||
reply=$(get "${port}" "${path}" "the server wedged on ${path}")
|
||||
case "${reply}" in
|
||||
"HTTP/1.0 404 "*) ;;
|
||||
*) fail "${path} did not answer 404: $(firstline "${reply}")" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# The accept loop is single-threaded: one spin denies every later request.
|
||||
alive "${srv}" || fail "the server died: $(cat "${log}")"
|
||||
served "${port}" /website/probe.txt "the server stopped answering after a directory request"
|
||||
|
||||
# This fopen() is reached before any guard, so its read loop must stop on ferror().
|
||||
mkdir -p "${base}/proj2/hts-cache/winprofile.ini"
|
||||
reply=$(post "${port}" "sid=${sid}&path=${base}&loadprojname=proj2") ||
|
||||
fail "a directory winprofile.ini wedged the server"
|
||||
case "${reply}" in
|
||||
"HTTP/1.0 3"*) ;;
|
||||
*) fail "loading a project did not redirect: $(firstline "${reply}")" ;;
|
||||
esac
|
||||
|
||||
alive "${srv}" || fail "the server died: $(cat "${log}")"
|
||||
served "${port}" /website/probe.txt "the server stopped answering after a project load"
|
||||
|
||||
echo "PASS"
|
||||
37
tests/92_local-proxytrack-ndx-fields.test
Normal file
37
tests/92_local-proxytrack-ndx-fields.test
Normal file
@@ -0,0 +1,37 @@
|
||||
#!/bin/bash
|
||||
# The two halves of an .ndx entry's URL share one buffer, so the second must be
|
||||
# bounded by what the first left. The sanitizer CI build turns a regression
|
||||
# here into a hard stack-buffer-overflow.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=tests/testlib.sh
|
||||
. "${testdir}/testlib.sh"
|
||||
|
||||
python=$(find_python) || {
|
||||
echo "python3 not found; skipping" >&2
|
||||
exit 77
|
||||
}
|
||||
|
||||
dir=$(mktemp -d)
|
||||
trap 'rm -rf "$dir"' EXIT
|
||||
|
||||
# each field fills binput's own bound, so together they run one past line[]
|
||||
"$python" - "$dir/foo.ndx" <<'EOF'
|
||||
import sys
|
||||
|
||||
with open(sys.argv[1], "w") as f:
|
||||
f.write("CACHE-1.4\n")
|
||||
f.write("Wed, 01 Jan 2025 00:00:00 GMT\n")
|
||||
f.write("h" * 1024 + "\n" + "f" * 1024 + "\n" + "0\n")
|
||||
# a trailing entry keeps the walk inside the buffer, so this exercises the
|
||||
# field bound alone
|
||||
f.write("tail/entry\nx\n1\n")
|
||||
EOF
|
||||
: >"$dir/foo.dat" # the reader only proceeds when the sibling .dat opens
|
||||
|
||||
proxytrack --convert "$dir/out.arc" "$dir/foo.ndx" >/dev/null 2>&1 || {
|
||||
echo "FAIL: proxytrack crashed on an .ndx with two full-length URL fields" >&2
|
||||
exit 1
|
||||
}
|
||||
@@ -32,6 +32,7 @@ TESTS = \
|
||||
00_runnable.test \
|
||||
01_engine-charset.test \
|
||||
01_engine-cmdline.test \
|
||||
01_engine-cmdline-split.test \
|
||||
01_engine-cookies.test \
|
||||
01_engine-copyopt.test \
|
||||
01_engine-crange.test \
|
||||
@@ -176,6 +177,18 @@ TESTS = \
|
||||
77_webhttrack-redirect.test \
|
||||
78_webhttrack-sid.test \
|
||||
79_local-proxytrack-webdav-mime.test \
|
||||
80_engine-crash-symbolize.test
|
||||
80_engine-crash-symbolize.test \
|
||||
81_webhttrack-maxsize.test \
|
||||
82_webhttrack-browse-links.test \
|
||||
83_webhttrack-argescape.test \
|
||||
84_webhttrack-mirror-verbatim.test \
|
||||
85_webhttrack-projpath.test \
|
||||
86_local-proxytrack-cache-longfields.test \
|
||||
87_local-proxytrack-nodate.test \
|
||||
88_local-proxytrack-badmtime.test \
|
||||
89_webhttrack-error-overflow.test \
|
||||
90_webhttrack-checkbox-clear.test \
|
||||
91_webhttrack-directory.test \
|
||||
92_local-proxytrack-ndx-fields.test
|
||||
|
||||
CLEANFILES = check-network_sh.cache
|
||||
|
||||
Reference in New Issue
Block a user