Compare commits

..

8 Commits

Author SHA1 Message Date
Xavier Roche
e914e0ecd3 Symbolize once when the handler itself faults
A fault inside the handler re-enters the printer, which interleaved a second
symbolized trace on the same fd and spent a second budget: 3.04s and two
overlapping reports, measured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 15:32:42 +02:00
Xavier Roche
bf7d45f74c Merge origin/master into feat/crash-symbolize
# Conflicts:
#	tests/Makefile.am
2026-07-26 15:21:02 +02:00
Xavier Roche
02599e0b7a Merge origin/master into feat/crash-symbolize
# Conflicts:
#	tests/Makefile.am
2026-07-26 14:39:12 +02:00
Xavier Roche
7ae16147c0 Stop discarding local symbols, which made the names wrong
--discard-all drops the local symbol of every static function, so addr2line
attributes the frame to the nearest surviving global: the abort frame read
dns_timeout_selftests instead of abortf_, with the file and line still right.
A wrong name is worse than none, and because the symbols go at link time no
-dbgsym package can recover them. Costs 21784 bytes on libhttrack.so.3, 0.6%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 14:31:54 +02:00
Xavier Roche
c09b2fb290 Skip pseudo-modules with no file on disk
A frame in linux-vdso.so.1 made addr2line complain instead of resolving, so
the arm64 leg lost its symbolized output entirely. Renumber the test too:
77 landed on master with #700.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 14:17:40 +02:00
Xavier Roche
795ffe03fb Test: match glibc's backtrace format with or without the space
backtrace_symbols_fd() prints the trailing "[0xADDR]" with a leading space on
some glibc versions and without on others (Ubuntu 24.04), so the raw-frame
assertion failed everywhere but the dev box. Match only up to the offset.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 14:06:10 +02:00
Xavier Roche
f5198e1037 Move the crash backtrace printer into src/htsbacktrace.c
httrack.c keeps only the two call sites. The symbolizer needs _GNU_SOURCE for
dladdr(), which is now confined to its own translation unit instead of being
forced on the whole CLI front-end.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 14:01:39 +02:00
Xavier Roche
26bb361b8d Symbolize fatal-signal backtraces through addr2line
backtrace_symbols_fd() resolves names from .dynsym only, and
-fvisibility=hidden keeps every engine frame out of it, so a crash report
arrived as a column of bare module+offset. The handler now emits the raw trace
first and unconditionally, then groups the frames per module and runs addr2line
(or llvm-symbolizer) over the offsets, which reads DWARF and names the static
frames plus their inline chain.

-rdynamic is dropped: it only populated .dynsym and bought exactly one named
frame. -Wl,--build-id replaces it, so a trace from a stripped build can be
matched to its debug symbols.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-26 13:50:28 +02:00
56 changed files with 342 additions and 2911 deletions

View File

@@ -226,9 +226,8 @@ jobs:
# Every gate here exits 77, so an all-skipped suite would report green having
# tested nothing: pin the skips, and floor the passes in case the glob empties.
# footer-overflow skips on Windows (needs a path past MAX_PATH); crange pending #581;
# webdav-mime needs a reapable background listener, which MSYS cannot give it;
# badmtime needs a filesystem that stores an mtime past gmtime's range.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test 88_local-proxytrack-badmtime.test"
# webdav-mime needs a reapable background listener, which MSYS cannot give it.
expected_skips=" 01_engine-footer-overflow.test 48_local-crange-memresume.test 71_local-crange-repaircache.test 79_local-proxytrack-webdav-mime.test"
[ "$pass" -ge 90 ] || { echo "::error::only $pass tests passed ($skip skipped)"; exit 1; }
[ "$skipped" = "$expected_skips" ] || { echo "::error::unexpected skips:$skipped"; exit 1; }
[ "$fail" -eq 0 ] || { echo "::error::failing:$failed"; exit 1; }

View File

@@ -26,12 +26,6 @@ the operational checklist: toolchain, invariants, and how to ship a change.
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
- Never assert with `cmd | grep -q MARKER && fail`. Under `pipefail` the
pipeline is non-zero both when `cmd` fails and when `grep -q` matches early
and SIGPIPEs it, so the `&&` never fires and a probe that proved nothing reads
as "marker absent". Capture the reply, assert the status line it must carry
(an empty, truncated or redirected one is marker-free too), then match with a
here-string.
## Hard invariants
- **Generated autotools files are NOT in git.** `configure`, every
@@ -54,16 +48,6 @@ the operational checklist: toolchain, invariants, and how to ship a change.
- Bounds-check every copy. Overflow-safe form: put the untrusted value alone,
`untrusted < limit - controlled` — never `controlled + untrusted < limit`,
which can wrap and pass.
- **Abort or clip is a decision, not a default.** The `*_safe_` helpers
(`strcpybuff`, `strlcpybuff`, `strcatbuff`) **abort** on overflow. Right for
our own data, wrong for anything read back from a cache, a header or the
wire, where it trades a memory smash for a crash on malformed input. Clip
with `dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1)`.
- **A warning class is not the unsafe set.** `-Wformat-truncation` fires only on
a *bounded* `snprintf` whose return is discarded, so an unbounded `sprintf`
into the same buffer never appears on it. Before scoping a hardening pass off
compiler output, grep the unguarded forms yourself (`\bsprintf\s*\(`,
`\bstrcpy\s*\(`, `\bstrcat\s*\(`).
## C conventions
- **Use the `*t` allocator wrappers, never raw libc** (`htssafe.h`):
@@ -100,17 +84,6 @@ Before pushing, and when reviewing others, don't skim for bugs:
layout/ABI, cache/wire format, or a security path? A static or unit check
isn't enough; exercise the wrong behavior at runtime. Claude Code:
`/review-recipe`.
- **Poison a canary, never compare it against zero.** Checking that a
neighbouring field is still `'\0'` cannot see the stray NUL an off-by-one
terminator writes — the exact bug the canary is there for. Fill it with a
non-zero byte, and prove it by killing both the stray-`'X'` and the
stray-NUL mutant. Neither ASan nor `_FORTIFY_SOURCE` sees an overflow that
lands inside the same struct.
- **Overshoot every destination, not one.** A bounds test that oversizes a
single field cannot tell a per-field bound from a one-size-fits-all one, nor
from a fix that bounds that field and leaves its neighbours raw. Exercise
each destination the path touches, spanning at least two capacities, and
check what the code actually emits before writing the expected values.
## Commits
- **Sign-off is mandatory.** Every commit carries a `Signed-off-by` trailer:

View File

@@ -97,8 +97,9 @@ ${do:end-if}
</pre>
${LANG_G8} :
${/* an http: page cannot navigate to file:, so the mirror is reached through the server */}
<a href="/website/index.html" target="_new">
${do:output-mode:html-urlescaped}
<a href="file://${path}/${projname}/" target="_new">
${do:output-mode:}
${path}/${projname}
</a></li>
<ul>

View File

@@ -98,9 +98,6 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="ftpprox" value="">
${LANG_PROXYTYPE}
<select name="proxytype"
title='${html:LANG_PROXYTYPETIP}' onMouseOver="info('${html:LANG_PROXYTYPETIP}'); return true" onMouseOut="info('&nbsp;'); return true"

View File

@@ -98,13 +98,6 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="errpage" value="">
<input type="hidden" name="external" value="">
<input type="hidden" name="hidepwd" value="">
<input type="hidden" name="hidequery" value="">
<input type="hidden" name="nopurge" value="">
${LANG_I33}
<br>
<select name="build"

View File

@@ -103,7 +103,7 @@ ${LANG_Q3}
<tr><td>
<table width="100%">
<tr><td align="left">
<input type="submit" value="${LANG_OK}"
<input type="submit" value="${LANG_OK]"
${do:output-mode:html-urlescaped}
onClick="if (confirm(str_replace(str_replace('${LANG_DIAL7}', '%20', ' '), '%0a', ' '))) { form.closeme.value=1; form.submit(); } return false;"
${do:output-mode:}

View File

@@ -98,9 +98,6 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="windebug" value="">
${LANG_I40c}
<br>

View File

@@ -98,11 +98,6 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="ka" value="">
<input type="hidden" name="remt" value="">
<input type="hidden" name="rems" value="">
<table border="0" width="100%" cellspacing="0">
<tr><td>

View File

@@ -98,17 +98,6 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="cookies" value="">
<input type="hidden" name="parsejava" value="">
<input type="hidden" name="updhack" value="">
<input type="hidden" name="urlhack" value="">
<input type="hidden" name="keepwww" value="">
<input type="hidden" name="keepslashes" value="">
<input type="hidden" name="keepqueryorder" value="">
<input type="hidden" name="toler" value="">
<input type="hidden" name="http10" value="">
<input type="checkbox" name="cookies" ${checked:cookies}
title='${html:LANG_I1b}' onMouseOver="info('${html:LANG_I1b}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I58}

View File

@@ -98,13 +98,6 @@ ${do:end-if}
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="warc" value="">
<input type="hidden" name="norecatch" value="">
<input type="hidden" name="logf" value="">
<input type="hidden" name="index" value="">
<input type="hidden" name="index2" value="">
<input type="checkbox" name="cache2" ${checked:cache2}
title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I61}

View File

@@ -114,16 +114,16 @@ ${do:end-if}
${/* Real commands and ini file generated below */}
<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
<!-- engine commandline -->
${do:output-mode:html}
<textarea name="command" cols="50" rows="4" style="visibility:hidden">
httrack \
--quiet \
--build-top-index \
${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
${unquoted:urls}
${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
--path "${arg:path}/${arg:projname}"
${urls}
${test:filelist:-%L "}${filelist}${test:filelist:"}
--path "${html:path}/${html:projname}"
\
${test:parseall:--near}
${test:link:--test}
@@ -131,7 +131,7 @@ httrack \
${test:htmlfirst::--priority=7}
\
${do:if-not-empty:BuildString}
--structure "${arg:BuildString}"
--structure "${BuildString}"
${do:end-if}
${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
\
@@ -150,52 +150,51 @@ ${do:end-if}
${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
${test:windebug:--debug-headers}
\
${test:connexion:--sockets=}${unquoted:connexion}
${test:connexion:--sockets=}${connexion}
${test:ka:--keep-alive}
${test:timeout:--timeout=}${unquoted:timeout}
${test:timeout:--timeout=}${timeout}
${test:remt:--host-control=1}
${test:retry:--retries=}${unquoted:retry}
${test:rate:--min-rate=}${unquoted:rate}
${test:retry:--retries=}${retry}
${test:rate:--min-rate=}${rate}
${test:rems:--host-control=2}
\
${test:depth:--depth=}${unquoted:depth}
${test:depth2:--ext-depth=}${unquoted:depth2}
${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
${test:othermax:--max-files=}${unquoted:othermax}
${test:maxhtml:--max-files=,}${unquoted:maxhtml}
${test:sizemax:--max-size=}${unquoted:sizemax}
${test:pausebytes:--max-pause=}${unquoted:pausebytes}
${test:maxtime:--max-time=}${unquoted:maxtime}
${test:maxrate:--max-rate=}${unquoted:maxrate}
${test:maxconn:--connection-per-second=}${unquoted:maxconn}
${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
${test:depth:--depth=}${depth}
${test:depth2:--ext-depth=}${depth2}
${test:maxhtml:--max-files=,}${maxhtml}
${test:othermax:--max-files=}${othermax}
${test:sizemax:--max-files=}${sizemax}
${test:pausebytes:--max-pause=}${pausebytes}
${test:maxtime:--max-time=}${maxtime}
${test:maxrate:--max-rate=}${maxrate}
${test:maxconn:--connection-per-second=}${maxconn}
${test:maxlinks:--advanced-maxlinks=}${maxlinks}
\
--user-agent "${arg:user}"
--footer "${arg:footer}"
--user-agent "${html:user}"
--footer "${html:footer}"
\
${unquoted:url2}
${url2}
\
${ztest:cookies:--cookies=0:}
${ztest:parsejava:--parse-java=0:}
${test:cookies:--cookies=0:}
${test:parsejava:--parse-java=0:}
${test:updhack:--updatehack}
${test:urlhack:--urlhack=0:--urlhack}
${test:keepwww:--keep-www-prefix}
${test:keepslashes:--keep-double-slashes}
${test:keepqueryorder:--keep-query-order}
${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
${test:cookiesfile:--cookies-file "}${html:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${html:stripquery}${test:stripquery:"}
${test:toler:--tolerant}
${test:http10:--http-10}
${test:cache2:--store-all-in-cache}
${test:warc:--warc}
${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
${test:norecatch:--do-not-recatch}
${test:logf:--single-log}
${test:logtype:::--extra-log:--debug-log}
${test:index:--index=0:}
${test:index2:--search-index=0:--search-index}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${prox}${test:prox:\3A}${portprox}${test:prox:"}
${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
</textarea>
@@ -214,7 +213,7 @@ ParseAll=${ztest:parseall:0:1}
HTMLFirst=${ztest:htmlfirst:0:1}
Cache=${ztest:cache:0:1}
NoRecatch=${ztest:norecatch:0:1}
Dos=${dos}
Dos=${dos
Index=${ztest:index:0:1}
WordIndex=${ztest:index2:0:1}
Log=${ztest:logf:0:1:2}

View File

@@ -71,9 +71,7 @@ static int mysavename(t_hts_callbackarg * carg, httrackp * opt,
for(j = 0; iisBogus[i][j] == a[j] && iisBogus[i][j] != '\0'; j++) ;
if (iisBogus[i][j] == '\0'
&& (a[j] == '\0' || a[j] == '/' || a[j] == '\\')) {
/* j bytes matched, so j fit: copying j cannot overrun whatever the
table holds, and the tail must survive untouched */
memcpy(a, iisBogusReplace[i], (size_t) j);
strncpy(a, iisBogusReplace[i], strlen(iisBogusReplace[i]));
break;
}
}

View File

@@ -48,8 +48,7 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
lib_LTLIBRARIES = libhttrack.la
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h htsstats.h \
htscmdline.c htscmdline.h \
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
htsurlport.c htsurlport.h
proxytrack_SOURCES = proxy/main.c \
proxy/proxytrack.c proxy/store.c \
@@ -63,7 +62,7 @@ whttrackrun_SCRIPTS = webhttrack
libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htscache_selftest.c htsdns_selftest.c htsselftest.c \
htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
htscmdline.c htshelp.c htslib.c htsurlport.c htscoremain.c \
htshelp.c htslib.c htsurlport.c htscoremain.c \
htsname.c htsrobots.c htstools.c htswizard.c \
htsalias.c htsthread.c htsindex.c htsbauth.c \
htsmd5.c htscodec.c htswarc.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
@@ -72,7 +71,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c \
hts-indextmpl.h htsalias.h htsback.h htsbase.h htssafe.h \
htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h \
htscmdline.h htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
htsmodules.h htsname.h htsnet.h htssniff.h \
@@ -87,7 +86,7 @@ libhttrack_la_LIBADD = $(THREADS_LIBS) $(ZLIB_LIBS) $(BROTLI_LIBS) $(ZSTD_LIBS)
libhttrack_la_CFLAGS = $(AM_CFLAGS) -DLIBHTTRACK_EXPORTS -DZLIB_CONST
libhttrack_la_LDFLAGS = $(AM_LDFLAGS) -version-info $(VERSION_INFO)
EXTRA_DIST = httrack.h htsstats.h webhttrack \
EXTRA_DIST = httrack.h webhttrack \
version.rc \
libhttrack.rc \
httrack.rc \

View File

@@ -125,22 +125,6 @@ void back_free(struct_back ** sback) {
above a normal handshake. The last candidate still gets the full timeout. */
#define HTS_CONNECT_FALLBACK_TIMEOUT 10
void back_read_ftp_result(FILE *fp, htsblk *r) {
size_t j = 0;
if (fscanf(fp, "%d ", &r->statuscode) != 1)
r->statuscode = STATUSCODE_INVALID;
// an external helper writes this file: stop at capacity, not at EOF
while (j + 1 < sizeof(r->msg)) {
const int c = fgetc(fp);
if (c == EOF)
break;
r->msg[j++] = (char) c;
}
r->msg[j] = '\0';
}
int back_connect_fallback_due(int addr_index, int addr_count, int elapsed,
int timeout) {
int deadline;
@@ -558,14 +542,8 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
// do not use tempnam() but a regular filename
back->tmpfile_buffer[0] = '\0';
if (back->url_sav[0] != '\0') {
/* same capacity as url_sav, so truncation drops the extension and aliases
the temp name onto the live file that back_finalize_backup() UNLINKs */
if (!sprintfbuff(back->tmpfile_buffer, "%s.%s", back->url_sav, ext)) {
hts_log_print(opt, LOG_WARNING, "temporary filename too long for %s",
back->url_sav);
back->tmpfile_buffer[0] = '\0';
return -1;
}
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s.%s",
back->url_sav, ext);
back->tmpfile = back->tmpfile_buffer;
if (structcheck(back->tmpfile) != 0) {
hts_log_print(opt, LOG_WARNING, "can not create directory to %s",
@@ -573,15 +551,8 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
return -1;
}
} else {
/* truncation here would collide distinct tmpnameid's onto one name */
if (!sprintfbuff(back->tmpfile_buffer, "%s/tmp%d.%s",
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++,
ext)) {
hts_log_print(opt, LOG_WARNING, "temporary filename too long in %s",
StringBuff(opt->path_html_utf8));
back->tmpfile_buffer[0] = '\0';
return -1;
}
snprintf(back->tmpfile_buffer, sizeof(back->tmpfile_buffer), "%s/tmp%d.%s",
StringBuff(opt->path_html_utf8), opt->state.tmpnameid++, ext);
back->tmpfile = back->tmpfile_buffer;
}
/* OK */
@@ -2976,7 +2947,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
back[i].r.msg[0] = '\0';
strncatbuff(back[i].r.msg, tmp, sizeof(back[i].r.msg) - 2);
if (!strnotempty(back[i].r.msg)) {
htsblk_failf(&back[i].r, "SSL/TLS error %d", err_code);
sprintf(back[i].r.msg, "SSL/TLS error %d", err_code);
}
deletehttp(&back[i].r);
back[i].r.soc = INVALID_SOCKET;
@@ -3049,7 +3020,16 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
FOPEN(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"),
"rb");
if (fp) {
back_read_ftp_result(fp, &back[i].r);
int j = 0;
fscanf(fp, "%d ", &(back[i].r.statuscode));
while(!feof(fp)) {
int c = fgetc(fp);
if (c != EOF)
back[i].r.msg[j++] = c;
}
back[i].r.msg[j++] = '\0';
fclose(fp);
UNLINK(fconcat(OPT_GET_BUFF(opt), back[i].location_buffer, ".ok"));
strcpybuff(fconcat
@@ -3350,11 +3330,10 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
deleteaddr(&back[i].r);
if (back[i].r.size < back[i].r.totalsize)
back[i].r.statuscode = STATUSCODE_CONNERROR; // recatch
htsblk_failf(&back[i].r,
"Incorrect length (" LLintP " Bytes, " LLintP
" expected)",
(LLint) back[i].r.size,
(LLint) back[i].r.totalsize);
sprintf(back[i].r.msg,
"Incorrect length (" LLintP " Bytes, " LLintP
" expected)", (LLint) back[i].r.size,
(LLint) back[i].r.totalsize);
} else {
// Un warning suffira..
hts_log_print(opt, LOG_WARNING,

View File

@@ -74,10 +74,6 @@ void back_free(struct_back ** sback);
// backing
#define BACK_ADD_TEST "(dummy)"
#define BACK_ADD_TEST2 "(dummy2)"
/* Parse an external FTP helper's "<statuscode> <message>" result file into r,
clipping the message to r->msg. */
void back_read_ftp_result(FILE *fp, htsblk *r);
int back_index(httrackp * opt, struct_back * sback, const char *adr, const char *fil,
const char *sav);
int back_available(const struct_back * sback);

View File

@@ -142,12 +142,10 @@ struct cache_back_zip_entry {
int compressionMethod;
};
/* A corrupt cache can carry a field wider than ours; clipping it keeps the
entry, where aborting would take the crawl down. */
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(void) strclipbuff(refvalue, refvalue_size, value); \
strlcpybuff(refvalue, value, refvalue_size); \
line[0] = '\0'; \
} \
} while (0)

View File

@@ -1195,12 +1195,6 @@ int cache_legacy_refused_selftest(httrackp *opt, const char *dir) {
/* --- read-side corruption injection --------------------------------------- */
/* 100 'A's: a placeholder header line long enough to be overwritten by a
forged, over-long X-StatusMessage of the same byte length. */
#define CORRUPT_LONG_ETAG \
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" \
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
/* canary read back intact after each corruption; victim gets the byte surgery
*/
#define CORRUPT_ADR "corrupt.example.com"
@@ -1225,24 +1219,6 @@ static void corrupt_build(httrackp *opt) {
selftest_close(&cache);
}
/* Like corrupt_build, but the victim carries a 100-char Etag placeholder. */
static void corrupt_build_longetag(httrackp *opt) {
cache_back cache;
memset(corrupt_body_a, 'a', sizeof(corrupt_body_a) - 1);
memset(corrupt_body_b, 'b', sizeof(corrupt_body_b) - 1);
remove(reconcile_st_path(opt, "hts-cache/new.zip"));
remove(reconcile_st_path(opt, "hts-cache/old.zip"));
selftest_open_for_write(&cache, opt);
store_entry(opt, &cache, CORRUPT_ADR, "/canary.html", "canary.html", 200,
"OK", "text/html", "utf-8", "", "", "", "", corrupt_body_a,
strlen(corrupt_body_a));
store_entry(opt, &cache, CORRUPT_ADR, "/victim.html", "victim.html", 200,
"OK", "text/html", "utf-8", "", CORRUPT_LONG_ETAG, "", "",
corrupt_body_b, strlen(corrupt_body_b));
selftest_close(&cache);
}
/* Like corrupt_build, but the victim carries a 20-char Etag whose header line
is later overwritten with a forged oversized X-Size (same byte length). */
static void corrupt_build_etag(httrackp *opt) {
@@ -1427,48 +1403,6 @@ static int corrupt_expect_disk_header(httrackp *opt, LLint wantsize,
return fail;
}
/* An over-long field from a foreign cache must clip, not abort: the entry
still reads, clipped to capacity, and the canary survives. */
static int corrupt_expect_victim_clipped(httrackp *opt, size_t wantmsg,
size_t wantlastmod, const char *what) {
cache_back cache;
htsblk v, c;
char BIGSTK lv[HTS_URLMAXSIZE * 2];
char BIGSTK lc[HTS_URLMAXSIZE * 2];
int fail = 0;
selftest_open_for_read(&cache, opt);
lv[0] = lc[0] = '\0';
v = cache_readex(opt, &cache, CORRUPT_ADR, "/victim.html", "", lv, NULL, 1);
if (v.statuscode != 200) {
fprintf(stderr, "%s: %s: status %d, expected 200\n", selftest_tag, what,
v.statuscode);
fail++;
}
if (wantmsg != (size_t) -1 && strlen(v.msg) != wantmsg) {
fprintf(stderr, "%s: %s: msg len %u, expected %u\n", selftest_tag, what,
(unsigned) strlen(v.msg), (unsigned) wantmsg);
fail++;
}
if (wantlastmod != (size_t) -1 && strlen(v.lastmodified) != wantlastmod) {
fprintf(stderr, "%s: %s: lastmodified len %u, expected %u\n", selftest_tag,
what, (unsigned) strlen(v.lastmodified), (unsigned) wantlastmod);
fail++;
}
c = cache_readex(opt, &cache, CORRUPT_ADR, "/canary.html", "", lc, NULL, 1);
if (c.statuscode != 200) {
fprintf(stderr, "%s: %s: canary tainted (status %d)\n", selftest_tag, what,
c.statuscode);
fail++;
}
if (v.adr != NULL)
freet(v.adr);
if (c.adr != NULL)
freet(c.adr);
selftest_close(&cache);
return fail;
}
/* One zip corruption case: build, patch, then check victim+canary in-session.
*/
static int corrupt_case_zip(httrackp *opt, const char *pat, const char *rep,
@@ -1505,31 +1439,6 @@ int cache_corruption_selftest(httrackp *opt, const char *dir) {
failures += corrupt_expect_victim(opt, "Cache Read Error : Read Data",
"garbled deflate stream");
/* A corrupt cache can hold a field wider than ours. Clipping keeps the
entry; aborting would take the crawl down. Overwrite the placeholder Etag
line in place, same byte length, so the zip offsets stay intact. */
corrupt_build_longetag(opt);
corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
"X-StatusMessage: " /* 17 + 89 = 106 */
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
1, 1);
failures +=
corrupt_expect_victim_clipped(opt, sizeof(((htsblk *) 0)->msg) - 1,
(size_t) -1, "over-long X-StatusMessage");
/* lastmodified[64] is narrower than msg[80]: one hardcoded clip length
cannot satisfy both. */
corrupt_build_longetag(opt);
corrupt_patch(opt, "Etag: " CORRUPT_LONG_ETAG, 106,
"Last-Modified: " /* 15 + 91 = 106 */
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
1, 1);
failures += corrupt_expect_victim_clipped(
opt, (size_t) -1, sizeof(((htsblk *) 0)->lastmodified) - 1,
"over-long Last-Modified");
/* An X-Size above INT_MAX is positive as int64 (slips a bare sign check) but
truncates negative in the (int) cast the malloc uses: a wraparound alloc.
cache_add asserts size fits an int, so such a value only reaches the reader

View File

@@ -1,95 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: command line splitter, shared by the engine and */
/* htsserver */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#include "htscmdline.h"
#include "htssafe.h"
#include <limits.h>
#include <stdint.h>
char **hts_split_cmdline(char *cmd, int *nargs) {
size_t nsep = 0;
size_t capacity;
size_t r;
size_t w;
int argc = 0;
hts_boolean quoted = HTS_FALSE;
char **argv;
*nargs = 0;
/* fold the other separators, so counting them sizes the vector exactly */
for (r = 0; cmd[r] != '\0'; r++) {
if (cmd[r] == '\t' || cmd[r] == '\r' || cmd[r] == '\n') {
cmd[r] = ' ';
}
if (cmd[r] == ' ') {
nsep++;
}
}
/* at most one argument per separator, plus the leading one and the NULL */
if (nsep > (size_t) INT_MAX - 1 || nsep > SIZE_MAX / sizeof(char *) - 2) {
return NULL;
}
capacity = nsep + 2;
argv = (char **) malloct(capacity * sizeof(char *));
if (argv == NULL) {
return NULL;
}
argv[argc++] = cmd;
for (r = 0, w = 0; cmd[r] != '\0';) {
if (quoted && cmd[r] == '\\' &&
(cmd[r + 1] == '\\' || cmd[r + 1] == '\"')) {
r++;
cmd[w++] = cmd[r++];
} else if (cmd[r] == '\"') {
quoted = !quoted;
cmd[w++] = cmd[r++];
} else if (cmd[r] == ' ' && !quoted) {
cmd[w++] = '\0';
assertf((size_t) argc < capacity - 1); /* the last slot holds the NULL */
argv[argc++] = cmd + w;
r++;
} else {
cmd[w++] = cmd[r++];
}
}
cmd[w] = '\0';
argv[argc] = NULL; /* callers may rely on argv[argc] == NULL */
*nargs = argc;
return argv;
}

View File

@@ -1,45 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: command line splitter, shared by the engine and */
/* htsserver */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTSCMDLINE_DEFH
#define HTSCMDLINE_DEFH
#include "htsglobal.h"
/* Split "cmd" in place into a NULL-terminated argv vector of *nargs entries,
argv[0] being the program name and quotes left for the engine to strip.
Returns a malloct'ed vector of pointers into cmd (freet the vector, never its
entries), or NULL when it cannot be sized or allocated. */
char **hts_split_cmdline(char *cmd, int *nargs);
#endif

View File

@@ -447,22 +447,13 @@ void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
const char *fil) {
if (!*makeindex_done) {
if (*makeindex_fp) {
/* sized off link_escaped below: at the old flat 1024 a long first link
produced a redirect to a clipped URL */
char BIGSTK tempo[HTS_URLMAXSIZE * 2 + 64];
char BIGSTK tempo[1024];
if (makeindex_links == 1) {
char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
/* no redirect beats one pointing at a clipped URL */
if (!sprintfbuff(
tempo,
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped)) {
hts_log_print(opt, LOG_WARNING,
"index redirect omitted: first link too long (%s)",
makeindex_firstlink);
tempo[0] = '\0';
}
snprintf(tempo, sizeof(tempo),
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped);
} else
tempo[0] = '\0';
hts_template_format(*makeindex_fp, template_footer,

View File

@@ -251,7 +251,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// folding a nonsense port into 1..65535 fetches one the link never named;
// an empty "host:" just means the default (#614)
if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
htsblk_failf(&back->r, "Invalid port: %s", a + 1);
snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
_HALT_FTP return 0;
}
@@ -262,7 +262,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// récupérer adresse résolue
strcpybuff(back->info, "host name");
if (hts_dns_resolve2(opt, _adr, &server, &error) == NULL) {
htsblk_failf(&back->r, "Unable to get server's address: %s", error);
snprintf(back->r.msg, sizeof(back->r.msg),
"Unable to get server's address: %s", error);
back->r.statuscode = STATUSCODE_NON_FATAL;
_HALT_FTP return 0;
}
@@ -331,15 +332,18 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
} else {
htsblk_failf(&back->r, "Bad password: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "Bad password: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
htsblk_failf(&back->r, "Bad user name: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "Bad user name: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
htsblk_failf(&back->r, "Connection refused: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "Connection refused: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
@@ -406,7 +410,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
// -- fin analyse de l'adresse IP et du port --
} else {
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
@@ -437,11 +442,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
}
}
} else {
htsblk_failf(&back->r, "EPSV incorrect: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "EPSV incorrect: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
htsblk_failf(&back->r, "PASV/EPSV error: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "PASV/EPSV error: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
}
@@ -547,8 +554,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
deletesoc(soc_dat);
soc_dat = INVALID_SOCKET;
//
htsblk_failf(&back->r, "RETR command error: %s",
linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg),
"RETR command error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
@@ -566,12 +573,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
htsblk_failf(&back->r, "Unable to resolve IP %s: %s", adr_ip,
error);
snprintf(back->r.msg, sizeof(back->r.msg),
"Unable to resolve IP %s: %s", adr_ip, error);
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
} else {
htsblk_failf(&back->r, "PASV incorrect: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "PASV incorrect: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
} // sinon on est prêts
#else
@@ -595,11 +603,13 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
htsblk_failf(&back->r, "RETR command error: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg),
"RETR command error: %s", linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {
htsblk_failf(&back->r, "PORT command error: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "PORT command error: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
#ifdef _WIN32
@@ -642,7 +652,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
len = 0; // fin
break;
case 0:
htsblk_failf(&back->r, "Time out (%d)", timeout);
snprintf(back->r.msg, sizeof(back->r.msg), "Time out (%d)",
timeout);
back->r.statuscode = STATUSCODE_INVALID;
len = 0; // fin
break;
@@ -705,7 +716,8 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
strcpybuff(back->r.msg, "OK");
back->r.statuscode = HTTP_OK;
} else {
htsblk_failf(&back->r, "RETR incorrect: %s", linejmp(line));
snprintf(back->r.msg, sizeof(back->r.msg), "RETR incorrect: %s",
linejmp(line));
back->r.statuscode = STATUSCODE_INVALID;
}
} else {

View File

@@ -72,8 +72,7 @@ Please visit our Website: http://www.httrack.com
HTS_UNUSED: suppress unused-symbol warnings. HTS_STATIC: an unused-safe
static. HTS_PRINTF_FUN(fmt, arg): mark a printf-like function so the
compiler type-checks the format string at argument index fmt against the
varargs starting at arg. HTS_CHECK_RESULT: the return value carries the only
error signal, so dropping it is a bug; a (void) cast does not silence it. */
varargs starting at arg. */
#ifndef HTS_UNUSED
#ifdef __GNUC__
#define HTS_UNUSED __attribute__((unused))
@@ -81,13 +80,10 @@ Please visit our Website: http://www.httrack.com
#define HTS_STATIC static __attribute__((unused))
#define HTS_PRINTF_FUN(fmt, arg) __attribute__((format(printf, fmt, arg)))
#define HTS_CHECK_RESULT __attribute__((warn_unused_result))
#else
#define HTS_UNUSED
#define HTS_STATIC static
#define HTS_PRINTF_FUN(fmt, arg)
#define HTS_CHECK_RESULT
#endif
#endif

View File

@@ -124,9 +124,7 @@ typedef struct help_wizard_buffers {
char stropt[2048]; // options
char stropt2[2048]; // options longues
char strwild[2048]; // wildcards
/* holds all four of the above plus separators: at 4096 a long answer set
clipped the filters off the command line */
char cmd[HTS_URLMAXSIZE * 2 + 3 * 2048 + 4];
char cmd[4096];
char str[256];
char *argv[256];
} help_wizard_buffers;
@@ -310,14 +308,7 @@ void help_wizard(httrackp * opt) {
printf("\n");
if (strlen(stropt) == 1)
stropt[0] = '\0'; // aucune
/* the tail is the filter list, and cmd is split into the argv handed to
hts_main() below: a clipped line would silently widen the crawl */
if (!sprintfbuff(cmd, "%s %s %s %s", urls, stropt, stropt2, strwild)) {
printf("* command line too long (%d bytes max)\n",
(int) sizeof(cmd) - 1);
freet(buffers);
return;
}
snprintf(cmd, sizeof(cmd), "%s %s %s %s", urls, stropt, stropt2, strwild);
printf("---> Wizard command line: httrack %s\n\n", cmd);
printf("Ready to launch the mirror? (Y/n) :");
fflush(stdout);
@@ -433,8 +424,7 @@ void help_catchurl(const char *dest_path) {
}
// former URL!
{
/* url and dest are each HTS_URLMAXSIZE*2, plus the POSTTOK marker */
char BIGSTK finalurl[HTS_URLMAXSIZE * 4 + 32];
char BIGSTK finalurl[HTS_URLMAXSIZE * 2];
inplace_escape_check_url(dest, sizeof(dest));
snprintf(finalurl, sizeof(finalurl), "%s" POSTTOK "file:%s", url, dest);

View File

@@ -708,11 +708,11 @@ T_SOC http_xfopen(httrackp *opt, int mode, int treat, int waitconnect,
#ifdef _WIN32
int last_errno = WSAGetLastError();
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
#else
int last_errno = errno;
htsblk_failf(retour, "Connect error: %s", strerror(last_errno));
sprintf(retour->msg, "Connect error: %s", strerror(last_errno));
#endif
}
}
@@ -2240,13 +2240,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
#ifdef _WIN32
int last_errno = WSAGetLastError();
htsblk_failf(retour, "Unable to create a socket: %s",
strerror(last_errno));
sprintf(retour->msg, "Unable to create a socket: %s",
strerror(last_errno));
#else
int last_errno = errno;
htsblk_failf(retour, "Unable to create a socket: %s",
strerror(last_errno));
sprintf(retour->msg, "Unable to create a socket: %s",
strerror(last_errno));
#endif
}
return INVALID_SOCKET; // erreur création socket impossible
@@ -2312,13 +2312,13 @@ T_SOC newhttp_addr(httrackp *opt, const char *_iadr, htsblk *retour, int port,
#ifdef _WIN32
const int last_errno = WSAGetLastError();
htsblk_failf(retour, "Unable to connect to the server: %s",
strerror(last_errno));
sprintf(retour->msg, "Unable to connect to the server: %s",
strerror(last_errno));
#else
const int last_errno = errno;
htsblk_failf(retour, "Unable to connect to the server: %s",
strerror(last_errno));
sprintf(retour->msg, "Unable to connect to the server: %s",
strerror(last_errno));
#endif
}
/* Close the socket and notify the error!!! */

View File

@@ -199,11 +199,6 @@ T_SOC newhttp(httrackp * opt, const char *iadr, htsblk * retour, int port,
etc.). */
T_SOC newhttp_addr(httrackp *opt, const char *iadr, htsblk *retour, int port,
int waitconnect, int addr_index, int *addr_count);
/* Clips the formatted failure reason into r->msg, which also round-trips
through the cache as X-StatusMessage. Leaves r->statuscode to the caller. */
#define htsblk_failf(R, ...) \
slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
HTS_INLINE void deletehttp(htsblk * r);
HTS_INLINE int deleteaddr(htsblk * r);
HTS_INLINE void deletesoc(T_SOC soc);

View File

@@ -173,8 +173,8 @@ int http_proxy_tunnel(httrackp *opt, htsblk *retour, const char *adr,
if (sscanf(line, "HTTP/%*d.%*d %d", &code) < 1)
code = 0;
if (code < 200 || code >= 300) {
htsblk_failf(retour, "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
snprintf(retour->msg, sizeof(retour->msg), "proxy CONNECT refused: %s",
strnotempty(line) ? line : "(no status)");
return 0;
}

View File

@@ -33,7 +33,6 @@ Please visit our Website: http://www.httrack.com
#ifndef HTSSAFE_DEFH
#define HTSSAFE_DEFH
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -460,85 +459,6 @@ static HTS_INLINE HTS_UNUSED const char *htsbuff_str(const htsbuff *b) {
return b->buf;
}
/**
* Copy src into dest (capacity size, NUL included), truncating to fit and
* always NUL-terminating. Unlike strlcpybuff() it never aborts, so it suits a
* value read back from a cache, a header or the wire, where refusing the whole
* record is worse than keeping a clipped one. Returns HTS_TRUE if it all fit;
* callers that clip on purpose ignore that, so it is not HTS_CHECK_RESULT.
*/
static HTS_INLINE HTS_UNUSED hts_boolean strclipbuff(char *dest, size_t size,
const char *src) {
size_t len, copy;
assertf(dest != NULL && src != NULL && size != 0);
len = strlen(src);
copy = len < size ? len : size - 1;
memcpy(dest, src, copy);
dest[copy] = '\0';
return copy == len ? HTS_TRUE : HTS_FALSE;
}
/**
* Callers that deliberately ignore truncation use this instead of
* slprintfbuff(), so it is not HTS_CHECK_RESULT.
*/
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 0) hts_boolean
vslprintfbuff(char *dest, size_t size, const char *fmt, va_list args) {
int ret;
assertf(dest != NULL && size != 0);
/* a vsnprintf failing outright may write nothing at all, leaving whatever
the caller had on the stack for it to publish */
dest[0] = '\0';
ret = vsnprintf(dest, size, fmt, args);
/* pre-C99 runtimes (msvcrt _vsnprintf) return -1 and do not terminate */
dest[size - 1] = '\0';
return ret >= 0 && (size_t) ret < size ? HTS_TRUE : HTS_FALSE;
}
/**
* Formatted print into dest (capacity size, NUL included), truncating to fit
* and always NUL-terminating. Returns HTS_TRUE if the whole output fit; the
* result is the only truncation signal, so it must be acted on. Unlike
* strcpybuff() it never aborts, so it suits text built from remote input.
*/
static HTS_INLINE HTS_UNUSED HTS_CHECK_RESULT HTS_PRINTF_FUN(3, 4) hts_boolean
slprintfbuff(char *dest, size_t size, const char *fmt, ...) {
va_list args;
hts_boolean ret;
va_start(args, fmt);
ret = vslprintfbuff(dest, size, fmt, args);
va_end(args);
return ret;
}
/**
* slprintfbuff() for diagnostics quoting remote or client text, which are
* meant to be clipped: nothing to act on, hence not HTS_CHECK_RESULT. A (void)
* cast on slprintfbuff() is no substitute, GCC warns through it.
*/
static HTS_INLINE HTS_UNUSED HTS_PRINTF_FUN(3, 4) void slprintfbuff_clip(
char *dest, size_t size, const char *fmt, ...) {
va_list args;
va_start(args, fmt);
(void) vslprintfbuff(dest, size, fmt, args);
va_end(args);
}
/**
* slprintfbuff() over the in-scope array ARR (capacity = sizeof(ARR)).
* On GCC/Clang a pointer is a compile error; use slprintfbuff() for those.
*/
#if (defined(__GNUC__) && !defined(__cplusplus))
#define sprintfbuff(ARR, ...) \
slprintfbuff((ARR), sizeof(ARR) + htsbuff_must_be_array_(ARR), __VA_ARGS__)
#else
#define sprintfbuff(ARR, ...) slprintfbuff((ARR), sizeof(ARR), __VA_ARGS__)
#endif
/* Thin aliases over the libc allocator/memcpy (historical "t" suffix); no
added bounds checking. freet() also NULLs the freed pointer and tolerates
NULL. memcpybuff() despite the name is a raw memcpy: the caller owns the

View File

@@ -43,7 +43,6 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htsback.h"
#include "htsdefines.h"
#include "htslib.h"
#include "htsalias.h"
@@ -52,7 +51,6 @@ Please visit our Website: http://www.httrack.com
#include "htscache_selftest.h"
#include "htsdns_selftest.h"
#include "htscharset.h"
#include "htscmdline.h"
#include "htsencoding.h"
#include "htsftp.h"
#include "htsmd5.h"
@@ -457,12 +455,6 @@ static void basic_selftests(void) {
// link one level up -> a "../" prefix
assertf(lienrelatif(s, sizeof(s), "a.html", "dir/index.html") == 0);
assertf(strcmp(s, "../a.html") == 0);
// an empty current path: the trim used to walk off the front of it, which
// "?x" reaches too because the query pre-pass hands on the part before it
assertf(lienrelatif(s, sizeof(s), "dir/page.html", "") == 0);
assertf(strcmp(s, "dir/page.html") == 0);
assertf(lienrelatif(s, sizeof(s), "dir/page.html", "?x") == 0);
assertf(strcmp(s, "dir/page.html") == 0);
}
}
@@ -576,214 +568,6 @@ static int string_safety_selftests(void) {
return 1;
}
/* sprintfbuff: truncate-and-report. Must never abort (its callers format
remote banners) nor write past the array, which the canary catches. */
{
struct {
char dst[8];
char canary[8];
} s;
const char *const big = "0123456789abcdefghijklmnopqrstuvwxyz";
/* repoison before every call, or an implementation that measures first and
writes nothing "passes" the truncating cases on the previous content */
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
memset(&s, '#', sizeof(s));
if (!sprintfbuff(s.dst, "%s-%d", "ab", 42) || strcmp(s.dst, "ab-42") != 0)
return 1;
/* exact fit: 7 characters plus the NUL */
POISON_DST();
if (!sprintfbuff(s.dst, "%s", "1234567") || strcmp(s.dst, "1234567") != 0)
return 1;
/* one over, then far over: truncated to the prefix, terminated, reported */
POISON_DST();
if (sprintfbuff(s.dst, "%s", "12345678") || strcmp(s.dst, "1234567") != 0)
return 1;
POISON_DST();
if (sprintfbuff(s.dst, "%s", big) || strcmp(s.dst, "0123456") != 0)
return 1;
/* explicit-capacity form, down to the degenerate size 1 */
{
char *const p = s.dst;
POISON_DST();
if (slprintfbuff(p, 1, "%s", "x") || p[0] != '\0')
return 1;
POISON_DST();
if (!slprintfbuff(p, sizeof(s.dst), "%s", "ok") || strcmp(p, "ok") != 0)
return 1;
}
#undef POISON_DST
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
return 1;
}
/* strclipbuff: truncate-and-report, never abort. Same canary shape; the
destination is poisoned before every call so a case cannot pass on the
previous one's bytes. */
{
struct {
char dst[8];
char canary[8];
} s;
memset(&s, '#', sizeof(s));
#define POISON_DST() memset(s.dst, '#', sizeof(s.dst))
/* well under capacity: no padding, nothing eaten off the end */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "abc") || strcmp(s.dst, "abc") != 0)
return 1;
/* exact fit: capacity - 1 characters plus the NUL */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "1234567") ||
strcmp(s.dst, "1234567") != 0)
return 1;
/* one over, then far over: clipped, terminated, reported. The expected
bytes differ from the case above, so a write-nothing implementation
cannot pass on the leftovers. */
POISON_DST();
if (strclipbuff(s.dst, sizeof(s.dst), "abcdefgh") ||
strcmp(s.dst, "abcdefg") != 0)
return 1;
POISON_DST();
if (strclipbuff(s.dst, sizeof(s.dst), "0123456789abcdef") ||
strcmp(s.dst, "0123456") != 0)
return 1;
/* degenerate capacity 1: only the NUL fits. Capacity 2 pins the boundary
between that and the sizes above: one character plus the NUL. */
POISON_DST();
if (strclipbuff(s.dst, 1, "x") || s.dst[0] != '\0')
return 1;
POISON_DST();
if (strclipbuff(s.dst, 2, "yz") || strcmp(s.dst, "y") != 0)
return 1;
/* the empty string fits any non-zero capacity */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "") || s.dst[0] != '\0')
return 1;
/* a byte over 0x7f must not end the copy early */
POISON_DST();
if (!strclipbuff(s.dst, sizeof(s.dst), "\xff\xfe") ||
strcmp(s.dst, "\xff\xfe") != 0)
return 1;
#undef POISON_DST
if (memcmp(s.canary, "########", sizeof(s.canary)) != 0)
return 1;
}
/* htsblk_failf: clips a reason quoted from a remote reply into msg[] and
touches nothing else in the block */
{
htsblk r;
char expect[sizeof(r.msg)];
char big[4 * sizeof(r.msg)];
/* contenttype abuts msg, so a one-past-the-end store lands in it rather
than in padding. Poison it: a stray NUL is invisible against zeroes,
and a stray NUL is exactly what an off-by-one terminator writes. */
#define NEIGHBOURS_INTACT() (r.contenttype[0] == '#' && r.statuscode == 1234)
memset(&r, 0, sizeof(r));
memset(r.contenttype, '#', sizeof(r.contenttype));
r.statuscode = 1234;
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "PASV incorrect: %s", "220 ok");
if (strcmp(r.msg, "PASV incorrect: 220 ok") != 0 || !NEIGHBOURS_INTACT())
return 1;
/* exact fit: capacity - 1 characters plus the NUL */
memset(expect, 'y', sizeof(expect) - 1);
expect[sizeof(expect) - 1] = '\0';
memcpy(expect, "Bad password: ", sizeof("Bad password: ") - 1);
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "%s", expect);
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
return 1;
/* far over: the expected bytes differ from the cases above, so writing
nothing cannot pass on the leftovers */
memset(big, 'z', sizeof(big) - 1);
big[sizeof(big) - 1] = '\0';
memset(expect, 'z', sizeof(expect) - 1);
expect[sizeof(expect) - 1] = '\0';
memcpy(expect, "Bad user name: ", sizeof("Bad user name: ") - 1);
memset(r.msg, '#', sizeof(r.msg));
htsblk_failf(&r, "Bad user name: %s", big);
if (strcmp(r.msg, expect) != 0 || !NEIGHBOURS_INTACT())
return 1;
#undef NEIGHBOURS_INTACT
}
/* back_read_ftp_result: the helper's result file is external input, so an
over-long message must stop at msg[]'s capacity */
{
htsblk r;
size_t k;
/* poisoned so a short message cannot pass on leftovers, and so a stray
NUL past msg[] is visible in the neighbour */
#define FTP_RESULT_CASE(BODY) \
do { \
FILE *fp_ = tmpfile(); \
\
if (fp_ == NULL) \
return 1; \
BODY; \
rewind(fp_); \
memset(&r, 0, sizeof(r)); \
memset(r.msg, '#', sizeof(r.msg)); \
memset(r.contenttype, '#', sizeof(r.contenttype)); \
back_read_ftp_result(fp_, &r); \
fclose(fp_); \
if (r.contenttype[0] != '#') \
return 1; \
} while (0)
/* over capacity: clipped to 79 payload bytes plus the NUL */
FTP_RESULT_CASE({
fprintf(fp_, "226 ");
for (k = 0; k < 4 * sizeof(r.msg); k++)
fputc('q', fp_);
});
if (r.statuscode != 226 || strlen(r.msg) != sizeof(r.msg) - 1)
return 1;
for (k = 0; k < sizeof(r.msg) - 1; k++) {
if (r.msg[k] != 'q')
return 1;
}
/* well under capacity: nothing padded, nothing eaten off the end */
FTP_RESULT_CASE(fprintf(fp_, "550 no such file"));
if (r.statuscode != 550 || strcmp(r.msg, "no such file") != 0)
return 1;
/* a byte over 0x7f must not read as EOF and cut the message short */
FTP_RESULT_CASE(fprintf(fp_, "226 \xff ok"));
if (r.statuscode != 226 || strcmp(r.msg, "\xff ok") != 0)
return 1;
/* unparseable status: the message still loads, the code reports failure */
FTP_RESULT_CASE(fprintf(fp_, "not-a-number here"));
if (r.statuscode != STATUSCODE_INVALID)
return 1;
#undef FTP_RESULT_CASE
}
/* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
n_eval == 2 on a double-eval macro. */
{
@@ -1342,128 +1126,6 @@ static int st_unescape_bounds(httrackp *opt, int argc, char **argv) {
return 0;
}
// hts_split_cmdline(): the vector must grow with the argument count, and a
// quote inside a value must not end the argument and hand -V to the parser.
static int st_cmdlinesplit(httrackp *opt, int argc, char **argv) {
char line[512];
char **args;
int nargs = 0;
(void) opt;
(void) argc;
(void) argv;
// control: every separator splits, and argv[0] is the program name
strcpybuff(line, "httrack http://x/ --quiet\t-c8\n-O out");
args = hts_split_cmdline(line, &nargs);
assertf(args != NULL && nargs == 6);
assertf(args[nargs] == NULL); // callers may walk to the terminator
assertf(strcmp(args[0], "httrack") == 0);
assertf(strcmp(args[1], "http://x/") == 0);
assertf(strcmp(args[2], "--quiet") == 0);
assertf(strcmp(args[3], "-c8") == 0);
assertf(strcmp(args[4], "-O") == 0);
assertf(strcmp(args[5], "out") == 0);
freet(args);
// the template pads with whitespace: empty arguments are kept (the engine
// skips them), so the count is one per separator
strcpybuff(line, "httrack --quiet");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3 && args[1][0] == '\0');
assertf(strcmp(args[2], "--quiet") == 0);
freet(args);
// a quoted run keeps both its spaces and its quotes: the engine unquotes
strcpybuff(line, "httrack --user-agent \"Mozilla 5.0\" -c8");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 4);
assertf(strcmp(args[2], "\"Mozilla 5.0\"") == 0);
assertf(strcmp(args[3], "-c8") == 0);
freet(args);
// an escaped quote is a literal quote, not the end of the argument: the
// engine strips only the outer pair
strcpybuff(line, "httrack --user-agent \"x\\\" -V \\\"touch /tmp/pwn\" -c8");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 4);
assertf(strcmp(args[2], "\"x\" -V \"touch /tmp/pwn\"") == 0);
assertf(strcmp(args[3], "-c8") == 0);
freet(args);
// \\ is a literal backslash, so a Windows path survives
strcpybuff(line, "httrack --path \"C:\\\\dir\\\\sub\"");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[2], "\"C:\\dir\\sub\"") == 0);
freet(args);
// outside a quoted run a backslash is literal: the url and wildcard-filter
// fields, which the wizard cannot quote, read as before
strcpybuff(line, "httrack -*\\** +*.png");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[1], "-*\\**") == 0);
assertf(strcmp(args[2], "+*.png") == 0);
freet(args);
// a quoted run leaves slots unused, so the terminator has to be written and
// not inherited: size the vector from a full line first, so freeing it hands
// the same chunk back with stale pointers in those slots
strcpybuff(line, "httrack a b c d e");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 6);
freet(args);
strcpybuff(line, "httrack \"a b c d e\"");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 2);
assertf(args[nargs] == NULL);
freet(args);
// an unterminated quote protects the rest of the line, as one argument
strcpybuff(line, "httrack --footer \"unbalanced -V x");
args = hts_split_cmdline(line, &nargs);
assertf(nargs == 3);
assertf(strcmp(args[2], "\"unbalanced -V x") == 0);
freet(args);
// past the 1024 entries the vector used to hold: distinct arguments, so a
// write beyond the allocation cannot read back as the expected parse
{
const int n = 2000;
const size_t size = 16 * (size_t) n + 16;
char *big = malloct(size);
size_t pos = 0;
int i;
assertf(big != NULL);
pos = (size_t) snprintf(big, size, "httrack");
assertf(pos < size);
for (i = 0; i < n; i++) {
// snprintf returns what it wanted to write, so accumulating it blind
// would let the next size argument wrap
const int len = snprintf(big + pos, size - pos, " a%d", i);
assertf(len > 0 && (size_t) len < size - pos);
pos += (size_t) len;
}
args = hts_split_cmdline(big, &nargs);
assertf(args != NULL && nargs == n + 1);
assertf(args[nargs] == NULL);
for (i = 0; i < n; i++) {
char expect[16];
snprintf(expect, sizeof(expect), "a%d", i);
assertf(strcmp(args[i + 1], expect) == 0);
}
freet(args);
freet(big);
}
printf("cmdline-split self-test OK\n");
return 0;
}
static int st_hashtable(httrackp *opt, int argc, char **argv) {
char *snum;
unsigned long count = 0;
@@ -1585,7 +1247,7 @@ static int st_hashtable(httrackp *opt, int argc, char **argv) {
size_t i;
for (i = bench[loop].offset; i < (size_t) count;
i += bench[loop].modulus) {
int result = 0; /* no final else: an unknown type reports failure */
int result;
FMT();
if (bench[loop].type == DO_ADD || bench[loop].type == DO_DRY_ADD) {
size_t k;
@@ -3100,32 +2762,6 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
assertf(strstr(buf, "Refresh") != NULL);
assertf(strstr(buf, "example.com") != NULL);
/* a first link whose escaped form overruns the old flat 1024-byte tempo: the
redirect must carry the whole URL, not a clipped prefix */
{
char BIGSTK link[HTS_URLMAXSIZE * 2];
char *p = link;
strcpybuff(link, "http://example.com/");
p += strlen(link);
memset(p, 'a', 1200);
p += 1200;
strcpy(p, "/end.html");
done = 0;
fp = fopen(path, "wb");
assertf(fp != NULL);
hts_finish_makeindex(opt, &done, &fp, 1, link, "%s%s", "", "");
assertf(fp == NULL);
fp = fopen(path, "rb");
assertf(fp != NULL);
n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
buf[n] = '\0';
/* the closing quote proves the URL was not clipped mid-way */
assertf(strstr(buf, "/end.html\">") != NULL);
}
/* no single link: footer only, no refresh meta */
done = 0;
fp = fopen(path, "wb");
@@ -5165,10 +4801,10 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
char fpath[HTS_URLMAXSIZE * 2];
char file[HTS_URLMAXSIZE * 2];
assertf(sprintfbuff(fpath, "%s/", dir)); /* IE glob wants a trailing sep */
snprintf(fpath, sizeof(fpath), "%s/", dir); /* IE glob wants a trailing sep */
/* cookies.txt: one Netscape record (host, _, path, _, _, name, value). */
assertf(sprintfbuff(file, "%scookies.txt", fpath));
snprintf(file, sizeof(file), "%scookies.txt", fpath);
{
FILE *fp = FOPEN(file, "wb");
@@ -5178,7 +4814,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
}
/* A copied IE cookie u@v.txt: name, value, url, then 6 unused fields. */
assertf(sprintfbuff(file, "%su@v.txt", fpath));
snprintf(file, sizeof(file), "%su@v.txt", fpath);
{
FILE *fp = FOPEN(file, "wb");
@@ -5200,7 +4836,7 @@ static int st_cookieimport(httrackp *opt, int argc, char **argv) {
#endif
(void) UNLINK(file); /* u@v.txt (already gone on Windows) */
assertf(sprintfbuff(file, "%scookies.txt", fpath));
snprintf(file, sizeof(file), "%scookies.txt", fpath);
(void) UNLINK(file);
dir[dirlen] = '\0';
while (strlen(dir) > base) {
@@ -5274,9 +4910,6 @@ static const struct selftest_entry {
st_footerfmt},
{"unescape-bounds", "", "unescapers reserve the NUL byte (no 1-byte OOB)",
st_unescape_bounds},
{"cmdline-split", "",
"webhttrack command-line to argv split (bounds, quoting)",
st_cmdlinesplit},
{"hashtable", "<count|file>", "coucal hashtable stress test", st_hashtable},
{"strsafe", "[overflow|overflow-buff|overflow-src [str]]",
"bounded string-op self-test", st_strsafe},

View File

@@ -318,18 +318,6 @@ typedef struct {
error_redirect = "/server/error.html"; \
} while(0)
/* Longest error message shown on the error page; the rest is clipped. */
#define ERROR_MESSAGE_MAX 1024
/* SET_ERROR() with a printf format. Clips: these messages quote posted fields,
whose length the client picks. */
#define SET_ERRORF(...) \
do { \
char errbuf[ERROR_MESSAGE_MAX]; \
slprintfbuff_clip(errbuf, sizeof(errbuf), __VA_ARGS__); \
SET_ERROR(errbuf); \
} while (0)
/* Longest "sid" value worth unescaping: the expected one is an md5 hex digest,
so anything near this is already invalid and is rejected unread. */
#define SID_VALUE_MAX 64
@@ -375,59 +363,6 @@ static hts_boolean body_sid_is_valid(const char *body, const char *expected) {
return seen;
}
/** Append src to the NUL-terminated dst of capacity size (NUL included).
False, leaving dst untouched, if it would not fit: unlike strcatbuff() this
never aborts, because every piece appended here is client-supplied. */
static hts_boolean path_append(char *dst, size_t size, const char *src) {
const size_t used = strlen(dst);
const size_t len = strlen(src);
/* dst holds at most size-1 bytes, so "size - used" is >= 1 and the untrusted
len stays alone: "used + len < size" could wrap and pass. */
if (len >= size - used) {
return HTS_FALSE;
}
memcpy(dst + used, src, len + 1);
return HTS_TRUE;
}
/* Append c to dst as an HTML entity, or return HTS_FALSE if it needs none. */
static hts_boolean cat_html_escaped(String *dst, char c) {
switch (c) {
case '<':
StringCat(*dst, "&lt;");
break;
case '>':
StringCat(*dst, "&gt;");
break;
case '&':
StringCat(*dst, "&amp;");
break;
case '\'':
StringCat(*dst, "&#39;");
break;
default:
return HTS_FALSE;
}
return HTS_TRUE;
}
/* Append the value of a double-quoted command-line argument: escaped for HTML,
which the browser undoes when it posts the command line back, and for the
argv splitter, which does not. */
static void cat_cmdline_arg(String *output, const char *value) {
const char *a;
for (a = value; *a != '\0'; a++) {
if (*a == '\\' || *a == '\"') {
StringCat(*output, "\\");
}
if (!cat_html_escaped(output, *a)) {
StringMemcat(*output, a, 1);
}
}
}
int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
int timeout = 30;
int retour = 0;
@@ -439,9 +374,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
String tmpbuff = STRING_EMPTY;
String tmpbuff2 = STRING_EMPTY;
String fspath = STRING_EMPTY;
/* Project directory this server set up; the only root /website/ serves from,
and deliberately not cleared between requests. */
String website = STRING_EMPTY;
char catbuff[CATBUFF_SIZE];
/* Load strings */
@@ -737,7 +669,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
fp = fopen(StringBuff(fspath), "rb");
if (fp) {
/* Read file */
while (!feof(fp) && !ferror(fp)) {
while(!feof(fp)) {
char *str = line;
char *pos;
@@ -853,11 +785,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
if (!structcheck(StringBuff(tmpbuff))) {
FILE *fp;
/* Both halves of fspath come from posted fields, so a ".."
in them would escape the mirror once served. */
if (strstr(StringBuff(fspath), "..") == NULL) {
StringCopy(website, StringBuff(fspath));
}
StringCat(tmpbuff, "winprofile.ini");
fp = fopen(StringBuff(tmpbuff), "wb");
if (fp != NULL) {
@@ -891,18 +818,28 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
commandEnd = 1;
}
} else {
SET_ERRORF(
"Unable to write %d bytes in the the init file %s",
count, StringBuff(fspath));
char tmp[1024];
sprintf(tmp,
"Unable to write %d bytes in the the init file %s",
count, StringBuff(fspath));
SET_ERROR(tmp);
}
fclose(fp);
} else {
SET_ERRORF("Unable to create the init file %s",
StringBuff(fspath));
char tmp[1024];
sprintf(tmp, "Unable to create the init file %s",
StringBuff(fspath));
SET_ERROR(tmp);
}
} else {
SET_ERRORF("Unable to create the directory structure in %s",
StringBuff(fspath));
char tmp[1024];
sprintf(tmp,
"Unable to create the directory structure in %s",
StringBuff(fspath));
SET_ERROR(tmp);
}
} else {
@@ -920,7 +857,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
/* Response */
if (meth) {
hts_boolean virtualpath = HTS_FALSE;
int virtualpath = 0;
char *pos;
char *url = strchr(line1, ' ');
@@ -931,11 +868,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
char *qpos;
/* get the URL */
fsfile[0] = '\0';
if (error_redirect == NULL) {
if ((qpos = strchr(url, '?'))) {
*qpos = '\0';
}
fsfile[0] = '\0';
if (strcmp(url, "/") == 0) {
file = "/server/index.html";
meth = 2;
@@ -948,7 +885,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
if (strncmp(file, "/website/", 9) == 0) {
virtualpath = HTS_TRUE;
virtualpath = 1;
}
/* override */
@@ -962,28 +899,20 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
/* the override above may have swapped a mirror path for a GUI page */
virtualpath = strncmp(file, "/website/", 9) == 0;
if (strlen(path) + strlen(file) + 32 < sizeof(fsfile)) {
if (strncmp(file, "/website/", 9) != 0) {
sprintf(fsfile, "%shtml%s", path, file);
} else {
intptr_t adr = 0;
if (!virtualpath) {
if (!path_append(fsfile, sizeof(fsfile), path) ||
!path_append(fsfile, sizeof(fsfile), "html") ||
!path_append(fsfile, sizeof(fsfile), file)) {
fsfile[0] = '\0';
}
} else if (StringNotEmpty(website)) {
/* Never the posted "projpath": a client root reads any file. */
if (!path_append(fsfile, sizeof(fsfile), StringBuff(website)) ||
!path_append(fsfile, sizeof(fsfile), "/") ||
!path_append(fsfile, sizeof(fsfile), file + 9)) {
fsfile[0] = '\0';
if (coucal_readptr(NewLangList, "projpath", &adr)) {
sprintf(fsfile, "%s%s", (char *) adr, file + 9);
}
}
}
/* Regular files only: reading a directory or FIFO never ends, and
"path" may hold "..", so only the untrusted halves are checked. */
if (fsfile[0] && strstr(file, "..") == NULL && fexist(fsfile) &&
(fp = fopen(fsfile, "rb"))) {
if (fsfile[0] && strstr(file, "..") == NULL
&& (fp = fopen(fsfile, "rb"))) {
char ok[] =
"HTTP/1.0 200 OK\r\n" "Connection: close\r\n"
"Server: httrack-small-server\r\n" "Content-type: text/html\r\n"
@@ -1036,13 +965,11 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringCat(headers, "\r\n");
}
coucal_write(NewLangList, "redirect", (intptr_t) NULL);
} else if (!virtualpath && is_html(file)) {
/* GUI templates only: ${_sid} in a mirrored page would hand the
crawled site the session id that authenticates commands */
} else if (is_html(file)) {
int outputmode = 0;
StringMemcat(headers, ok, sizeof(ok) - 1);
while (!feof(fp) && !ferror(fp)) {
while(!feof(fp)) {
char *str = line;
int prevlen = (int) StringLength(output);
int nocr = 0;
@@ -1066,7 +993,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
int p;
int format = 0;
int listDefault = 0;
hts_boolean unquoted = HTS_FALSE;
name[0] = '\0';
strlncatbuff(name, str, sizeof(name_), n);
@@ -1076,12 +1002,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
} else if ((p = strfield(name, "html:"))) {
name += p;
format = 1;
} else if ((p = strfield(name, "unquoted:"))) {
name += p;
unquoted = HTS_TRUE;
} else if ((p = strfield(name, "arg:"))) {
name += p;
format = 5;
} else if ((p = strfield(name, "list:"))) {
name += p;
format = 2;
@@ -1218,8 +1138,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
test:<if ==0>:<if ==1>:<if == 2>..
ztest:<if == 0 || !exist>:<if == 1>:<if == 2>..
*/
else if ((p = strfield(name, "test:")) ||
(p = strfield(name, "ztest:"))) {
else if ((p = strfield(name, "test:"))
|| (p = strfield(name, "ztest:"))) {
intptr_t adr = 0;
char *pos2;
int ztest = (name[0] == 'z');
@@ -1322,12 +1242,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
}
/* consumed here: it shares nothing with the list and
option formats below */
if (format == 5 && langstr != NULL && outputmode != -1) {
cat_cmdline_arg(&output, langstr);
langstr = NULL;
}
if (langstr && outputmode != -1) {
switch (format) {
case 0:
@@ -1345,18 +1259,18 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringMemcat(output, &c, 1);
}
a += 2;
} else if (unquoted && a[0] == '\"') {
/* the browser posts an entity back as a raw
quote, which would open a quoted run in the
argv splitter; a URI cannot hold one anyway */
StringCat(output, "%22");
} else if (outputmode &&
cat_html_escaped(&output, a[0])) {
/* appended as an entity */
} else if (outputmode && a[0] == '<') {
StringCat(output, "&lt;");
} else if (outputmode && a[0] == '>') {
StringCat(output, "&gt;");
} else if (outputmode && a[0] == '&') {
StringCat(output, "&amp;");
} else if (outputmode && a[0] == '\'') {
StringCat(output, "&#39;");
} else if (outputmode == 3 && a[0] == ' ') {
StringCat(output, "%20");
} else if (outputmode >= 2 &&
((unsigned char) a[0]) < 32) {
} else if (outputmode >= 2
&& ((unsigned char) a[0]) < 32) {
char tmp[32];
sprintf(tmp, "%%%02x", (unsigned char) a[0]);
@@ -1417,10 +1331,20 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
StringClear(tmpbuff);
break;
case '<':
StringCat(tmpbuff, "&lt;");
break;
case '>':
StringCat(tmpbuff, "&gt;");
break;
case '&':
StringCat(tmpbuff, "&amp;");
break;
case '\'':
StringCat(tmpbuff, "&#39;");
break;
default:
if (!cat_html_escaped(&tmpbuff, *fstr)) {
StringMemcat(tmpbuff, fstr, 1);
}
StringMemcat(tmpbuff, fstr, 1);
break;
}
fstr++;
@@ -1460,9 +1384,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
#endif
} else {
if (is_html(file)) {
StringMemcat(headers, ok, sizeof(ok) - 1);
} else if (is_text(file)) {
if (is_text(file)) {
StringMemcat(headers, ok_text, sizeof(ok_text) - 1);
} else if (is_js(file)) {
StringMemcat(headers, ok_js, sizeof(ok_js) - 1);
@@ -1476,15 +1398,14 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
while(!feof(fp)) {
int n = (int) fread(line, 1, sizeof(line) - 2, fp);
if (n <= 0) {
break; /* short read: EOF or error, never a retry */
if (n > 0) {
StringMemcat(output, line, n);
}
StringMemcat(output, line, n);
}
}
fclose(fp);
} else if (strcmp(file, "/ping") == 0 ||
strncmp(file, "/ping?", 6) == 0) {
} else if (strcmp(file, "/ping") == 0
|| strncmp(file, "/ping?", 6) == 0) {
char error_hdr[] =
"HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
"Content-type: text/html\r\n";
@@ -1570,7 +1491,6 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
StringFree(tmpbuff);
StringFree(tmpbuff2);
StringFree(fspath);
StringFree(website);
if (buffer)
free(buffer);

View File

@@ -1,63 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: in-progress display rows, shared by httrack and htsserver .h */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTS_STATS_DEFH
#define HTS_STATS_DEFH
#include "htsglobal.h"
/* One row of the "in progress" display, shared so the CLI (httrack) and the
web GUI (htsserver) cannot drift apart: each fills its own array. */
#define NStatsBuffer 14
#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
#define HTS_DEF_FWSTRUCT_t_StatsBuffer
typedef struct t_StatsBuffer t_StatsBuffer;
#endif
struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[288]; // a short label plus back->info[256]
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
LLint size;
LLint sizetot;
int offset;
//
int back;
//
int actived; // pour disabled
};
#endif

View File

@@ -308,10 +308,8 @@ int lienrelatif(char *s, size_t ssize, const char *link, const char *curr_fil) {
// copy only the current path
curr = _curr;
strlcpybuff(curr, curr_fil, sizeof(_curr));
if ((a = strchr(curr, '?')) == NULL) { // cut at the ? (query parameters)
// an empty path has no last character: curr-1 would read before the buffer
a = curr[0] != '\0' ? curr + strlen(curr) - 1 : curr;
}
if ((a = strchr(curr, '?')) == NULL) // couper au ? (params)
a = curr + strlen(curr) - 1; // pas de params: aller à la fin
while((*a != '/') && (a > curr))
a--; // chercher dernier / du chemin courant
if (*a == '/')

View File

@@ -62,7 +62,6 @@ Please visit our Website: http://www.httrack.com
#include "htsmd5.c"
#include "md5.c"
#include "htscmdline.h"
#include "htsserver.h"
#include "htsurlport.h"
#include "htsweb.h"
@@ -320,8 +319,10 @@ int main(int argc, char *argv[]) {
static int webhttrack_runmain(httrackp * opt, int argc, char **argv);
static void back_launch_cmd(void *pP) {
char *cmd = (char *) pP;
char **argv;
char **argv = (char **) malloct(1024 * sizeof(char *));
int argc = 0;
int i = 0;
int g = 0;
//
httrackp *opt;
@@ -332,19 +333,28 @@ static void back_launch_cmd(void *pP) {
commandReturnCmdl = strdup(cmd);
/* split */
argv = hts_split_cmdline(cmd, &argc);
if (argv == NULL) {
if (commandReturnMsg)
free(commandReturnMsg);
commandReturnMsg = strdup("could not parse the command line");
commandReturn = -1;
commandRunning = 0;
commandEnd = 1;
free(cmd);
return;
argv[0] = strdup("webhttrack");
argv[1] = cmd;
argc++;
i = 0;
while(cmd[i]) {
if (cmd[i] == '\t' || cmd[i] == '\r' || cmd[i] == '\n') {
cmd[i] = ' ';
}
i++;
}
i = 0;
while(cmd[i]) {
if (cmd[i] == '\"')
g = !g;
if (cmd[i] == ' ') {
if (!g) {
cmd[i] = '\0';
argv[argc++] = cmd + i + 1;
}
}
i++;
}
/* drop the program name the posted command line carries */
argv[0] = strdupt("webhttrack");
/* init */
hts_init();
@@ -373,7 +383,6 @@ static void back_launch_cmd(void *pP) {
/* free */
free(cmd);
freet(argv[0]);
freet(argv);
return;
}

View File

@@ -35,10 +35,26 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htsstats.h"
#define NStatsBuffer 14
#define MAX_LEN_INPROGRESS 40
typedef struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[256];
char url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char url_adr[HTS_URLMAXSIZE * 2];
char url_fil[HTS_URLMAXSIZE * 2];
LLint size;
LLint sizetot;
int offset;
//
int back;
//
int actived; // pour disabled
} t_StatsBuffer;
typedef struct t_InpInfo {
int ask_refresh;
int refresh;

View File

@@ -230,7 +230,8 @@ static hts_boolean vt_size_refresh(void) {
*/
#define STYLE_STATVALUES VT_BOLD
#define STYLE_STATTEXT VT_UNBOLD
#define STYLE_STATRESET VT_UNBOLD
#define STYLE_STATRESET VT_UNBOLD
#define NStatsBuffer 14
/* Rows the stats block and "Current job" take above the in-progress list. */
#define NStatsHeaderRows 7

View File

@@ -36,7 +36,26 @@ Please visit our Website: http://www.httrack.com
#include "htsglobal.h"
#include "htscore.h"
#include "htssafe.h"
#include "htsstats.h"
#ifndef HTS_DEF_FWSTRUCT_t_StatsBuffer
#define HTS_DEF_FWSTRUCT_t_StatsBuffer
typedef struct t_StatsBuffer t_StatsBuffer;
#endif
struct t_StatsBuffer {
char name[1024];
char file[1024];
char state[256];
char BIGSTK url_sav[HTS_URLMAXSIZE * 2]; // pour cancel
char BIGSTK url_adr[HTS_URLMAXSIZE * 2];
char BIGSTK url_fil[HTS_URLMAXSIZE * 2];
LLint size;
LLint sizetot;
int offset;
//
int back;
//
int actived; // pour disabled
};
#ifndef HTS_DEF_FWSTRUCT_t_InpInfo
#define HTS_DEF_FWSTRUCT_t_InpInfo

View File

@@ -122,7 +122,6 @@
<ClCompile Include="htshelp.c" />
<ClCompile Include="htsindex.c" />
<ClCompile Include="htslib.c" />
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="htsmodules.c" />

View File

@@ -688,7 +688,7 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
PT_Element elt = PT_ElementNew();
elt->statuscode = 405;
strcpybuff(elt->msg, "Method Not Allowed");
strcpy(elt->msg, "Method Not Allowed");
return elt;
}
@@ -820,9 +820,9 @@ static PT_Element proxytrack_process_DAV_Request(PT_Indexes indexes,
elt->size = StringLength(response);
elt->adr = StringAcquire(&response);
elt->statuscode = 207; /* Multi-Status */
strcpybuff(elt->charset, "utf-8");
strcpybuff(elt->contenttype, "text/xml");
strcpybuff(elt->msg, "Multi-Status");
strcpy(elt->charset, "utf-8");
strcpy(elt->contenttype, "text/xml");
strcpy(elt->msg, "Multi-Status");
StringFree(response);
fprintf(stderr, "RESPONSE:\n%s\n", elt->adr);
@@ -877,9 +877,9 @@ static PT_Element proxytrack_process_HTTP_List(PT_Indexes indexes,
elt->size = StringLength(html);
elt->adr = StringAcquire(&html);
elt->statuscode = HTTP_OK;
strcpybuff(elt->charset, "iso-8859-1");
strcpybuff(elt->contenttype, "text/html");
strcpybuff(elt->msg, "OK");
strcpy(elt->charset, "iso-8859-1");
strcpy(elt->contenttype, "text/html");
strcpy(elt->msg, "OK");
StringFree(html);
return elt;
}

View File

@@ -368,11 +368,7 @@ HTS_UNUSED static struct tm PT_GetTime(time_t t) {
if (tm != NULL)
return *tm;
else {
/* an all-zero tm has tm_mday == 0, which the ARC date field prints as a
day of "00"; the epoch is the conventional "date unknown" */
memset(&tmbuf, 0, sizeof(tmbuf));
tmbuf.tm_year = 70;
tmbuf.tm_mday = 1;
return tmbuf;
}
}

View File

@@ -405,9 +405,9 @@ PT_Element PT_Index_HTML_BuildRootInfo(PT_Indexes indexes) {
elt->size = StringLength(html);
elt->adr = StringAcquire(&html);
elt->statuscode = HTTP_OK;
strcpybuff(elt->charset, "iso-8859-1");
strcpybuff(elt->contenttype, "text/html");
strcpybuff(elt->msg, "OK");
strcpy(elt->charset, "iso-8859-1");
strcpy(elt->contenttype, "text/html");
strcpy(elt->msg, "OK");
StringFree(html);
return elt;
}
@@ -579,10 +579,9 @@ PT_Index PT_LoadCache(const char *filename) {
if (chain != NULL) {
const char *scheme = link_has_authority(chain->name) ? "" : "http://";
/* dropped rather than truncated: empty already reads as "unset" */
if (!sprintfbuff(index->slots.common.startUrl, "%s%s", scheme,
(const char *) chain->name))
index->slots.common.startUrl[0] = '\0';
snprintf(index->slots.common.startUrl,
sizeof(index->slots.common.startUrl), "%s%s", scheme,
(const char *) chain->name);
}
}
}
@@ -828,10 +827,6 @@ PT_Element PT_ElementNew(void) {
return r;
}
/* ProxyTrack's htsblk_failf(): a clipped, diagnostic-only failure reason. */
#define PT_Element_failf(R, ...) \
slprintfbuff_clip((R)->msg, sizeof((R)->msg), __VA_ARGS__)
PT_Element PT_ReadCache(PT_Index index, const char *url, int flags) {
if (index != NULL && SAFE_INDEX(index)) {
return _IndexFuncts[index->type].PT_ReadCache(index, url, flags);
@@ -870,15 +865,12 @@ static PT_Element PT_ReadCache__New(PT_Index index, const char *url, int flags)
sprintf(headers + headersSize, "%s: "LLintP"\r\n", field, (LLint)(value)); \
(headersSize) += (int) strlen(headers + headersSize); \
} while(0)
/* refvalue_size is mandatory: the cache line is bounded only by the line
buffer, not by the destination. */
#define ZIP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(void) strclipbuff(refvalue, refvalue_size, value); \
line[0] = '\0'; \
} \
} while (0)
#define ZIP_READFIELD_STRING(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strcpy(refvalue, value); \
line[0] = '\0'; \
} \
} while(0)
#define ZIP_READFIELD_INT(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
int intval = 0; \
@@ -980,12 +972,9 @@ int PT_LoadCache__New(PT_Index index_, const char *filename) {
const char *scheme =
link_has_authority(filenameIndex) ? "" : "http://";
/* dropped rather than truncated; try the next entry */
if (sprintfbuff(index->startUrl, "%s%s", scheme,
filenameIndex))
firstSeen = 1;
else
index->startUrl[0] = '\0';
firstSeen = 1;
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
scheme, filenameIndex);
}
}
} else {
@@ -1032,7 +1021,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
previous_save[0] = previous_save_[0] = '\0';
memset(r, 0, sizeof(_PT_Element));
r->location = location_default;
r->location[0] = '\0';
strcpy(r->location, "");
if (strncmp(url, "http://", 7) == 0)
url += 7;
hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -1081,23 +1070,16 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
value++;
ZIP_READFIELD_INT(line, value, "X-In-Cache", dataincache);
ZIP_READFIELD_INT(line, value, "X-Statuscode", r->statuscode);
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg,
sizeof(r->msg));
ZIP_READFIELD_STRING(line, value, "X-StatusMessage", r->msg); // msg
ZIP_READFIELD_INT(line, value, "X-Size", r->size); // size
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
sizeof(r->contenttype));
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset,
sizeof(r->charset));
ZIP_READFIELD_STRING(line, value, "Last-Modified",
r->lastmodified, sizeof(r->lastmodified));
ZIP_READFIELD_STRING(line, value, "Etag", r->etag,
sizeof(r->etag));
ZIP_READFIELD_STRING(line, value, "Location", r->location,
sizeof(location_default));
ZIP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
ZIP_READFIELD_STRING(line, value, "X-Charset", r->charset); // contenttype
ZIP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
ZIP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
ZIP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
ZIP_READFIELD_STRING(line, value, "Content-Disposition",
r->cdispo, sizeof(r->cdispo));
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_,
sizeof(previous_save_));
r->cdispo); // Content-disposition
ZIP_READFIELD_STRING(line, value, "X-Save", previous_save_); // Original save filename
if (line[0] != '\0') {
int len = r->headers ? ((int) strlen(r->headers)) : 0;
int nlen =
@@ -1117,7 +1099,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
int pathLen = (int) strlen(index->path);
if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) { // old (<3.40) buggy format
strcpybuff(previous_save, previous_save_);
strcpy(previous_save, previous_save_);
}
// relative ? (hack)
else if (index->safeCache || (previous_save_[0] != '/' // /home/foo/bar.gif
@@ -1153,9 +1135,8 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
snprintf(previous_save, sizeof(previous_save), "%s%s",
index->path, previous_save_ + index->fixedPath);
} else {
PT_Element_failf(
r, "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
r->statuscode = STATUSCODE_INVALID;
}
} else {
@@ -1174,7 +1155,7 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
// Peut-on stocker le fichier directement sur disque?
if (ok) {
if (r->msg[0] == '\0') {
strcpybuff(r->msg, "Cache Read Error : Unexpected error");
strcpy(r->msg, "Cache Read Error : Unexpected error");
}
} else { // lire en mémoire
@@ -1192,27 +1173,24 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
int last_errno = errno;
r->statuscode = STATUSCODE_INVALID;
PT_Element_failf(r,
"Read error in cache disk data: %s",
strerror(last_errno));
sprintf(r->msg, "Read error in cache disk data: %s",
strerror(last_errno));
}
r->adr[r->size] = '\0';
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
strcpy(r->msg,
"Read error (memory exhausted) from cache");
}
fclose(fp);
} else {
r->statuscode = STATUSCODE_INVALID;
PT_Element_failf(
r, "Read error (can't open '%s') from cache",
file_convert(catbuff, sizeof(catbuff),
previous_save));
snprintf(r->msg, sizeof(r->msg), "Read error (can't open '%s') from cache",
file_convert(catbuff, sizeof(catbuff), previous_save));
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cached file name is invalid");
strcpy(r->msg, "Cached file name is invalid");
}
}
} else {
@@ -1224,12 +1202,12 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
free(r->adr);
r->adr = NULL;
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Read Data");
strcpy(r->msg, "Cache Read Error : Read Data");
} else
*(r->adr + r->size) = '\0';
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Memory Error");
strcpy(r->msg, "Cache Memory Error");
}
}
}
@@ -1237,21 +1215,21 @@ static PT_Element PT_ReadCache__New_u(PT_Index index_, const char *url,
} // si save==null, ne rien charger (juste en tête)
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Read Header Data");
strcpy(r->msg, "Cache Read Error : Read Header Data");
}
unzCloseCurrentFile(index->zFile);
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Open File");
strcpy(r->msg, "Cache Read Error : Open File");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Bad Offset");
strcpy(r->msg, "Cache Read Error : Bad Offset");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "File Cache Entry Not Found");
strcpy(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -1544,7 +1522,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
cache->version = (int) (firstline[8] - '0'); // cache 1.x
if (cache->version <= 5) {
a += cache_brstr(a, firstline, sizeof(firstline));
strcpybuff(cache->lastmodified, firstline);
strcpy(cache->lastmodified, firstline);
} else {
fclose(cache->dat);
cache->dat = NULL;
@@ -1561,7 +1539,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
} else { // Vieille version du cache
/* */
cache->version = 0; // cache 1.0
strcpybuff(cache->lastmodified, firstline);
strcpy(cache->lastmodified, firstline);
}
/* Create hash table for the cache (MUCH FASTER!) */
@@ -1577,14 +1555,7 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
a++;
/* read "host/file" */
a += binput(a, line, HTS_URLMAXSIZE);
{
/* binput writes its NUL at s[max], so the second field must be
bounded by what is left of line[], not by the same constant
*/
const size_t used = strlen(line);
a += binput(a, line + used, (int) (sizeof(line) - used - 1));
}
a += binput(a, line + strlen(line), HTS_URLMAXSIZE);
/* read position */
a += binput(a, linepos, 200);
sscanf(linepos, "%d", &pos);
@@ -1599,11 +1570,9 @@ static int PT_LoadCache__Old(PT_Index index_, const char *filename) {
const char *scheme =
link_has_authority(line) ? "" : "http://";
/* dropped rather than truncated; try the next entry */
if (sprintfbuff(index->startUrl, "%s%s", scheme, line))
firstSeen = 1;
else
index->startUrl[0] = '\0';
firstSeen = 1;
snprintf(index->startUrl, sizeof(index->startUrl), "%s%s",
scheme, line);
}
}
@@ -1681,7 +1650,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
previous_save[0] = previous_save_[0] = '\0';
memset(r, 0, sizeof(_PT_Element));
r->location = location_default;
r->location[0] = '\0';
strcpy(r->location, "");
if (strncmp(url, "http://", 7) == 0)
url += 7;
hash_pos_return = coucal_read(cache->hash, url, &hash_pos);
@@ -1788,7 +1757,7 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
int pathLen = (int) strlen(index->path);
if (pathLen > 0 && strncmp(previous_save_, index->path, pathLen) == 0) { // old (<3.40) buggy format
strcpybuff(previous_save, previous_save_);
strcpy(previous_save, previous_save_);
}
// relative ? (hack)
else if (index->safeCache || (previous_save_[0] != '/' // /home/foo/bar.gif
@@ -1824,9 +1793,8 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
snprintf(previous_save, sizeof(previous_save), "%s%s",
index->path, previous_save_ + index->fixedPath);
} else {
PT_Element_failf(r,
"Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
snprintf(r->msg, sizeof(r->msg), "Bogus fixePath prefix for %s (prefixLen=%d)",
previous_save_, (int) index->fixedPath);
r->statuscode = STATUSCODE_INVALID;
}
} else {
@@ -1852,18 +1820,17 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
if (r->adr != NULL) {
if (r->size > 0 && fread(r->adr, 1, r->size, fp) != r->size) {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Read error in cache disk data");
strcpy(r->msg, "Read error in cache disk data");
}
r->adr[r->size] = '\0';
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
strcpy(r->msg, "Read error (memory exhausted) from cache");
}
fclose(fp);
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Previous cache file not found (2)");
strcpy(r->msg, "Previous cache file not found (2)");
}
}
} else {
@@ -1875,30 +1842,30 @@ static PT_Element PT_ReadCache__Old_u(PT_Index index_, const char *url,
free(r->adr);
r->adr = NULL;
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Read Data");
strcpy(r->msg, "Cache Read Error : Read Data");
} else
r->adr[r->size] = '\0';
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Memory Error");
strcpy(r->msg, "Cache Memory Error");
}
}
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Bad Data");
strcpy(r->msg, "Cache Read Error : Bad Data");
}
} else { // erreur
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Read Header");
strcpy(r->msg, "Cache Read Error : Read Header");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Seek Failed");
strcpy(r->msg, "Cache Read Error : Seek Failed");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "File Cache Entry Not Found");
strcpy(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -2161,14 +2128,12 @@ int PT_LoadCache__Arc(PT_Index index_, const char *filename) {
return 0;
}
/* Same contract as ZIP_READFIELD_STRING, for the ARC reader's header lines. */
#define HTTP_READFIELD_STRING(line, value, refline, refvalue, refvalue_size) \
do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
(void) strclipbuff(refvalue, refvalue_size, value); \
line[0] = '\0'; \
} \
} while (0)
#define HTTP_READFIELD_STRING(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
strcpy(refvalue, value); \
line[0] = '\0'; \
} \
} while(0)
#define HTTP_READFIELD_INT(line, value, refline, refvalue) do { \
if (line[0] != '\0' && strfield2(line, refline)) { \
int intval = 0; \
@@ -2204,7 +2169,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
location_default[0] = '\0';
memset(r, 0, sizeof(_PT_Element));
r->location = location_default;
r->location[0] = '\0';
strcpy(r->location, "");
if (strncmp(url, "http://", 7) == 0)
url += 7;
hash_pos_return = coucal_read(index->hash, url, &hash_pos);
@@ -2237,16 +2202,11 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
*value = '\0';
for(value++; *value == ' ' || *value == '\t'; value++) ;
HTTP_READFIELD_INT(line, value, "Content-Length", r->size); // size
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype,
sizeof(r->contenttype));
HTTP_READFIELD_STRING(line, value, "Last-Modified",
r->lastmodified, sizeof(r->lastmodified));
HTTP_READFIELD_STRING(line, value, "Etag", r->etag,
sizeof(r->etag));
HTTP_READFIELD_STRING(line, value, "Location", r->location,
sizeof(location_default));
HTTP_READFIELD_STRING(line, value, "Content-Disposition",
r->cdispo, sizeof(r->cdispo));
HTTP_READFIELD_STRING(line, value, "Content-Type", r->contenttype); // contenttype
HTTP_READFIELD_STRING(line, value, "Last-Modified", r->lastmodified); // last-modified
HTTP_READFIELD_STRING(line, value, "Etag", r->etag); // Etag
HTTP_READFIELD_STRING(line, value, "Location", r->location); // 'location' pour moved
HTTP_READFIELD_STRING(line, value, "Content-Disposition", r->cdispo); // Content-disposition
if (line[0] != '\0') {
int len = r->headers ? ((int) strlen(r->headers)) : 0;
int nlen =
@@ -2286,7 +2246,7 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
fetchSize = dataLength - metaSize;
} else if (fetchSize > dataLength - metaSize) {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Truncated Data");
strcpy(r->msg, "Cache Read Error : Truncated Data");
}
r->size = 0;
if (r->statuscode != STATUSCODE_INVALID) {
@@ -2299,13 +2259,12 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
int last_errno = errno;
r->statuscode = STATUSCODE_INVALID;
PT_Element_failf(r, "Read error in cache disk data: %s",
strerror(last_errno));
sprintf(r->msg, "Read error in cache disk data: %s",
strerror(last_errno));
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg,
"Read error (memory exhausted) from cache");
strcpy(r->msg, "Read error (memory exhausted) from cache");
}
}
}
@@ -2313,21 +2272,21 @@ static PT_Element PT_ReadCache__Arc_u(PT_Index index_, const char *url,
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
strcpy(r->msg, "Cache Read Error : Read Header Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Read Header Error");
strcpy(r->msg, "Cache Read Error : Read Header Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "Cache Read Error : Seek Error");
strcpy(r->msg, "Cache Read Error : Seek Error");
}
} else {
r->statuscode = STATUSCODE_INVALID;
strcpybuff(r->msg, "File Cache Entry Not Found");
strcpy(r->msg, "File Cache Entry Not Found");
}
if (r->location[0] != '\0') {
r->location = strdup(r->location);
@@ -2376,18 +2335,8 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
PT_SaveCache__Arc_t *st = (PT_SaveCache__Arc_t *) arg;
FILE *const fp = st->fp;
struct tm *tm = convert_time_rfc822(&st->buff, element->lastmodified);
struct tm unknown_date;
int size_headers;
/* a cached entry with no parseable Last-Modified must not take the writer
down; the epoch is the conventional "date unknown" */
if (tm == NULL) {
memset(&unknown_date, 0, sizeof(unknown_date));
unknown_date.tm_year = 70;
unknown_date.tm_mday = 1;
tm = &unknown_date;
}
sprintf(st->headers,
"HTTP/1.0 %d %s"
"\r\n"
@@ -2423,7 +2372,7 @@ static int PT_SaveCache__Arc_Fun(void *arg, const char *url, PT_Element element)
if (element->adr != NULL) {
domd5mem(element->adr, element->size, st->md5, 1);
} else {
strcpybuff(st->md5, "-");
strcpy(st->md5, "-");
}
fprintf(fp,
/* nl */

View File

@@ -97,7 +97,6 @@
<ItemGroup>
<ClCompile Include="htsserver.c" />
<ClCompile Include="htsweb.c" />
<ClCompile Include="htscmdline.c" />
<ClCompile Include="htsurlport.c" />
</ItemGroup>

View File

@@ -1,8 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# webhttrack posts its command line as one string: the argv split must grow past
# 1024 arguments and keep a quote inside a value out of the option parser.
httrack -O /dev/null -#test=cmdline-split run | grep -q "cmdline-split self-test OK"

View File

@@ -146,13 +146,11 @@ sid=$(scrape_sid "${port}")
resp=$(post_redirect '/foo
X-Injected: pwned' "${port}" "${sid}") || fail "no response to a CRLF redirect value"
stop
# Here-string, not a pipe: grep -q SIGPIPEs the producer on the first match, and
# pipefail then turns the hit into a silent pass.
grep -qi 'X-Injected' <<<"${resp}" &&
printf '%s' "${resp}" | grep -qi 'X-Injected' &&
fail "CRLF in the redirect value reached the response"
test "$(grep -ci '^Location:' <<<"${resp}")" -eq 0 ||
test "$(printf '%s' "${resp}" | grep -ci '^Location:')" -eq 0 ||
fail "CRLF redirect still emitted a Location header"
test "$(grep -c '^HTTP/1\.' <<<"${resp}")" -eq 1 ||
test "$(printf '%s' "${resp}" | grep -c '^HTTP/1\.')" -eq 1 ||
fail "CRLF redirect did not yield exactly one status line"
# Loopback unless asked otherwise. Assert the socket, not the announcement.

View File

@@ -103,10 +103,9 @@ sid=$(scrape_sid "${port}")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
# Accept: the legitimate flow still works. "redirect" is the cheapest field
# with a reply that is visible in the headers. Matched from a here-string, not a
# pipe: grep -q SIGPIPEs the producer, and pipefail then buries the verdict.
# with a reply that is visible in the headers.
resp=$(request "${port}" "sid=${sid}&redirect=/accepted")
grep -q '^Location: /accepted' <<<"${resp}" ||
printf '%s' "${resp}" | grep -q '^Location: /accepted' ||
fail "a body carrying the correct sid was refused"
# Refuse: missing and wrong. A missing one is the regression under test — the
@@ -115,12 +114,12 @@ grep -q '^Location: /accepted' <<<"${resp}" ||
for bad in "redirect=/nosid" "sid=&redirect=/empty" \
"sid=00000000000000000000000000000000&redirect=/wrong"; do
resp=$(request "${port}" "${bad}")
grep -q '^Location:' <<<"${resp}" &&
printf '%s' "${resp}" | grep -q '^Location:' &&
fail "body accepted without a valid sid: ${bad}"
# A refusal has to be a well-formed reply, not a headerless fragment: the
# release build used to emit only Content-length, which reads as a protocol
# error to any client and hides the reason.
grep -q '^HTTP/1\.0 403 ' <<<"${resp}" ||
printf '%s' "${resp}" | grep -q '^HTTP/1\.0 403 ' ||
fail "refusal was not a 403: ${bad}"
done
@@ -128,23 +127,16 @@ done
# applied to one global key store that later requests render from, and the
# command dispatcher reads it from there. Probe the store itself — step3
# interpolates ${projname} into its title — rather than the refused reply.
store_page() {
# a dead probe or a non-page reply must not read as "the store was not written"
page=$(request "$1" "" step3) || fail "the key store probe failed"
grep -q '^HTTP/1\.0 200 ' <<<"${page}" ||
fail "the key store probe did not answer: '$(head -1 <<<"${page}")'"
}
title() { request "$1" "" step3; }
request "${port}" "projname=UNAUTHWRITE" >/dev/null 2>&1 || true
store_page "${port}"
grep -q 'UNAUTHWRITE' <<<"${page}" &&
title "${port}" | grep -q 'UNAUTHWRITE' &&
fail "a body without a valid sid was written to the key store"
# Paired accept case: without it the assertion above passes even if the server
# simply ignores every body, which would prove nothing.
request "${port}" "sid=${sid}&projname=AUTHWRITE" >/dev/null 2>&1 || true
store_page "${port}"
grep -q 'AUTHWRITE' <<<"${page}" ||
title "${port}" | grep -q 'AUTHWRITE' ||
fail "a body carrying the correct sid was not written to the key store"
echo "PASS"

View File

@@ -1,142 +0,0 @@
#!/bin/bash
#
# The wizard's size fields must render as distinct caps: sizemax is -M, othermax
# then maxhtml share -m.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_maxsize.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
srv=
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# webhttrack server on a pre-picked port; an isolated HOME keeps a stray
# ~/.httrack.ini out of it.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
# Audit what step4.html renders back; without command_do nothing is launched.
"${python}" - "${url}" <<'PY' || fail "wizard rendered the wrong options (see above)"
import re, sys, urllib.parse, urllib.request
url = sys.argv[1]
html, other, site = "111111", "222222", "333333"
def post(fields):
# The body is refused without the session id the server puts in each form.
form = urllib.request.urlopen(url + "server/index.html", timeout=20).read()
m = re.search(rb'name="sid" value="([0-9a-f]+)"', form)
if not m:
raise SystemExit("no session id in server/index.html")
fields = [("sid", m.group(1).decode())] + fields
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in fields)
req = urllib.request.Request(url + "server/step4.html",
data=body.encode("latin-1"), method="POST")
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
def textarea(page, name):
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
if m is None:
sys.exit("no %s textarea in the rendered step4.html" % name)
return m.group(1)
def want(ok, msg, ctx):
if not ok:
sys.exit("%s\nrendered:%s" % (msg, ctx))
page = post([("maxhtml", html), ("othermax", other), ("sizemax", site),
("dos", "2")])
cmd = textarea(page, "command")
want("--max-size=" + site in cmd, "site cap not emitted as --max-size", cmd)
want("--max-files=" + site not in cmd, "site cap still an --max-files", cmd)
# Positive controls: both per-file caps still ride -m, one option each.
want("--max-files=" + other in cmd, "non-HTML cap not emitted as --max-files",
cmd)
want("--max-files=," + html in cmd, "HTML cap not emitted as --max-files=,",
cmd)
want(cmd.count("--max-files=") == 2, "unexpected --max-files count", cmd)
want(cmd.index("--max-files=" + other) < cmd.index("--max-files=," + html),
"the bare --max-files must precede the --max-files=, form", cmd)
# winprofile.ini feeds the same three caps to the Windows GUI, one key each.
ini = textarea(page, "winprofile").replace("\r\n", "\n") # the ini is CRLF
want("\nDos=2" in ini, "Dos not written from the dos field", ini)
want("\nMaxHtml=" + html + "\n" in ini, "MaxHtml not written from maxhtml", ini)
want("\nMaxOther=" + other + "\n" in ini, "MaxOther not written from othermax",
ini)
want("\nMaxAll=" + site + "\n" in ini, "MaxAll not written from sizemax", ini)
# A bare --max-files= (an unguarded empty field) parses as -m with no digits,
# silently clearing the html cap.
page = post([("maxhtml", ""), ("othermax", ""), ("sizemax", ""), ("dos", "2")])
cmd = textarea(page, "command")
want("--max-rate=" in cmd, "empty size fields rendered no command line", cmd)
want("--max-files" not in cmd and "--max-size" not in cmd,
"empty size fields still rendered a size option", cmd)
# A mistyped ${LANG_OK] key renders the OK button's label empty.
opts = urllib.request.urlopen(url + "server/option2b.html",
timeout=20).read().decode("latin-1")
want("LANG_OK" not in opts, "unexpanded LANG_OK in option2b.html", "")
want('<input type="submit" value="OK"' in opts, "no OK label in option2b.html",
"")
PY
cleanup
# A bare -m<n> resets the html limit, so only the bare-then-comma order keeps
# both caps live. basic.html is 487 bytes, well past the 10-byte html cap.
crawl() {
bash "${distdir}/tests/local-crawl.sh" "$@"
}
crawl --errors 1 --log-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=,10'
crawl --errors 1 --log-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=500000' '--max-files=,10'
crawl --errors 0 --found 'simple/basic.html' --log-not-found 'File too big' \
httrack 'BASEURL/simple/basic.html' '--max-files=,10' '--max-files=500000'
echo "PASS"

View File

@@ -1,135 +0,0 @@
#!/bin/bash
#
# A browser will not follow an http: page to a file: URL, so the GUI's "browse
# the mirror" link has to go through the server's own /website/ route.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
# Control: on a path that does not resolve, grep "passes" without reading anything.
test -f "${distdir}/html/server/finished.html" || fail "no GUI pages under ${distdir}"
bad=$(grep -l 'file://' "${distdir}"/html/server/finished.html || true)
test -z "${bad}" || fail "file: link left in: ${bad}"
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_browse.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
srvpid=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# Stand in for a finished mirror: the crawl itself is not under test.
proj="${work}/websites/proj"
mkdir -p "${proj}"
printf 'MIRROR-PROBE-OK\n' >"${proj}/probe.txt"
printf '<html><body>MIRROR-INDEX-OK</body></html>\n' >"${proj}/index.html"
# An isolated HOME keeps a stray ~/.httrack.ini out of the server's settings.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
# htsserver resolves the posted paths itself, so hand it native ones.
"${python}" - "${url}" "$(nativepath "${work}/websites")" <<'PY' || fail "browse-link checks failed"
import re, sys, urllib.error, urllib.parse, urllib.request
url, base = sys.argv[1].rstrip("/"), sys.argv[2]
rc = 0
def check(ok, what):
global rc
print(("ok: " if ok else "FAIL: ") + what)
if not ok:
rc = 1
def get(path):
# The UI is served ISO-8859-1, so stay in bytes.
return urllib.request.urlopen(url + path, timeout=20).read()
# No crawl has run, so no commandRunning/commandEnd override swaps the page out.
finished = get("/server/finished.html").decode("latin-1")
# Control: an empty or unrendered page would pass "no file:" on its own.
check("HTTrack Website Copier" in finished, "finished.html rendered")
check("file://" not in finished, "finished.html has no file: link")
# Control: /website/ 404s until a project is set, so the fetches below are real.
try:
get("/website/probe.txt")
check(False, "/website/ served with no project set")
except urllib.error.HTTPError as e:
check(e.code == 404, "/website/ is bound to a project (got %d)" % e.code)
# Point the server at the mirror the way step4.html does; a body without the
# session id is refused outright. Only a profile save records the served root,
# so post one; command_do=save runs no crawl.
sid = re.search(r'name="sid" value="([0-9a-f]+)"', finished)
if sid is None:
print("FAIL: no sid in the rendered form")
sys.exit(1)
fields = [("sid", sid.group(1)), ("path", base), ("projname", "proj"),
("command", "httrack"), ("command_do", "save"), ("winprofile", "x")]
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe="")) for k, v in fields)
urllib.request.urlopen(urllib.request.Request(
url + "/server/step4.html", data=body.encode("latin-1"), method="POST"),
timeout=20).read()
check(get("/website/probe.txt") == b"MIRROR-PROBE-OK\n", "mirror file over http")
check("MIRROR-INDEX-OK" in get("/website/index.html").decode("latin-1"),
"mirror index over http")
body = get("/server/finished.html").decode("latin-1")
# Match the anchor by its mirror-path label: the list below it links /website/
# too, so a bare "is the route mentioned" check passes on the unfixed page.
check(re.search(r'href="/website/index\.html"[^>]*>\s*' +
re.escape(base + "/proj"), body) is not None,
"the mirror-path link points at the served mirror")
check("file://" not in body, "finished.html has no file: link")
sys.exit(rc)
PY
# A leaked htsserver wedges the parallel harness behind a green log.
cleanup
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
echo "PASS"

View File

@@ -1,148 +0,0 @@
#!/bin/bash
#
# The wizard renders its httrack command line as one string, which the engine
# splits back into argv: a double quote in a field must reach the split escaped,
# or the rest of the value is parsed as fresh options (-V runs a shell command).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1; $2 is the body ("" for a GET of the $3 page,
# default index). Prints the reply.
request() {
python3 -c 'import socket, sys
port, body = int(sys.argv[1]), sys.argv[2]
page = sys.argv[3] if len(sys.argv) > 3 else "index"
if body:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
else:
req = ("GET /server/%s.html HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % page)
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" ${3+"$3"}
}
scrape_sid() {
request "$1" "" | sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1
}
# urlencode the key=value pairs given as arguments
formencode() {
python3 -c 'import sys, urllib.parse
print(urllib.parse.urlencode([tuple(a.split("=", 1)) for a in sys.argv[1:]]))' "$@"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
sid=$(scrape_sid "${port}")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid from the page (got '${sid}')"
# Fill the wizard fields the command line quotes, then read back the generated
# command line: the user-agent carries a break-out attempt, the footer a
# backslash (which must survive the escape round trip), the project name a plain
# value.
body=$(formencode "sid=${sid}" 'user=Moz" -V "touch /tmp/pwn' 'footer=a\b"c' \
"path=/tmp/p" "projname=plain proj" 'urls=http://x/a"b' 'url2=+*.png"')
request "${port}" "${body}" >/dev/null
cmdline=$(request "${port}" "" step4 |
sed -n '/<textarea name="command"/,/<\/textarea>/p')
# Control: without the fields in the page every assertion below is vacuous.
grep -q -- '--user-agent' <<<"${cmdline}" ||
fail "no --user-agent in the generated command line (probe blind)"
# A plain value is passed through untouched: the escaping must not mangle the
# ordinary case.
grep -qF -- '--path "/tmp/p/plain proj"' <<<"${cmdline}" ||
fail "a plain quoted value was not passed through: ${cmdline}"
# The quote is escaped, so the split keeps it inside the value...
grep -qF -- '--user-agent "Moz\" -V \"touch /tmp/pwn"' <<<"${cmdline}" ||
fail "the quote in the user-agent was not escaped: ${cmdline}"
# ...and the raw form, which would end the argument and hand -V to the option
# parser, is gone.
grep -qF -- '--user-agent "Moz" -V "touch' <<<"${cmdline}" &&
fail "the user-agent still closes its argument early: ${cmdline}"
# A backslash is escaped too, or the split would eat it along with the quote
# that follows.
grep -qF -- '--footer "a\\b\"c"' <<<"${cmdline}" ||
fail "the backslash in the footer was not escaped: ${cmdline}"
# The url and filter fields sit outside quotes, where a backslash cannot escape
# anything: one quote there flips the parity of every quote after it, so the
# escaping above would protect nothing. They must not emit a raw quote at all.
grep -qF -- 'http://x/a%22b' <<<"${cmdline}" ||
fail "the quote in the url field was not neutralised: ${cmdline}"
grep -qF -- '+*.png%22' <<<"${cmdline}" ||
fail "the quote in the filter field was not neutralised: ${cmdline}"
grep -qF -- 'http://x/a"b' <<<"${cmdline}" &&
fail "the url field still emits a raw quote: ${cmdline}"
echo "PASS"

View File

@@ -1,167 +0,0 @@
#!/bin/bash
#
# htsserver serves the crawled mirror under /website/ alongside its own GUI:
# mirrored pages must skip the ${...} expander, or ${_sid} leaks the session id.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
log=$(mktemp)
work=$(mktemp -d)
csrv=
# start() runs in a command substitution, so its $! never reaches this shell. A
# missed kill leaves an orphan holding the CI job open long after a green suite.
srvpid() { sed -n 's/^PID=//p' "${log}" 2>/dev/null | head -1; }
cleanup() {
local pid
pid=$(srvpid)
test -z "${pid}" || kill -9 "${pid}" 2>/dev/null || true
test -z "${csrv}" || kill -9 "${csrv}" 2>/dev/null || true
wait "${csrv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${log}" "${work}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
portof() { echo "${1##*:}" | tr -d /; }
# GET $2 from 127.0.0.1:$1, headers into $3 and the body, byte for byte, into $4.
fetch() {
python3 -c 'import socket, sys
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(20)
s.sendall(("GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % sys.argv[2]).encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
head, _, body = out.partition(b"\r\n\r\n")
open(sys.argv[3], "wb").write(head)
open(sys.argv[4], "wb").write(body)' "$1" "$2" "$3" "$4"
}
# POST the remaining args as urlencoded key=value fields to 127.0.0.1:$1.
post() {
local port=$1
shift
python3 -c 'import socket, sys, urllib.parse
body = "&".join("%s=%s" % (k, urllib.parse.quote(v, safe=""))
for k, v in (a.split("=", 1) for a in sys.argv[2:])).encode()
s = socket.create_connection(("127.0.0.1", int(sys.argv[1])), 10)
s.settimeout(30)
s.sendall(b"POST /step4.html HTTP/1.0\r\nHost: 127.0.0.1\r\n"
b"Content-type: application/x-www-form-urlencoded\r\n"
b"Content-length: %d\r\n\r\n" % len(body) + body)
while s.recv(65536):
pass
s.close()' "${port}" "$@" >/dev/null
}
# LF-only, and a line ending in a backslash: the expander rewrites both, so a
# mangled reply fails the byte comparison even where no directive is present.
mirror="${work}/proj/hostile.html"
hdr="${work}/hdr"
body="${work}/body"
# The server merges $HOME/.httrack.ini into the same store on the first request;
# point it somewhere empty so a developer's own file cannot shadow the fields.
export HOME="${work}"
url=$(start)
port=$(portof "${url}")
# Positive control: the GUI's own templates must still expand. This is also
# where the real token comes from, so its absence below can be asserted.
fetch "${port}" /server/index.html "${hdr}" "${body}"
sid=$(sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' "${body}" | head -1)
test "${#sid}" -eq 32 || fail "GUI page did not expand \${sid} (got '${sid}')"
# /website/ serves only the root the server itself recorded, so save a profile
# (no command_do=start, so nothing crawls) to create it, then plant the file.
post "${port}" "sid=${sid}" command=httrack command_do=save winprofile=x \
"path=${work}" projname=proj
test -f "${work}/proj/hts-cache/winprofile.ini" ||
fail "profile save did not create the project: $(cat "${log}")"
mkdir -p "$(dirname "${mirror}")"
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
printf 'Hostile mirrored page.\nsid=${_sid} copy=${sid}\ntrailing backslash: \\\n' \
>"${mirror}"
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
grep -q '^HTTP/1\.0 200 ' "${hdr}" || fail "mirrored page not served: $(head -1 "${hdr}")"
grep -qF "${sid}" "${body}" &&
fail "the session id was expanded into mirrored content"
# shellcheck disable=SC2016 # the directives are the payload, not shell expansions
grep -qF '${_sid}' "${body}" || fail "\${_sid} did not survive verbatim"
# shellcheck disable=SC2016
grep -qF '${sid}' "${body}" || fail "\${sid} did not survive verbatim"
cmp -s "${mirror}" "${body}" || fail "mirrored file not served byte for byte"
# The mirror stays browsable: verbatim must not mean served as a download.
grep -qi '^Content-type: text/html' "${hdr}" ||
fail "mirrored page lost its text/html type: $(cat "${hdr}")"
# The other direction: while a crawl runs, every .html request is overridden to
# the GUI's own refresh page, so a /website/ URL stops naming mirrored content.
# /trickle/ dribbles for a minute, which holds the crawl open for the probe.
clog="${work}/content.log"
python3 "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
csrv=$!
for _ in $(seq 1 40); do
cport=$(sed -n 's/^PORT //p' "${clog}") && test -n "${cport}" && break
kill -0 "${csrv}" 2>/dev/null || break
sleep 0.25
done
test -n "${cport:-}" || fail "content server did not come up: $(cat "${clog}")"
post "${port}" "sid=${sid}" "path=${work}" projname=crawl winprofile=x \
command_do=start \
"command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/trickle/ -O ${work}/crawl"
fetch "${port}" /website/hostile.html "${hdr}" "${body}"
grep -q '^HTTP/1\.0 200 ' "${hdr}" ||
fail "running crawl: /website/ was not overridden to the GUI page: $(head -1 "${hdr}")"
grep -qF "'crawl' - HTTrack Website Copier" "${body}" ||
fail "the overridden GUI page was not the expanded refresh page"
echo "PASS"

View File

@@ -1,183 +0,0 @@
#!/bin/bash
#
# /website/ is served from the project directory htsserver set up itself, never
# from a root the request body names, and composing that path must stay bounded.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
base=$(mktemp -d)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
# The server reports its own pid; the aliveness assertion below hangs off it.
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
# $2 is empty. Prints the reply.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
get() { request "$1" "$2" ""; }
post() { request "$1" "" "$2"; }
# GET $2 into ${reply}, requiring status $3. Captured, not piped: a dead request
# reads as marker-absent, and so do a truncated body and a 302 to the file.
reply=
fetch() {
reply=$(get "$1" "$2") || fail "the request for $2 failed"
grep -q "^HTTP/1\.0 $3 " <<<"${reply}" ||
fail "$2: wanted a $3 reply, got '$(head -1 <<<"${reply}")'"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Every request body is gated by the session id (78_webhttrack-sid.test).
sid=$(get "${port}" /server/index.html |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
# A file the mirror must never expose, next to the project that may.
echo "SECRETMARKER" >"${base}/secret.txt"
mkdir -p "${base}/proj"
echo "LOGMARKER" >"${base}/proj/hts-log.txt"
# error_redirect is the branch that skipped the fsfile clearing, so fail the save
# with a component over NAME_MAX (mkdir refuses it whatever the uid). Must precede
# any successful save: commandEnd then swaps the error page for the finished one.
get "${port}" /server/style.css >/dev/null # leaves a path behind in fsfile
saved=$(post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/$(printf '%0300d' 0)&projname=p")
grep -q '^Location: /server/error.html' <<<"${saved}" ||
fail "the refused save did not redirect to the error page"
# No project yet, so no root to serve from.
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
fetch "${port}" /website/secret.txt 404
grep -q SECRETMARKER <<<"${reply}" &&
fail "a posted projpath served a file outside any project"
# Positive control: step4's "save settings" flow registers the project without
# crawling, and browsing its mirror keeps working. Without it the assertions
# around it would pass on a server that never serves /website/ at all.
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
body="${body}&path=${base}&projname=proj&projpath=${base}/proj/"
post "${port}" "${body}" >/dev/null
test -f "${base}/proj/hts-cache/winprofile.ini" ||
fail "the project was not registered: $(cat "${log}")"
fetch "${port}" /website/hts-log.txt 200
grep -q LOGMARKER <<<"${reply}" ||
fail "the registered project's mirror is not served"
# Same request with the root repointed: the project is legitimate, projpath is
# not what decides where the bytes come from.
post "${port}" "sid=${sid}&projpath=${base}/" >/dev/null
fetch "${port}" /website/secret.txt 404
grep -q SECRETMARKER <<<"${reply}" &&
fail "a posted projpath repointed the served root"
post "${port}" "sid=${sid}&projpath=/etc/" >/dev/null
fetch "${port}" /website/passwd 404
grep -q '^root:' <<<"${reply}" &&
fail "a posted projpath read an arbitrary system file"
# A ".." anywhere in the recorded root would escape the mirror on every later
# request, so the save must be refused and the previous root kept.
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${base}/proj&projname=.." >/dev/null
fetch "${port}" /website/secret.txt 404
grep -q SECRETMARKER <<<"${reply}" &&
fail "a '..' in the saved project path escaped the mirror root"
fetch "${port}" /website/hts-log.txt 200
grep -q LOGMARKER <<<"${reply}" ||
fail "rejecting the '..' root also lost the previous one"
# The root is now the server's own, but it is still built from two posted
# fields: 800-odd bytes of them used to be sprintf'd into a 1024-byte buffer.
seg=$(printf '%0200d' 0)
longpath="${base}/${seg}/${seg}/${seg}/${seg}"
fspath="${longpath}/proj"
# structcheck() refuses a root over HTS_URLMAXSIZE, so the URL carries the rest.
test "$((${#fspath} + 11))" -le 1024 ||
fail "the long project path (${#fspath}) would not pass structcheck"
longurl=$(printf '%0800d' 0)
test "$((${#fspath} + 1 + ${#longurl}))" -gt 1024 ||
fail "the composed path (${#fspath} + ${#longurl}) would not overflow"
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=x&path=${longpath}&projname=proj" >/dev/null
test -f "${fspath}/hts-cache/winprofile.ini" ||
fail "the long-path project was not registered: $(cat "${log}")"
get "${port}" "/website/${longurl}" >/dev/null 2>&1 || true
alive "${srv}" || fail "an over-long project path crashed the server: $(cat "${log}")"
# Not just alive: still answering.
fetch "${port}" /server/index.html 200
echo "PASS"

View File

@@ -1,111 +0,0 @@
#!/bin/bash
# A cached header longer than proxytrack's field must be clipped to that field,
# not overflow into the next one (contenttype[64] abuts the location pointer).
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
pad() { printf "%${1}s" '' | tr ' ' "$2"; }
# Longest surviving run of char $2 in file $1, or 0.
runlen() {
grep -ao "$2\\+" "$1" | awk '{ print length($0) }' | sort -rn | head -n1 || true
}
# --- ARC reader (HTTP_READFIELD_STRING), no python needed -------------------
# Each header overshoots its own destination, so a per-field clip is the only
# way to get every expected length right at once.
{
printf 'HTTP/1.1 200 OK\r\n'
printf 'Content-Type: text/html%s\r\n' "$(pad 400 A)"
printf 'Etag: %s\r\n' "$(pad 400 E)"
printf 'Content-Disposition: %s\r\n' "$(pad 400 D)"
printf 'Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n'
printf 'Content-Length: 5\r\n\r\n'
} >"$dir/hdr"
printf 'hello' >"$dir/body"
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
{
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
printf '2 0 test\n'
printf '\n\n'
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
cat "$dir/hdr" "$dir/body"
} >"$dir/in.arc"
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed reading an ARC with over-long headers" >&2
exit 1
}
test -s "$dir/out.arc" || {
echo "FAIL: ARC entry dropped instead of clipped" >&2
exit 1
}
# Only contenttype survives into the ARC output; the over-long Etag and
# Content-Disposition above are still load-bearing, since bounding just one
# field leaves the others smashing the element (the run above would crash).
got=$(runlen "$dir/out.arc" A)
test "${got:-0}" = 54 || {
echo "FAIL: ARC content-type clipped to ${got:-0}, expected 54" >&2
exit 1
}
# --- ZIP reader (ZIP_READFIELD_STRING) --------------------------------------
python=$(find_python) || {
echo "python3 missing; ARC half only" >&2
exit 0
}
"$python" - "$dir/in.zip" <<'EOF'
import sys, zipfile
body = b"hello world"
meta = (
"X-In-Cache: 1\r\n"
"X-StatusCode: 200\r\n"
"X-StatusMessage: ok%s\r\n"
"X-Size: %d\r\n"
"Content-Type: text/html%s\r\n"
"X-Charset: %s\r\n"
"Etag: %s\r\n"
"Content-Disposition: %s\r\n"
# a parseable date is required: the ARC writer dereferences it unchecked
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
"X-Save: out/page.html\r\n"
% ("M" * 400, len(body), "A" * 400, "C" * 400, "E" * 400, "D" * 400)
).encode()
zi = zipfile.ZipInfo("example.com/page.html")
zi.compress_type = zipfile.ZIP_STORED
zi.extra = meta
with zipfile.ZipFile(sys.argv[1], "w") as z:
z.writestr(zi, body)
EOF
proxytrack --convert "$dir/out2.arc" "$dir/in.zip" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed reading a zip cache with over-long headers" >&2
exit 1
}
test -s "$dir/out2.arc" || {
echo "FAIL: zip entry dropped instead of clipped" >&2
exit 1
}
# three destinations, two capacities: msg[1024] keeps all 400 M's where a
# one-size-fits-all clip would cut it to 63 like the others
while read -r ch want; do
got=$(runlen "$dir/out2.arc" "$ch")
test "${got:-0}" = "$want" || {
echo "FAIL: zip field '$ch' clipped to ${got:-0}, expected $want" >&2
exit 1
}
done <<'EOF'
A 54
C 63
M 400
EOF

View File

@@ -1,51 +0,0 @@
#!/bin/bash
# A cached entry whose Last-Modified is missing or unparseable must still
# convert: the ARC writer used to dereference the parsed date unchecked.
set -euo pipefail
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
# $1 Last-Modified value (empty = omit the header), $2 expected archive date,
# $3 label. The date is asserted exactly: a guard that fires unconditionally
# would clobber the valid case, and one that skips the year/day would emit a
# month and day of 00.
run_case() {
lastmod=$1
wantdate=$2
what=$3
{
printf 'HTTP/1.1 200 OK\r\n'
printf 'Content-Type: text/html\r\n'
test -z "$lastmod" || printf 'Last-Modified: %s\r\n' "$lastmod"
printf 'Content-Length: 5\r\n\r\n'
} >"$dir/hdr"
printf 'hello' >"$dir/body"
alen=$(($(wc -c <"$dir/hdr") + $(wc -c <"$dir/body")))
{
printf 'filedesc://t.arc 0.0.0.0 20250101000000 text/plain 200 - - 0 t.arc 9\n'
printf '2 0 test\n'
printf '\n\n'
printf 'http://example.com/p.html 0.0.0.0 20250101000000 text/html 200 - - 0 t.arc %d\n' "$alen"
cat "$dir/hdr" "$dir/body"
} >"$dir/in.arc"
proxytrack --convert "$dir/out.arc" "$dir/in.arc" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed on $what" >&2
exit 1
}
grep -aq "^http://example.com/p.html 0.0.0.0 ${wantdate} " "$dir/out.arc" || {
echo "FAIL: $what: expected archive date ${wantdate}, got:" >&2
grep -a '^http://example.com/' "$dir/out.arc" >&2 || echo "(entry dropped)" >&2
exit 1
}
}
# the epoch stands in for "date unknown"
run_case '' 19700101000000 'a missing Last-Modified'
run_case 'not a date at all' 19700101000000 'an unparseable Last-Modified'
run_case '0' 19700101000000 'a bare 0 Last-Modified'
# a parseable date must still come through untouched
run_case 'Sun, 06 Nov 1994 08:49:37 GMT' 19941106084937 'a valid Last-Modified'

View File

@@ -1,69 +0,0 @@
#!/bin/bash
# A cache file whose mtime is beyond what gmtime can represent must not put a
# day of "00" in the ARC filedesc date: PT_GetTime fell back to an all-zero tm.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
rc=0
"$python" - "$dir/in.zip" <<'EOF' || rc=$?
import os, sys, zipfile
body = b"hello world"
meta = (
"X-In-Cache: 1\r\n"
"X-StatusCode: 200\r\n"
"X-StatusMessage: OK\r\n"
"X-Size: %d\r\n"
"Content-Type: text/html\r\n"
"Last-Modified: Wed, 01 Jan 2025 00:00:00 GMT\r\n"
"X-Save: out/page.html\r\n" % len(body)
).encode()
zi = zipfile.ZipInfo("example.com/page.html")
zi.compress_type = zipfile.ZIP_STORED
zi.extra = meta
with zipfile.ZipFile(sys.argv[1], "w") as z:
z.writestr(zi, body)
# past gmtime's range; the index timestamp is this file's mtime
huge = 4611686018427387903
try:
os.utime(sys.argv[1], (huge, huge))
except OSError:
sys.exit(77)
if int(os.stat(sys.argv[1]).st_mtime) < huge:
sys.exit(77) # filesystem clamped it, nothing to test
EOF
test "$rc" -eq 0 || {
test "$rc" -eq 77 || {
echo "FAIL: could not build the cache (python exit $rc)" >&2
exit 1
}
echo "filesystem will not hold an out-of-range mtime; skipping" >&2
exit 77
}
proxytrack --convert "$dir/out.arc" "$dir/in.zip" >/dev/null 2>&1 || {
echo "FAIL: proxytrack failed on a cache with an out-of-range mtime" >&2
exit 1
}
# field 3 of the filedesc line is YYYYMMDDhhmmss; the epoch stands in for
# "unknown", and 14 digits with a real day is the point (00 is what broke)
date=$(head -n1 "$dir/out.arc" | awk '{ print $3 }')
test "$date" = 19700101000000 || {
echo "FAIL: filedesc date is '$date', expected 19700101000000" >&2
exit 1
}

View File

@@ -1,174 +0,0 @@
#!/bin/bash
#
# The "save settings" failure messages quote a project path the request body
# supplies, so composing them must clip instead of running off a fixed buffer.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
# ERROR_MESSAGE_MAX - 1 in htsserver.c.
msgmax=1023
srv=
log=$(mktemp)
# Physical path: macOS resolves TMPDIR through /var -> private/var, and the
# paths built below sit within a few bytes of its 1024-byte PATH_MAX.
base=$(cd "$(mktemp -d)" && pwd -P)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU XFSZ
# Caps regular-file writes, which is how the failing-write branch below
# is reached without /dev/full; the announce rides a pipe, exempt.
ulimit -f 8
exec htsserver "${distdir}/" --port "${port}" 2>&1
) | cat >"${log}" &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
srvpid() { sed -n 's/^PID=//p' "${log}" | head -1; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# Raw request to 127.0.0.1:$1: GET the path $2, or POST the body $3 to / when
# $2 is empty. Prints the reply.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(30)
s.sendall(req.encode())
out = b""
while True:
b = s.recv(65536)
if not b:
break
out += b
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
get() { request "$1" "$2" ""; }
post() { request "$1" "" "$2"; }
# Echo a path under the root $1 exactly $2 bytes long, in NAME_MAX-sized parts.
padpath() {
local root=$1 want=$2 p=$1 last
while test $((${#p} + 203)) -lt "${want}"; do
p="${p}/$(printf '%0200d' 0)"
done
last=$((want - ${#p} - 1))
test "${last}" -ge 1 || fail "cannot build a ${want}-byte path under ${root}"
printf '%s/%0*d' "${p}" "${last}" 0
}
# Fail unless the error page renders the message opening with $1 about project
# path $2, clipped to msgmax. error.html puts ${error} alone on its own line.
check_clipped() {
local prefix=$1 full=$((${#1} + ${#2})) page line shown
test "${full}" -gt "${msgmax}" ||
fail "a ${full}-byte message fits in ${msgmax}: it cannot show a clip"
page=$(get "${port}" /server/error.html | tr -d '\r')
line=$(printf '%s\n' "${page}" | grep "^${prefix}") || {
shown=$(printf '%s\n' "${page}" | grep '^Unable to ' | cut -c1-80 || true)
fail "the error page reports '${shown:-nothing}', want: ${prefix}"
}
test "${#line}" -eq "${msgmax}" ||
fail "the message rendered ${#line} bytes, want ${full} clipped to ${msgmax}"
}
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Every request body is gated by the session id (78_webhttrack-sid.test).
sid=$(get "${port}" /server/index.html |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p' | head -1)
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
save() {
post "${port}" "sid=${sid}&command=httrack&command_do=save&winprofile=$2&path=$1&projname=$3"
}
# A project path too long to be created at all.
path=$(padpath "${base}/nodir" 1098)
save "${path}" x p >/dev/null
alive "${srv}" ||
fail "an over-long project path crashed the server: $(cat "${log}")"
check_clipped "Unable to create the directory structure in " "${path}/p"
# A creatable project path whose hts-cache is not a directory, so the init file
# under it cannot be opened.
path=$(padpath "${base}/isdir" 993)
mkdir -p "${path}/p"
ln -s /dev/null "${path}/p/hts-cache"
save "${path}" x p >/dev/null
alive "${srv}" || fail "an unwritable init file crashed the server: $(cat "${log}")"
check_clipped "Unable to create the init file " "${path}/p"
# The init file opens, then the write fails: the profile is past both stdio's
# buffer, so fwrite() reaches the descriptor, and the server's file-size limit.
path=$(padpath "${base}/full" 993)
mkdir -p "${path}/p/hts-cache"
profile=$(printf '%01024d' 0)
profile=${profile}${profile}${profile}${profile}
profile=${profile}${profile}${profile}${profile}
test "${#profile}" -eq 16384 || fail "built a ${#profile}-byte profile, want 16384"
save "${path}" "${profile}" p >/dev/null
alive "${srv}" || fail "a short init file write crashed the server: $(cat "${log}")"
written=$(wc -c <"${path}/p/hts-cache/winprofile.ini" | tr -d ' ')
test "${written}" -lt "${#profile}" ||
fail "the file-size limit did not stop the write (${written} bytes landed)"
check_clipped "Unable to write ${#profile} bytes in the the init file " "${path}/p"
get "${port}" /server/index.html | grep -q '200 OK' ||
fail "the server stopped answering after the refused saves"
echo "PASS"

View File

@@ -1,195 +0,0 @@
#!/bin/bash
#
# An unchecked box posts nothing and the stored "1" survives: hence a hidden
# companion per box, plus a disabling flag for the default-on ones.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_checkbox.XXXXXX") || fail "no tmpdir"
srvlog=$(mktemp)
srv=
srvpid=
cleanup() {
# htsserver keeps SIGTERM ignored across its exec, so only -9 reaps it.
test -z "${srvpid}" || kill -9 "${srvpid}" 2>/dev/null || true
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true # absorb bash's async "Killed" notice
rm -rf "${work}" "${srvlog}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# An isolated HOME keeps a stray ~/.httrack.ini out of the served settings.
sport=$("${python}" -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
(
trap '' TERM TTOU
export HOME="${work}"
exec htsserver "${distdir}/" --port "${sport}" >"${srvlog}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${srvlog}") && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not start: $(cat "${srvlog}")"
srvpid=$(sed -n 's/^PID=//p' "${srvlog}") # absent on Windows
"${python}" - "${url}" "${distdir}" <<'PY' || fail "checkbox clearing is broken (see above)"
import glob, os, re, sys, urllib.parse, urllib.request
url, srcdir = sys.argv[1].rstrip("/"), sys.argv[2]
rc = 0
def check(ok, what):
global rc
print(("ok: " if ok else "FAIL: ") + what)
if not ok:
rc = 1
# cache/cache2 are left to the separate rework of the dead "Cache" seed.
skip = {"cache", "cache2"}
page_of = {}
boxes = 0
for path in sorted(glob.glob(os.path.join(srcdir, "html", "server", "option*.html"))):
page = open(path, "rb").read().decode("latin-1")
cleared = dict((m.group(1), m.start()) for m in
re.finditer(r'<input type="hidden" name="([^"]+)" value="">', page))
for m in re.finditer(r'<input type="checkbox" name="([^"]+)"', page):
boxes += 1
if m.group(1) in skip:
continue
page_of[m.group(1)] = os.path.basename(path)
at = cleared.get(m.group(1))
check(at is not None and at < m.start(), "%s: %s is cleared before it is drawn"
% (os.path.basename(path), m.group(1)))
# Control: a regex that stopped matching would leave nothing to assert on.
check(boxes >= 25, "the option pages were scanned (%d checkboxes)" % boxes)
sid = None
def get(path):
# The UI is served ISO-8859-1, so decode, do not assume UTF-8.
return urllib.request.urlopen(url + path, timeout=20).read().decode("latin-1")
def post(fields):
global sid
if sid is None:
m = re.search(r'name="sid" value="([0-9a-f]+)"', get("/server/index.html"))
if m is None:
sys.exit("no session id in server/index.html")
sid = m.group(1)
body = "&".join("%s=%s" % (k, urllib.parse.quote(v)) for k, v in
[("sid", sid)] + fields)
req = urllib.request.Request(url + "/server/step4.html",
data=body.encode("latin-1"), method="POST")
return urllib.request.urlopen(req, timeout=20).read().decode("latin-1")
def textarea(page, name):
m = re.search(r'<textarea name="%s".*?>(.*?)</textarea>' % name, page, re.S)
if m is None:
sys.exit("no %s textarea in the rendered step4.html" % name)
return m.group(1)
def checked(page, name):
return re.search(r'name="%s"[ \t]*checked' % name, get("/server/" + page)) is not None
# What each state puts on the command line (None: nothing) and its profile key.
# Only a value-taking alias can carry a disabling flag: -I and -%I are "single"
# (htsalias.c), so a --index=0 would read back as the bare enabling flag.
BOXES = [
# field profile key set cleared
("parseall", "ParseAll", "--near", None),
("link", "Near", "--test", None),
("testall", "Test", "--extended-parsing", None),
# htmlfirst emits --priority=7, which the scan-priority list also emits.
("htmlfirst", "HTMLFirst", None, None),
("errpage", "NoErrorPages", "--generate-errors=0", None),
("external", "NoExternalPages", "--replace-external", None),
("hidepwd", "NoPwdInPages", "--disable-passwords", None),
("hidequery", "NoQueryStrings", "--include-query-string=0", None),
("nopurge", "NoPurgeOldFiles", "--purge-old=0", None),
("windebug", None, "--debug-headers", None),
("ka", "KeepAlive", "--keep-alive", None),
("remt", "RemoveTimeout", "--host-control=1", None),
("rems", "RemoveRateout", "--host-control=2", None),
("cookies", "Cookies", None, "--cookies=0"),
("parsejava", "ParseJava", None, "--parse-java=0"),
("updhack", "UpdateHack", "--updatehack", None),
("urlhack", "URLHack", "--urlhack", None),
("keepwww", "KeepWww", "--keep-www-prefix", None),
("keepslashes", "KeepSlashes", "--keep-double-slashes", None),
("keepqueryorder", "KeepQueryOrder", "--keep-query-order", None),
("toler", "TolerantRequests", "--tolerant", None),
("http10", "HTTP10", "--http-10", None),
("warc", "Warc", "--warc", None),
("norecatch", "NoRecatch", "--do-not-recatch", None),
("logf", "Log", "--single-log", None),
("index", "Index", None, None),
("index2", "WordIndex", "--search-index", None),
("ftpprox", "UseHTTPProxyForFTP", "--httpproxy-ftp", None),
]
for field, key, when_set, when_clear in BOXES:
for value, want, unwanted, stored in (("on", when_set, when_clear, "1"),
("", when_clear, when_set, "0")):
rendered = post([(field, value)])
cmd = textarea(rendered, "command").split()
label = "%s %s" % (field, "set" if value else "cleared")
if want:
check(want in cmd, "%s emits %s" % (label, want))
if unwanted:
check(unwanted not in cmd, "%s drops %s" % (label, unwanted))
if key:
# The Windows GUI reads the same option out of the profile.
check("%s=%s" % (key, stored) in textarea(rendered, "winprofile").splitlines(),
"%s writes %s=%s" % (label, key, stored))
check(checked(page_of[field], field) == bool(value),
"%s draws %s box" % (label, "a ticked" if value else "an empty"))
missing = sorted(set(page_of) - set(b[0] for b in BOXES))
check(not missing, "every checkbox is exercised (missing %s)" % missing)
# A browser posts the companion and the ticked box under one name, so the fix
# rests on the body loop keeping the last value; both orders pin that down.
cmd = textarea(post([("cookies", ""), ("cookies", "on")]), "command").split()
check("--cookies=0" not in cmd, "cookies=&cookies=on keeps cookies set")
check(checked("option8.html", "cookies"), "cookies=&cookies=on draws a ticked box")
cmd = textarea(post([("cookies", "on"), ("cookies", "")]), "command").split()
check("--cookies=0" in cmd, "cookies=on&cookies= clears cookies")
check(not checked("option8.html", "cookies"), "cookies=on&cookies= draws an empty box")
sys.exit(rc)
PY
# A leaked htsserver wedges the parallel harness behind a green log.
cleanup
! kill -0 "${srv}" 2>/dev/null || fail "htsserver ${srv} survived"
echo "PASS"

View File

@@ -1,181 +0,0 @@
#!/bin/bash
#
# fopen() succeeds on a directory on POSIX and reading one never reaches EOF; a
# FIFO blocks in fopen() instead. Either used to wedge the single-threaded server.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
log=$(mktemp)
base=$(mktemp -d)
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
rm -f "${log}"
rm -rf "${base}"
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
# First line only. A "| head -1" would close the pipe early and, under pipefail,
# SIGPIPE the producer into a spurious failure.
firstline() { echo "${1%%$'\n'*}"; }
freeport() {
python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()'
}
# Echo the announced URL. Runs in a command substitution, so it is a
# subshell and cannot export the pid: the caller reads it back with srvpid.
start() {
local port url
port=$(freeport)
: >"${log}"
(
trap '' TERM TTOU
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
echo "${url}"
}
# The server reports its own pid; the aliveness assertion below hangs off it.
srvpid() { firstline "$(sed -n 's/^PID=//p' "${log}")"; }
alive() { kill -0 "$1" 2>/dev/null; }
portof() { echo "${1##*:}" | tr -d /; }
# GET the path $2 from 127.0.0.1:$1, or POST the body $3 to / when $2 is empty.
# The socket timeout is what makes an unfixed tree fail rather than wedge CI.
request() {
python3 -c 'import socket, sys
port, path, body = int(sys.argv[1]), sys.argv[2], sys.argv[3]
if path:
req = "GET %s HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n" % path
else:
req = ("POST / HTTP/1.0\r\nHost: 127.0.0.1\r\n"
"Content-type: application/x-www-form-urlencoded\r\n"
"Content-length: %d\r\n\r\n%s" % (len(body), body))
s = socket.create_connection(("127.0.0.1", port), 10)
s.settimeout(15)
s.sendall(req.encode())
out = b""
try:
while True:
b = s.recv(65536)
if not b:
break
out += b
except socket.timeout:
sys.stderr.write("timed out after %d bytes\n" % len(out))
sys.exit(9)
s.close()
sys.stdout.write(out.decode("latin-1"))' "$1" "$2" "$3"
}
# GET $2, or fail with $3 when the reply never comes.
get() {
local out
out=$(request "$1" "$2" "") || fail "$2 did not answer: $3"
echo "${out}"
}
post() { request "$1" "" "$2"; }
url=$(start)
port=$(portof "${url}")
srv=$(srvpid)
test -n "${srv}" || fail "htsserver did not report its pid"
# Needs no session id and no project: html/server is a directory of the install.
reply=$(get "${port}" /server/ "an unauthenticated request wedged the server")
case "${reply}" in
"HTTP/1.0 404 "*) ;;
*) fail "a GUI directory did not answer 404: $(firstline "${reply}")" ;;
esac
# Every request body is gated by the session id.
reply=$(get "${port}" /server/index.html "the GUI is not served")
sid=$(firstline "$(echo "${reply}" |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p')")
test "${#sid}" -eq 32 || fail "did not scrape a 32-hex sid (got '${sid}')"
mkdir -p "${base}/proj/sub"
echo "PROBEMARKER" >"${base}/proj/probe.txt"
# mkfifo and ln -s may not work on a Windows checkout, so both stay optional.
refuse=(/website/ /website/sub/)
if mkfifo "${base}/proj/fifo.txt" 2>/dev/null; then
refuse+=(/website/fifo.txt)
fi
ln -s probe.txt "${base}/proj/link.txt" 2>/dev/null || true
# step4's "save settings" flow registers the project without crawling, and that
# is what arms the /website/ root.
body="sid=${sid}&command=httrack&command_do=save&winprofile=x"
body="${body}&path=${base}&projname=proj"
post "${port}" "${body}" >/dev/null || fail "the save request did not answer"
test -f "${base}/proj/hts-cache/winprofile.ini" ||
fail "the project was not registered: $(cat "${log}")"
# Positive control: every assertion below would pass on a server serving nothing.
served() {
case "$(get "$1" "$2" "$3")" in
*PROBEMARKER*) ;;
*) fail "$3" ;;
esac
}
served "${port}" /website/probe.txt "the registered project's mirror is not served"
if test -L "${base}/proj/link.txt"; then
# The guard stats rather than lstats, so a symlinked mirror file still serves.
served "${port}" /website/link.txt "a symlink to a mirror file is not served"
fi
# /website/ is where the GUI's own "browse mirrored site" link points.
for path in "${refuse[@]}"; do
reply=$(get "${port}" "${path}" "the server wedged on ${path}")
case "${reply}" in
"HTTP/1.0 404 "*) ;;
*) fail "${path} did not answer 404: $(firstline "${reply}")" ;;
esac
done
# The accept loop is single-threaded: one spin denies every later request.
alive "${srv}" || fail "the server died: $(cat "${log}")"
served "${port}" /website/probe.txt "the server stopped answering after a directory request"
# This fopen() is reached before any guard, so its read loop must stop on ferror().
mkdir -p "${base}/proj2/hts-cache/winprofile.ini"
reply=$(post "${port}" "sid=${sid}&path=${base}&loadprojname=proj2") ||
fail "a directory winprofile.ini wedged the server"
case "${reply}" in
"HTTP/1.0 3"*) ;;
*) fail "loading a project did not redirect: $(firstline "${reply}")" ;;
esac
alive "${srv}" || fail "the server died: $(cat "${log}")"
served "${port}" /website/probe.txt "the server stopped answering after a project load"
echo "PASS"

View File

@@ -1,37 +0,0 @@
#!/bin/bash
# The two halves of an .ndx entry's URL share one buffer, so the second must be
# bounded by what the first left. The sanitizer CI build turns a regression
# here into a hard stack-buffer-overflow.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
python=$(find_python) || {
echo "python3 not found; skipping" >&2
exit 77
}
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
# each field fills binput's own bound, so together they run one past line[]
"$python" - "$dir/foo.ndx" <<'EOF'
import sys
with open(sys.argv[1], "w") as f:
f.write("CACHE-1.4\n")
f.write("Wed, 01 Jan 2025 00:00:00 GMT\n")
f.write("h" * 1024 + "\n" + "f" * 1024 + "\n" + "0\n")
# a trailing entry keeps the walk inside the buffer, so this exercises the
# field bound alone
f.write("tail/entry\nx\n1\n")
EOF
: >"$dir/foo.dat" # the reader only proceeds when the sibling .dat opens
proxytrack --convert "$dir/out.arc" "$dir/foo.ndx" >/dev/null 2>&1 || {
echo "FAIL: proxytrack crashed on an .ndx with two full-length URL fields" >&2
exit 1
}

View File

@@ -32,7 +32,6 @@ TESTS = \
00_runnable.test \
01_engine-charset.test \
01_engine-cmdline.test \
01_engine-cmdline-split.test \
01_engine-cookies.test \
01_engine-copyopt.test \
01_engine-crange.test \
@@ -177,18 +176,6 @@ TESTS = \
77_webhttrack-redirect.test \
78_webhttrack-sid.test \
79_local-proxytrack-webdav-mime.test \
80_engine-crash-symbolize.test \
81_webhttrack-maxsize.test \
82_webhttrack-browse-links.test \
83_webhttrack-argescape.test \
84_webhttrack-mirror-verbatim.test \
85_webhttrack-projpath.test \
86_local-proxytrack-cache-longfields.test \
87_local-proxytrack-nodate.test \
88_local-proxytrack-badmtime.test \
89_webhttrack-error-overflow.test \
90_webhttrack-checkbox-clear.test \
91_webhttrack-directory.test \
92_local-proxytrack-ndx-fields.test
80_engine-crash-symbolize.test
CLEANFILES = check-network_sh.cache