rms/httrack
1
0
Fork 0
mirror of https://github.com/xroche/httrack.git synced 2026-07-28 11:32:48 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
2fccb170583d9c36738aff0f106eeb6be71c19fa
httrack/html/server/step4.html
Xavier Roche 9e29c1e159 Sitemap files are never read, so URLs nothing links to are never found (#718) 
* Read sitemap files so URLs nothing links to are found

HTTrack finds URLs only by parsing links, so anything a site publishes solely
in its sitemap stayed invisible: robots.txt was already parsed, but its
Sitemap: lines were ignored and nothing else in the tree touched sitemaps.

Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and
falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document.
Handles <urlset> and nested <sitemapindex>, plain or gzipped. Discovered URLs
enter with the full depth budget but still go through the wizard, so filters
and scope rules decide; a sitemap is not a filter bypass.

The parser reads attacker-controlled XML off the network, so it is capped on
URL count, index nesting, decompressed size and decompression ratio, and child
sitemaps must stay on the host that named them.

Closes #712

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: distinguish the sitemapindex log line from a urlset one

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: fix an out-of-bounds read, tighten the parser and the tests

lienrelatif() walked back from the last character of its current-path
argument without checking the path was non-empty, reading one byte before
the stack buffer. htsAddLink is the first caller to pass an empty savename,
which sitemap documents have because they are ingested rather than mirrored,
so ASan caught it on the new crawl test.

The parser drops a value whose numeric character reference decodes outside
printable ASCII, rather than leaving the reference verbatim and seeding a URL
the site never published, and classifies a document by its real root element,
so a comment naming the other one no longer flips urlset and sitemapindex.
The robots.txt line reader is bounded by the body size instead of relying on
a NUL terminator.

The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only,
because an uninstrumented libz floods it with false positives.

Tests gain the assertions the earlier ones were missing: which of the
robots.txt route and the /sitemap.xml fallback was taken, that the sitemap
documents stay out of the mirror, that the off-host child sitemap is refused,
the sitemapindex nesting cap, the per-document URL cap at its production
value, and copy_htsopt coverage for the two new fields.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: state what the decompression cap actually binds on

deflate tops out near 1032:1, so hts_codec_maxout never binds before the
64 MiB cap; the old comment implied a ratio guard that cannot fire.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: a child sitemap is a fetch, so filters and robots.txt must gate it

An adversarial review found that a <sitemapindex> <loc>, and a robots.txt
Sitemap: line, went straight to hts_record_link: the request went out even
when a -* rule or robots.txt Disallow covered it. Only the <urlset> half ran
through the wizard. Gate the document itself on the filters and on
robots.txt, which is all that can apply: the wizard proper wants a referring
link, and its up/down travel rules would judge a child sitemap against the
parent sitemap's own directory. The robots.txt probe is exempt, being the
request that fetches the rules.

A 301 also used to end ingestion silently, since the engine re-queues the
target as a fresh link that carried no sitemap marking. That hit any site
redirecting http to https. The marking now follows the redirect.

The "N URL(s) added" counter reported what the scanner emitted rather than
what was taken, which hid both of the above; it now reads "N of M". The
fallback to /sitemap.xml keys on the same corrected count, so a robots.txt
whose only Sitemap: line is off-host or filtered still falls back. Root
classification skips a UTF-8 BOM and an XML namespace prefix, and the doc
list is cleared when a mirror starts rather than only when it ends.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: gate each fetch by who asked for it, and anchor travel on the start URL

A nine-agent review found a scope escape: the sitemap document was its own
`premier`, so the wizard measured travel from wherever the site chose to put
its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the
whole host. The ingester now points the wizard at the crawl's own start link
and lets each seeded URL become its own anchor, which is what a command-line
seed gets.

Robots handling was both mistimed and undifferentiated. The Sitemap: lines are
now collected by robots_parse, on the same body in the same fetch, and acted on
after the parsed rules are installed rather than before; and the decision comes
from a new hts_robots_forbids extracted out of the wizard, so the sitemap path
inherits the -s1 filters-win override instead of a stricter hand-rolled check.
The four fetches are no longer treated alike: a sitemap the user names is user
intent, one the site declares invites the fetch, only the guessed /sitemap.xml
obeys a Disallow, and the URLs listed inside stay fully gated.

Also: hts_unescapeEntities replaces the private entity decoder, whose guard
tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead,
which is only defined under HTS_USEZLIB; the composed URL buffer now fits two
maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the
bounded search is promoted to htstools as hts_memstr; and the live state moves
from httrackp into htsoptstate, leaving two installed fields rather than three.

Tests gain the scope escape, the three robots cases, a cap-boundary control,
the handler invocation count and a compression-bomb decode. Every one was
checked against a deliberately broken build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: add the fuzz harness the parser was missing, and drop truncated Sitemap: lines

The file header called the scanner fuzzable while fuzz/ registered ten
harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies
and truncated streams off a heap copy of exactly the input size, so an overread
is an ASan report rather than a quiet pass, with a four-file seed corpus.
60000 runs clean under ASan+UBSan.

robots_parse now drops a Sitemap: line that filled its scratch buffer instead
of handing on the half URL it was truncated to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* fuzz: keep only the four sitemap seed inputs

A libFuzzer run writes its finds into the first corpus directory, and 191 of
them were committed with the harness.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* selftest: bound the sitemap document builders' snprintf accumulation

snprintf returns the length it wanted to write, so accumulating it blind
lets the next offset and size argument walk past the buffer. Guard each
step the way the argv builder above already does, and give the per-URL
loop a real remaining-space bound instead of a fixed 33.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: date the new files 2026

The headers were copied from an existing file and kept its 1998 year.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: keep the ingestion state out of htsoptstate

htsoptstate is embedded by value as httrackp.state, so a field at its tail
shifts every httrackp member declared after it: an offsetof probe put
warc_file at 141752 on master and 141760 on the branch. Move the pointer to
httrackp's own tail, where every existing offset holds and copy_htsopt still
ignores it.

Also renumber the crawl test to 89, master having taken 87 and 90, and give
the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear
requires, plus its row in that test's table.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: renumber the sitemap crawl test to 95

Master took 88 through 93 and #720 claims 94.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: realign the htsopt.h comments after the single-file merge

Master's longer LLint declarator moved the block's comment column.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: translate the new GUI strings into the remaining 28 locales (#738)

The feature PR added the four LANG_SITEMAP* entries to lang.def with English
and Francais only; every other locale fell back to English in the WebHTTrack
form. Each file is written in its own declared charset.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 07:59:59 +02:00

351 lines
11 KiB
HTML
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" lang="${LANGUAGE_ISO}">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=${LANGUAGE_CHARSET}" />
<meta name="description" content="${LANG_METADESC}" />
<meta name="keywords" content="${LANG_METAKEYW}" />
<title>'${projname}' - HTTrack Website Copier</title>
<link rel="stylesheet" type="text/css" media="screen" href="style.css">
<script type="text/javascript" src="ping.js" defer></script>
<script language="javascript">
<!--
function do_load() {
window.status=' ';
}
function do_unload() {
}
function key_event(event) {
if (event && event.keyCode && event.keyCode == 10) {
form.nextBtn.click();
return false;
}
return true;
}
function info(str) {
window.status = str;
}
// -->
</script>
</head>
<body onLoad="do_load();" onUnload="do_unload();" onKeyPress="return key_event(event);">
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td width="90%" id="subTitle">${LANG_OSFWB}</td>
<td id="subTitle" align="right">
<a href="/server/file.html" target="_blank"
onClick="window.open('/server/file.html', 'help', 'toolbar=no, location=no, directories=no, status=yes, menubar=no, scrollbars=yes, resizable=yes, width=640, height=480'); return false"
title='${html:LANG_O1}' onMouseOver="info('${html:LANG_O1}'); return true" onMouseOut="info('&nbsp;'); return true"
style="color:#FFFFFF"
>
${LANG_O1}
</a>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}
<td id="subTitle">|</td>
<td id="subTitle" align="right">
<a href="/index.html" target="_blank"
onClick="window.open('/server/help.html', 'help', 'toolbar=no, location=no, directories=no, status=yes, menubar=no, scrollbars=yes, resizable=yes, width=640, height=480'); return false"
title='${html:LANG_TIPHELP}' onMouseOver="info('${html:LANG_TIPHELP}'); return true" onMouseOut="info('&nbsp;'); return true"
style="color:#FFFFFF"
>
${LANG_O5}
</a>
</td>
${do:end-if}
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_J9}</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}
<td>
<a href="/step3.html" target="_blank"
onClick="window.open('/step3.html', 'help', 'toolbar=no, location=no, directories=no, status=yes, menubar=no, scrollbars=yes, resizable=yes, width=640, height=480'); return false"
title='${html:LANG_TIPHELP}' onMouseOver="info('${html:LANG_TIPHELP}'); return true" onMouseOut="info('&nbsp;'); return true"
>${LANG_TIPHELP}</a>
</td>
${do:end-if}
</tr></table>
<br>
<form method="POST" name="form" action="step4.html">
<input type="hidden" name="sid" value="${sid}">
<input type="hidden" name="redirect" value="">
<input type="hidden" name="projpath" value="${path}/${projname}/">
<input type="hidden" name="command_arg_profile" value="path,lang">
<table border="0" width="100%">
<tr><td>
</td></tr>
<tr><td>
<input type="radio" name="command_do" value="start" checked><b>${LANG_J10}</b>
</td></tr>
<tr><td>
<input type="radio" name="command_do" value="save">${LANG_J10b}
</td></tr>
<tr><td>
${/* Real commands and ini file generated below */}
<!-- engine commandline; ztest so a cleared default-on option still emits its disabling flag -->
${do:output-mode:html}
<textarea name="command" cols="50" rows="4" style="visibility:hidden">
httrack \
--quiet \
--build-top-index \
${test:todo:--mirror:--mirror:--mirror-wizard:--get:--mirrorlinks:--testlinks:--continue:--update}
${unquoted:urls}
${test:filelist:-%L "}${arg:filelist}${test:filelist:"}
--path "${arg:path}/${arg:projname}"
\
${test:parseall:--near}
${test:link:--test}
${test:testall:--extended-parsing}
${test:htmlfirst::--priority=7}
\
${do:if-not-empty:BuildString}
--structure "${arg:BuildString}"
${do:end-if}
${test:build:-N0:-N0:-N1:-N2:-N3:-N4:-N5:-N100:-N101:-N102:-N103:-N104:-N105:-N99:-N199:}
\
${ztest:dos::--long-names=0:--long-names=2}
${test:robots:--robots=0:--robots=0:--robots=1:--robots=2}
${test:errpage:--generate-errors:--generate-errors=0}
${test:external:--replace-external}
${test:hidepwd:--disable-passwords}
${test:hidequery:--include-query-string=1:--include-query-string=0}
${test:nopurge:--purge-old=1:--purge-old=0}
\
${test:cache:--cache=0:}
${test:filter::--priority=0:--priority=1:--priority=2:--priority=7}
${test:travel::--stay-on-same-dir:--can-go-down:--can-go-up:--can-go-up-and-down}
${test:travel2::--stay-on-same-address:--stay-on-same-domain:--stay-on-same-tld:--go-everywhere}
${test:travel3::--keep-links=0:--keep-links:--keep-links=3:--keep-links=4}
${test:windebug:--debug-headers}
\
${test:connexion:--sockets=}${unquoted:connexion}
${test:ka:--keep-alive}
${test:timeout:--timeout=}${unquoted:timeout}
${test:remt:--host-control=1}
${test:retry:--retries=}${unquoted:retry}
${test:rate:--min-rate=}${unquoted:rate}
${test:rems:--host-control=2}
\
${test:depth:--depth=}${unquoted:depth}
${test:depth2:--ext-depth=}${unquoted:depth2}
${/* -m<n> resets the html limit, so the bare form must precede the -m,<n> one */}
${test:othermax:--max-files=}${unquoted:othermax}
${test:maxhtml:--max-files=,}${unquoted:maxhtml}
${test:sizemax:--max-size=}${unquoted:sizemax}
${test:pausebytes:--max-pause=}${unquoted:pausebytes}
${test:maxtime:--max-time=}${unquoted:maxtime}
${test:maxrate:--max-rate=}${unquoted:maxrate}
${test:maxconn:--connection-per-second=}${unquoted:maxconn}
${test:maxlinks:--advanced-maxlinks=}${unquoted:maxlinks}
\
--user-agent "${arg:user}"
--footer "${arg:footer}"
\
${unquoted:url2}
\
${ztest:cookies:--cookies=0:}
${ztest:parsejava:--parse-java=0:}
${test:updhack:--updatehack}
${test:urlhack:--urlhack=0:--urlhack}
${test:keepwww:--keep-www-prefix}
${test:keepslashes:--keep-double-slashes}
${test:keepqueryorder:--keep-query-order}
${test:cookiesfile:--cookies-file "}${arg:cookiesfile}${test:cookiesfile:"}
${test:pausefiles:--pause "}${arg:pausefiles}${test:pausefiles:"}
${test:stripquery:--strip-query "}${arg:stripquery}${test:stripquery:"}
${test:toler:--tolerant}
${test:http10:--http-10}
${test:cache2:--store-all-in-cache}
${test:sitemap:--sitemap}
${test:sitemapurl:--sitemap-url "}${html:sitemapurl}${test:sitemapurl:"}
${test:warc:--warc}
${test:warcfile:--warc-file "}${arg:warcfile}${test:warcfile:"}
${test:changes:--changes}
${test:singlefile:--single-file}
${test:singlefilemax:--single-file-max-size=}${unquoted:singlefilemax}
${test:norecatch:--do-not-recatch}
${test:logf:--single-log}
${test:logtype:::--extra-log:--debug-log}
${test:index:--index=0:}
${test:index2:--search-index=0:--search-index}
${test:prox:--proxy "}${do:if-not-empty:prox}${test:proxytype::socks5:connect}${test:proxytype:\3A//}${do:end-if}${do:output-mode:html}${arg:prox}${test:prox:\3A}${arg:portprox}${test:prox:"}
${test:ftpprox:--httpproxy-ftp=0:--httpproxy-ftp}
</textarea>
<!-- winprofile.ini -->
${do:output-mode:inifile}
<textarea name="winprofile" cols="50" rows="4" style="visibility:hidden">
CurrentUrl=${urls}
Category=${projcateg}
CurrentAction=${todo}
CurrentURLList=${filelist}
Proxy=${proxy}
Port=${port}
Near=${ztest:link:0:1}
Test=${ztest:testall:0:1}
ParseAll=${ztest:parseall:0:1}
HTMLFirst=${ztest:htmlfirst:0:1}
Cache=${ztest:cache:0:1}
NoRecatch=${ztest:norecatch:0:1}
Dos=${dos}
Index=${ztest:index:0:1}
WordIndex=${ztest:index2:0:1}
Log=${ztest:logf:0:1:2}
RemoveTimeout=${ztest:remt:0:1}
RemoveRateout=${ztest:rems:0:1}
KeepAlive=${ztest:ka:0:1}
FollowRobotsTxt=${robots}
NoErrorPages=${ztest:errpage:0:1}
NoExternalPages=${ztest:external:0:1}
NoPwdInPages=${ztest:hidepwd:0:1}
NoQueryStrings=${ztest:hidequery:0:1}
NoPurgeOldFiles=${ztest:nopurge:0:1}
Cookies=${ztest:cookies:0:1}
CookiesFile=${cookiesfile}
CheckType=${ztest:checktype:0:1:2}
ParseJava=${ztest:parsejava:0:1}
HTTP10=${ztest:http10:0:1}
TolerantRequests=${ztest:toler:0:1}
UpdateHack=${ztest:updhack:0:1}
URLHack=${ztest:urlhack:0:1}
KeepWww=${ztest:keepwww:0:1}
KeepSlashes=${ztest:keepslashes:0:1}
KeepQueryOrder=${ztest:keepqueryorder:0:1}
StripQuery=${stripquery}
StoreAllInCache=${ztest:cache2:0:1}
Sitemap=${ztest:sitemap:0:1}
SitemapUrl=${sitemapurl}
Warc=${ztest:warc:0:1}
WarcFile=${warcfile}
Changes=${ztest:changes:0:1}
SingleFile=${ztest:singlefile:0:1}
SingleFileMaxSize=${singlefilemax}
LogType=${logtype}
UseHTTPProxyForFTP=${ztest:ftpprox:0:1}
ProxyType=${proxytype}
Build=${build}
PrimaryScan=${filter}
Travel=${travel}
GlobalTravel=${travel2}
RewriteLinks=${travel3}
BuildString=${BuildString}
MaxHtml=${maxhtml}
MaxOther=${othermax}
MaxAll=${sizemax}
MaxWait=${pausebytes}
PauseFiles=${pausefiles}
Sockets=${connexion}
Retry=${retry}
MaxTime=${maxtime}
TimeOut=${timeout}
RateOut=${rate}
UserID=${user}
Footer=${footer}
MaxRate=${maxrate}
WildCardFilters=${url2}
Proxy=${proxy}
Port=${port}
Depth=${depth}
ExtDepth=${depth2}
MaxConn=${maxconn}
MaxLinks=${maxlinks}
MIMEDefsExt1=${ext1}
MIMEDefsExt2=${ext2}
MIMEDefsExt3=${ext3}
MIMEDefsExt4=${ext4}
MIMEDefsExt5=${ext5}
MIMEDefsExt6=${ext6}
MIMEDefsExt7=${ext7}
MIMEDefsExt8=${ext8}
MIMEDefsMime1=${mime1}
MIMEDefsMime2=${mime2}
MIMEDefsMime3=${mime3}
MIMEDefsMime4=${mime4}
MIMEDefsMime5=${mime5}
MIMEDefsMime6=${mime6}
MIMEDefsMime7=${mime7}
MIMEDefsMime8=${mime8}
</textarea>
${do:output-mode:}
<!-- winprofile.ini -->
<!-- httrack.ini -->
${do:output-mode:inifile}
<textarea name="userprofile" cols="50" rows="4" style="visibility:hidden">
path=${path}
lang=${lang_}
</textarea>
${do:output-mode:}
<!-- httrack.ini -->
</td></tr><tr><td>
<table width="100%" border="0"><tr><td align="left">
<input type="submit" value=" << ${LANG_PREVIOUS} " onClick="form.command.value=''; form.redirect.value='step3.html'; form.submit()"
title='${html:LANG_TIPPREV}' onMouseOver="info('${html:LANG_TIPPREV}'); return true" onMouseOut="info('&nbsp;'); return true"
>
</td><td align="right">
<input name="nextBtn" type="submit" value=" ${LANG_J9} >> "
title='${html:LANG_TIPNEXT}' onMouseOver="info('${html:LANG_TIPNEXT}'); return true" onMouseOut="info('&nbsp;'); return true"
>
</td></tr></table>
</td></tr>
</table>
</form>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
</tr>
</table>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink