rms/httrack
1
0
Fork 0
mirror of https://github.com/xroche/httrack.git synced 2026-07-28 11:32:48 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
2fccb170583d9c36738aff0f106eeb6be71c19fa
httrack/html/cmdguide.html

564 lines
34 KiB
HTML
Raw Normal View History

Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
<title>HTTrack Website Copier - Offline Browser</title>
<style type="text/css">
<!--
body {
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
background: #77b;
}
body, td {
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
}
#subTitle {
background: #000; color: #fff; padding: 4px; font-weight: bold;
}
#siteNavigation a, #siteNavigation .current {
font-weight: bold; color: #448;
}
#siteNavigation a:link { text-decoration: none; }
#siteNavigation a:visited { text-decoration: none; }
#siteNavigation .current { background-color: #ccd; }
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
a:link { text-decoration: underline; color: #00f; }
a:visited { text-decoration: underline; color: #000; }
a:hover { text-decoration: underline; color: #c00; }
a:active { text-decoration: underline; }
#pageContent {
clear: both;
border-bottom: 6px solid #000;
padding: 10px; padding-top: 20px;
line-height: 1.65em;
background-image: url(images/bg_rings.gif);
background-repeat: no-repeat;
background-position: top right;
}
#pageContent, #siteNavigation {
background-color: #ccd;
}
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
h1 { margin: 0; font-weight: bold; font-size: 2em; }
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
.blak { background-color: #000; }
.hide { display: none; }
.tableWidth { min-width: 400px; }
.tblRegular { border-collapse: collapse; }
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
.tblNoBorder td { border: 0; }
// -->
</style>
</head>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">Open Source offline browser</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<h2 align="center"><em>Command-Line Guide</em></h2>
<p>This is a task-oriented guide to the <tt>httrack</tt> command line: how to do the
things people actually ask for, and the handful of defaults that surprise
newcomers. It sits on top of the
<a href="httrack.man.html">generated manual page</a>, which lists every option in
full. When you want the exhaustive detail for a flag, that page is the reference;
this one is the map.</p>
<p>Two habits before anything else. First, HTTrack has its own options: they are not
wget or curl flags, so reach for the tables here rather than guessing. Second,
when a mirror does something you did not expect, the answer is almost always in
the log. Every project writes <tt>hts-log.txt</tt> (and <tt>hts-err.txt</tt>) into
its output directory, and those files name every URL that was refused, redirected,
or filtered out. Read them first.</p>
<h4>On this page</h4>
<ul class="tblNoBorder">
<li><a href="#quickstart">1. Quick start</a></li>
<li><a href="#scope">2. Scope: how far the crawl reaches</a></li>
<li><a href="#filters">3. Filters and scan rules</a></li>
<li><a href="#limits">4. Limits and politeness</a></li>
<li><a href="#names">5. File names and types</a></li>
<li><a href="#links">6. Links and page building</a></li>
<li><a href="#identity">7. Identity, cookies and login</a></li>
<li><a href="#proxy">8. Proxy and network</a></li>
<li><a href="#update">9. Update and cache</a></li>
<li><a href="#experts">10. Experts and scripting</a></li>
<li><a href="#recipes">11. Recipes</a></li>
</ul>
<h3 id="quickstart">1. Quick start</h3>
<p>A mirror is one command: a start URL and an output directory.</p>
<p><tt>httrack https://example.com/ --path mydir</tt></p>
<p>With no other options HTTrack mirrors that site, stays on the same host, follows
links to any depth, rebuilds them to browse offline, and stores everything under
<tt>mydir</tt>. The same directory also holds the log files and the
<tt>hts-cache/</tt> folder that makes a later update or resume possible.</p>
<p>Two defaults are worth knowing up front, because both catch people out:</p>
<ul>
<li>HTTrack throttles itself to about <b>100 KB/s</b> even when you pass no rate
option. If a mirror feels slow, that is why. See
<a href="#limits">Limits</a> for how to lift it.</li>
<li>The download proceeds as a well-behaved robot: it identifies itself as
<tt>HTTrack</tt>, obeys <tt>robots.txt</tt>, and sends a Referer with each
request. A site that blocks that behavior needs the levers in
<a href="#identity">Identity</a>, not brute force.</li>
</ul>
<h3 id="scope">2. Scope: how far the crawl reaches</h3>
<p>Scope decides which links HTTrack is even willing to follow, before any filter
you write. Get this right and most "it downloaded too much" or "it only grabbed
the index" problems disappear.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--depth (-r)</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
<tr><td><tt>--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
<tr><td><tt>--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
<tr><td><tt>--near (-n)</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
<tr><td><tt>--ext-depth (-%e)</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
<tr><td><tt>--test (-t)</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
Sitemap files are never read, so URLs nothing links to are never found (#718) * Read sitemap files so URLs nothing links to are found HTTrack finds URLs only by parsing links, so anything a site publishes solely in its sitemap stayed invisible: robots.txt was already parsed, but its Sitemap: lines were ignored and nothing else in the tree touched sitemaps. Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document. Handles <urlset> and nested <sitemapindex>, plain or gzipped. Discovered URLs enter with the full depth budget but still go through the wizard, so filters and scope rules decide; a sitemap is not a filter bypass. The parser reads attacker-controlled XML off the network, so it is capped on URL count, index nesting, decompressed size and decompression ratio, and child sitemaps must stay on the host that named them. Closes #712 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: distinguish the sitemapindex log line from a urlset one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: fix an out-of-bounds read, tighten the parser and the tests lienrelatif() walked back from the last character of its current-path argument without checking the path was non-empty, reading one byte before the stack buffer. htsAddLink is the first caller to pass an empty savename, which sitemap documents have because they are ingested rather than mirrored, so ASan caught it on the new crawl test. The parser drops a value whose numeric character reference decodes outside printable ASCII, rather than leaving the reference verbatim and seeding a URL the site never published, and classifies a document by its real root element, so a comment naming the other one no longer flips urlset and sitemapindex. The robots.txt line reader is bounded by the body size instead of relying on a NUL terminator. The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only, because an uninstrumented libz floods it with false positives. Tests gain the assertions the earlier ones were missing: which of the robots.txt route and the /sitemap.xml fallback was taken, that the sitemap documents stay out of the mirror, that the off-host child sitemap is refused, the sitemapindex nesting cap, the per-document URL cap at its production value, and copy_htsopt coverage for the two new fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: state what the decompression cap actually binds on deflate tops out near 1032:1, so hts_codec_maxout never binds before the 64 MiB cap; the old comment implied a ratio guard that cannot fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: a child sitemap is a fetch, so filters and robots.txt must gate it An adversarial review found that a <sitemapindex> <loc>, and a robots.txt Sitemap: line, went straight to hts_record_link: the request went out even when a -* rule or robots.txt Disallow covered it. Only the <urlset> half ran through the wizard. Gate the document itself on the filters and on robots.txt, which is all that can apply: the wizard proper wants a referring link, and its up/down travel rules would judge a child sitemap against the parent sitemap's own directory. The robots.txt probe is exempt, being the request that fetches the rules. A 301 also used to end ingestion silently, since the engine re-queues the target as a fresh link that carried no sitemap marking. That hit any site redirecting http to https. The marking now follows the redirect. The "N URL(s) added" counter reported what the scanner emitted rather than what was taken, which hid both of the above; it now reads "N of M". The fallback to /sitemap.xml keys on the same corrected count, so a robots.txt whose only Sitemap: line is off-host or filtered still falls back. Root classification skips a UTF-8 BOM and an XML namespace prefix, and the doc list is cleared when a mirror starts rather than only when it ends. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: gate each fetch by who asked for it, and anchor travel on the start URL A nine-agent review found a scope escape: the sitemap document was its own `premier`, so the wizard measured travel from wherever the site chose to put its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the whole host. The ingester now points the wizard at the crawl's own start link and lets each seeded URL become its own anchor, which is what a command-line seed gets. Robots handling was both mistimed and undifferentiated. The Sitemap: lines are now collected by robots_parse, on the same body in the same fetch, and acted on after the parsed rules are installed rather than before; and the decision comes from a new hts_robots_forbids extracted out of the wizard, so the sitemap path inherits the -s1 filters-win override instead of a stricter hand-rolled check. The four fetches are no longer treated alike: a sitemap the user names is user intent, one the site declares invites the fetch, only the guessed /sitemap.xml obeys a Disallow, and the URLs listed inside stay fully gated. Also: hts_unescapeEntities replaces the private entity decoder, whose guard tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead, which is only defined under HTS_USEZLIB; the composed URL buffer now fits two maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the bounded search is promoted to htstools as hts_memstr; and the live state moves from httrackp into htsoptstate, leaving two installed fields rather than three. Tests gain the scope escape, the three robots cases, a cap-boundary control, the handler invocation count and a compression-bomb decode. Every one was checked against a deliberately broken build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: add the fuzz harness the parser was missing, and drop truncated Sitemap: lines The file header called the scanner fuzzable while fuzz/ registered ten harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies and truncated streams off a heap copy of exactly the input size, so an overread is an ASan report rather than a quiet pass, with a four-file seed corpus. 60000 runs clean under ASan+UBSan. robots_parse now drops a Sitemap: line that filled its scratch buffer instead of handing on the half URL it was truncated to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * fuzz: keep only the four sitemap seed inputs A libFuzzer run writes its finds into the first corpus directory, and 191 of them were committed with the harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * selftest: bound the sitemap document builders' snprintf accumulation snprintf returns the length it wanted to write, so accumulating it blind lets the next offset and size argument walk past the buffer. Guard each step the way the argv builder above already does, and give the per-URL loop a real remaining-space bound instead of a fixed 33. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: date the new files 2026 The headers were copied from an existing file and kept its 1998 year. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: keep the ingestion state out of htsoptstate htsoptstate is embedded by value as httrackp.state, so a field at its tail shifts every httrackp member declared after it: an offsetof probe put warc_file at 141752 on master and 141760 on the branch. Move the pointer to httrackp's own tail, where every existing offset holds and copy_htsopt still ignores it. Also renumber the crawl test to 89, master having taken 87 and 90, and give the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear requires, plus its row in that test's table. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: renumber the sitemap crawl test to 95 Master took 88 through 93 and #720 claims 94. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: realign the htsopt.h comments after the single-file merge Master's longer LLint declarator moved the block's comment column. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: translate the new GUI strings into the remaining 28 locales (#738) The feature PR added the four LANG_SITEMAP* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Signed-off-by: Xavier Roche <xroche@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 07:59:59 +02:00
<tr><td><tt>--sitemap (-%m), --sitemap-url URL (-%mu)</tt></td><td>Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
</table>
Sitemap files are never read, so URLs nothing links to are never found (#718) * Read sitemap files so URLs nothing links to are found HTTrack finds URLs only by parsing links, so anything a site publishes solely in its sitemap stayed invisible: robots.txt was already parsed, but its Sitemap: lines were ignored and nothing else in the tree touched sitemaps. Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document. Handles <urlset> and nested <sitemapindex>, plain or gzipped. Discovered URLs enter with the full depth budget but still go through the wizard, so filters and scope rules decide; a sitemap is not a filter bypass. The parser reads attacker-controlled XML off the network, so it is capped on URL count, index nesting, decompressed size and decompression ratio, and child sitemaps must stay on the host that named them. Closes #712 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: distinguish the sitemapindex log line from a urlset one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: fix an out-of-bounds read, tighten the parser and the tests lienrelatif() walked back from the last character of its current-path argument without checking the path was non-empty, reading one byte before the stack buffer. htsAddLink is the first caller to pass an empty savename, which sitemap documents have because they are ingested rather than mirrored, so ASan caught it on the new crawl test. The parser drops a value whose numeric character reference decodes outside printable ASCII, rather than leaving the reference verbatim and seeding a URL the site never published, and classifies a document by its real root element, so a comment naming the other one no longer flips urlset and sitemapindex. The robots.txt line reader is bounded by the body size instead of relying on a NUL terminator. The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only, because an uninstrumented libz floods it with false positives. Tests gain the assertions the earlier ones were missing: which of the robots.txt route and the /sitemap.xml fallback was taken, that the sitemap documents stay out of the mirror, that the off-host child sitemap is refused, the sitemapindex nesting cap, the per-document URL cap at its production value, and copy_htsopt coverage for the two new fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: state what the decompression cap actually binds on deflate tops out near 1032:1, so hts_codec_maxout never binds before the 64 MiB cap; the old comment implied a ratio guard that cannot fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: a child sitemap is a fetch, so filters and robots.txt must gate it An adversarial review found that a <sitemapindex> <loc>, and a robots.txt Sitemap: line, went straight to hts_record_link: the request went out even when a -* rule or robots.txt Disallow covered it. Only the <urlset> half ran through the wizard. Gate the document itself on the filters and on robots.txt, which is all that can apply: the wizard proper wants a referring link, and its up/down travel rules would judge a child sitemap against the parent sitemap's own directory. The robots.txt probe is exempt, being the request that fetches the rules. A 301 also used to end ingestion silently, since the engine re-queues the target as a fresh link that carried no sitemap marking. That hit any site redirecting http to https. The marking now follows the redirect. The "N URL(s) added" counter reported what the scanner emitted rather than what was taken, which hid both of the above; it now reads "N of M". The fallback to /sitemap.xml keys on the same corrected count, so a robots.txt whose only Sitemap: line is off-host or filtered still falls back. Root classification skips a UTF-8 BOM and an XML namespace prefix, and the doc list is cleared when a mirror starts rather than only when it ends. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: gate each fetch by who asked for it, and anchor travel on the start URL A nine-agent review found a scope escape: the sitemap document was its own `premier`, so the wizard measured travel from wherever the site chose to put its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the whole host. The ingester now points the wizard at the crawl's own start link and lets each seeded URL become its own anchor, which is what a command-line seed gets. Robots handling was both mistimed and undifferentiated. The Sitemap: lines are now collected by robots_parse, on the same body in the same fetch, and acted on after the parsed rules are installed rather than before; and the decision comes from a new hts_robots_forbids extracted out of the wizard, so the sitemap path inherits the -s1 filters-win override instead of a stricter hand-rolled check. The four fetches are no longer treated alike: a sitemap the user names is user intent, one the site declares invites the fetch, only the guessed /sitemap.xml obeys a Disallow, and the URLs listed inside stay fully gated. Also: hts_unescapeEntities replaces the private entity decoder, whose guard tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead, which is only defined under HTS_USEZLIB; the composed URL buffer now fits two maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the bounded search is promoted to htstools as hts_memstr; and the live state moves from httrackp into htsoptstate, leaving two installed fields rather than three. Tests gain the scope escape, the three robots cases, a cap-boundary control, the handler invocation count and a compression-bomb decode. Every one was checked against a deliberately broken build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: add the fuzz harness the parser was missing, and drop truncated Sitemap: lines The file header called the scanner fuzzable while fuzz/ registered ten harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies and truncated streams off a heap copy of exactly the input size, so an overread is an ASan report rather than a quiet pass, with a four-file seed corpus. 60000 runs clean under ASan+UBSan. robots_parse now drops a Sitemap: line that filled its scratch buffer instead of handing on the half URL it was truncated to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * fuzz: keep only the four sitemap seed inputs A libFuzzer run writes its finds into the first corpus directory, and 191 of them were committed with the harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * selftest: bound the sitemap document builders' snprintf accumulation snprintf returns the length it wanted to write, so accumulating it blind lets the next offset and size argument walk past the buffer. Guard each step the way the argv builder above already does, and give the per-URL loop a real remaining-space bound instead of a fixed 33. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: date the new files 2026 The headers were copied from an existing file and kept its 1998 year. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: keep the ingestion state out of htsoptstate htsoptstate is embedded by value as httrackp.state, so a field at its tail shifts every httrackp member declared after it: an offsetof probe put warc_file at 141752 on master and 141760 on the branch. Move the pointer to httrackp's own tail, where every existing offset holds and copy_htsopt still ignores it. Also renumber the crawl test to 89, master having taken 87 and 90, and give the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear requires, plus its row in that test's table. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: renumber the sitemap crawl test to 95 Master took 88 through 93 and #720 claims 94. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: realign the htsopt.h comments after the single-file merge Master's longer LLint declarator moved the block's comment column. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * sitemap: translate the new GUI strings into the remaining 28 locales (#738) The feature PR added the four LANG_SITEMAP* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Signed-off-by: Xavier Roche <xroche@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 07:59:59 +02:00
<p>Link-following only finds what something links to. Anything a site publishes
solely in its sitemap is invisible to HTTrack unless you ask for it.
<tt>--sitemap</tt> reads the start host's <tt>robots.txt</tt> for
<tt>Sitemap:</tt> lines and falls back to <tt>/sitemap.xml</tt>;
<tt>--sitemap-url</tt> names one directly. Nested <tt>sitemapindex</tt> files
and gzipped <tt>.xml.gz</tt> sitemaps are followed. The URLs found become start
URLs with the full depth budget, but they still go through your filters and
scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is
off by default because a sitemap can list thousands of pages nothing links
to.</p>
<p>One surprise worth knowing: a sitemap you name with <tt>--sitemap-url</tt>,
and one the site itself declares in <tt>robots.txt</tt>, are fetched even when
<tt>robots.txt</tt> disallows that path, because naming or declaring a sitemap
is an invitation to read it. Only the guessed <tt>/sitemap.xml</tt> obeys a
<tt>Disallow</tt>. The URLs listed inside are gated normally either way.</p>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<p>The single most common surprise is "only the home page came down." That is
usually not a scope option at all: it is an off-host redirect. A start URL of
<tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
lands you on a different host, and same-host scope stops the crawl there. Start
from the final URL, or add a filter that re-admits the real host (see
<a href="#filters">Filters</a>). The log will show the redirect.</p>
<p><tt>-n</tt> is the fix for pages that render locally without their images or
stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note
that its embedded-asset handling (following <tt>img</tt>, <tt>link</tt>,
<tt>script</tt>, <tt>style</tt> and HTML5 <tt>source</tt>/<tt>track</tt> targets
past the normal depth and filter limits) applies only when <tt>-n</tt> is on; it
is not automatic. It can also over-fetch by dragging in a whole external host from
a single link, in which case name the assets you want with a filter instead.</p>
<h3 id="filters">3. Filters and scan rules</h3>
<p>Filters are the number-one source of confusion, and also the tool that solves
most scope problems once you understand them. A filter is a rule that accepts
(<tt>+</tt>) or rejects (<tt>-</tt>) URLs by pattern. The sign is mandatory:
<tt>+pattern</tt> adds, <tt>-pattern</tt> removes, and a bare pattern is an error.</p>
<p>The rules that matter:</p>
<ul>
<li><b>Last match wins.</b> Rules are applied in order and the last one that
matches a URL decides its fate. Order your rules from general to specific.</li>
<li><b>Wildcards.</b> <tt>*</tt> matches any run of characters;
<tt>*[a-z]</tt>, <tt>*[0-9]</tt> and similar classes match sets. So
<tt>+*.pdf</tt> means "any URL ending in .pdf".</li>
<li><b>Whitelisting.</b> To keep one site and nothing else, deny everything then
re-admit the host: <tt>"-*" "+example.com/*"</tt>. A lone <tt>+</tt> rule only
adds to the default scope; it never restricts.</li>
<li><b>Size rules.</b> <tt>*[&gt;100000]</tt> and <tt>*[&lt;1000]</tt> filter by
byte size. Because size is only known once the transfer starts, an oversize file
is fetched partway and then aborted, not skipped for free.</li>
<li><b>mime: rules.</b> A rule like <tt>-mime:video/*</tt> matches the
<tt>Content-Type</tt>. That type is only known after the response headers arrive,
so a mime rule <b>cannot stop a request</b>; it can only abort the body. Use a
URL pattern when you want to avoid the fetch entirely.</li>
</ul>
<p>Quote your filters. Shells treat <tt>*</tt>, <tt>[</tt> and sometimes <tt>+</tt>
specially, so wrap each rule in quotes as shown above. The full pattern language,
with tables for wildcards, size and mime, is in
<a href="filters.html">the filters page</a>, and the
<a href="faq.html">FAQ</a> has a worked tutorial.</p>
<p><b>robots.txt.</b> By default HTTrack obeys <tt>robots.txt</tt> (<tt>-s2</tt>).
<tt>-s0</tt> ignores it entirely, <tt>-s1</tt> obeys it but lets one of your
<tt>+</tt> filters override a disallow for a URL you explicitly asked for. Note
that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
options will not help there. That is an
<a href="#identity">identity</a> problem.</p>
Document the filter wildcards in the command-line guide (#659) Add a "Filter wildcards" subsection listing the bracket wildcard forms the matcher (src/htsfilters.c) implements: *, *[file]/*[name], *[path], *[param], character sets and ranges, literal escapes, size rules, and the *[] end anchor. It links to the full filters page for the rest. Added as an h4 under the filters section so section numbering and the page ToC are untouched. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:59:50 +02:00
<h4>Filter wildcards</h4>
<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
bracket forms match narrower sets. The full table, with size and mime rules, is on
<a href="filters.html">the filters page</a>.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> &mdash; any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> &mdash; a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
<tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
<tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
<tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
<tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
<tr><td><tt>*[&lt;NN]</tt>, <tt>*[&gt;NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[&lt;5]</tt> skips GIFs under 5&nbsp;KB</td></tr>
<tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
</table>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<h3 id="limits">4. Limits and politeness</h3>
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
accident, and the <a href="abuse.html">abuse page</a> is worth a read. The limits
below let you go faster when you own the target, and slower when you do not.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--max-rate (-A)</tt></td><td>Maximum transfer rate in bytes/sec. <b>The default is about 100 KB/s even without this flag.</b> Raise it to go faster.</td></tr>
<tr><td><tt>--sockets (-c)</tt></td><td>Number of parallel connections (default 4). <tt>--tiny</tt>, <tt>--wide</tt> and <tt>--ultrawide</tt> are presets.</td></tr>
<tr><td><tt>--connection-per-second (-%c)</tt></td><td>New connections opened per second (default 5).</td></tr>
<tr><td><tt>--max-size (-M)</tt></td><td>Stop after N bytes <b>received from the network</b> across the whole mirror (this counts what was transferred, not what was saved).</td></tr>
<tr><td><tt>--max-time (-E)</tt></td><td>Stop after N seconds of wall-clock time.</td></tr>
<tr><td><tt>--max-files (-m)</tt></td><td>Per-file size caps.</td></tr>
<tr><td><tt>--timeout (-T), --retries (-R), --min-rate (-J), --host-control (-H)</tt></td><td>Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.</td></tr>
Advertise -%N's long option and document the real long forms (#652) * Advertise -%N's long option and document the real long forms The help display appends each option's long alias by looking it up in the alias table, but it first stripped a trailing N to turn placeholders like cN into c. That also turned -%N into -%, so --delayed-type-check was never advertised in --help (nor in the generated man page). Try the flag as-is first and only strip a trailing N on a miss; -%N now shows its long form, and cN still resolves to --sockets. The command-line guide had several rows marked short-only that in fact have long aliases (I had read them off a stale installed binary): fill in --pause, --strip-query, --disable-compression, the three --keep-* dedup opts, and --delayed-type-check. Regenerate man/httrack.1 to match, and drop the long-dead commented-out -%O chroot help line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Bound the option-token sscanf to the buffer size CodeQL flagged the %s read into cmd[32] as an unbounded copy on the line the previous commit touched. The tokens come from HTTrack's own help strings, not hostile input, so it was not reachable in practice, but the copy should be bounded regardless. Limit it to %30s (the buffer holds the leading '-' plus 30 chars and a NUL). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:21:50 +02:00
<tr><td><tt>--max-pause (-G), --pause (-%G)</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
</table>
<p><b>The security clamps.</b> To keep an accidental typo from turning into a flood,
HTTrack silently caps a few values: at most 8 connections (<tt>-c</tt>), at most
10 MB/s (<tt>-A</tt>), and at most 5 new connections per second (<tt>-%c</tt>).
Ask for more and you get the ceiling, quietly. The single flag
<tt>--disable-security-limits</tt> lifts all three (the short form <tt>-%!</tt>
also works, but the bare <tt>!</tt> is awkward to type safely in a shell). Use it
only against infrastructure you are allowed to load that hard.</p>
<h3 id="names">5. File names and types</h3>
<p>Where local files land, and what they are called, is controlled by the naming
options. This is the second-biggest source of "why did it do that" questions,
usually about a URL like <tt>/article?id=42</tt> or a <tt>.php</tt> page that is
really HTML.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
<tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
<tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
Advertise -%N's long option and document the real long forms (#652) * Advertise -%N's long option and document the real long forms The help display appends each option's long alias by looking it up in the alias table, but it first stripped a trailing N to turn placeholders like cN into c. That also turned -%N into -%, so --delayed-type-check was never advertised in --help (nor in the generated man page). Try the flag as-is first and only strip a trailing N on a miss; -%N now shows its long form, and cN still resolves to --sockets. The command-line guide had several rows marked short-only that in fact have long aliases (I had read them off a stale installed binary): fill in --pause, --strip-query, --disable-compression, the three --keep-* dedup opts, and --delayed-type-check. Regenerate man/httrack.1 to match, and drop the long-dead commented-out -%O chroot help line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Bound the option-token sscanf to the buffer size CodeQL flagged the %s read into cmd[32] as an unbounded copy on the line the previous commit touched. The tokens come from HTTrack's own help strings, not hostile input, so it was not reachable in practice, but the copy should be bounded regardless. Limit it to %30s (the buffer holds the leading '-' plus 30 chars and a NUL). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:21:50 +02:00
<tr><td><tt>--delayed-type-check (-%N), --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
<tr><td><tt>--include-query-string (-%q), --strip-query (-%g)</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
</table>
<p>The <tt>-N</tt> presets are built from modular arithmetic on the name fields, so
undocumented number combinations often "work" by accident. If you care about the
exact layout, use an explicit template (the <tt>%h %p %n %q %t</tt> placeholders)
rather than a magic number, and check the result on a small crawl first.</p>
<p>A dynamic page served as <tt>.php</tt> or <tt>.asp</tt> that is actually HTML is
the classic case: without help it can be saved with an extension a browser will
not open locally. <tt>--assume php=text/html</tt> fixes both the extension and the
naming.</p>
<h3 id="links">6. Links and page building</h3>
<p>After a page is downloaded, HTTrack parses it for more links and rewrites the
ones it kept so the local copy browses offline. These options tune both halves.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--keep-links (-K)</tt></td><td>How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare <tt>-K</tt> keeps <b>absolute</b> URLs, and <tt>-K0</tt> is the <b>relative</b> default. <tt>-K3</tt> keeps absolute URIs, <tt>-K4</tt> keeps the original links.</td></tr>
<tr><td><tt>--replace-external (-x), --generate-errors (-o)</tt></td><td>Replace external links with an error page, and generate an error page for links that failed.</td></tr>
<tr><td><tt>--preserve (-%p), --disable-passwords (-%x)</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
<tr><td><tt>--extended-parsing (-%P), --parse-java (-j)</tt></td><td>Aggressive link discovery, and how much script content is parsed for links.</td></tr>
<tr><td><tt>--mime-html (-%M)</tt></td><td>Save the whole mirror as a single MIME-encapsulated <tt>.mht</tt> archive (<tt>index.mht</tt>).</td></tr>
No single-file HTML output with inlined assets (#720) * Single-file output is MHT, which browsers no longer open Add --single-file (-%Z): after the mirror completes, rewrite every saved page in place with its stylesheets, scripts, images and fonts embedded as data: URIs, while links between pages stay relative. The mirror remains a browsable tree and each page also stands alone. This cannot reuse the -%M path: MHT streams a MIME part per file as it is saved, but a data: URI needs the asset's bytes when the page is written, and pages are normally saved before their assets are fetched. The new pass runs over the finished tree at the tail of httpmirror(), after the update purge. Audio, video, page-to-page links and anything over --single-file-max-size (10 MB default) keep an ordinary link. References carrying a scheme are skipped, which covers data: and makes a second --update run a no-op. Resolution is clamped to the mirror root: the HTML is hostile input. htsopt.h gains two tail-appended fields; VERSION_INFO is untouched. Closes #713 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: renumber the single-file test to 87, run the self-test last Master took 82 through 85 while this branch was open. The engine self-test also moves to the end of the script: it and the crawl assertions cover different ground, and failing first hid the crawl half. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * Fix the parser and path bugs review found, and cover them sf_relative_from() indexed one byte past from_dir's terminator whenever the page's own directory was a prefix of the asset path, which is the ordinary mirror layout: an out-of-bounds read that also miscounted the ../ prefix and emitted a broken link. Two review agents hit the same ASan trace; the branch had no test because every nested asset in the fixture was under the cap. Also from review: an escaped quote no longer ends a CSS string early (which exposed its contents to the url()/@import scanner), @import url(...) now inlines like the quoted form, a raw-text element ends only on a real end tag rather than any prefix of one, an over-wide tag is copied through by the same quote-aware scan instead of a second quote-blind one, <!--> is an empty comment, whitespace before a tag's > survives, and a page cannot inline more than SINGLEFILE_MAX_PAGE_SIZE, which bounds the multiplicative @import fan-out. A failed encode no longer leaves a payload-less data: prefix behind, and base_dir is matched against the root on a component boundary. sf_readfile now wraps a new readfile2_utf8() rather than being a fifth copy of the readfile family. Docs place --single-file beside -%M instead of implying it supersedes it: MHT stays the better container, this wins on opening anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * singlefile: tighten two comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: cover the option plumbing review found untested copy_htsopt's two new fields, including the >0 guard that must not let an unset source clear the target's default; -%Z0; and a rejected --single-file-max-size argument falling back to the built-in cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: stand the single-file GUI check down where htsserver is absent The Windows job builds httrack.exe only, and reaches this file through its *_local-*.test glob, so requiring htsserver failed the whole test there even though every crawl assertion had passed. Skip that half instead, and run the engine self-test before it so the skip cannot swallow it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: check the sprintfbuff master just made warn_unused_result #722 gave slprintfbuff the attribute, so the over-wide-tag fixture's call became the one warning this branch adds over master's baseline. Handle it the way that PR's own self-test code does; the buffer cannot truncate here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * single-file: translate the new GUI strings into the remaining 28 locales (#739) The feature PR added the four LANG_SINGLEFILE* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * single-file: restore the mirror file mode, and give the guards real coverage The rewrite spools to a .sfnew temp and renames it over the page, which bypasses filecreate() and with it the HTS_ACCESS_FILE chmod the engine puts on every other mirrored file. Under a restrictive umask the pages came out 0600 while their assets stayed 0644, so a mirror served by a webserver or shared with a group lost read access on exactly the pages. chmod the spool before the rename. Two guards had no coverage: deleting the scheme/data: check in sf_resolve left both the self-test and the crawl test passing, because the fixtures resolved to paths that were absent either way, and the per-page inline budget was never exercised. The fixtures now plant a file where each guard's removal would land the walk, and a self-importing stylesheet measures the budget against a large-budget control. Charging that budget after the nested rewrite instead of before let an @import chain spend what its ancestors had already claimed and drove it negative; charge it up front and refund on failure. The attribute table missed the lazy-loading attributes hts_detect[] already downloads, so a modern page inlined almost nothing: add data-src, data-srcset, lowsrc, object@data and embed@src, and record why the rest stay links. The new web GUI checkbox had no hidden companion input, so it could be ticked but never cleared, which is what #725 fixed for every other box. Master's test 90 catches it once the branch merges. Adds a libFuzzer harness over the rewriter, since it re-serializes hostile HTML, and renames the crawl test to 91 now that master holds 87 and 90. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * single-file: document the inlined-stylesheet limitation in the CLI guide Recorded in htssinglefile.h already; the user-facing guide is where someone raising --single-file-max-size will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: move the Windows note down to the gate it explains Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * ci: register the single-file test's Windows skip Its GUI half needs htsserver, which the Windows job does not build, so the test now exits 77 there instead of reporting a pass for assertions it never ran. The skip list is pinned, so it has to be declared. Both Windows jobs reported fail=0; only the list check was red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <xroche@gmail.com> Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 07:17:13 +02:00
<tr><td><tt>--single-file (-%Z), --single-file-max-size N</tt></td><td>Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded as <tt>data:</tt> URIs. Assets over the cap (10&nbsp;MB by default) keep their link, as do audio, video, and the links from one page to another. A sibling of <tt>-%M</tt>, not a replacement: see the recipe below for which to pick.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<tr><td><tt>--index (-I), --build-top-index (-%i), --search-index (-%I)</tt></td><td>Build a per-mirror index, a top index across projects, and a searchable keyword index.</td></tr>
</table>
<p>HTTrack finds links by parsing HTML and CSS. It does not run JavaScript, so any
URL a page builds at runtime in script (a lazy-loaded image, a
JavaScript-assembled path) is invisible to the crawler and will be missing from
the mirror. There is no flag that fixes this; the asset has to appear in the
static HTML or CSS to be found. <tt>-%P</tt> widens discovery for links that are
present but awkwardly formatted, not for links that do not exist until script
runs.</p>
<h3 id="identity">7. Identity, cookies and login</h3>
<p>By default HTTrack is an honest robot: it sends a <tt>User-Agent</tt> of
<tt>HTTrack</tt>, a Referer with each link (which reveals the crawl path to the
server), and obeys robots. Plenty of sites filter exactly that profile. These
options control what HTTrack says about itself.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--user-agent (-F)</tt></td><td>The <tt>User-Agent</tt>. Set a browser string to get past crawler blocks; <tt>--user-agent ""</tt> sends none.</td></tr>
<tr><td><tt>--referer (-%R), --from (-%E), --language (-%l), --accept (-%a)</tt></td><td>Referer, From, Accept-Language and Accept headers.</td></tr>
<tr><td><tt>--headers (-%X)</tt></td><td>Add raw header lines to every request.</td></tr>
Named footer fields for -%F (#667) * Add named {addr}{path}{date}{version} footer fields alongside the legacy %s form The -%F footer was a bare positional printf: each %s consumed the next of a fixed addr/path/date/version arg list. It is order-coupled and inexpressive (you cannot reach {date} without emitting addr and path first, cannot reorder, and a wrong %s count silently yields "???" or shifted values), and the help text advertised a cryptic bracket syntax. hts_footer_format now dispatches on content: a footer containing %s keeps the legacy positional model byte-for-byte (the default footer and every existing -%F string are unaffected), while a footer without %s uses named fields {addr} {path} {date} {version}, with "{{"/"}}" for a literal brace and any unrecognized {...} left verbatim so typos stay visible. Sanitization is unchanged: addr/path still pass through html_inline_safe() at the call site, so the comment-injection guard (#165) still holds. Driven by a new -#test=footerfmt self-test (tests/01_engine-footerfmt.test) covering both models, brace escaping, the mixed-mode dispatch boundary, and the overflow/zero-size return paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Regenerate html/httrack.man.html for the -%F help change The man/html sync CI guard requires html/httrack.man.html to change whenever man/httrack.1 does; the -%F help-text update regenerated the man page but not its html rendering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Add {url}{lastmodified}{mime}{charset}{status}{size} footer fields Extends the named footer set beyond addr/path/date/version. hts_footer_format now takes a {name,value} table instead of four fixed parameters, so the field set is data rather than signature and the legacy %s path looks addr/path/date/ version up by name (order-independent, still byte-for-byte). The call site supplies the new fields from the current response: {url} (scheme + host + path, credentials stripped like {addr}), {lastmodified}, {mime}, {charset}, {status} and {size}. Every network-derived string is html_inline_safe()'d, since the footer sits inside an HTML comment and a value holding "-->" would otherwise close it and inject markup (#165); {status}/{size} are formatted integers and need none. Documented in --help, the man pages and the Command-Line Guide, and covered by the -#test=footerfmt self-test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 18:15:12 +02:00
<tr><td><tt>--footer (-%F)</tt></td><td>A footer written into saved pages (on disk, not a network header). See <b>Footer fields</b> below.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<tr><td><tt>--cookies (-b), --cookies-file (-%K)</tt></td><td>Accept cookies, and preload a Netscape <tt>cookies.txt</tt>.</td></tr>
</table>
Named footer fields for -%F (#667) * Add named {addr}{path}{date}{version} footer fields alongside the legacy %s form The -%F footer was a bare positional printf: each %s consumed the next of a fixed addr/path/date/version arg list. It is order-coupled and inexpressive (you cannot reach {date} without emitting addr and path first, cannot reorder, and a wrong %s count silently yields "???" or shifted values), and the help text advertised a cryptic bracket syntax. hts_footer_format now dispatches on content: a footer containing %s keeps the legacy positional model byte-for-byte (the default footer and every existing -%F string are unaffected), while a footer without %s uses named fields {addr} {path} {date} {version}, with "{{"/"}}" for a literal brace and any unrecognized {...} left verbatim so typos stay visible. Sanitization is unchanged: addr/path still pass through html_inline_safe() at the call site, so the comment-injection guard (#165) still holds. Driven by a new -#test=footerfmt self-test (tests/01_engine-footerfmt.test) covering both models, brace escaping, the mixed-mode dispatch boundary, and the overflow/zero-size return paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Regenerate html/httrack.man.html for the -%F help change The man/html sync CI guard requires html/httrack.man.html to change whenever man/httrack.1 does; the -%F help-text update regenerated the man page but not its html rendering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Add {url}{lastmodified}{mime}{charset}{status}{size} footer fields Extends the named footer set beyond addr/path/date/version. hts_footer_format now takes a {name,value} table instead of four fixed parameters, so the field set is data rather than signature and the legacy %s path looks addr/path/date/ version up by name (order-independent, still byte-for-byte). The call site supplies the new fields from the current response: {url} (scheme + host + path, credentials stripped like {addr}), {lastmodified}, {mime}, {charset}, {status} and {size}. Every network-derived string is html_inline_safe()'d, since the footer sits inside an HTML comment and a value holding "-->" would otherwise close it and inject markup (#165); {status}/{size} are formatted integers and need none. Documented in --help, the man pages and the Command-Line Guide, and covered by the -#test=footerfmt self-test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 18:15:12 +02:00
<p><b>Footer fields.</b> A footer with no <tt>%s</tt> may reference named fields:
<tt>{addr}</tt>, <tt>{path}</tt>, <tt>{url}</tt>, <tt>{date}</tt> (mirror time),
<tt>{lastmodified}</tt> (the page's Last-Modified), <tt>{version}</tt>,
<tt>{mime}</tt>, <tt>{charset}</tt>, <tt>{status}</tt> and <tt>{size}</tt>; write
<tt>{{</tt> or <tt>}}</tt> for a literal brace. A footer that contains <tt>%s</tt>
keeps the older positional form (host, path, date in that order). Example:
<tt>-%F "&lt;!-- Mirrored from {url} on {date} --&gt;"</tt>.</p>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<p><b>Login.</b> For HTTP Basic auth, put the credentials in the URL:
<tt>http://user:pass@host/</tt>. An <tt>@</tt> inside the username must be written
<tt>%40</tt>. Only Basic is supported, not Digest.</p>
<p>For cookie or form logins, the simplest path is to log in with a browser, export
its <tt>cookies.txt</tt>, and drop that file in the project directory so HTTrack
sends the session cookie. For a form that needs a POST, <tt>--catchurl</tt> can
capture the exact request your browser sends and replay it. A few cookie caveats
to know: expiry is ignored, there is a silent cap of about 8 cookies sent per
request, and <tt>-b0</tt> disables cookies and the reuse of Basic credentials
across links at the same time.</p>
<h3 id="proxy">8. Proxy and network</h3>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
<tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
<tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
Advertise -%N's long option and document the real long forms (#652) * Advertise -%N's long option and document the real long forms The help display appends each option's long alias by looking it up in the alias table, but it first stripped a trailing N to turn placeholders like cN into c. That also turned -%N into -%, so --delayed-type-check was never advertised in --help (nor in the generated man page). Try the flag as-is first and only strip a trailing N on a miss; -%N now shows its long form, and cN still resolves to --sockets. The command-line guide had several rows marked short-only that in fact have long aliases (I had read them off a stale installed binary): fill in --pause, --strip-query, --disable-compression, the three --keep-* dedup opts, and --delayed-type-check. Regenerate man/httrack.1 to match, and drop the long-dead commented-out -%O chroot help line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Bound the option-token sscanf to the buffer size CodeQL flagged the %s read into cmd[32] as an unbounded copy on the line the previous commit touched. The tokens come from HTTrack's own help strings, not hostile input, so it was not reachable in practice, but the copy should be bounded regardless. Limit it to %30s (the buffer holds the leading '-' plus 30 chars and a NUL). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:21:50 +02:00
<tr><td><tt>--http-10 (-%h), --keep-alive (-%k), --disable-compression (-%z)</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<tr><td><tt>--bind (-%b), --tolerant (-%B)</tt></td><td>Bind to a local address, and accept technically-bogus responses some servers send.</td></tr>
</table>
<p>Two network facts worth stating plainly. Over SOCKS5, HTTrack always resolves
host names at the proxy (remote DNS) for both <tt>socks5://</tt> and
<tt>socks5h://</tt>, so your local resolver is never consulted. And HTTrack does
not verify TLS certificates: HTTPS gives you an encrypted transport, but not an
authenticated one. That is a deliberate choice for a mirroring tool, not a bug,
but it is worth knowing if you are relying on it for trust.</p>
<h3 id="update">9. Update and cache</h3>
<p>Every project keeps a cache under <tt>hts-cache/</tt>. It records every URL that
was fetched, together with the options you used, and it is what makes resuming and
updating possible. It is not a size-limited scratch area you can delete: throw it
away and you lose the ability to continue or update the mirror.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--continue</tt></td><td>Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.</td></tr>
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
<tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
<tr><td><tt>--cache (-C)</tt></td><td>Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.</td></tr>
Advertise -%N's long option and document the real long forms (#652) * Advertise -%N's long option and document the real long forms The help display appends each option's long alias by looking it up in the alias table, but it first stripped a trailing N to turn placeholders like cN into c. That also turned -%N into -%, so --delayed-type-check was never advertised in --help (nor in the generated man page). Try the flag as-is first and only strip a trailing N on a miss; -%N now shows its long form, and cN still resolves to --sockets. The command-line guide had several rows marked short-only that in fact have long aliases (I had read them off a stale installed binary): fill in --pause, --strip-query, --disable-compression, the three --keep-* dedup opts, and --delayed-type-check. Regenerate man/httrack.1 to match, and drop the long-dead commented-out -%O chroot help line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Bound the option-token sscanf to the buffer size CodeQL flagged the %s read into cmd[32] as an unbounded copy on the line the previous commit touched. The tokens come from HTTrack's own help strings, not hostile input, so it was not reachable in practice, but the copy should be bounded regardless. Limit it to %30s (the buffer holds the leading '-' plus 30 chars and a NUL). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:21:50 +02:00
<tr><td><tt>--urlhack (-%u), --keep-www-prefix (-%j), --keep-double-slashes (-%o), --keep-query-order (-%y), --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<tr><td><tt>--debug-cache (-#C), --repair-cache (-#R), --clean</tt></td><td>Inspect the cache, repair its ZIP, and erase cache plus logs.</td></tr>
</table>
<p><b>The purge trap.</b> An <tt>--update</tt> run rebuilds the list of files the
mirror should contain, then deletes any previously-mirrored file that is not on the
new list. This is what keeps a mirror in sync with a shrinking site, but it means a
partial or interrupted update can delete files you meant to keep. If an update might
not complete cleanly, add <tt>-X0</tt> to protect the existing tree, and expect
dynamic pages to look "changed" on every run. See the
<a href="cache.html">cache page</a> for the details.</p>
<h3 id="experts">10. Experts and scripting</h3>
<p>HTTrack is also a scriptable fetch-and-scan tool. These options turn off the
mirror behavior and expose the engine.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
<tr><td><tt>--get URL</tt></td><td>Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.</td></tr>
<tr><td><tt>--spider --testlinks --skeleton</tt></td><td>Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.</td></tr>
<tr><td><tt>--userdef-cmd (-V)</tt></td><td>Run a shell command on each downloaded file; <tt>$0</tt> is the file path. Good for on-the-fly processing.</td></tr>
<tr><td><tt>--callback (-%W)</tt></td><td>Load an external callback module to hook the engine.</td></tr>
<tr><td><tt>--do-not-log (-Q), --quiet (-q), --verbose (-v), --file-log (-f), --extra-log (-z), --debug-log (-Z)</tt></td><td>Logging: quiet, no questions, verbose on screen, and the various log-to-file levels.</td></tr>
</table>
<p>The <a href="dev.html">developer page</a> covers the callback API and batch use
in more depth.</p>
<h3 id="recipes">11. Recipes</h3>
<p>Copy-ready command lines for the tasks people ask about most. Each has the one
gotcha that trips it up.</p>
<h4>Mirror one site, nothing off-host</h4>
<p><tt>httrack https://example.com/ --path mydir</tt><br>
<small>Same-host is already the default. The usual failure is a
<tt>www.</tt>-to-apex or <tt>http</tt>-to-<tt>https</tt> redirect that moves you off
host and stops the crawl; start from the final URL, or add
<tt>"+finalhost/*"</tt>.</small></p>
<h4>One level of links only</h4>
<p><tt>httrack https://example.com/ --depth=2 --path mydir</tt><br>
<small>Depth counts the start page as level 1, so one level out is <tt>--depth=2</tt>.</small></p>
<h4>This site only, deny everything else</h4>
<p><tt>httrack https://example.com/ "-*" "+example.com/*" --path mydir</tt><br>
<small>You must deny-all first; a lone <tt>+</tt> only adds, and the last matching
rule wins.</small></p>
<h4>Download the PDFs on a site</h4>
Fix the download-PDFs example in the command-line guide (#658) * Fix the download-PDFs example in the command-line guide The old example ran httrack with no filters, so it mirrored the whole site rather than downloading its PDFs. The corrected command restricts to the site, admits the HTML pages as scaffolding for link discovery (including directory-index pages via *[file]/ so sub-directory PDFs are found), keeps the PDFs, and drops everything else. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Use *[path]/ so the PDF recipe follows nested directory indexes *[file] cannot cross a slash, so *[file]/ only admits a single directory level and silently drops PDFs linked under nested paths (dated archives, doc trees). *[path]/ spans slashes and follows directory-index pages at any depth, with no over-admission (only trailing-slash URLs match, never files). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 12:01:02 +02:00
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<h4>Keep page requisites, including off-host images</h4>
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>
<small><tt>--near</tt> can over-fetch by pulling in an entire external host from one
link; if it does, name the assets with <tt>"+cdn.example.com/*"</tt> instead.</small></p>
<h4>Resume an interrupted crawl</h4>
<p><tt>httrack --continue --path mydir</tt><br>
<small>Needs an intact <tt>hts-cache/</tt>. Deleting it loses the URL list and your
options.</small></p>
<h4>Update a mirror, downloading only what changed</h4>
<p><tt>httrack --update --path mydir</tt><br>
<small>Purge deletes any previously-mirrored file not seen this run; add
<tt>--purge-old=0</tt> to protect the tree against a partial update.</small></p>
<h4>Reach a section behind a login cookie</h4>
<p><small>Export your browser's <tt>cookies.txt</tt> into the project directory,
then:</small><br>
<tt>httrack https://example.com/members/ --path mydir</tt><br>
<small>The file must be Netscape <tt>cookies.txt</tt> in the project folder;
<tt>--cookies=0</tt> would disable cookies and Basic-auth reuse together.</small></p>
<h4>Get past a crawler-UA block</h4>
<p><tt>httrack https://example.com/ --user-agent "Mozilla/5.0 (Windows NT 10.0)" --path mydir</tt><br>
<small>A 403 is server-side, not robots: change the User-Agent, and add
<tt>--http-10</tt> for fragile CGI. Do not reach for <tt>--robots=0</tt>.</small></p>
<h4>Full-speed mirror on your own server</h4>
<p><tt>httrack https://example.com/ --max-rate=2000000 --sockets=8 --disable-security-limits --path mydir</tt><br>
<small>The default throttles to about 100 KB/s. <tt>--disable-security-limits</tt>
lifts the built-in caps; use it only against infrastructure you are allowed to
load.</small></p>
cmdguide: add a WARC/WACZ archive recipe (#680) The command-line guide's recipe section had no entry for WARC output, so the five --warc* flags were discoverable only through --help or the man page. Add a §11 recipe leading with the gotcha that trips people: the archive is written alongside the browsable mirror, not instead of it. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 08:09:16 +02:00
<h4>Save a WARC archive of the crawl</h4>
<p><tt>httrack https://example.com/ --warc --path mydir</tt><br>
<small>Writes a standard WARC/1.1 file (<tt>httrack-&lt;timestamp&gt;.warc.gz</tt>) in
the project folder alongside the browsable mirror, not instead of it. Set the name
with <tt>--warc-file NAME</tt> and split a large crawl with <tt>--warc-max-size N</tt>;
add <tt>--warc-cdx</tt> for a sorted CDXJ index, or <tt>--wacz</tt> to bundle the
archive, index and pages into one WACZ for replay tools such as
replayweb.page.</small></p>
No way to see what changed between two mirrors (#721) * Report what a crawl changed against the previous mirror (--changes) --update already knows which resources were new, which changed and which the server called unchanged, and throws it away: the flags reach file_notify() and go no further than a log line, while deletions exist only as a side effect of purging. --changes (-%d) keeps all of it and writes hts-changes.json plus a one-line summary in the log. "Changed" means the bytes differ, not that the server re-sent the resource. Comparing the mirrored files directly would not work: HTTrack stamps every parsed page with the crawl date via the footer, so those bytes differ on every run. Payloads are compared instead, the previous one coming from the cache for parsed pages and from the local copy sampled just before it is overwritten for everything else. The mirror-relative path, not the URL, is the accumulator's key, so a redirect and its target that share a save name are one entry; and only the first notify for a file samples its pre-run state, so a retried transfer is not counted twice. What counts as already mirrored comes from the previous run's file index rather than from the file's presence on disk: a partial left by this crawl's own failed attempt is on disk but was never part of the previous mirror. The deleted set is now computed whether or not purging is enabled; unlinking still happens only under --purge-old. Closes #714 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: never leave a stale report behind A crawl that mirrored nothing created no accumulator, so hts_changes_close_opt returned without writing and the previous run's report stayed on disk as if it described this one. Write it whenever --changes is on. The no-data rollback is the deliberate exception, and is now documented: it restores the previous cache generation, so leaving the matching report alone is the consistent behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: drop em dashes from the format page Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: fix the review's three findings The size shortcut compared rendered on-disk sizes even for parsed pages, whose payload digests describe something else entirely, so it decided the outcome before the payload comparison could run: a page whose payload never changed but whose rewritten links moved read as changed. It now only applies when both digests describe the file on disk. file_notify() reaches the accumulator from the FTP download thread as well as the main one, and the lazy allocation, the coucal write and the entries realloc were all unguarded. Every entry point now takes a mutex, and the HTML hook does its cache read before taking it, since that read can itself re-enter file_notify() and move the array. Two fixtures cover what nothing did: a gzipped direct-to-disk body that changes at constant length (without the pre-sample before the decoded temp is renamed, it reads as unchanged), and a page with a fixed payload behind a redirect whose target is renamed, so its file on disk changes length while its bytes do not. Both were checked against builds with the respective fix reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: document the renamed-file case in the format page Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: refresh two stale test comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * Merge origin/master into feat/change-report Both sides appended to tests/Makefile.am's TESTS; kept master's 86_local-proxytrack-cache-longfields.test alongside 88_local-changes.test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: translate the new GUI strings into the remaining 28 locales (#740) The feature PR added the two LANG_CHANGES* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * changes: fix the review's blocking findings Lock the report path against the FTP thread the crawl never joins, seal the accumulator once the report is written, key entries off the project directory so the report survives --cache=0, stop calling a file gone when the crawl only failed to re-fetch it, and skip the hook's work entirely when --changes is off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * changes: prove the fixes, and give the report a cache-off mode Adds a changes-race self-test (the FTP shape: notifier threads against the report path), fixtures for a transfer the crawl never completes, for a leftover file at a name the crawl mirrors fresh, and for a cache-off mirror, plus a pass with purging on. Registers the web GUI's --changes box with the clearing companion master's #725 now requires, and documents the degraded mode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * changes: keep the failed-transfer case portable A connection killed before the status line surfaces differently on macOS, where it truncates the mirrored file to zero (#748). Assert only what holds on both: it is never reported gone, and its file survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <xroche@gmail.com> Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 06:58:33 +02:00
<h4>See what a re-crawl changed</h4>
<p><tt>httrack https://example.com/ --update --changes --path mydir</tt><br>
<small>Writes <tt>hts-changes.json</tt> in the project folder, listing every
mirrored file as new, changed, unchanged or gone, plus a one-line summary in the
log. &quot;Changed&quot; means the bytes really differ: a server that answers 200
with the same content it served last time lands in <tt>unchanged</tt>. Deletions
are reported whether or not <tt>--purge-old</tt> is deleting them. The format is
documented in <a href="changes.html">the change report specification</a>.</small></p>
No single-file HTML output with inlined assets (#720) * Single-file output is MHT, which browsers no longer open Add --single-file (-%Z): after the mirror completes, rewrite every saved page in place with its stylesheets, scripts, images and fonts embedded as data: URIs, while links between pages stay relative. The mirror remains a browsable tree and each page also stands alone. This cannot reuse the -%M path: MHT streams a MIME part per file as it is saved, but a data: URI needs the asset's bytes when the page is written, and pages are normally saved before their assets are fetched. The new pass runs over the finished tree at the tail of httpmirror(), after the update purge. Audio, video, page-to-page links and anything over --single-file-max-size (10 MB default) keep an ordinary link. References carrying a scheme are skipped, which covers data: and makes a second --update run a no-op. Resolution is clamped to the mirror root: the HTML is hostile input. htsopt.h gains two tail-appended fields; VERSION_INFO is untouched. Closes #713 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: renumber the single-file test to 87, run the self-test last Master took 82 through 85 while this branch was open. The engine self-test also moves to the end of the script: it and the crawl assertions cover different ground, and failing first hid the crawl half. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * Fix the parser and path bugs review found, and cover them sf_relative_from() indexed one byte past from_dir's terminator whenever the page's own directory was a prefix of the asset path, which is the ordinary mirror layout: an out-of-bounds read that also miscounted the ../ prefix and emitted a broken link. Two review agents hit the same ASan trace; the branch had no test because every nested asset in the fixture was under the cap. Also from review: an escaped quote no longer ends a CSS string early (which exposed its contents to the url()/@import scanner), @import url(...) now inlines like the quoted form, a raw-text element ends only on a real end tag rather than any prefix of one, an over-wide tag is copied through by the same quote-aware scan instead of a second quote-blind one, <!--> is an empty comment, whitespace before a tag's > survives, and a page cannot inline more than SINGLEFILE_MAX_PAGE_SIZE, which bounds the multiplicative @import fan-out. A failed encode no longer leaves a payload-less data: prefix behind, and base_dir is matched against the root on a component boundary. sf_readfile now wraps a new readfile2_utf8() rather than being a fifth copy of the readfile family. Docs place --single-file beside -%M instead of implying it supersedes it: MHT stays the better container, this wins on opening anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * singlefile: tighten two comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: cover the option plumbing review found untested copy_htsopt's two new fields, including the >0 guard that must not let an unset source clear the target's default; -%Z0; and a rejected --single-file-max-size argument falling back to the built-in cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: stand the single-file GUI check down where htsserver is absent The Windows job builds httrack.exe only, and reaches this file through its *_local-*.test glob, so requiring htsserver failed the whole test there even though every crawl assertion had passed. Skip that half instead, and run the engine self-test before it so the skip cannot swallow it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * tests: check the sprintfbuff master just made warn_unused_result #722 gave slprintfbuff the attribute, so the over-wide-tag fixture's call became the one warning this branch adds over master's baseline. Handle it the way that PR's own self-test code does; the buffer cannot truncate here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * single-file: translate the new GUI strings into the remaining 28 locales (#739) The feature PR added the four LANG_SINGLEFILE* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * single-file: restore the mirror file mode, and give the guards real coverage The rewrite spools to a .sfnew temp and renames it over the page, which bypasses filecreate() and with it the HTS_ACCESS_FILE chmod the engine puts on every other mirrored file. Under a restrictive umask the pages came out 0600 while their assets stayed 0644, so a mirror served by a webserver or shared with a group lost read access on exactly the pages. chmod the spool before the rename. Two guards had no coverage: deleting the scheme/data: check in sf_resolve left both the self-test and the crawl test passing, because the fixtures resolved to paths that were absent either way, and the per-page inline budget was never exercised. The fixtures now plant a file where each guard's removal would land the walk, and a self-importing stylesheet measures the budget against a large-budget control. Charging that budget after the nested rewrite instead of before let an @import chain spend what its ancestors had already claimed and drove it negative; charge it up front and refund on failure. The attribute table missed the lazy-loading attributes hts_detect[] already downloads, so a modern page inlined almost nothing: add data-src, data-srcset, lowsrc, object@data and embed@src, and record why the rest stay links. The new web GUI checkbox had no hidden companion input, so it could be ticked but never cleared, which is what #725 fixed for every other box. Master's test 90 catches it once the branch merges. Adds a libFuzzer harness over the rewriter, since it re-serializes hostile HTML, and renames the crawl test to 91 now that master holds 87 and 90. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * single-file: document the inlined-stylesheet limitation in the CLI guide Recorded in htssinglefile.h already; the user-facing guide is where someone raising --single-file-max-size will look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * tests: move the Windows note down to the gate it explains Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * ci: register the single-file test's Windows skip Its GUI half needs htsserver, which the Windows job does not build, so the test now exits 77 there instead of reporting a pass for assertions it never ran. The skip list is pinned, so it has to be declared. Both Windows jobs reported fail=0; only the list check was red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <xroche@gmail.com> Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 07:17:13 +02:00
<h4>Pages you can hand to someone as one file</h4>
<p><tt>httrack https://example.com/ --single-file --path mydir</tt><br>
<small>Rewrites each saved page after the crawl so its stylesheets, scripts,
images and fonts are embedded as <tt>data:</tt> URIs. The mirror stays a normal
browsable tree, with links between pages relative and the assets still on disk,
but any single <tt>.html</tt> file can now be mailed or archived on its own.
Raise or lower the 10&nbsp;MB per-asset limit with
<tt>--single-file-max-size N</tt>; anything over it, plus audio and video, keeps
its link. One caveat if you raise it: an inlined stylesheet becomes a
<tt>data:</tt> URL, whose path is opaque, so an asset it referenced that stayed
over the cap no longer resolves from inside it. Raising the cap past that asset
embeds it too and the question goes away.<br>
Reach for this when the file has to open for someone you cannot make assumptions
about: it is plain HTML and needs no add-on. Reach for <tt>--mime-html</tt>
instead when a Chromium-family browser is a given and the mirror is large: MIME
carries text parts without the base64 tax, keeps each resource's original URL,
and stores a shared asset once rather than re-embedding it in every page that
uses it.</small></p>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<h4>HTTrack as a fetch tool</h4>
<p><tt>httrack --get https://host/file.bin --path tmp</tt><br>
<small><tt>--get</tt> fetches one file with cache, index, depth, cookies and robots
all disabled.</small></p>
<p><br>For the complete, always-current option list, see
<a href="httrack.man.html">the manual page</a>. For the filter language, see
<a href="filters.html">filters</a>; for the cache and updates, see
No way to see what changed between two mirrors (#721) * Report what a crawl changed against the previous mirror (--changes) --update already knows which resources were new, which changed and which the server called unchanged, and throws it away: the flags reach file_notify() and go no further than a log line, while deletions exist only as a side effect of purging. --changes (-%d) keeps all of it and writes hts-changes.json plus a one-line summary in the log. "Changed" means the bytes differ, not that the server re-sent the resource. Comparing the mirrored files directly would not work: HTTrack stamps every parsed page with the crawl date via the footer, so those bytes differ on every run. Payloads are compared instead, the previous one coming from the cache for parsed pages and from the local copy sampled just before it is overwritten for everything else. The mirror-relative path, not the URL, is the accumulator's key, so a redirect and its target that share a save name are one entry; and only the first notify for a file samples its pre-run state, so a retried transfer is not counted twice. What counts as already mirrored comes from the previous run's file index rather than from the file's presence on disk: a partial left by this crawl's own failed attempt is on disk but was never part of the previous mirror. The deleted set is now computed whether or not purging is enabled; unlinking still happens only under --purge-old. Closes #714 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: never leave a stale report behind A crawl that mirrored nothing created no accumulator, so hts_changes_close_opt returned without writing and the previous run's report stayed on disk as if it described this one. Write it whenever --changes is on. The no-data rollback is the deliberate exception, and is now documented: it restores the previous cache generation, so leaving the matching report alone is the consistent behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: drop em dashes from the format page Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: fix the review's three findings The size shortcut compared rendered on-disk sizes even for parsed pages, whose payload digests describe something else entirely, so it decided the outcome before the payload comparison could run: a page whose payload never changed but whose rewritten links moved read as changed. It now only applies when both digests describe the file on disk. file_notify() reaches the accumulator from the FTP download thread as well as the main one, and the lazy allocation, the coucal write and the entries realloc were all unguarded. Every entry point now takes a mutex, and the HTML hook does its cache read before taking it, since that read can itself re-enter file_notify() and move the array. Two fixtures cover what nothing did: a gzipped direct-to-disk body that changes at constant length (without the pre-sample before the decoded temp is renamed, it reads as unchanged), and a page with a fixed payload behind a redirect whose target is renamed, so its file on disk changes length while its bytes do not. Both were checked against builds with the respective fix reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: document the renamed-file case in the format page Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: refresh two stale test comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * Merge origin/master into feat/change-report Both sides appended to tests/Makefile.am's TESTS; kept master's 86_local-proxytrack-cache-longfields.test alongside 88_local-changes.test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com> * changes: translate the new GUI strings into the remaining 28 locales (#740) The feature PR added the two LANG_CHANGES* entries to lang.def with English and Francais only; every other locale fell back to English in the WebHTTrack form. Each file is written in its own declared charset. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * changes: fix the review's blocking findings Lock the report path against the FTP thread the crawl never joins, seal the accumulator once the report is written, key entries off the project directory so the report survives --cache=0, stop calling a file gone when the crawl only failed to re-fetch it, and skip the hook's work entirely when --changes is off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * changes: prove the fixes, and give the report a cache-off mode Adds a changes-race self-test (the FTP shape: notifier threads against the report path), fixtures for a transfer the crawl never completes, for a leftover file at a name the crawl mirrors fresh, and for a cache-off mirror, plus a pass with purging on. Registers the web GUI's --changes box with the clearing companion master's #725 now requires, and documents the degraded mode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * changes: keep the failed-transfer case portable A connection killed before the status line surfaces differently on macOS, where it truncates the mirrored file to zero (#748). Assert only what holds on both: it is never reported gone, and its file survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <xroche@gmail.com> Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 06:58:33 +02:00
<a href="cache.html">cache</a>; for the change report, see
<a href="changes.html">changes</a>.</p>
Add a task-oriented command-line guide to the offline docs (#649) * Add a task-oriented command-line guide to the offline docs The command-line docs so far are the option list and the generated manual page: exhaustive, but organized by flag, not by task. Newcomers arrive expecting wget/curl syntax and hit the same walls (only the index came down, filters that do the opposite of what they read, an update that deletes files), because the reference answers "what does -X do", not "how do I do the thing I want". cmdguide.html is a task-oriented layer on top of the manual page: quick start, scope, filters, limits, naming, links, identity/login, proxy, update/cache, scripting, then eleven copy-ready recipes with the one gotcha each. It foregrounds the defaults that actually surprise people (the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge semantics, the depth off-by-one, mime filters running after headers) and links the manual page for per-option detail. Linked from cmddoc.html and the index. Every documented default and behavior was checked against the engine source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Correct four scope/option descriptions in the guide Adversarial review against the engine source caught four mislabels: -d is "same principal domain" and -l is "same TLD" (not "same directory" and "same domain"); -U goes up only and -B goes both ways (the guide had -U as up-and-down and -B as "anywhere"); -%M archives the whole mirror into one index.mht, not each page; and -t HEAD-tests links outside scope rather than reporting "what a scope would reach". The rest of the guide's documented behavior verified against code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Fix the PDF recipe, prefer long options, drop the section rules Review feedback from Xavier: - The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages that carry the PDF links, so the crawl only keeps PDFs linked from the entry page (verified on a local fixture). There is no PDF-only crawl: HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse, with the off-host case handled by a `+host/*.pdf` rule. - Recipes now use long options throughout. `-%!` in particular is replaced by `--disable-security-limits`: the bare `!` triggers shell history expansion and is easy to fumble, and the long name says what it does. - Dropped the per-section `<hr>` rules; the sibling doc pages don't use them and the section headings already separate the content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Key the reference tables by long option name, short in parens Follows the recipe conversion: every table row now leads with the long option (--depth, --stay-on-same-domain, ...) and carries the short flag in parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y, -%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command snippets in descriptions to long form (--assume, --structure, --user-agent ""). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:27:04 +02:00
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>
</body>
</html>
Reference in New Issue Copy Permalink