rms/httrack
1
0
Fork 0
mirror of https://github.com/xroche/httrack.git synced 2026-07-28 03:22:12 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
2fccb170583d9c36738aff0f106eeb6be71c19fa
httrack/html/httrack.man.html
Xavier Roche 9e29c1e159 Sitemap files are never read, so URLs nothing links to are never found (#718) 
* Read sitemap files so URLs nothing links to are found

HTTrack finds URLs only by parsing links, so anything a site publishes solely
in its sitemap stayed invisible: robots.txt was already parsed, but its
Sitemap: lines were ignored and nothing else in the tree touched sitemaps.

Adds opt-in --sitemap (-%m), which probes the start host's robots.txt and
falls back to /sitemap.xml, and --sitemap-url (-%mu) for an explicit document.
Handles <urlset> and nested <sitemapindex>, plain or gzipped. Discovered URLs
enter with the full depth budget but still go through the wizard, so filters
and scope rules decide; a sitemap is not a filter bypass.

The parser reads attacker-controlled XML off the network, so it is capped on
URL count, index nesting, decompressed size and decompression ratio, and child
sitemaps must stay on the host that named them.

Closes #712

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: distinguish the sitemapindex log line from a urlset one

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: fix an out-of-bounds read, tighten the parser and the tests

lienrelatif() walked back from the last character of its current-path
argument without checking the path was non-empty, reading one byte before
the stack buffer. htsAddLink is the first caller to pass an empty savename,
which sitemap documents have because they are ingested rather than mirrored,
so ASan caught it on the new crawl test.

The parser drops a value whose numeric character reference decodes outside
printable ASCII, rather than leaving the reference verbatim and seeding a URL
the site never published, and classifies a document by its real root element,
so a comment naming the other one no longer flips urlset and sitemapindex.
The robots.txt line reader is bounded by the body size instead of relying on
a NUL terminator.

The self-test moves to 01_zlib-sitemap.test: MSan runs 01_engine-* only,
because an uninstrumented libz floods it with false positives.

Tests gain the assertions the earlier ones were missing: which of the
robots.txt route and the /sitemap.xml fallback was taken, that the sitemap
documents stay out of the mirror, that the off-host child sitemap is refused,
the sitemapindex nesting cap, the per-document URL cap at its production
value, and copy_htsopt coverage for the two new fields.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: state what the decompression cap actually binds on

deflate tops out near 1032:1, so hts_codec_maxout never binds before the
64 MiB cap; the old comment implied a ratio guard that cannot fire.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: a child sitemap is a fetch, so filters and robots.txt must gate it

An adversarial review found that a <sitemapindex> <loc>, and a robots.txt
Sitemap: line, went straight to hts_record_link: the request went out even
when a -* rule or robots.txt Disallow covered it. Only the <urlset> half ran
through the wizard. Gate the document itself on the filters and on
robots.txt, which is all that can apply: the wizard proper wants a referring
link, and its up/down travel rules would judge a child sitemap against the
parent sitemap's own directory. The robots.txt probe is exempt, being the
request that fetches the rules.

A 301 also used to end ingestion silently, since the engine re-queues the
target as a fresh link that carried no sitemap marking. That hit any site
redirecting http to https. The marking now follows the redirect.

The "N URL(s) added" counter reported what the scanner emitted rather than
what was taken, which hid both of the above; it now reads "N of M". The
fallback to /sitemap.xml keys on the same corrected count, so a robots.txt
whose only Sitemap: line is off-host or filtered still falls back. Root
classification skips a UTF-8 BOM and an XML namespace prefix, and the doc
list is cleared when a mirror starts rather than only when it ends.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: gate each fetch by who asked for it, and anchor travel on the start URL

A nine-agent review found a scope escape: the sitemap document was its own
`premier`, so the wizard measured travel from wherever the site chose to put
its sitemap. A root /sitemap.xml therefore widened a /deep/dir/ crawl to the
whole host. The ingester now points the wizard at the crawl's own start link
and lets each seeded URL become its own anchor, which is what a command-line
seed gets.

Robots handling was both mistimed and undifferentiated. The Sitemap: lines are
now collected by robots_parse, on the same body in the same fetch, and acted on
after the parsed rules are installed rather than before; and the decision comes
from a new hts_robots_forbids extracted out of the wizard, so the sitemap path
inherits the -s1 filters-win override instead of a stricter hand-rolled check.
The four fetches are no longer treated alike: a sitemap the user names is user
intent, one the site declares invites the fetch, only the guessed /sitemap.xml
obeys a Disallow, and the URLs listed inside stay fully gated.

Also: hts_unescapeEntities replaces the private entity decoder, whose guard
tests and fuzz corpus it silently forfeited; hts_codec_head replaces hts_zhead,
which is only defined under HTS_USEZLIB; the composed URL buffer now fits two
maximal components plus a scheme, which a 2046-byte --sitemap-url reached; the
bounded search is promoted to htstools as hts_memstr; and the live state moves
from httrackp into htsoptstate, leaving two installed fields rather than three.

Tests gain the scope escape, the three robots cases, a cap-boundary control,
the handler invocation count and a compression-bomb decode. Every one was
checked against a deliberately broken build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: add the fuzz harness the parser was missing, and drop truncated Sitemap: lines

The file header called the scanner fuzzable while fuzz/ registered ten
harnesses and none for it. fuzz-sitemap feeds it raw XML, gzip-framed bodies
and truncated streams off a heap copy of exactly the input size, so an overread
is an ASan report rather than a quiet pass, with a four-file seed corpus.
60000 runs clean under ASan+UBSan.

robots_parse now drops a Sitemap: line that filled its scratch buffer instead
of handing on the half URL it was truncated to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* fuzz: keep only the four sitemap seed inputs

A libFuzzer run writes its finds into the first corpus directory, and 191 of
them were committed with the harness.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* selftest: bound the sitemap document builders' snprintf accumulation

snprintf returns the length it wanted to write, so accumulating it blind
lets the next offset and size argument walk past the buffer. Guard each
step the way the argv builder above already does, and give the per-URL
loop a real remaining-space bound instead of a fixed 33.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: date the new files 2026

The headers were copied from an existing file and kept its 1998 year.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: keep the ingestion state out of htsoptstate

htsoptstate is embedded by value as httrackp.state, so a field at its tail
shifts every httrackp member declared after it: an offsetof probe put
warc_file at 141752 on master and 141760 on the branch. Move the pointer to
httrackp's own tail, where every existing offset holds and copy_htsopt still
ignores it.

Also renumber the crawl test to 89, master having taken 87 and 90, and give
the new option8 checkbox the hidden companion that 90_webhttrack-checkbox-clear
requires, plus its row in that test's table.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* tests: renumber the sitemap crawl test to 95

Master took 88 through 93 and #720 claims 94.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: realign the htsopt.h comments after the single-file merge

Master's longer LLint declarator moved the block's comment column.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* sitemap: translate the new GUI strings into the remaining 28 locales (#738)

The feature PR added the four LANG_SITEMAP* entries to lang.def with English
and Francais only; every other locale fell back to English in the WebHTTrack
form. Each file is written in its own declared charset.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 07:59:59 +02:00

2423 lines
51 KiB
HTML
Raw Blame History

<!-- Creator : groff -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta name="generator" content="groff -Thtml, see www.gnu.org">
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<meta name="Content-Style" content="text/css">
<style type="text/css">
p { margin-top: 0; margin-bottom: 0; vertical-align: top }
pre { margin-top: 0; margin-bottom: 0; vertical-align: top }
table { margin-top: 0; margin-bottom: 0; vertical-align: top }
h1 { text-align: center }
</style>
<title>httrack</title>
</head>
<body>
<h1 align="center">httrack</h1>
<a href="#NAME">NAME</a><br>
<a href="#SYNOPSIS">SYNOPSIS</a><br>
<a href="#DESCRIPTION">DESCRIPTION</a><br>
<a href="#EXAMPLES">EXAMPLES</a><br>
<a href="#OPTIONS">OPTIONS</a><br>
<a href="#General options:">General options:</a><br>
<a href="#Action options:">Action options:</a><br>
<a href="#Proxy options:">Proxy options:</a><br>
<a href="#Limits options:">Limits options:</a><br>
<a href="#Flow control:">Flow control:</a><br>
<a href="#Links options:">Links options:</a><br>
<a href="#Build options:">Build options:</a><br>
<a href="#Spider options:">Spider options:</a><br>
<a href="#Browser ID:">Browser ID:</a><br>
<a href="#Log, index, cache">Log, index, cache</a><br>
<a href="#Expert options:">Expert options:</a><br>
<a href="#Guru options: (do NOT use if possible)">Guru options: (do NOT use if possible)</a><br>
<a href="#Dangerous options: (do NOT use unless you exactly know what you are doing)">Dangerous options: (do NOT use unless you exactly know what you are doing)</a><br>
<a href="#Command-line specific options:">Command-line specific options:</a><br>
<a href="#Details: Option N">Details: Option N</a><br>
<a href="#Details: User-defined option N">Details: User-defined option N</a><br>
<a href="#Details: User-defined option N and advanced variable extraction">Details: User-defined option N and advanced variable extraction</a><br>
<a href="#Details: Option K">Details: Option K</a><br>
<a href="#Shortcuts:">Shortcuts:</a><br>
<a href="#Details: Option %W: External callbacks prototypes">Details: Option %W: External callbacks prototypes</a><br>
<a href="#see htsdefines.h">see htsdefines.h</a><br>
<a href="#FILES">FILES</a><br>
<a href="#ENVIRONMENT">ENVIRONMENT</a><br>
<a href="#DIAGNOSTICS">DIAGNOSTICS</a><br>
<a href="#LIMITS">LIMITS</a><br>
<a href="#BUGS">BUGS</a><br>
<a href="#COPYRIGHT">COPYRIGHT</a><br>
<a href="#AVAILABILITY">AVAILABILITY</a><br>
<a href="#AUTHOR">AUTHOR</a><br>
<a href="#SEE ALSO">SEE ALSO</a><br>
<hr>
<h2>NAME
<a name="NAME"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">httrack -
offline browser : copy websites to a local directory</p>
<h2>SYNOPSIS
<a name="SYNOPSIS"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em"><b>httrack [ url
]... [ -filter ]... [ +filter ]...</b> [ <b>-O, --path</b> ]
[ <b>-w, --mirror</b> ] [ <b>-W, --mirror-wizard</b> ] [
<b>-g, --get-files</b> ] [ <b>-i, --continue</b> ] [ <b>-Y,
--mirrorlinks</b> ] [ <b>-P, --proxy</b> ] [ <b>-%f,
--httpproxy-ftp[=N]</b> ] [ <b>-%b, --bind</b> ] [ <b>-rN,
--depth[=N]</b> ] [ <b>-%eN, --ext-depth[=N]</b> ] [ <b>-mN,
--max-files[=N]</b> ] [ <b>-MN, --max-size[=N]</b> ] [
<b>-EN, --max-time[=N]</b> ] [ <b>-AN, --max-rate[=N]</b> ]
[ <b>-%cN, --connection-per-second[=N]</b> ] [ <b>-%G,
--pause</b> ] [ <b>-GN, --max-pause[=N]</b> ] [ <b>-cN,
--sockets[=N]</b> ] [ <b>-TN, --timeout[=N]</b> ] [ <b>-RN,
--retries[=N]</b> ] [ <b>-JN, --min-rate[=N]</b> ] [ <b>-HN,
--host-control[=N]</b> ] [ <b>-%P,
--extended-parsing[=N]</b> ] [ <b>-n, --near</b> ] [ <b>-t,
--test</b> ] [ <b>-%L, --list</b> ] [ <b>-%S, --urllist</b>
] [ <b>-%m, --sitemap</b> ] [ <b>-NN, --structure[=N]</b> ]
[ <b>-%N, --delayed-type-check</b> ] [ <b>-%D,
--cached-delayed-type-check</b> ] [ <b>-%M, --mime-html</b>
] [ <b>-%Z, --single-file</b> ] [ <b>-LN,
--long-names[=N]</b> ] [ <b>-KN, --keep-links[=N]</b> ] [
<b>-x, --replace-external</b> ] [ <b>-%x,
--disable-passwords</b> ] [ <b>-%q,
--include-query-string</b> ] [ <b>-%g, --strip-query</b> ] [
<b>-o, --generate-errors</b> ] [ <b>-X, --purge-old[=N]</b>
] [ <b>-%p, --preserve</b> ] [ <b>-%T, --utf8-conversion</b>
] [ <b>-bN, --cookies[=N]</b> ] [ <b>-%K, --cookies-file</b>
] [ <b>-%Y, --why</b> ] [ <b>-u, --check-type[=N]</b> ] [
<b>-j, --parse-java[=N]</b> ] [ <b>-sN, --robots[=N]</b> ] [
<b>-%h, --http-10</b> ] [ <b>-%k, --keep-alive</b> ] [
<b>-%z, --disable-compression</b> ] [ <b>-%B, --tolerant</b>
] [ <b>-%s, --updatehack</b> ] [ <b>-%u, --urlhack</b> ] [
<b>-%A, --assume</b> ] [ <b>-@iN, --protocol[=N]</b> ] [
<b>-%w, --disable-module</b> ] [ <b>-F, --user-agent</b> ] [
<b>-%R, --referer</b> ] [ <b>-%E, --from</b> ] [ <b>-%F,
--footer</b> ] [ <b>-%l, --language</b> ] [ <b>-%a,
--accept</b> ] [ <b>-%X, --headers</b> ] [ <b>-C,
--cache[=N]</b> ] [ <b>-k, --store-all-in-cache</b> ] [
<b>-%r, --warc</b> ] [ <b>-%d, --changes</b> ] [ <b>-%n,
--do-not-recatch</b> ] [ <b>-%v, --display</b> ] [ <b>-Q,
--do-not-log</b> ] [ <b>-q, --quiet</b> ] [ <b>-z,
--extra-log</b> ] [ <b>-Z, --debug-log</b> ] [ <b>-v,
--verbose</b> ] [ <b>-f, --file-log</b> ] [ <b>-f2,
--single-log</b> ] [ <b>-I, --index</b> ] [ <b>-%i,
--build-top-index</b> ] [ <b>-%I, --search-index</b> ] [
<b>-pN, --priority[=N]</b> ] [ <b>-S, --stay-on-same-dir</b>
] [ <b>-D, --can-go-down</b> ] [ <b>-U, --can-go-up</b> ] [
<b>-B, --can-go-up-and-down</b> ] [ <b>-a,
--stay-on-same-address</b> ] [ <b>-d,
--stay-on-same-domain</b> ] [ <b>-l, --stay-on-same-tld</b>
] [ <b>-e, --go-everywhere</b> ] [ <b>-%H,
--debug-headers</b> ] [ <b>-%!,
--disable-security-limits</b> ] [ <b>-V, --userdef-cmd</b> ]
[ <b>-%W, --callback</b> ] [ <b>-y,
--background-on-suspend</b> ] [ <b>-K, --keep-links[=N]</b>
]</p>
<h2>DESCRIPTION
<a name="DESCRIPTION"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em"><b>httrack</b>
allows you to download a World Wide Web site from the
Internet to a local directory, building recursively all
directories, getting HTML, images, and other files from the
server to your computer. HTTrack arranges the original
site&rsquo;s relative link-structure. Simply open a page of
the &quot;mirrored&quot; website in your browser, and you
can browse the site from link to link, as if you were
viewing it online. HTTrack can also update an existing
mirrored site, and resume interrupted downloads.</p>
<h2>EXAMPLES
<a name="EXAMPLES"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em"><b>httrack
www.example.com/bob/</b></p>
<p style="margin-left:18%;">mirror site
www.example.com/bob/ and only this site</p>
<p style="margin-left:9%;"><b>httrack www.example.com/bob/
www.anothertest.com/mike/ +*.com/*.jpg <br>
-mime:application/*</b></p>
<p style="margin-left:18%;">mirror the two sites together
(with shared links) and accept any .jpg files on .com
sites</p>
<p style="margin-left:9%;"><b>httrack
www.example.com/bob/bobby.html +* -r6 <br>
httrack www.example.com/bob/bobby.html --spider -P <br>
proxy.myhost.com:8080 <br>
httrack --update <br>
httrack <br>
httrack --continue</b></p>
<h2>OPTIONS
<a name="OPTIONS"></a>
</h2>
<h3>General options:
<a name="General options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="3%">
<p style="margin-top: 1em">-O</p></td>
<td width="6%"></td>
<td width="82%">
<p style="margin-top: 1em">path for mirror/logfiles+cache
(-O path_mirror[,path_cache_and_logfiles]) (--path
&lt;param&gt;)</p> </td></tr>
</table>
<h3>Action options:
<a name="Action options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="3%">
<p style="margin-top: 1em">-w</p></td>
<td width="6%"></td>
<td width="82%">
<p style="margin-top: 1em">*mirror web sites (--mirror)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="3%">
<p>-W</p></td>
<td width="6%"></td>
<td width="82%">
<p>mirror web sites, semi-automatic (asks questions)
(--mirror-wizard)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="3%">
<p>-g</p></td>
<td width="6%"></td>
<td width="82%">
<p>just get files (saved in the current directory)
(--get-files)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="3%">
<p>-i</p></td>
<td width="6%"></td>
<td width="82%">
<p>continue an interrupted mirror using the cache
(--continue)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="3%">
<p>-Y</p></td>
<td width="6%"></td>
<td width="82%">
<p>mirror ALL links located in the first level pages
(mirror links) (--mirrorlinks)</p></td></tr>
</table>
<h3>Proxy options:
<a name="Proxy options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-P</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">proxy use (-P
[socks5://|connect://][user:pass@]proxy:port) (--proxy
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%f</p></td>
<td width="5%"></td>
<td width="82%">
<p>*use proxy for ftp (f0 don&rsquo;t use)
(--httpproxy-ftp[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%b</p></td>
<td width="5%"></td>
<td width="82%">
<p>use this local hostname to make/send requests (-%b
hostname) (--bind &lt;param&gt;)</p></td></tr>
</table>
<h3>Limits options:
<a name="Limits options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p style="margin-top: 1em">-rN</p></td>
<td width="1%"></td>
<td width="82%">
<p style="margin-top: 1em">set the mirror depth to N (*
r9999) (--depth[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-%eN</p></td>
<td width="1%"></td>
<td width="82%">
<p>set the external links depth to N (* %e0)
(--ext-depth[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-mN</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum file length for a non-html file
(--max-files[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-mN,N2</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum file length for non html (N) and html (N2)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-MN</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum overall size that can be uploaded/scanned
(--max-size[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-EN</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum mirror time in seconds (60=1 minute, 3600=1
hour) (--max-time[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-AN</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum transfer rate in bytes/seconds (1000=1KB/s max)
(--max-rate[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-%cN</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum number of connections/seconds (*%c5)
(--connection-per-second[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-%G</p></td>
<td width="1%"></td>
<td width="82%">
<p>random pause of MIN[:MAX] seconds between files (e.g.
%G5:10) (--pause &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-GN</p></td>
<td width="1%"></td>
<td width="82%">
<p>pause transfer if N bytes reached, and wait until lock
file is deleted (--max-pause[=N])</p></td></tr>
</table>
<h3>Flow control:
<a name="Flow control:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-cN</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">number of multiple connections
(*c4) (--sockets[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-TN</p></td>
<td width="5%"></td>
<td width="82%">
<p>timeout, number of seconds after a non-responding link
is shutdown; also bounds host name resolution
(--timeout[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-RN</p></td>
<td width="5%"></td>
<td width="82%">
<p>number of retries, in case of timeout or non-fatal
errors (*R1) (--retries[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-JN</p></td>
<td width="5%"></td>
<td width="82%">
<p>traffic jam control, minimum transfert rate
(bytes/seconds) tolerated for a link (--min-rate[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-HN</p></td>
<td width="5%"></td>
<td width="82%">
<p>host is abandoned if: 0=never, 1=timeout, 2=slow,
3=timeout or slow (--host-control[=N])</p></td></tr>
</table>
<h3>Links options:
<a name="Links options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-%P</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">*extended parsing, attempt to
parse all links, even in unknown tags or Javascript (%P0
don&rsquo;t use) (--extended-parsing[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-n</p></td>
<td width="5%"></td>
<td width="82%">
<p>get non-html files &rsquo;near&rsquo; an html file (ex:
an image located outside) (--near)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-t</p></td>
<td width="5%"></td>
<td width="82%">
<p>test all URLs (even forbidden ones) (--test)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%L</p></td>
<td width="5%"></td>
<td width="82%">
<p>&lt;file&gt; add all URL located in this text file (one
URL per line) (--list &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%S</p></td>
<td width="5%"></td>
<td width="82%">
<p>&lt;file&gt; add all scan rules located in this text
file (one scan rule per line) (--urllist &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%m</p></td>
<td width="5%"></td>
<td width="82%">
<p>seed the crawl from the site&rsquo;s sitemap (robots.txt
Sitemap:, then /sitemap.xml); --sitemap-url URL names one
explicitly. A sitemap you name, or one the site declares, is
fetched even under robots.txt Disallow; only the guessed
/sitemap.xml obeys it. The URLs found still pass every
filter and scope rule (--sitemap)</p></td></tr>
</table>
<h3>Build options:
<a name="Build options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-NN</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">structure type (0 *original
structure, 1+: see below) (--structure[=N])</p></td></tr>
</table>
<p style="margin-left:18%;">or user defined structure (-N
&quot;%h%p/%n%q.%t&quot;)</p>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-%N</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">delayed type check, don&rsquo;t
make any link test but wait for files download to start
instead (experimental) (%N0 don&rsquo;t use, %N1 use for
unknown extensions, * %N2 always use)
(--delayed-type-check)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%D</p></td>
<td width="5%"></td>
<td width="82%">
<p>cached delayed type check, don&rsquo;t wait for remote
type during updates, to speedup them (%D0 wait, * %D1
don&rsquo;t wait) (--cached-delayed-type-check)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%M</p></td>
<td width="5%"></td>
<td width="82%">
<p>generate a RFC MIME-encapsulated full-archive (.mht)
(--mime-html)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%Z</p></td>
<td width="5%"></td>
<td width="82%">
<p>after the mirror, rewrite each saved page with its
stylesheets, scripts, images and fonts inlined as data:
URIs, so any page opens by double-click anywhere (links
between pages stay relative; audio and video stay links);
--single-file-max-size N caps each asset (default 10485760
bytes). %M is the better container where a Chromium-family
browser is a given: one archive, no base64 tax on text, a
shared asset stored once (--single-file)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%t</p></td>
<td width="5%"></td>
<td width="82%">
<p>keep the original file extension, don&rsquo;t rewrite it
from the MIME type (%t0 rewrite)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-LN</p></td>
<td width="5%"></td>
<td width="82%">
<p>long names (L1 *long names / L0 8-3 conversion / L2
ISO9660 compatible) (--long-names[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-KN</p></td>
<td width="5%"></td>
<td width="82%">
<p>keep original links (e.g. http://www.adr/link) (K0
*relative link, K absolute links, K4 original links, K3
absolute URI links, K5 transparent proxy link)
(--keep-links[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-x</p></td>
<td width="5%"></td>
<td width="82%">
<p>replace external html links by error pages
(--replace-external)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%x</p></td>
<td width="5%"></td>
<td width="82%">
<p>do not include any password for external password
protected websites (%x0 include) (--disable-passwords)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%q</p></td>
<td width="5%"></td>
<td width="82%">
<p>*include query string for local files (useless, for
information purpose only) (%q0 don&rsquo;t include)
(--include-query-string)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%g</p></td>
<td width="5%"></td>
<td width="82%">
<p>strip query keys for dedup
([host/pattern=]key1,key2,...) (--strip-query
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-o</p></td>
<td width="5%"></td>
<td width="82%">
<p>*generate output html file in case of error (404..) (o0
don&rsquo;t generate) (--generate-errors)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-X</p></td>
<td width="5%"></td>
<td width="82%">
<p>*purge old files after update (X0 keep delete)
(--purge-old[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%p</p></td>
<td width="5%"></td>
<td width="82%">
<p>preserve html files &rsquo;as is&rsquo; (identical to
&rsquo;-K4 -%F &quot;&quot;&rsquo;) (--preserve)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%T</p></td>
<td width="5%"></td>
<td width="82%">
<p>links conversion to UTF-8 (--utf8-conversion)</p></td></tr>
</table>
<h3>Spider options:
<a name="Spider options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-bN</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">accept cookies in cookies.txt
(0=do not accept,* 1=accept) (--cookies[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%K</p></td>
<td width="5%"></td>
<td width="82%">
<p>load extra cookies from a Netscape cookies.txt
(--cookies-file &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%Y</p></td>
<td width="5%"></td>
<td width="82%">
<p>explain which filter rule accepts or rejects a URL, then
exit (--why &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-u</p></td>
<td width="5%"></td>
<td width="82%">
<p>check document type if unknown (cgi,asp..) (u0
don&rsquo;t check, * u1 check but /, u2 check always)
(--check-type[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-j</p></td>
<td width="5%"></td>
<td width="82%">
<p>*parse scripts (j0 don&rsquo;t parse, bitmask: |1 parse
default, |4 don&rsquo;t parse .js |8 don&rsquo;t be
aggressive) (--parse-java[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-sN</p></td>
<td width="5%"></td>
<td width="82%">
<p>follow robots.txt and meta robots tags
(0=never,1=sometimes,* 2=always, 3=always (even strict
rules)) (--robots[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%h</p></td>
<td width="5%"></td>
<td width="82%">
<p>force HTTP/1.0 requests (reduce update features, only
for old servers or proxies) (--http-10)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%k</p></td>
<td width="5%"></td>
<td width="82%">
<p>use keep-alive if possible, greately reducing latency
for small files and test requests (%k0 don&rsquo;t use)
(--keep-alive)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%z</p></td>
<td width="5%"></td>
<td width="82%">
<p>do not request compressed content (%z0 request)
(--disable-compression)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%B</p></td>
<td width="5%"></td>
<td width="82%">
<p>tolerant requests (accept bogus responses on some
servers, but not standard!) (--tolerant)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%s</p></td>
<td width="5%"></td>
<td width="82%">
<p>update hacks: various hacks to limit re-transfers when
updating (identical size, bogus response..)
(--updatehack)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%u</p></td>
<td width="5%"></td>
<td width="82%">
<p>url hacks: various hacks to limit duplicate URLs (strip
//, www.foo.com==foo.com..) (--urlhack)</p></td></tr>
</table>
<p style="margin-left:18%;">opt out of one url-hack part:
--keep-www-prefix (www.foo.com&lt;&gt;foo.com),
--keep-double-slashes (//), --keep-query-order
(?b&amp;a)</p>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-%A</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">assume that a type (cgi,asp..)
is always linked with a mime type (-%A
php3,cgi=text/html;dat,bin=application/x-zip) (--assume
&lt;param&gt;)</p> </td></tr>
</table>
<p style="margin-left:18%;">shortcut: &rsquo;--assume
standard&rsquo; is equivalent to -%A php2 php3 php4 php cgi
asp jsp pl cfm nsf=text/html <br>
can also be used to force a specific file type: --assume
foo.cgi=text/html</p>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="5%">
<p style="margin-top: 1em">-@iN</p></td>
<td width="4%"></td>
<td width="82%">
<p style="margin-top: 1em">internet protocol (0=both
ipv6+ipv4, 4=ipv4 only, 6=ipv6 only) (--protocol[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="5%">
<p>-%w</p></td>
<td width="4%"></td>
<td width="82%">
<p>disable a specific external mime module (-%w
httrack-plugin) (--disable-module &lt;param&gt;)</p></td></tr>
</table>
<h3>Browser ID:
<a name="Browser ID:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-F</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">user-agent field sent in HTTP
headers (-F &quot;user-agent name&quot;) (--user-agent
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%R</p></td>
<td width="5%"></td>
<td width="82%">
<p>default referer field sent in HTTP headers (--referer
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%E</p></td>
<td width="5%"></td>
<td width="82%">
<p>from email address sent in HTTP headers (--from
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%F</p></td>
<td width="5%"></td>
<td width="82%">
<p>footer string in Html code (-%F &quot;Mirrored from
{url} on {date}&quot;; fields {addr} {path} {url} {date}
{lastmodified} {version} {mime} {charset} {status} {size},
or legacy %s) (--footer &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%l</p></td>
<td width="5%"></td>
<td width="82%">
<p>preferred language (-%l &quot;fr, en, jp, *&quot;
(--language &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%a</p></td>
<td width="5%"></td>
<td width="82%">
<p>accepted formats (-%a
&quot;text/html,image/png;q=0.9,*/*;q=0.1&quot; (--accept
&lt;param&gt;)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%X</p></td>
<td width="5%"></td>
<td width="82%">
<p>additional HTTP header line (-%X &quot;X-Magic: 42&quot;
(--headers &lt;param&gt;)</p></td></tr>
</table>
<h3>Log, index, cache
<a name="Log, index, cache"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-C</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">create/use a cache for updates
and retries (C0 no cache,C1 cache is prioritary,* C2 test
update before) (--cache[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-k</p></td>
<td width="5%"></td>
<td width="82%">
<p>store all files in cache (not useful if files on disk)
(--store-all-in-cache)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%r</p></td>
<td width="5%"></td>
<td width="82%">
<p>write an ISO-28500 WARC/1.1 archive; --warc-file NAME
sets the output name, --warc-max-size N rotates segments
past N bytes, --warc-cdx also writes a sorted CDXJ index,
--wacz packages it all as a WACZ file (--warc)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%d</p></td>
<td width="5%"></td>
<td width="82%">
<p>write hts-changes.json listing what this crawl left new,
changed, unchanged and gone compared to the previous mirror
(--changes)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%n</p></td>
<td width="5%"></td>
<td width="82%">
<p>do not re-download locally erased files
(--do-not-recatch)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%v</p></td>
<td width="5%"></td>
<td width="82%">
<p>display on screen filenames downloaded (in realtime) - *
%v1 short version - %v2 full animation (--display)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-Q</p></td>
<td width="5%"></td>
<td width="82%">
<p>no log - quiet mode (--do-not-log)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-q</p></td>
<td width="5%"></td>
<td width="82%">
<p>no questions - quiet mode (--quiet)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-z</p></td>
<td width="5%"></td>
<td width="82%">
<p>log - extra infos (--extra-log)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-Z</p></td>
<td width="5%"></td>
<td width="82%">
<p>log - debug (--debug-log)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-v</p></td>
<td width="5%"></td>
<td width="82%">
<p>log on screen (--verbose)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-f</p></td>
<td width="5%"></td>
<td width="82%">
<p>*log in files (--file-log)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-f2</p></td>
<td width="5%"></td>
<td width="82%">
<p>one single log file (--single-log)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-I</p></td>
<td width="5%"></td>
<td width="82%">
<p>*make an index (I0 don&rsquo;t make) (--index)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%i</p></td>
<td width="5%"></td>
<td width="82%">
<p>make a top index for a project folder (* %i0 don&rsquo;t
make) (--build-top-index)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%I</p></td>
<td width="5%"></td>
<td width="82%">
<p>make an searchable index for this mirror (* %I0
don&rsquo;t make) (--search-index)</p></td></tr>
</table>
<h3>Expert options:
<a name="Expert options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-pN</p></td>
<td width="5%"></td>
<td width="49%">
<p style="margin-top: 1em">priority mode: (* p3)
(--priority[=N])</p> </td>
<td width="33%">
</td></tr>
</table>
<p style="margin-left:18%;">p0 just scan, don&rsquo;t save
anything (for checking links) <br>
p1 save only html files <br>
p2 save only non html files <br>
*p3 save all files <br>
p7 get html files before, then treat other files</p>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-S</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">stay on the same directory
(--stay-on-same-dir)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-D</p></td>
<td width="5%"></td>
<td width="82%">
<p>*can only go down into subdirs (--can-go-down)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-U</p></td>
<td width="5%"></td>
<td width="82%">
<p>can only go to upper directories (--can-go-up)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-B</p></td>
<td width="5%"></td>
<td width="82%">
<p>can both go up&amp;down into the directory structure
(--can-go-up-and-down)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-a</p></td>
<td width="5%"></td>
<td width="82%">
<p>*stay on the same address (--stay-on-same-address)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-d</p></td>
<td width="5%"></td>
<td width="82%">
<p>stay on the same principal domain
(--stay-on-same-domain)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-l</p></td>
<td width="5%"></td>
<td width="82%">
<p>stay on the same TLD (eg: .com) (--stay-on-same-tld)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-e</p></td>
<td width="5%"></td>
<td width="82%">
<p>go everywhere on the web (--go-everywhere)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%H</p></td>
<td width="5%"></td>
<td width="82%">
<p>debug HTTP headers in logfile (--debug-headers)</p></td></tr>
</table>
<h3>Guru options: (do NOT use if possible)
<a name="Guru options: (do NOT use if possible)"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p style="margin-top: 1em">-#test</p></td>
<td width="1%"></td>
<td width="82%">
<p style="margin-top: 1em">list engine self-tests (run one
with -#test=NAME [args])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#C</p></td>
<td width="1%"></td>
<td width="82%">
<p>cache list (-#C &rsquo;*.com/spider*.gif&rsquo;
(--debug-cache &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#R</p></td>
<td width="1%"></td>
<td width="82%">
<p>cache repair (damaged cache) (--repair-cache)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#d</p></td>
<td width="1%"></td>
<td width="82%">
<p>debug parser (--debug-parsing)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#E</p></td>
<td width="1%"></td>
<td width="82%">
<p>extract new.zip cache meta-data in meta.zip</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#f</p></td>
<td width="1%"></td>
<td width="82%">
<p>always flush log files (--advanced-flushlogs)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#FN</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum number of filters
(--advanced-maxfilters[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#h</p></td>
<td width="1%"></td>
<td width="82%">
<p>version info (--version)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#K</p></td>
<td width="1%"></td>
<td width="82%">
<p>scan stdin (debug) (--debug-scanstdin)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#L</p></td>
<td width="1%"></td>
<td width="82%">
<p>maximum number of links (-#L1000000)
(--advanced-maxlinks[=N])</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#p</p></td>
<td width="1%"></td>
<td width="82%">
<p>display ugly progress information
(--advanced-progressinfo)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#P</p></td>
<td width="1%"></td>
<td width="82%">
<p>catch URL (--catch-url)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#T</p></td>
<td width="1%"></td>
<td width="82%">
<p>generate transfer ops. log every minutes
(--debug-xfrstats)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#u</p></td>
<td width="1%"></td>
<td width="82%">
<p>wait time (--advanced-wait)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-#Z</p></td>
<td width="1%"></td>
<td width="82%">
<p>generate transfer rate statistics every minutes
(--debug-ratestats)</p> </td></tr>
</table>
<h3>Dangerous options: (do NOT use unless you exactly know what you are doing)
<a name="Dangerous options: (do NOT use unless you exactly know what you are doing)"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-%!</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">bypass built-in security limits
aimed to avoid bandwidth abuses (bandwidth, simultaneous
connections) (--disable-security-limits)</p></td></tr>
</table>
<p style="margin-left:18%;">IMPORTANT NOTE: DANGEROUS
OPTION, ONLY SUITABLE FOR EXPERTS <br>
USE IT WITH EXTREME CARE</p>
<h3>Command-line specific options:
<a name="Command-line specific options:"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-V</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">execute system command after
each files ($0 is the filename: -V &quot;rm \$0&quot;)
(--userdef-cmd &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%W</p></td>
<td width="5%"></td>
<td width="82%">
<p>use an external library function as a wrapper (-%W
myfoo.so[,myparameters]) (--callback &lt;param&gt;)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-y</p></td>
<td width="5%"></td>
<td width="82%">
<p>go to background when suspended (y0 don&rsquo;t)
(--background-on-suspend)</p> </td></tr>
</table>
<h3>Details: Option N
<a name="Details: Option N"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p style="margin-top: 1em">-N0</p></td>
<td width="1%"></td>
<td width="82%">
<p style="margin-top: 1em">Site-structure (default)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1</p></td>
<td width="1%"></td>
<td width="82%">
<p>HTML in web/, images/other files in web/images/</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N2</p></td>
<td width="1%"></td>
<td width="82%">
<p>HTML in web/HTML, images/other in web/images</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N3</p></td>
<td width="1%"></td>
<td width="82%">
<p>HTML in web/, images/other in web/</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N4</p></td>
<td width="1%"></td>
<td width="82%">
<p>HTML in web/, images/other in web/xxx, where xxx is the
file extension (all gif will be placed onto web/gif, for
example)</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N5</p></td>
<td width="1%"></td>
<td width="82%">
<p>Images/other in web/xxx and HTML in web/HTML</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N99</p></td>
<td width="1%"></td>
<td width="82%">
<p>All files in web/, with random names (gadget !)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N100</p></td>
<td width="1%"></td>
<td width="82%">
<p>Site-structure, without www.domain.xxx/</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N101</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N1 except that &quot;web&quot; is replaced
by the site&rsquo;s name</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N102</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N2 except that &quot;web&quot; is replaced
by the site&rsquo;s name</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N103</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N3 except that &quot;web&quot; is replaced
by the site&rsquo;s name</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N104</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N4 except that &quot;web&quot; is replaced
by the site&rsquo;s name</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N105</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N5 except that &quot;web&quot; is replaced
by the site&rsquo;s name</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N199</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N99 except that &quot;web&quot; is replaced
by the site&rsquo;s name</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1001</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N1 except that there is no &quot;web&quot;
directory</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1002</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N2 except that there is no &quot;web&quot;
directory</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1003</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N3 except that there is no &quot;web&quot;
directory (option set for g option)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1004</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N4 except that there is no &quot;web&quot;
directory</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1005</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N5 except that there is no &quot;web&quot;
directory</p> </td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>-N1099</p></td>
<td width="1%"></td>
<td width="82%">
<p>Identical to N99 except that there is no &quot;web&quot;
directory</p> </td></tr>
</table>
<h3>Details: User-defined option N
<a name="Details: User-defined option N"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-%n</p></td>
<td width="5%"></td>
<td width="63%">
<p style="margin-top: 1em">Name of file without file type
(ex: image)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%N</p></td>
<td width="5%"></td>
<td width="63%">
<p>Name of file, including file type (ex: image.gif)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%t</p></td>
<td width="5%"></td>
<td width="63%">
<p>File type (ex: gif)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%p</p></td>
<td width="5%"></td>
<td width="63%">
<p>Path [without ending /] (ex: /someimages)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%h</p></td>
<td width="5%"></td>
<td width="63%">
<p>Host name (ex: www.example.com)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%M</p></td>
<td width="5%"></td>
<td width="63%">
<p>URL MD5 (128 bits, 32 ascii bytes)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%Q</p></td>
<td width="5%"></td>
<td width="63%">
<p>query string MD5 (128 bits, 32 ascii bytes)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%k</p></td>
<td width="5%"></td>
<td width="63%">
<p>full query string</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%r</p></td>
<td width="5%"></td>
<td width="63%">
<p>protocol name (ex: http)</p></td>
<td width="19%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-%q</p></td>
<td width="5%"></td>
<td width="63%">
<p>small query string MD5 (16 bits, 4 ascii bytes)</p></td>
<td width="19%">
</td></tr>
</table>
<p style="margin-left:18%;">&rsquo;%s?&rsquo; Short name
version (ex: %sN)</p>
<p style="margin-left:9%;">-%[param]</p>
<p style="margin-left:18%;">param variable in query
string</p>
<p style="margin-left:9%;">-%[param:before:after:empty:notfound]</p>
<p style="margin-left:18%;">advanced variable
extraction</p>
<h3>Details: User-defined option N and advanced variable extraction
<a name="Details: User-defined option N and advanced variable extraction"></a>
</h3>
<p style="margin-left:9%; margin-top: 1em">%[param:before:after:empty:notfound]
<br>
param : parameter name <br>
before : string to prepend if the parameter was found <br>
after : string to append if the parameter was found <br>
notfound : string replacement if the parameter could not be
found <br>
empty : string replacement if the parameter was empty <br>
all fields, except the first one (the parameter name), can
be empty</p>
<h3>Details: Option K
<a name="Details: Option K"></a>
</h3>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p style="margin-top: 1em">-K0</p></td>
<td width="5%"></td>
<td width="82%">
<p style="margin-top: 1em">foo.cgi?q=45 -&gt;
foo4B54.html?q=45 (relative URI, default)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-K</p></td>
<td width="5%"></td>
<td width="82%">
<p>-&gt; http://www.foobar.com/folder/foo.cgi?q=45
(absolute URL) (--keep-links[=N])</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-K3</p></td>
<td width="5%"></td>
<td width="82%">
<p>-&gt; /folder/foo.cgi?q=45 (absolute URI)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-K4</p></td>
<td width="5%"></td>
<td width="82%">
<p>-&gt; foo.cgi?q=45 (original URL)</p></td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="4%">
<p>-K5</p></td>
<td width="5%"></td>
<td width="82%">
<p>-&gt; http://www.foobar.com/folder/foo4B54.html?q=45
(transparent proxy URL)</p></td></tr>
</table>
<h3>Shortcuts:
<a name="Shortcuts:"></a>
</h3>
<p style="margin-left:9%; margin-top: 1em">--mirror</p>
<p style="margin-left:18%;">&lt;URLs&gt; *make a mirror of
site(s) (default)</p>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>--get</p></td>
<td width="1%"></td>
<td width="78%">
<p>&lt;URLs&gt; get the files indicated, do not seek other
URLs (-qg)</p></td>
<td width="4%">
</td></tr>
<tr valign="top" align="left">
<td width="9%"></td>
<td width="8%">
<p>--list</p></td>
<td width="1%"></td>
<td width="78%">
<p>&lt;text file&gt; add all URL located in this text file
(-%L)</p> </td>
<td width="4%">
</td></tr>
</table>
<p style="margin-left:9%;">--mirrorlinks</p>
<p style="margin-left:18%;">&lt;URLs&gt; mirror all links
in 1st level pages (-Y)</p>
<p style="margin-left:9%;">--testlinks</p>
<p style="margin-left:18%;">&lt;URLs&gt; test links in
pages (-r1p0C0I0t)</p>
<p style="margin-left:9%;">--spider</p>
<p style="margin-left:18%;">&lt;URLs&gt; spider site(s), to
test links: reports Errors &amp; Warnings (-p0C0I0t)</p>
<p style="margin-left:9%;">--testsite</p>
<p style="margin-left:18%;">&lt;URLs&gt; identical to
--spider</p>
<p style="margin-left:9%;">--skeleton</p>
<p style="margin-left:18%;">&lt;URLs&gt; make a mirror, but
gets only html files (-p1)</p>
<p style="margin-left:9%;">--update</p>
<p style="margin-left:18%;">update a mirror, without
confirmation (-iC2)</p>
<p style="margin-left:9%;">--continue</p>
<p style="margin-left:18%;">continue a mirror, without
confirmation (-iC1)</p>
<p style="margin-left:9%;">--catchurl</p>
<p style="margin-left:18%;">create a temporary proxy to
capture an URL or a form post URL</p>
<p style="margin-left:9%;">--clean</p>
<p style="margin-left:18%;">erase cache &amp; log files</p>
<p style="margin-left:9%;">--http10</p>
<p style="margin-left:18%;">force http/1.0 requests
(-%h)</p>
<h3>Details: Option %W: External callbacks prototypes
<a name="Details: Option %W: External callbacks prototypes"></a>
</h3>
<h3>see htsdefines.h
<a name="see htsdefines.h"></a>
</h3>
<h2>FILES
<a name="FILES"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em"><i>/etc/httrack.conf</i></p>
<p style="margin-left:18%;">The system wide configuration
file.</p>
<h2>ENVIRONMENT
<a name="ENVIRONMENT"></a>
</h2>
<table width="100%" border="0" rules="none" frame="void"
cellspacing="0" cellpadding="0">
<tr valign="top" align="left">
<td width="9%"></td>
<td width="5%">
<p style="margin-top: 1em">HOME</p></td>
<td width="4%"></td>
<td width="82%">
<p style="margin-top: 1em">Is being used if you defined in
/etc/httrack.conf the line <i>path &#732;/websites/#</i></p></td></tr>
</table>
<h2>DIAGNOSTICS
<a name="DIAGNOSTICS"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">Errors/Warnings
are reported to <i>hts-log.txt</i> by default, or to stderr
if the <i>-v</i> option was specified.</p>
<h2>LIMITS
<a name="LIMITS"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">These are the
principals limits of HTTrack for that moment. Note that we
did not heard about any other utility that would have solved
them. <small>- Several scripts generating complex filenames
may not find them (ex:
img.src=&rsquo;image&rsquo;+a+Mobj.dst+&rsquo;.gif&rsquo;) -
Cgi-bin links may not work properly in some cases
(parameters needed). To avoid them: use filters like
-*cgi-bin*</small></p>
<h2>BUGS
<a name="BUGS"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">Please reports
bugs to <b>&lt;bugs@httrack.com&gt;.</b> Include a complete,
self-contained example that will allow the bug to be
reproduced, and say which version of httrack you are using.
Do not forget to detail options used, OS version, and any
other information you deem necessary.</p>
<h2>COPYRIGHT
<a name="COPYRIGHT"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">Copyright (C)
1998-2026 Xavier Roche and other contributors</p>
<p style="margin-left:9%; margin-top: 1em">This program is
free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as
published by the Free Software Foundation, either version 3
of the License, or (at your option) any later version.</p>
<p style="margin-left:9%; margin-top: 1em">This program is
distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.</p>
<p style="margin-left:9%; margin-top: 1em">You should have
received a copy of the GNU General Public License along with
this program. If not, see
&lt;http://www.gnu.org/licenses/&gt;.</p>
<h2>AVAILABILITY
<a name="AVAILABILITY"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">The most recent
released version of httrack can be found at:
<b>http://www.httrack.com</b></p>
<h2>AUTHOR
<a name="AUTHOR"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">Xavier Roche
&lt;roche@httrack.com&gt;</p>
<h2>SEE ALSO
<a name="SEE ALSO"></a>
</h2>
<p style="margin-left:9%; margin-top: 1em">The <b>HTML</b>
documentation (available online at
<b>http://www.httrack.com/html/</b> ) contains more detailed
information. Please also refer to the <b>httrack FAQ</b>
(available online at
<b>http://www.httrack.com/html/faq.html</b> )</p>
<hr>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink