|
|
|
|
@@ -0,0 +1,496 @@
|
|
|
|
|
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
|
|
|
|
|
|
|
|
|
|
<head>
|
|
|
|
|
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
|
|
|
|
|
<meta name="description" content="HTTrack is an easy-to-use website mirror utility. It allows you to download a World Wide website from the Internet to a local directory,building recursively all structures, getting html, images, and other files from the server to your computer. Links are rebuiltrelatively so that you can freely browse to the local site (works with any browser). You can mirror several sites together so that you can jump from one toanother. You can, also, update an existing mirror site, or resume an interrupted download. The robot is fully configurable, with an integrated help" />
|
|
|
|
|
<meta name="keywords" content="httrack, HTTRACK, HTTrack, winhttrack, WINHTTRACK, WinHTTrack, offline browser, web mirror utility, aspirateur web, surf offline, web capture, www mirror utility, browse offline, local site builder, website mirroring, aspirateur www, internet grabber, capture de site web, internet tool, hors connexion, unix, dos, windows 95, windows 98, solaris, ibm580, AIX 4.0, HTS, HTGet, web aspirator, web aspirateur, libre, GPL, GNU, free software" />
|
|
|
|
|
<title>HTTrack Website Copier - Offline Browser</title>
|
|
|
|
|
|
|
|
|
|
<style type="text/css">
|
|
|
|
|
<!--
|
|
|
|
|
|
|
|
|
|
body {
|
|
|
|
|
margin: 0; padding: 0; margin-bottom: 15px; margin-top: 8px;
|
|
|
|
|
background: #77b;
|
|
|
|
|
}
|
|
|
|
|
body, td {
|
|
|
|
|
font: 14px "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#subTitle {
|
|
|
|
|
background: #000; color: #fff; padding: 4px; font-weight: bold;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#siteNavigation a, #siteNavigation .current {
|
|
|
|
|
font-weight: bold; color: #448;
|
|
|
|
|
}
|
|
|
|
|
#siteNavigation a:link { text-decoration: none; }
|
|
|
|
|
#siteNavigation a:visited { text-decoration: none; }
|
|
|
|
|
|
|
|
|
|
#siteNavigation .current { background-color: #ccd; }
|
|
|
|
|
|
|
|
|
|
#siteNavigation a:hover { text-decoration: none; background-color: #fff; color: #000; }
|
|
|
|
|
#siteNavigation a:active { text-decoration: none; background-color: #ccc; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
a:link { text-decoration: underline; color: #00f; }
|
|
|
|
|
a:visited { text-decoration: underline; color: #000; }
|
|
|
|
|
a:hover { text-decoration: underline; color: #c00; }
|
|
|
|
|
a:active { text-decoration: underline; }
|
|
|
|
|
|
|
|
|
|
#pageContent {
|
|
|
|
|
clear: both;
|
|
|
|
|
border-bottom: 6px solid #000;
|
|
|
|
|
padding: 10px; padding-top: 20px;
|
|
|
|
|
line-height: 1.65em;
|
|
|
|
|
background-image: url(images/bg_rings.gif);
|
|
|
|
|
background-repeat: no-repeat;
|
|
|
|
|
background-position: top right;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#pageContent, #siteNavigation {
|
|
|
|
|
background-color: #ccd;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.imgLeft { float: left; margin-right: 10px; margin-bottom: 10px; }
|
|
|
|
|
.imgRight { float: right; margin-left: 10px; margin-bottom: 10px; }
|
|
|
|
|
|
|
|
|
|
hr { height: 1px; color: #000; background-color: #000; margin-bottom: 15px; }
|
|
|
|
|
|
|
|
|
|
h1 { margin: 0; font-weight: bold; font-size: 2em; }
|
|
|
|
|
h2 { margin: 0; font-weight: bold; font-size: 1.6em; }
|
|
|
|
|
h3 { margin: 0; font-weight: bold; font-size: 1.3em; }
|
|
|
|
|
h4 { margin: 0; font-weight: bold; font-size: 1.18em; }
|
|
|
|
|
|
|
|
|
|
.blak { background-color: #000; }
|
|
|
|
|
.hide { display: none; }
|
|
|
|
|
.tableWidth { min-width: 400px; }
|
|
|
|
|
|
|
|
|
|
.tblRegular { border-collapse: collapse; }
|
|
|
|
|
.tblRegular td { padding: 6px; background-image: url(fade.gif); border: 2px solid #99c; }
|
|
|
|
|
.tblHeaderColor, .tblHeaderColor td { background: #99c; }
|
|
|
|
|
.tblNoBorder td { border: 0; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
// -->
|
|
|
|
|
</style>
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
|
|
|
|
|
|
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
|
|
|
|
|
<tr>
|
|
|
|
|
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
|
|
|
|
|
</tr>
|
|
|
|
|
</table>
|
|
|
|
|
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
|
|
|
|
|
<tr>
|
|
|
|
|
<td id="subTitle">Open Source offline browser</td>
|
|
|
|
|
</tr>
|
|
|
|
|
</table>
|
|
|
|
|
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
|
|
|
|
|
<tr class="blak">
|
|
|
|
|
<td>
|
|
|
|
|
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
|
|
|
|
|
<tr>
|
|
|
|
|
<td colspan="6">
|
|
|
|
|
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
|
|
|
|
|
<tr>
|
|
|
|
|
<td id="pageContent">
|
|
|
|
|
<!-- ==================== End prologue ==================== -->
|
|
|
|
|
|
|
|
|
|
<h2 align="center"><em>Command-Line Guide</em></h2>
|
|
|
|
|
|
|
|
|
|
<p>This is a task-oriented guide to the <tt>httrack</tt> command line: how to do the
|
|
|
|
|
things people actually ask for, and the handful of defaults that surprise
|
|
|
|
|
newcomers. It sits on top of the
|
|
|
|
|
<a href="httrack.man.html">generated manual page</a>, which lists every option in
|
|
|
|
|
full. When you want the exhaustive detail for a flag, that page is the reference;
|
|
|
|
|
this one is the map.</p>
|
|
|
|
|
|
|
|
|
|
<p>Two habits before anything else. First, HTTrack has its own options: they are not
|
|
|
|
|
wget or curl flags, so reach for the tables here rather than guessing. Second,
|
|
|
|
|
when a mirror does something you did not expect, the answer is almost always in
|
|
|
|
|
the log. Every project writes <tt>hts-log.txt</tt> (and <tt>hts-err.txt</tt>) into
|
|
|
|
|
its output directory, and those files name every URL that was refused, redirected,
|
|
|
|
|
or filtered out. Read them first.</p>
|
|
|
|
|
|
|
|
|
|
<h4>On this page</h4>
|
|
|
|
|
<ul class="tblNoBorder">
|
|
|
|
|
<li><a href="#quickstart">1. Quick start</a></li>
|
|
|
|
|
<li><a href="#scope">2. Scope: how far the crawl reaches</a></li>
|
|
|
|
|
<li><a href="#filters">3. Filters and scan rules</a></li>
|
|
|
|
|
<li><a href="#limits">4. Limits and politeness</a></li>
|
|
|
|
|
<li><a href="#names">5. File names and types</a></li>
|
|
|
|
|
<li><a href="#links">6. Links and page building</a></li>
|
|
|
|
|
<li><a href="#identity">7. Identity, cookies and login</a></li>
|
|
|
|
|
<li><a href="#proxy">8. Proxy and network</a></li>
|
|
|
|
|
<li><a href="#update">9. Update and cache</a></li>
|
|
|
|
|
<li><a href="#experts">10. Experts and scripting</a></li>
|
|
|
|
|
<li><a href="#recipes">11. Recipes</a></li>
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="quickstart">1. Quick start</h3>
|
|
|
|
|
|
|
|
|
|
<p>A mirror is one command: a start URL and an output directory.</p>
|
|
|
|
|
|
|
|
|
|
<p><tt>httrack https://example.com/ -O mydir</tt></p>
|
|
|
|
|
|
|
|
|
|
<p>With no other options HTTrack mirrors that site, stays on the same host, follows
|
|
|
|
|
links to any depth, rebuilds them to browse offline, and stores everything under
|
|
|
|
|
<tt>mydir</tt>. The same directory also holds the log files and the
|
|
|
|
|
<tt>hts-cache/</tt> folder that makes a later update or resume possible.</p>
|
|
|
|
|
|
|
|
|
|
<p>Two defaults are worth knowing up front, because both catch people out:</p>
|
|
|
|
|
<ul>
|
|
|
|
|
<li>HTTrack throttles itself to about <b>100 KB/s</b> even when you pass no rate
|
|
|
|
|
option. If a mirror feels slow, that is why. See
|
|
|
|
|
<a href="#limits">Limits</a> for how to lift it.</li>
|
|
|
|
|
<li>The download proceeds as a well-behaved robot: it identifies itself as
|
|
|
|
|
<tt>HTTrack</tt>, obeys <tt>robots.txt</tt>, and sends a Referer with each
|
|
|
|
|
request. A site that blocks that behavior needs the levers in
|
|
|
|
|
<a href="#identity">Identity</a>, not brute force.</li>
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="scope">2. Scope: how far the crawl reaches</h3>
|
|
|
|
|
|
|
|
|
|
<p>Scope decides which links HTTrack is even willing to follow, before any filter
|
|
|
|
|
you write. Get this right and most "it downloaded too much" or "it only grabbed
|
|
|
|
|
the index" problems disappear.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>-rN</tt></td><td>Maximum link depth. <b>The start page is level 1</b>, so one level of links out is <tt>-r2</tt>, not <tt>-r1</tt>.</td></tr>
|
|
|
|
|
<tr><td><tt>-a -d -l -e</tt></td><td>How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.</td></tr>
|
|
|
|
|
<tr><td><tt>-D -U -S -B</tt></td><td>Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.</td></tr>
|
|
|
|
|
<tr><td><tt>-n</tt></td><td>Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.</td></tr>
|
|
|
|
|
<tr><td><tt>-%eN</tt></td><td>How many levels of external links to follow once the crawl leaves your scope (default 0).</td></tr>
|
|
|
|
|
<tr><td><tt>-t</tt></td><td>Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p>The single most common surprise is "only the home page came down." That is
|
|
|
|
|
usually not a scope option at all: it is an off-host redirect. A start URL of
|
|
|
|
|
<tt>http://example.com/</tt> that redirects to <tt>https://www.example.com/</tt>
|
|
|
|
|
lands you on a different host, and same-host scope stops the crawl there. Start
|
|
|
|
|
from the final URL, or add a filter that re-admits the real host (see
|
|
|
|
|
<a href="#filters">Filters</a>). The log will show the redirect.</p>
|
|
|
|
|
|
|
|
|
|
<p><tt>-n</tt> is the fix for pages that render locally without their images or
|
|
|
|
|
stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note
|
|
|
|
|
that its embedded-asset handling (following <tt>img</tt>, <tt>link</tt>,
|
|
|
|
|
<tt>script</tt>, <tt>style</tt> and HTML5 <tt>source</tt>/<tt>track</tt> targets
|
|
|
|
|
past the normal depth and filter limits) applies only when <tt>-n</tt> is on; it
|
|
|
|
|
is not automatic. It can also over-fetch by dragging in a whole external host from
|
|
|
|
|
a single link, in which case name the assets you want with a filter instead.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="filters">3. Filters and scan rules</h3>
|
|
|
|
|
|
|
|
|
|
<p>Filters are the number-one source of confusion, and also the tool that solves
|
|
|
|
|
most scope problems once you understand them. A filter is a rule that accepts
|
|
|
|
|
(<tt>+</tt>) or rejects (<tt>-</tt>) URLs by pattern. The sign is mandatory:
|
|
|
|
|
<tt>+pattern</tt> adds, <tt>-pattern</tt> removes, and a bare pattern is an error.</p>
|
|
|
|
|
|
|
|
|
|
<p>The rules that matter:</p>
|
|
|
|
|
<ul>
|
|
|
|
|
<li><b>Last match wins.</b> Rules are applied in order and the last one that
|
|
|
|
|
matches a URL decides its fate. Order your rules from general to specific.</li>
|
|
|
|
|
<li><b>Wildcards.</b> <tt>*</tt> matches any run of characters;
|
|
|
|
|
<tt>*[a-z]</tt>, <tt>*[0-9]</tt> and similar classes match sets. So
|
|
|
|
|
<tt>+*.pdf</tt> means "any URL ending in .pdf".</li>
|
|
|
|
|
<li><b>Whitelisting.</b> To keep one site and nothing else, deny everything then
|
|
|
|
|
re-admit the host: <tt>"-*" "+example.com/*"</tt>. A lone <tt>+</tt> rule only
|
|
|
|
|
adds to the default scope; it never restricts.</li>
|
|
|
|
|
<li><b>Size rules.</b> <tt>*[>100000]</tt> and <tt>*[<1000]</tt> filter by
|
|
|
|
|
byte size. Because size is only known once the transfer starts, an oversize file
|
|
|
|
|
is fetched partway and then aborted, not skipped for free.</li>
|
|
|
|
|
<li><b>mime: rules.</b> A rule like <tt>-mime:video/*</tt> matches the
|
|
|
|
|
<tt>Content-Type</tt>. That type is only known after the response headers arrive,
|
|
|
|
|
so a mime rule <b>cannot stop a request</b>; it can only abort the body. Use a
|
|
|
|
|
URL pattern when you want to avoid the fetch entirely.</li>
|
|
|
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
<p>Quote your filters. Shells treat <tt>*</tt>, <tt>[</tt> and sometimes <tt>+</tt>
|
|
|
|
|
specially, so wrap each rule in quotes as shown above. The full pattern language,
|
|
|
|
|
with tables for wildcards, size and mime, is in
|
|
|
|
|
<a href="filters.html">the filters page</a>, and the
|
|
|
|
|
<a href="faq.html">FAQ</a> has a worked tutorial.</p>
|
|
|
|
|
|
|
|
|
|
<p><b>robots.txt.</b> By default HTTrack obeys <tt>robots.txt</tt> (<tt>-s2</tt>).
|
|
|
|
|
<tt>-s0</tt> ignores it entirely, <tt>-s1</tt> obeys it but lets one of your
|
|
|
|
|
<tt>+</tt> filters override a disallow for a URL you explicitly asked for. Note
|
|
|
|
|
that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
|
|
|
|
|
options will not help there. That is an
|
|
|
|
|
<a href="#identity">identity</a> problem.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="limits">4. Limits and politeness</h3>
|
|
|
|
|
|
|
|
|
|
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
|
|
|
|
|
accident, and the <a href="abuse.html">abuse page</a> is worth a read. The limits
|
|
|
|
|
below let you go faster when you own the target, and slower when you do not.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>-AN</tt></td><td>Maximum transfer rate in bytes/sec. <b>The default is about 100 KB/s even without this flag.</b> Raise it to go faster.</td></tr>
|
|
|
|
|
<tr><td><tt>-cN</tt></td><td>Number of parallel connections (default 4). <tt>--tiny</tt>, <tt>--wide</tt> and <tt>--ultrawide</tt> are presets.</td></tr>
|
|
|
|
|
<tr><td><tt>-%cN</tt></td><td>New connections opened per second (default 5).</td></tr>
|
|
|
|
|
<tr><td><tt>-MN</tt></td><td>Stop after N bytes <b>received from the network</b> across the whole mirror (this counts what was transferred, not what was saved).</td></tr>
|
|
|
|
|
<tr><td><tt>-EN</tt></td><td>Stop after N seconds of wall-clock time.</td></tr>
|
|
|
|
|
<tr><td><tt>-mN</tt></td><td>Per-file size caps.</td></tr>
|
|
|
|
|
<tr><td><tt>-TN -RN -JN -HN</tt></td><td>Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.</td></tr>
|
|
|
|
|
<tr><td><tt>-GN -%GN</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p><b>The security clamps.</b> To keep an accidental typo from turning into a flood,
|
|
|
|
|
HTTrack silently caps a few values: at most 8 connections (<tt>-c</tt>), at most
|
|
|
|
|
10 MB/s (<tt>-A</tt>), and at most 5 new connections per second (<tt>-%c</tt>).
|
|
|
|
|
Ask for more and you get the ceiling, quietly. The single flag <tt>-%!</tt> lifts
|
|
|
|
|
all three. Use it only against infrastructure you are allowed to load that hard.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="names">5. File names and types</h3>
|
|
|
|
|
|
|
|
|
|
<p>Where local files land, and what they are called, is controlled by the naming
|
|
|
|
|
options. This is the second-biggest source of "why did it do that" questions,
|
|
|
|
|
usually about a URL like <tt>/article?id=42</tt> or a <tt>.php</tt> page that is
|
|
|
|
|
really HTML.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>-NN</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>-N "%h%p/%n%q.%t"</tt>.</td></tr>
|
|
|
|
|
<tr><td><tt>-LN</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
|
|
|
|
|
<tr><td><tt>-%A ext=type</tt></td><td>Assume a MIME type for an extension, for example <tt>-%A php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
|
|
|
|
|
<tr><td><tt>-%N -%D -u -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
|
|
|
|
|
<tr><td><tt>-%q -%g</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p>The <tt>-N</tt> presets are built from modular arithmetic on the name fields, so
|
|
|
|
|
undocumented number combinations often "work" by accident. If you care about the
|
|
|
|
|
exact layout, use an explicit template (the <tt>%h %p %n %q %t</tt> placeholders)
|
|
|
|
|
rather than a magic number, and check the result on a small crawl first.</p>
|
|
|
|
|
|
|
|
|
|
<p>A dynamic page served as <tt>.php</tt> or <tt>.asp</tt> that is actually HTML is
|
|
|
|
|
the classic case: without help it can be saved with an extension a browser will
|
|
|
|
|
not open locally. <tt>-%A php=text/html</tt> fixes both the extension and the
|
|
|
|
|
naming.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="links">6. Links and page building</h3>
|
|
|
|
|
|
|
|
|
|
<p>After a page is downloaded, HTTrack parses it for more links and rewrites the
|
|
|
|
|
ones it kept so the local copy browses offline. These options tune both halves.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>-KN</tt></td><td>How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare <tt>-K</tt> keeps <b>absolute</b> URLs, and <tt>-K0</tt> is the <b>relative</b> default. <tt>-K3</tt> keeps absolute URIs, <tt>-K4</tt> keeps the original links.</td></tr>
|
|
|
|
|
<tr><td><tt>-x -o</tt></td><td>Replace external links with an error page, and generate an error page for links that failed.</td></tr>
|
|
|
|
|
<tr><td><tt>-%p -%x</tt></td><td>Leave HTML untouched (no rewriting), and strip passwords out of saved links.</td></tr>
|
|
|
|
|
<tr><td><tt>-%P -j</tt></td><td>Aggressive link discovery, and how much script content is parsed for links.</td></tr>
|
|
|
|
|
<tr><td><tt>-%M</tt></td><td>Save the whole mirror as a single MIME-encapsulated <tt>.mht</tt> archive (<tt>index.mht</tt>).</td></tr>
|
|
|
|
|
<tr><td><tt>-I -%i -%I</tt></td><td>Build a per-mirror index, a top index across projects, and a searchable keyword index.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p>HTTrack finds links by parsing HTML and CSS. It does not run JavaScript, so any
|
|
|
|
|
URL a page builds at runtime in script (a lazy-loaded image, a
|
|
|
|
|
JavaScript-assembled path) is invisible to the crawler and will be missing from
|
|
|
|
|
the mirror. There is no flag that fixes this; the asset has to appear in the
|
|
|
|
|
static HTML or CSS to be found. <tt>-%P</tt> widens discovery for links that are
|
|
|
|
|
present but awkwardly formatted, not for links that do not exist until script
|
|
|
|
|
runs.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="identity">7. Identity, cookies and login</h3>
|
|
|
|
|
|
|
|
|
|
<p>By default HTTrack is an honest robot: it sends a <tt>User-Agent</tt> of
|
|
|
|
|
<tt>HTTrack</tt>, a Referer with each link (which reveals the crawl path to the
|
|
|
|
|
server), and obeys robots. Plenty of sites filter exactly that profile. These
|
|
|
|
|
options control what HTTrack says about itself.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>-F "string"</tt></td><td>The <tt>User-Agent</tt>. Set a browser string to get past crawler blocks; <tt>-F ""</tt> sends none.</td></tr>
|
|
|
|
|
<tr><td><tt>-%R -%E -%l -%a</tt></td><td>Referer, From, Accept-Language and Accept headers.</td></tr>
|
|
|
|
|
<tr><td><tt>-%X "line"</tt></td><td>Add raw header lines to every request.</td></tr>
|
|
|
|
|
<tr><td><tt>-%F</tt></td><td>A footer written into saved pages (on disk, not a network header).</td></tr>
|
|
|
|
|
<tr><td><tt>-b -%K file</tt></td><td>Accept cookies, and preload a Netscape <tt>cookies.txt</tt>.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p><b>Login.</b> For HTTP Basic auth, put the credentials in the URL:
|
|
|
|
|
<tt>http://user:pass@host/</tt>. An <tt>@</tt> inside the username must be written
|
|
|
|
|
<tt>%40</tt>. Only Basic is supported, not Digest.</p>
|
|
|
|
|
|
|
|
|
|
<p>For cookie or form logins, the simplest path is to log in with a browser, export
|
|
|
|
|
its <tt>cookies.txt</tt>, and drop that file in the project directory so HTTrack
|
|
|
|
|
sends the session cookie. For a form that needs a POST, <tt>--catchurl</tt> can
|
|
|
|
|
capture the exact request your browser sends and replay it. A few cookie caveats
|
|
|
|
|
to know: expiry is ignored, there is a silent cap of about 8 cookies sent per
|
|
|
|
|
request, and <tt>-b0</tt> disables cookies and the reuse of Basic credentials
|
|
|
|
|
across links at the same time.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="proxy">8. Proxy and network</h3>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>-P proxy:port</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
|
|
|
|
|
<tr><td><tt>-%f</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
|
|
|
|
|
<tr><td><tt>-@iN</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
|
|
|
|
|
<tr><td><tt>-%h -%k -%z</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
|
|
|
|
|
<tr><td><tt>-%b -%B</tt></td><td>Bind to a local address, and accept technically-bogus responses some servers send.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p>Two network facts worth stating plainly. Over SOCKS5, HTTrack always resolves
|
|
|
|
|
host names at the proxy (remote DNS) for both <tt>socks5://</tt> and
|
|
|
|
|
<tt>socks5h://</tt>, so your local resolver is never consulted. And HTTrack does
|
|
|
|
|
not verify TLS certificates: HTTPS gives you an encrypted transport, but not an
|
|
|
|
|
authenticated one. That is a deliberate choice for a mirroring tool, not a bug,
|
|
|
|
|
but it is worth knowing if you are relying on it for trust.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="update">9. Update and cache</h3>
|
|
|
|
|
|
|
|
|
|
<p>Every project keeps a cache under <tt>hts-cache/</tt>. It records every URL that
|
|
|
|
|
was fetched, together with the options you used, and it is what makes resuming and
|
|
|
|
|
updating possible. It is not a size-limited scratch area you can delete: throw it
|
|
|
|
|
away and you lose the ability to continue or update the mirror.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>--continue</tt></td><td>Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.</td></tr>
|
|
|
|
|
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
|
|
|
|
|
<tr><td><tt>-X0</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>-X0</tt> keeps them.</td></tr>
|
|
|
|
|
<tr><td><tt>-C</tt></td><td>Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.</td></tr>
|
|
|
|
|
<tr><td><tt>-%u -%j -%o -%y -%n -%s -k</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
|
|
|
|
|
<tr><td><tt>-#C -#R --clean</tt></td><td>Inspect the cache, repair its ZIP, and erase cache plus logs.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p><b>The purge trap.</b> An <tt>--update</tt> run rebuilds the list of files the
|
|
|
|
|
mirror should contain, then deletes any previously-mirrored file that is not on the
|
|
|
|
|
new list. This is what keeps a mirror in sync with a shrinking site, but it means a
|
|
|
|
|
partial or interrupted update can delete files you meant to keep. If an update might
|
|
|
|
|
not complete cleanly, add <tt>-X0</tt> to protect the existing tree, and expect
|
|
|
|
|
dynamic pages to look "changed" on every run. See the
|
|
|
|
|
<a href="cache.html">cache page</a> for the details.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="experts">10. Experts and scripting</h3>
|
|
|
|
|
|
|
|
|
|
<p>HTTrack is also a scriptable fetch-and-scan tool. These options turn off the
|
|
|
|
|
mirror behavior and expose the engine.</p>
|
|
|
|
|
|
|
|
|
|
<table class="tblRegular tableWidth" border="0">
|
|
|
|
|
<tr class="tblHeaderColor"><td><b>Option</b></td><td><b>What it controls</b></td></tr>
|
|
|
|
|
<tr><td><tt>--get URL</tt></td><td>Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.</td></tr>
|
|
|
|
|
<tr><td><tt>--spider --testlinks --skeleton</tt></td><td>Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.</td></tr>
|
|
|
|
|
<tr><td><tt>-V "cmd"</tt></td><td>Run a shell command on each downloaded file; <tt>$0</tt> is the file path. Good for on-the-fly processing.</td></tr>
|
|
|
|
|
<tr><td><tt>-%W module</tt></td><td>Load an external callback module to hook the engine.</td></tr>
|
|
|
|
|
<tr><td><tt>-Q -q -v -f -z -Z</tt></td><td>Logging: quiet, no questions, verbose on screen, and the various log-to-file levels.</td></tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<p>The <a href="dev.html">developer page</a> covers the callback API and batch use
|
|
|
|
|
in more depth.</p>
|
|
|
|
|
|
|
|
|
|
<hr>
|
|
|
|
|
|
|
|
|
|
<h3 id="recipes">11. Recipes</h3>
|
|
|
|
|
|
|
|
|
|
<p>Copy-ready command lines for the tasks people ask about most. Each has the one
|
|
|
|
|
gotcha that trips it up.</p>
|
|
|
|
|
|
|
|
|
|
<h4>Mirror one site, nothing off-host</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/ -O mydir</tt><br>
|
|
|
|
|
<small>Same-host is already the default. The usual failure is a
|
|
|
|
|
<tt>www.</tt>-to-apex or <tt>http</tt>-to-<tt>https</tt> redirect that moves you off
|
|
|
|
|
host and stops the crawl; start from the final URL, or add
|
|
|
|
|
<tt>"+finalhost/*"</tt>.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>One level of links only</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/ -r2 -O mydir</tt><br>
|
|
|
|
|
<small>Depth counts the start page as level 1, so one level out is <tt>-r2</tt>.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>This site only, deny everything else</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/ "-*" "+example.com/*" -O mydir</tt><br>
|
|
|
|
|
<small>You must deny-all first; a lone <tt>+</tt> only adds, and the last matching
|
|
|
|
|
rule wins.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Grab every PDF under a site</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/ "-*" "+example.com/*.pdf" -O mydir</tt><br>
|
|
|
|
|
<small>Always scope the host. A bare <tt>+*.pdf</tt> invites the crawler onto the
|
|
|
|
|
whole web.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Keep page requisites, including off-host images</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/blog/ -n -O mydir</tt><br>
|
|
|
|
|
<small><tt>-n</tt> can over-fetch by pulling in an entire external host from one
|
|
|
|
|
link; if it does, name the assets with <tt>"+cdn.example.com/*"</tt> instead.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Resume an interrupted crawl</h4>
|
|
|
|
|
<p><tt>httrack --continue -O mydir</tt><br>
|
|
|
|
|
<small>Needs an intact <tt>hts-cache/</tt>. Deleting it loses the URL list and your
|
|
|
|
|
options.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Update a mirror, downloading only what changed</h4>
|
|
|
|
|
<p><tt>httrack --update -O mydir</tt><br>
|
|
|
|
|
<small>Purge deletes any previously-mirrored file not seen this run; add
|
|
|
|
|
<tt>-X0</tt> to protect the tree against a partial update.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Reach a section behind a login cookie</h4>
|
|
|
|
|
<p><small>Export your browser's <tt>cookies.txt</tt> into the project directory,
|
|
|
|
|
then:</small><br>
|
|
|
|
|
<tt>httrack https://example.com/members/ -O mydir</tt><br>
|
|
|
|
|
<small>The file must be Netscape <tt>cookies.txt</tt> in the project folder;
|
|
|
|
|
<tt>-b0</tt> would disable cookies and Basic-auth reuse together.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Get past a crawler-UA block</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/ -F "Mozilla/5.0 (Windows NT 10.0)" -O mydir</tt><br>
|
|
|
|
|
<small>A 403 is server-side, not robots: change the User-Agent, and add
|
|
|
|
|
<tt>-%h</tt> for fragile CGI. Do not reach for <tt>-s0</tt>.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>Full-speed mirror on your own server</h4>
|
|
|
|
|
<p><tt>httrack https://example.com/ -A2000000 -c8 -%! -O mydir</tt><br>
|
|
|
|
|
<small>The default throttles to about 100 KB/s. <tt>-%!</tt> lifts the built-in
|
|
|
|
|
caps; use it only against infrastructure you are allowed to load.</small></p>
|
|
|
|
|
|
|
|
|
|
<h4>HTTrack as a fetch tool</h4>
|
|
|
|
|
<p><tt>httrack --get https://host/file.bin -O tmp</tt><br>
|
|
|
|
|
<small><tt>--get</tt> fetches one file with cache, index, depth, cookies and robots
|
|
|
|
|
all disabled.</small></p>
|
|
|
|
|
|
|
|
|
|
<p><br>For the complete, always-current option list, see
|
|
|
|
|
<a href="httrack.man.html">the manual page</a>. For the filter language, see
|
|
|
|
|
<a href="filters.html">filters</a>; for the cache and updates, see
|
|
|
|
|
<a href="cache.html">cache</a>.</p>
|
|
|
|
|
|
|
|
|
|
<!-- ==================== Start epilogue ==================== -->
|
|
|
|
|
</td>
|
|
|
|
|
</tr>
|
|
|
|
|
</table>
|
|
|
|
|
</td>
|
|
|
|
|
</tr>
|
|
|
|
|
</table>
|
|
|
|
|
</td>
|
|
|
|
|
</tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
|
|
|
|
|
<tr>
|
|
|
|
|
<td id="footer"><small>© 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
|
|
|
|
|
</tr>
|
|
|
|
|
</table>
|
|
|
|
|
|
|
|
|
|
</body>
|
|
|
|
|
|
|
|
|
|
</html>
|
|
|
|
|
|
