Command-Line Guide

This is a task-oriented guide to the httrack command line: how to do the things people actually ask for, and the handful of defaults that surprise newcomers. It sits on top of the generated manual page, which lists every option in full. When you want the exhaustive detail for a flag, that page is the reference; this one is the map.

Two habits before anything else. First, HTTrack has its own options: they are not wget or curl flags, so reach for the tables here rather than guessing. Second, when a mirror does something you did not expect, the answer is almost always in the log. Every project writes hts-log.txt (and hts-err.txt) into its output directory, and those files name every URL that was refused, redirected, or filtered out. Read them first.

On this page

• 1. Quick start
• 2. Scope: how far the crawl reaches
• 3. Filters and scan rules
• 4. Limits and politeness
• 5. File names and types
• 6. Links and page building
• 7. Identity, cookies and login
• 8. Proxy and network
• 9. Update and cache
• 10. Experts and scripting
• 11. Recipes

1. Quick start

A mirror is one command: a start URL and an output directory.

httrack https://example.com/ --path mydir

With no other options HTTrack mirrors that site, stays on the same host, follows links to any depth, rebuilds them to browse offline, and stores everything under mydir. The same directory also holds the log files and the hts-cache/ folder that makes a later update or resume possible.

Two defaults are worth knowing up front, because both catch people out:

• HTTrack throttles itself to about 100 KB/s even when you pass no rate option. If a mirror feels slow, that is why. See Limits for how to lift it.
• The download proceeds as a well-behaved robot: it identifies itself as HTTrack, obeys robots.txt, and sends a Referer with each request. A site that blocks that behavior needs the levers in Identity, not brute force.

2. Scope: how far the crawl reaches

Scope decides which links HTTrack is even willing to follow, before any filter you write. Get this right and most "it downloaded too much" or "it only grabbed the index" problems disappear.

The single most common surprise is "only the home page came down." That is usually not a scope option at all: it is an off-host redirect. A start URL of http://example.com/ that redirects to https://www.example.com/ lands you on a different host, and same-host scope stops the crawl there. Start from the final URL, or add a filter that re-admits the real host (see Filters). The log will show the redirect.

-n is the fix for pages that render locally without their images or stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note that its embedded-asset handling (following img, link, script, style and HTML5 source/track targets past the normal depth and filter limits) applies only when -n is on; it is not automatic. It can also over-fetch by dragging in a whole external host from a single link, in which case name the assets you want with a filter instead.

3. Filters and scan rules

Filters are the number-one source of confusion, and also the tool that solves most scope problems once you understand them. A filter is a rule that accepts (+) or rejects (-) URLs by pattern. The sign is mandatory: +pattern adds, -pattern removes, and a bare pattern is an error.

The rules that matter:

• Last match wins. Rules are applied in order and the last one that matches a URL decides its fate. Order your rules from general to specific.
• Wildcards. * matches any run of characters; *[a-z], *[0-9] and similar classes match sets. So +*.pdf means "any URL ending in .pdf".
• Whitelisting. To keep one site and nothing else, deny everything then re-admit the host: "-*" "+example.com/*". A lone + rule only adds to the default scope; it never restricts.
• Size rules. *[>100000] and *[<1000] filter by byte size. Because size is only known once the transfer starts, an oversize file is fetched partway and then aborted, not skipped for free.
• mime: rules. A rule like -mime:video/* matches the Content-Type. That type is only known after the response headers arrive, so a mime rule cannot stop a request; it can only abort the body. Use a URL pattern when you want to avoid the fetch entirely.

Quote your filters. Shells treat *, [ and sometimes + specially, so wrap each rule in quotes as shown above. The full pattern language, with tables for wildcards, size and mime, is in the filters page, and the FAQ has a worked tutorial.

robots.txt. By default HTTrack obeys robots.txt (-s2). -s0 ignores it entirely, -s1 obeys it but lets one of your + filters override a disallow for a URL you explicitly asked for. Note that a 403 Forbidden is a server refusal, not a robots rule: robots options will not help there. That is an identity problem.

4. Limits and politeness

HTTrack ships cautious on purpose: it is easy to hammer a small site by accident, and the abuse page is worth a read. The limits below let you go faster when you own the target, and slower when you do not.

The security clamps. To keep an accidental typo from turning into a flood, HTTrack silently caps a few values: at most 8 connections (-c), at most 10 MB/s (-A), and at most 5 new connections per second (-%c). Ask for more and you get the ceiling, quietly. The single flag --disable-security-limits lifts all three (the short form -%! also works, but the bare ! is awkward to type safely in a shell). Use it only against infrastructure you are allowed to load that hard.

5. File names and types

Where local files land, and what they are called, is controlled by the naming options. This is the second-biggest source of "why did it do that" questions, usually about a URL like /article?id=42 or a .php page that is really HTML.

The -N presets are built from modular arithmetic on the name fields, so undocumented number combinations often "work" by accident. If you care about the exact layout, use an explicit template (the %h %p %n %q %t placeholders) rather than a magic number, and check the result on a small crawl first.

A dynamic page served as .php or .asp that is actually HTML is the classic case: without help it can be saved with an extension a browser will not open locally. --assume php=text/html fixes both the extension and the naming.

6. Links and page building

After a page is downloaded, HTTrack parses it for more links and rewrites the ones it kept so the local copy browses offline. These options tune both halves.

HTTrack finds links by parsing HTML and CSS. It does not run JavaScript, so any URL a page builds at runtime in script (a lazy-loaded image, a JavaScript-assembled path) is invisible to the crawler and will be missing from the mirror. There is no flag that fixes this; the asset has to appear in the static HTML or CSS to be found. -%P widens discovery for links that are present but awkwardly formatted, not for links that do not exist until script runs.

7. Identity, cookies and login

By default HTTrack is an honest robot: it sends a User-Agent of HTTrack, a Referer with each link (which reveals the crawl path to the server), and obeys robots. Plenty of sites filter exactly that profile. These options control what HTTrack says about itself.

Login. For HTTP Basic auth, put the credentials in the URL: http://user:pass@host/. An @ inside the username must be written %40. Only Basic is supported, not Digest.

For cookie or form logins, the simplest path is to log in with a browser, export its cookies.txt, and drop that file in the project directory so HTTrack sends the session cookie. For a form that needs a POST, --catchurl can capture the exact request your browser sends and replay it. A few cookie caveats to know: expiry is ignored, there is a silent cap of about 8 cookies sent per request, and -b0 disables cookies and the reuse of Basic credentials across links at the same time.

8. Proxy and network

Two network facts worth stating plainly. Over SOCKS5, HTTrack always resolves host names at the proxy (remote DNS) for both socks5:// and socks5h://, so your local resolver is never consulted. And HTTrack does not verify TLS certificates: HTTPS gives you an encrypted transport, but not an authenticated one. That is a deliberate choice for a mirroring tool, not a bug, but it is worth knowing if you are relying on it for trust.

9. Update and cache

Every project keeps a cache under hts-cache/. It records every URL that was fetched, together with the options you used, and it is what makes resuming and updating possible. It is not a size-limited scratch area you can delete: throw it away and you lose the ability to continue or update the mirror.

The purge trap. An --update run rebuilds the list of files the mirror should contain, then deletes any previously-mirrored file that is not on the new list. This is what keeps a mirror in sync with a shrinking site, but it means a partial or interrupted update can delete files you meant to keep. If an update might not complete cleanly, add -X0 to protect the existing tree, and expect dynamic pages to look "changed" on every run. See the cache page for the details.

10. Experts and scripting

HTTrack is also a scriptable fetch-and-scan tool. These options turn off the mirror behavior and expose the engine.

The developer page covers the callback API and batch use in more depth.

11. Recipes

Copy-ready command lines for the tasks people ask about most. Each has the one gotcha that trips it up.

Mirror one site, nothing off-host

httrack https://example.com/ --path mydir
Same-host is already the default. The usual failure is a www.-to-apex or http-to-https redirect that moves you off host and stops the crawl; start from the final URL, or add "+finalhost/*".

One level of links only

httrack https://example.com/ --depth=2 --path mydir
Depth counts the start page as level 1, so one level out is --depth=2.

This site only, deny everything else

httrack https://example.com/ "-*" "+example.com/*" --path mydir
You must deny-all first; a lone + only adds, and the last matching rule wins.

Download the PDFs on a site

httrack https://example.com/ --path mydir
There is no PDF-only crawl. HTTrack discovers PDF links by parsing the site's HTML pages, so a "-*" "+example.com/*.pdf" filter blocks the very pages that carry the links and grabs only the PDFs linked from the front page. Let it crawl the site: the HTML pages come along as the scaffolding, and every reachable PDF is saved with them. If some PDFs live on another host (a CDN or a docs subdomain), allow that host too, for example "+docs.example.com/*.pdf".

Keep page requisites, including off-host images

httrack https://example.com/blog/ --near --path mydir
--near can over-fetch by pulling in an entire external host from one link; if it does, name the assets with "+cdn.example.com/*" instead.

Resume an interrupted crawl

httrack --continue --path mydir
Needs an intact hts-cache/. Deleting it loses the URL list and your options.

Update a mirror, downloading only what changed

httrack --update --path mydir
Purge deletes any previously-mirrored file not seen this run; add --purge-old=0 to protect the tree against a partial update.

Reach a section behind a login cookie

Export your browser's cookies.txt into the project directory, then:
httrack https://example.com/members/ --path mydir
The file must be Netscape cookies.txt in the project folder; --cookies=0 would disable cookies and Basic-auth reuse together.

Get past a crawler-UA block

httrack https://example.com/ --user-agent "Mozilla/5.0 (Windows NT 10.0)" --path mydir
A 403 is server-side, not robots: change the User-Agent, and add --http-10 for fragile CGI. Do not reach for --robots=0.

Full-speed mirror on your own server

httrack https://example.com/ --max-rate=2000000 --sockets=8 --disable-security-limits --path mydir
The default throttles to about 100 KB/s. --disable-security-limits lifts the built-in caps; use it only against infrastructure you are allowed to load.

HTTrack as a fetch tool

httrack --get https://host/file.bin --path tmp
--get fetches one file with cache, index, depth, cookies and robots all disabled.

For the complete, always-current option list, see the manual page. For the filter language, see filters; for the cache and updates, see cache.