Compare commits

..

2 Commits

Author SHA1 Message Date
Xavier Roche
793a3375b7 Merge master to refresh format base 2026-07-22 09:37:43 +02:00
Xavier Roche
6099653139 Refresh the stale man-page HTML and guard it against future drift
html/httrack.man.html is groff-rendered from man/httrack.1 and committed, but
had drifted far behind the roff page (missing the option table-of-contents and
many options such as --pause and --delayed-type-check). Regenerate it, and make
regen-man-html strip groff's version-stamp and creation-date comments so the
committed file no longer churns across groff versions or rebuilds.

Add a lightweight CI guard: a PR that changes man/httrack.1 must also change
html/httrack.man.html, catching the "regenerated the roff page, forgot the HTML"
case that caused this drift. Rendering needs the full groff html device, so CI
verifies the two move together rather than regenerating in-CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:24:45 +02:00
3 changed files with 10 additions and 29 deletions

View File

@@ -219,23 +219,6 @@ that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
options will not help there. That is an
<a href="#identity">identity</a> problem.</p>
<h4>Filter wildcards</h4>
<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
bracket forms match narrower sets. The full table, with size and mime rules, is on
<a href="filters.html">the filters page</a>.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> &mdash; any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> &mdash; a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
<tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
<tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
<tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
<tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
<tr><td><tt>*[&lt;NN]</tt>, <tt>*[&gt;NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[&lt;5]</tt> skips GIFs under 5&nbsp;KB</td></tr>
<tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
</table>
<h3 id="limits">4. Limits and politeness</h3>
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
@@ -421,16 +404,14 @@ host and stops the crawl; start from the final URL, or add
rule wins.</small></p>
<h4>Download the PDFs on a site</h4>
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
<p><tt>httrack https://example.com/ --path mydir</tt><br>
<small>There is no PDF-only crawl. HTTrack discovers PDF links by parsing the
site's HTML pages, so a <tt>"-*" "+example.com/*.pdf"</tt> filter blocks the very
pages that carry the links and grabs only the PDFs linked from the front page. Let
it crawl the site: the HTML pages come along as the scaffolding, and every reachable
PDF is saved with them. If some PDFs live on another host (a CDN or a docs
subdomain), allow that host too, for example
<tt>"+docs.example.com/*.pdf"</tt>.</small></p>
<h4>Keep page requisites, including off-host images</h4>
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>

View File

@@ -95,7 +95,7 @@ ${do:end-if}
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_G44}</em></h2>
<h2 align="center"><em>Select URLs</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}

View File

@@ -77,7 +77,7 @@ ${do:end-if}
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_J9}</em></h2>
<h2 align="center"><em>Start</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}