Compare commits

..

3 Commits

Author SHA1 Message Date
Xavier Roche
16485da5c7 Use *[path]/ so the PDF recipe follows nested directory indexes
*[file] cannot cross a slash, so *[file]/ only admits a single directory level and
silently drops PDFs linked under nested paths (dated archives, doc trees). *[path]/
spans slashes and follows directory-index pages at any depth, with no over-admission
(only trailing-slash URLs match, never files).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:48:49 +02:00
Xavier Roche
719a785c60 Merge master to refresh format base 2026-07-22 09:38:11 +02:00
Xavier Roche
4eb7334b1f Fix the download-PDFs example in the command-line guide
The old example ran httrack with no filters, so it mirrored the whole
site rather than downloading its PDFs. The corrected command restricts
to the site, admits the HTML pages as scaffolding for link discovery
(including directory-index pages via *[file]/ so sub-directory PDFs are
found), keeps the PDFs, and drops everything else.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:27:39 +02:00
6 changed files with 839 additions and 982 deletions

View File

@@ -618,31 +618,3 @@ jobs:
echo "Fix locally with: git clang-format --binary clang-format-19 $base"
exit 1 ;;
esac
man-page-sync:
name: man page / html in sync
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
# html/httrack.man.html is groff-rendered from man/httrack.1 and committed.
# Rendering needs the full groff html device, so CI can't regenerate it;
# instead require the two to move together: a PR that touches httrack.1
# must also touch the html, catching the "regenerated roff, forgot html".
- name: httrack.1 changes must include html/httrack.man.html
run: |
set -euo pipefail
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
changed="$(git diff --name-only "$base"...HEAD)"
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
if has man/httrack.1 && ! has html/httrack.man.html; then
echo "::error::man/httrack.1 changed but html/httrack.man.html did not."
echo "Regenerate it with: make -C man regen-man-html (needs the full groff package)."
exit 1
fi
echo "man/html sync OK."

View File

@@ -219,23 +219,6 @@ that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
options will not help there. That is an
<a href="#identity">identity</a> problem.</p>
<h4>Filter wildcards</h4>
<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
bracket forms match narrower sets. The full table, with size and mime rules, is on
<a href="filters.html">the filters page</a>.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> &mdash; any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> &mdash; a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
<tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
<tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
<tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
<tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
<tr><td><tt>*[&lt;NN]</tt>, <tt>*[&gt;NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[&lt;5]</tt> skips GIFs under 5&nbsp;KB</td></tr>
<tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
</table>
<h3 id="limits">4. Limits and politeness</h3>
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by

File diff suppressed because it is too large Load Diff

View File

@@ -95,7 +95,7 @@ ${do:end-if}
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_G44}</em></h2>
<h2 align="center"><em>Select URLs</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}

View File

@@ -77,7 +77,7 @@ ${do:end-if}
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_J9}</em></h2>
<h2 align="center"><em>Start</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}

View File

@@ -16,11 +16,6 @@ regen-man: makeman.sh $(top_builddir)/src/httrack$(EXEEXT)
# Render html/httrack.man.html from httrack.1. Needs the groff html device
# (Debian: full "groff" package, not "groff-base"). Run by hand: make -C man regen-man-html
# Strip groff's version-stamp and creation-date comments so the committed file
# doesn't churn across groff versions or rebuilds; the ci man-page-sync guard
# only requires this file to change whenever httrack.1 does.
regen-man-html: httrack.1
groff -t -man -Thtml $(srcdir)/httrack.1 \
| sed -e 's/groff version [0-9][0-9.]*/groff/' -e '/^<!-- CreationDate:/d' \
> $(top_srcdir)/html/httrack.man.html
groff -t -man -Thtml $(srcdir)/httrack.1 > $(top_srcdir)/html/httrack.man.html
.PHONY: regen-man-html