Compare commits

..

3 Commits

Author SHA1 Message Date
Xavier Roche
16485da5c7 Use *[path]/ so the PDF recipe follows nested directory indexes
*[file] cannot cross a slash, so *[file]/ only admits a single directory level and
silently drops PDFs linked under nested paths (dated archives, doc trees). *[path]/
spans slashes and follows directory-index pages at any depth, with no over-admission
(only trailing-slash URLs match, never files).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:48:49 +02:00
Xavier Roche
719a785c60 Merge master to refresh format base 2026-07-22 09:38:11 +02:00
Xavier Roche
4eb7334b1f Fix the download-PDFs example in the command-line guide
The old example ran httrack with no filters, so it mirrored the whole
site rather than downloading its PDFs. The corrected command restricts
to the site, admits the HTML pages as scaffolding for link discovery
(including directory-index pages via *[file]/ so sub-directory PDFs are
found), keeps the PDFs, and drops everything else.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:27:39 +02:00
4 changed files with 847 additions and 971 deletions

View File

@@ -618,31 +618,3 @@ jobs:
echo "Fix locally with: git clang-format --binary clang-format-19 $base"
exit 1 ;;
esac
man-page-sync:
name: man page / html in sync
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
# html/httrack.man.html is groff-rendered from man/httrack.1 and committed.
# Rendering needs the full groff html device, so CI can't regenerate it;
# instead require the two to move together: a PR that touches httrack.1
# must also touch the html, catching the "regenerated roff, forgot html".
- name: httrack.1 changes must include html/httrack.man.html
run: |
set -euo pipefail
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
changed="$(git diff --name-only "$base"...HEAD)"
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
if has man/httrack.1 && ! has html/httrack.man.html; then
echo "::error::man/httrack.1 changed but html/httrack.man.html did not."
echo "Regenerate it with: make -C man regen-man-html (needs the full groff package)."
exit 1
fi
echo "man/html sync OK."

View File

@@ -404,14 +404,16 @@ host and stops the crawl; start from the final URL, or add
rule wins.</small></p>
<h4>Download the PDFs on a site</h4>
<p><tt>httrack https://example.com/ --path mydir</tt><br>
<small>There is no PDF-only crawl. HTTrack discovers PDF links by parsing the
site's HTML pages, so a <tt>"-*" "+example.com/*.pdf"</tt> filter blocks the very
pages that carry the links and grabs only the PDFs linked from the front page. Let
it crawl the site: the HTML pages come along as the scaffolding, and every reachable
PDF is saved with them. If some PDFs live on another host (a CDN or a docs
subdomain), allow that host too, for example
<tt>"+docs.example.com/*.pdf"</tt>.</small></p>
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
<h4>Keep page requisites, including off-host images</h4>
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>

File diff suppressed because it is too large Load Diff

View File

@@ -16,11 +16,6 @@ regen-man: makeman.sh $(top_builddir)/src/httrack$(EXEEXT)
# Render html/httrack.man.html from httrack.1. Needs the groff html device
# (Debian: full "groff" package, not "groff-base"). Run by hand: make -C man regen-man-html
# Strip groff's version-stamp and creation-date comments so the committed file
# doesn't churn across groff versions or rebuilds; the ci man-page-sync guard
# only requires this file to change whenever httrack.1 does.
regen-man-html: httrack.1
groff -t -man -Thtml $(srcdir)/httrack.1 \
| sed -e 's/groff version [0-9][0-9.]*/groff/' -e '/^<!-- CreationDate:/d' \
> $(top_srcdir)/html/httrack.man.html
groff -t -man -Thtml $(srcdir)/httrack.1 > $(top_srcdir)/html/httrack.man.html
.PHONY: regen-man-html