Compare commits

..

2 Commits

Author SHA1 Message Date
Xavier Roche
793a3375b7 Merge master to refresh format base 2026-07-22 09:37:43 +02:00
Xavier Roche
6099653139 Refresh the stale man-page HTML and guard it against future drift
html/httrack.man.html is groff-rendered from man/httrack.1 and committed, but
had drifted far behind the roff page (missing the option table-of-contents and
many options such as --pause and --delayed-type-check). Regenerate it, and make
regen-man-html strip groff's version-stamp and creation-date comments so the
committed file no longer churns across groff versions or rebuilds.

Add a lightweight CI guard: a PR that changes man/httrack.1 must also change
html/httrack.man.html, catching the "regenerated the roff page, forgot the HTML"
case that caused this drift. Rendering needs the full groff html device, so CI
verifies the two move together rather than regenerating in-CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:24:45 +02:00
4 changed files with 981 additions and 857 deletions

View File

@@ -618,3 +618,31 @@ jobs:
echo "Fix locally with: git clang-format --binary clang-format-19 $base"
exit 1 ;;
esac
man-page-sync:
name: man page / html in sync
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
# html/httrack.man.html is groff-rendered from man/httrack.1 and committed.
# Rendering needs the full groff html device, so CI can't regenerate it;
# instead require the two to move together: a PR that touches httrack.1
# must also touch the html, catching the "regenerated roff, forgot html".
- name: httrack.1 changes must include html/httrack.man.html
run: |
set -euo pipefail
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
changed="$(git diff --name-only "$base"...HEAD)"
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
if has man/httrack.1 && ! has html/httrack.man.html; then
echo "::error::man/httrack.1 changed but html/httrack.man.html did not."
echo "Regenerate it with: make -C man regen-man-html (needs the full groff package)."
exit 1
fi
echo "man/html sync OK."

View File

@@ -404,16 +404,14 @@ host and stops the crawl; start from the final URL, or add
rule wins.</small></p>
<h4>Download the PDFs on a site</h4>
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
<p><tt>httrack https://example.com/ --path mydir</tt><br>
<small>There is no PDF-only crawl. HTTrack discovers PDF links by parsing the
site's HTML pages, so a <tt>"-*" "+example.com/*.pdf"</tt> filter blocks the very
pages that carry the links and grabs only the PDFs linked from the front page. Let
it crawl the site: the HTML pages come along as the scaffolding, and every reachable
PDF is saved with them. If some PDFs live on another host (a CDN or a docs
subdomain), allow that host too, for example
<tt>"+docs.example.com/*.pdf"</tt>.</small></p>
<h4>Keep page requisites, including off-host images</h4>
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>

File diff suppressed because it is too large Load Diff

View File

@@ -16,6 +16,11 @@ regen-man: makeman.sh $(top_builddir)/src/httrack$(EXEEXT)
# Render html/httrack.man.html from httrack.1. Needs the groff html device
# (Debian: full "groff" package, not "groff-base"). Run by hand: make -C man regen-man-html
# Strip groff's version-stamp and creation-date comments so the committed file
# doesn't churn across groff versions or rebuilds; the ci man-page-sync guard
# only requires this file to change whenever httrack.1 does.
regen-man-html: httrack.1
groff -t -man -Thtml $(srcdir)/httrack.1 > $(top_srcdir)/html/httrack.man.html
groff -t -man -Thtml $(srcdir)/httrack.1 \
| sed -e 's/groff version [0-9][0-9.]*/groff/' -e '/^<!-- CreationDate:/d' \
> $(top_srcdir)/html/httrack.man.html
.PHONY: regen-man-html