*[file] cannot cross a slash, so *[file]/ only admits a single directory level and
silently drops PDFs linked under nested paths (dated archives, doc trees). *[path]/
spans slashes and follows directory-index pages at any depth, with no over-admission
(only trailing-slash URLs match, never files).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The old example ran httrack with no filters, so it mirrored the whole
site rather than downloading its PDFs. The corrected command restricts
to the site, admits the HTML pages as scaffolding for link discovery
(including directory-index pages via *[file]/ so sub-directory PDFs are
found), keeps the PDFs, and drops everything else.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Advertise -%N's long option and document the real long forms
The help display appends each option's long alias by looking it up in the
alias table, but it first stripped a trailing N to turn placeholders like
cN into c. That also turned -%N into -%, so --delayed-type-check was never
advertised in --help (nor in the generated man page). Try the flag as-is
first and only strip a trailing N on a miss; -%N now shows its long form,
and cN still resolves to --sockets.
The command-line guide had several rows marked short-only that in fact
have long aliases (I had read them off a stale installed binary): fill in
--pause, --strip-query, --disable-compression, the three --keep-* dedup
opts, and --delayed-type-check. Regenerate man/httrack.1 to match, and
drop the long-dead commented-out -%O chroot help line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Bound the option-token sscanf to the buffer size
CodeQL flagged the %s read into cmd[32] as an unbounded copy on the line
the previous commit touched. The tokens come from HTTrack's own help
strings, not hostile input, so it was not reachable in practice, but the
copy should be bounded regardless. Limit it to %30s (the buffer holds the
leading '-' plus 30 chars and a NUL).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add a task-oriented command-line guide to the offline docs
The command-line docs so far are the option list and the generated manual
page: exhaustive, but organized by flag, not by task. Newcomers arrive
expecting wget/curl syntax and hit the same walls (only the index came
down, filters that do the opposite of what they read, an update that
deletes files), because the reference answers "what does -X do", not "how
do I do the thing I want".
cmdguide.html is a task-oriented layer on top of the manual page: quick
start, scope, filters, limits, naming, links, identity/login, proxy,
update/cache, scripting, then eleven copy-ready recipes with the one
gotcha each. It foregrounds the defaults that actually surprise people
(the ~100 KB/s rate cap, the -c/-A/-%c security clamps, --update purge
semantics, the depth off-by-one, mime filters running after headers) and
links the manual page for per-option detail. Linked from cmddoc.html and
the index. Every documented default and behavior was checked against the
engine source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Correct four scope/option descriptions in the guide
Adversarial review against the engine source caught four mislabels:
-d is "same principal domain" and -l is "same TLD" (not "same directory"
and "same domain"); -U goes up only and -B goes both ways (the guide had
-U as up-and-down and -B as "anywhere"); -%M archives the whole mirror
into one index.mht, not each page; and -t HEAD-tests links outside scope
rather than reporting "what a scope would reach". The rest of the guide's
documented behavior verified against code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the PDF recipe, prefer long options, drop the section rules
Review feedback from Xavier:
- The "grab every PDF" recipe was wrong. `-* +*.pdf` blocks the HTML pages
that carry the PDF links, so the crawl only keeps PDFs linked from the
entry page (verified on a local fixture). There is no PDF-only crawl:
HTTrack finds PDFs by parsing HTML. Reworded to let the site traverse,
with the off-host case handled by a `+host/*.pdf` rule.
- Recipes now use long options throughout. `-%!` in particular is replaced
by `--disable-security-limits`: the bare `!` triggers shell history
expansion and is easy to fumble, and the long name says what it does.
- Dropped the per-section `<hr>` rules; the sibling doc pages don't use
them and the section headings already separate the content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Key the reference tables by long option name, short in parens
Follows the recipe conversion: every table row now leads with the long
option (--depth, --stay-on-same-domain, ...) and carries the short flag in
parentheses. The guru %-flags that have no long form (-%g, -%j, -%o, -%y,
-%z, -%G, -%t, -%N) stay short. Also switches the remaining inline command
snippets in descriptions to long form (--assume, --structure,
--user-agent "").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--help (and the manual page generated from it) claimed the default socket
count is 8 and the default new-connections-per-second is 10. The engine
actually defaults to 4 sockets (htslib.c) and 5 connections/second, which
is also the -%c ceiling. The 8 in the old text is the -c clamp, not the
default, so the line conflated the two.
Correct the two markers to (*c4) and (*%c5) in htshelp.c and regenerate
man/httrack.1 via man/makeman.sh. html/httrack.man.html carries the same
two-token fix applied directly, since regenerating it needs groff's html
device (Debian: full groff, not groff-base); a later full regen on such a
host reproduces the identical output.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The command-line documentation (cmddoc.html, options.html, and the generated httrack.man.html) was unreachable from the documentation index; they only linked each other. Link cmddoc.html into the "How to Use" list so the whole set is reachable, and replace options.html's stale ~2007 option list with a pointer to the generated man page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Adds html/android.html, a step-by-step guide for the HTTrack Android app (install, project, address, options, run, browse, storage), with screenshots of the shipped app and an options section covering all eleven tabs and the settings that are missing or fixed on Android. Linked from the documentation index between the WinHTTrack/WebHTTrack and Fred Cohen entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Bumps the copyright year in the html/ documentation footers (and the inline notice in contact.html) from 2007 to 1998-2026. Footer text only, no content changes; fcguide.html (Fred Cohen, upstream) is left untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Four factual bugs in the shipped html/ help pages: the FAQ's stale "SOCKS? Not yet!" answer (SOCKS5 and HTTP CONNECT proxies have shipped), cache.html's false ">4GiB ZIP64 not supported" claim, two -mime:video/* example rows in filters.html mislabeled "application/", and a compile-breaking fprintf(stder, ...) in plug.html's sample module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
HTTPS already reaches the origin through a CONNECT tunnel (#85), but plain HTTP still used the absolute-URI request form, which a CONNECT-only proxy like Tor's HTTPTunnelPort rejects. Adds a `connect://` proxy scheme (`-P connect://host:port`) that drives the proxy with CONNECT for every request: httrack opens a tunnel to the origin (port 80 for http, 443 for https), then sends the request origin-form with no Proxy-Authorization, the same way https does.
It can't be the default (CONNECT-to-:80 breaks proxies that only allow CONNECT on 443) and auto-detection is ruled out, so the mode rides the proxy scheme like socks5:// does. That keeps it ABI- and cache-safe (the scheme lives in proxy.name, no new struct field) and drops into the webhttrack GUI as one more entry beside HTTP and SOCKS5. The tooltip was reworded and retranslated across the lang/*.txt files.
Test 57 fronts a plain-http origin with a CONNECT-only proxy (it 501s anything but CONNECT) and checks the request arrives origin-form, that dropping connect:// gets the classic form refused, that credentials ride the CONNECT and never reach the origin, that a portless origin tunnels to :80, and that a flooding proxy does not hang.
Closes#564.
The web GUI carried per-control help via onMouseOver="info(...)", which
only wrote window.status. Browsers stopped honoring scripted window.status
writes years ago, so those hints render nowhere.
Emit each hint as a native title='...' attribute alongside the existing
handler, reusing the same ${html:LANG_*} string. Single-quoted so the
filter's ' -> ' (and <>&) escaping keeps the attribute well-formed; a
literal " is harmless inside single quotes. The window.status calls stay
(inert) to keep the diff to added attributes.
The webhttrack smoke test now also fetches an option page and requires a
rendered title=' tooltip.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The web GUI was frozen before the engine gained SOCKS5 proxying, --cookies-file,
--pause, --strip-query, and the three --keep-* URL-hack opt-outs. Wire each into
the option tabs, the generated httrack command, the winprofile.ini persistence,
and the reload remap so they reach parity with WinHTTrack (xroche/httrack-windows#31).
The SOCKS5 selector reuses the single proxy field and prepends socks5:// to -P;
it is gated on a non-empty host so a SOCKS5-selected-but-blank config can't emit a
stray scheme token. winprofile.ini key names match WinHTTrack so projects round-trip
across both GUIs. Labels/tooltips reuse the LANG_ strings already added in #588.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove the obsolete Java-applet .class parser and dead SWF vestiges
Java applets stopped running in every mainstream browser years ago (NPAPI
dropped by Chrome in 2015 and Firefox in 2017; the JDK browser plugin gone
since JDK 11), so htsjava -- a hand-rolled parser of hostile .class bytecode
fetched off the network -- chased dependencies no live site serves while
carrying real attack surface for no functional gain. The SWF module was
already gone; only vestiges remained (a dead libhtsswf.so.1 dlopen entry and
help-text examples).
htsjava was a dlopen plugin, never linked into libhttrack, so its removal
leaves libhttrack's ABI untouched: libhttrack.so.3 is unchanged and the
libhttrack3/-dev packages just stop shipping libhtsjava.so.3*. That is a
plain file drop dpkg removes on upgrade -- no package rename, no
Replaces/Breaks, and nothing ever linked the library (it was dlopened), so
there are no reverse dependencies. The parsejava/-j option stays; it also
gates JavaScript parsing.
Drops the plugin sources, the build/config/vcproj wiring, the java self-test
and its two engine tests, and the now-unused hts_count_fits helper (whose
only caller was the plugin), and regenerates the man page. The <applet> URL
rewriting in the HTML parser and the .class codebase bookkeeping are left
intact, so applet pages are still mirrored as plain files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Scrub the removed java/swf modules from the shipped HTML docs
Follow-up to the htsjava/SWF removal: the pre-generated docs in html/ (shipped
in httrack-doc) still named the gone modules. Update the two stale --help
mirrors in httrack.man.html by hand (a full groff regen would rewrite the whole
file under a newer groff) and drop the htsjava.c plugin-example reference from
plug.html.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Finish scrubbing the java-class docs across the guides and man source
The -j option still exists (it drives JavaScript parsing), so every "parse
Java Classes" summary in the HTML guides becomes "parse scripts", and the
obsolete .class-parsing descriptions and java-applet troubleshooting entries
are reworded to drop the removed capability. The "Some java classes may not
find..." engine-limit line is dropped at its source (README, from which
makeman.sh regenerates the man LIMITS section), and httrack.1 regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* filters: decode escaped chars correctly inside *[...] classes
The escape branch in strjoker probed joker[i+2] instead of the current
char, so a backslash escape only worked as the first class member:
'*[\[\]]' (documented as "the [ or ] character") matched only ']', and
'*[a,\[]' dropped the 'a'. The loop also treated any ']' as the class
terminator, so an escaped ']' could never be a member.
Decode the escape first in the loop body: a backslash takes the next char
as the literal member (only that char, not also the backslash the old code
added), and an escaped ']' is consumed before the terminator check. So
'*[\[\]]' now matches both brackets, and escape precedes the range/size
checks ('\-' '\,' '\<' become literal members). The self-test previously
pinned the buggy output as expected; it now asserts the documented
behavior and fails against the old matcher.
Closes#148
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* filters: fix a 1-byte over-read on a truncated range *[a-
The *[...] class parser's range arm does i += 3 unconditionally, so a
pattern ending in a dangling '-' (e.g. *[a-) read one byte past the NUL:
joker[i+2] is the NUL, i jumps to len+1, and the separator skip and loop
guard then read joker[len+1]. Guard the range arm on joker[i+2] != '\0'
so a truncated range falls through to the literal-member path instead of
overshooting.
The filter self-test now copies the pattern and string into exact-size
heap buffers so a sanitizer traps such over-reads; the pattern previously
came straight from argv (no redzone), which is why this stayed invisible.
A *[a- test case exercises it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Fix the shellcheck findings the shfmt pass left behind, all proven
behaviour-preserving:
- Quote single-value expansions, drop the redundant ${} in arithmetic,
add read -r, and use printf '%s' instead of variables in format
strings, across the generators, crawl-test.sh, run-all-tests.sh and
search.sh.
- crawl-test.sh / webhttrack: turn the deliberately word-split search
lists into bash arrays (space-safe, no scattered disables) and replace
the numeric trap signal lists with names, dropping the un-trappable
KILL/STOP that bash silently ignored anyway.
- search.sh: drop the bogus \" escapes that made grep search for a
literal-quoted pattern.
The generators are exercised by hand and ship their committed output
(htscodepages.h, htsentities.h); a differential run on synthetic input
confirms byte-identical output before and after. crawl-test.sh and
webhttrack were run end to end against a local server / a faked install,
the latter also proving the array search now survives spaces in paths.
SC2153/SC2120 false positives carry a scoped disable with a reason.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Mechanical pass: run shfmt -i 4 over the whole tracked shell tree (the
test harness .test files, the regen generators, webhttrack, the CGI
search helper, and the build/dist scripts) so they share one style.
shfmt also normalised backticks to $(...) and $[..] to $((..)).
No behaviour change: arithmetic is preserved exactly, non-ASCII bytes
are untouched, and the full make check suite still passes. The tab
indented .test files become 4-space indented, hence the wide diff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The generated build system (configure, every Makefile.in, config.h.in,
ltmain.sh, config.guess/sub, the aux scripts) was committed so a bare git
clone could build without autotools. Nothing downstream relied on the
committed copies: CI runs autoreconf -fi, Debian regenerates via
dh_autoreconf, and the release tarball is built by make dist, which
regenerates them regardless. The only cost was a recurring footgun: a stale
Makefile.in after a *_SOURCES edit silently broke the plain build (undefined
reference to cache_selftests), and CI could not catch it.
Treat them as build products. They are now .gitignored and regenerated from
configure.ac/Makefile.am by the new ./bootstrap (autoreconf -fi), and shipped
only inside make dist tarballs so tarball users still need no autotools.
build.sh is a one-shot wrapper (bootstrap + configure + make) that runs
configure via /bin/sh, so it survives a noexec source tree. Both scripts join
EXTRA_DIST. INSTALL.Linux, README.md and AGENTS.md document the git flow:
./bootstrap before ./configure, autotools required for a git build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
-Wl,--push-state,--no-as-needed,-lc,--pop-state forces libc back into
DT_NEEDED for libraries that reach it only through libhttrack: the
libhtsjava JNI wrapper and the libtest callback examples. The flag is
GNU-ld-specific; Apple's ld rejects it ("ld: unknown options:
--push-state --no-as-needed --pop-state"), breaking the macOS build, and
doesn't need it (every dylib links libSystem anyway).
Probe it once with AX_CHECK_LINK_FLAG and emit it via LIBC_FORCE_LINK
only where the linker accepts it. On GNU/Linux the flag is still applied
and libc.so.6 stays in DT_NEEDED, so behavior is unchanged there.
Signed-off-by: Xavier Roche <roche@httrack.com>
The Debian AppStream generator flagged both webhttrack desktop entries as
no-metainfo: with no MetaInfo file, the catalog entry is synthesized from
the .desktop file and the package description, which is deprecated and risks
the app being dropped from the metadata catalog.
Add com.httrack.WebHTTrack.metainfo.xml (installed to share/metainfo) for the
main app, launching WebHTTrack.desktop. Mark the secondary "Browse Mirrored
Websites" launcher with X-AppStream-Ignore=true so it doesn't produce a
duplicate, metadata-less catalog entry.
Validated with appstreamcli validate and desktop-file-validate.
Signed-off-by: Xavier Roche <roche@httrack.com>
The tracked generated files (configure, */Makefile.in, ltmain.sh, aux
scripts) had drifted stale: src/Makefile.in predated #343, which added
htscache_selftest.c to the library sources, so a plain
`bash configure && make` link-failed with "undefined reference to
cache_selftests". CI regenerates with `autoreconf -fi` on every run, so it
never saw the staleness.
Regenerate with `autoreconf -fi` so a from-checkout build needs no autotools
installed, as intended. This also bumps the generators (autoconf 2.71->2.72,
automake 1.17, libtool 2.4.7), hence the large but purely generated diff.
Also commit the automake test-driver aux script (its siblings compile,
depcomp, missing, install-sh are already committed); without it `make check`
on a fresh checkout could not find $(top_srcdir)/test-driver. A clean
checkout now builds and passes `make check` with no autotools installed.
Signed-off-by: Xavier Roche <roche@httrack.com>
Out-of-tree builds were broken in two ways, and "make check" could not run
when the source tree sits on a noexec filesystem. Fix both so a plain
`mkdir build && cd build && bash <srcdir>/configure && make && make check`
works without copying the tree.
libtest: -I../src is relative to the build dir, so out-of-tree it pointed at
build/src (generated files only) and missed the source header
httrack-library.h. Use -I$(top_srcdir)/src.
Wildcard DATA lists (libtest, lang, html, m4) used bare globs like "*.html".
Make expands a wildcard prerequisite against the build dir, so out-of-tree the
glob matches nothing and stays literal ("No rule to make target '*.html'").
Glob against $(srcdir) instead. Explicit filenames (e.g. ../history.txt) are
left as-is; they resolve through VPATH.
make check: automake's driver execve()s each tests/*.test, which fails with
"Permission denied" when the source tree is on a noexec mount. Run them through
bash via TEST_LOG_COMPILER = $(BASH) (detected by configure); this also drops
any reliance on the scripts' executable bit and works on a normal tree too.
Verified end to end from a noexec source tree: out-of-tree make builds the full
tree including libtest, and "make check" runs (14 pass, online crawl tests skip
offline, 0 fail).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
someweb.com is a real registrable domain; example.com is reserved for
documentation (RFC 2606). Replace it across the HTML guides, the CLI
--help text (htshelp.c) and code comments, then regenerate man/httrack.1
and the rendered html/httrack.man.html. Other placeholder domains are
left alone: they appear inside filter/wildcard examples where the host
interacts with the pattern.
Escape the literal <URLs>, <FILTERs>, <param>, <filter>, <file> and
related placeholders in fcguide.html so they render instead of being
swallowed as unknown HTML tags; several were also missing their closing
'>'. Use --recurse-submodules in the README clone command. Relabel
lang/Ukrainian.txt as windows-1251, which is what its bytes actually
are (ISO-8859-5 decodes them to garbage).
closes#132, closes#103, closes#167
* closes:#53
Also fixed HTML-escaping issues inside webhttrack
Rationale: The webhttrack script made the wrong assumption that once the "browse" command returned, it meant the user killed the navigation window, and it had to kill the server itself. However, modern browsers tend to "attach" to an existing session (creating a new tab, for example, within an existing window), causing the browsing command to return immediately, thus causing the server to be killed immediately by the webhttrack script. I have rewritten the logic behind, and now the server is able to kill himself if the parent script dies, AND if the browsing client did not make any activity for two minutes. The "activity" can be any browser/refreshed page, or the internal "ping" iframe (which pings the server every 30 seconds). With this model, we *should* be compatible with old browsers, and modern ones.
Removed unused CXX
added the following default compiler flags:
-Wdeclaration-after-statement
-Wsequence-point
-Wparentheses
-Winit-self
-Wuninitialized
-Wformat
-fstrict-aliasing -Wstrict-aliasing=2
added the following default linker flags:
-Wl,--discard-all
-Wl,--no-undefined
Depending on autoconf-archive because using AX_CHECK_COMPILE_FLAG and AX_CHECK_LINK_FLAG
