mirror of
https://github.com/xroche/httrack.git
synced 2026-07-25 01:59:27 +03:00
bd7e0989f6
The robots.txt handler only did substring Disallow matching against a flat token blob: no Allow:, no path wildcards. Sites using "Disallow: /" plus "Allow: /public/" were over-blocked, since Allow was never parsed. Move the body parsing into robots_parse() (htsrobots.c) so both the crawler and a self-test feed raw robots.txt. Rules are stored Allow/Disallow-tagged and consulted with RFC 9309 precedence: the longest matching path pattern wins, Allow breaking ties. Pattern matching supports '*' (any run) and a trailing '$' (end-of-path anchor) via a linear two-pointer matcher with a single resumable star position, so hostile patterns cannot trigger exponential backtracking. Path matching is now case-sensitive per the RFC. robots_wizard is internal (not in DevIncludes_DATA, no HTSEXT_API; htsopt.h holds only an opaque pointer), so the in-memory format changed without an ABI break. Sitemap:/Crawl-delay: lines are tolerated but ignored, as before. New -#test=robots self-test plus tests/01_engine-robots.test cover the Allow-over-Disallow longest match, the equal-length Allow tie, '*'/'$' wildcards, and httrack-group selection. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
8 lines
181 B
Bash
Executable File
8 lines
181 B
Bash
Executable File
#!/bin/bash
|
|
#
|
|
|
|
set -euo pipefail
|
|
|
|
# robots.txt RFC 9309 Allow/Disallow precedence (#452): longest match wins.
|
|
httrack -O /dev/null -#test=robots run | grep -q "robots self-test OK"
|