mirror of
https://github.com/xroche/httrack.git
synced 2026-07-12 11:57:30 +03:00
68897b09ce
* Re-fetch a self-redirect cookie wall instead of dropping it (#15) A page that 302-redirects to itself only to set a cookie (a consent or session "cookie wall") was lost: the self-redirect guard treated the loop as a crazy server and never re-issued the request, so the real content behind the cookie was never fetched. The redirect's Set-Cookie is already folded into the shared jar before the guard runs, so a re-issue would carry it. In the delayed-type loop (hts_wait_delayed) snapshot the jar before the request; when a self-redirect changed it, evict the cached fast-header for that URL and re-fetch once with the new cookie. Termination is bounded: the jar stops changing once the cookie is satisfied (real walls resolve in two requests), and the existing loops<7 cap backstops a server that mints a fresh cookie on every hit. Under the default HARD delayed-type mode every URL routes through this loop, so both unknown-ext (wall.php) and known-ext (wall.html) walls are covered. A wall reached only under -%N0/1 still takes the hts_mirror_check_moved path, which cannot see the Set-Cookie without an ABI change; left as a known limitation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Test the cookie-wall give-up and retry-cap paths (#15) The happy-path test only drove a wall that sets its cookie on the first hit, so an engine that dropped the jar-changed gate and retried every self-redirect would still pass. Add two cases that pin the gate and the bound: - cookiewall3: a self-redirect that sets no cookie must give up at once (asserts the "loop to same filename" guard fires and the page is not mirrored). A dropped gate would retry and skip that log line. - cookiewall4: a self-redirect that mints a fresh cookie every hit must stop at the loops<7 cap, not spin (asserts the crawl terminates, takes the retry path, and does not mirror the wall). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> * Scope the cookie-wall retry to this URL's own cookies (#15) The retry trigger compared the whole shared cookie jar before and after the request. Because the jar is one process-wide store mutated by every in-flight fetch, a concurrent slot's Set-Cookie landing mid-loop could flip the compare and force a needless (though bounded) re-fetch of an unrelated self-redirect. Compare instead the Cookie header THIS url would send, built for its own host/path. A Set-Cookie from another host no longer trips the retry; a change in this URL's own outgoing cookies (its self-redirect setting one) still does, which is exactly the wall we want to re-fetch. No jar-wide snapshot, and the same natural termination: once the cookie stabilizes the header stops changing. Reuses append_cookie_header via http_cookie_header (renamed from the _selftest-suffixed wrapper, now that engine code calls it too). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com> --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HTTrack Website Copier - Development Repository
About
Copy websites to your computer (Offline browser)
HTTrack is an offline browser utility, allowing you to download a World Wide website from the Internet to a local directory, building recursively all directories, getting html, images, and other files from the server to your computer.
HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online.
HTTrack can also update an existing mirrored site, and resume interrupted downloads. HTTrack is fully configurable, and has an integrated help system.
WinHTTrack is the Windows 2000/XP/Vista/Seven release of HTTrack, and WebHTTrack the Linux/Unix/BSD release.
Website
Main Website: http://www.httrack.com/
Compile trunk release
A git checkout ships only the autotools sources, so ./bootstrap (which runs
autoreconf) regenerates configure first; this needs autoconf, automake and
libtool. Released tarballs already include configure, so building from a
tarball skips ./bootstrap.
git clone https://github.com/xroche/httrack.git --recurse-submodules
cd httrack
./bootstrap
./configure --prefix=$HOME/usr && make -j8 && make install
Or use the one-shot wrapper (bootstrap + configure + make), which forwards its
arguments to configure:
./build.sh --prefix=$HOME/usr