mirror of
https://github.com/xroche/httrack.git
synced 2026-07-01 06:33:47 +03:00
Compare commits
1 Commits
3.49.10
...
selftest-n
| Author | SHA1 | Date | |
|---|---|---|---|
|
0a21ec9acf |
45
.github/workflows/ci.yml
vendored
45
.github/workflows/ci.yml
vendored
@@ -188,51 +188,6 @@ jobs:
|
||||
if: failure()
|
||||
run: cat tests/test-suite.log 2>/dev/null || true
|
||||
|
||||
# MemorySanitizer catches reads of uninitialized memory (#143's stack-garbage
|
||||
# size filter) that ASan/UBSan miss. It flags any byte an uninstrumented lib
|
||||
# wrote, so the job stays in our own code: offline self-tests only, no openssl
|
||||
# (--disable-https), no zlib cache tests, static (the runtime is not in .so's).
|
||||
msan:
|
||||
name: msan (MemorySanitizer, clang)
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: recursive
|
||||
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y --no-install-recommends \
|
||||
build-essential clang autoconf automake libtool autoconf-archive \
|
||||
zlib1g-dev
|
||||
|
||||
- name: Configure (MSan, static, no https)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
autoreconf -fi
|
||||
./configure CC=clang \
|
||||
CFLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-sanitize-recover=all -g -O1 -fno-omit-frame-pointer" \
|
||||
LDFLAGS="-fsanitize=memory" \
|
||||
--disable-https --disable-shared --enable-static
|
||||
|
||||
- name: Build
|
||||
run: make -j"$(nproc)"
|
||||
|
||||
- name: Test (offline self-tests under MSan)
|
||||
env:
|
||||
MSAN_OPTIONS: abort_on_error=1:halt_on_error=1
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Engine self-tests only; the cache trio pulls in uninstrumented zlib.
|
||||
tests="$(cd tests && ls 01_engine-*.test | grep -v -- '-cache' | tr '\n' ' ')"
|
||||
make check TESTS="$tests"
|
||||
|
||||
- name: Print the test log on failure
|
||||
if: failure()
|
||||
run: cat tests/test-suite.log 2>/dev/null || true
|
||||
|
||||
# Optional-dependency build: compile and test with HTTPS/OpenSSL disabled --
|
||||
# the configuration users on minimal systems build, and one libssl is not even
|
||||
# installed here so configure cannot silently re-enable it. The matrix above
|
||||
|
||||
@@ -39,10 +39,6 @@ Welcome, and nothing to disclose. Two rules:
|
||||
|
||||
The sign-off covers AI-assisted code too.
|
||||
|
||||
## Translations
|
||||
|
||||
Interface strings live in [`lang/`](lang/). See [lang/README.md](lang/README.md) for the file format and how to add or update a language.
|
||||
|
||||
## Bugs
|
||||
|
||||
Open an issue with the version, OS, command used, and expected vs actual result.
|
||||
|
||||
10
configure.ac
10
configure.ac
@@ -1,6 +1,6 @@
|
||||
AC_PREREQ([2.71])
|
||||
|
||||
AC_INIT([httrack], [3.49.10], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
|
||||
AC_INIT([httrack], [3.49.9], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
|
||||
AC_COPYRIGHT([
|
||||
HTTrack Website Copier, Offline Browser for Windows and Unix
|
||||
Copyright (C) 1998-2015 Xavier Roche and other contributors
|
||||
@@ -29,10 +29,10 @@ AC_CONFIG_SRCDIR(src/httrack.c)
|
||||
AC_CONFIG_MACRO_DIR([m4])
|
||||
AC_CONFIG_HEADERS(config.h)
|
||||
AM_INIT_AUTOMAKE([subdir-objects])
|
||||
# 3:2:0: 3.49.10 only appends tail fields to the options struct (no existing
|
||||
# symbol or offset changed vs 3.49.9), so it stays soname .so.3; bump revision.
|
||||
# (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
|
||||
VERSION_INFO="3:2:0"
|
||||
# 3:1:0: 3.49.9 changed code but not the exported interface vs 3.49.8 (same 164
|
||||
# symbols, no struct-layout change), so bump revision only. (3:0:0 was the htsblk
|
||||
# mime-buffer widening, an ABI break that moved the soname .so.2 -> .so.3.)
|
||||
VERSION_INFO="3:1:0"
|
||||
AM_MAINTAINER_MODE
|
||||
AC_USE_SYSTEM_EXTENSIONS
|
||||
|
||||
|
||||
13
debian/changelog
vendored
13
debian/changelog
vendored
@@ -1,16 +1,3 @@
|
||||
httrack (3.49.10-1) unstable; urgency=medium
|
||||
|
||||
* New upstream release: new download-pacing and URL-handling options plus a
|
||||
batch of crawl and robustness fixes (full list in history.txt).
|
||||
* Rewrite debian/copyright in machine-readable DEP-5 format, crediting the
|
||||
bundled minizip, md5 and coucal sources (#415).
|
||||
* Lead the webhttrack browser dependency with chromium so httrack is not
|
||||
dragged into the firefox-esr autoremoval cascade (#436).
|
||||
* Override the embedded-library lint for the bundled minizip (#419).
|
||||
* Bump Standards-Version to 4.7.4 (no changes required).
|
||||
|
||||
-- Xavier Roche <xavier@debian.org> Sun, 28 Jun 2026 14:01:53 +0200
|
||||
|
||||
httrack (3.49.9-1) unstable; urgency=medium
|
||||
|
||||
* New upstream release: Content-Type and file-type detection fixes (trust a
|
||||
|
||||
4
debian/control
vendored
4
debian/control
vendored
@@ -2,7 +2,7 @@ Source: httrack
|
||||
Section: web
|
||||
Priority: optional
|
||||
Maintainer: Xavier Roche <roche@httrack.com>
|
||||
Standards-Version: 4.7.4
|
||||
Standards-Version: 4.7.0
|
||||
Build-Depends: debhelper-compat (= 13), autoconf, autoconf-archive, automake, libtool, zlib1g-dev, libssl-dev
|
||||
Rules-Requires-Root: no
|
||||
Homepage: http://www.httrack.com
|
||||
@@ -30,7 +30,7 @@ Description: Copy websites to your computer (Offline browser)
|
||||
Package: webhttrack
|
||||
Architecture: any
|
||||
Multi-Arch: foreign
|
||||
Depends: ${misc:Depends}, ${shlibs:Depends}, webhttrack-common, sensible-utils, chromium | firefox-esr | www-browser
|
||||
Depends: ${misc:Depends}, ${shlibs:Depends}, webhttrack-common, sensible-utils, firefox-esr | chromium | www-browser
|
||||
Replaces: webhttrack-common (<< 3.43.9-2)
|
||||
Breaks: webhttrack-common (<< 3.43.9-2)
|
||||
Suggests: httrack, httrack-doc
|
||||
|
||||
20
history.txt
20
history.txt
@@ -4,25 +4,7 @@ HTTrack Website Copier release history:
|
||||
|
||||
This file lists all changes and fixes that have been made for HTTrack
|
||||
|
||||
3.49-10
|
||||
+ New: --cookies-file to preload a Netscape cookies.txt before crawling (#215)
|
||||
+ New: --pause to space out file downloads by a random delay (#185)
|
||||
+ New: --strip-query to drop selected query keys from the dedup naming (#112)
|
||||
+ Changed: split the -%u URL hacks into independent --keep-www-prefix, --keep-double-slashes and --keep-query-order toggles (#271)
|
||||
+ Fixed: follow a redirect Location after dropping its #fragment, instead of requesting the fragment and polluting the saved name (#204)
|
||||
+ Fixed: escaped brackets inside a *[...] filter character class (#148)
|
||||
+ Fixed: honor the server's Content-Range when resuming a partial download, instead of appending overlapping bytes (#198)
|
||||
+ Fixed: abort the download as soon as the response type is excluded by -mime:, instead of fetching then discarding the body (#58)
|
||||
+ Fixed: keep size-based filter rules neutral until the file size is known (#143)
|
||||
+ Fixed: stop the mirror with a clean fatal error on a cache write failure, instead of crashing (#174, #219)
|
||||
+ Fixed: stop the 412/416 partial re-get loop on --continue and --update (#206)
|
||||
+ Fixed: keep an unrecognized URL tail instead of mangling it to .html (#115)
|
||||
+ Fixed: honor --tolerant (-%B) on a broken Content-Length, and fix an out-of-bounds read it exposed (#32, #41)
|
||||
+ Fixed: fall back to the next resolved address when a connection fails or stalls, instead of hanging on a dead IPv6 address
|
||||
+ Fixed: report why a -%L URL list could not be loaded (#49)
|
||||
+ Changed: multiple internal hardening, build and CI improvements
|
||||
|
||||
.49-9
|
||||
3.49-9
|
||||
+ Fixed: file-type detection from the Content-Type header: trust a declared type over a binary URL extension, honor --assume under the delayed type check, and keep a known extension against a bogus or empty Content-Type (#267, #29, #56)
|
||||
+ Fixed: an uninitialized-buffer read when the Content-Type is empty (#411)
|
||||
+ Fixed: restored C++ source-compatibility of the installed headers so reverse dependencies (httraqt) build again (#413)
|
||||
|
||||
@@ -247,7 +247,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
<td>the \ character</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td nowrap><tt>*[\[,\]]</tt></td>
|
||||
<td nowrap><tt>*[\[\]]</tt></td>
|
||||
<td>the [ or ] character</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
||||
@@ -295,7 +295,7 @@ Max Depth
|
||||
Maximum external depth:
|
||||
Maximum external depth:
|
||||
Filters (refuse/accept links) :
|
||||
Filters (refuse/accept links):
|
||||
Filters (refuse/accept links) :
|
||||
Paths
|
||||
Paths
|
||||
Save prefs
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
# Translating HTTrack
|
||||
|
||||
Interface strings live here, one `.txt` file per language. `English.txt` is the reference: every other file maps each English string to its translation.
|
||||
|
||||
## File format
|
||||
|
||||
Plain text, entries in consecutive pairs of lines:
|
||||
|
||||
```
|
||||
<English string>
|
||||
<translation>
|
||||
```
|
||||
|
||||
The first line of a pair is the lookup key and must stay identical to the one in `English.txt`; translate only the second line. Missing entries fall back to the English text at runtime, so a partial translation works.
|
||||
|
||||
Preserve any `\r\n`, `\t` and `printf` placeholders (`%s`, `%d`, ...) in the translation.
|
||||
|
||||
A few `LANGUAGE_*` entries at the top describe the file itself:
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `LANGUAGE_NAME` | Name shown in the language picker, in its own language (`Deutsch`, not `German`) |
|
||||
| `LANGUAGE_ISO` | ISO 639 code, with region if needed (`de`, `pt_BR`) |
|
||||
| `LANGUAGE_CHARSET` | Encoding the file is saved in (`ISO-8859-1`, `windows-1251`, `UTF-8`, ...) |
|
||||
| `LANGUAGE_AUTHOR` | Your name and contact |
|
||||
| `LANGUAGE_WINDOWSID` | Windows locale name used by WinHTTrack (`German (Standard)`) |
|
||||
|
||||
Save the file in exactly its declared `LANGUAGE_CHARSET`; an editor that rewrites it as UTF-8 will corrupt the non-ASCII bytes.
|
||||
|
||||
## Adding or updating a language
|
||||
|
||||
1. Copy `English.txt` to `<Language>.txt`, or edit the existing file.
|
||||
2. Translate each second line; leave the English keys untouched.
|
||||
3. Fill in the `LANGUAGE_*` header for a new file.
|
||||
4. Open a pull request, or attach the file to a GitHub issue.
|
||||
|
||||
When new strings land in `English.txt` they show up untranslated (as English) until a translator fills them in.
|
||||
@@ -3,7 +3,7 @@
|
||||
.\"
|
||||
.\" This file is generated by man/makeman.sh; do not edit by hand.
|
||||
.\" SPDX-License-Identifier: GPL-3.0-or-later
|
||||
.TH httrack 1 "27 June 2026" "httrack website copier"
|
||||
.TH httrack 1 "26 June 2026" "httrack website copier"
|
||||
.SH NAME
|
||||
httrack \- offline browser : copy websites to a local directory
|
||||
.SH SYNOPSIS
|
||||
@@ -24,7 +24,6 @@ httrack \- offline browser : copy websites to a local directory
|
||||
[ \fB\-EN, \-\-max\-time[=N]\fR ]
|
||||
[ \fB\-AN, \-\-max\-rate[=N]\fR ]
|
||||
[ \fB\-%cN, \-\-connection\-per\-second[=N]\fR ]
|
||||
[ \fB\-%G, \-\-pause\fR ]
|
||||
[ \fB\-GN, \-\-max\-pause[=N]\fR ]
|
||||
[ \fB\-cN, \-\-sockets[=N]\fR ]
|
||||
[ \fB\-TN, \-\-timeout[=N]\fR ]
|
||||
@@ -44,13 +43,11 @@ httrack \- offline browser : copy websites to a local directory
|
||||
[ \fB\-x, \-\-replace\-external\fR ]
|
||||
[ \fB\-%x, \-\-disable\-passwords\fR ]
|
||||
[ \fB\-%q, \-\-include\-query\-string\fR ]
|
||||
[ \fB\-%g, \-\-strip\-query\fR ]
|
||||
[ \fB\-o, \-\-generate\-errors\fR ]
|
||||
[ \fB\-X, \-\-purge\-old[=N]\fR ]
|
||||
[ \fB\-%p, \-\-preserve\fR ]
|
||||
[ \fB\-%T, \-\-utf8\-conversion\fR ]
|
||||
[ \fB\-bN, \-\-cookies[=N]\fR ]
|
||||
[ \fB\-%K, \-\-cookies\-file\fR ]
|
||||
[ \fB\-u, \-\-check\-type[=N]\fR ]
|
||||
[ \fB\-j, \-\-parse\-java[=N]\fR ]
|
||||
[ \fB\-sN, \-\-robots[=N]\fR ]
|
||||
@@ -156,8 +153,6 @@ maximum mirror time in seconds (60=1 minute, 3600=1 hour) (\-\-max\-time[=N])
|
||||
maximum transfer rate in bytes/seconds (1000=1KB/s max) (\-\-max\-rate[=N])
|
||||
.IP \-%cN
|
||||
maximum number of connections/seconds (*%c10) (\-\-connection\-per\-second[=N])
|
||||
.IP \-%G
|
||||
random pause of MIN[:MAX] seconds between files (e.g. %G5:10) (\-\-pause <param>)
|
||||
.IP \-GN
|
||||
pause transfer if N bytes reached, and wait until lock file is deleted (\-\-max\-pause[=N])
|
||||
.SS Flow control:
|
||||
@@ -203,8 +198,6 @@ replace external html links by error pages (\-\-replace\-external)
|
||||
do not include any password for external password protected websites (%x0 include) (\-\-disable\-passwords)
|
||||
.IP \-%q
|
||||
*include query string for local files (useless, for information purpose only) (%q0 don't include) (\-\-include\-query\-string)
|
||||
.IP \-%g
|
||||
strip query keys for dedup ([host/pattern=]key1,key2,...) (\-\-strip\-query <param>)
|
||||
.IP \-o
|
||||
*generate output html file in case of error (404..) (o0 don't generate) (\-\-generate\-errors)
|
||||
.IP \-X
|
||||
@@ -216,8 +209,6 @@ links conversion to UTF\-8 (\-\-utf8\-conversion)
|
||||
.SS Spider options:
|
||||
.IP \-bN
|
||||
accept cookies in cookies.txt (0=do not accept,* 1=accept) (\-\-cookies[=N])
|
||||
.IP \-%K
|
||||
load extra cookies from a Netscape cookies.txt (\-\-cookies\-file <param>)
|
||||
.IP \-u
|
||||
check document type if unknown (cgi,asp..) (u0 don't check, * u1 check but /, u2 check always) (\-\-check\-type[=N])
|
||||
.IP \-j
|
||||
@@ -234,8 +225,6 @@ tolerant requests (accept bogus responses on some servers, but not standard!) (\
|
||||
update hacks: various hacks to limit re\-transfers when updating (identical size, bogus response..) (\-\-updatehack)
|
||||
.IP \-%u
|
||||
url hacks: various hacks to limit duplicate URLs (strip //, www.foo.com==foo.com..) (\-\-urlhack)
|
||||
.br
|
||||
opt out of one url\-hack part: \-\-keep\-www\-prefix (www.foo.com<>foo.com), \-\-keep\-double\-slashes (//), \-\-keep\-query\-order (?b&a)
|
||||
.IP \-%A
|
||||
assume that a type (cgi,asp..) is always linked with a mime type (\-%A php3,cgi=text/html;dat,bin=application/x\-zip) (\-\-assume <param>)
|
||||
.br
|
||||
|
||||
@@ -60,9 +60,6 @@ Please visit our Website: http://www.httrack.com
|
||||
param1 : this option must be alone, and needs one distinct parameter (-P <path>)
|
||||
param0 : this option must be alone, but the parameter should be put together (+*.gif)
|
||||
*/
|
||||
/* clang-format off: hand-aligned table; clang-format reflows the whole
|
||||
initializer (2->4 space) on any edit, churning every untouched row. */
|
||||
/* clang-format off */
|
||||
const char *hts_optalias[][4] = {
|
||||
/* {"","","",""}, */
|
||||
{"path", "-O", "param1", "output path"},
|
||||
@@ -110,12 +107,6 @@ const char *hts_optalias[][4] = {
|
||||
{"disable-passwords", "-%x", "single", ""}, {"disable-password", "-%x",
|
||||
"single", ""},
|
||||
{"include-query-string", "-%q", "single", ""},
|
||||
{"strip-query", "-%g", "param1",
|
||||
"strip [host/pattern=]key1,key2,... from URLs"},
|
||||
{"cookies-file", "-%K", "param1",
|
||||
"load extra cookies from a Netscape cookies.txt"},
|
||||
{"pause", "-%G", "param1",
|
||||
"random pause of MIN[:MAX] seconds between files"},
|
||||
{"generate-errors", "-o", "single", ""},
|
||||
{"do-not-generate-errors", "-o0", "single", ""},
|
||||
{"purge-old", "-X", "param", ""},
|
||||
@@ -132,9 +123,6 @@ const char *hts_optalias[][4] = {
|
||||
{"tolerant", "-%B", "single", ""},
|
||||
{"updatehack", "-%s", "single", ""}, {"sizehack", "-%s", "single", ""},
|
||||
{"urlhack", "-%u", "single", ""},
|
||||
{"keep-www-prefix", "-%j", "single", ""},
|
||||
{"keep-double-slashes", "-%o", "single", ""},
|
||||
{"keep-query-order", "-%y", "single", ""},
|
||||
{"user-agent", "-F", "param1", "user-agent identity"},
|
||||
{"referer", "-%R", "param1", "default referer URL"},
|
||||
{"from", "-%E", "param1", "from email address"},
|
||||
@@ -253,7 +241,6 @@ const char *hts_optalias[][4] = {
|
||||
|
||||
{"", "", "", ""}
|
||||
};
|
||||
/* clang-format on */
|
||||
|
||||
/*
|
||||
Check for alias in command-line
|
||||
|
||||
120
src/htsback.c
120
src/htsback.c
@@ -57,10 +57,7 @@ Please visit our Website: http://www.httrack.com
|
||||
// DOS
|
||||
#include <process.h> /* _beginthread, _endthread */
|
||||
#endif
|
||||
#include <io.h> /* _chsize_s */
|
||||
#define HTS_FTRUNCATE(fp, sz) _chsize_s(_fileno(fp), (sz))
|
||||
#else
|
||||
#define HTS_FTRUNCATE(fp, sz) ftruncate(fileno(fp), (sz))
|
||||
#endif
|
||||
|
||||
#define VT_CLREOL "\33[K"
|
||||
@@ -3766,27 +3763,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
}
|
||||
#endif
|
||||
/********** **************************** ********** */
|
||||
}
|
||||
// MIME type excluded by a -mime: filter: abort, don't fetch
|
||||
// the body (#58)
|
||||
else if (HTTP_IS_OK(back[i].r.statuscode) &&
|
||||
!back[i].testmode &&
|
||||
strnotempty(back[i].r.contenttype) &&
|
||||
hts_acceptmime(opt, 0, back[i].url_adr,
|
||||
back[i].url_fil,
|
||||
back[i].r.contenttype) == 1) {
|
||||
deletehttp(&back[i].r);
|
||||
back[i].r.soc = INVALID_SOCKET;
|
||||
back[i].status = STATUS_READY;
|
||||
back_set_finished(sback, i);
|
||||
back[i].r.statuscode = STATUSCODE_EXCLUDED;
|
||||
strcpybuff(back[i].r.msg, "Excluded by MIME type filter");
|
||||
hts_log_print(
|
||||
opt, LOG_NOTICE,
|
||||
"File excluded by MIME type filter (%s): %s%s",
|
||||
back[i].r.contenttype, back[i].url_adr,
|
||||
back[i].url_fil);
|
||||
} else { // il faut aller le chercher
|
||||
} else { // il faut aller le chercher
|
||||
|
||||
// effacer buffer (requète)
|
||||
if (!noFreebuff) {
|
||||
@@ -3797,70 +3774,35 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
// xxc SI CHUNK VERIFIER QUE CA MARCHE??
|
||||
if (back[i].r.statuscode == 206) { // on nous envoie un morceau (la fin) coz une partie sur disque!
|
||||
off_t sz = fsize_utf8(back[i].url_sav);
|
||||
/* RFC 7233: resume at the server's Content-Range start,
|
||||
not the offset we requested; a server may resume
|
||||
earlier and appending the overlap duplicates bytes
|
||||
(#198). */
|
||||
const LLint resume = back[i].r.crange_start;
|
||||
const hts_boolean range_ok =
|
||||
back[i].r.crange > 0 && resume >= 0 &&
|
||||
resume <= (LLint) sz &&
|
||||
back[i].r.crange_end + 1 == back[i].r.crange &&
|
||||
(back[i].r.totalsize < 0 ||
|
||||
back[i].r.totalsize ==
|
||||
back[i].r.crange_end - resume + 1);
|
||||
|
||||
#if HDEBUG
|
||||
printf("partial content: " LLintP " on disk..\n",
|
||||
(LLint) sz);
|
||||
#endif
|
||||
if (sz >= 0 && range_ok) {
|
||||
if (sz >= 0) {
|
||||
if (!is_hypertext_mime(opt, back[i].r.contenttype, back[i].url_sav)) { // pas HTML
|
||||
if (opt->getmode & HTS_GETMODE_NONHTML) {
|
||||
filenote(&opt->state.strc, back[i].url_sav, NULL); // noter fichier comme connu
|
||||
file_notify(opt, back[i].url_adr, back[i].url_fil,
|
||||
back[i].url_sav, 0, 1,
|
||||
back[i].r.notmodified);
|
||||
back[i].r.out =
|
||||
FOPEN(fconv(catbuff, sizeof(catbuff),
|
||||
back[i].url_sav),
|
||||
"r+b"); // resume in place
|
||||
back[i].r.out = FOPEN(fconv(catbuff, sizeof(catbuff), back[i].url_sav), "ab"); // append
|
||||
if (back[i].r.out && opt->cache != 0) {
|
||||
back[i].r.is_write = 1;
|
||||
back[i].r.size = resume; // bytes already on disk
|
||||
back[i].r.statuscode = HTTP_OK; // force 'OK'
|
||||
back[i].r.is_write = 1; // écrire
|
||||
back[i].r.size = sz; // déja écrit
|
||||
back[i].r.statuscode = HTTP_OK; // Forcer 'OK'
|
||||
if (back[i].r.totalsize >= 0)
|
||||
back[i].r.totalsize += resume; // -> full size
|
||||
// drop bytes past the resume point; a silent
|
||||
// failure could leave a stale tail, so on error
|
||||
// drop the partial and refetch the whole file
|
||||
if (HTS_FTRUNCATE(back[i].r.out,
|
||||
(off_t) resume) != 0) {
|
||||
fclose(back[i].r.out);
|
||||
back[i].r.out = NULL;
|
||||
url_savename_refname_remove(
|
||||
opt, back[i].url_adr, back[i].url_fil);
|
||||
UNLINK(back[i].url_sav);
|
||||
back[i].status = STATUS_READY;
|
||||
back_set_finished(sback, i);
|
||||
strcpybuff(back[i].r.msg,
|
||||
"Can not truncate partial file, "
|
||||
"restarting");
|
||||
} else {
|
||||
fseeko(back[i].r.out, (off_t) resume, SEEK_SET);
|
||||
/* create a temporary reference file in case of
|
||||
* broken mirror */
|
||||
if (back_serialize_ref(opt, &back[i]) != 0) {
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
"Could not create temporary "
|
||||
"reference file for %s%s",
|
||||
back[i].url_adr,
|
||||
back[i].url_fil);
|
||||
}
|
||||
#if HDEBUG
|
||||
printf("continue interrupted file\n");
|
||||
#endif
|
||||
back[i].r.totalsize += sz; // plus en fait
|
||||
fseek(back[i].r.out, 0, SEEK_END); // à la fin
|
||||
/* create a temporary reference file in case of broken mirror */
|
||||
if (back_serialize_ref(opt, &back[i]) != 0) {
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
"Could not create temporary reference file for %s%s",
|
||||
back[i].url_adr, back[i].url_fil);
|
||||
}
|
||||
#if HDEBUG
|
||||
printf("continue interrupted file\n");
|
||||
#endif
|
||||
} else { // On est dans la m**
|
||||
back[i].status = STATUS_READY; // terminé (voir plus loin)
|
||||
back_set_finished(sback, i);
|
||||
@@ -3872,18 +3814,17 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
FILE *fp =
|
||||
FOPEN(fconv(catbuff, sizeof(catbuff), back[i].url_sav), "rb");
|
||||
if (fp) {
|
||||
LLint alloc_mem = resume + 1;
|
||||
LLint alloc_mem = sz + 1;
|
||||
|
||||
if (back[i].r.totalsize >= 0)
|
||||
alloc_mem += back[i].r.totalsize; // AJOUTER RESTANT!
|
||||
if (deleteaddr(&back[i].r)
|
||||
&& (back[i].r.adr =
|
||||
(char *) malloct((size_t) alloc_mem))) {
|
||||
back[i].r.size = resume;
|
||||
back[i].r.size = sz;
|
||||
if (back[i].r.totalsize >= 0)
|
||||
back[i].r.totalsize += resume; // -> full size
|
||||
if ((fread(back[i].r.adr, 1, (size_t) resume,
|
||||
fp)) != (size_t) resume) {
|
||||
back[i].r.totalsize += sz; // plus en fait
|
||||
if ((fread(back[i].r.adr, 1, sz, fp)) != sz) {
|
||||
back[i].status = STATUS_READY; // terminé (voir plus loin)
|
||||
back_set_finished(sback, i);
|
||||
strcpybuff(back[i].r.msg,
|
||||
@@ -3901,30 +3842,14 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
"No memory for partial file");
|
||||
}
|
||||
fclose(fp);
|
||||
} else { // open failed
|
||||
} else { // Argh..
|
||||
back[i].status = STATUS_READY; // terminé (voir plus loin)
|
||||
back_set_finished(sback, i);
|
||||
strcpybuff(back[i].r.msg,
|
||||
"Can not open partial file");
|
||||
}
|
||||
}
|
||||
} else if (sz >=
|
||||
0) { // unusable range -> restart whole file
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
"Unusable partial-content range for %s%s "
|
||||
"(have " LLintP " bytes, got " LLintP
|
||||
"-" LLintP "/" LLintP "), restarting",
|
||||
back[i].url_adr, back[i].url_fil,
|
||||
(LLint) sz, back[i].r.crange_start,
|
||||
back[i].r.crange_end, back[i].r.crange);
|
||||
url_savename_refname_remove(opt, back[i].url_adr,
|
||||
back[i].url_fil);
|
||||
UNLINK(back[i].url_sav);
|
||||
back[i].status = STATUS_READY;
|
||||
back_set_finished(sback, i);
|
||||
strcpybuff(back[i].r.msg,
|
||||
"Unusable partial content, restarting");
|
||||
} else { // partial not found
|
||||
} else { // Non trouvé??
|
||||
back[i].status = STATUS_READY; // terminé (voir plus loin)
|
||||
back_set_finished(sback, i);
|
||||
strcpybuff(back[i].r.msg, "Can not find partial file");
|
||||
@@ -4005,6 +3930,7 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/*} */
|
||||
|
||||
@@ -146,8 +146,7 @@ typedef enum BackStatusCode {
|
||||
STATUSCODE_NON_FATAL = -5,
|
||||
STATUSCODE_SSL_HANDSHAKE = -6,
|
||||
STATUSCODE_TOO_BIG = -7,
|
||||
STATUSCODE_TEST_OK = -10,
|
||||
STATUSCODE_EXCLUDED = -11 /* aborted: MIME excluded by a -mime: filter */
|
||||
STATUSCODE_TEST_OK = -10
|
||||
} BackStatusCode;
|
||||
|
||||
/** HTTrack status ('status' member of of 'lien_back') **/
|
||||
|
||||
@@ -35,7 +35,6 @@ Please visit our Website: http://www.httrack.com
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <ctype.h>
|
||||
#include <stdint.h> /* uint64_t for the pause mixer (already a hard dep via md5.h) */
|
||||
|
||||
/* File defs */
|
||||
#include "htscore.h"
|
||||
@@ -524,12 +523,9 @@ int httpmirror(char *url1, httrackp * opt) {
|
||||
opt->cookie = &cookie;
|
||||
cookie.max_len = 30000; // max len
|
||||
strcpybuff(cookie.data, "");
|
||||
// Load the mirror's cookies.txt, then the one in the current directory
|
||||
// Charger cookies.txt par défaut ou cookies.txt du miroir
|
||||
cookie_load(opt->cookie, StringBuff(opt->path_log), "cookies.txt");
|
||||
cookie_load(opt->cookie, "", "cookies.txt");
|
||||
// A user-supplied cookie file is merged last so it wins on conflicts
|
||||
if (strnotempty(StringBuff(opt->cookies_file)))
|
||||
cookie_load(opt->cookie, "", StringBuff(opt->cookies_file));
|
||||
} else
|
||||
opt->cookie = NULL;
|
||||
|
||||
@@ -740,39 +736,26 @@ int httpmirror(char *url1, httrackp * opt) {
|
||||
/* OPTIMIZED for fast load */
|
||||
if (StringNotEmpty(opt->filelist)) {
|
||||
char *filelist_buff = NULL;
|
||||
size_t filelist_sz = 0;
|
||||
const char *filelist_err = NULL; /* failure reason, NULL on success */
|
||||
const off_t fs = fsize(StringBuff(opt->filelist));
|
||||
const size_t filelist_sz = off_t_to_size_t(fsize(StringBuff(opt->filelist)));
|
||||
|
||||
if (fs < 0) {
|
||||
/* fsize() hides the cause; redo stat() for a precise errno (#49) */
|
||||
struct stat st;
|
||||
filelist_err = stat(StringBuff(opt->filelist), &st) != 0
|
||||
? strerror(errno)
|
||||
: "not a regular file";
|
||||
} else if ((filelist_sz = off_t_to_size_t(fs)) == (size_t) -1) {
|
||||
filelist_err = "file too large";
|
||||
filelist_sz = 0;
|
||||
} else {
|
||||
if (filelist_sz != (size_t) -1) {
|
||||
FILE *fp = fopen(StringBuff(opt->filelist), "rb");
|
||||
|
||||
if (fp == NULL) {
|
||||
filelist_err = strerror(errno);
|
||||
} else {
|
||||
if (fp) {
|
||||
filelist_buff = malloct(filelist_sz + 1);
|
||||
if (filelist_buff == NULL) {
|
||||
filelist_err = "out of memory";
|
||||
} else if (fread(filelist_buff, 1, filelist_sz, fp) != filelist_sz) {
|
||||
freet(filelist_buff);
|
||||
filelist_err = "read error";
|
||||
} else {
|
||||
filelist_buff[filelist_sz] = '\0';
|
||||
if (filelist_buff) {
|
||||
if (fread(filelist_buff, 1, filelist_sz, fp) != filelist_sz) {
|
||||
freet(filelist_buff);
|
||||
filelist_buff = NULL;
|
||||
} else {
|
||||
*(filelist_buff + filelist_sz) = '\0';
|
||||
}
|
||||
}
|
||||
fclose(fp);
|
||||
}
|
||||
}
|
||||
|
||||
if (filelist_buff != NULL) {
|
||||
if (filelist_buff) {
|
||||
int filelist_ptr = 0;
|
||||
int n = 0;
|
||||
char BIGSTK line[HTS_URLMAXSIZE * 2];
|
||||
@@ -797,8 +780,8 @@ int httpmirror(char *url1, httrackp * opt) {
|
||||
// Free buffer
|
||||
freet(filelist_buff);
|
||||
} else {
|
||||
hts_log_print(opt, LOG_ERROR, "Could not include URL list \"%s\": %s",
|
||||
StringBuff(opt->filelist), filelist_err);
|
||||
hts_log_print(opt, LOG_ERROR, "Could not include URL list: %s",
|
||||
StringBuff(opt->filelist));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3315,21 +3298,6 @@ HTS_INLINE int back_fillmax(struct_back * sback, httrackp * opt,
|
||||
return -1; /* plus de place */
|
||||
}
|
||||
|
||||
/* Seed-derived: stable within a gap, rerolls per launch; a per-call rand()
|
||||
would bias the delay toward min_ms (see header). Jitter, not crypto. */
|
||||
int hts_pause_target_ms(TStamp seed, int min_ms, int max_ms) {
|
||||
uint64_t z = (uint64_t) seed;
|
||||
|
||||
if (max_ms <= min_ms)
|
||||
return min_ms;
|
||||
/* SplitMix64 finalizer: scrambles the low-entropy ms timestamp. */
|
||||
z += 0x9E3779B97F4A7C15ULL;
|
||||
z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
|
||||
z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
|
||||
z ^= z >> 31;
|
||||
return min_ms + (int) (z % (uint64_t) (max_ms - min_ms + 1));
|
||||
}
|
||||
|
||||
int back_pluggable_sockets_strict(struct_back * sback, httrackp * opt) {
|
||||
int n = opt->maxsoc - back_nsoc(sback);
|
||||
|
||||
@@ -3350,18 +3318,6 @@ int back_pluggable_sockets_strict(struct_back * sback, httrackp * opt) {
|
||||
}
|
||||
}
|
||||
|
||||
// #185 randomized inter-file pause: non-blocking, one launch per gap
|
||||
if (n > 0 && opt->pause_max_ms > 0 && HTS_STAT.last_connect > 0) {
|
||||
TStamp opTime =
|
||||
HTS_STAT.last_request ? HTS_STAT.last_request : HTS_STAT.last_connect;
|
||||
TStamp lap = mtime_local() - opTime;
|
||||
|
||||
if (lap < hts_pause_target_ms(opTime, opt->pause_min_ms, opt->pause_max_ms))
|
||||
n = 0;
|
||||
else
|
||||
n = 1;
|
||||
}
|
||||
|
||||
return n;
|
||||
}
|
||||
|
||||
@@ -3770,17 +3726,6 @@ HTSEXT_API int copy_htsopt(const httrackp * from, httrackp * to) {
|
||||
if (StringNotEmpty(from->user_agent))
|
||||
StringCopyS(to->user_agent, from->user_agent);
|
||||
|
||||
if (StringNotEmpty(from->strip_query))
|
||||
StringCopyS(to->strip_query, from->strip_query);
|
||||
|
||||
if (StringNotEmpty(from->cookies_file))
|
||||
StringCopyS(to->cookies_file, from->cookies_file);
|
||||
|
||||
if (from->pause_max_ms > 0) {
|
||||
to->pause_min_ms = from->pause_min_ms;
|
||||
to->pause_max_ms = from->pause_max_ms;
|
||||
}
|
||||
|
||||
if (from->retry > -1)
|
||||
to->retry = from->retry;
|
||||
|
||||
|
||||
@@ -234,12 +234,8 @@ struct hash_struct {
|
||||
coucal adrfil;
|
||||
/* former address+path -> link index (renamed/moved entries) */
|
||||
coucal former_adrfil;
|
||||
/* effective urlhack sub-flags: www.==host / // collapse / query-arg sort */
|
||||
hts_boolean norm_host;
|
||||
hts_boolean norm_slash;
|
||||
hts_boolean norm_query;
|
||||
/* query-strip keys (not owned); set from opt->strip_query at hash_init */
|
||||
const char *strip_query;
|
||||
/* scratch buffers reused across lookups (not reentrant) */
|
||||
int normalized;
|
||||
char normfil[HTS_URLMAXSIZE * 2];
|
||||
char normfil2[HTS_URLMAXSIZE * 2];
|
||||
char catbuff[CATBUFF_SIZE];
|
||||
@@ -368,22 +364,6 @@ int fspc(httrackp * opt, FILE * fp, const char *type);
|
||||
|
||||
char *next_token(char *p, int flag);
|
||||
|
||||
/* Like fil_normalized(), but first drops query keys in STRIP (comma-separated,
|
||||
"*" = all); STRIP NULL/empty behaves exactly like fil_normalized(). */
|
||||
char *fil_normalized_filtered(const char *source, char *dest,
|
||||
const char *strip);
|
||||
|
||||
/* As fil_normalized_filtered(), but DO_SLASH/DO_QUERY gate the // collapse and
|
||||
the query-argument sort independently (the urlhack sub-flags). */
|
||||
char *fil_normalized_filtered_ex(const char *source, char *dest,
|
||||
const char *strip, int do_slash, int do_query);
|
||||
|
||||
/* For URL ADR/FIL, return (in DEST) the comma keylist to strip from the
|
||||
'\n'-separated "[pattern=]keys" RULES (patterns matched on host/path via
|
||||
strjoker, last wins); NULL if none match. Feeds fil_normalized_filtered(). */
|
||||
const char *hts_query_strip_keys(const char *rules, const char *adr,
|
||||
const char *fil, char *dest, size_t destsize);
|
||||
|
||||
/* Read a whole file into a freshly malloc'd, NUL-terminated buffer; the caller
|
||||
owns it and must release it with freet(). Return NULL on missing/unreadable
|
||||
file (readfile_or substitutes defaultdata instead). The byte content is NOT
|
||||
@@ -418,10 +398,6 @@ int back_pluggable_sockets(struct_back * sback, httrackp * opt);
|
||||
|
||||
int back_pluggable_sockets_strict(struct_back * sback, httrackp * opt);
|
||||
|
||||
/* Randomized inter-file pause target in [min_ms,max_ms] (#185), derived from a
|
||||
timestamp seed so it is stable within one gap and rerolls per launch. */
|
||||
int hts_pause_target_ms(TStamp seed, int min_ms, int max_ms);
|
||||
|
||||
/* Schedule more links from the heap into free slots. Returns the number queued,
|
||||
or <=0 if none could be added (no free slot / paused / stopped). */
|
||||
int back_fill(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
|
||||
@@ -1570,30 +1570,6 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
|
||||
com++;
|
||||
}
|
||||
break; // url hack
|
||||
case 'j':
|
||||
opt->no_www_dedup =
|
||||
HTS_TRUE; // --keep-www-prefix: keep www.X != X
|
||||
if (*(com + 1) == '0') {
|
||||
opt->no_www_dedup = HTS_FALSE;
|
||||
com++;
|
||||
}
|
||||
break;
|
||||
case 'o':
|
||||
opt->no_slash_dedup =
|
||||
HTS_TRUE; // --keep-double-slashes: keep //
|
||||
if (*(com + 1) == '0') {
|
||||
opt->no_slash_dedup = HTS_FALSE;
|
||||
com++;
|
||||
}
|
||||
break;
|
||||
case 'y':
|
||||
opt->no_query_dedup =
|
||||
HTS_TRUE; // --keep-query-order: keep ?b&a order
|
||||
if (*(com + 1) == '0') {
|
||||
opt->no_query_dedup = HTS_FALSE;
|
||||
com++;
|
||||
}
|
||||
break;
|
||||
case 'v':
|
||||
opt->verbosedisplay = HTS_VERBOSE_FULL;
|
||||
if (isdigit((unsigned char) *(com + 1))) {
|
||||
@@ -1961,66 +1937,6 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
|
||||
}
|
||||
break;
|
||||
|
||||
case 'g': // strip-query: accumulate "[pattern=]keys" entries
|
||||
if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
|
||||
HTS_PANIC_PRINTF("Option strip-query needs a blank space and "
|
||||
"[host/pattern=]key1,key2,...");
|
||||
printf("Example: --strip-query "
|
||||
"\"www.example.com/*=utm_source,sid\"\n");
|
||||
htsmain_free();
|
||||
return -1;
|
||||
} else {
|
||||
na++;
|
||||
if (StringNotEmpty(opt->strip_query))
|
||||
StringCat(opt->strip_query, "\n");
|
||||
StringCat(opt->strip_query, argv[na]);
|
||||
}
|
||||
break;
|
||||
case 'K': // cookies-file: extra Netscape cookies.txt to preload
|
||||
if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
|
||||
HTS_PANIC_PRINTF(
|
||||
"Option cookies-file needs a blank space and "
|
||||
"a cookies.txt path");
|
||||
printf("Example: --cookies-file \"/home/me/cookies.txt\"\n");
|
||||
htsmain_free();
|
||||
return -1;
|
||||
} else {
|
||||
na++;
|
||||
if (strlen(argv[na]) >= 1024) {
|
||||
HTS_PANIC_PRINTF("Cookie file path too long");
|
||||
htsmain_free();
|
||||
return -1;
|
||||
}
|
||||
StringCopy(opt->cookies_file, argv[na]);
|
||||
}
|
||||
break;
|
||||
case 'G': // pause: randomized inter-file delay MIN[:MAX] seconds
|
||||
if ((na + 1 >= argc) || (argv[na + 1][0] == '-')) {
|
||||
HTS_PANIC_PRINTF("Option pause needs a blank space and a "
|
||||
"delay in seconds (MIN[:MAX])");
|
||||
printf("Example: --pause 5:10\n");
|
||||
htsmain_free();
|
||||
return -1;
|
||||
} else {
|
||||
double pmin = 0, pmax = 0;
|
||||
int nf;
|
||||
|
||||
na++;
|
||||
nf = sscanf(argv[na], "%lf:%lf", &pmin, &pmax);
|
||||
if (nf < 2)
|
||||
pmax = pmin; /* a single value means a fixed delay */
|
||||
/* positive-form bounds: NaN fails every comparison, so this
|
||||
rejects it before the undefined (int)(NaN*1000) cast */
|
||||
if (nf < 1 || !(pmin >= 0 && pmax >= pmin && pmax <= 86400)) {
|
||||
HTS_PANIC_PRINTF("Invalid --pause range (expected "
|
||||
"MIN[:MAX] seconds, 0<=MIN<=MAX<=86400)");
|
||||
htsmain_free();
|
||||
return -1;
|
||||
}
|
||||
opt->pause_min_ms = (int) (pmin * 1000.0);
|
||||
opt->pause_max_ms = (int) (pmax * 1000.0);
|
||||
}
|
||||
break;
|
||||
case 't': /* do not change type (ending) of filenames according to the MIME type */
|
||||
opt->no_type_change = 1;
|
||||
if (*(com+1)=='0') { opt->no_type_change = 0; com++; }
|
||||
|
||||
@@ -76,8 +76,7 @@ int fa_strjoker(int type, char **filters, int nfil, const char *nom, LLint * siz
|
||||
}
|
||||
if (size)
|
||||
sz = *size;
|
||||
/* size unknown (scan time): no size pointer => size tests stay neutral */
|
||||
if (strjoker(nom, filters[i] + filteroffs, size ? &sz : NULL, size_flag)) {
|
||||
if (strjoker(nom, filters[i] + filteroffs, &sz, size_flag)) { // reconnu
|
||||
if (size)
|
||||
if (sz != *size)
|
||||
sizelimit = sz;
|
||||
@@ -193,12 +192,7 @@ HTS_INLINE const char *strjoker(const char *chaine, const char *joker, LLint * s
|
||||
int len = (int) strlen(joker);
|
||||
|
||||
while((joker[i] != RIGHT) && (joker[i]) && (i < len)) {
|
||||
// '\' escapes the next char as a literal member, e.g. *[\[\]]
|
||||
if (joker[i] == '\\' && joker[i + 1] != '\0') {
|
||||
i++;
|
||||
pass[(int) (unsigned char) joker[i]] = 1;
|
||||
i++;
|
||||
} else if ((joker[i] == '<') || (joker[i] == '>')) { // *[<10]
|
||||
if ((joker[i] == '<') || (joker[i] == '>')) { // *[<10]
|
||||
int lsize = 0;
|
||||
int lverdict;
|
||||
|
||||
@@ -226,9 +220,7 @@ HTS_INLINE const char *strjoker(const char *chaine, const char *joker, LLint * s
|
||||
while(isdigit((unsigned char) joker[i]))
|
||||
i++;
|
||||
}
|
||||
} else if (joker[i + 1] == '-' && joker[i + 2] != '\0') {
|
||||
// range *[A-Z]; the '\0' guard rejects a truncated *[a- (else
|
||||
// i+=3 overshoots the NUL)
|
||||
} else if (joker[i + 1] == '-') { // 2 car, ex: *[A-Z]
|
||||
if ((int) (unsigned char) joker[i + 2] >
|
||||
(int) (unsigned char) joker[i]) {
|
||||
int j;
|
||||
@@ -240,7 +232,10 @@ HTS_INLINE const char *strjoker(const char *chaine, const char *joker, LLint * s
|
||||
}
|
||||
// else err=1;
|
||||
i += 3;
|
||||
} else { // 1 car, ex: *[ ]
|
||||
} else { // 1 car, ex: *[ ]
|
||||
if (joker[i + 2] == '\\' && joker[i + 3] != 0) { // escaped char, such as *[\[] or *[\]]
|
||||
i++;
|
||||
}
|
||||
pass[(int) (unsigned char) joker[i]] = 1;
|
||||
i++;
|
||||
}
|
||||
|
||||
@@ -43,8 +43,8 @@ Please visit our Website: http://www.httrack.com
|
||||
configure.ac, decoupled from these). VERSION is the display form, VERSIONID
|
||||
the dotted numeric form, AFF_VERSION the short form shown in footers,
|
||||
LIB_VERSION the data/cache format generation. */
|
||||
#define HTTRACK_VERSION "3.49-10"
|
||||
#define HTTRACK_VERSIONID "3.49.10"
|
||||
#define HTTRACK_VERSION "3.49-9"
|
||||
#define HTTRACK_VERSIONID "3.49.9"
|
||||
#define HTTRACK_AFF_VERSION "3.x"
|
||||
#define HTTRACK_LIB_VERSION "2.0"
|
||||
|
||||
|
||||
@@ -106,10 +106,10 @@ static coucal_hashkeys key_adrfil_hashes_generic(void *arg,
|
||||
const lien_url*const lien = (const lien_url*) value;
|
||||
const char *const adr = !former ? lien->adr : lien->former_adr;
|
||||
const char *const fil = !former ? lien->fil : lien->former_fil;
|
||||
const char *const adr_norm =
|
||||
adr != NULL ? (hash->norm_host ? jump_normalized_const(adr)
|
||||
: jump_identification_const(adr))
|
||||
: NULL;
|
||||
const char *const adr_norm = adr != NULL ?
|
||||
( hash->normalized ? jump_normalized_const(adr)
|
||||
: jump_identification_const(adr) )
|
||||
: NULL;
|
||||
|
||||
// copy address
|
||||
assertf(adr_norm != NULL);
|
||||
@@ -117,18 +117,10 @@ static coucal_hashkeys key_adrfil_hashes_generic(void *arg,
|
||||
|
||||
// copy link
|
||||
assertf(fil != NULL);
|
||||
{
|
||||
/* resolve the per-URL strip keys; strip applies even when urlhack is off */
|
||||
char BIGSTK keybuf[HTS_URLMAXSIZE];
|
||||
const char *const keys = hts_query_strip_keys(hash->strip_query, adr, fil,
|
||||
keybuf, sizeof(keybuf));
|
||||
|
||||
if (hash->norm_slash || hash->norm_query || keys != NULL) {
|
||||
fil_normalized_filtered_ex(fil, &hash->normfil[strlen(hash->normfil)],
|
||||
keys, hash->norm_slash, hash->norm_query);
|
||||
} else {
|
||||
strcpy(&hash->normfil[strlen(hash->normfil)], fil);
|
||||
}
|
||||
if (hash->normalized) {
|
||||
fil_normalized(fil, &hash->normfil[strlen(hash->normfil)]);
|
||||
} else {
|
||||
strcpy(&hash->normfil[strlen(hash->normfil)], fil);
|
||||
}
|
||||
|
||||
// hash
|
||||
@@ -140,7 +132,8 @@ static int key_adrfil_equals_generic(void *arg,
|
||||
coucal_key_const a_,
|
||||
coucal_key_const b_,
|
||||
const int former) {
|
||||
hash_struct *const hash = (hash_struct *) arg;
|
||||
hash_struct *const hash = (hash_struct*) arg;
|
||||
const int normalized = hash->normalized;
|
||||
const lien_url*const a = (const lien_url*) a_;
|
||||
const lien_url*const b = (const lien_url*) b_;
|
||||
const char *const a_adr = !former ? a->adr : a->former_adr;
|
||||
@@ -157,10 +150,10 @@ static int key_adrfil_equals_generic(void *arg,
|
||||
assertf(b_fil != NULL);
|
||||
|
||||
// skip scheme and authentication to the domain (possibly without www.)
|
||||
ja = hash->norm_host ? jump_normalized_const(a_adr)
|
||||
: jump_identification_const(a_adr);
|
||||
jb = hash->norm_host ? jump_normalized_const(b_adr)
|
||||
: jump_identification_const(b_adr);
|
||||
ja = normalized
|
||||
? jump_normalized_const(a_adr) : jump_identification_const(a_adr);
|
||||
jb = normalized
|
||||
? jump_normalized_const(b_adr) : jump_identification_const(b_adr);
|
||||
assertf(ja != NULL);
|
||||
assertf(jb != NULL);
|
||||
if (strcasecmp(ja, jb) != 0) {
|
||||
@@ -168,23 +161,12 @@ static int key_adrfil_equals_generic(void *arg,
|
||||
}
|
||||
|
||||
// now compare pathes
|
||||
{
|
||||
char BIGSTK ka[HTS_URLMAXSIZE], kb[HTS_URLMAXSIZE];
|
||||
const char *const keysa =
|
||||
hts_query_strip_keys(hash->strip_query, a_adr, a_fil, ka, sizeof(ka));
|
||||
const char *const keysb =
|
||||
hts_query_strip_keys(hash->strip_query, b_adr, b_fil, kb, sizeof(kb));
|
||||
|
||||
if (hash->norm_slash || hash->norm_query || keysa != NULL ||
|
||||
keysb != NULL) {
|
||||
fil_normalized_filtered_ex(a_fil, hash->normfil, keysa, hash->norm_slash,
|
||||
hash->norm_query);
|
||||
fil_normalized_filtered_ex(b_fil, hash->normfil2, keysb, hash->norm_slash,
|
||||
hash->norm_query);
|
||||
return strcmp(hash->normfil, hash->normfil2) == 0;
|
||||
} else {
|
||||
return strcmp(a_fil, b_fil) == 0;
|
||||
}
|
||||
if (normalized) {
|
||||
fil_normalized(a_fil, hash->normfil);
|
||||
fil_normalized(b_fil, hash->normfil2);
|
||||
return strcmp(hash->normfil, hash->normfil2) == 0;
|
||||
} else {
|
||||
return strcmp(a_fil, b_fil) == 0;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -240,17 +222,11 @@ static int key_former_adrfil_equals(void *arg,
|
||||
return key_adrfil_equals_generic(arg, a, b, 1);
|
||||
}
|
||||
|
||||
void hash_init(httrackp *opt, hash_struct *hash, hts_boolean normalized) {
|
||||
void hash_init(httrackp *opt, hash_struct * hash, int normalized) {
|
||||
hash->sav = coucal_new(0);
|
||||
hash->adrfil = coucal_new(0);
|
||||
hash->former_adrfil = coucal_new(0);
|
||||
/* urlhack is the umbrella; per-feature negatives opt out of each part */
|
||||
hash->norm_host = normalized && !opt->no_www_dedup;
|
||||
hash->norm_slash = normalized && !opt->no_slash_dedup;
|
||||
hash->norm_query = normalized && !opt->no_query_dedup;
|
||||
/* snapshot the query-strip list (not owned; valid for the hash lifetime) */
|
||||
hash->strip_query =
|
||||
StringNotEmpty(opt->strip_query) ? StringBuff(opt->strip_query) : NULL;
|
||||
hash->normalized = normalized;
|
||||
|
||||
hts_set_hash_handler(hash->sav, opt);
|
||||
hts_set_hash_handler(hash->adrfil, opt);
|
||||
@@ -306,26 +282,6 @@ void hash_free(hash_struct *hash) {
|
||||
}
|
||||
}
|
||||
|
||||
/* Test helper: do the two URLs dedupe to the same key under opt's urlhack
|
||||
flags? Exercises the live hash compare (norm_host/slash/query resolution). */
|
||||
hts_boolean hash_url_equals(httrackp *opt, const char *adra, const char *fila,
|
||||
const char *adrb, const char *filb) {
|
||||
hash_struct hash;
|
||||
lien_url la, lb;
|
||||
hts_boolean eq;
|
||||
|
||||
memset(&la, 0, sizeof(la));
|
||||
memset(&lb, 0, sizeof(lb));
|
||||
la.adr = key_duphandler(NULL, adra);
|
||||
la.fil = key_duphandler(NULL, fila);
|
||||
lb.adr = key_duphandler(NULL, adrb);
|
||||
lb.fil = key_duphandler(NULL, filb);
|
||||
hash_init(opt, &hash, opt->urlhack);
|
||||
eq = key_adrfil_equals(&hash, &la, &lb);
|
||||
hash_free(&hash);
|
||||
return eq;
|
||||
}
|
||||
|
||||
// retour: position ou -1 si non trouvé
|
||||
int hash_read(const hash_struct * hash, const char *nom1, const char *nom2,
|
||||
hash_struct_type type) {
|
||||
|
||||
@@ -51,12 +51,8 @@ typedef enum hash_struct_type {
|
||||
} hash_struct_type;
|
||||
|
||||
// tables de hachage
|
||||
void hash_init(httrackp *opt, hash_struct *hash, hts_boolean normalized);
|
||||
void hash_init(httrackp *opt, hash_struct *hash, int normalized);
|
||||
void hash_free(hash_struct *hash);
|
||||
/* Test helper: HTS_TRUE if the two URLs dedupe together under opt's urlhack
|
||||
flags. */
|
||||
hts_boolean hash_url_equals(httrackp *opt, const char *adra, const char *fila,
|
||||
const char *adrb, const char *filb);
|
||||
int hash_read(const hash_struct * hash, const char *nom1, const char *nom2,
|
||||
hash_struct_type type);
|
||||
void hash_write(hash_struct * hash, size_t lpos);
|
||||
|
||||
@@ -521,7 +521,6 @@ void help(const char *app, int more) {
|
||||
infomsg(" EN maximum mirror time in seconds (60=1 minute, 3600=1 hour)");
|
||||
infomsg(" AN maximum transfer rate in bytes/seconds (1000=1KB/s max)");
|
||||
infomsg(" %cN maximum number of connections/seconds (*%c10)");
|
||||
infomsg(" %G random pause of MIN[:MAX] seconds between files (e.g. %G5:10)");
|
||||
infomsg
|
||||
(" GN pause transfer if N bytes reached, and wait until lock file is deleted");
|
||||
infomsg("");
|
||||
@@ -564,7 +563,6 @@ void help(const char *app, int more) {
|
||||
(" %x do not include any password for external password protected websites (%x0 include)");
|
||||
infomsg
|
||||
(" %q *include query string for local files (useless, for information purpose only) (%q0 don't include)");
|
||||
infomsg(" %g strip query keys for dedup ([host/pattern=]key1,key2,...)");
|
||||
infomsg
|
||||
(" o *generate output html file in case of error (404..) (o0 don't generate)");
|
||||
infomsg(" X *purge old files after update (X0 keep delete)");
|
||||
@@ -573,7 +571,6 @@ void help(const char *app, int more) {
|
||||
infomsg("");
|
||||
infomsg("Spider options:");
|
||||
infomsg(" bN accept cookies in cookies.txt (0=do not accept,* 1=accept)");
|
||||
infomsg(" %K load extra cookies from a Netscape cookies.txt");
|
||||
infomsg
|
||||
(" u check document type if unknown (cgi,asp..) (u0 don't check, * u1 check but /, u2 check always)");
|
||||
infomsg
|
||||
@@ -590,9 +587,6 @@ void help(const char *app, int more) {
|
||||
(" %s update hacks: various hacks to limit re-transfers when updating (identical size, bogus response..)");
|
||||
infomsg
|
||||
(" %u url hacks: various hacks to limit duplicate URLs (strip //, www.foo.com==foo.com..)");
|
||||
infomsg(" opt out of one url-hack part: --keep-www-prefix "
|
||||
"(www.foo.com<>foo.com), --keep-double-slashes (//), "
|
||||
"--keep-query-order (?b&a)");
|
||||
infomsg
|
||||
(" %A assume that a type (cgi,asp..) is always linked with a mime type (-%A php3,cgi=text/html;dat,bin=application/x-zip)");
|
||||
infomsg(" shortcut: '--assume standard' is equivalent to -%A "
|
||||
|
||||
167
src/htslib.c
167
src/htslib.c
@@ -3610,10 +3610,7 @@ static int sortNormFnc(const void *a_, const void *b_) {
|
||||
return strcmp(*a + 1, *b + 1);
|
||||
}
|
||||
|
||||
/* Path normalizer core: optionally collapse redundant '//' (DO_SLASH) and/or
|
||||
sort query arguments (DO_QUERY) so equivalent URLs dedupe. */
|
||||
static char *fil_normalized_ex(const char *source, char *dest, int do_slash,
|
||||
int do_query) {
|
||||
HTSEXT_API char *fil_normalized(const char *source, char *dest) {
|
||||
char lastc = 0;
|
||||
int gotquery = 0;
|
||||
int ampargs = 0;
|
||||
@@ -3623,8 +3620,8 @@ static char *fil_normalized_ex(const char *source, char *dest, int do_slash,
|
||||
for(i = j = 0; source[i] != '\0'; i++) {
|
||||
if (!gotquery && source[i] == '?')
|
||||
gotquery = ampargs = 1;
|
||||
if (do_slash && !gotquery && lastc == '/' && source[i] == '/') {
|
||||
// foo//bar -> foo/bar
|
||||
if ((!gotquery && lastc == '/' && source[i] == '/') // foo//bar -> foo/bar
|
||||
) {
|
||||
} else {
|
||||
if (gotquery && source[i] == '&') {
|
||||
ampargs++;
|
||||
@@ -3636,7 +3633,7 @@ static char *fil_normalized_ex(const char *source, char *dest, int do_slash,
|
||||
dest[j++] = '\0';
|
||||
|
||||
/* Sort arguments (&foo=1&bar=2 == &bar=2&foo=1) */
|
||||
if (do_query && ampargs > 1) {
|
||||
if (ampargs > 1) {
|
||||
char **amps = malloct(ampargs * sizeof(char *));
|
||||
char *copyBuff = NULL;
|
||||
size_t qLen = 0;
|
||||
@@ -3684,153 +3681,6 @@ static char *fil_normalized_ex(const char *source, char *dest, int do_slash,
|
||||
return dest;
|
||||
}
|
||||
|
||||
HTSEXT_API char *fil_normalized(const char *source, char *dest) {
|
||||
return fil_normalized_ex(source, dest, 1, 1);
|
||||
}
|
||||
|
||||
/* Is query key ARG[0..keylen) in the comma-separated STRIP list? "*" = all;
|
||||
case-sensitive, space-trimmed tokens. */
|
||||
static int hts_query_key_stripped(const char *arg, size_t keylen,
|
||||
const char *strip) {
|
||||
const char *p = strip;
|
||||
|
||||
while (*p != '\0') {
|
||||
const char *start = p;
|
||||
size_t toklen;
|
||||
|
||||
while (*p != '\0' && *p != ',')
|
||||
p++;
|
||||
toklen = (size_t) (p - start);
|
||||
while (toklen > 0 && *start == ' ') {
|
||||
start++;
|
||||
toklen--;
|
||||
}
|
||||
while (toklen > 0 && start[toklen - 1] == ' ')
|
||||
toklen--;
|
||||
if (toklen == 1 && start[0] == '*')
|
||||
return 1;
|
||||
if (toklen == keylen && strncmp(start, arg, keylen) == 0)
|
||||
return 1;
|
||||
if (*p == ',')
|
||||
p++;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* see htscore.h */
|
||||
char *fil_normalized_filtered_ex(const char *source, char *dest,
|
||||
const char *strip, int do_slash,
|
||||
int do_query) {
|
||||
const char *query;
|
||||
char BIGSTK tmp[HTS_URLMAXSIZE * 2];
|
||||
htsbuff cb;
|
||||
int wrote = 0;
|
||||
|
||||
/* No strip list, or no query: plain normalization. */
|
||||
if (strip == NULL || *strip == '\0' ||
|
||||
(query = strchr(source, '?')) == NULL) {
|
||||
return fil_normalized_ex(source, dest, do_slash, do_query);
|
||||
}
|
||||
|
||||
/* Copy the path, re-emit kept query args, let fil_normalized() sort. Walk
|
||||
every field incl. empty/trailing ("a&","?&&") so the result is a fixpoint
|
||||
(the read re-normalizes it; a dropped empty arg would miss dedup). */
|
||||
cb = htsbuff_ptr(tmp, sizeof(tmp));
|
||||
htsbuff_catn(&cb, source, (size_t) (query - source));
|
||||
for (query++;;) {
|
||||
const char *const arg = query;
|
||||
const char *eq = NULL;
|
||||
size_t keylen, arglen;
|
||||
|
||||
while (*query != '\0' && *query != '&') {
|
||||
if (eq == NULL && *query == '=')
|
||||
eq = query;
|
||||
query++;
|
||||
}
|
||||
arglen = (size_t) (query - arg);
|
||||
keylen = eq != NULL ? (size_t) (eq - arg) : arglen;
|
||||
if (!hts_query_key_stripped(arg, keylen, strip)) {
|
||||
htsbuff_catc(&cb, wrote ? '&' : '?');
|
||||
htsbuff_catn(&cb, arg, arglen);
|
||||
wrote = 1;
|
||||
}
|
||||
if (*query == '\0')
|
||||
break;
|
||||
query++;
|
||||
}
|
||||
return fil_normalized_ex(tmp, dest, do_slash, do_query);
|
||||
}
|
||||
|
||||
/* see htscore.h */
|
||||
char *fil_normalized_filtered(const char *source, char *dest,
|
||||
const char *strip) {
|
||||
return fil_normalized_filtered_ex(source, dest, strip, 1, 1);
|
||||
}
|
||||
|
||||
/* see htscore.h */
|
||||
const char *hts_query_strip_keys(const char *rules, const char *adr,
|
||||
const char *fil, char *dest, size_t destsize) {
|
||||
const char *p, *q;
|
||||
const char *result = NULL;
|
||||
char BIGSTK url[HTS_URLMAXSIZE * 2];
|
||||
|
||||
if (rules == NULL || *rules == '\0' || destsize == 0)
|
||||
return NULL;
|
||||
|
||||
/* Match string = normalized host/path, query removed. jump_normalized_const
|
||||
collapses www+scheme/auth so read and write (double-normalized) agree;
|
||||
query excluded keeps the decision on host/path only. */
|
||||
url[0] = '\0';
|
||||
strcatbuff(url, jump_normalized_const(adr));
|
||||
if (fil[0] != '/')
|
||||
strcatbuff(url, "/");
|
||||
q = strchr(fil, '?');
|
||||
if (q != NULL)
|
||||
strncatbuff(url, fil, (int) (q - fil));
|
||||
else
|
||||
strcatbuff(url, fil);
|
||||
|
||||
/* Walk the '\n' entries; last match wins (like the +/- filter eval). Each is
|
||||
"pattern=keys"; no '=' is the bare form, pattern "*". */
|
||||
for (p = rules; *p != '\0';) {
|
||||
const char *const line = p;
|
||||
const char *eol, *eq, *keys;
|
||||
char BIGSTK pat[HTS_URLMAXSIZE * 2];
|
||||
|
||||
while (*p != '\0' && *p != '\n')
|
||||
p++;
|
||||
eol = p;
|
||||
if (*p == '\n')
|
||||
p++;
|
||||
if (eol == line)
|
||||
continue;
|
||||
eq = memchr(line, '=', (size_t) (eol - line));
|
||||
if (eq != NULL) {
|
||||
size_t patlen = (size_t) (eq - line);
|
||||
|
||||
if (patlen >= sizeof(pat))
|
||||
patlen = sizeof(pat) - 1;
|
||||
memcpy(pat, line, patlen);
|
||||
pat[patlen] = '\0';
|
||||
keys = eq + 1;
|
||||
} else {
|
||||
pat[0] = '*';
|
||||
pat[1] = '\0';
|
||||
keys = line;
|
||||
}
|
||||
if (strjoker(url, pat, NULL, NULL) != NULL) {
|
||||
size_t klen = (size_t) (eol - keys);
|
||||
|
||||
if (klen >= destsize)
|
||||
klen = destsize - 1;
|
||||
memcpy(dest, keys, klen);
|
||||
dest[klen] = '\0';
|
||||
result = dest;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
#define endwith(a) ( (len >= (sizeof(a)-1)) ? ( strncmp(dest, a+len-(sizeof(a)-1), sizeof(a)-1) == 0 ) : 0 );
|
||||
HTSEXT_API char *adr_normalized_sized(const char *source, char *dest,
|
||||
size_t destsize) {
|
||||
@@ -6040,14 +5890,7 @@ HTSEXT_API httrackp *hts_create_opt(void) {
|
||||
opt->verbosedisplay = HTS_VERBOSE_NONE; // no text animation
|
||||
opt->sizehack = HTS_FALSE;
|
||||
opt->urlhack = HTS_TRUE;
|
||||
opt->no_www_dedup = HTS_FALSE;
|
||||
opt->no_slash_dedup = HTS_FALSE;
|
||||
opt->no_query_dedup = HTS_FALSE;
|
||||
StringCopy(opt->footer, HTS_DEFAULT_FOOTER);
|
||||
StringCopy(opt->strip_query, "");
|
||||
StringCopy(opt->cookies_file, "");
|
||||
opt->pause_min_ms = 0;
|
||||
opt->pause_max_ms = 0;
|
||||
opt->ftp_proxy = HTS_TRUE;
|
||||
opt->convert_utf8 = HTS_TRUE;
|
||||
StringCopy(opt->filelist, "");
|
||||
@@ -6192,8 +6035,6 @@ HTSEXT_API void hts_free_opt(httrackp * opt) {
|
||||
StringFree(opt->urllist);
|
||||
StringFree(opt->footer);
|
||||
StringFree(opt->mod_blacklist);
|
||||
StringFree(opt->strip_query);
|
||||
StringFree(opt->cookies_file);
|
||||
|
||||
StringFree(opt->path_html);
|
||||
StringFree(opt->path_html_utf8);
|
||||
|
||||
@@ -198,13 +198,6 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
// copy of fil, used for lookups (see urlhack)
|
||||
const char *normadr = adr;
|
||||
const char *normfil = fil_complete;
|
||||
/* query keys to strip for this URL (NULL = none); decoupled from urlhack */
|
||||
char BIGSTK stripkeys[HTS_URLMAXSIZE];
|
||||
const char *const strip =
|
||||
StringNotEmpty(opt->strip_query)
|
||||
? hts_query_strip_keys(StringBuff(opt->strip_query), adr,
|
||||
fil_complete, stripkeys, sizeof(stripkeys))
|
||||
: NULL;
|
||||
const char *const print_adr = jump_protocol_const(adr);
|
||||
const char *start_pos = NULL, *nom_pos = NULL, *dot_pos = NULL; // Position nom et point
|
||||
|
||||
@@ -237,13 +230,9 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
// www-42.foo.com -> foo.com
|
||||
// foo.com/bar//foobar -> foo.com/bar/foobar
|
||||
if (opt->urlhack) {
|
||||
// dedup-lookup key; honor the per-feature negatives like htshash.c so
|
||||
// distinct URLs keep distinct savenames (else keep normadr = adr)
|
||||
if (!opt->no_www_dedup)
|
||||
normadr = adr_normalized_sized(adr, normadr_, sizeof(normadr_));
|
||||
normfil =
|
||||
fil_normalized_filtered_ex(fil_complete, normfil_, strip,
|
||||
!opt->no_slash_dedup, !opt->no_query_dedup);
|
||||
// copy of adr (without protocol), used for lookups (see urlhack)
|
||||
normadr = adr_normalized_sized(adr, normadr_, sizeof(normadr_));
|
||||
normfil = fil_normalized(fil_complete, normfil_);
|
||||
} else {
|
||||
if (link_has_authority(adr_complete)) { // https or other protocols : in "http/" subfolder
|
||||
char *pos = strchr(adr_complete, ':');
|
||||
@@ -256,11 +245,6 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
normadr = normadr_;
|
||||
}
|
||||
}
|
||||
// strip still applies with urlhack off (host left untouched); no // or
|
||||
// query-sort here, to match the hash key (norm_slash/norm_query are 0 when
|
||||
// urlhack is off) so a URL is looked up under the key it was stored with
|
||||
if (strip != NULL)
|
||||
normfil = fil_normalized_filtered_ex(fil_complete, normfil_, strip, 0, 0);
|
||||
}
|
||||
|
||||
// à afficher sans ftp://
|
||||
|
||||
10
src/htsopt.h
10
src/htsopt.h
@@ -529,16 +529,6 @@ struct httrackp {
|
||||
htslibhandles libHandles; /**< loaded external module handles */
|
||||
//
|
||||
htsoptstate state; /**< embedded live engine state */
|
||||
String strip_query; /**< query keys to drop when deduping URLs (-strip-query);
|
||||
appended at the tail to keep field offsets stable */
|
||||
hts_boolean
|
||||
no_www_dedup; /**< with urlhack, keep www.host distinct from host */
|
||||
hts_boolean no_slash_dedup; /**< with urlhack, keep redundant // in paths */
|
||||
hts_boolean no_query_dedup; /**< with urlhack, keep query-argument order */
|
||||
String cookies_file; /**< extra Netscape cookies.txt to preload
|
||||
(--cookies-file) */
|
||||
int pause_min_ms; /**< inter-file pause lower bound, ms (0=off, #185) */
|
||||
int pause_max_ms; /**< inter-file pause upper bound, ms */
|
||||
};
|
||||
|
||||
/* Running statistics for a mirror. */
|
||||
|
||||
@@ -302,14 +302,6 @@ static HTS_INLINE char html_prevc(const char *html, const char *start) {
|
||||
return html > start ? html[-1] : ' ';
|
||||
}
|
||||
|
||||
/* Drop a redirect Location's #fragment: a UA anchor, never part of the fetched
|
||||
* resource (#204). */
|
||||
static void url_drop_fragment(char *const url) {
|
||||
char *const frag = strchr(url, '#');
|
||||
if (frag != NULL)
|
||||
*frag = '\0';
|
||||
}
|
||||
|
||||
/* True if [s, s+len) is exactly an HTTP method token (XHR.open's first
|
||||
argument is a method, not a URL: #218). Case-insensitive. */
|
||||
static int is_http_method(const char *s, size_t len) {
|
||||
@@ -3604,35 +3596,22 @@ int hts_mirror_check_moved(htsmoduleStruct * str,
|
||||
//
|
||||
|
||||
strcpybuff(mov_url, r->location);
|
||||
url_drop_fragment(mov_url);
|
||||
|
||||
// url qque -> adresse+fichier
|
||||
if ((reponse =
|
||||
ident_url_relatif(mov_url, urladr(), urlfil(), moved)) >= 0) {
|
||||
int set_prio_to = 0; // pas de priotité fixéd par wizard
|
||||
|
||||
// check whether URLHack is harmless or not (per the effective
|
||||
// sub-flags)
|
||||
if (opt->urlhack && (!opt->no_www_dedup || !opt->no_slash_dedup ||
|
||||
!opt->no_query_dedup)) {
|
||||
const int norm_host = !opt->no_www_dedup;
|
||||
const int norm_slash = !opt->no_slash_dedup;
|
||||
const int norm_query = !opt->no_query_dedup;
|
||||
// check whether URLHack is harmless or not
|
||||
if (opt->urlhack) {
|
||||
char BIGSTK n_adr[HTS_URLMAXSIZE * 2], n_fil[HTS_URLMAXSIZE * 2];
|
||||
char BIGSTK pn_adr[HTS_URLMAXSIZE * 2], pn_fil[HTS_URLMAXSIZE * 2];
|
||||
|
||||
strlcpybuff(n_adr,
|
||||
norm_host ? jump_normalized_const(moved->adr)
|
||||
: jump_identification_const(moved->adr),
|
||||
sizeof(n_adr));
|
||||
strlcpybuff(pn_adr,
|
||||
norm_host ? jump_normalized_const(urladr())
|
||||
: jump_identification_const(urladr()),
|
||||
sizeof(pn_adr));
|
||||
fil_normalized_filtered_ex(moved->fil, n_fil, NULL, norm_slash,
|
||||
norm_query);
|
||||
fil_normalized_filtered_ex(urlfil(), pn_fil, NULL, norm_slash,
|
||||
norm_query);
|
||||
n_adr[0] = n_fil[0] = '\0';
|
||||
(void) adr_normalized_sized(moved->adr, n_adr, sizeof(n_adr));
|
||||
(void) fil_normalized(moved->fil, n_fil);
|
||||
(void) adr_normalized_sized(urladr(), pn_adr, sizeof(pn_adr));
|
||||
(void) fil_normalized(urlfil(), pn_fil);
|
||||
if (strcasecmp(n_adr, pn_adr) == 0
|
||||
&& strcasecmp(n_fil, pn_fil) == 0) {
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
@@ -4812,7 +4791,6 @@ int hts_wait_delayed(htsmoduleStruct * str, lien_adrfilsave *afs,
|
||||
|
||||
mov_url[0] = '\0';
|
||||
strcpybuff(mov_url, back[b].r.location); // copier URL
|
||||
url_drop_fragment(mov_url);
|
||||
|
||||
/* Remove (temporarily created) file if it was created */
|
||||
UNLINK(fconv(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), back[b].url_sav));
|
||||
|
||||
@@ -512,47 +512,15 @@ static int string_safety_selftests(void) {
|
||||
/* ------------------------------------------------------------ */
|
||||
|
||||
static int st_filter(httrackp *opt, int argc, char **argv) {
|
||||
char *str, *pat;
|
||||
int matched;
|
||||
|
||||
(void) opt;
|
||||
if (argc < 2) {
|
||||
fprintf(stderr, "filter: needs a filter pattern and a string\n");
|
||||
return 1;
|
||||
}
|
||||
/* exact-size heap copies so a sanitizer traps any over-read of the pattern */
|
||||
str = strdupt(argv[1]);
|
||||
pat = strdupt(argv[0]);
|
||||
matched = strjoker(str, pat, NULL, NULL) != NULL;
|
||||
printf("%s does %s %s\n", argv[1], matched ? "match" : "NOT match", argv[0]);
|
||||
freet(str);
|
||||
freet(pat);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Size-aware filter verdict via fa_strjoker: a negative <size> means the size
|
||||
is still unknown (scan time), so a size rule like -*.jpg*[<10] must stay
|
||||
neutral. */
|
||||
static int st_filtersize(httrackp *opt, int argc, char **argv) {
|
||||
LLint sz;
|
||||
int size_flag = 0, verdict, known;
|
||||
|
||||
(void) opt;
|
||||
if (argc < 3) {
|
||||
fprintf(stderr, "filtersize: needs <size> <string> <filter> [filter...]\n");
|
||||
return 1;
|
||||
}
|
||||
known = (argv[0][0] != '-'); /* "-1"/"-" => size unknown */
|
||||
sz = -1;
|
||||
if (known)
|
||||
sscanf(argv[0], LLintP, &sz);
|
||||
verdict = fa_strjoker(0, &argv[2], argc - 2, argv[1], known ? &sz : NULL,
|
||||
known ? &size_flag : NULL, NULL);
|
||||
printf("verdict=%s size_flag=%d\n",
|
||||
verdict > 0 ? "allowed"
|
||||
: verdict < 0 ? "forbidden"
|
||||
: "unknown",
|
||||
size_flag);
|
||||
if (strjoker(argv[1], argv[0], NULL, NULL))
|
||||
printf("%s does match %s\n", argv[1], argv[0]);
|
||||
else
|
||||
printf("%s does NOT match %s\n", argv[1], argv[0]);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -905,71 +873,12 @@ static int st_copyopt(httrackp *opt, int argc, char **argv) {
|
||||
if (to->parseall != HTS_TRUE)
|
||||
err = 1;
|
||||
|
||||
/* String field: a non-empty source deep-copies across, an empty source
|
||||
leaves the target intact (StringNotEmpty guard). Covers the exported
|
||||
copy_htsopt String path that no crawl test reaches. */
|
||||
StringCopy(from->cookies_file, "/tmp/jar.txt");
|
||||
StringCopy(to->cookies_file, "");
|
||||
copy_htsopt(from, to);
|
||||
if (strcmp(StringBuff(to->cookies_file), "/tmp/jar.txt") != 0)
|
||||
err = 1;
|
||||
StringCopy(from->cookies_file, "");
|
||||
copy_htsopt(from, to);
|
||||
if (strcmp(StringBuff(to->cookies_file), "/tmp/jar.txt") != 0)
|
||||
err = 1;
|
||||
|
||||
/* #185 pause pair: copied when enabled (max>0), the 0 sentinel skips */
|
||||
from->pause_min_ms = 5000;
|
||||
from->pause_max_ms = 10000;
|
||||
to->pause_min_ms = to->pause_max_ms = 0;
|
||||
copy_htsopt(from, to);
|
||||
if (to->pause_min_ms != 5000 || to->pause_max_ms != 10000)
|
||||
err = 1;
|
||||
from->pause_min_ms = from->pause_max_ms = 0;
|
||||
copy_htsopt(from, to);
|
||||
if (to->pause_min_ms != 5000 || to->pause_max_ms != 10000)
|
||||
err = 1;
|
||||
|
||||
hts_free_opt(from);
|
||||
hts_free_opt(to);
|
||||
printf("copy-htsopt: %s\n", err ? "FAIL" : "OK");
|
||||
return err;
|
||||
}
|
||||
|
||||
static int st_pause(httrackp *opt, int argc, char **argv) {
|
||||
int err = 0, i, seen_low = 0, seen_high = 0;
|
||||
|
||||
(void) opt;
|
||||
(void) argc;
|
||||
(void) argv;
|
||||
/* Consecutive-ms seeds (production shape: launch timestamps a few ms apart)
|
||||
must stay in range and spread, not collapse to a bound -- worst case for a
|
||||
weak low-bit mixer. */
|
||||
for (i = 0; i < 10000; i++) {
|
||||
int t = hts_pause_target_ms((TStamp) (1719500000000LL + i), 5000, 10000);
|
||||
|
||||
if (t < 5000 || t > 10000)
|
||||
err = 1;
|
||||
seen_low |= (t < 6000);
|
||||
seen_high |= (t > 9000);
|
||||
}
|
||||
if (!seen_low || !seen_high)
|
||||
err = 1;
|
||||
if (hts_pause_target_ms(12345, 8000, 8000) != 8000) /* equal bounds = fixed */
|
||||
err = 1;
|
||||
/* deterministic: a seed yields the same target even after an intervening call
|
||||
with another seed (no global PRNG state to perturb it) */
|
||||
{
|
||||
int a = hts_pause_target_ms(99, 5000, 10000);
|
||||
|
||||
(void) hts_pause_target_ms(54321, 5000, 10000);
|
||||
if (hts_pause_target_ms(99, 5000, 10000) != a)
|
||||
err = 1;
|
||||
}
|
||||
printf("pause: %s\n", err ? "FAIL" : "OK");
|
||||
return err;
|
||||
}
|
||||
|
||||
static int st_relative(httrackp *opt, int argc, char **argv) {
|
||||
char s[HTS_URLMAXSIZE * 2];
|
||||
|
||||
@@ -1117,173 +1026,6 @@ static int st_cookies(httrackp *opt, int argc, char **argv) {
|
||||
return err;
|
||||
}
|
||||
|
||||
/* --strip-query: resolver + fil_normalized_filtered, end to end. */
|
||||
static int st_stripquery(httrackp *opt, int argc, char **argv) {
|
||||
char dest[1024], keys[256], ref[1024];
|
||||
const char *k;
|
||||
|
||||
(void) opt;
|
||||
(void) argc;
|
||||
(void) argv;
|
||||
|
||||
/* empty rules == plain fil_normalized */
|
||||
assertf(hts_query_strip_keys(NULL, "h.com", "/p?a=1", keys, sizeof(keys)) ==
|
||||
NULL);
|
||||
assertf(hts_query_strip_keys("", "h.com", "/p?a=1", keys, sizeof(keys)) ==
|
||||
NULL);
|
||||
assertf(strcmp(fil_normalized_filtered("/p?b=2&a=1", dest, NULL),
|
||||
fil_normalized("/p?b=2&a=1", ref)) == 0);
|
||||
|
||||
/* bare form (*=keys): strip the key everywhere, keep+sort the rest */
|
||||
k = hts_query_strip_keys("sid", "any.com", "/p?b=2&sid=x&a=1", keys,
|
||||
sizeof(keys));
|
||||
assertf(k != NULL && strcmp(k, "sid") == 0);
|
||||
assertf(strcmp(fil_normalized_filtered("/p?b=2&sid=x&a=1", dest, k),
|
||||
"/p?a=1&b=2") == 0);
|
||||
|
||||
/* reordered variant + an extra stripped key == the clean URL */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?sid=y&a=1&b=2", dest, "sid"),
|
||||
fil_normalized("/p?a=1&b=2", ref)) == 0);
|
||||
|
||||
/* host pattern matches only that host, incl. its www-normalized forms */
|
||||
assertf(hts_query_strip_keys("ex.com/*=utm", "other.com", "/p?utm=1", keys,
|
||||
sizeof(keys)) == NULL);
|
||||
assertf(hts_query_strip_keys("ex.com/*=utm", "ex.com", "/p?utm=1", keys,
|
||||
sizeof(keys)) != NULL);
|
||||
assertf(hts_query_strip_keys("ex.com/*=utm", "www.ex.com", "/p?utm=1", keys,
|
||||
sizeof(keys)) != NULL);
|
||||
assertf(hts_query_strip_keys("ex.com/*=utm", "http://www-3.ex.com",
|
||||
"/p?utm=1", keys, sizeof(keys)) != NULL);
|
||||
|
||||
/* last match wins, wholesale: host rule overrides global, no union */
|
||||
k = hts_query_strip_keys("*=sid\nex.com/*=utm", "ex.com",
|
||||
"/p?sid=1&utm=2&a=3", keys, sizeof(keys));
|
||||
assertf(k != NULL && strcmp(k, "utm") == 0);
|
||||
assertf(strcmp(fil_normalized_filtered("/p?sid=1&utm=2&a=3", dest, k),
|
||||
"/p?a=3&sid=1") == 0);
|
||||
k = hts_query_strip_keys("*=sid\nex.com/*=utm", "z.com", "/p?sid=1&a=3", keys,
|
||||
sizeof(keys));
|
||||
assertf(k != NULL && strcmp(k, "sid") == 0);
|
||||
|
||||
/* whole-key match, not prefix: "utm" must not strip utm_source */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?utm_source=x&a=1", dest, "utm"),
|
||||
"/p?a=1&utm_source=x") == 0);
|
||||
|
||||
/* "*" drops every param; a fully-stripped single-arg query loses its '?' */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?a=1&b=2", dest, "*"), "/p") == 0);
|
||||
assertf(strcmp(fil_normalized_filtered("/p?utm=1", dest, "utm"), "/p") == 0);
|
||||
|
||||
/* degenerate forms a=, b, c== (key 'c'); strip c keeps a= and b */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?a=&b&c==", dest, "c"),
|
||||
"/p?a=&b") == 0);
|
||||
/* short key must not strip a longer one: 'c' must not touch 'cc' */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?cc=1&c=2", dest, "c"),
|
||||
"/p?cc=1") == 0);
|
||||
|
||||
/* repeated key: every occurrence is stripped, not just the first */
|
||||
assertf(
|
||||
strcmp(fil_normalized_filtered("/p?foo=42&bar=13&foo=43", dest, "foo"),
|
||||
"/p?bar=13") == 0);
|
||||
/* repeated key mixing missing/empty values */
|
||||
assertf(
|
||||
strcmp(fil_normalized_filtered("/p?foo&bar=13&foo=42&foo=", dest, "foo"),
|
||||
"/p?bar=13") == 0);
|
||||
/* repeated key kept (no match): all occurrences retained, then sorted */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?foo=42&bar=13&foo=43", dest, "z"),
|
||||
"/p?bar=13&foo=42&foo=43") == 0);
|
||||
|
||||
/* value containing '=': the key is only the part before the first '='. Strip
|
||||
'foo' drops "foo=42=17" whole; the '=' in the value is not a delimiter. */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?foo=42=17&bar=", dest, "foo"),
|
||||
"/p?bar=") == 0);
|
||||
/* keeping it preserves the embedded '=' verbatim */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?foo=42=17&bar=", dest, "bar"),
|
||||
"/p?foo=42=17") == 0);
|
||||
/* a value segment is not a key: stripping "42" must not touch foo=42=17 */
|
||||
assertf(strcmp(fil_normalized_filtered("/p?foo=42=17", dest, "42"),
|
||||
"/p?foo=42=17") == 0);
|
||||
|
||||
/* Idempotency: the read path re-normalizes an already-normalized fil, so the
|
||||
result must be a fixpoint or dedup misses (catches a dropped empty/trailing
|
||||
arg like "?&&", "a&"). */
|
||||
{
|
||||
static const char *const qs[] = {"/p?a=&b&c==",
|
||||
"/p?a&&b",
|
||||
"/p?&a",
|
||||
"/p?a&",
|
||||
"/p?",
|
||||
"/p?=v",
|
||||
"/p?&&",
|
||||
"/p?b=2&a=1",
|
||||
"/p?utm=x&",
|
||||
"/p?&utm=x",
|
||||
"/p?foo=42&bar=13&foo=43",
|
||||
"/p?foo&bar=13&foo=42&foo=",
|
||||
"/p?foo=42=17&bar="};
|
||||
static const char *const strips[] = {NULL, "z", "utm", "*", "a", "foo"};
|
||||
char once[1024], twice[1024];
|
||||
size_t i, j;
|
||||
|
||||
for (i = 0; i < sizeof(qs) / sizeof(qs[0]); i++) {
|
||||
for (j = 0; j < sizeof(strips) / sizeof(strips[0]); j++) {
|
||||
fil_normalized_filtered(qs[i], once, strips[j]);
|
||||
fil_normalized_filtered(once, twice, strips[j]);
|
||||
assertf(strcmp(once, twice) == 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
printf("strip-query self-test OK\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* -%u url-hack split (#271): each sub-flag must toggle independently. */
|
||||
static int st_urlhack(httrackp *opt, int argc, char **argv) {
|
||||
(void) argc;
|
||||
(void) argv;
|
||||
#define EQ(aa, fa, ab, fb) hash_url_equals(opt, aa, fa, ab, fb)
|
||||
/* urlhack on, no opt-outs: www, // and query order all collapse */
|
||||
opt->urlhack = HTS_TRUE;
|
||||
opt->no_www_dedup = opt->no_slash_dedup = opt->no_query_dedup = HTS_FALSE;
|
||||
assertf(EQ("www.foo.com", "/a", "foo.com", "/a"));
|
||||
assertf(EQ("foo.com", "/a//b", "foo.com", "/a/b"));
|
||||
assertf(EQ("foo.com", "/p?b=2&a=1", "foo.com", "/p?a=1&b=2"));
|
||||
|
||||
/* keep-www-prefix: host off; // and query still collapse */
|
||||
opt->no_www_dedup = HTS_TRUE;
|
||||
assertf(!EQ("www.foo.com", "/a", "foo.com", "/a"));
|
||||
assertf(EQ("foo.com", "/a//b", "foo.com", "/a/b"));
|
||||
assertf(EQ("foo.com", "/p?b=2&a=1", "foo.com", "/p?a=1&b=2"));
|
||||
opt->no_www_dedup = HTS_FALSE;
|
||||
|
||||
/* keep-double-slashes: // significant; www, query order still collapse */
|
||||
opt->no_slash_dedup = HTS_TRUE;
|
||||
assertf(!EQ("foo.com", "/a//b", "foo.com", "/a/b"));
|
||||
assertf(EQ("www.foo.com", "/a", "foo.com", "/a"));
|
||||
assertf(EQ("foo.com", "/p?b=2&a=1", "foo.com", "/p?a=1&b=2"));
|
||||
opt->no_slash_dedup = HTS_FALSE;
|
||||
|
||||
/* keep-query-order: query order significant; www and // still collapse */
|
||||
opt->no_query_dedup = HTS_TRUE;
|
||||
assertf(!EQ("foo.com", "/p?b=2&a=1", "foo.com", "/p?a=1&b=2"));
|
||||
assertf(EQ("www.foo.com", "/a", "foo.com", "/a"));
|
||||
assertf(EQ("foo.com", "/a//b", "foo.com", "/a/b"));
|
||||
opt->no_query_dedup = HTS_FALSE;
|
||||
|
||||
/* all opt-outs == urlhack off entirely */
|
||||
opt->no_www_dedup = opt->no_slash_dedup = opt->no_query_dedup = HTS_TRUE;
|
||||
assertf(!EQ("www.foo.com", "/a", "foo.com", "/a"));
|
||||
assertf(!EQ("foo.com", "/a//b", "foo.com", "/a/b"));
|
||||
assertf(!EQ("foo.com", "/p?b=2&a=1", "foo.com", "/p?a=1&b=2"));
|
||||
opt->urlhack = HTS_FALSE;
|
||||
opt->no_www_dedup = opt->no_slash_dedup = opt->no_query_dedup = HTS_FALSE;
|
||||
assertf(!EQ("www.foo.com", "/a", "foo.com", "/a"));
|
||||
assertf(!EQ("foo.com", "/a//b", "foo.com", "/a/b"));
|
||||
#undef EQ
|
||||
printf("urlhack self-test OK\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------ */
|
||||
/* Registry: name -> handler, with a usage hint and a one-line description. */
|
||||
/* ------------------------------------------------------------ */
|
||||
@@ -1296,14 +1038,7 @@ static const struct selftest_entry {
|
||||
} selftests[] = {
|
||||
{"filter", "<pattern> <string>", "match a string against a wildcard filter",
|
||||
st_filter},
|
||||
{"filtersize", "<size> <string> <filter>...",
|
||||
"size-aware filter verdict (negative size = unknown/scan time)",
|
||||
st_filtersize},
|
||||
{"simplify", "<path>", "collapse ./ and ../ in a path", st_simplify},
|
||||
{"stripquery", "", "--strip-query pattern/key stripping self-test",
|
||||
st_stripquery},
|
||||
{"urlhack", "", "-%u url-hack sub-flag (www/slash/query) self-test",
|
||||
st_urlhack},
|
||||
{"mime", "<filename>", "MIME type for a filename", st_mime},
|
||||
{"charset", "<charset> <string>",
|
||||
"convert a string to UTF-8 from a charset", st_charset},
|
||||
@@ -1316,7 +1051,6 @@ static const struct selftest_entry {
|
||||
{"strsafe", "[overflow|overflow-buff [str]]", "bounded string-op self-test",
|
||||
st_strsafe},
|
||||
{"copyopt", "", "copy_htsopt option-copy self-test", st_copyopt},
|
||||
{"pause", "", "randomized inter-file pause target self-test", st_pause},
|
||||
{"relative", "<link> <curr-file>", "relative link between two paths",
|
||||
st_relative},
|
||||
{"resolve", "<link> <adr> <fil>", "resolve a link against an origin",
|
||||
|
||||
@@ -90,16 +90,4 @@ refused "dangling-quote argument not refused cleanly"
|
||||
run_only "$tmp/q-lone" '"'
|
||||
refused "lone-quote argument not refused cleanly"
|
||||
|
||||
# --pause (#185): valid MIN[:MAX] accepted; malformed, reversed, over-range and
|
||||
# non-finite values refused cleanly. NaN defeats naive `<`/`>` checks (it
|
||||
# compares false to everything), so it must not slip through to the int cast.
|
||||
run "$tmp/pause-ok" --pause 0.2:0.4
|
||||
accepted "$tmp/pause-ok" "#185: valid --pause range rejected"
|
||||
run "$tmp/pause-fix" --pause 0.2
|
||||
accepted "$tmp/pause-fix" "#185: valid fixed --pause rejected"
|
||||
for bad in nan nan:5 5:nan inf 10:5 99999; do
|
||||
run "$tmp/pause-bad" --pause "$bad"
|
||||
refused "#185: invalid --pause '$bad' not refused cleanly"
|
||||
done
|
||||
|
||||
exit 0
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# -%L URL-list loading (#49): a readable list is honored; an unusable one fails
|
||||
# with the reason (errno / not-a-regular-file), not a bare "Could not include
|
||||
# URL list". Offline: file:// fixture, no server. Asserts on httrack's own
|
||||
# strings and the message shape, so it is locale-independent.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_filelist.XXXXXX") || exit 1
|
||||
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
echo '<html><body>hi</body></html>' >"$tmp/index.html"
|
||||
|
||||
# run httrack with the given -%L target; structured log lands in $out/hts-log.txt
|
||||
run() {
|
||||
local out="$1" list="$2"
|
||||
rm -rf "$out"
|
||||
mkdir -p "$out"
|
||||
httrack -O "$out" --quiet -n "-%L" "$list" >"$out/.stdout" 2>&1 || true
|
||||
LOG="$out/hts-log.txt"
|
||||
}
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $1"
|
||||
cat "$LOG"
|
||||
exit 1
|
||||
}
|
||||
loghas() {
|
||||
grep -Eq "$1" "$LOG" || fail "expected /$1/ in $LOG"
|
||||
}
|
||||
lognot() {
|
||||
if grep -Eq "$1" "$LOG"; then fail "unexpected /$1/ in $LOG"; fi
|
||||
}
|
||||
|
||||
# readable list: its one URL is loaded and counted (count must be non-zero)
|
||||
printf 'file://%s/index.html\n' "$tmp" >"$tmp/urls.txt"
|
||||
run "$tmp/ok" "$tmp/urls.txt"
|
||||
loghas '[1-9][0-9]* links added from'
|
||||
|
||||
# missing file: quoted name + a non-empty reason, never the old reasonless
|
||||
# "Could not include URL list: <name>". The reason is the stat() errno, not the
|
||||
# directory fallback literal (guards against dropping the errno lookup).
|
||||
run "$tmp/miss" "$tmp/nope.txt"
|
||||
loghas 'Could not include URL list "[^"]+": .+'
|
||||
lognot 'Could not include URL list: '
|
||||
lognot 'not a regular file'
|
||||
|
||||
# a directory is rejected with our own reason (locale-independent)
|
||||
mkdir -p "$tmp/adir"
|
||||
run "$tmp/dir" "$tmp/adir"
|
||||
loghas 'Could not include URL list "[^"]+": not a regular file'
|
||||
|
||||
# unreadable regular file: the fopen() errno arm fires, distinct from the
|
||||
# directory branch. Root bypasses mode 000, so skip it there.
|
||||
if test "$(id -u)" -ne 0; then
|
||||
: >"$tmp/noperm.txt"
|
||||
chmod 000 "$tmp/noperm.txt"
|
||||
run "$tmp/perm" "$tmp/noperm.txt"
|
||||
chmod 644 "$tmp/noperm.txt"
|
||||
loghas 'Could not include URL list "[^"]+": .+'
|
||||
lognot 'not a regular file'
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -50,75 +50,24 @@ match '*foo*bar' 'foozbar'
|
||||
# '?' is the query-string marker, not a single-char wildcard
|
||||
nomatch 'a?c' 'abc'
|
||||
|
||||
# Inside a class, backslash escapes the next char as a literal member (#148):
|
||||
# '\X' matches X only (not '\'), and an escaped ']' is a member, not the terminator.
|
||||
# backslash escapes a metacharacter inside a class so it is matched literally.
|
||||
# Quirk: the decoder also adds the backslash itself to the set, so '\X' matches
|
||||
# both X and '\'. These assertions pin that behavior.
|
||||
match '*[\*]' '*'
|
||||
nomatch '*[\*]' "\\"
|
||||
match '*[\*]' "\\"
|
||||
nomatch '*[\*]' 'a'
|
||||
match '*[\\]' "\\"
|
||||
nomatch '*[\\]' '*'
|
||||
nomatch '*[\\]' 'a'
|
||||
match '*[\[]' '['
|
||||
nomatch '*[\[]' "\\"
|
||||
match '*[\]]' ']'
|
||||
nomatch '*[\]]' "\\"
|
||||
match '*[\[]' "\\"
|
||||
nomatch '*[\[]' 'a'
|
||||
|
||||
# '*[\[\]]' is "the [ or ] character", as the filter guide documents.
|
||||
match '*[\[\]]' '['
|
||||
match '*[\[\]]' ']'
|
||||
nomatch '*[\[\]]' 'a'
|
||||
match '*[\[,\]]' '[' # comma between members is optional
|
||||
match '*[\[,\]]' ']'
|
||||
match '*[a,\[]' 'a' # an escaped member no longer eats the preceding one
|
||||
match '*[a,\[]' '['
|
||||
|
||||
# Escape is decoded before the range/separator/size checks, so '\-' '\,' '\<'
|
||||
# are literal members, not operators.
|
||||
match '*[a\-z]' 'a'
|
||||
match '*[a\-z]' 'z'
|
||||
nomatch '*[a\-z]' 'b' # not the a..z range
|
||||
match '*[\,]' ','
|
||||
nomatch '*[\,]' "\\" # the escape must not leak '\' into the class
|
||||
match '*[\<]' '<'
|
||||
nomatch '*[\<]' "\\"
|
||||
match '*[\[,\],a]' '['
|
||||
match '*[\[,\],a]' ']'
|
||||
match '*[\[,\],a]' 'a'
|
||||
|
||||
# A truncated range '*[a-' is the literal members {a,-}; the parser must not
|
||||
# read past the end decoding it (was a 1-byte heap over-read in the range arm).
|
||||
match '*[a-' 'a'
|
||||
nomatch '*[a-' 'b'
|
||||
|
||||
# *(...) matches exactly one char from the class; *[...] matches a run.
|
||||
match '*(a,b)' 'a'
|
||||
nomatch '*(a,b)' 'aa'
|
||||
nomatch '*(a,b)' 'c'
|
||||
|
||||
# documented composite filters (filters.html)
|
||||
match 'www.*[path].com/*[path].zip' 'www.foo.com/a/b.zip'
|
||||
nomatch 'www.*[path].com/*[path].zip' 'www.foo.com/a/b.tar'
|
||||
match '*.html*[]' 'page.html'
|
||||
nomatch '*.html*[]' 'page.html?x=1' # *[] forbids the trailing query
|
||||
|
||||
# Size-based rules (-#test=filtersize <size> <string> <filter...>): a negative size
|
||||
# means the size is still unknown (scan time). A size exclusion must stay neutral
|
||||
# then, so the file is fetched and only cancelled once its size is known (#143).
|
||||
fsize() {
|
||||
local want="$1"
|
||||
shift
|
||||
test "$(httrack -O /dev/null -#test=filtersize "$@")" == "$want" || exit 1
|
||||
}
|
||||
fsize 'verdict=allowed size_flag=0' -1 foo.jpg -* '+*.jpg' '-*.jpg*[<10]' # scan time: keep
|
||||
fsize 'verdict=forbidden size_flag=1' 5 foo.jpg -* '+*.jpg' '-*.jpg*[<10]' # <10KB: cancel
|
||||
fsize 'verdict=allowed size_flag=1' 20 foo.jpg -* '+*.jpg' '-*.jpg*[<10]' # >=10KB: keep
|
||||
fsize 'verdict=forbidden size_flag=0' -1 foo.txt -* '+*.jpg' '-*.jpg*[<10]' # not a jpg
|
||||
# the '>' operator is just as neutral at scan time, and fires once size is known
|
||||
fsize 'verdict=allowed size_flag=0' -1 foo.jpg -* '+*.jpg' '-*.jpg*[>10]' # scan time: keep
|
||||
fsize 'verdict=forbidden size_flag=1' 20 foo.jpg -* '+*.jpg' '-*.jpg*[>10]' # >10KB: cancel
|
||||
|
||||
# [name]/[file]/[path] never span '?' mid-string; a trailing query is still
|
||||
# tolerated by the global '?' rule (same as plain *.aspx), not the class (#144).
|
||||
nomatch '*[path]/end' 'a?b/end'
|
||||
nomatch '*[file]end' 'foo?xend'
|
||||
nomatch '*[name]X' 'abc?X'
|
||||
match '*[file]' 'foo?x=1' # trailing query: tolerated, as for *.aspx
|
||||
match '*.aspx' 'page.aspx?y=2'
|
||||
# A literal ']' cannot be a class member: the class parser stops at the first
|
||||
# ']', escaped or not. So '*[\[\]]' does NOT mean "the [ or ] character" as the
|
||||
# filter guide claims (GitHub #148); it parses as the class {'[','\'} followed
|
||||
# by a trailing literal ']'. These assertions document the current (buggy)
|
||||
# behavior so any future matcher fix is a deliberate, visible change.
|
||||
nomatch '*[\[\]]' '[' # not matched, despite the docs
|
||||
match '*[\[\]]' ']' # only via the empty class-match + trailing ']'
|
||||
match '*[\[\]]' '[]' # one of {'[','\'} then the trailing ']'
|
||||
nomatch '*[\[\]]' '[]x'
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# --pause (#185): the inter-file pause target must stay in [min,max] and spread
|
||||
# across it (a per-call rand() would collapse it toward min). Driven by the
|
||||
# in-process 'httrack -#test=pause' test. POSIX-portable ($(BASH) is /bin/sh on macOS).
|
||||
|
||||
set -eu
|
||||
|
||||
# 'run' is an ignored placeholder argument.
|
||||
out=$(httrack -#test=pause run)
|
||||
|
||||
test "$out" = "pause: OK" || {
|
||||
echo "expected 'pause: OK', got: $out" >&2
|
||||
exit 1
|
||||
}
|
||||
@@ -1,8 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# --strip-query: pattern-scoped query-key stripping for dedup. All assertions
|
||||
# live in the engine self-test (hts_query_strip_keys + fil_normalized_filtered).
|
||||
httrack -O /dev/null -#test=stripquery | grep -q "strip-query self-test OK"
|
||||
@@ -1,8 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# -%u url-hack split (#271): www / // / query-order dedup toggle independently.
|
||||
# All assertions live in the engine self-test (hash compare flag resolution).
|
||||
httrack -O /dev/null -#test=urlhack run | grep -q "urlhack self-test OK"
|
||||
@@ -1,109 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Issue #198: on a resumed download the server may answer the Range with a 206
|
||||
# that starts *before* the offset we asked for (block-aligned ranges). httrack
|
||||
# must honor the returned Content-Range, not blindly append, or the overlap
|
||||
# bytes get duplicated and the file grows (corrupt PDFs). Pass 1 interrupts
|
||||
# flaky.bin mid-body (partial + temp-ref); pass 2 resumes against a 206 that
|
||||
# backs up 8 bytes. The result must equal the same bytes fetched whole (full.bin).
|
||||
set -eu
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
testdir=$(cd "$(dirname "$0")" && pwd)
|
||||
server="${testdir}/local-server.py"
|
||||
|
||||
command -v python3 >/dev/null || ! echo "python3 not found; skipping" || exit 77
|
||||
|
||||
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_198.XXXXXX") || exit 1
|
||||
serverpid=
|
||||
crawlpid=
|
||||
cleanup() {
|
||||
if test -n "$crawlpid"; then kill -9 "$crawlpid" 2>/dev/null || true; fi
|
||||
if test -n "$serverpid"; then
|
||||
kill "$serverpid" 2>/dev/null || true
|
||||
wait "$serverpid" 2>/dev/null || true
|
||||
fi
|
||||
rm -rf "$tmpdir"
|
||||
}
|
||||
trap cleanup EXIT HUP INT QUIT PIPE TERM
|
||||
|
||||
# OVERLAP_COUNTER gets a byte per flaky.bin request so pass 1 knows when to interrupt.
|
||||
serverlog="${tmpdir}/server.log"
|
||||
counter="${tmpdir}/hits"
|
||||
resumed="${tmpdir}/resumed" # gets a byte when the server serves a resume 206
|
||||
OVERLAP_COUNTER="$counter" OVERLAP_RESUMED="$resumed" \
|
||||
python3 "$server" --root "${testdir}/server-root" \
|
||||
>"$serverlog" 2>&1 &
|
||||
serverpid=$!
|
||||
port=
|
||||
for _ in $(seq 1 50); do
|
||||
line=$(head -n1 "$serverlog" 2>/dev/null)
|
||||
if test "${line%% *}" == "PORT"; then
|
||||
port="${line#PORT }"
|
||||
break
|
||||
fi
|
||||
kill -0 "$serverpid" 2>/dev/null || {
|
||||
echo "server exited early: $(cat "$serverlog")"
|
||||
exit 1
|
||||
}
|
||||
sleep 0.1
|
||||
done
|
||||
test -n "$port" || {
|
||||
echo "could not discover server port"
|
||||
exit 1
|
||||
}
|
||||
base="http://127.0.0.1:${port}"
|
||||
|
||||
which httrack >/dev/null || {
|
||||
echo "could not find httrack"
|
||||
exit 1
|
||||
}
|
||||
out="${tmpdir}/crawl"
|
||||
common=(-O "$out" --quiet --disable-security-limits --robots=0 --timeout=30 --retries=0 -c1)
|
||||
refdir="${out}/hts-cache/ref"
|
||||
|
||||
# pass 1: interrupt once flaky.bin's prefix is streaming (partial + temp-ref).
|
||||
printf '[pass 1: interrupt flaky.bin] ..\t'
|
||||
httrack "${common[@]}" "${base}/overlap/index.html" >"${tmpdir}/log1" 2>&1 &
|
||||
crawlpid=$!
|
||||
for _ in $(seq 1 300); do
|
||||
test -s "$counter" && break
|
||||
kill -0 "$crawlpid" 2>/dev/null || break
|
||||
sleep 0.1
|
||||
done
|
||||
sleep 0.5
|
||||
kill -TERM "$crawlpid" 2>/dev/null || true
|
||||
wait "$crawlpid" 2>/dev/null || true
|
||||
crawlpid=
|
||||
test -n "$(find "$refdir" -name '*.ref' 2>/dev/null)" || {
|
||||
echo "FAIL: no temp-ref survived pass 1; cannot drive the resume"
|
||||
exit 1
|
||||
}
|
||||
echo "OK (temp-ref present)"
|
||||
|
||||
# pass 2: --continue -> resume Range -> 206 that starts 8 bytes early.
|
||||
printf '[pass 2: resume flaky.bin] ..\t'
|
||||
httrack "${common[@]}" --continue "${base}/overlap/index.html" >"${tmpdir}/log2" 2>&1 || true
|
||||
echo "OK"
|
||||
|
||||
# Guard against a silent full re-download: the byte-compare below only tests the
|
||||
# fix if pass 2 actually went through the resume Range -> 206 path.
|
||||
printf '[resume path was exercised] ..\t'
|
||||
if ! test -s "$resumed"; then
|
||||
echo "FAIL: pass 2 never triggered a resume 206; the overlap fix was not exercised"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK"
|
||||
|
||||
printf '[resumed file is not corrupted] ..\t'
|
||||
dir=$(find "$out" -maxdepth 1 -type d -name '127.0.0.1*' | head -1)
|
||||
flaky="${dir}/overlap/flaky.bin"
|
||||
full="${dir}/overlap/full.bin"
|
||||
if ! test -f "$flaky" || ! test -f "$full"; then
|
||||
echo "FAIL: flaky.bin or full.bin missing after pass 2"
|
||||
exit 1
|
||||
fi
|
||||
if ! cmp -s "$flaky" "$full"; then
|
||||
echo "FAIL: resumed flaky.bin ($(wc -c <"$flaky")) != full.bin ($(wc -c <"$full")); overlap duplicated"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK ($(wc -c <"$flaky") bytes, byte-identical)"
|
||||
@@ -1,16 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# A -mime: exclusion must abort the transfer on the response Content-Type, not
|
||||
# fetch the whole 1 MB body then discard it (#58). The bytes-received guard is
|
||||
# the real one: the file is absent either way, but only the fix keeps the count
|
||||
# tiny (header only) instead of pulling the body. Match it positively (a small,
|
||||
# <=4-digit count) so a vanished/reworded summary line fails rather than passes.
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 0 \
|
||||
--found 'mimex/real.html' \
|
||||
--not-found 'mimex/blob.pdf' \
|
||||
--log-found 'excluded by MIME type filter' \
|
||||
--log-found '\[[0-9]{1,4} bytes received' \
|
||||
httrack 'BASEURL/mimex/index.html' '-mime:application/pdf'
|
||||
@@ -1,23 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# End-to-end --strip-query (#112): two links to one resource differing only by
|
||||
# ?utm_source dedup to a single saved file (2 files written: index + resource);
|
||||
# the control crawl without the option keeps both variants (3 files). Locks the
|
||||
# CLI->opt->hash plumbing the engine self-test can't reach.
|
||||
|
||||
set -e
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
# stripped: the two ?utm_source variants collapse to one resource
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 0 --files 2 \
|
||||
httrack 'BASEURL/stripquery/index.html' --strip-query 'utm_source'
|
||||
|
||||
# control: no stripping -> both query-named variants are saved
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 0 --files 3 \
|
||||
httrack 'BASEURL/stripquery/index.html'
|
||||
|
||||
# strip still applies with url-hack off (-%u0): exercises the urlhack-off
|
||||
# savename branch, which must normalize the dedup key the same way the hash does
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 0 --files 2 \
|
||||
httrack 'BASEURL/stripquery/index.html' -%u0 --strip-query 'utm_source'
|
||||
@@ -1,22 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# End-to-end --cookies-file (#215): /gated/secret.php needs a cookie no page
|
||||
# ever Set-Cookies, so it is reachable only when the option preloads it from a
|
||||
# Netscape cookies.txt. Locks the CLI->opt->cookie_load->wire plumbing.
|
||||
|
||||
set -e
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
# preloaded cookie -> secret page is served. -o0 means a 500 leaves no file, so
|
||||
# --found/--files only hold when the secret is genuinely fetched (200).
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --cookie 'session=opensesame' \
|
||||
--errors 0 --files 2 \
|
||||
--found 'gated/index.html' --found 'gated/secret.html' \
|
||||
httrack 'BASEURL/gated/index.php' -o0
|
||||
|
||||
# control: without the cookie the secret 500s; -o0 suppresses the error page so
|
||||
# its absence is real (error + missing file)
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 1 \
|
||||
--found 'gated/index.html' --not-found 'gated/secret.html' \
|
||||
httrack 'BASEURL/gated/index.php' -o0
|
||||
@@ -1,29 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# --pause (#185): a fixed inter-file delay must slow a multi-file crawl. Measure
|
||||
# the same crawl with and without --pause and compare: the harness overhead
|
||||
# cancels, leaving only the pause. Integer seconds keep it portable (BSD date
|
||||
# has no %N); a lower bound is not timing-flaky since a pause only adds time.
|
||||
|
||||
set -e
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
run() { # echoes the wall-clock seconds of one crawl
|
||||
local t0 t1
|
||||
t0=$(date +%s)
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 0 \
|
||||
httrack 'BASEURL/types/index.html' -c1 "$@" >/dev/null 2>&1
|
||||
t1=$(date +%s)
|
||||
echo $((t1 - t0))
|
||||
}
|
||||
|
||||
base=$(run)
|
||||
paused=$(run --pause 0.5)
|
||||
delta=$((paused - base))
|
||||
|
||||
echo "crawl: ${base}s, with --pause 0.5: ${paused}s (delta ${delta}s)"
|
||||
if [ "$delta" -lt 2 ]; then
|
||||
echo "FAIL: --pause did not delay the crawl (delta ${delta}s)" >&2
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,11 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Issue #204: a 302 Location with a #fragment must drop the fragment before the
|
||||
# target is fetched. The server is strict (400 on a '#' in the request-target),
|
||||
# so a leaked fragment logs an error and the target is never saved.
|
||||
set -e
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --errors 0 \
|
||||
--found 'redir/target.html' \
|
||||
httrack 'BASEURL/redir/index.html'
|
||||
@@ -5,7 +5,6 @@ EXTRA_DIST = $(TESTS) crawl-test.sh run-all-tests.sh check-network.sh \
|
||||
proxy-https-server.py \
|
||||
local-crawl.sh local-server.py server.crt server.key \
|
||||
server-root/simple/basic.html server-root/simple/link.html \
|
||||
server-root/stripquery/index.html server-root/stripquery/a.html \
|
||||
fixtures/cache-golden/hts-cache/new.zip
|
||||
|
||||
TESTS_ENVIRONMENT =
|
||||
@@ -35,21 +34,17 @@ TESTS = \
|
||||
01_engine-dns.test \
|
||||
01_engine-doitlog.test \
|
||||
01_engine-entities.test \
|
||||
01_engine-filelist.test \
|
||||
01_engine-filter.test \
|
||||
01_engine-hashtable.test \
|
||||
01_engine-idna.test \
|
||||
01_engine-mime.test \
|
||||
01_engine-parse.test \
|
||||
01_engine-pause.test \
|
||||
01_engine-rcfile.test \
|
||||
01_engine-relative.test \
|
||||
01_engine-savename.test \
|
||||
01_engine-selftest-dispatch.test \
|
||||
01_engine-simplify.test \
|
||||
01_engine-stripquery.test \
|
||||
01_engine-strsafe.test \
|
||||
01_engine-urlhack.test \
|
||||
02_manpage-regen.test \
|
||||
02_update-cache.test \
|
||||
10_crawl-simple.test \
|
||||
@@ -70,12 +65,6 @@ TESTS = \
|
||||
20_local-resume-loop.test \
|
||||
21_local-intl-update.test \
|
||||
22_local-broken-size.test \
|
||||
23_local-errpage.test \
|
||||
24_local-resume-overlap.test \
|
||||
25_local-mime-exclude.test \
|
||||
26_local-strip-query.test \
|
||||
27_local-cookies-file.test \
|
||||
28_local-pause.test \
|
||||
29_local-redirect-fragment.test
|
||||
23_local-errpage.test
|
||||
|
||||
CLEANFILES = check-network_sh.cache
|
||||
|
||||
@@ -12,14 +12,11 @@
|
||||
# the mirror directory name.
|
||||
#
|
||||
# Usage:
|
||||
# bash local-crawl.sh [--tls] [--root DIR] [--cookie NAME=VALUE ...] \
|
||||
# bash local-crawl.sh [--tls] [--root DIR] \
|
||||
# --errors N --files N --found PATH ... --directory PATH ... \
|
||||
# --log-found REGEX ... --log-not-found REGEX ... \
|
||||
# httrack BASEURL/some/path [httrack-args...]
|
||||
# --log-found/--log-not-found grep (ERE) the crawl's hts-log.txt.
|
||||
# --cookie writes a Netscape cookies.txt (scoped to the discovered host:port,
|
||||
# which the ephemeral port forces into the cookie domain) and passes it to
|
||||
# httrack via --cookies-file, to exercise preloaded cookies.
|
||||
|
||||
set -u
|
||||
|
||||
@@ -88,7 +85,6 @@ tmpdir=$(mktemp -d "${tmptopdir}/httrack_local.XXXXXX") || die "could not create
|
||||
|
||||
# --- parse leading control flags --------------------------------------------
|
||||
declare -a audit=()
|
||||
declare -a cookies=()
|
||||
scheme=http
|
||||
pos=0
|
||||
args=("$@")
|
||||
@@ -109,10 +105,6 @@ while test "$pos" -lt "$nargs"; do
|
||||
pos=$((pos + 1))
|
||||
root="${args[$pos]}"
|
||||
;;
|
||||
--cookie)
|
||||
pos=$((pos + 1))
|
||||
cookies+=("${args[$pos]}")
|
||||
;;
|
||||
--errors | --files)
|
||||
audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
|
||||
pos=$((pos + 1))
|
||||
@@ -166,17 +158,6 @@ while test "$pos" -lt "$nargs"; do
|
||||
pos=$((pos + 1))
|
||||
done
|
||||
|
||||
# --- materialize any --cookie entries into a cookies.txt ---------------------
|
||||
if test "${#cookies[@]}" -gt 0; then
|
||||
jar="${tmpdir}/cookies.txt"
|
||||
: >"$jar"
|
||||
for spec in "${cookies[@]}"; do
|
||||
printf '127.0.0.1:%s\tTRUE\t/\tFALSE\t1999999999\t%s\t%s\n' \
|
||||
"$port" "${spec%%=*}" "${spec#*=}" >>"$jar"
|
||||
done
|
||||
hts+=(--cookies-file "$jar")
|
||||
fi
|
||||
|
||||
# --- run httrack -------------------------------------------------------------
|
||||
which httrack >/dev/null || die "could not find httrack"
|
||||
ver=$(httrack -O /dev/null --version | sed -e 's/HTTrack version //')
|
||||
|
||||
@@ -110,19 +110,6 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
return self.fail_cookie("badger")
|
||||
self.send_html("\tThis is a test.")
|
||||
|
||||
# --cookies-file (#215): the secret page needs a cookie no page ever sets,
|
||||
# so it is reachable only when --cookies-file preloads it.
|
||||
GATE_COOKIE = ("session", "opensesame")
|
||||
|
||||
def route_gated_index(self):
|
||||
self.send_html('\tThis is a <a href="secret.php">link</a>')
|
||||
|
||||
def route_gated_secret(self):
|
||||
name, value = self.GATE_COOKIE
|
||||
if self.request_cookies().get(name) != value:
|
||||
return self.fail_cookie(name)
|
||||
self.send_html("\tThis is the secret.")
|
||||
|
||||
def route_robots(self):
|
||||
body = b"User-agent: *\nDisallow:\n"
|
||||
self.send_response(200)
|
||||
@@ -190,24 +177,6 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
body, ctype = self.TYPE_MATRIX[path]
|
||||
self.send_raw(body, ctype)
|
||||
|
||||
# --- MIME-type exclusion abort (issue #58) -----------------------------
|
||||
# A -mime:application/pdf filter must abort the transfer once the header
|
||||
# arrives, not download the whole body and discard it.
|
||||
def route_mimex_index(self):
|
||||
self.send_html(
|
||||
'\t<a href="blob.pdf">pdf</a>\n' '\t<a href="real.html">real</a>\n'
|
||||
)
|
||||
|
||||
# 1 MB body: the fix aborts after the header, so httrack's "bytes received"
|
||||
# stays tiny; without it the engine reads the body and the count jumps.
|
||||
MIMEX_BLOB = b"%PDF-1.4\n" + b"\x00" * (1024 * 1024)
|
||||
|
||||
def route_mimex_blob(self):
|
||||
self.send_raw(self.MIMEX_BLOB, "application/pdf")
|
||||
|
||||
def route_mimex_real(self):
|
||||
self.send_raw(b"<html><body>real</body></html>", "text/html")
|
||||
|
||||
# --- special chars in URLs across an update (issue #157) ---------------
|
||||
# A dotless, accented basename served as text/html (MediaWiki style). The
|
||||
# name the first crawl picks (.html) must survive the update pass.
|
||||
@@ -256,71 +225,6 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
self.send_header("Content-Length", "0")
|
||||
self.end_headers()
|
||||
|
||||
# 206 resume must honor the server's Content-Range, not the offset we asked
|
||||
# for (#198): a server resuming a few bytes *before* the request must not
|
||||
# leave httrack duplicating the overlap onto the partial. flaky.bin
|
||||
# interrupts once then resumes OVERLAP_EARLY bytes early; full.bin serves
|
||||
# the identical bytes in one shot, so the test can compare the two.
|
||||
OVERLAP_BLOB = b"%PDF-1.4\n" + bytes((i * 37 + 11) % 256 for i in range(8000))
|
||||
OVERLAP_EARLY = 8
|
||||
OVERLAP_PREFIX_LEN = 4000 # flushed before the stall
|
||||
_overlap_started = False
|
||||
|
||||
def route_overlap_index(self):
|
||||
self.send_html('\t<a href="flaky.bin">flaky</a>\n\t<a href="full.bin">full</a>')
|
||||
|
||||
def route_overlap_full(self):
|
||||
self.send_raw(self.OVERLAP_BLOB, "application/octet-stream")
|
||||
|
||||
def route_overlap(self):
|
||||
counter = os.environ.get("OVERLAP_COUNTER")
|
||||
if counter:
|
||||
with open(counter, "a") as fp:
|
||||
fp.write("x")
|
||||
blob = self.OVERLAP_BLOB
|
||||
rng = self.headers.get("Range")
|
||||
# First GET: stream a prefix then stall, so the crawl can be interrupted
|
||||
# mid-body (partial + temp-ref on disk).
|
||||
if rng is None and not Handler._overlap_started:
|
||||
Handler._overlap_started = True
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/octet-stream")
|
||||
self.send_header("Content-Length", str(len(blob)))
|
||||
self.send_header("Accept-Ranges", "bytes")
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(blob[: self.OVERLAP_PREFIX_LEN])
|
||||
self.wfile.flush()
|
||||
try:
|
||||
while True:
|
||||
time.sleep(3600)
|
||||
except OSError:
|
||||
pass
|
||||
return
|
||||
if rng is None: # no resume request: serve the whole file
|
||||
return self.route_overlap_full()
|
||||
# Resume: honor the Range, but back up OVERLAP_EARLY bytes.
|
||||
start = (
|
||||
int(rng[len("bytes=") :].split("-")[0]) if rng.startswith("bytes=") else 0
|
||||
)
|
||||
start = max(0, start - self.OVERLAP_EARLY)
|
||||
# Signal that the resume Range -> 206 path actually fired, so the test
|
||||
# can prove it was exercised (not a silent full re-download).
|
||||
resumed = os.environ.get("OVERLAP_RESUMED")
|
||||
if resumed:
|
||||
with open(resumed, "a") as fp:
|
||||
fp.write("x")
|
||||
part = blob[start:]
|
||||
self.send_response(206, "Partial Content")
|
||||
self.send_header("Content-Type", "application/octet-stream")
|
||||
self.send_header("Content-Length", str(len(part)))
|
||||
self.send_header(
|
||||
"Content-Range", "bytes %d-%d/%d" % (start, len(blob) - 1, len(blob))
|
||||
)
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(part)
|
||||
|
||||
# error pages / 0-byte files (#17): -o0 ("no error pages") must keep 4xx/5xx
|
||||
# bodies off disk; a genuine 0-byte 200 is a valid file and stays.
|
||||
def route_errpage_index(self):
|
||||
@@ -354,27 +258,10 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(body)
|
||||
|
||||
# 302 whose Location carries a #fragment (#204): the fragment is a UA anchor
|
||||
# that must be dropped before the target is fetched. A leaked '#' reaches the
|
||||
# strict-server guard below and 400s.
|
||||
def route_redir_index(self):
|
||||
self.send_html('\t<a href="go.php">go</a>')
|
||||
|
||||
def route_redir_go(self):
|
||||
self.send_response(302, "Found")
|
||||
self.send_header("Location", "target.html#section")
|
||||
self.send_header("Content-Length", "0")
|
||||
self.end_headers()
|
||||
|
||||
def route_redir_target(self):
|
||||
self.send_raw(b"<html><body>redirect target</body></html>\n", "text/html")
|
||||
|
||||
ROUTES = {
|
||||
"/cookies/entrance.php": route_entrance,
|
||||
"/cookies/second.php": route_second,
|
||||
"/cookies/third.php": route_third,
|
||||
"/gated/index.php": route_gated_index,
|
||||
"/gated/secret.php": route_gated_secret,
|
||||
"/robots.txt": route_robots,
|
||||
"/types/index.html": route_types_index,
|
||||
"/types/control.php": route_types,
|
||||
@@ -394,35 +281,16 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
"/intl/" + INTL_NAME: route_intl_page,
|
||||
"/resume/index.html": route_resume_index,
|
||||
"/resume/blob.txt": route_resume,
|
||||
"/overlap/index.html": route_overlap_index,
|
||||
"/overlap/flaky.bin": route_overlap,
|
||||
"/overlap/full.bin": route_overlap_full,
|
||||
"/size/index.html": route_size_index,
|
||||
"/size/oversize.bin": route_size_oversize,
|
||||
"/errpage/index.html": route_errpage_index,
|
||||
"/errpage/good.html": route_errpage_good,
|
||||
"/errpage/missing.html": route_errpage_missing,
|
||||
"/errpage/empty.html": route_errpage_empty,
|
||||
"/mimex/index.html": route_mimex_index,
|
||||
"/mimex/blob.pdf": route_mimex_blob,
|
||||
"/mimex/real.html": route_mimex_real,
|
||||
"/redir/index.html": route_redir_index,
|
||||
"/redir/go.php": route_redir_go,
|
||||
"/redir/target.html": route_redir_target,
|
||||
}
|
||||
|
||||
# --- dispatch ----------------------------------------------------------
|
||||
|
||||
def reject_fragment(self):
|
||||
# Strict server: a '#' in the request-target is the client failing to
|
||||
# drop a fragment (#204). RFC 3986 forbids it on the wire; answer 400.
|
||||
if "#" in self.path:
|
||||
self.send_response(400, "Bad Request")
|
||||
self.send_header("Content-Length", "0")
|
||||
self.end_headers()
|
||||
return True
|
||||
return False
|
||||
|
||||
def dispatch(self):
|
||||
self._set_cookies = []
|
||||
path = urlsplit(self.path).path
|
||||
@@ -434,14 +302,10 @@ class Handler(SimpleHTTPRequestHandler):
|
||||
return False
|
||||
|
||||
def do_GET(self):
|
||||
if self.reject_fragment():
|
||||
return
|
||||
if not self.dispatch():
|
||||
super().do_GET()
|
||||
|
||||
def do_HEAD(self):
|
||||
if self.reject_fragment():
|
||||
return
|
||||
if not self.dispatch():
|
||||
super().do_HEAD()
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
<html><body>resource A</body></html>
|
||||
@@ -1,5 +0,0 @@
|
||||
<html><body>
|
||||
Two links to one resource, differing only by a tracking parameter.
|
||||
<a href="a.html?utm_source=x">x</a>
|
||||
<a href="a.html?utm_source=y">y</a>
|
||||
</body></html>
|
||||
Reference in New Issue
Block a user