Compare commits

..

2 Commits

Author SHA1 Message Date
Xavier Roche
b6f794b404 Use hts_boolean and HTS_TRUE/HTS_FALSE for the refetch-whole flag
The #581 fix carried the whole-file refetch flag under three inconsistent
types (short int, char, int) set with bare 1/0. Normalize all three to the
house hts_boolean type and use the HTS_TRUE/HTS_FALSE macros for the literal
sets. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:07:17 +02:00
Xavier Roche
c791e73ef7 Force a whole-file refetch when a rejected 206 resume can loop (#581)
Resuming an interrupted download, the engine sends a Range and, if the
server answers an unusable 206 (Content-Range not matching the partial on
disk), drops the partial and retries the whole file. On Windows a hard-killed
first pass leaves a truncated cache that the next pass must repair; the retry
can then re-derive a Range from a partial or temp-ref that outlived the
restart, meet the same unusable 206, and loop until retries run out. The
partial is removed but never refetched, so the file is lost from the mirror.

Carry a refetch-whole signal from the restart-whole decision onto the
requeued link, so the retry's back_add drops any stale temp-ref and skips the
partial/temp-ref resume branches: it sends no Range and GETs the whole file,
regardless of whether a partial survived the restart. On POSIX the restart
already removes both sources, so the retry was already Range-less and behavior
is unchanged; test 71 still recovers the file whole.

back_add gains an internal (hidden, non-exported) parameter; htsblk and
lien_url gain one trailing field each. No exported symbol changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 08:49:12 +02:00
11 changed files with 141 additions and 205 deletions

View File

@@ -55,9 +55,6 @@ jobs:
query-filters:
- exclude:
id: cpp/world-writable-file-creation
# Models auth-bypass-by-spoofing; httrack has no auth surface, its +/- crawl filter is a mirror boundary, not a security one.
- exclude:
id: cpp/user-controlled-bypass
# Manual build: CodeQL traces the compiler, so build exactly what ships.
- name: Build

View File

@@ -234,7 +234,7 @@ below let you go faster when you own the target, and slower when you do not.</p>
<tr><td><tt>--max-time (-E)</tt></td><td>Stop after N seconds of wall-clock time.</td></tr>
<tr><td><tt>--max-files (-m)</tt></td><td>Per-file size caps.</td></tr>
<tr><td><tt>--timeout (-T), --retries (-R), --min-rate (-J), --host-control (-H)</tt></td><td>Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.</td></tr>
<tr><td><tt>--max-pause (-G), --pause (-%G)</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
<tr><td><tt>--max-pause (-G), -%G</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
</table>
<p><b>The security clamps.</b> To keep an accidental typo from turning into a flood,
@@ -257,8 +257,8 @@ really HTML.</p>
<tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
<tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
<tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
<tr><td><tt>--delayed-type-check (-%N), --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
<tr><td><tt>--include-query-string (-%q), --strip-query (-%g)</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
<tr><td><tt>-%N, --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
<tr><td><tt>--include-query-string (-%q), -%g</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
</table>
<p>The <tt>-N</tt> presets are built from modular arithmetic on the name fields, so
@@ -329,7 +329,7 @@ across links at the same time.</p>
<tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
<tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
<tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
<tr><td><tt>--http-10 (-%h), --keep-alive (-%k), --disable-compression (-%z)</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
<tr><td><tt>--http-10 (-%h), --keep-alive (-%k), -%z</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
<tr><td><tt>--bind (-%b), --tolerant (-%B)</tt></td><td>Bind to a local address, and accept technically-bogus responses some servers send.</td></tr>
</table>
@@ -353,7 +353,7 @@ away and you lose the ability to continue or update the mirror.</p>
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
<tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
<tr><td><tt>--cache (-C)</tt></td><td>Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.</td></tr>
<tr><td><tt>--urlhack (-%u), --keep-www-prefix (-%j), --keep-double-slashes (-%o), --keep-query-order (-%y), --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
<tr><td><tt>--urlhack (-%u), -%j, -%o, -%y, --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
<tr><td><tt>--debug-cache (-#C), --repair-cache (-#R), --clean</tt></td><td>Inspect the cache, repair its ZIP, and erase cache plus logs.</td></tr>
</table>
@@ -404,16 +404,14 @@ host and stops the crawl; start from the final URL, or add
rule wins.</small></p>
<h4>Download the PDFs on a site</h4>
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
<p><tt>httrack https://example.com/ --path mydir</tt><br>
<small>There is no PDF-only crawl. HTTrack discovers PDF links by parsing the
site's HTML pages, so a <tt>"-*" "+example.com/*.pdf"</tt> filter blocks the very
pages that carry the links and grabs only the PDFs linked from the front page. Let
it crawl the site: the HTML pages come along as the scaffolding, and every reachable
PDF is saved with them. If some PDFs live on another host (a CDN or a docs
subdomain), allow that host too, for example
<tt>"+docs.example.com/*.pdf"</tt>.</small></p>
<h4>Keep page requisites, including off-host images</h4>
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>

View File

@@ -3,7 +3,7 @@
.\"
.\" This file is generated by man/makeman.sh; do not edit by hand.
.\" SPDX-License-Identifier: GPL-3.0-or-later
.TH httrack 1 "22 July 2026" "httrack website copier"
.TH httrack 1 "21 July 2026" "httrack website copier"
.SH NAME
httrack \- offline browser : copy websites to a local directory
.SH SYNOPSIS
@@ -37,7 +37,6 @@ httrack \- offline browser : copy websites to a local directory
[ \fB\-%L, \-\-list\fR ]
[ \fB\-%S, \-\-urllist\fR ]
[ \fB\-NN, \-\-structure[=N]\fR ]
[ \fB\-%N, \-\-delayed\-type\-check\fR ]
[ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
[ \fB\-%M, \-\-mime\-html\fR ]
[ \fB\-LN, \-\-long\-names[=N]\fR ]
@@ -190,7 +189,7 @@ structure type (0 *original structure, 1+: see below) (\-\-structure[=N])
.br
or user defined structure (\-N "%h%p/%n%q.%t")
.IP \-%N
delayed type check, don't make any link test but wait for files download to start instead (experimental) (%N0 don't use, %N1 use for unknown extensions, * %N2 always use) (\-\-delayed\-type\-check)
delayed type check, don't make any link test but wait for files download to start instead (experimental) (%N0 don't use, %N1 use for unknown extensions, * %N2 always use)
.IP \-%D
cached delayed type check, don't wait for remote type during updates, to speedup them (%D0 wait, * %D1 don't wait) (\-\-cached\-delayed\-type\-check)
.IP \-%M

View File

@@ -856,41 +856,7 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
// si save==null alors test unqiquement
static int hts_rename(httrackp * opt, const char *a, const char *b) {
hts_log_print(opt, LOG_DEBUG, "Cache: rename %s -> %s (%p %p)", a, b, a, b);
return RENAME(a, b);
}
/* Open the cache ZIP via hts_fopen_utf8 so a non-ASCII path_log isn't mangled
to ANSI (#630); 64-bit funcs keep multi-GB caches whole on Windows LLP64. */
static voidpf ZCALLBACK hts_zip_fopen_utf8(voidpf opaque, const void *filename,
int mode) {
const char *mode_fopen = NULL;
(void) opaque;
if ((mode & ZLIB_FILEFUNC_MODE_READWRITEFILTER) == ZLIB_FILEFUNC_MODE_READ)
mode_fopen = "rb";
else if (mode & ZLIB_FILEFUNC_MODE_EXISTING)
mode_fopen = "r+b";
else if (mode & ZLIB_FILEFUNC_MODE_CREATE)
mode_fopen = "wb";
if (filename == NULL || mode_fopen == NULL)
return NULL;
return (voidpf) FOPEN((const char *) filename, mode_fopen);
}
static unzFile hts_unzOpen_utf8(const char *path) {
zlib_filefunc64_def ff;
fill_fopen64_filefunc(&ff);
ff.zopen64_file = hts_zip_fopen_utf8;
return unzOpen2_64(path, &ff);
}
static zipFile hts_zipOpen_utf8(const char *path, int append) {
zlib_filefunc64_def ff;
fill_fopen64_filefunc(&ff);
ff.zopen64_file = hts_zip_fopen_utf8;
return zipOpen2_64(path, append, NULL, &ff);
return rename(a, b);
}
/* Pathname of a file inside the mirror dir (rotating concat buffer). */
@@ -908,9 +874,9 @@ static char *reconcile_path(httrackp *opt, const char *name) {
/* Replace the new-generation file by the old one, when the old one exists. */
static void reconcile_promote(httrackp *opt, const char *oldname,
const char *newname) {
if (fexist_utf8(reconcile_path(opt, oldname))) {
UNLINK(reconcile_path(opt, newname));
RENAME(reconcile_path(opt, oldname), reconcile_path(opt, newname));
if (fexist(reconcile_path(opt, oldname))) {
remove(reconcile_path(opt, newname));
rename(reconcile_path(opt, oldname), reconcile_path(opt, newname));
}
}
@@ -919,7 +885,7 @@ void hts_cache_reconcile(httrackp *opt, hts_cache_reconcile_mode mode) {
case CACHE_RECONCILE_PROMOTE:
/* Previous run rotated new.* to old.* then died before writing: promote
the old generation back, whichever format it uses. */
if (!fexist_utf8(reconcile_path(opt, "hts-cache/new.zip")))
if (!fexist(reconcile_path(opt, "hts-cache/new.zip")))
reconcile_promote(opt, "hts-cache/old.zip", "hts-cache/new.zip");
break;
case CACHE_RECONCILE_INTERRUPTED:
@@ -928,17 +894,16 @@ void hts_cache_reconcile(httrackp *opt, hts_cache_reconcile_mode mode) {
is -1 for a missing file, which would spuriously pass the "< TINY" test
and overwrite a solid old generation that PROMOTE/ROLLBACK should keep.
*/
if (!opt->cache ||
!fexist_utf8(reconcile_path(opt, "hts-in_progress.lock")))
if (!opt->cache || !fexist(reconcile_path(opt, "hts-in_progress.lock")))
break;
if (fexist_utf8(reconcile_path(opt, "hts-cache/new.zip")) &&
fexist_utf8(reconcile_path(opt, "hts-cache/old.zip")) &&
fsize_utf8(reconcile_path(opt, "hts-cache/new.zip")) <
if (fexist(reconcile_path(opt, "hts-cache/new.zip")) &&
fexist(reconcile_path(opt, "hts-cache/old.zip")) &&
fsize(reconcile_path(opt, "hts-cache/new.zip")) <
CACHE_RECONCILE_NEW_TINY &&
fsize_utf8(reconcile_path(opt, "hts-cache/old.zip")) >
fsize(reconcile_path(opt, "hts-cache/old.zip")) >
CACHE_RECONCILE_OLD_SOLID &&
fsize_utf8(reconcile_path(opt, "hts-cache/old.zip")) >
fsize_utf8(reconcile_path(opt, "hts-cache/new.zip")))
fsize(reconcile_path(opt, "hts-cache/old.zip")) >
fsize(reconcile_path(opt, "hts-cache/new.zip")))
reconcile_promote(opt, "hts-cache/old.zip", "hts-cache/new.zip");
break;
case CACHE_RECONCILE_ROLLBACK:
@@ -975,26 +940,24 @@ void cache_init(cache_back * cache, httrackp * opt) {
#endif
if (!cache->ro) {
#ifdef _WIN32
/* Windows mkdir takes no mode; use the UTF-8 wrapper for #630. */
MKDIR(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache"));
mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache"));
#else
/* keep the cache dir 0700, not MKDIR's 0755. */
mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache"),
mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache"),
HTS_PROTECT_FOLDER);
#endif
if ((fexist_utf8(fconcat(
if ((fexist(fconcat(
OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/new.zip")))) { // a previous cache exists.. rename it
/* Remove OLD cache */
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.zip"))) {
if (UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.zip")) !=
0) {
if (fexist
(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.zip"))) {
if (remove
(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.zip")) != 0) {
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
"Cache: error while moving previous cache");
}
@@ -1017,27 +980,33 @@ void cache_init(cache_back * cache, httrackp * opt) {
hts_log_print(opt, LOG_DEBUG, "Cache: no cache found");
}
hts_log_print(opt, LOG_DEBUG, "Cache: size %d",
(int) fsize_utf8(
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.zip")));
(int)
fsize(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.zip")));
// charger index cache précédent
if ((!cache->ro &&
fsize_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.zip")) >
0) ||
(cache->ro &&
fsize_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.zip")) >
0)) {
if ((!cache->ro
&&
fsize(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.zip")) > 0)
|| (cache->ro
&&
fsize(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.zip")) > 0)
) {
if (!cache->ro) {
cache->zipInput = hts_unzOpen_utf8(
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.zip"));
cache->zipInput =
unzOpen(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.zip"));
} else {
cache->zipInput = hts_unzOpen_utf8(
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.zip"));
cache->zipInput =
unzOpen(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.zip"));
}
// Corrupted ZIP file ? Try to repair!
@@ -1057,8 +1026,6 @@ void cache_init(cache_back * cache, httrackp * opt) {
}
hts_log_print(opt, LOG_WARNING,
"Cache: damaged cache, trying to repair");
/* mztools has no UTF-8 hook, so repairing a corrupt cache under a
non-ASCII path_log fails cleanly (re-crawl), never forks a twin. */
if (unzRepair
(name,
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
@@ -1066,11 +1033,11 @@ void cache_init(cache_back * cache, httrackp * opt) {
StringBuff(opt->path_log),
"hts-cache/repair.tmp"),
&repaired, &repairedBytes) == Z_OK) {
UNLINK(name);
RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/repair.zip"),
name);
cache->zipInput = hts_unzOpen_utf8(name);
unlink(name);
rename(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/repair.zip"), name);
cache->zipInput = unzOpen(name);
hts_log_print(opt, LOG_WARNING,
"Cache: %d bytes successfully recovered in %d entries",
(int) repairedBytes, (int) repaired);
@@ -1166,8 +1133,8 @@ void cache_init(cache_back * cache, httrackp * opt) {
"Cache: error trying to open the cache");
}
} else if (fsize_utf8(reconcile_path(opt, "hts-cache/old.ndx")) > 0 ||
fsize_utf8(reconcile_path(opt, "hts-cache/new.ndx")) > 0) {
} else if (fsize(reconcile_path(opt, "hts-cache/old.ndx")) > 0 ||
fsize(reconcile_path(opt, "hts-cache/new.ndx")) > 0) {
/* pre-3.31 (2003) .dat/.ndx cache: import support removed */
hts_log_print(opt, LOG_ERROR,
"Cache: the pre-3.31 .dat/.ndx cache format is no longer "
@@ -1183,58 +1150,67 @@ void cache_init(cache_back * cache, httrackp * opt) {
#endif
if (!cache->ro) {
// ouvrir caches actuels
structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/"));
structcheck(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache/"));
{
/* Create ZIP file cache */
cache->zipOutput = (void *) hts_zipOpen_utf8(
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.zip"),
0);
cache->zipOutput =
(void *)
zipOpen(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.zip"), 0);
if (cache->zipOutput != NULL) {
// supprimer old.lst
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.lst")))
UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.lst"));
if (fexist
(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.lst")))
remove(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.lst"));
// renommer
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/new.lst")))
RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.lst"),
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.lst"));
if (fexist
(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.lst")))
rename(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.lst"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.lst"));
// ouvrir
cache->lst =
FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.lst"),
"wb");
fopen(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.lst"), "wb");
strcpybuff(opt->state.strc.path, StringBuff(opt->path_html));
opt->state.strc.lst = cache->lst;
// supprimer old.txt
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.txt")))
UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.txt"));
if (fexist
(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.txt")))
remove(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/old.txt"));
// renommer
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/new.txt")))
RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.txt"),
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/old.txt"));
if (fexist
(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.txt")))
rename(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.txt"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log),
"hts-cache/old.txt"));
// ouvrir
cache->txt =
FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "hts-cache/new.txt"),
"wb");
fopen(fconcat
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
"hts-cache/new.txt"), "wb");
if (cache->txt) {
fprintf(cache->txt,
"date\tsize'/'remotesize\tflags(request:Update,Range state:File response:Modified,Chunked,gZipped)\t");

View File

@@ -77,17 +77,17 @@ void infomsg(const char *msg) {
if (msg[2] != ' ') {
if ((msg[3] == ' ') || (msg[4] == ' ')) {
char cmd[32] = "-";
int p;
int p = 0;
sscanf(msg, "%30s", cmd + strlen(cmd));
/* try the flag as-is, then strip a trailing N as the numeric-arg
placeholder (cN -> c); this order keeps -%N from becoming -% */
while(cmd[p] == ' ')
p++;
sscanf(msg + p, "%s", cmd + strlen(cmd));
/* clears cN -> c */
if ((p = (int) strlen(cmd)) > 2)
if (cmd[p - 1] == 'N')
cmd[p - 1] = '\0';
/* finds alias (if any) */
p = optreal_find(cmd);
if (p < 0 && (int) strlen(cmd) > 2 &&
cmd[strlen(cmd) - 1] == 'N') {
cmd[strlen(cmd) - 1] = '\0';
p = optreal_find(cmd);
}
if (p >= 0) {
/* fings type of parameter: number,param,param concatenated,single cmd */
if (strcmp(opttype_value(p), "param") == 0)
@@ -648,8 +648,9 @@ void help(const char *app, int more) {
infomsg("Command-line specific options:");
infomsg
(" V execute system command after each files ($0 is the filename: -V \"rm \\$0\")");
infomsg(" %W use an external library function as a wrapper (-%W "
"myfoo.so[,myparameters])");
infomsg
(" %W use an external library function as a wrapper (-%W myfoo.so[,myparameters])");
/* infomsg(" %O do a chroot before setuid"); */
infomsg("");
infomsg("Details: Option N");
infomsg(" N0 Site-structure (default)");

View File

@@ -315,18 +315,9 @@ static void escape_url_parens(char *const s, const size_t size) {
strlcpybuff(s, buff, size);
}
/* Default port for lien's scheme (case-insensitive), 80 if absent/unknown; so
schemeless and protocol-relative //host links default to 80 (known gap). */
static int scheme_default_port(const char *lien) {
if (strfield(lien, "https:"))
return 443;
if (strfield(lien, "ftp:"))
return 21;
return 80;
}
/* Strip the scheme's own default port (80 http, 443 https, 21 ftp) from lien's
authority in place; :80 on https/ftp stays as a real port (#638, #614). */
/* Strip a default ":80" from lien's authority in place. Any spelling that
range-parses to 80 (":80", ":080") is dropped by its matched length, not a
hardcoded 3 chars; a value that merely wraps to 80 as an int (#614) stays. */
void hts_strip_default_port(char *lien, size_t size) {
char *a;
@@ -348,8 +339,7 @@ void hts_strip_default_port(char *lien, size_t size) {
b++;
saved = *b;
*b = '\0';
is_default =
hts_parse_url_port(a + 1, &port) && port == scheme_default_port(lien);
is_default = hts_parse_url_port(a + 1, &port) && port == 80;
*b = saved;
if (is_default) { // default port, strip it
char BIGSTK tempo[HTS_URLMAXSIZE * 2];

View File

@@ -1589,8 +1589,10 @@ static int st_identabs(httrackp *opt, int argc, char **argv) {
return 0;
}
/* Default-port strip is scheme-aware (#638), overflow-safe (#614): a scheme's
own default (any spelling) is dropped, a real port stays; guards #627. */
/* Default-port strip (#627): a genuine 80 (any spelling) is removed by its
matched length, host preserved; a non-80 port or one that only wraps to 80 as
a 32-bit int (#614) is left intact. Guards the old bug where ":080"/":0080"
dropped a hardcoded 3 chars and glued the leftover digits onto the host. */
static int st_stripport(httrackp *opt, int argc, char **argv) {
static const struct {
const char *in, *out;
@@ -1604,13 +1606,6 @@ static int st_stripport(httrackp *opt, int argc, char **argv) {
{"http://127.0.0.1:8080/x", "http://127.0.0.1:8080/x"},
{"http://127.0.0.1:4294967376/x", "http://127.0.0.1:4294967376/x"},
{"http://127.0.0.1/x", "http://127.0.0.1/x"},
{"https://127.0.0.1:443/x", "https://127.0.0.1/x"},
{"https://127.0.0.1:80/x", "https://127.0.0.1:80/x"},
// Scheme match is case-insensitive: HTTPS' default is 443, so :80 stays.
{"HTTPS://127.0.0.1:80/x", "HTTPS://127.0.0.1:80/x"},
{"ftp://127.0.0.1:21/x", "ftp://127.0.0.1/x"},
{"ftp://127.0.0.1:80/x", "ftp://127.0.0.1:80/x"},
{"http://127.0.0.1:443/x", "http://127.0.0.1:443/x"},
};
size_t k;

View File

@@ -1,6 +1,6 @@
--- mztools.c.orig 2024-01-27 14:07:18.636193212 +0100
+++ mztools.c 2024-01-27 14:09:55.356620093 +0100
@@ -10,10 +10,11 @@
@@ -10,6 +10,7 @@
#include <string.h>
#include "zlib.h"
#include "unzip.h"
@@ -8,11 +8,6 @@
#define READ_8(adr) ((unsigned char)*(adr))
#define READ_16(adr) ( READ_8(adr) | (READ_8(adr+1) << 8) )
-#define READ_32(adr) ( READ_16(adr) | (READ_16((adr)+2) << 16) )
+#define READ_32(adr) ((uLong) READ_16(adr) | ((uLong) READ_16((adr) + 2) << 16))
#define WRITE_8(buff, n) do { \
*((unsigned char*)(buff)) = (unsigned char) ((n) & 0xff); \
@@ -141,8 +142,8 @@
/* Central directory entry */
{

View File

@@ -6,7 +6,6 @@ set -euo pipefail
# Stripping a default :80 from a crawled link used to skip a hardcoded 3 chars
# (#627): ":080"/":0080" lost only ":80" and glued the rest onto the host
# (127.0.0.1:080/x -> 127.0.0.10/x), and a value wrapping to 80 as a 32-bit int
# (#614) was stripped as if it were the default. The default is scheme-aware
# (#638): an explicit :80 on https/ftp stays, :443/:21 strip under their own
# scheme. All assertions live in the engine self-test.
# (#614) was stripped as if it were the default. All assertions live in the
# engine self-test.
httrack -O /dev/null -#test=stripport | grep -q "stripport self-test OK"

View File

@@ -1,17 +1,15 @@
#!/bin/bash
#
# A single non-ASCII -O sets both path_html and path_log to "café" (#630). The
# logs (hts-log.txt/hts-err.txt) and the hts-cache must land there, not in an
# ANSI-mangled twin: on Windows path_log holds UTF-8 bytes the raw file calls
# read as the codepage. The --log-found audit greps logroot=café/hts-log.txt and
# --cache-under-logroot checks café/hts-cache/new.zip, so anything written to the
# twin fails them. POSIX has no twin, so this bites on the Windows CI leg (test
# 64), exercising only the code path elsewhere.
# logs (hts-log.txt/hts-err.txt) must land there, not in an ANSI-mangled twin:
# on Windows path_log holds UTF-8 bytes the raw file calls read as the codepage.
# The --errors/--files/--log-found audits all grep logroot=café/hts-log.txt, so
# a log written to the twin fails them. POSIX has no twin, so this bites on the
# Windows CI leg (like test 64). The cache twin is a separate, larger fix.
: "${top_srcdir:=..}"
bash "$top_srcdir/tests/local-crawl.sh" --outdir-intl 'café' --errors 0 --files 5 \
--found 'simple/basic.html' \
--log-found 'mirror complete in' \
--cache-under-logroot \
httrack 'BASEURL/simple/basic.html'

View File

@@ -118,9 +118,6 @@ while test "$pos" -lt "$nargs"; do
nopurge=1
audit+=("--no-purge")
;;
--cache-under-logroot)
audit+=("--cache-under-logroot")
;;
--tls)
tls=1
scheme=https
@@ -382,15 +379,6 @@ while test "$i" -lt "${#audit[@]}"; do
sed -e 's/.*[[:space:]]\([^ ]*\)[[:space:]]files written.*/\1/g')
assert_equals "checking files" "${audit[$i]}" "$nFiles"
;;
--cache-under-logroot)
# The cache must sit under path_log (logroot), not an ANSI-mangled twin
# of a non-ASCII -O dir (#630 cache half). Bites only on the Windows leg.
info "checking cache under logroot"
if test -e "${logroot}/hts-cache/new.zip"; then result "OK"; else
result "cache not under logroot (mangled twin?)"
exit 1
fi
;;
--found)
i=$((i + 1))
info "checking for ${audit[$i]}"