Compare commits

..

3 Commits

Author SHA1 Message Date
Xavier Roche
bf0c210d73 Tighten the #630 comments
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 09:12:27 +02:00
Xavier Roche
d5ab135649 Cache UTF-8 filefunc must keep 64-bit offsets and the cache dir 0700
The #630 cache-UTF-8 hook built a 32-bit zlib_filefunc_def via
fill_fopen_filefunc and called unzOpen2/zipOpen2. Minizip runs a 32-bit
filefunc through fill_zlib_filefunc64_32_def_from_filefunc32, which NULLs the
64-bit seek so every seek truncates the offset to uLong: on Windows LLP64 a
cache >=4GB hard-fails and >=2GB corrupts. The ANSI code this replaced went
through fill_fopen64_filefunc and never truncated. Keep 64-bit: fill the
zlib_filefunc64_def and override only zopen64_file with the UTF-8 opener, then
call unzOpen2_64/zipOpen2_64.

Also restore the POSIX cache-dir mode: the MKDIR macro creates with
HTS_ACCESS_FOLDER (0755), loosening the 0700 the explicit mkdir used. There is
no UTF-8 mkdir-with-mode, so keep the platform split - UTF-8 MKDIR on Windows
(which ignores the mode), mkdir(HTS_PROTECT_FOLDER) on POSIX.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 08:57:57 +02:00
Xavier Roche
4dfb512e10 Non-ASCII single -O drops the hts-cache into a mangled twin directory on Windows
PR #636 fixed the log half of #630 and left the cache for a follow-up. With a
single -O café, path_log holds UTF-8 bytes, but cache_init created hts-cache and
opened new.zip through ANSI calls, so on Windows the cache landed in a second
mangled café-twin directory beside the mirror, and a later --update read the
cache from the wrong place.

This routes every path_log filesystem op in cache_init and the reconcile helpers
through the UTF-8 wrappers (structcheck_utf8, MKDIR, FOPEN, UNLINK, RENAME,
fexist_utf8, fsize_utf8), and opens the cache ZIP through minizip's filefunc
entry points (zipOpen2/unzOpen2) backed by hts_fopen_utf8. On POSIX the wrappers
resolve to the same libc calls, so only Windows changes. The conversion is
all-or-nothing: a directory created UTF-8 but a ZIP opened ANSI would split the
cache, so the whole cache-open path flips together.

The corrupt-cache repair path stays ANSI. mztools' unzRepair takes no filefunc,
so a corrupt cache under a non-ASCII path fails to repair (the site is
re-crawled) rather than forking a twin.

Test 69 now also asserts the cache lands under the café directory through a new
--cache-under-logroot audit. Like the log assertion it only bites on the Windows
CI leg; on POSIX it exercises the code path without reproducing the mojibake.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 08:40:58 +02:00
19 changed files with 909 additions and 1107 deletions

View File

@@ -618,31 +618,3 @@ jobs:
echo "Fix locally with: git clang-format --binary clang-format-19 $base"
exit 1 ;;
esac
man-page-sync:
name: man page / html in sync
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
# html/httrack.man.html is groff-rendered from man/httrack.1 and committed.
# Rendering needs the full groff html device, so CI can't regenerate it;
# instead require the two to move together: a PR that touches httrack.1
# must also touch the html, catching the "regenerated roff, forgot html".
- name: httrack.1 changes must include html/httrack.man.html
run: |
set -euo pipefail
git fetch --no-tags origin \
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
base="origin/${{ github.base_ref }}"
changed="$(git diff --name-only "$base"...HEAD)"
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
if has man/httrack.1 && ! has html/httrack.man.html; then
echo "::error::man/httrack.1 changed but html/httrack.man.html did not."
echo "Regenerate it with: make -C man regen-man-html (needs the full groff package)."
exit 1
fi
echo "man/html sync OK."

View File

@@ -55,9 +55,6 @@ jobs:
query-filters:
- exclude:
id: cpp/world-writable-file-creation
# Models auth-bypass-by-spoofing; httrack has no auth surface, its +/- crawl filter is a mirror boundary, not a security one.
- exclude:
id: cpp/user-controlled-bypass
# Manual build: CodeQL traces the compiler, so build exactly what ships.
- name: Build

View File

@@ -219,23 +219,6 @@ that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
options will not help there. That is an
<a href="#identity">identity</a> problem.</p>
<h4>Filter wildcards</h4>
<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
bracket forms match narrower sets. The full table, with size and mime rules, is on
<a href="filters.html">the filters page</a>.</p>
<table class="tblRegular tableWidth" border="0">
<tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> &mdash; any URL ending <tt>.pdf</tt></td></tr>
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> &mdash; a directory-index page</td></tr>
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
<tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
<tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
<tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
<tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
<tr><td><tt>*[&lt;NN]</tt>, <tt>*[&gt;NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[&lt;5]</tt> skips GIFs under 5&nbsp;KB</td></tr>
<tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
</table>
<h3 id="limits">4. Limits and politeness</h3>
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
@@ -251,7 +234,7 @@ below let you go faster when you own the target, and slower when you do not.</p>
<tr><td><tt>--max-time (-E)</tt></td><td>Stop after N seconds of wall-clock time.</td></tr>
<tr><td><tt>--max-files (-m)</tt></td><td>Per-file size caps.</td></tr>
<tr><td><tt>--timeout (-T), --retries (-R), --min-rate (-J), --host-control (-H)</tt></td><td>Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.</td></tr>
<tr><td><tt>--max-pause (-G), --pause (-%G)</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
<tr><td><tt>--max-pause (-G), -%G</tt></td><td>Pause the mirror at N bytes, or pause between files, to spread the load.</td></tr>
</table>
<p><b>The security clamps.</b> To keep an accidental typo from turning into a flood,
@@ -274,8 +257,8 @@ really HTML.</p>
<tr><td><tt>--structure (-N)</tt></td><td>The local path and name layout. Presets are numeric, and you can also give a template such as <tt>--structure "%h%p/%n%q.%t"</tt>.</td></tr>
<tr><td><tt>--long-names (-L)</tt></td><td>Long names, 8.3 names, or ISO9660 for CD masters.</td></tr>
<tr><td><tt>--assume (-%A)</tt></td><td>Assume a MIME type for an extension, for example <tt>--assume php=text/html</tt>. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.</td></tr>
<tr><td><tt>--delayed-type-check (-%N), --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
<tr><td><tt>--include-query-string (-%q), --strip-query (-%g)</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
<tr><td><tt>-%N, --cached-delayed-type-check (-%D), --check-type (-u), -%t</tt></td><td>When and how the content type is checked, and whether the original extension is kept.</td></tr>
<tr><td><tt>--include-query-string (-%q), -%g</tt></td><td>Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.</td></tr>
</table>
<p>The <tt>-N</tt> presets are built from modular arithmetic on the name fields, so
@@ -346,7 +329,7 @@ across links at the same time.</p>
<tr><td><tt>--proxy (-P)</tt></td><td>Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: <tt>-P host:8080</tt>, <tt>-P socks5://host:1080</tt>, <tt>-P connect://host:443</tt>, with optional <tt>user:pass@</tt>.</td></tr>
<tr><td><tt>--httpproxy-ftp (-%f)</tt></td><td>Send FTP requests through the HTTP proxy.</td></tr>
<tr><td><tt>--protocol (-@i)</tt></td><td>Prefer IPv4 or IPv6.</td></tr>
<tr><td><tt>--http-10 (-%h), --keep-alive (-%k), --disable-compression (-%z)</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
<tr><td><tt>--http-10 (-%h), --keep-alive (-%k), -%z</tt></td><td>Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.</td></tr>
<tr><td><tt>--bind (-%b), --tolerant (-%B)</tt></td><td>Bind to a local address, and accept technically-bogus responses some servers send.</td></tr>
</table>
@@ -370,7 +353,7 @@ away and you lose the ability to continue or update the mirror.</p>
<tr><td><tt>--update</tt></td><td>Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.</td></tr>
<tr><td><tt>--purge-old=0 (-X0)</tt></td><td>Do not purge. By default an update deletes local files that are no longer part of the mirror; <tt>--purge-old=0</tt> keeps them.</td></tr>
<tr><td><tt>--cache (-C)</tt></td><td>Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.</td></tr>
<tr><td><tt>--urlhack (-%u), --keep-www-prefix (-%j), --keep-double-slashes (-%o), --keep-query-order (-%y), --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
<tr><td><tt>--urlhack (-%u), -%j, -%o, -%y, --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)</tt></td><td>URL-deduplication behavior and cache storage details.</td></tr>
<tr><td><tt>--debug-cache (-#C), --repair-cache (-#R), --clean</tt></td><td>Inspect the cache, repair its ZIP, and erase cache plus logs.</td></tr>
</table>
@@ -421,16 +404,14 @@ host and stops the crawl; start from the final URL, or add
rule wins.</small></p>
<h4>Download the PDFs on a site</h4>
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
<p><tt>httrack https://example.com/ --path mydir</tt><br>
<small>There is no PDF-only crawl. HTTrack discovers PDF links by parsing the
site's HTML pages, so a <tt>"-*" "+example.com/*.pdf"</tt> filter blocks the very
pages that carry the links and grabs only the PDFs linked from the front page. Let
it crawl the site: the HTML pages come along as the scaffolding, and every reachable
PDF is saved with them. If some PDFs live on another host (a CDN or a docs
subdomain), allow that host too, for example
<tt>"+docs.example.com/*.pdf"</tt>.</small></p>
<h4>Keep page requisites, including off-host images</h4>
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>

File diff suppressed because it is too large Load Diff

View File

@@ -95,7 +95,7 @@ ${do:end-if}
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_G44}</em></h2>
<h2 align="center"><em>Select URLs</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}

View File

@@ -77,7 +77,7 @@ ${do:end-if}
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_J9}</em></h2>
<h2 align="center"><em>Start</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}

View File

@@ -16,11 +16,6 @@ regen-man: makeman.sh $(top_builddir)/src/httrack$(EXEEXT)
# Render html/httrack.man.html from httrack.1. Needs the groff html device
# (Debian: full "groff" package, not "groff-base"). Run by hand: make -C man regen-man-html
# Strip groff's version-stamp and creation-date comments so the committed file
# doesn't churn across groff versions or rebuilds; the ci man-page-sync guard
# only requires this file to change whenever httrack.1 does.
regen-man-html: httrack.1
groff -t -man -Thtml $(srcdir)/httrack.1 \
| sed -e 's/groff version [0-9][0-9.]*/groff/' -e '/^<!-- CreationDate:/d' \
> $(top_srcdir)/html/httrack.man.html
groff -t -man -Thtml $(srcdir)/httrack.1 > $(top_srcdir)/html/httrack.man.html
.PHONY: regen-man-html

View File

@@ -3,7 +3,7 @@
.\"
.\" This file is generated by man/makeman.sh; do not edit by hand.
.\" SPDX-License-Identifier: GPL-3.0-or-later
.TH httrack 1 "22 July 2026" "httrack website copier"
.TH httrack 1 "21 July 2026" "httrack website copier"
.SH NAME
httrack \- offline browser : copy websites to a local directory
.SH SYNOPSIS
@@ -37,7 +37,6 @@ httrack \- offline browser : copy websites to a local directory
[ \fB\-%L, \-\-list\fR ]
[ \fB\-%S, \-\-urllist\fR ]
[ \fB\-NN, \-\-structure[=N]\fR ]
[ \fB\-%N, \-\-delayed\-type\-check\fR ]
[ \fB\-%D, \-\-cached\-delayed\-type\-check\fR ]
[ \fB\-%M, \-\-mime\-html\fR ]
[ \fB\-LN, \-\-long\-names[=N]\fR ]
@@ -190,7 +189,7 @@ structure type (0 *original structure, 1+: see below) (\-\-structure[=N])
.br
or user defined structure (\-N "%h%p/%n%q.%t")
.IP \-%N
delayed type check, don't make any link test but wait for files download to start instead (experimental) (%N0 don't use, %N1 use for unknown extensions, * %N2 always use) (\-\-delayed\-type\-check)
delayed type check, don't make any link test but wait for files download to start instead (experimental) (%N0 don't use, %N1 use for unknown extensions, * %N2 always use)
.IP \-%D
cached delayed type check, don't wait for remote type during updates, to speedup them (%D0 wait, * %D1 don't wait) (\-\-cached\-delayed\-type\-check)
.IP \-%M

View File

@@ -1617,15 +1617,14 @@ int back_add_if_not_exists(struct_back * sback, httrackp * opt,
back_clean(opt, cache, sback); /* first cleanup the backlog to ensure that we have some entry left */
if (!back_exist(sback, opt, adr, fil, save)) {
return back_add(sback, opt, cache, adr, fil, save, referer_adr, referer_fil,
test, HTS_FALSE);
test);
}
return 0;
}
int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
const char *adr, const char *fil, const char *save,
const char *referer_adr, const char *referer_fil, int test,
hts_boolean refetch_whole) {
int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char *adr,
const char *fil, const char *save, const char *referer_adr, const char *referer_fil,
int test) {
lien_back *const back = sback->lnk;
const int back_max = sback->count;
int p = 0;
@@ -1702,12 +1701,6 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
else if (strcmp(back[p].url_sav, BACK_ADD_TEST2) == 0) // test en GET
back[p].head_request = 2; // test en get
/* Forced whole refetch (#581): drop the stale temp-ref and skip the resume
branches below, so a surviving partial can't Range-loop. */
if (refetch_whole) {
url_savename_refname_remove(opt, adr, fil);
}
/* Stop requested - abort backing */
/* For update mode: second check after cache lookup not to lose all previous cache data ! */
if (opt->state.stop && !opt->is_update) {
@@ -1963,9 +1956,8 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
}
}
/* Not in cache ; maybe in temporary cache ? Warning: non-movable
"url_sav" (skipped on a forced whole refetch, #581) */
else if (!refetch_whole &&
back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
"url_sav" */
else if (back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
const LLint file_size = fsize_utf8(itemback->url_sav);
/* Found file on disk */
@@ -1999,9 +1991,8 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
freet(itemback); /* delete item */
itemback = NULL;
}
/* Not in cache or temporary cache ; found on disk ? (hack)
(skipped on a forced whole refetch, #581) */
else if (!refetch_whole && fexist_utf8(save)) {
/* Not in cache or temporary cache ; found on disk ? (hack) */
else if (fexist_utf8(save)) {
const LLint sz = fsize_utf8(save);
// Bon, là il est possible que le fichier ait été partiellement transféré
@@ -3856,8 +3847,6 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
deletehttp(&back[i].r);
back[i].r.soc = INVALID_SOCKET;
back[i].r.statuscode = STATUSCODE_NON_FATAL;
back[i].r.refetch_wholefile =
HTS_TRUE; // retry whole, no Range (#581)
strcpybuff(back[i].r.msg,
"Bogus 304 on resume, restarting");
back[i].status = STATUS_READY;
@@ -4129,9 +4118,6 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
}
back[i].r.soc = INVALID_SOCKET;
back[i].r.statuscode = STATUSCODE_NON_FATAL;
// the resume was rejected: the retry must GET the whole
// file, never re-Range a surviving partial/ref (#581)
back[i].r.refetch_wholefile = HTS_TRUE;
if (strnotempty(back[i].r.msg))
strcpybuff(back[i].r.msg,
"Error attempting to solve status 206 (partial file)");

View File

@@ -83,12 +83,9 @@ HTS_INLINE int back_exist(struct_back * sback, httrackp * opt, const char *adr,
const char *fil, const char *sav);
int back_nsoc(const struct_back * sback);
int back_nsoc_overall(const struct_back * sback);
/* refetch_whole: force a whole-file GET, ignoring any partial/temp-ref resume
(set when a prior 206 was rejected as unusable, #581). */
int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
const char *adr, const char *fil, const char *save,
const char *referer_adr, const char *referer_fil, int test,
hts_boolean refetch_whole);
int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char *adr,
const char *fil, const char *save, const char *referer_adr, const char *referer_fil,
int test);
int back_add_if_not_exists(struct_back * sback, httrackp * opt,
cache_back * cache, const char *adr, const char *fil, const char *save,
const char *referer_adr, const char *referer_fil, int test);

View File

@@ -3293,10 +3293,10 @@ int back_fill(struct_back * sback, httrackp * opt, cache_back * cache,
if (ok) {
if (!back_exist
(sback, opt, heap(p)->adr, heap(p)->fil, heap(p)->sav)) {
if (back_add(sback, opt, cache, heap(p)->adr, heap(p)->fil,
heap(p)->sav, heap(heap(p)->precedent)->adr,
heap(heap(p)->precedent)->fil, heap(p)->testmode,
heap(p)->refetch_whole) == -1) {
if (back_add
(sback, opt, cache, heap(p)->adr, heap(p)->fil, heap(p)->sav,
heap(heap(p)->precedent)->adr, heap(heap(p)->precedent)->fil,
heap(p)->testmode) == -1) {
hts_log_print(opt, LOG_DEBUG,
"error: unable to add more links through back_add for back_fill");
#if BDEBUG==1

View File

@@ -77,17 +77,17 @@ void infomsg(const char *msg) {
if (msg[2] != ' ') {
if ((msg[3] == ' ') || (msg[4] == ' ')) {
char cmd[32] = "-";
int p;
int p = 0;
sscanf(msg, "%30s", cmd + strlen(cmd));
/* try the flag as-is, then strip a trailing N as the numeric-arg
placeholder (cN -> c); this order keeps -%N from becoming -% */
while(cmd[p] == ' ')
p++;
sscanf(msg + p, "%s", cmd + strlen(cmd));
/* clears cN -> c */
if ((p = (int) strlen(cmd)) > 2)
if (cmd[p - 1] == 'N')
cmd[p - 1] = '\0';
/* finds alias (if any) */
p = optreal_find(cmd);
if (p < 0 && (int) strlen(cmd) > 2 &&
cmd[strlen(cmd) - 1] == 'N') {
cmd[strlen(cmd) - 1] = '\0';
p = optreal_find(cmd);
}
if (p >= 0) {
/* fings type of parameter: number,param,param concatenated,single cmd */
if (strcmp(opttype_value(p), "param") == 0)
@@ -648,8 +648,9 @@ void help(const char *app, int more) {
infomsg("Command-line specific options:");
infomsg
(" V execute system command after each files ($0 is the filename: -V \"rm \\$0\")");
infomsg(" %W use an external library function as a wrapper (-%W "
"myfoo.so[,myparameters])");
infomsg
(" %W use an external library function as a wrapper (-%W myfoo.so[,myparameters])");
/* infomsg(" %O do a chroot before setuid"); */
infomsg("");
infomsg("Details: Option N");
infomsg(" N0 Site-structure (default)");

View File

@@ -615,9 +615,9 @@ int url_savename(lien_adrfilsave *const afs,
strcpybuff(current.fil, fil_complete);
// ajouter dans le backing le fichier en mode test
// savename: rien car en mode test
if (back_add(sback, opt, cache, current.adr, current.fil,
BACK_ADD_TEST, referer_adr, referer_fil, 1,
HTS_FALSE) != -1) {
if (back_add
(sback, opt, cache, current.adr, current.fil, BACK_ADD_TEST,
referer_adr, referer_fil, 1) != -1) {
int b;
b = back_index(opt, sback, current.adr, current.fil, BACK_ADD_TEST);
@@ -706,10 +706,7 @@ int url_savename(lien_adrfilsave *const afs,
if (!hts_wait_available_socket(sback, opt,
cache, ptr))
return -1;
if (back_add(sback, opt, cache, moved.adr,
moved.fil, methode, referer_adr,
referer_fil, 1,
HTS_FALSE) != -1) { // OK
if (back_add(sback, opt, cache, moved.adr, moved.fil, methode, referer_adr, referer_fil, 1) != -1) { // OK
hts_log_print(opt, LOG_DEBUG,
"(during prefetch) %s (%d) to link %s at %s%s",
back[b].r.msg,
@@ -728,8 +725,7 @@ int url_savename(lien_adrfilsave *const afs,
has_been_moved = 1; // sinon ne pas forcer has_been_moved car non déplacé
petits_tours++;
//
} else { // sinon on fait rien et on s'en va..
// (ftp etc)
} else { // sinon on fait rien et on s'en va.. (ftp etc)
hts_log_print(opt, LOG_DEBUG,
"Warning: Savename redirect backing error at %s%s",
moved.adr, moved.fil);
@@ -800,6 +796,7 @@ int url_savename(lien_adrfilsave *const afs,
hts_log_print(opt, LOG_ERROR,
"Unexpected savename backing error at %s%s", adr,
fil_complete);
}
// restaurer
opt->state._hts_in_html_parsing = hihp;

View File

@@ -658,9 +658,6 @@ struct htsblk {
int debugid; /**< connection debug id */
/* */
htsrequest req; /**< parameters used for the request */
/* Restart-whole signal: a resume this response rejected (unusable 206) must
retry with no Range, else a surviving partial/temp-ref loops (#581). */
hts_boolean refetch_wholefile;
/*char digest[32+2]; // md5 digest generated by the engine ("" if none) */
};
@@ -684,9 +681,6 @@ struct lien_url {
char link_import; /**< imported after a move; skip the usual up/down rules */
int retry; /**< remaining retries */
int testmode; /**< test only: send just a HEAD */
hts_boolean
refetch_whole; /**< force a whole-file GET, ignoring any partial/temp-ref
resume, so a rejected 206 can't loop (#581) */
};
/* A file being fetched in the background. */

View File

@@ -315,18 +315,9 @@ static void escape_url_parens(char *const s, const size_t size) {
strlcpybuff(s, buff, size);
}
/* Default port for lien's scheme (case-insensitive), 80 if absent/unknown; so
schemeless and protocol-relative //host links default to 80 (known gap). */
static int scheme_default_port(const char *lien) {
if (strfield(lien, "https:"))
return 443;
if (strfield(lien, "ftp:"))
return 21;
return 80;
}
/* Strip the scheme's own default port (80 http, 443 https, 21 ftp) from lien's
authority in place; :80 on https/ftp stays as a real port (#638, #614). */
/* Strip a default ":80" from lien's authority in place. Any spelling that
range-parses to 80 (":80", ":080") is dropped by its matched length, not a
hardcoded 3 chars; a value that merely wraps to 80 as an int (#614) stays. */
void hts_strip_default_port(char *lien, size_t size) {
char *a;
@@ -348,8 +339,7 @@ void hts_strip_default_port(char *lien, size_t size) {
b++;
saved = *b;
*b = '\0';
is_default =
hts_parse_url_port(a + 1, &port) && port == scheme_default_port(lien);
is_default = hts_parse_url_port(a + 1, &port) && port == 80;
*b = saved;
if (is_default) { // default port, strip it
char BIGSTK tempo[HTS_URLMAXSIZE * 2];
@@ -3756,9 +3746,6 @@ int hts_mirror_check_moved(htsmoduleStruct * str,
heap_top()->retry = heap(ptr)->retry - 1; // moins 1 retry!
heap_top()->premier = heap(ptr)->premier;
heap_top()->precedent = heap(ptr)->precedent;
// a rejected resume (unusable 206) must refetch whole, no Range
// (#581)
heap_top()->refetch_whole = r->refetch_wholefile;
} else { // oups erreur, plus de mémoire!!
return 0;
}
@@ -3990,17 +3977,18 @@ int hts_mirror_wait_for_next_file(htsmoduleStruct * str,
#if BDEBUG==1
printf("crash backing: %s%s\n", heap(ptr)->adr, heap(ptr)->fil);
#endif
if (back_add(sback, opt, cache, urladr(), urlfil(), savename(),
heap(heap(ptr)->precedent)->adr,
heap(heap(ptr)->precedent)->fil, heap(ptr)->testmode,
heap(ptr)->refetch_whole) == -1) {
if (back_add
(sback, opt, cache, urladr(), urlfil(), savename(),
heap(heap(ptr)->precedent)->adr, heap(heap(ptr)->precedent)->fil,
heap(ptr)->testmode) == -1) {
printf("PANIC! : Crash adding error, unexpected error found.. [%d]\n",
__LINE__);
#if BDEBUG==1
printf("error while crash adding\n");
#endif
hts_log_print(opt, LOG_ERROR, "Unexpected backing error for %s%s",
urladr(), urlfil());
hts_log_print(opt, LOG_ERROR, "Unexpected backing error for %s%s", urladr(),
urlfil());
}
}
#if BDEBUG==1

View File

@@ -1589,8 +1589,10 @@ static int st_identabs(httrackp *opt, int argc, char **argv) {
return 0;
}
/* Default-port strip is scheme-aware (#638), overflow-safe (#614): a scheme's
own default (any spelling) is dropped, a real port stays; guards #627. */
/* Default-port strip (#627): a genuine 80 (any spelling) is removed by its
matched length, host preserved; a non-80 port or one that only wraps to 80 as
a 32-bit int (#614) is left intact. Guards the old bug where ":080"/":0080"
dropped a hardcoded 3 chars and glued the leftover digits onto the host. */
static int st_stripport(httrackp *opt, int argc, char **argv) {
static const struct {
const char *in, *out;
@@ -1604,13 +1606,6 @@ static int st_stripport(httrackp *opt, int argc, char **argv) {
{"http://127.0.0.1:8080/x", "http://127.0.0.1:8080/x"},
{"http://127.0.0.1:4294967376/x", "http://127.0.0.1:4294967376/x"},
{"http://127.0.0.1/x", "http://127.0.0.1/x"},
{"https://127.0.0.1:443/x", "https://127.0.0.1/x"},
{"https://127.0.0.1:80/x", "https://127.0.0.1:80/x"},
// Scheme match is case-insensitive: HTTPS' default is 443, so :80 stays.
{"HTTPS://127.0.0.1:80/x", "HTTPS://127.0.0.1:80/x"},
{"ftp://127.0.0.1:21/x", "ftp://127.0.0.1/x"},
{"ftp://127.0.0.1:80/x", "ftp://127.0.0.1:80/x"},
{"http://127.0.0.1:443/x", "http://127.0.0.1:443/x"},
};
size_t k;

View File

@@ -1,6 +1,6 @@
--- mztools.c.orig 2024-01-27 14:07:18.636193212 +0100
+++ mztools.c 2024-01-27 14:09:55.356620093 +0100
@@ -10,10 +10,11 @@
@@ -10,6 +10,7 @@
#include <string.h>
#include "zlib.h"
#include "unzip.h"
@@ -8,11 +8,6 @@
#define READ_8(adr) ((unsigned char)*(adr))
#define READ_16(adr) ( READ_8(adr) | (READ_8(adr+1) << 8) )
-#define READ_32(adr) ( READ_16(adr) | (READ_16((adr)+2) << 16) )
+#define READ_32(adr) ((uLong) READ_16(adr) | ((uLong) READ_16((adr) + 2) << 16))
#define WRITE_8(buff, n) do { \
*((unsigned char*)(buff)) = (unsigned char) ((n) & 0xff); \
@@ -141,8 +142,8 @@
/* Central directory entry */
{

View File

@@ -6,7 +6,6 @@ set -euo pipefail
# Stripping a default :80 from a crawled link used to skip a hardcoded 3 chars
# (#627): ":080"/":0080" lost only ":80" and glued the rest onto the host
# (127.0.0.1:080/x -> 127.0.0.10/x), and a value wrapping to 80 as a 32-bit int
# (#614) was stripped as if it were the default. The default is scheme-aware
# (#638): an explicit :80 on https/ftp stays, :443/:21 strip under their own
# scheme. All assertions live in the engine self-test.
# (#614) was stripped as if it were the default. All assertions live in the
# engine self-test.
httrack -O /dev/null -#test=stripport | grep -q "stripport self-test OK"

View File

@@ -16,12 +16,11 @@ root=$(nativepath "${testdir}/server-root")
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
# Windows-only repro of #581 (a hard-killed pass 1 + repaired cache loses the
# file). The engine now forces a no-Range whole refetch on the restart, but that
# fix is unverified on Windows CI, and this test's pass-1 interrupt/port-race
# behavior there is unconfirmed; lift this skip on a Windows runner to verify.
# On Windows the pass-1 interrupt is a hard kill (MSYS can't signal a native
# exe) and the restart-whole path fails on the repaired cache (#581) -- the very
# bug this exercises; skip until the engine fix lands.
if is_windows; then
echo "Windows: #581 fix unverified on CI, skipping"
echo "Windows: restart-whole fails on a repaired cache (#581), skipping"
exit 77
fi