mirror of
https://github.com/xroche/httrack.git
synced 2026-07-23 09:09:05 +03:00
Compare commits
2 Commits
prose-typo
...
fix-help-N
| Author | SHA1 | Date | |
|---|---|---|---|
|
44036f99c0 | ||
|
|
555d79f56c |
28
.github/workflows/ci.yml
vendored
28
.github/workflows/ci.yml
vendored
@@ -618,31 +618,3 @@ jobs:
|
||||
echo "Fix locally with: git clang-format --binary clang-format-19 $base"
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
man-page-sync:
|
||||
name: man page / html in sync
|
||||
if: github.event_name == 'pull_request'
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
# html/httrack.man.html is groff-rendered from man/httrack.1 and committed.
|
||||
# Rendering needs the full groff html device, so CI can't regenerate it;
|
||||
# instead require the two to move together: a PR that touches httrack.1
|
||||
# must also touch the html, catching the "regenerated roff, forgot html".
|
||||
- name: httrack.1 changes must include html/httrack.man.html
|
||||
run: |
|
||||
set -euo pipefail
|
||||
git fetch --no-tags origin \
|
||||
"+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
|
||||
base="origin/${{ github.base_ref }}"
|
||||
changed="$(git diff --name-only "$base"...HEAD)"
|
||||
has() { printf '%s\n' "$changed" | grep -qx "$1"; }
|
||||
if has man/httrack.1 && ! has html/httrack.man.html; then
|
||||
echo "::error::man/httrack.1 changed but html/httrack.man.html did not."
|
||||
echo "Regenerate it with: make -C man regen-man-html (needs the full groff package)."
|
||||
exit 1
|
||||
fi
|
||||
echo "man/html sync OK."
|
||||
|
||||
3
.github/workflows/codeql.yml
vendored
3
.github/workflows/codeql.yml
vendored
@@ -55,9 +55,6 @@ jobs:
|
||||
query-filters:
|
||||
- exclude:
|
||||
id: cpp/world-writable-file-creation
|
||||
# Models auth-bypass-by-spoofing; httrack has no auth surface, its +/- crawl filter is a mirror boundary, not a security one.
|
||||
- exclude:
|
||||
id: cpp/user-controlled-bypass
|
||||
|
||||
# Manual build: CodeQL traces the compiler, so build exactly what ships.
|
||||
- name: Build
|
||||
|
||||
@@ -217,7 +217,7 @@ school or
|
||||
shows. They might do that because they are connected through expensive modem connection,
|
||||
or because they would like to consult pages while travelling, or archive sites that may be
|
||||
removed
|
||||
one day, make some data mining, compiling information ("if only I could find this
|
||||
one day, make some data mining, comiling information ("if only I could find this
|
||||
website I saw one day.."). <br>
|
||||
There are many good reasons to mirror websites, and this helps many good people.<br>
|
||||
As a webmaster, you might be interested to use such tools, too: test broken links, move a
|
||||
@@ -229,7 +229,7 @@ test the webserver response and performances, index it..<br>
|
||||
Anyway, bandwidth abuse can be a problem. If your site is regularly "clobbered"
|
||||
by evil downloaders, you have <br>
|
||||
various solutions. You have radical solutions, and intermediate solutions. I strongly
|
||||
recommend not to use<br>
|
||||
recomment not to use<br>
|
||||
radical solutions, because of the previous remarks (good people often mirror websites).<br>
|
||||
<br>
|
||||
In general, for all solutions,<br>
|
||||
@@ -244,7 +244,7 @@ or, to be extreme: if you unplug the wire, there will be no bandwidth abuse<br>
|
||||
Good: Will work with good people. Many good people just don't KNOW that they can slow down
|
||||
a network.<br>
|
||||
Bad: Will **only** work with good people<br>
|
||||
How to do: Obvious - place a note, a warning, an article, a draw, a poem or whatever you
|
||||
How to do: Obvious - place a note, a warning, an article, a draw, a poeme or whatever you
|
||||
want<br>
|
||||
<br>
|
||||
</li><li>Use "robots.txt" file<br>
|
||||
@@ -266,7 +266,7 @@ Good: Efficient<br>
|
||||
Bad: Multiple users behind proxies will be slow down, not really easy to setup<br>
|
||||
How to do: Depends on webserver. Might be done with low-level IP rules (QoS)<br>
|
||||
<br>
|
||||
</li><li>Prioritize small files, against large files<br>
|
||||
</li><li>Priorize small files, against large files<br>
|
||||
Good: Efficient if large files are the cause of abuse<br>
|
||||
Bad: Not always efficient<br>
|
||||
How to do: Depends on the webserver<br>
|
||||
@@ -283,7 +283,7 @@ How to do: Use routine QoS (fair queuing), or webserver options<br>
|
||||
<br>
|
||||
</li><li>Use technical tricks (like javascript) to hide URLs<br>
|
||||
Good: Efficient<br>
|
||||
Bad: The most efficient tricks will also cause your website to be heavy, and not
|
||||
Bad: The most efficient tricks will also cause your website to he heavy, and not
|
||||
user-friendly (and therefore less attractive, even for surfing users). Remember: clients
|
||||
or visitors might want to consult offline your website. Advanced users will also be still
|
||||
able to note the URLs and catch them. Will not work on non-javascript browsers. It will
|
||||
@@ -335,7 +335,7 @@ Example: Use things like
|
||||
</li><li>Use technical tricks to temporarily ban IPs<br>
|
||||
Good: Efficient<br>
|
||||
Bad: Radical (your site will only be available online for all users), not easy to setup<br>
|
||||
How to do: Create fake links with "killing" targets<br>
|
||||
How to to: Create fake links with "killing" targets<br>
|
||||
Example: Use things like <a href="killme.cgi"><nothing></a>
|
||||
(again an example in php:)<br>
|
||||
<tt>
|
||||
@@ -417,7 +417,7 @@ Example:<br>
|
||||
</li><li>Another one is to create images of emails<br>
|
||||
Good: Efficient, does not require javascript<br>
|
||||
Bad: There is still the problem of the link (mailto:), images are bigger than text, and it can cause problems for blind people (a good solution is use an ALT attribute with the email written like "smith at mycompany dot com")<br>
|
||||
How to do: Not so obvious if you do not want to create images by yourself<br>
|
||||
How to do: Not so obvious of you do not want to create images by yourself<br>
|
||||
Example: (php, Unix)<br>
|
||||
|
||||
<tt>
|
||||
@@ -491,7 +491,7 @@ echo <br>
|
||||
</li><li>You can also create temporary email aliases, each week, for all users<br>
|
||||
Good: Efficient, and you can give your real email in your reply-to address<br>
|
||||
Bad: Temporary emails<br>
|
||||
How to do: Not so hard to do<br>
|
||||
How to do: Not so hard todo<br>
|
||||
Example: (script & php, Unix)<br>
|
||||
|
||||
<tt>
|
||||
|
||||
@@ -219,23 +219,6 @@ that a <tt>403 Forbidden</tt> is a server refusal, not a robots rule: robots
|
||||
options will not help there. That is an
|
||||
<a href="#identity">identity</a> problem.</p>
|
||||
|
||||
<h4>Filter wildcards</h4>
|
||||
<p>Inside a filter pattern, <tt>*</tt> matches any run of characters; a few
|
||||
bracket forms match narrower sets. The full table, with size and mime rules, is on
|
||||
<a href="filters.html">the filters page</a>.</p>
|
||||
<table class="tblRegular tableWidth" border="0">
|
||||
<tr class="tblHeaderColor"><td><b>Wildcard</b></td><td><b>Matches</b></td><td><b>Example</b></td></tr>
|
||||
<tr><td><tt>*</tt></td><td>any run of characters</td><td><tt>+*.pdf</tt> — any URL ending <tt>.pdf</tt></td></tr>
|
||||
<tr><td><tt>*[file]</tt>, <tt>*[name]</tt></td><td>one path segment (any char but <tt>/</tt> and <tt>?</tt>)</td><td><tt>example.com/*[file]/</tt> — a directory-index page</td></tr>
|
||||
<tr><td><tt>*[path]</tt></td><td>a path, slashes allowed (any char but <tt>?</tt>)</td><td><tt>example.com/*[path].zip</tt></td></tr>
|
||||
<tr><td><tt>*[param]</tt></td><td>an optional query string</td><td><tt>page.html*[param]</tt> matches with or without <tt>?...</tt></td></tr>
|
||||
<tr><td><tt>*[a,b,c]</tt></td><td>any one character in the set</td><td><tt>*[a,b,c].txt</tt></td></tr>
|
||||
<tr><td><tt>*[a-z]</tt></td><td>any one character in the range</td><td><tt>img*[0-9].gif</tt></td></tr>
|
||||
<tr><td><tt>*[\x]</tt></td><td>the literal character x (escapes <tt>* [ ] \</tt>)</td><td><tt>*[\*]</tt> matches a real <tt>*</tt></td></tr>
|
||||
<tr><td><tt>*[<NN]</tt>, <tt>*[>NN]</tt></td><td>file size in KB below / above NN</td><td><tt>-*.gif*[<5]</tt> skips GIFs under 5 KB</td></tr>
|
||||
<tr><td><tt>*[]</tt></td><td>end anchor: nothing may follow</td><td><tt>*.html*[]</tt> rejects <tt>i.html?p=1</tt></td></tr>
|
||||
</table>
|
||||
|
||||
<h3 id="limits">4. Limits and politeness</h3>
|
||||
|
||||
<p>HTTrack ships cautious on purpose: it is easy to hammer a small site by
|
||||
@@ -421,16 +404,14 @@ host and stops the crawl; start from the final URL, or add
|
||||
rule wins.</small></p>
|
||||
|
||||
<h4>Download the PDFs on a site</h4>
|
||||
<p><tt>httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir</tt><br>
|
||||
<small>HTTrack finds PDFs by parsing the site's HTML, so a plain
|
||||
<tt>"-*" "+example.com/*.pdf"</tt> is wrong: it prunes the pages that carry the
|
||||
links and keeps only PDFs reachable from the front page. Instead admit the HTML as
|
||||
scaffolding (<tt>*.html</tt> and <tt>*[path]/</tt> for directory-index pages at any
|
||||
depth, e.g. <tt>docs/</tt> or <tt>a/b/deep/</tt>; <tt>*[file]/</tt> would stop at one
|
||||
level), keep the PDFs, and let <tt>-*</tt> drop everything else (images,
|
||||
archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not
|
||||
included by default; allow that host too, e.g. <tt>"+docs.example.com/*.pdf"</tt>,
|
||||
or widen to <tt>"+*.pdf"</tt> for PDFs anywhere.</small></p>
|
||||
<p><tt>httrack https://example.com/ --path mydir</tt><br>
|
||||
<small>There is no PDF-only crawl. HTTrack discovers PDF links by parsing the
|
||||
site's HTML pages, so a <tt>"-*" "+example.com/*.pdf"</tt> filter blocks the very
|
||||
pages that carry the links and grabs only the PDFs linked from the front page. Let
|
||||
it crawl the site: the HTML pages come along as the scaffolding, and every reachable
|
||||
PDF is saved with them. If some PDFs live on another host (a CDN or a docs
|
||||
subdomain), allow that host too, for example
|
||||
<tt>"+docs.example.com/*.pdf"</tt>.</small></p>
|
||||
|
||||
<h4>Keep page requisites, including off-host images</h4>
|
||||
<p><tt>httrack https://example.com/blog/ --near --path mydir</tt><br>
|
||||
|
||||
@@ -111,7 +111,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
starts links, the default mode is to mirror these links - i.e. if one of your start page is
|
||||
www.example.com/test/index.html, all links starting with www.example.com/test/ will be
|
||||
accepted. But links directly in www.example.com/.. will not be accepted, however, because
|
||||
they are in a higher structure. This prevent HTTrack from mirroring the whole site. (All
|
||||
they are in a higher strcuture. This prevent HTTrack from mirroring the whole site. (All
|
||||
files in structure levels equal or lower than the primary links will be retrieved.)<br>
|
||||
</i>
|
||||
<br>
|
||||
@@ -127,15 +127,13 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
an authorization filter, like <b><tt>+*.gif</tt></b>. The pattern is a plus (this one: <b><tt>+</tt></b>),
|
||||
followed by a pattern composed of letters and wildcards (this one: <b><tt>*</tt></b>).
|
||||
<br><br>
|
||||
To forbid a family of links, define
|
||||
To forbide a family of links, define
|
||||
an authorization filter, like <b><tt>-*.gif</tt></b>. The pattern is a dash (this one: <b><tt>-</tt></b>),
|
||||
followed by a the same kind of pattern as for the authorization filter.
|
||||
<br><br>
|
||||
Example: +*.gif will accept all files finished by .gif<br>
|
||||
Example: -*.gif will refuse all files finished by .gif<br>
|
||||
<br>
|
||||
To see which rule accepted or blocked a given URL, run HTTrack with the <b><tt>--why</tt></b> (<b><tt>-%Y</tt></b>) option, described in <a href="httrack.man.html#OPTIONS">the manual page</a>.<br>
|
||||
<br>
|
||||
|
||||
<p>
|
||||
<h4>Scan rules based on size (e.g. accept or refuse files bigger/smaller than a certain size)</h4>
|
||||
@@ -145,7 +143,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
size to ensure that you won't reach a defined limit.
|
||||
|
||||
Example: You may want to accept all files on the domain www.example.com, using '+www.example.com/*',
|
||||
including gif files inside this domain and outside (external images), but not take to large images,
|
||||
including gif files inside this domain and outside (eternal images), but not take to large images,
|
||||
or too small ones (thumbnails)<br>
|
||||
Excluding gif images smaller than 5KB and images larger than 100KB is therefore a good option;
|
||||
+www.example.com +*.gif -*.gif*[<5] -*.gif*[>100]
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -16,11 +16,6 @@ regen-man: makeman.sh $(top_builddir)/src/httrack$(EXEEXT)
|
||||
|
||||
# Render html/httrack.man.html from httrack.1. Needs the groff html device
|
||||
# (Debian: full "groff" package, not "groff-base"). Run by hand: make -C man regen-man-html
|
||||
# Strip groff's version-stamp and creation-date comments so the committed file
|
||||
# doesn't churn across groff versions or rebuilds; the ci man-page-sync guard
|
||||
# only requires this file to change whenever httrack.1 does.
|
||||
regen-man-html: httrack.1
|
||||
groff -t -man -Thtml $(srcdir)/httrack.1 \
|
||||
| sed -e 's/groff version [0-9][0-9.]*/groff/' -e '/^<!-- CreationDate:/d' \
|
||||
> $(top_srcdir)/html/httrack.man.html
|
||||
groff -t -man -Thtml $(srcdir)/httrack.1 > $(top_srcdir)/html/httrack.man.html
|
||||
.PHONY: regen-man-html
|
||||
|
||||
@@ -1617,15 +1617,14 @@ int back_add_if_not_exists(struct_back * sback, httrackp * opt,
|
||||
back_clean(opt, cache, sback); /* first cleanup the backlog to ensure that we have some entry left */
|
||||
if (!back_exist(sback, opt, adr, fil, save)) {
|
||||
return back_add(sback, opt, cache, adr, fil, save, referer_adr, referer_fil,
|
||||
test, HTS_FALSE);
|
||||
test);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
|
||||
const char *adr, const char *fil, const char *save,
|
||||
const char *referer_adr, const char *referer_fil, int test,
|
||||
hts_boolean refetch_whole) {
|
||||
int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char *adr,
|
||||
const char *fil, const char *save, const char *referer_adr, const char *referer_fil,
|
||||
int test) {
|
||||
lien_back *const back = sback->lnk;
|
||||
const int back_max = sback->count;
|
||||
int p = 0;
|
||||
@@ -1702,12 +1701,6 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
|
||||
else if (strcmp(back[p].url_sav, BACK_ADD_TEST2) == 0) // test en GET
|
||||
back[p].head_request = 2; // test en get
|
||||
|
||||
/* Forced whole refetch (#581): drop the stale temp-ref and skip the resume
|
||||
branches below, so a surviving partial can't Range-loop. */
|
||||
if (refetch_whole) {
|
||||
url_savename_refname_remove(opt, adr, fil);
|
||||
}
|
||||
|
||||
/* Stop requested - abort backing */
|
||||
/* For update mode: second check after cache lookup not to lose all previous cache data ! */
|
||||
if (opt->state.stop && !opt->is_update) {
|
||||
@@ -1963,9 +1956,8 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
|
||||
}
|
||||
}
|
||||
/* Not in cache ; maybe in temporary cache ? Warning: non-movable
|
||||
"url_sav" (skipped on a forced whole refetch, #581) */
|
||||
else if (!refetch_whole &&
|
||||
back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
|
||||
"url_sav" */
|
||||
else if (back_unserialize_ref(opt, adr, fil, &itemback) == 0) {
|
||||
const LLint file_size = fsize_utf8(itemback->url_sav);
|
||||
|
||||
/* Found file on disk */
|
||||
@@ -1999,9 +1991,8 @@ int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
|
||||
freet(itemback); /* delete item */
|
||||
itemback = NULL;
|
||||
}
|
||||
/* Not in cache or temporary cache ; found on disk ? (hack)
|
||||
(skipped on a forced whole refetch, #581) */
|
||||
else if (!refetch_whole && fexist_utf8(save)) {
|
||||
/* Not in cache or temporary cache ; found on disk ? (hack) */
|
||||
else if (fexist_utf8(save)) {
|
||||
const LLint sz = fsize_utf8(save);
|
||||
|
||||
// Bon, là il est possible que le fichier ait été partiellement transféré
|
||||
@@ -3856,8 +3847,6 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
deletehttp(&back[i].r);
|
||||
back[i].r.soc = INVALID_SOCKET;
|
||||
back[i].r.statuscode = STATUSCODE_NON_FATAL;
|
||||
back[i].r.refetch_wholefile =
|
||||
HTS_TRUE; // retry whole, no Range (#581)
|
||||
strcpybuff(back[i].r.msg,
|
||||
"Bogus 304 on resume, restarting");
|
||||
back[i].status = STATUS_READY;
|
||||
@@ -4129,9 +4118,6 @@ void back_wait(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
}
|
||||
back[i].r.soc = INVALID_SOCKET;
|
||||
back[i].r.statuscode = STATUSCODE_NON_FATAL;
|
||||
// the resume was rejected: the retry must GET the whole
|
||||
// file, never re-Range a surviving partial/ref (#581)
|
||||
back[i].r.refetch_wholefile = HTS_TRUE;
|
||||
if (strnotempty(back[i].r.msg))
|
||||
strcpybuff(back[i].r.msg,
|
||||
"Error attempting to solve status 206 (partial file)");
|
||||
|
||||
@@ -83,12 +83,9 @@ HTS_INLINE int back_exist(struct_back * sback, httrackp * opt, const char *adr,
|
||||
const char *fil, const char *sav);
|
||||
int back_nsoc(const struct_back * sback);
|
||||
int back_nsoc_overall(const struct_back * sback);
|
||||
/* refetch_whole: force a whole-file GET, ignoring any partial/temp-ref resume
|
||||
(set when a prior 206 was rejected as unusable, #581). */
|
||||
int back_add(struct_back *sback, httrackp *opt, cache_back *cache,
|
||||
const char *adr, const char *fil, const char *save,
|
||||
const char *referer_adr, const char *referer_fil, int test,
|
||||
hts_boolean refetch_whole);
|
||||
int back_add(struct_back * sback, httrackp * opt, cache_back * cache, const char *adr,
|
||||
const char *fil, const char *save, const char *referer_adr, const char *referer_fil,
|
||||
int test);
|
||||
int back_add_if_not_exists(struct_back * sback, httrackp * opt,
|
||||
cache_back * cache, const char *adr, const char *fil, const char *save,
|
||||
const char *referer_adr, const char *referer_fil, int test);
|
||||
|
||||
220
src/htscache.c
220
src/htscache.c
@@ -856,41 +856,7 @@ static htsblk cache_readex_new(httrackp * opt, cache_back * cache,
|
||||
// si save==null alors test unqiquement
|
||||
static int hts_rename(httrackp * opt, const char *a, const char *b) {
|
||||
hts_log_print(opt, LOG_DEBUG, "Cache: rename %s -> %s (%p %p)", a, b, a, b);
|
||||
return RENAME(a, b);
|
||||
}
|
||||
|
||||
/* Open the cache ZIP via hts_fopen_utf8 so a non-ASCII path_log isn't mangled
|
||||
to ANSI (#630); 64-bit funcs keep multi-GB caches whole on Windows LLP64. */
|
||||
static voidpf ZCALLBACK hts_zip_fopen_utf8(voidpf opaque, const void *filename,
|
||||
int mode) {
|
||||
const char *mode_fopen = NULL;
|
||||
|
||||
(void) opaque;
|
||||
if ((mode & ZLIB_FILEFUNC_MODE_READWRITEFILTER) == ZLIB_FILEFUNC_MODE_READ)
|
||||
mode_fopen = "rb";
|
||||
else if (mode & ZLIB_FILEFUNC_MODE_EXISTING)
|
||||
mode_fopen = "r+b";
|
||||
else if (mode & ZLIB_FILEFUNC_MODE_CREATE)
|
||||
mode_fopen = "wb";
|
||||
if (filename == NULL || mode_fopen == NULL)
|
||||
return NULL;
|
||||
return (voidpf) FOPEN((const char *) filename, mode_fopen);
|
||||
}
|
||||
|
||||
static unzFile hts_unzOpen_utf8(const char *path) {
|
||||
zlib_filefunc64_def ff;
|
||||
|
||||
fill_fopen64_filefunc(&ff);
|
||||
ff.zopen64_file = hts_zip_fopen_utf8;
|
||||
return unzOpen2_64(path, &ff);
|
||||
}
|
||||
|
||||
static zipFile hts_zipOpen_utf8(const char *path, int append) {
|
||||
zlib_filefunc64_def ff;
|
||||
|
||||
fill_fopen64_filefunc(&ff);
|
||||
ff.zopen64_file = hts_zip_fopen_utf8;
|
||||
return zipOpen2_64(path, append, NULL, &ff);
|
||||
return rename(a, b);
|
||||
}
|
||||
|
||||
/* Pathname of a file inside the mirror dir (rotating concat buffer). */
|
||||
@@ -908,9 +874,9 @@ static char *reconcile_path(httrackp *opt, const char *name) {
|
||||
/* Replace the new-generation file by the old one, when the old one exists. */
|
||||
static void reconcile_promote(httrackp *opt, const char *oldname,
|
||||
const char *newname) {
|
||||
if (fexist_utf8(reconcile_path(opt, oldname))) {
|
||||
UNLINK(reconcile_path(opt, newname));
|
||||
RENAME(reconcile_path(opt, oldname), reconcile_path(opt, newname));
|
||||
if (fexist(reconcile_path(opt, oldname))) {
|
||||
remove(reconcile_path(opt, newname));
|
||||
rename(reconcile_path(opt, oldname), reconcile_path(opt, newname));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -919,7 +885,7 @@ void hts_cache_reconcile(httrackp *opt, hts_cache_reconcile_mode mode) {
|
||||
case CACHE_RECONCILE_PROMOTE:
|
||||
/* Previous run rotated new.* to old.* then died before writing: promote
|
||||
the old generation back, whichever format it uses. */
|
||||
if (!fexist_utf8(reconcile_path(opt, "hts-cache/new.zip")))
|
||||
if (!fexist(reconcile_path(opt, "hts-cache/new.zip")))
|
||||
reconcile_promote(opt, "hts-cache/old.zip", "hts-cache/new.zip");
|
||||
break;
|
||||
case CACHE_RECONCILE_INTERRUPTED:
|
||||
@@ -928,17 +894,16 @@ void hts_cache_reconcile(httrackp *opt, hts_cache_reconcile_mode mode) {
|
||||
is -1 for a missing file, which would spuriously pass the "< TINY" test
|
||||
and overwrite a solid old generation that PROMOTE/ROLLBACK should keep.
|
||||
*/
|
||||
if (!opt->cache ||
|
||||
!fexist_utf8(reconcile_path(opt, "hts-in_progress.lock")))
|
||||
if (!opt->cache || !fexist(reconcile_path(opt, "hts-in_progress.lock")))
|
||||
break;
|
||||
if (fexist_utf8(reconcile_path(opt, "hts-cache/new.zip")) &&
|
||||
fexist_utf8(reconcile_path(opt, "hts-cache/old.zip")) &&
|
||||
fsize_utf8(reconcile_path(opt, "hts-cache/new.zip")) <
|
||||
if (fexist(reconcile_path(opt, "hts-cache/new.zip")) &&
|
||||
fexist(reconcile_path(opt, "hts-cache/old.zip")) &&
|
||||
fsize(reconcile_path(opt, "hts-cache/new.zip")) <
|
||||
CACHE_RECONCILE_NEW_TINY &&
|
||||
fsize_utf8(reconcile_path(opt, "hts-cache/old.zip")) >
|
||||
fsize(reconcile_path(opt, "hts-cache/old.zip")) >
|
||||
CACHE_RECONCILE_OLD_SOLID &&
|
||||
fsize_utf8(reconcile_path(opt, "hts-cache/old.zip")) >
|
||||
fsize_utf8(reconcile_path(opt, "hts-cache/new.zip")))
|
||||
fsize(reconcile_path(opt, "hts-cache/old.zip")) >
|
||||
fsize(reconcile_path(opt, "hts-cache/new.zip")))
|
||||
reconcile_promote(opt, "hts-cache/old.zip", "hts-cache/new.zip");
|
||||
break;
|
||||
case CACHE_RECONCILE_ROLLBACK:
|
||||
@@ -975,26 +940,24 @@ void cache_init(cache_back * cache, httrackp * opt) {
|
||||
#endif
|
||||
if (!cache->ro) {
|
||||
#ifdef _WIN32
|
||||
/* Windows mkdir takes no mode; use the UTF-8 wrapper for #630. */
|
||||
MKDIR(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache"));
|
||||
mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache"));
|
||||
#else
|
||||
/* keep the cache dir 0700, not MKDIR's 0755. */
|
||||
mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache"),
|
||||
mkdir(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache"),
|
||||
HTS_PROTECT_FOLDER);
|
||||
#endif
|
||||
if ((fexist_utf8(fconcat(
|
||||
if ((fexist(fconcat(
|
||||
OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/new.zip")))) { // a previous cache exists.. rename it
|
||||
/* Remove OLD cache */
|
||||
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/old.zip"))) {
|
||||
if (UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.zip")) !=
|
||||
0) {
|
||||
if (fexist
|
||||
(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.zip"))) {
|
||||
if (remove
|
||||
(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.zip")) != 0) {
|
||||
hts_log_print(opt, LOG_WARNING | LOG_ERRNO,
|
||||
"Cache: error while moving previous cache");
|
||||
}
|
||||
@@ -1017,27 +980,33 @@ void cache_init(cache_back * cache, httrackp * opt) {
|
||||
hts_log_print(opt, LOG_DEBUG, "Cache: no cache found");
|
||||
}
|
||||
hts_log_print(opt, LOG_DEBUG, "Cache: size %d",
|
||||
(int) fsize_utf8(
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.zip")));
|
||||
(int)
|
||||
fsize(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.zip")));
|
||||
|
||||
// charger index cache précédent
|
||||
if ((!cache->ro &&
|
||||
fsize_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.zip")) >
|
||||
0) ||
|
||||
(cache->ro &&
|
||||
fsize_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.zip")) >
|
||||
0)) {
|
||||
if ((!cache->ro
|
||||
&&
|
||||
fsize(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.zip")) > 0)
|
||||
|| (cache->ro
|
||||
&&
|
||||
fsize(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.zip")) > 0)
|
||||
) {
|
||||
if (!cache->ro) {
|
||||
cache->zipInput = hts_unzOpen_utf8(
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.zip"));
|
||||
cache->zipInput =
|
||||
unzOpen(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.zip"));
|
||||
} else {
|
||||
cache->zipInput = hts_unzOpen_utf8(
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.zip"));
|
||||
cache->zipInput =
|
||||
unzOpen(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.zip"));
|
||||
}
|
||||
|
||||
// Corrupted ZIP file ? Try to repair!
|
||||
@@ -1057,8 +1026,6 @@ void cache_init(cache_back * cache, httrackp * opt) {
|
||||
}
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
"Cache: damaged cache, trying to repair");
|
||||
/* mztools has no UTF-8 hook, so repairing a corrupt cache under a
|
||||
non-ASCII path_log fails cleanly (re-crawl), never forks a twin. */
|
||||
if (unzRepair
|
||||
(name,
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
@@ -1066,11 +1033,11 @@ void cache_init(cache_back * cache, httrackp * opt) {
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/repair.tmp"),
|
||||
&repaired, &repairedBytes) == Z_OK) {
|
||||
UNLINK(name);
|
||||
RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/repair.zip"),
|
||||
name);
|
||||
cache->zipInput = hts_unzOpen_utf8(name);
|
||||
unlink(name);
|
||||
rename(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/repair.zip"), name);
|
||||
cache->zipInput = unzOpen(name);
|
||||
hts_log_print(opt, LOG_WARNING,
|
||||
"Cache: %d bytes successfully recovered in %d entries",
|
||||
(int) repairedBytes, (int) repaired);
|
||||
@@ -1166,8 +1133,8 @@ void cache_init(cache_back * cache, httrackp * opt) {
|
||||
"Cache: error trying to open the cache");
|
||||
}
|
||||
|
||||
} else if (fsize_utf8(reconcile_path(opt, "hts-cache/old.ndx")) > 0 ||
|
||||
fsize_utf8(reconcile_path(opt, "hts-cache/new.ndx")) > 0) {
|
||||
} else if (fsize(reconcile_path(opt, "hts-cache/old.ndx")) > 0 ||
|
||||
fsize(reconcile_path(opt, "hts-cache/new.ndx")) > 0) {
|
||||
/* pre-3.31 (2003) .dat/.ndx cache: import support removed */
|
||||
hts_log_print(opt, LOG_ERROR,
|
||||
"Cache: the pre-3.31 .dat/.ndx cache format is no longer "
|
||||
@@ -1183,58 +1150,67 @@ void cache_init(cache_back * cache, httrackp * opt) {
|
||||
#endif
|
||||
if (!cache->ro) {
|
||||
// ouvrir caches actuels
|
||||
structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/"));
|
||||
structcheck(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "hts-cache/"));
|
||||
|
||||
{
|
||||
/* Create ZIP file cache */
|
||||
cache->zipOutput = (void *) hts_zipOpen_utf8(
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.zip"),
|
||||
0);
|
||||
cache->zipOutput =
|
||||
(void *)
|
||||
zipOpen(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.zip"), 0);
|
||||
|
||||
if (cache->zipOutput != NULL) {
|
||||
// supprimer old.lst
|
||||
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/old.lst")))
|
||||
UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.lst"));
|
||||
if (fexist
|
||||
(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.lst")))
|
||||
remove(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.lst"));
|
||||
// renommer
|
||||
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/new.lst")))
|
||||
RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.lst"),
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.lst"));
|
||||
if (fexist
|
||||
(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.lst")))
|
||||
rename(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.lst"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/old.lst"));
|
||||
// ouvrir
|
||||
cache->lst =
|
||||
FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.lst"),
|
||||
"wb");
|
||||
fopen(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.lst"), "wb");
|
||||
strcpybuff(opt->state.strc.path, StringBuff(opt->path_html));
|
||||
opt->state.strc.lst = cache->lst;
|
||||
|
||||
// supprimer old.txt
|
||||
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/old.txt")))
|
||||
UNLINK(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.txt"));
|
||||
if (fexist
|
||||
(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.txt")))
|
||||
remove(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/old.txt"));
|
||||
// renommer
|
||||
if (fexist_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/new.txt")))
|
||||
RENAME(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.txt"),
|
||||
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/old.txt"));
|
||||
if (fexist
|
||||
(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.txt")))
|
||||
rename(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.txt"), fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log),
|
||||
"hts-cache/old.txt"));
|
||||
// ouvrir
|
||||
cache->txt =
|
||||
FOPEN(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
|
||||
StringBuff(opt->path_log), "hts-cache/new.txt"),
|
||||
"wb");
|
||||
fopen(fconcat
|
||||
(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log),
|
||||
"hts-cache/new.txt"), "wb");
|
||||
if (cache->txt) {
|
||||
fprintf(cache->txt,
|
||||
"date\tsize'/'remotesize\tflags(request:Update,Range state:File response:Modified,Chunked,gZipped)\t");
|
||||
|
||||
@@ -3293,10 +3293,10 @@ int back_fill(struct_back * sback, httrackp * opt, cache_back * cache,
|
||||
if (ok) {
|
||||
if (!back_exist
|
||||
(sback, opt, heap(p)->adr, heap(p)->fil, heap(p)->sav)) {
|
||||
if (back_add(sback, opt, cache, heap(p)->adr, heap(p)->fil,
|
||||
heap(p)->sav, heap(heap(p)->precedent)->adr,
|
||||
heap(heap(p)->precedent)->fil, heap(p)->testmode,
|
||||
heap(p)->refetch_whole) == -1) {
|
||||
if (back_add
|
||||
(sback, opt, cache, heap(p)->adr, heap(p)->fil, heap(p)->sav,
|
||||
heap(heap(p)->precedent)->adr, heap(heap(p)->precedent)->fil,
|
||||
heap(p)->testmode) == -1) {
|
||||
hts_log_print(opt, LOG_DEBUG,
|
||||
"error: unable to add more links through back_add for back_fill");
|
||||
#if BDEBUG==1
|
||||
|
||||
@@ -615,9 +615,9 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
strcpybuff(current.fil, fil_complete);
|
||||
// ajouter dans le backing le fichier en mode test
|
||||
// savename: rien car en mode test
|
||||
if (back_add(sback, opt, cache, current.adr, current.fil,
|
||||
BACK_ADD_TEST, referer_adr, referer_fil, 1,
|
||||
HTS_FALSE) != -1) {
|
||||
if (back_add
|
||||
(sback, opt, cache, current.adr, current.fil, BACK_ADD_TEST,
|
||||
referer_adr, referer_fil, 1) != -1) {
|
||||
int b;
|
||||
|
||||
b = back_index(opt, sback, current.adr, current.fil, BACK_ADD_TEST);
|
||||
@@ -706,10 +706,7 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
if (!hts_wait_available_socket(sback, opt,
|
||||
cache, ptr))
|
||||
return -1;
|
||||
if (back_add(sback, opt, cache, moved.adr,
|
||||
moved.fil, methode, referer_adr,
|
||||
referer_fil, 1,
|
||||
HTS_FALSE) != -1) { // OK
|
||||
if (back_add(sback, opt, cache, moved.adr, moved.fil, methode, referer_adr, referer_fil, 1) != -1) { // OK
|
||||
hts_log_print(opt, LOG_DEBUG,
|
||||
"(during prefetch) %s (%d) to link %s at %s%s",
|
||||
back[b].r.msg,
|
||||
@@ -728,8 +725,7 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
has_been_moved = 1; // sinon ne pas forcer has_been_moved car non déplacé
|
||||
petits_tours++;
|
||||
//
|
||||
} else { // sinon on fait rien et on s'en va..
|
||||
// (ftp etc)
|
||||
} else { // sinon on fait rien et on s'en va.. (ftp etc)
|
||||
hts_log_print(opt, LOG_DEBUG,
|
||||
"Warning: Savename redirect backing error at %s%s",
|
||||
moved.adr, moved.fil);
|
||||
@@ -800,6 +796,7 @@ int url_savename(lien_adrfilsave *const afs,
|
||||
hts_log_print(opt, LOG_ERROR,
|
||||
"Unexpected savename backing error at %s%s", adr,
|
||||
fil_complete);
|
||||
|
||||
}
|
||||
// restaurer
|
||||
opt->state._hts_in_html_parsing = hihp;
|
||||
|
||||
@@ -658,9 +658,6 @@ struct htsblk {
|
||||
int debugid; /**< connection debug id */
|
||||
/* */
|
||||
htsrequest req; /**< parameters used for the request */
|
||||
/* Restart-whole signal: a resume this response rejected (unusable 206) must
|
||||
retry with no Range, else a surviving partial/temp-ref loops (#581). */
|
||||
hts_boolean refetch_wholefile;
|
||||
/*char digest[32+2]; // md5 digest generated by the engine ("" if none) */
|
||||
};
|
||||
|
||||
@@ -684,9 +681,6 @@ struct lien_url {
|
||||
char link_import; /**< imported after a move; skip the usual up/down rules */
|
||||
int retry; /**< remaining retries */
|
||||
int testmode; /**< test only: send just a HEAD */
|
||||
hts_boolean
|
||||
refetch_whole; /**< force a whole-file GET, ignoring any partial/temp-ref
|
||||
resume, so a rejected 206 can't loop (#581) */
|
||||
};
|
||||
|
||||
/* A file being fetched in the background. */
|
||||
|
||||
@@ -315,18 +315,9 @@ static void escape_url_parens(char *const s, const size_t size) {
|
||||
strlcpybuff(s, buff, size);
|
||||
}
|
||||
|
||||
/* Default port for lien's scheme (case-insensitive), 80 if absent/unknown; so
|
||||
schemeless and protocol-relative //host links default to 80 (known gap). */
|
||||
static int scheme_default_port(const char *lien) {
|
||||
if (strfield(lien, "https:"))
|
||||
return 443;
|
||||
if (strfield(lien, "ftp:"))
|
||||
return 21;
|
||||
return 80;
|
||||
}
|
||||
|
||||
/* Strip the scheme's own default port (80 http, 443 https, 21 ftp) from lien's
|
||||
authority in place; :80 on https/ftp stays as a real port (#638, #614). */
|
||||
/* Strip a default ":80" from lien's authority in place. Any spelling that
|
||||
range-parses to 80 (":80", ":080") is dropped by its matched length, not a
|
||||
hardcoded 3 chars; a value that merely wraps to 80 as an int (#614) stays. */
|
||||
void hts_strip_default_port(char *lien, size_t size) {
|
||||
char *a;
|
||||
|
||||
@@ -348,8 +339,7 @@ void hts_strip_default_port(char *lien, size_t size) {
|
||||
b++;
|
||||
saved = *b;
|
||||
*b = '\0';
|
||||
is_default =
|
||||
hts_parse_url_port(a + 1, &port) && port == scheme_default_port(lien);
|
||||
is_default = hts_parse_url_port(a + 1, &port) && port == 80;
|
||||
*b = saved;
|
||||
if (is_default) { // default port, strip it
|
||||
char BIGSTK tempo[HTS_URLMAXSIZE * 2];
|
||||
@@ -3756,9 +3746,6 @@ int hts_mirror_check_moved(htsmoduleStruct * str,
|
||||
heap_top()->retry = heap(ptr)->retry - 1; // moins 1 retry!
|
||||
heap_top()->premier = heap(ptr)->premier;
|
||||
heap_top()->precedent = heap(ptr)->precedent;
|
||||
// a rejected resume (unusable 206) must refetch whole, no Range
|
||||
// (#581)
|
||||
heap_top()->refetch_whole = r->refetch_wholefile;
|
||||
} else { // oups erreur, plus de mémoire!!
|
||||
return 0;
|
||||
}
|
||||
@@ -3990,17 +3977,18 @@ int hts_mirror_wait_for_next_file(htsmoduleStruct * str,
|
||||
#if BDEBUG==1
|
||||
printf("crash backing: %s%s\n", heap(ptr)->adr, heap(ptr)->fil);
|
||||
#endif
|
||||
if (back_add(sback, opt, cache, urladr(), urlfil(), savename(),
|
||||
heap(heap(ptr)->precedent)->adr,
|
||||
heap(heap(ptr)->precedent)->fil, heap(ptr)->testmode,
|
||||
heap(ptr)->refetch_whole) == -1) {
|
||||
if (back_add
|
||||
(sback, opt, cache, urladr(), urlfil(), savename(),
|
||||
heap(heap(ptr)->precedent)->adr, heap(heap(ptr)->precedent)->fil,
|
||||
heap(ptr)->testmode) == -1) {
|
||||
printf("PANIC! : Crash adding error, unexpected error found.. [%d]\n",
|
||||
__LINE__);
|
||||
#if BDEBUG==1
|
||||
printf("error while crash adding\n");
|
||||
#endif
|
||||
hts_log_print(opt, LOG_ERROR, "Unexpected backing error for %s%s",
|
||||
urladr(), urlfil());
|
||||
hts_log_print(opt, LOG_ERROR, "Unexpected backing error for %s%s", urladr(),
|
||||
urlfil());
|
||||
|
||||
}
|
||||
}
|
||||
#if BDEBUG==1
|
||||
|
||||
@@ -1589,8 +1589,10 @@ static int st_identabs(httrackp *opt, int argc, char **argv) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Default-port strip is scheme-aware (#638), overflow-safe (#614): a scheme's
|
||||
own default (any spelling) is dropped, a real port stays; guards #627. */
|
||||
/* Default-port strip (#627): a genuine 80 (any spelling) is removed by its
|
||||
matched length, host preserved; a non-80 port or one that only wraps to 80 as
|
||||
a 32-bit int (#614) is left intact. Guards the old bug where ":080"/":0080"
|
||||
dropped a hardcoded 3 chars and glued the leftover digits onto the host. */
|
||||
static int st_stripport(httrackp *opt, int argc, char **argv) {
|
||||
static const struct {
|
||||
const char *in, *out;
|
||||
@@ -1604,13 +1606,6 @@ static int st_stripport(httrackp *opt, int argc, char **argv) {
|
||||
{"http://127.0.0.1:8080/x", "http://127.0.0.1:8080/x"},
|
||||
{"http://127.0.0.1:4294967376/x", "http://127.0.0.1:4294967376/x"},
|
||||
{"http://127.0.0.1/x", "http://127.0.0.1/x"},
|
||||
{"https://127.0.0.1:443/x", "https://127.0.0.1/x"},
|
||||
{"https://127.0.0.1:80/x", "https://127.0.0.1:80/x"},
|
||||
// Scheme match is case-insensitive: HTTPS' default is 443, so :80 stays.
|
||||
{"HTTPS://127.0.0.1:80/x", "HTTPS://127.0.0.1:80/x"},
|
||||
{"ftp://127.0.0.1:21/x", "ftp://127.0.0.1/x"},
|
||||
{"ftp://127.0.0.1:80/x", "ftp://127.0.0.1:80/x"},
|
||||
{"http://127.0.0.1:443/x", "http://127.0.0.1:443/x"},
|
||||
};
|
||||
|
||||
size_t k;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
--- mztools.c.orig 2024-01-27 14:07:18.636193212 +0100
|
||||
+++ mztools.c 2024-01-27 14:09:55.356620093 +0100
|
||||
@@ -10,10 +10,11 @@
|
||||
@@ -10,6 +10,7 @@
|
||||
#include <string.h>
|
||||
#include "zlib.h"
|
||||
#include "unzip.h"
|
||||
@@ -8,11 +8,6 @@
|
||||
|
||||
#define READ_8(adr) ((unsigned char)*(adr))
|
||||
#define READ_16(adr) ( READ_8(adr) | (READ_8(adr+1) << 8) )
|
||||
-#define READ_32(adr) ( READ_16(adr) | (READ_16((adr)+2) << 16) )
|
||||
+#define READ_32(adr) ((uLong) READ_16(adr) | ((uLong) READ_16((adr) + 2) << 16))
|
||||
|
||||
#define WRITE_8(buff, n) do { \
|
||||
*((unsigned char*)(buff)) = (unsigned char) ((n) & 0xff); \
|
||||
@@ -141,8 +142,8 @@
|
||||
/* Central directory entry */
|
||||
{
|
||||
|
||||
@@ -6,7 +6,6 @@ set -euo pipefail
|
||||
# Stripping a default :80 from a crawled link used to skip a hardcoded 3 chars
|
||||
# (#627): ":080"/":0080" lost only ":80" and glued the rest onto the host
|
||||
# (127.0.0.1:080/x -> 127.0.0.10/x), and a value wrapping to 80 as a 32-bit int
|
||||
# (#614) was stripped as if it were the default. The default is scheme-aware
|
||||
# (#638): an explicit :80 on https/ftp stays, :443/:21 strip under their own
|
||||
# scheme. All assertions live in the engine self-test.
|
||||
# (#614) was stripped as if it were the default. All assertions live in the
|
||||
# engine self-test.
|
||||
httrack -O /dev/null -#test=stripport | grep -q "stripport self-test OK"
|
||||
|
||||
@@ -1,17 +1,15 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# A single non-ASCII -O sets both path_html and path_log to "café" (#630). The
|
||||
# logs (hts-log.txt/hts-err.txt) and the hts-cache must land there, not in an
|
||||
# ANSI-mangled twin: on Windows path_log holds UTF-8 bytes the raw file calls
|
||||
# read as the codepage. The --log-found audit greps logroot=café/hts-log.txt and
|
||||
# --cache-under-logroot checks café/hts-cache/new.zip, so anything written to the
|
||||
# twin fails them. POSIX has no twin, so this bites on the Windows CI leg (test
|
||||
# 64), exercising only the code path elsewhere.
|
||||
# logs (hts-log.txt/hts-err.txt) must land there, not in an ANSI-mangled twin:
|
||||
# on Windows path_log holds UTF-8 bytes the raw file calls read as the codepage.
|
||||
# The --errors/--files/--log-found audits all grep logroot=café/hts-log.txt, so
|
||||
# a log written to the twin fails them. POSIX has no twin, so this bites on the
|
||||
# Windows CI leg (like test 64). The cache twin is a separate, larger fix.
|
||||
|
||||
: "${top_srcdir:=..}"
|
||||
|
||||
bash "$top_srcdir/tests/local-crawl.sh" --outdir-intl 'café' --errors 0 --files 5 \
|
||||
--found 'simple/basic.html' \
|
||||
--log-found 'mirror complete in' \
|
||||
--cache-under-logroot \
|
||||
httrack 'BASEURL/simple/basic.html'
|
||||
|
||||
@@ -16,12 +16,11 @@ root=$(nativepath "${testdir}/server-root")
|
||||
|
||||
python=$(find_python) || ! echo "python3 not found; skipping" >&2 || exit 77
|
||||
|
||||
# Windows-only repro of #581 (a hard-killed pass 1 + repaired cache loses the
|
||||
# file). The engine now forces a no-Range whole refetch on the restart, but that
|
||||
# fix is unverified on Windows CI, and this test's pass-1 interrupt/port-race
|
||||
# behavior there is unconfirmed; lift this skip on a Windows runner to verify.
|
||||
# On Windows the pass-1 interrupt is a hard kill (MSYS can't signal a native
|
||||
# exe) and the restart-whole path fails on the repaired cache (#581) -- the very
|
||||
# bug this exercises; skip until the engine fix lands.
|
||||
if is_windows; then
|
||||
echo "Windows: #581 fix unverified on CI, skipping"
|
||||
echo "Windows: restart-whole fails on a repaired cache (#581), skipping"
|
||||
exit 77
|
||||
fi
|
||||
|
||||
|
||||
@@ -118,9 +118,6 @@ while test "$pos" -lt "$nargs"; do
|
||||
nopurge=1
|
||||
audit+=("--no-purge")
|
||||
;;
|
||||
--cache-under-logroot)
|
||||
audit+=("--cache-under-logroot")
|
||||
;;
|
||||
--tls)
|
||||
tls=1
|
||||
scheme=https
|
||||
@@ -382,15 +379,6 @@ while test "$i" -lt "${#audit[@]}"; do
|
||||
sed -e 's/.*[[:space:]]\([^ ]*\)[[:space:]]files written.*/\1/g')
|
||||
assert_equals "checking files" "${audit[$i]}" "$nFiles"
|
||||
;;
|
||||
--cache-under-logroot)
|
||||
# The cache must sit under path_log (logroot), not an ANSI-mangled twin
|
||||
# of a non-ASCII -O dir (#630 cache half). Bites only on the Windows leg.
|
||||
info "checking cache under logroot"
|
||||
if test -e "${logroot}/hts-cache/new.zip"; then result "OK"; else
|
||||
result "cache not under logroot (mangled twin?)"
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
--found)
|
||||
i=$((i + 1))
|
||||
info "checking for ${audit[$i]}"
|
||||
|
||||
Reference in New Issue
Block a user