Compare commits

..

21 Commits

Author SHA1 Message Date
Xavier Roche
417d4cc467 Test 295 expects a wizard verdict string that #1254 retired (#1267)
An answer that does not forbid now resolves to 0 in hts_wizard_apply_verdict(),
so an accepting answer logs "allowed" where the test still expected "no
verdict". Each PR was green alone; the pair is red.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 16:05:37 +00:00
Xavier Roche
492c239dcd The credit tag in every mirrored page is dated 2014 (#1243)
* The credit tag in every mirrored page is dated 2014

HTTRACK_AFF_AUTHORS reads "[XR&CO'2014]" and lands in every mirrored page's
footer, every generated index and the version banner, so a year nobody has
maintained since 2014 is stamped on the output of a 2026 crawl. Drop the year
rather than move it: the next bump would go stale the same way, and the release
already dates itself through the footer's {date}.

The index templates carried their own copies, which had drifted apart in the
meantime: the compiled-in ones said 2014 and the shipped templates/ ones 2008,
with the web designer's name reversed between them. Both now credit 1998, the
start year the source headers already use, and the name matches the served
WebHTTrack pages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Collapse the clang-format note to one line

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the clang-format note under the column limit

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 14:39:06 +00:00
Xavier Roche
2472f8f417 The per-option argument caps are literals the front ends have to copy
`htscoremain.c` held the caps for six option arguments as bare `254` and `62` literals at the option sites. The GUI front ends carry their own copies of three of them, and nothing links the two, so a changed cap shows up only when a user's mirror dies on a value the GUI accepted.

They move to `htsglobal.h`, which already reaches the front ends through `HTTrackInterface.h`, and keep the exclusive bound the existing checks use, so no boundary shifts by one. Naming all six rather than the three asked for avoids leaving a shared-looking `254` next to them for someone to cross-wire later.

No behavior change: every boundary was probed against master, one byte under and exactly at each cap, with identical results. `299_option-arg-caps.test` reads each limit out of the header rather than copying it, so it pins the engine against `htsglobal.h` instead of against a second copy of the numbers.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 14:30:24 +00:00
Xavier Roche
f94da62bad Strings added since 3.49 are still English in every catalog (#1255)
`LANG_M10` and `LANG_M11` join `LANG_M1`..`LANG_M9` for the wizard's two domain-scope answers, which #1239 left hardcoded in the CLI menu with no key for WinHTTrack (xroche/httrack-windows#96). They are worded exactly like that menu, and test 62 now requires both to appear verbatim in `English.txt`, `lang.def` and `src/httrack.c`, so the front ends cannot drift the way #1117 warned about.

Two other sets were still English everywhere: the host-alias field and tooltip #1166 copied into every catalog, and the fifteen Android strings that had only ever reached Danish, French and Brazilian Portuguese, which `tests/62_lang-integrity.test` waived by name. Both are translated across the thirty catalogs now and the waiver is gone, so the next gap reds the test.

Each file keeps its own on-disk charset and line endings. Uzbek stays LF, Romanian still spells s-comma, t-comma and a-breve with the Latin-1 lookalikes it uses throughout, and the catalogs whose charset has no typographic quote pair take ASCII ones. Three mutants prove test 62 catches drift: a dropped msgid, a stray UTF-8 byte in a Latin-1 file, and `LANG_M10` pointed at a msgid missing from `English.txt`.

A review pass over the translations themselves fixed a feminine pronoun standing in for the masculine host in the Czech, Slovak, Croatian and Slovene scope answers, plus per-language defects in nine more catalogs. `Chinese-BIG5` also spelled `Ignore all` with the wrong character in the same wizard block; that is corrected here.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 14:03:24 +00:00
Xavier Roche
2294aa4019 The footer field names are copied by hand in three places (#1252)
The ten `-%F` footer field names were written out twice inside the formatter's own reach, in htsparse.c and in the htsselftest.c fixture, and a third time in WinHTTrack's preset validator with nothing to link against: `hts_footer_format` sits behind `HTS_INTERNAL_BYTECODE` and the field table was function-local. That third copy could catch a GUI typo and nothing else; an engine rename sailed straight past it.

The formatter's two callers now share one table in htstools.c and pass a positional `values[]` indexed by a new `hts_footer_field_id`, and front ends get `HTSEXT_API hts_boolean hts_footer_field_ok(const char *name)` to test a name against it. The export is additive and `hts_footer_format` is internal, so nothing about the ABI moves. The help text, man page and cmdguide.html still list the names by hand; that is #1256, not this PR.

Asked for by the httrack-windows session, which wires the GUI half once this lands.

Review caught the trap in the positional design and the second commit closes it: reordering the enum without touching the name table passed all 326 tests, because the named path resolves by name while the crawl assigns by enum symbol, so `{mime}` would have carried the charset. The fixture now fills `values[]` by id the way htsparse.c does, which makes the test's expected strings pin each name to its slot. A name missing from the table is a compile error instead of a NULL slot. Both mutants were confirmed to red, and a dropped name to fail the build.

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 13:54:22 +00:00
Xavier Roche
4729035a6e doit.log writes a leading quote raw and swallows the options after it
httrack wrote an argument into `hts-cache/doit.log` unquoted and unescaped whenever its first character was already a quote. Against the raw command line that guard makes sense. But by the time the writer runs, the `-O` pass has stripped one layer of surrounding quotes in place, so a leading quote is data, not syntax. Written raw it opens a quote that never closes, and the next run pulls everything after it into that one argument.

A footer of `"<!-- x -->` loses every option recorded after `-%F`. The reprise then rewrites doit.log from the corrupted parse, so the loss carries into every later update, and the mirrored page footer differs between the first run and the second. Reaching it takes two leading quotes on the command line, since the `-O` pass eats one pair, which is the shape the GUI front ends produce when they wrap an argument in literal quotes (xroche/httrack-windows#122).

One class is not fixed and changes failure mode, so it is worth stating plainly. A plain argument (not an option value) whose value starts with a quote and does not end with one now aborts the reprise with `Missing quote`, where master silently dropped the quote instead. The abort precedes the writer, so the no-argument update stays dead until the mirror is re-run with its URL. Option values are unaffected: they bypass the second strip pass. It takes three leading quotes typed by hand to reach, since the single-pair shape a front end emits already panics on the first run, on master too. #1263 has the asymmetry behind it.

Section 4 of `01_engine-doitlog.test` checks the footer on the mirrored page rather than the doit.log text alone, so a writer and reader that agree on a wrong value still fail it. A differential over 11 argv shapes moves only the leading-quote cases.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 13:45:32 +00:00
Xavier Roche
ed30cde2a4 A wizard session leaves no record in hts-log.txt (#1249)
Two problems, same log. Driving the wizard by hand left no trace: hts-log.txt recorded the crawl going on around the questions, but not what was asked or what the user replied, so a session could not be reviewed afterwards. The levels themselves were mislabelled too. `hts_log_vprint` mapped LOG_NOTICE onto the "warning" prefix, so every notice read as a warning, while the footer's message count tracked LOG_INFO, which the default `opt->debug` filters out. That count was always zero.

The engine now writes one line per wizard answer at LOG_NOTICE: the link, the raw reply, the parsed answer number, the verdict, and the filters that answer inserted, read back from the slots they were written to. `ask_continue()` gets the same line for the yes/no prompts, and nothing is logged unless a front end registered a callback to answer with, so a non-interactive crawl gains no noise. LOG_NOTICE also gets its own arm in the level switch, mapping to "info", so notices print `Info:` and move from the warning column of `(N errors, N warnings, N messages)` to the message one. `html/faq.html` already documents the robots.txt notice as an `Info:` line. Two notice sites report a failure rather than progress, and this promotes them to LOG_WARNING so they render as before.

`tests/295_wizard-log.test` drives three answers through a `file://` crawl under `-W` and checks the line each one produces, the two-filter host-scope answer included.


Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-08-14 13:53:28 +02:00
Xavier Roche
5e8fea2a72 A link the wizard authorized is mirrored under its .delayed placeholder name (#1254)
Answer the wizard with anything that takes the link (4, 5, 6, 7, 50, or a host-scope include) and the verdict came back undecided at -1 rather than 0. The parser records a link on `!= 1` but only resolves its delayed type on `== 0`, so an authorized link went into the hash still called `<base>.<hexid>.delayed`. Every later reference gets that stored name back, `hts_wait_delayed()` then finds the link's own record and gives up ("Duplicate entry in hts_wait_delayed() cancelled", then "link is probably looping, type unknown, aborting"), and the page lands in the mirror as a placeholder file no browser opens. Answering "mirror this domain and below" on httrack.com is enough.

The question is only asked while the verdict is undecided, so an answer that did not forbid has authorized. `hts_wizard_apply_verdict()` now resolves that once at the end instead of in six cases plus the scope-include branch, and it still never overturns a verdict the crawl had already computed. An unreadable reply (-999) used to arrive here undecided too; #1259 makes it refuse the link, so it reaches this normalization already set to 1 and the two changes compose in either merge order.

Two other effects move with it, both in the same direction. `htsAddLink()` (htscore.c:3786) gates on `if (!forbidden_url)`, so under -1 a link handed back by an external module parser was never saved: it logged "(module): file not caught" and shipped as an absolute external URL. It is now saved and rewritten local, and the function returns 1 where it returned 0. The `check_link` callback also sees this value, and html/plug.html documents -1 as "no decision has yet been taken by the engine", so a plugin keyed on -1 used to get a say on wizard-answered links and now defers. The user answering is a decision, so I read that as a correction rather than a regression, but it crosses the public plugin surface.

The two gates are unchanged. Outside `-W` the verdict reaching them is already 0 or 1, so aligning them would change no link I could build. `--addurl` is a second door onto the same state, filed as #1253. The "looks like binary" and octet-stream pair from the field log did not reproduce here and is untouched.

tests/296 crawls two references to one authorized foreign link and checks the mirror: the page is there with its body, both hrefs point at it, and no placeholder survives. tests/293's battery now drives every answer from the undecided verdict the question is actually asked in. The mutant that matters most, an unconditional `*forbidden_url = 0` erasing a refusal, is caught by 293 alone. 296 never crawls a link the crawl had already refused, so it cannot see that one.


Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-08-14 13:53:12 +02:00
Xavier Roche
c5c931093a The interactive wizard spins forever when stdin reaches EOF (#1248)
`httrack -W <url> < /dev/null` never returned. At EOF `linput()` hands back an empty line and reports no failure, so the answer loop in `htsshow_query3()` re-prompted forever at 100% CPU. Any non-interactive wizard run hits it: a piped answer script that runs out of answers, or a service whose stdin is closed.

An unanswerable prompt now reads as the `*` answer, which refuses the link and stops asking. No further answer can arrive, and refusing mirrors nothing the user did not ask for. The guard tests both stream flags, since a closed descriptor sets only `ferror` while EOF sets only `feof`. On a closed stdin the old loop printed 19 million prompts and 58 MB in 25 seconds.

tests/294 crawls a page linking twice to a second local server, once with stdin on `/dev/null` and once with it closed. It asserts the question was asked exactly once, so an answer that refuses this link but keeps asking fails, and the foreign host is served rather than a black hole, so an answer that mirrors it fails too. Each case needs its own half of the guard to pass.

Closes #1247

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 11:39:27 +02:00
Xavier Roche
935ea5a89b An LTO build collapses the installed-header symbol test's candidate list (#1246)
Ubuntu's 3.49.21-2 failed on four architectures (amd64, ppc64el, riscv64, s390x) with `207_install-headers-symbols.test` reporting "only 1 symbols reached the link probe, the candidate list is broken". Debian built the same source everywhere. Ubuntu compiles with `-flto=auto -ffat-lto-objects` by default and Debian does not.

The test picks its probe candidates by matching identifiers in the installed headers against the library's hidden symbol names. Under LTO, GCC renames a local to `abortf_.lto_priv.0`, the names stop matching, and the candidate list empties, so the `probed >= 5` floor fires. The floor was right; what fed it was broken. Stripping the clone suffixes before the comparison takes an `-O3 -flto` build from 1 candidate to 58, and an ordinary build from 32 to 65 by recovering its `.isra` and `.constprop` clones.

That floor turns out to be a weak detector on its own: a strip handling only `.lto_priv` still passed while losing 30 of 54 candidates. The canary header now declares one clone-only name per suffix and requires each to be reported, so dropping any single suffix fails the test.

Nothing in CI builds with LTO, which is how this reached the archive. The deb job runs inside debian:sid and gets Debian's flags; every other leg uses configure's defaults. The matrix gains an `-O3 -flto` leg.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 08:30:21 +00:00
Xavier Roche
3a0cf262d6 The wizard verdict switch is unreachable by any test (#1245)
The verdict half of a wizard answer moves into `hts_wizard_apply_verdict()`, leaving `hts_acceptlink_()` one call where a switch used to be. #1239 made the filter half a testable function and left this one unreachable: `hts_acceptlink_()` is static, and only the interactive `query3` callback gets there.

The self-test drives the new function against a real `httrackp`, so it asserts what an answer does to the crawl rather than what it means. An accepting answer must not clear a refusal the crawl already computed, answer 4 caps the recursion instead of deciding the link, `*` switches the wizard to automatic, and an answer in no known range only warns. Eight mutants each turn tests/293 red, including the three shapes the call site used to hide.

Behavior is unchanged for every int answer, both scope-range boundaries included. The dead commented-out blocks under answers 3 and 4 went with the move. The widened filter reservation is still untested: reaching it needs a crawl driving `query3`.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 10:22:44 +02:00
Xavier Roche
47a41fb39e webhttrack's browser dependency drags httrack toward autoremoval (#1235)
The autoremoval gatherer reads only the first alternative of a disjunction and never looks at Recommends, so webhttrack's `chromium | firefox-esr | www-browser` in Depends put the whole httrack source on chromium's RC bug clock. [#1128867](https://bugs.debian.org/1128867) has 3.49.14-1 marked for removal from testing on 9 September. Nothing here wants chromium in particular: `src/webhttrack.in` searches `x-www-browser`, then `www-browser`, then a dozen binaries by name.

#436 flipped this list the other way round in June, back when firefox-esr's RC bugs were doing the same to us, and six weeks later chromium got one. Whichever real browser leads the list is big and bug-prone, so the disjunction moves to Recommends, where the gatherer cannot follow it. Apt installs Recommends by default, so a normal install is unchanged. The `webhttrack` wrapper does still exit when it finds no browser (`src/webhttrack.in:97`), but the package also ships `htsserver`, which serves the same UI over HTTP to a browser on another machine, so the dependency is strong rather than absolute.

This only beats the removal if it is uploaded and migrates before 9 September. Testing is still on 3.49.14-1, with i386 and riscv64 at Needs-Build.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 07:10:54 +00:00
Xavier Roche
fabbf80ea5 The wizard can only answer about one exact host (#1239)
The interactive wizard could only answer about the exact host it asked
about, so a mirror of www.example.com stopped again at
download.example.com, and Skip All stayed unusable while any unknown
subdomain might still turn up. Reported against the GUI as
xroche/httrack-windows#96.

The engine now enumerates the domain scopes for a host and takes two new
answer ranges over them. hts_wizard_host_scope() does the splitting so a
front end never has to: it derives the scopes from the same question
string it was handed, which may carry a protocol, credentials and a port.
Answers HTS_WIZARD_SCOPE_INCLUDE+k and HTS_WIZARD_SCOPE_EXCLUDE+k then
take or drop the k-th scope. An index is only safe instead of a finished
pattern because of that helper: the menu label and the applied filter come
out of the same code and cannot drift.

Each answer emits two filters, since +*.example.co.uk/* misses the apex,
which also meant widening the filter-array reservation before the answer
switch. It made room for exactly one insert while HT_INSERT_FILTERS0
asserts on overflow, so the second insert would have aborted on the
boundary at filptr == maxfilter - 2. hts_wizard_scope_answer() carries the
range decision for both the filter and the verdict halves, so the two
cannot disagree.

No public suffix list is involved: every suffix down to a two-label domain
is offered and the user picks the boundary. A bare TLD is never offered,
an IP literal has no scopes, and a fully-qualified www.foo.com. stops at
foo.com. instead of offering the root label as com.

Only the CLI and WinHTTrack need the new menu entries. WebHTTrack answers
"" to every question and Android registers no query3 at all, which is
#1237.

Closes #1117

Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 08:51:10 +02:00
Xavier Roche
af0e36e326 A -%F footer that exactly fills the page buffer aborts the crawl (#1241)
* A footer that exactly fills the page buffer aborts the crawl

#670 dropped an oversized footer instead of emitting an unterminated buffer,
but left the success path alone: it appends the closing newline with
strcatbuff, which aborts rather than clips, so the one expansion length that
fills tempo exactly has nowhere to put that byte. Reserve the newline before
formatting, so an expansion that no longer fits takes the existing drop path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Format the touched call to clang-format

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Size the sweep's path instead of inheriting the temp one

The footer template caps at 253 chars, so the number of {path} references and
the literal padding both hang off the path's length. On the Windows runner the
temp directory is 80 chars, which pushed the template past the cap and failed
the test on its own guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Sweep both line endings and assert the footer arrives whole

The fixture was LF-only, so strlen(eol) was always 1 and a fix reserving one
byte rather than the eol's length passed while CRLF pages still aborted. The
emitted length was unchecked too, so a clipping fix read as success.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fail on a temp path too long, and name the length that died

A skip would red the Windows leg anyway, since it compares the skip set
exactly, while silently dropping the coverage on every other platform. The
wider path leaves more room before that bites. The abort message named the
template, identical at every step, rather than the expansion that aborted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 01:14:40 +00:00
Xavier Roche
2dc30ab490 The default footer still uses the legacy %s form (#1240)
* Default footer uses named fields instead of %s

The named-field footer (-%F "{url}") has been the documented model since #667,
but the default template still carried the positional %s form, so any field a
user added to it was silently ignored: a "%s" anywhere selects legacy mode for
the whole string. WebHTTrack carried its own copy of that literal; it now takes
the engine macro.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Point the WebHTTrack footer test at the macro's own comment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 00:55:41 +00:00
Xavier Roche
d681929d1a An abandoned WebHTTrack server never stops, holding the macOS app open (#1236)
* An abandoned WebHTTrack server never stops on its own

The watchdog only started consulting the heartbeat after the launcher died,
and on macOS the launcher waits on "open -W", which returns when the whole
browser quits. Closing the window therefore left htsserver running with the
app bundle open, so the disk image could not be ejected without a force kill.

Bind the session to the user's attention when idle and to the work when busy:
the page now says goodbye on pagehide, a running mirror vetoes every exit so a
crawl is never lost, and the launcher waits on the server rather than on the
browser. /ping was also the one reply served without cache headers, and
smallserver() returned 0 unconditionally, so every clean quit was reported as
"Unable to create the server".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Keep the idle fallback at 120s and settle the test before judging survivors

A backgrounded tab has its timers throttled, so the fallback stays where it
was; promptness is the goodbye's job. The two servers that must stay alive
started after the one whose death ends the wait, so the test judged them
before either had outlived a timeout of its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Report the killed server before the pings that failed because of it

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Count windows instead of racing a browser's throttled timers

A single pending-goodbye deadline cannot tell "the last window left" from "one
of several left", so closing one of two windows ended a session the other still
had open: a hidden tab's timers are throttled to as little as one wake-up a
minute, and its cancelling ping arrived after the countdown had fired. Each page
now carries an id, pings under it and drops it on pagehide, and the server exits
when the last one is gone.

Liveness is counted in watchdog ticks rather than wall clock, so a suspended
laptop no longer ages a session out from under the user. A bare connection no
longer cancels a departure, since any local peer can open one. And a failed
thread spawn left commandRunning latched, which the new mirror veto would have
turned into a server that never exits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Take a farewell only from a request holding the session id

/ping is a GET, so it clears neither the session-id gate nor the Origin check,
both of which are POST-only. That was harmless while a heartbeat could only
extend a session; ending one on an unauthenticated request handed every local
process, and every page the user visits, a way to close someone's WebHTTrack.
A full window table now refuses newcomers instead of evicting, so a flood of
ids cannot push the real window out either.

Also from review: query_alnum_value left a half-read value in the caller's
buffer when it returned false, the new predicates and flags were plain ints in
a tree that spells booleans hts_boolean, and --ping-timeout re-parsed with the
bare sscanf that #614 cost us on --port.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Prove a flood of window ids cannot evict the real one

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Name the launcher's pid with a type MSVC has

pid_t is POSIX, and the old code only ever named it inside an #ifndef _WIN32
block. Hoisting the parent check into a function signature put it where Windows
compiles it, and webhttrack.vcxproj stopped at seven errors on one line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-13 14:09:21 +00:00
Xavier Roche
1483289eec debian: record the 3.49.21-2 upload (#1234)
The revision carried the hppa FTBFS fix to the buildds as a quilt patch against
the frozen orig. The patch itself stays out of git, where the fix is already in
the source; only the changelog entry belongs here, so the file stays continuous
for the next upload.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 09:14:28 +00:00
Xavier Roche
1751c4b593 mkdeb.sh reaches debsign before it discovers the key is wrong (#1233)
The key is only resolved when debsign runs, which on a release is after the
tarball has been built. Ask gpg for it at startup instead, and name the
0x-prefixed fingerprint form in the error, since a short id is what people reach
for first and it is the ambiguous one.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 06:21:17 +00:00
Xavier Roche
bad28b629c A lost Windows runner leaves almost no telemetry behind (#1229)
The suite watchdog's commit statuses are the only evidence that survives a lost runner, and most of it was being thrown away: they went to the PR merge commit, whose SHA GitHub collects, leaving 3 of 146 recorded kills with any telemetry at all. They now go to the PR head, which costs us a line in the PR's checks list.

Each status carries more too. Loop lag and the number of posts that failed since one landed separate a box that stopped from a network that broke; `n=`/`f=`/`e=` come from one `GetTcpIPv4Statistics` call; `m=`/`c=`/`a=` are summed from the process array already enumerated for `p=` and `h=`. The cadence halves to 15s, and the in-flight test name is clipped to 30 characters to pay for the new fields. The widest real line measures 103 of the 140 characters GitHub keeps.

The rest closes two unsound spots in the Windows kill paths, neither of which explains the deaths. `stop_server` re-read `/proc/<pid>/winpid` after signalling its target, so the number it handed to `taskkill /F /T` could already belong to a stranger, and a hosted runner reissues a freed PID within a second rather than in theory. Nothing checked the image before firing either. Both the winpid and the image are now read while the target is alive and checked against `tasklist`, with a warning annotation when they disagree, which is also the meter for how often a tree kill would have left our own trees.

Refs #1228.
2026-08-13 08:08:03 +02:00
Xavier Roche
6584fded15 mkdeb.sh regenerates an orig that debian/patches cannot apply to (#1232)
A quilt patch is written against the tarball it will be applied to. Once the fix
it backports is upstream, HEAD carries it too, so an orig regenerated from HEAD
makes the patch fail or apply with fuzz. Require --orig whenever the series file
is non-empty, unsigned builds included: this one breaks the build, not policy.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 05:43:13 +00:00
Xavier Roche
d8f0cec17d Three suite tests fail on an emulated buildd for being slow, not broken (#1231)
* Three suite tests fail on an emulated buildd for being slow, not broken

hppa reported 3.49.21-1 as Build-Attempted with 105, 151 and 269 red, all three
on wall clock under qemu-user: the guard's own diagnostics dump took 31s of a
30s bound, one configure run passed a hard 300s cap, and the pairwise header
sweep outran the 600s harness budget mid-batch.

Each now measures the property instead of the host. 105 times the guard to its
DUMP announcement, leaving the dump (minutes, emulated) out of the bound. 151
watches configure's output for silence rather than capping its total, and skips
when the budget runs out while it is still making progress. The sweep runs in
slices so it can be given up on, paces itself against what 269 hands it, and 269
declares the larger budget its n^2 compiles need, so the emulated leg still runs
it to completion instead of pacing out.

The emulated leg is the CI counterpart of that buildd and was green on the same
commit: it prints test-suite.log now, so a test that skipped rather than ran
stops reading as coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Fix the review's two skip-masking holes and the unbounded budget raise

A hung configure could reach the budget skip before the silence window closed, so
a wedge reported SKIP; the skip now needs a run that is still writing and enough
budget left for the detector to speak. 151 also paced inside run(), before
accept/reject read the verdict, so a configure that answered wrongly could exit 77
instead of 1: pacing moved to the callers, which judge first.

The per-test raise is bounded and normalized through one budget parser, since bash
test errors rather than compares past intmax and would have left the guard unarmed,
and a leading zero read as octal in arithmetic and decimal in test. Renamed
TEST_TIMEOUT_AT_LEAST so each use site carries the upwards-only rule, documented in
AGENTS.md, and the sweep takes an explicit --budget rather than sniffing the
environment, so the MSVC job cannot report a paced skip as a header break.

Tests for each: 151 drives run() through a configure_cmd seam with a child that
hangs and one that only crawls, 105 requires the DUMP announcement to be seen while
the guard runs (its fallback made the latency bound vacuous) and pins the hostile
budget values, and the sweep counts the units that reached the compiler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* Two of the new assertions passed a mutant; make them bite

Mutation-testing the added tests found two that could not see the bug they were
written for. The verdict check stubbed run() out, so a pacer left inside the real
one stayed invisible: it now drives the real run() through a child that answers
wrongly with the budget spent. The announcement check only asked that the marker
appear while the guard ran, which a driver announcing after the dump still
satisfies: a slow ps widens the dump, and the marker must now precede it.

Writing the first exposed a third: reject takes one argument fewer than accept, so
the extra one reached run() as an env assignment and the child failed to exec,
which made the probe pass on the wrong answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* The verdict probe left the pacer disarmed

It numbered its run past the case count, so the pacer it exists to catch declined
to fire on a negative "steps left", and the mutant that puts pacing back inside
run() survived. Keep cases ahead of the run number.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* A killed configure left its child running and hung the macOS job

The macOS leg reported no test failure and then sat until its 20-minute step
timeout: bash 3.2 does not replace the subshell around the child with the child
itself, so killing that subshell left the run alive, and it outlived make check.
Give each run its own process group and kill the group.

The probe that exposed it now covers it: a child that spawns a child of its own,
and nothing of it left running afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 01:01:27 +02:00
89 changed files with 3458 additions and 847 deletions

View File

@@ -18,18 +18,29 @@ concurrency:
jobs:
build:
name: build (${{ matrix.arch }}, ${{ matrix.cc }})
name: build (${{ matrix.arch }}, ${{ matrix.cc }}${{ matrix.label }})
runs-on: ${{ matrix.runner }}
strategy:
fail-fast: false
matrix:
# cflags is spelled out everywhere: an exported empty CFLAGS reads as
# "set" to configure, which then drops its own -g -O2.
include:
- { arch: x86-64, runner: ubuntu-24.04, cc: gcc }
- { arch: x86-64, runner: ubuntu-24.04, cc: clang }
- { arch: arm64, runner: ubuntu-24.04-arm, cc: gcc }
- { arch: arm64, runner: ubuntu-24.04-arm, cc: clang }
- { arch: x86-64, runner: ubuntu-24.04, cc: gcc, cflags: -g -O2 }
- { arch: x86-64, runner: ubuntu-24.04, cc: clang, cflags: -g -O2 }
- { arch: arm64, runner: ubuntu-24.04-arm, cc: gcc, cflags: -g -O2 }
- { arch: arm64, runner: ubuntu-24.04-arm, cc: clang, cflags: -g -O2 }
# Ubuntu builds httrack this way; no other leg here does.
- {
arch: x86-64,
runner: ubuntu-24.04,
cc: gcc,
label: " -O3 -flto",
cflags: -g -O3 -flto=auto -ffat-lto-objects,
}
env:
CC: ${{ matrix.cc }}
CFLAGS: ${{ matrix.cflags }}
steps:
- uses: actions/checkout@v7
with:

View File

@@ -270,8 +270,9 @@ jobs:
# Through the environment, never argv, which the process list exposes.
WATCHDOG_TOKEN: ${{ secrets.GITHUB_TOKEN }}
WATCHDOG_REPO: ${{ github.repository }}
# github.sha here is the PR's merge commit, so statuses posted against it stay out of the PR's checks UI.
WATCHDOG_SHA: ${{ github.sha }}
# The PR head, not github.sha: a merge commit is garbage-collected, and
# these statuses are the only trace a lost runner leaves (#1228).
WATCHDOG_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
WATCHDOG_CONTEXT: windows-suite (${{ matrix.platform }}, ${{ matrix.configuration }})
WATCHDOG_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |

View File

@@ -1,364 +0,0 @@
# Scratch probe: does taskkill /F /T follow a recorded parent PID whose owner is
# long dead? Not part of the build gate; branch-scoped, delete when answered.
name: zz pid-reuse probe
on:
push:
branches: [zz-pidreuse-probe]
permissions:
contents: read
jobs:
probe:
runs-on: windows-2022
timeout-minutes: 15
steps:
- name: Probe
shell: pwsh
run: |
$ErrorActionPreference = 'Continue'
$ProgressPreference = 'SilentlyContinue'
$sum = $env:GITHUB_STEP_SUMMARY
function Say([string]$m) {
Write-Host $m
if ($sum) { try { Add-Content -LiteralPath $sum -Value $m -ErrorAction SilentlyContinue } catch { } }
}
Say "# PID-reuse probe on $env:COMPUTERNAME $(Get-Date -Format o)"
Say ""
Say '```'
# ---------------------------------------------------------------
# Part 1: is the hazard even present?
# ---------------------------------------------------------------
Say "== PART 1: recorded-parent state of the runner's own processes =="
$snap = @(Get-CimInstance Win32_Process)
$byPid = @{}
foreach ($p in $snap) { $byPid[[int]$p.ProcessId] = $p }
Say ("snapshot: {0} live processes" -f $snap.Count)
function Stamp($d) { if ($null -eq $d) { return '?' } else { return ([datetime]$d).ToString('HH:mm:ss.fff') } }
function Classify([int]$q) {
$c = $byPid[$q]
if ($null -eq $c) { return 'NOPROC' }
$par = $byPid[[int]$c.ParentProcessId]
if ($null -eq $par) { return 'DEAD-PARENT' }
if ($null -eq $par.CreationDate -or $null -eq $c.CreationDate) { return 'OK' }
if ($par.CreationDate -gt $c.CreationDate) { return 'RECYCLED-PARENT' }
return 'OK'
}
function Chain([int]$q) {
$out = @()
$seen = @{}
$cur = $q
for ($i = 0; $i -lt 24; $i++) {
$c = $byPid[$cur]
if ($null -eq $c) { $out += ("{0}=<no such process>" -f $cur); break }
$out += ("{0}/{1}@{2}" -f $c.ProcessId, $c.Name, (Stamp $c.CreationDate))
if ($seen.ContainsKey($cur)) { $out += '<cycle>'; break }
$seen[$cur] = $true
$st = Classify $cur
if ($st -ne 'OK') { $out += ("<<{0}>>" -f $st); break }
$cur = [int]$c.ParentProcessId
if ($cur -eq 0) { break }
}
return ($out -join ' <- ')
}
$interesting = @($snap | Where-Object {
$_.Name -like 'Runner.*' -or $_.Name -like 'Agent.*' -or $_.Name -like 'hosted-*' -or
$_.Name -eq 'bash.exe' -or $_.Name -eq 'pwsh.exe' -or
$_.Name -eq 'powershell.exe' -or $_.Name -eq 'dotnet.exe' -or
$_.Name -eq 'node.exe' -or $_.Name -eq 'cmd.exe' -or
$_.Name -eq 'conhost.exe' -or $_.Name -eq 'sh.exe'
} | Sort-Object Name, ProcessId)
$armed = 0
foreach ($p in $interesting) {
$st = Classify ([int]$p.ProcessId)
Say (" {0,-24} pid={1,-7} ppid={2,-7} {3,-16} {4}" -f $p.Name, $p.ProcessId, $p.ParentProcessId, $st, (Chain ([int]$p.ProcessId)))
if ($st -ne 'OK' -and ($p.Name -like 'Runner.*' -or $p.Name -like 'Agent.*' -or $p.Name -like 'hosted-*')) { $armed++ }
}
$dead = 0; $rec = 0; $ok = 0
$boxOrphan = @{}
foreach ($p in $snap) {
$st = Classify ([int]$p.ProcessId)
if ($st -eq 'DEAD-PARENT') {
$dead++
$k = [int]$p.ParentProcessId
if (-not $boxOrphan.ContainsKey($k)) { $boxOrphan[$k] = @() }
$boxOrphan[$k] += ("{0}/{1}" -f $p.ProcessId, $p.Name)
} elseif ($st -eq 'RECYCLED-PARENT') { $rec++ } else { $ok++ }
}
Say ""
Say ("box-wide: {0} processes, {1} with a DEAD recorded parent, {2} with a RECYCLED one, {3} sound" -f $snap.Count, $dead, $rec, $ok)
foreach ($k in ($boxOrphan.Keys | Sort-Object)) {
Say (" dead ppid {0,-7} still claimed as parent by: {1}" -f $k, ($boxOrphan[$k] -join ', '))
}
Say ""
if ($armed -gt 0) {
Say "PART 1 VERDICT: HAZARD PRESENT - $armed runner-agent process(es) record a parent PID that is dead or already recycled."
} elseif ($dead -gt 0) {
Say "PART 1 VERDICT: no Runner.*/Agent.* process has a stale recorded parent, but $dead other live processes do."
} else {
Say "PART 1 VERDICT: NOT ARMED - every live process's recorded parent is alive and older."
}
Say ""
# ---------------------------------------------------------------
# Part 2. Five sets of freed PIDs, released at the same instant, so
# that a null on the set the theory needs can be told apart from a
# probe that never recycles anything:
# A plain ping, killed, .NET handle disposed (baseline: does reuse happen?)
# C plain ping, killed, handle deliberately HELD (does a handle pin the PID?)
# D cmd orphan-maker, child killed (is the construction sound?)
# B cmd orphan-maker, ping child left ALIVE (the question)
# E cmd orphan-maker, wscript child left ALIVE (same, with a child that
# attaches to no console)
# Every process inherits this shell's console, so no console host survives
# to hold a handle on the dead cmd. Run 2 created 500 of those.
# ---------------------------------------------------------------
Say "== PART 2: can a dead parent's PID be handed out again, and does taskkill /T follow it? =="
$vbs = Join-Path $env:TEMP 'zzsleep.vbs'
Set-Content -LiteralPath $vbs -Value 'WScript.Sleep 600000' -Encoding ascii
function New-Ping {
$psi = New-Object System.Diagnostics.ProcessStartInfo
$psi.FileName = "$env:SystemRoot\System32\PING.EXE"
$psi.Arguments = '-n 600 127.0.0.1'
$psi.UseShellExecute = $false
$psi.CreateNoWindow = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
return [System.Diagnostics.Process]::Start($psi)
}
function New-Orphan([string]$childArgs) {
# cmd exits at once, leaving the child alive with cmd's now-free PID recorded as parent.
$psi = New-Object System.Diagnostics.ProcessStartInfo
$psi.FileName = "$env:ComSpec"
$psi.Arguments = $childArgs
$psi.UseShellExecute = $false
$psi.CreateNoWindow = $false
return [System.Diagnostics.Process]::Start($psi)
}
function Get-Proc([int]$q) {
return (Get-CimInstance Win32_Process -Filter "ProcessId = $q" -ErrorAction SilentlyContinue | Select-Object -First 1)
}
# --- control: a tree kill of a LIVE parent must kill its child, or the probe is blind.
Say ""
Say "-- control: taskkill /F /T on a live cmd whose child ping is genuine"
$ctl = New-Orphan '/c ping -n 600 127.0.0.1 > NUL'
Start-Sleep -Milliseconds 1500
$ctlKid = @(Get-CimInstance Win32_Process -Filter "ParentProcessId = $($ctl.Id)")
if ($ctlKid.Count -eq 0) {
Say " control INCONCLUSIVE: no child found under cmd $($ctl.Id)"
try { $ctl.Kill() } catch { }
} else {
$kp = [int]$ctlKid[0].ProcessId
Say (" cmd {0} -> child {1}/{2}" -f $ctl.Id, $kp, $ctlKid[0].Name)
$co = (& taskkill.exe /F /T /PID $ctl.Id 2>&1 | Out-String).Trim()
Say (" taskkill: " + ($co -replace "`r?`n", ' | '))
Start-Sleep -Milliseconds 800
$still = Get-Proc $kp
Say (" control result: child {0} {1}" -f $kp, $(if ($null -eq $still) { 'GONE - the probe can observe a tree kill' } else { 'STILL ALIVE - probe is blind, part 2 is inconclusive' }))
}
try { $ctl.Dispose() } catch { }
$conBefore = @(Get-Process conhost -ErrorAction SilentlyContinue).Count
# --- build the sets.
$NB = 300; $ND = 200; $NE = 200; $NA = 200; $NC = 200
Say ""
Say "-- building sets: B=$NB (ping child alive) E=$NE (wscript child alive) D=$ND (child killed) A=$NA (baseline) C=$NC (handle held)"
$bCmd = New-Object 'System.Collections.Generic.List[System.Diagnostics.Process]'
for ($i = 0; $i -lt $NB; $i++) { try { $bCmd.Add((New-Orphan '/c start /b ping -n 600 127.0.0.1 > NUL')) } catch { } }
$eCmd = New-Object 'System.Collections.Generic.List[System.Diagnostics.Process]'
for ($i = 0; $i -lt $NE; $i++) { try { $eCmd.Add((New-Orphan ('/c start /b wscript.exe //B //Nologo "' + $vbs + '"'))) } catch { } }
$dCmd = New-Object 'System.Collections.Generic.List[System.Diagnostics.Process]'
for ($i = 0; $i -lt $ND; $i++) { try { $dCmd.Add((New-Orphan '/c start /b ping -n 600 127.0.0.1 > NUL')) } catch { } }
$aProc = New-Object 'System.Collections.Generic.List[System.Diagnostics.Process]'
for ($i = 0; $i -lt $NA; $i++) { try { $aProc.Add((New-Ping)) } catch { } }
$pinned = New-Object 'System.Collections.Generic.List[System.Diagnostics.Process]'
for ($i = 0; $i -lt $NC; $i++) { try { $pinned.Add((New-Ping)) } catch { } }
$bPid = New-Object 'System.Collections.Generic.HashSet[int]'
foreach ($p in $bCmd) { [void]$bPid.Add([int]$p.Id) }
$ePid = New-Object 'System.Collections.Generic.HashSet[int]'
foreach ($p in $eCmd) { [void]$ePid.Add([int]$p.Id) }
$dPid = New-Object 'System.Collections.Generic.HashSet[int]'
foreach ($p in $dCmd) { [void]$dPid.Add([int]$p.Id) }
$aPid = New-Object 'System.Collections.Generic.HashSet[int]'
foreach ($p in $aProc) { [void]$aPid.Add([int]$p.Id) }
$cPid = New-Object 'System.Collections.Generic.HashSet[int]'
foreach ($p in $pinned) { [void]$cPid.Add([int]$p.Id) }
Say (" distinct PIDs held: B={0} E={1} D={2} A={3} C={4}" -f $bPid.Count, $ePid.Count, $dPid.Count, $aPid.Count, $cPid.Count)
if ($bPid.Count -eq 0 -or $aPid.Count -eq 0) { Say " set construction failed, aborting"; Say '```'; exit 1 }
Start-Sleep -Seconds 3
# Map the surviving children back to the cmd that made them.
$orphan = @{}
$dKids = New-Object 'System.Collections.Generic.List[int]'
$nB = 0; $nE = 0
foreach ($p in @(Get-CimInstance Win32_Process | Where-Object { $_.Name -eq 'ping.exe' -or $_.Name -eq 'wscript.exe' })) {
$pp = [int]$p.ParentProcessId
if ($bPid.Contains($pp) -or $ePid.Contains($pp)) {
if (-not $orphan.ContainsKey($pp)) { $orphan[$pp] = @() }
$orphan[$pp] += [pscustomobject]@{ Pid = [int]$p.ProcessId; Created = $p.CreationDate }
if ($bPid.Contains($pp)) { $nB++ } else { $nE++ }
} elseif ($dPid.Contains($pp)) {
$dKids.Add([int]$p.ProcessId)
}
}
Say (" surviving orphan children: {0} under B parents, {1} under E parents, {2} under D parents (to be killed)" -f $nB, $nE, $dKids.Count)
# Free every set at the same instant. Only C keeps its .NET handle open.
foreach ($p in $bCmd) { try { [void]$p.WaitForExit(2000) } catch { }; try { $p.Dispose() } catch { } }
foreach ($p in $eCmd) { try { [void]$p.WaitForExit(2000) } catch { }; try { $p.Dispose() } catch { } }
foreach ($p in $dCmd) { try { [void]$p.WaitForExit(2000) } catch { }; try { $p.Dispose() } catch { } }
foreach ($p in $aProc) { try { $p.Kill() } catch { } }
foreach ($p in $pinned) { try { $p.Kill() } catch { } }
Start-Sleep -Milliseconds 1000
foreach ($p in $aProc) { try { $p.Dispose() } catch { } }
$aProc.Clear(); $bCmd.Clear(); $eCmd.Clear(); $dCmd.Clear()
if ($dKids.Count -gt 0) { Stop-Process -Id $dKids.ToArray() -Force -ErrorAction SilentlyContinue }
[GC]::Collect(); [GC]::WaitForPendingFinalizers(); [GC]::Collect()
Start-Sleep -Seconds 2
$liveNow = New-Object 'System.Collections.Generic.HashSet[int]'
foreach ($p in @(Get-CimInstance Win32_Process)) { [void]$liveNow.Add([int]$p.ProcessId) }
function CountLive($s) { $n = 0; foreach ($x in $s) { if ($liveNow.Contains($x)) { $n++ } }; return $n }
Say (" set members still live (must be 0): B={0} E={1} D={2} A={3} C={4}" -f (CountLive $bPid), (CountLive $ePid), (CountLive $dPid), (CountLive $aPid), (CountLive $cPid))
$conAfter = @(Get-Process conhost -ErrorAction SilentlyContinue).Count
Say (" conhost.exe: {0} before, {1} after (a surviving console host would hold a handle on its dead creator and pin its PID)" -f $conBefore, $conAfter)
$kidsAlive = 0
foreach ($k in $orphan.Keys) { foreach ($o in $orphan[$k]) { if ($liveNow.Contains($o.Pid)) { $kidsAlive++ } } }
Say (" orphan children still alive under a dead B/E parent: {0}" -f $kidsAlive)
# --- hunt.
$hitA = New-Object 'System.Collections.Generic.HashSet[int]'
$hitB = New-Object 'System.Collections.Generic.HashSet[int]'
$hitC = New-Object 'System.Collections.Generic.HashSet[int]'
$hitD = New-Object 'System.Collections.Generic.HashSet[int]'
$hitE = New-Object 'System.Collections.Generic.HashSet[int]'
$boxHits = 0
$spawns = 0; $tested = 0; $followed = 0; $survived = 0
$KILLCAP = 25
$distinct = New-Object 'System.Collections.Generic.HashSet[int]'
$window = New-Object 'System.Collections.Generic.Queue[System.Diagnostics.Process]'
$minPid = [int]::MaxValue; $maxPid = 0
function Test-Hit([int]$np) {
if ($script:aPid.Contains($np)) { [void]$script:hitA.Add($np) }
if ($script:dPid.Contains($np)) { [void]$script:hitD.Add($np) }
if ($script:cPid.Contains($np)) {
if ($script:hitC.Add($np) -and $script:hitC.Count -le 5) { Say ("!! set C hit: PID $np was handed out although an open process handle on it is still held") }
}
if ($script:boxOrphan.ContainsKey($np)) {
$script:boxHits++
Say ""
Say ("!! REAL COLLISION: our spawn got PID $np, which live box process(es) still record as their parent: $($script:boxOrphan[$np] -join ', ')")
Say " (not tree-killing this one: it would reap a process we did not start)"
Say ""
}
$isB = $script:bPid.Contains($np)
$isE = $script:ePid.Contains($np)
if (-not ($isB -or $isE)) { return $false }
if ($isB) { [void]$script:hitB.Add($np) } else { [void]$script:hitE.Add($np) }
if ($script:tested -ge $script:KILLCAP) { return $false }
$live = @()
foreach ($o in $script:orphan[$np]) {
$cur = Get-Proc $o.Pid
if ($null -ne $cur -and $cur.CreationDate -eq $o.Created) { $live += $o }
}
if ($live.Count -eq 0) { return $false }
$script:tested++
Say ""
Say ("### HIT #$($script:tested) on set $(if ($isB) { 'B' } else { 'E' }): spawn #$($script:spawns) at t=$([int]$script:sw.Elapsed.TotalSeconds)s was handed PID $np, the PID of a dead parent whose child is still running")
foreach ($k in @(Get-CimInstance Win32_Process -Filter "ParentProcessId = $np")) {
Say (" the process table calls this a child of $np : {0}/{1} created {2}" -f $k.ProcessId, $k.Name, $k.CreationDate)
}
foreach ($o in $live) { Say (" stale-PPID orphan {0} is alive before the kill" -f $o.Pid) }
Say (" issuing: taskkill /F /T /PID $np")
$out = (& taskkill.exe /F /T /PID $np 2>&1 | Out-String).Trim()
Say (" taskkill said: " + ($out -replace "`r?`n", ' | '))
Start-Sleep -Milliseconds 1000
$anyDied = $false
foreach ($o in $live) {
$cur = Get-Proc $o.Pid
$alive = ($null -ne $cur -and $cur.CreationDate -eq $o.Created)
if (-not $alive) { $anyDied = $true }
Say (" stale-PPID orphan {0}: alive after the kill = {1}" -f $o.Pid, $alive)
}
if (-not $anyDied) { $script:survived++ }
if ($anyDied) {
$script:followed++
Say " >>> THE TREE WALK FOLLOWED A STALE PPID. taskkill /T does not validate that the recorded parent is the real one."
} else {
Say " >>> the orphan survived: taskkill /T did not follow the stale PPID."
}
Say ""
return $false
}
$sw = [Diagnostics.Stopwatch]::StartNew()
Say ""
Say "-- hunting: rolling window of 200 live spawns, 300s cap"
$done = $false
while (-not $done -and $sw.Elapsed.TotalSeconds -lt 300) {
try { $np = (New-Ping) } catch { Start-Sleep -Milliseconds 50; continue }
$window.Enqueue($np)
$spawns++
$id = [int]$np.Id
[void]$distinct.Add($id)
if ($id -lt $minPid) { $minPid = $id }
if ($id -gt $maxPid) { $maxPid = $id }
try { $done = Test-Hit $id } catch { Say ("Test-Hit failed on $id : " + $_.Exception.Message) }
while ($window.Count -gt 200) {
$old = $window.Dequeue()
try { $old.Kill() } catch { }
try { $old.Dispose() } catch { }
}
if (($spawns % 250) -eq 0) { Write-Host (" ... spawn $spawns, distinct $($distinct.Count), A=$($hitA.Count) B=$($hitB.Count) C=$($hitC.Count) D=$($hitD.Count) E=$($hitE.Count), t=$([int]$sw.Elapsed.TotalSeconds)s") }
}
Say ""
Say "== RESULT =="
Say ("spawns {0}, distinct PIDs {1}, range {2}..{3}, {4:N0}s" -f $spawns, $distinct.Count, $minPid, $maxPid, $sw.Elapsed.TotalSeconds)
Say ("distinct set PIDs reissued, by set:")
Say (" A killed, handle disposed, no child : {0} of {1}" -f $hitA.Count, $aPid.Count)
Say (" C killed, handle still OPEN : {0} of {1}" -f $hitC.Count, $cPid.Count)
Say (" D dead orphan-maker, child killed : {0} of {1}" -f $hitD.Count, $dPid.Count)
Say (" B dead orphan-maker, ping child ALIVE : {0} of {1}" -f $hitB.Count, $bPid.Count)
Say (" E dead orphan-maker, wscript ALIVE : {0} of {1}" -f $hitE.Count, $ePid.Count)
Say ("tree kills issued on a reissued stale-parent PID: {0}; reached the orphan: {1}; orphan survived: {2}" -f $tested, $followed, $survived)
Say ("collisions with a dead PPID recorded by a process we did NOT start: {0}" -f $boxHits)
if ($tested -gt 0 -and $followed -eq 0) {
Say "INTERPRETATION: a dead parent's PID IS reissued while a live process still records it as its parent, in $tested measured cases, and taskkill /F /T left that process alone every time. The tree walk does not follow a stale PPID: it skips a claimed child that predates the process now holding the PID."
} elseif ($hitA.Count -eq 0 -and $hitB.Count -eq 0 -and $hitE.Count -eq 0) {
Say "INTERPRETATION: nothing came back, so this run recycled nothing and says nothing either way."
} elseif ($tested -eq 0) {
Say "INTERPRETATION: PIDs recycle (A>0) but no PID with a live process recording it as parent ever came back. On this kernel such a PID stays reserved, so a tree kill cannot land on one and the stale-PPID mechanism cannot fire."
} elseif ($followed -gt 0) {
Say "INTERPRETATION: a dead parent's PID IS reissued while a live process records it, and taskkill /F /T reaped that process. The mechanism is real on this image."
} else {
Say "INTERPRETATION: the PID was reissued, but taskkill /F /T left the stale-PPID orphan alone: the tree walk validates parentage."
}
Say '```'
while ($window.Count -gt 0) { $o = $window.Dequeue(); try { $o.Kill() } catch { } }
try { Get-Process ping, wscript -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue } catch { }
Write-Host "probe done"

View File

@@ -30,6 +30,11 @@ the operational checklist: toolchain, invariants, and how to ship a change.
check`, or `PATH="<bld>/src:$PATH"` for a manual run.
- Give new `.test` scripts `set -e`: the older ones predate the rule, so several
`local-crawl.sh` calls with no `set -e` report PASS on any non-last failure.
- Each test runs under a 600s wall-clock guard that reports a wedge as 124. A test
whose own work outlasts it raises the budget with a `# TEST_TIMEOUT_AT_LEAST: N`
line, at column 0 within its first 40 lines, and paces itself with
`skip_if_out_of_budget` so a host too slow to finish skips instead. The value only
ever raises the budget: nothing can disarm the guard.
- Run teardown with errexit off: `trap 'set +e; cleanup' EXIT`. Under `set -e` a
failing cleanup command becomes the test's exit status (#773). Keep the other
signals on their own `trap` line, or errexit stays off for the rest of the run.

10
debian/changelog vendored
View File

@@ -1,3 +1,13 @@
httrack (3.49.21-2) unstable; urgency=medium
* Fix the FTBFS on hppa: three suite tests measured the build host rather
than the property they cover, and failed on the qemu-user buildd purely
for being slower there. Patched from upstream, which now skips a step a
host is too slow to finish instead of failing the build
(skip-emulated-host-test-failures.patch).
-- Xavier Roche <xavier@debian.org> Thu, 13 Aug 2026 07:26:38 +0200
httrack (3.49.21-1) unstable; urgency=medium
* New upstream release: a site answering to several hostnames can now be

5
debian/control vendored
View File

@@ -29,7 +29,10 @@ Description: Copy websites to your computer (Offline browser)
Package: webhttrack
Architecture: any
Multi-Arch: foreign
Depends: ${misc:Depends}, ${shlibs:Depends}, webhttrack-common, sensible-utils, chromium | firefox-esr | www-browser
Depends: ${misc:Depends}, ${shlibs:Depends}, webhttrack-common, sensible-utils
# Recommends, not Depends: the autoremoval gatherer follows only a disjunction's
# first alternative, which ties the httrack source to whichever browser leads it.
Recommends: firefox-esr | chromium | www-browser
Replaces: webhttrack-common (<< 3.43.9-2)
Breaks: webhttrack-common (<< 3.43.9-2)
Suggests: httrack, httrack-doc

View File

@@ -1,26 +1,85 @@
// Function aimed to ping the webhttrack server regularly to keep it alive
// If the browser window is closed, the server will eventually shutdown
function ping_server() {
var iframe = document.getElementById('pingiframe');
if (iframe && iframe.src) {
iframe.src = iframe.src;
setTimeout(ping_server, 30000);
}
// Tell the server this window is alive, so an abandoned server stops instead of
// outliving the session. The period is the one htsweb.c sizes its timeout from.
var PING_PERIOD = 5000;
// Identifies this window for as long as it is open. The server counts windows,
// so closing one of two must not read as the session ending.
var PING_WINDOW =
String(Math.random()).replace(/[^0-9]/g, "") + String(new Date().getTime());
function ping_url(extra) {
// Unique, or a cached response would never reach the server again.
return "/ping?w=" + PING_WINDOW + "&t=" + new Date().getTime() +
(extra ? "&" + extra : "");
}
// Create an invisible iframe to hold the server ping result
// Only modern browsers will support that, but old browsers are compatible
// with the legacy "wait for browser PID" mode
if (document && document.createElement && document.body
&& document.body.appendChild && document.getElementById) {
var iframe = document.createElement('iframe');
if (iframe) {
iframe.id = 'pingiframe';
iframe.style.display = "none";
iframe.style.visibility = "hidden";
iframe.width = iframe.height = 0;
iframe.src = "/ping";
document.body.appendChild(iframe);
ping_server();
}
// An iframe is the fallback only: reassigning its src can push a history entry,
// which would turn the Back button into a walk through past heartbeats.
function ping_send(url) {
if (window.fetch) {
fetch(url, {cache : "no-store"});
return true;
}
var iframe = document.getElementById('pingiframe');
if (!iframe) {
return false;
}
iframe.src = url;
return true;
}
function ping_server() {
if (ping_send(ping_url())) {
setTimeout(ping_server, PING_PERIOD);
}
}
// The session id this page carries, empty on the few pages that hold no form.
function ping_sid() {
var f = document.getElementsByName('sid');
return f && f.length ? f[0].value : "";
}
// Closing the window is the common case, and waiting out the timeout for it
// would hold the server open long after the user considers it gone. The server
// takes this only from a request holding the session id, so it goes as a POST;
// a page without one falls back to the timeout.
function ping_leaving() {
var sid = ping_sid();
if (!sid) {
return;
}
var url = ping_url("e=bye");
var body = "sid=" + encodeURIComponent(sid);
var type = "application/x-www-form-urlencoded";
if (navigator.sendBeacon) {
navigator.sendBeacon(url, new Blob([ body ], {type : type}));
} else if (window.XMLHttpRequest) {
// Synchronous: the page is going away, and an async send dies with it.
var x = new XMLHttpRequest();
x.open("POST", url, false);
x.setRequestHeader("Content-Type", type);
x.send(body);
}
}
// Old browsers reach none of this and stay on the legacy "wait for the launcher
// to die" mode.
if (document && document.createElement && document.body &&
document.body.appendChild && document.getElementById) {
if (!window.fetch) {
var iframe = document.createElement('iframe');
if (iframe) {
iframe.id = 'pingiframe';
iframe.style.display = "none";
iframe.style.visibility = "hidden";
iframe.width = iframe.height = 0;
document.body.appendChild(iframe);
}
}
ping_server();
// pagehide, not unload: Safari's back/forward cache never fires unload.
if (window.addEventListener) {
window.addEventListener('pagehide', ping_leaving, false);
}
}

View File

@@ -740,6 +740,10 @@ LANG_M8
Mirror domain
LANG_M9
Ignore all
LANG_M10
Mirror %s and every host below it
LANG_M11
Ignore %s and every host below it
LANG_N1
Wizard query
LANG_N2

View File

@@ -946,6 +946,36 @@ Server terminated
Ñúðâúðúò íå îòãîâàðÿ
A fatal error has occurred during this mirror
Ôàòàëíà ãðåøêà ïðè ñúçäàâàíåòî íà òîçè îãëåäàëåí ñàéò
View Documentation
Ïðåãëåä íà äîêóìåíòàöèÿòà
Go To HTTrack Website
Êúì óåáñàéòà íà HTTrack
Go To HTTrack Forum
Êúì ôîðóìà íà HTTrack
View License
Ïðåãëåä íà ëèöåíçà
Beware: you local browser might be unable to browse files with embedded filenames
Âíèìàíèå: âàøèÿò ëîêàëåí áðàóçúð ìîæå äà íå óñïåå äà îòâîðè ôàéëîâå ñ âãðàäåíè èìåíà íà ôàéëîâå
Recreated HTTrack internal cached resources
Âúòðåøíèòå êåøèðàíè ðåñóðñè íà HTTrack áÿõà ñúçäàäåíè íàíîâî
Could not create internal cached resources
Âúòðåøíèòå êåøèðàíè ðåñóðñè íå ìîæàõà äà áúäàò ñúçäàäåíè
Could not get the system external storage directory
Ñèñòåìíàòà âúíøíà ïàìåò íå ìîæà äà áúäå íàìåðåíà
Could not write to:
Íå ìîæå äà ñå ïèøå â:
Read-only media (SDCARD)
Íîñèòåë ñàìî çà ÷åòåíå (SDCARD)
No storage media (SDCARD)
Íÿìà íîñèòåë çà ñúõðàíåíèå (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Äîêàòî òîçè ïðîáëåì íå áúäå îòñòðàíåí, HTTrack ìîæå äà íå óñïåå äà ñâàëÿ óåáñàéòîâå
HTTrack: mirror '%s' stopped!
HTTrack: îãëåäàëíîòî êîïèå „%s“ áåøå ñïðÿíî!
Click on this notification to restart the interrupted mirror
Íàòèñíåòå òîâà èçâåñòèå, çà äà ïðîäúëæèòå ïðåêúñíàòîòî îãëåäàëíî êîïèå
HTTrack: could not save profile for '%s'!
HTTrack: ïðîôèëúò çà „%s“ íå ìîæà äà áúäå çàïàçåí!
Proxy type:
Òèï íà ïðîêñè:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC segment size limit:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Çàïî÷âàíå íà íîâ WARC ñåãìåíò, êîãàòî òåêóùèÿò íàäõâúðëè òîçè áðîé áàéòîâå; îñòàâåòå ïðàçíî èëè 0 çà åäèí ôàéë.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ïñåâäîíèìè íà õîñòà:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Äðóãè èìåíà íà õîñòîâå, êîèòî îáñëóæâàò ñúùèÿ ñàéò, îáåäèíåíè â åäíî, ïî åäíî ïðàâèëî íà ðåä (íàïð. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Íàïðàâè îãëåäàëíî êîïèå íà %s è íà âñè÷êè õîñòîâå ïîä íåãî
Ignore %s and every host below it
Èãíîðèðàé %s è âñè÷êè õîñòîâå ïîä íåãî

View File

@@ -946,6 +946,36 @@ Server terminated
Servidor desconectado
A fatal error has occurred during this mirror
Ha ocurrido un error fatal durante esta copia
View Documentation
Ver la documentación
Go To HTTrack Website
Ir al sitio web de HTTrack
Go To HTTrack Forum
Ir al foro de HTTrack
View License
Ver la licencia
Beware: you local browser might be unable to browse files with embedded filenames
Atención: puede que su navegador local no pueda abrir archivos con nombres de archivo incrustados
Recreated HTTrack internal cached resources
Recursos internos en caché de HTTrack recreados
Could not create internal cached resources
No se han podido crear los recursos internos en caché
Could not get the system external storage directory
No se ha podido acceder al almacenamiento externo del sistema
Could not write to:
No se ha podido escribir en:
Read-only media (SDCARD)
Soporte de solo lectura (SDCARD)
No storage media (SDCARD)
No hay soporte de almacenamiento (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Puede que HTTrack no pueda descargar sitios web mientras no se corrija este problema
HTTrack: mirror '%s' stopped!
HTTrack: ¡el volcado «%s» se ha detenido!
Click on this notification to restart the interrupted mirror
Pulse esta notificación para reanudar el volcado interrumpido
HTTrack: could not save profile for '%s'!
HTTrack: ¡no se ha podido guardar el perfil de «%s»!
Proxy type:
Tipo de proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Tama
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Empezar un nuevo segmento WARC cuando el actual supere este número de bytes; déjelo en blanco o 0 para un solo archivo.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Alias de host:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Otros nombres de host que sirven este mismo sitio, unificados en uno solo, una regla por línea (p. ej. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Volcar %s y todos los hosts por debajo
Ignore %s and every host below it
Ignorar %s y todos los hosts por debajo

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Zobrazit dokumentaci
Go To HTTrack Website
Pøejít na web HTTrack
Go To HTTrack Forum
Pøejít na fórum HTTrack
View License
Zobrazit licenci
Beware: you local browser might be unable to browse files with embedded filenames
Pozor: váš místní prohlížeè nemusí umìt otevøít soubory s vloženými názvy souborù
Recreated HTTrack internal cached resources
Interní zdroje v mezipamìti HTTrack byly znovu vytvoøeny
Could not create internal cached resources
Interní zdroje v mezipamìti se nepodaøilo vytvoøit
Could not get the system external storage directory
Nepodaøilo se najít systémové externí úložištì
Could not write to:
Nelze zapisovat do:
Read-only media (SDCARD)
Médium jen pro ètení (SDCARD)
No storage media (SDCARD)
Žádné úložné médium (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Dokud nebude tento problém vyøešen, nemusí HTTrack stahovat weby
HTTrack: mirror '%s' stopped!
HTTrack: zrcadlení „%s“ bylo zastaveno!
Click on this notification to restart the interrupted mirror
Klepnutím na toto oznámení obnovíte pøerušené zrcadlení
HTTrack: could not save profile for '%s'!
HTTrack: profil pro „%s“ se nepodaøilo uložit!
Proxy type:
Typ proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Nejv
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Zaèít nový segment WARC, jakmile aktuální pøekroèí tento poèet bajtù; ponechte prázdné nebo 0 pro jeden soubor.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Aliasy hostitele:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Další názvy hostitelù poskytující tentýž web, slouèené do jednoho, jedno pravidlo na øádek (napø. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Zrcadlit %s a všechny hostitele pod ním
Ignore %s and every host below it
Ignorovat %s a všechny hostitele pod ním

View File

@@ -679,7 +679,7 @@ Mirror site
Mirror domain
鏡像網域名稱
Ignore all
全部
全部
Wizard query
精靈提問
NO
@@ -946,6 +946,36 @@ Server terminated
伺服器已終止
A fatal error has occurred during this mirror
這鏡像發生了不可回復的錯誤
View Documentation
檢視說明文件
Go To HTTrack Website
前往 HTTrack 網站
Go To HTTrack Forum
前往 HTTrack 論壇
View License
檢視授權條款
Beware: you local browser might be unable to browse files with embedded filenames
注意:本機瀏覽器可能無法開啟檔名內嵌的檔案
Recreated HTTrack internal cached resources
已重新建立 HTTrack 內部快取資源
Could not create internal cached resources
無法建立內部快取資源
Could not get the system external storage directory
無法取得系統外部儲存空間目錄
Could not write to:
無法寫入:
Read-only media (SDCARD)
唯讀媒體 (SDCARD)
No storage media (SDCARD)
沒有儲存媒體 (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
在此問題修正前HTTrack 可能無法下載網站
HTTrack: mirror '%s' stopped!
HTTrack鏡像「%s」已停止
Click on this notification to restart the interrupted mirror
點選此通知以重新啟動中斷的鏡像
HTTrack: could not save profile for '%s'!
HTTrack無法儲存「%s」的設定檔
Proxy type:
proxy 類型:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
目前分段超過此位元組數時開始新的 WARC 分段;留空或填 0 則保持單一檔案。
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
主機別名:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
提供同一網站的其他主機名稱,合併為一個,每行一條規則(例如 www2.example.com,m.example.com=example.com
Mirror %s and every host below it
鏡像 %s 及其下所有主機
Ignore %s and every host below it
忽略 %s 及其下所有主機

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
查看文档
Go To HTTrack Website
前往 HTTrack 网站
Go To HTTrack Forum
前往 HTTrack 论坛
View License
查看许可协议
Beware: you local browser might be unable to browse files with embedded filenames
注意:本地浏览器可能无法打开文件名内嵌的文件
Recreated HTTrack internal cached resources
已重新创建 HTTrack 内部缓存资源
Could not create internal cached resources
无法创建内部缓存资源
Could not get the system external storage directory
无法获取系统外部存储目录
Could not write to:
无法写入:
Read-only media (SDCARD)
只读介质 (SDCARD)
No storage media (SDCARD)
没有存储介质 (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
在此问题修复前HTTrack 可能无法下载网站
HTTrack: mirror '%s' stopped!
HTTrack镜像“%s”已停止
Click on this notification to restart the interrupted mirror
点击此通知以重新启动中断的镜像
HTTrack: could not save profile for '%s'!
HTTrack无法保存“%s”的配置文件
Proxy type:
代理类型:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
当前分段超过该字节数时开始新的 WARC 分段;留空或填 0 则保持单个文件。
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
主机别名:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
提供同一网站的其他主机名,合并为一个,每行一条规则(例如 www2.example.com,m.example.com=example.com
Mirror %s and every host below it
镜像 %s 及其下所有主机
Ignore %s and every host below it
忽略 %s 及其下所有主机

View File

@@ -946,6 +946,36 @@ Server terminated
Poslužitelj je razriješen
A fatal error has occurred during this mirror
Tijekom ovog zrcaljenja je nastala fatalna pogreška
View Documentation
Prikaži dokumentaciju
Go To HTTrack Website
Otvori mrežno mjesto HTTrack
Go To HTTrack Forum
Otvori forum HTTrack
View License
Prikaži licencu
Beware: you local browser might be unable to browse files with embedded filenames
Pozor: vaš lokalni preglednik možda neæe moæi otvoriti datoteke s ugraðenim nazivima datoteka
Recreated HTTrack internal cached resources
Interni predmemorirani resursi HTTracka ponovno su stvoreni
Could not create internal cached resources
Interne predmemorirane resurse nije moguæe stvoriti
Could not get the system external storage directory
Nije moguæe pronaæi vanjsku pohranu sustava
Could not write to:
Nije moguæe pisati u:
Read-only media (SDCARD)
Medij samo za èitanje (SDCARD)
No storage media (SDCARD)
Nema medija za pohranu (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Dok se taj problem ne riješi, HTTrack možda neæe moæi preuzimati mrežna mjesta
HTTrack: mirror '%s' stopped!
HTTrack: zrcaljenje „%s” je zaustavljeno!
Click on this notification to restart the interrupted mirror
Dodirnite tu obavijest za nastavak prekinutog zrcaljenja
HTTrack: could not save profile for '%s'!
HTTrack: profil za „%s” nije moguæe spremiti!
Proxy type:
Vrsta posrednika:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Najve
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Zapoèni novi WARC segment kada trenutni prijeðe ovaj broj bajtova; ostavite prazno ili 0 za jednu datoteku.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Aliasi poslužitelja:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ostala imena poslužitelja koja poslužuju isto mrežno mjesto, sažeta u jedno, jedno pravilo po retku (npr. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Zrcali %s i sve poslužitelje ispod njega
Ignore %s and every host below it
Zanemari %s i sve poslužitelje ispod njega

View File

@@ -1045,6 +1045,10 @@ St
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Begynd et nyt WARC-segment, når det aktuelle overstiger dette antal byte; lad feltet stå tomt eller skriv 0 for én fil.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Værtsaliasser:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Andre værtsnavne der betjener det samme websted, samlet til ét, én regel pr. linje (f.eks. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Spejlkopiér %s og alle værter under den
Ignore %s and every host below it
Ignorer %s og alle værter under den

View File

@@ -946,6 +946,36 @@ Server terminated
Der Server wurde beendet
A fatal error has occurred during this mirror
Fataler Fehler während der Webseiten-Kopie
View Documentation
Dokumentation anzeigen
Go To HTTrack Website
Zur HTTrack-Website
Go To HTTrack Forum
Zum HTTrack-Forum
View License
Lizenz anzeigen
Beware: you local browser might be unable to browse files with embedded filenames
Achtung: Ihr lokaler Browser kann Dateien mit eingebetteten Dateinamen möglicherweise nicht öffnen
Recreated HTTrack internal cached resources
Interne zwischengespeicherte Ressourcen von HTTrack neu angelegt
Could not create internal cached resources
Interne zwischengespeicherte Ressourcen konnten nicht angelegt werden
Could not get the system external storage directory
Der externe Systemspeicher konnte nicht ermittelt werden
Could not write to:
Schreiben nicht möglich in:
Read-only media (SDCARD)
Schreibgeschützter Datenträger (SDCARD)
No storage media (SDCARD)
Kein Datenträger vorhanden (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Solange dieses Problem besteht, kann HTTrack möglicherweise keine Websites herunterladen
HTTrack: mirror '%s' stopped!
HTTrack: Kopie "%s" wurde angehalten!
Click on this notification to restart the interrupted mirror
Tippen Sie auf diese Benachrichtigung, um die unterbrochene Kopie fortzusetzen
HTTrack: could not save profile for '%s'!
HTTrack: Profil für "%s" konnte nicht gespeichert werden!
Proxy type:
Proxy-Typ:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Maximale Gr
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Ein neues WARC-Segment beginnen, sobald das aktuelle diese Anzahl Bytes überschreitet; leer lassen oder 0 für eine einzige Datei.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Host-Aliase:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Weitere Hostnamen derselben Website, auf einen zusammengefasst, eine Regel pro Zeile (z. B. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
%s und alle darunter liegenden Hosts kopieren
Ignore %s and every host below it
%s und alle darunter liegenden Hosts ignorieren

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Vaata dokumentatsiooni
Go To HTTrack Website
Ava HTTracki veebisait
Go To HTTrack Forum
Ava HTTracki foorum
View License
Vaata litsentsi
Beware: you local browser might be unable to browse files with embedded filenames
Tähelepanu: kohalik brauser ei pruugi suuta avada faile, mille nimi on faili sisse põimitud
Recreated HTTrack internal cached resources
HTTracki sisemised puhverdatud ressursid loodi uuesti
Could not create internal cached resources
Sisemisi puhverdatud ressursse ei õnnestunud luua
Could not get the system external storage directory
Süsteemi välist salvestusruumi ei leitud
Could not write to:
Ei saa kirjutada asukohta:
Read-only media (SDCARD)
Kirjutuskaitstud andmekandja (SDCARD)
No storage media (SDCARD)
Andmekandjat pole (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Kuni see probleem püsib, ei pruugi HTTrack veebisaite alla laadida
HTTrack: mirror '%s' stopped!
HTTrack: koopia "%s" peatati!
Click on this notification to restart the interrupted mirror
Katkenud koopia jätkamiseks puuduta seda teadet
HTTrack: could not save profile for '%s'!
HTTrack: profiili "%s" ei õnnestunud salvestada!
Proxy type:
Proxy tüüp:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC-segmendi suurim maht:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Alusta uut WARC-segmenti, kui praegune ületab selle baitide arvu; jäta tühjaks või 0, et hoida üks fail.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Hosti aliased:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Muud sama saiti teenindavad hostinimed, koondatud üheks, üks reegel rea kohta (nt www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Kopeeri %s ja kõik selle all olevad hostid
Ignore %s and every host below it
Ignoreeri %s ja kõiki selle all olevaid hoste

View File

@@ -1048,3 +1048,7 @@ Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Mirror %s and every host below it
Ignore %s and every host below it
Ignore %s and every host below it

View File

@@ -946,6 +946,36 @@ Server terminated
Palvelin lopetettu
A fatal error has occurred during this mirror
Tällä peilillä tapahtui vakava virhe
View Documentation
Näytä ohjeet
Go To HTTrack Website
Siirry HTTrackin sivustolle
Go To HTTrack Forum
Siirry HTTrackin foorumille
View License
Näytä lisenssi
Beware: you local browser might be unable to browse files with embedded filenames
Huomio: paikallinen selaimesi ei ehkä pysty avaamaan tiedostoja, joiden nimi on upotettu
Recreated HTTrack internal cached resources
HTTrackin sisäiset välimuistiresurssit luotiin uudelleen
Could not create internal cached resources
Sisäisiä välimuistiresursseja ei voitu luoda
Could not get the system external storage directory
Järjestelmän ulkoista tallennushakemistoa ei löytynyt
Could not write to:
Ei voi kirjoittaa kohteeseen:
Read-only media (SDCARD)
Vain luku -media (SDCARD)
No storage media (SDCARD)
Ei tallennusmediaa (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
HTTrack ei ehkä pysty lataamaan sivustoja ennen kuin tämä ongelma on korjattu
HTTrack: mirror '%s' stopped!
HTTrack: peilaus "%s" pysäytettiin!
Click on this notification to restart the interrupted mirror
Jatka keskeytynyttä peilausta napauttamalla tätä ilmoitusta
HTTrack: could not save profile for '%s'!
HTTrack: profiilia "%s" ei voitu tallentaa!
Proxy type:
Välityspalvelimen tyyppi:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC-osan enimm
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Aloita uusi WARC-osa, kun nykyinen ylittää tämän tavumäärän; jätä tyhjäksi tai 0, jolloin tiedosto pysyy yhtenä.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Palvelimen aliakset:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Muut samaa sivustoa palvelevat palvelinnimet yhdistettynä yhdeksi, yksi sääntö riviä kohden (esim. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Peilaa %s ja kaikki sen alla olevat palvelimet
Ignore %s and every host below it
Sivuuta %s ja kaikki sen alla olevat palvelimet

View File

@@ -1045,6 +1045,10 @@ Taille maximale d'un segment WARC :
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Commencer un nouveau segment WARC dès que le courant dépasse ce nombre d'octets ; laissez vide ou 0 pour un fichier unique.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Alias d'hôtes :
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Autres noms d'hôtes servant ce même site, ramenés à un seul, une règle par ligne (ex. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Copier %s et tous les hôtes en dessous
Ignore %s and every host below it
Ignorer %s et tous les hôtes en dessous

View File

@@ -946,6 +946,36 @@ Server terminated
Ακυρώθηκε από τον εξυπηρετητή
A fatal error has occurred during this mirror
Ένα καταστροφικό σφάλμα προκλήθηκε κατά την αντιγραφή αυτού του τόπου
View Documentation
Προβολή τεκμηρίωσης
Go To HTTrack Website
Μετάβαση στον ιστότοπο του HTTrack
Go To HTTrack Forum
Μετάβαση στο φόρουμ του HTTrack
View License
Προβολή άδειας χρήσης
Beware: you local browser might be unable to browse files with embedded filenames
Προσοχή: ο τοπικός σας browser ίσως δεν μπορεί να ανοίξει αρχεία με ενσωματωμένα ονόματα αρχείων
Recreated HTTrack internal cached resources
Οι εσωτερικοί προσωρινοί πόροι του HTTrack δημιουργήθηκαν ξανά
Could not create internal cached resources
Δεν ήταν δυνατή η δημιουργία των εσωτερικών προσωρινών πόρων
Could not get the system external storage directory
Δεν ήταν δυνατή η εύρεση του εξωτερικού χώρου αποθήκευσης του συστήματος
Could not write to:
Δεν ήταν δυνατή η εγγραφή στο:
Read-only media (SDCARD)
Μέσο μόνο για ανάγνωση (SDCARD)
No storage media (SDCARD)
Δεν υπάρχει μέσο αποθήκευσης (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Μέχρι να διορθωθεί αυτό το πρόβλημα, το HTTrack ίσως δεν μπορεί να κατεβάσει ιστότοπους
HTTrack: mirror '%s' stopped!
HTTrack: η αντιγραφή «%s» σταμάτησε!
Click on this notification to restart the interrupted mirror
Πατήστε αυτή την ειδοποίηση για να συνεχίσετε τη διακοπείσα αντιγραφή
HTTrack: could not save profile for '%s'!
HTTrack: δεν ήταν δυνατή η αποθήκευση του προφίλ «%s»!
Proxy type:
Τύπος proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC segment size limit:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Έναρξη νέου τμήματος WARC μόλις το τρέχον ξεπεράσει αυτόν τον αριθμό byte. Αφήστε κενό ή 0 για ένα ενιαίο αρχείο.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ψευδώνυμα host:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Άλλα ονόματα host που εξυπηρετούν την ίδια τοποθεσία, ενοποιημένα σε ένα, ένας κανόνας ανά γραμμή (π.χ. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Αντιγραφή του %s και κάθε host κάτω από αυτό
Ignore %s and every host below it
Αγνόηση του %s και κάθε host κάτω από αυτό

View File

@@ -946,6 +946,36 @@ Server terminated
Server disconnesso
A fatal error has occurred during this mirror
Si è verificato un errore fatale durante la copia
View Documentation
Visualizza la documentazione
Go To HTTrack Website
Vai al sito di HTTrack
Go To HTTrack Forum
Vai al forum di HTTrack
View License
Visualizza la licenza
Beware: you local browser might be unable to browse files with embedded filenames
Attenzione: il browser locale potrebbe non riuscire ad aprire i file con nomi di file incorporati
Recreated HTTrack internal cached resources
Risorse interne nella cache di HTTrack ricreate
Could not create internal cached resources
Impossibile creare le risorse interne nella cache
Could not get the system external storage directory
Impossibile individuare la memoria esterna di sistema
Could not write to:
Impossibile scrivere in:
Read-only media (SDCARD)
Supporto di sola lettura (SDCARD)
No storage media (SDCARD)
Nessun supporto di memorizzazione (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Finché il problema non è risolto, HTTrack potrebbe non riuscire a scaricare siti web
HTTrack: mirror '%s' stopped!
HTTrack: mirror "%s" interrotto!
Click on this notification to restart the interrupted mirror
Tocca questa notifica per riprendere il mirror interrotto
HTTrack: could not save profile for '%s'!
HTTrack: impossibile salvare il profilo di "%s"!
Proxy type:
Tipo di proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Dimensione massima del segmento WARC:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Inizia un nuovo segmento WARC quando quello corrente supera questo numero di byte; lascia vuoto o 0 per un solo file.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Alias host:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Altri nomi host che servono questo stesso sito, ricondotti a uno solo, una regola per riga (es. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Mirror di %s e di tutti gli host sottostanti
Ignore %s and every host below it
Ignora %s e tutti gli host sottostanti

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
ドキュメントの表示
Go To HTTrack Website
HTTrack のウェブサイトを開く
Go To HTTrack Forum
HTTrack のフォーラムを開く
View License
ライセンスの表示
Beware: you local browser might be unable to browse files with embedded filenames
注意: ローカルのブラウザではファイル名が埋め込まれたファイルを開けない場合があります
Recreated HTTrack internal cached resources
HTTrack の内部キャッシュを再作成しました
Could not create internal cached resources
内部キャッシュを作成できませんでした
Could not get the system external storage directory
システムの外部ストレージが見つかりませんでした
Could not write to:
次の場所に書き込めません:
Read-only media (SDCARD)
読み取り専用のメディア (SDCARD)
No storage media (SDCARD)
ストレージメディアがありません (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
この問題が解決するまで、HTTrack はサイトをダウンロードできない場合があります
HTTrack: mirror '%s' stopped!
HTTrack: ミラー「%s」を停止しました
Click on this notification to restart the interrupted mirror
中断したミラーを再開するには、この通知をタップしてください
HTTrack: could not save profile for '%s'!
HTTrack:「%s」のプロファイルを保存できませんでした
Proxy type:
プロキシの種類:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
現在のセグメントがこのバイト数を超えたら新しい WARC セグメントを開始します。空欄または 0 で単一ファイルのままにします。
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
ホストの別名:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
同じサイトを配信する他のホスト名を 1 つにまとめます。1 行につき 1 つの規則 (例: www2.example.com,m.example.com=example.com)。
Mirror %s and every host below it
%s とその配下のすべてのホストをコピー(ミラー)
Ignore %s and every host below it
%s とその配下のすべてのホストを無視

View File

@@ -946,6 +946,36 @@ Server terminated
Ñåðâåðîò å ïðåêèíàò
A fatal error has occurred during this mirror
Íàñòàíà ôàòàëíà ãðåøêà ïðè îâî¼ mirror
View Documentation
Ïðèêàæè ¼à äîêóìåíòàöè¼àòà
Go To HTTrack Website
Îäè íà âåá-ñòðàíèöàòà íà HTTrack
Go To HTTrack Forum
Îäè íà ôîðóìîò íà HTTrack
View License
Ïðèêàæè ¼à ëèöåíöàòà
Beware: you local browser might be unable to browse files with embedded filenames
Âíèìàíèå: âàøèîò ëîêàëåí ïðåëèñòóâà÷ ìîæåáè íåìà äà ìîæå äà îòâîðè äàòîòåêè ñî âãðàäåíè èìèœà íà äàòîòåêè
Recreated HTTrack internal cached resources
Âíàòðåøíèòå êåøèðàíè ðåñóðñè íà HTTrack ñå ñîçäàäåíè ïîâòîðíî
Could not create internal cached resources
Âíàòðåøíèòå êåøèðàíè ðåñóðñè íå ìîæåà äà ñå ñîçäàäàò
Could not get the system external storage directory
Ñèñòåìñêàòà íàäâîðåøíà ìåìîðè¼à íå ìîæåøå äà ñå íà¼äå
Could not write to:
Íå ìîæå äà ñå çàïèøå âî:
Read-only media (SDCARD)
Íîñà÷ ñàìî çà ÷èòàœå (SDCARD)
No storage media (SDCARD)
Íåìà íîñà÷ çà ñêëàäèðàœå (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Äîäåêà íå ñå ðåøè îâî¼ ïðîáëåì, HTTrack ìîæåáè íåìà äà ìîæå äà ïðåçåìà âåá-ñòðàíèöè
HTTrack: mirror '%s' stopped!
HTTrack: êîïè¼àòà „%s“ å çàïðåíà!
Click on this notification to restart the interrupted mirror
Äîïðåòå ãî îâà èçâåñòóâàœå çà äà ¼à ïðîäîëæèòå ïðåêèíàòàòà êîïè¼à
HTTrack: could not save profile for '%s'!
HTTrack: ïðîôèëîò çà „%s“ íå ìîæåøå äà ñå çà÷óâà!
Proxy type:
Òèï íà ïðîêñè:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC segment size limit:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Çàïî÷íè íîâ WARC ñåãìåíò êîãà òåêîâíèîò <20>å ãî íàäìèíå îâî¼ áðî¼ áà¼òè; îñòàâåòå ïðàçíî èëè 0 çà åäíà äàòîòåêà.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Àëè¼àñè íà õîñòîò:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Äðóãè èìèœà íà õîñòîâè øòî ãî îïñëóæóâààò èñòèîò ñà¼ò, ñïîåíè âî åäíî, ïî åäíî ïðàâèëî âî ðåä (ïð. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Êîïèð༠ãî %s è ñèòå õîñòîâè ïîä íåãî
Ignore %s and every host below it
Èãíîðèð༠ãî %s è ñèòå õîñòîâè ïîä íåãî

View File

@@ -946,6 +946,36 @@ Server terminated
A kiszolgáló befejezte a kapcsolatot
A fatal error has occurred during this mirror
Végzetes hiba történt a tükrözés közben
View Documentation
Dokumentáció megtekintése
Go To HTTrack Website
Ugrás a HTTrack webhelyére
Go To HTTrack Forum
Ugrás a HTTrack fórumára
View License
Licenc megtekintése
Beware: you local browser might be unable to browse files with embedded filenames
Figyelem: a helyi böngészõ nem biztos, hogy meg tudja nyitni a beágyazott fájlneveket tartalmazó fájlokat
Recreated HTTrack internal cached resources
A HTTrack belsõ gyorsítótárazott erõforrásai újra létrejöttek
Could not create internal cached resources
A belsõ gyorsítótárazott erõforrásokat nem sikerült létrehozni
Could not get the system external storage directory
A rendszer külsõ tárhelye nem található
Could not write to:
Nem lehet írni ide:
Read-only media (SDCARD)
Csak olvasható adathordozó (SDCARD)
No storage media (SDCARD)
Nincs adathordozó (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Amíg ez a hiba fennáll, a HTTrack esetleg nem tud webhelyeket letölteni
HTTrack: mirror '%s' stopped!
HTTrack: a(z) "%s" tükrözés leállt!
Click on this notification to restart the interrupted mirror
A megszakadt tükrözés folytatásához koppintson erre az értesítésre
HTTrack: could not save profile for '%s'!
HTTrack: a(z) "%s" profilját nem sikerült menteni!
Proxy type:
Proxy típusa:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC szegmens legnagyobb m
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Új WARC szegmens kezdése, amint az aktuális túllépi ezt a bájtszámot; hagyja üresen vagy adjon meg 0-t egyetlen fájlhoz.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Kiszolgálói álnevek:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ugyanazt a webhelyet kiszolgáló további kiszolgálónevek egybeolvasztva, soronként egy szabály (pl. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
%s és minden alatta lévõ kiszolgáló tükrözése
Ignore %s and every host below it
%s és minden alatta lévõ kiszolgáló kihagyása

View File

@@ -946,6 +946,36 @@ Server terminated
Server beeindigd
A fatal error has occurred during this mirror
Een fatale fout is opgetreden tijdens deze spiegeling
View Documentation
Documentatie bekijken
Go To HTTrack Website
Ga naar de HTTrack-website
Go To HTTrack Forum
Ga naar het HTTrack-forum
View License
Licentie bekijken
Beware: you local browser might be unable to browse files with embedded filenames
Let op: uw lokale browser kan bestanden met ingesloten bestandsnamen mogelijk niet openen
Recreated HTTrack internal cached resources
Interne buffer van HTTrack opnieuw aangemaakt
Could not create internal cached resources
Kan de interne buffer niet aanmaken
Could not get the system external storage directory
Kan de externe opslag van het systeem niet vinden
Could not write to:
Kan niet schrijven naar:
Read-only media (SDCARD)
Alleen-lezen medium (SDCARD)
No storage media (SDCARD)
Geen opslagmedium (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Zolang dit probleem niet is opgelost, kan HTTrack mogelijk geen websites downloaden
HTTrack: mirror '%s' stopped!
HTTrack: spiegeling "%s" gestopt!
Click on this notification to restart the interrupted mirror
Tik op deze melding om de onderbroken spiegeling te hervatten
HTTrack: could not save profile for '%s'!
HTTrack: kan het profiel voor "%s" niet opslaan!
Proxy type:
Proxytype:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Maximale grootte van een WARC-segment:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Een nieuw WARC-segment beginnen zodra het huidige dit aantal bytes overschrijdt; laat leeg of 0 voor één bestand.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Host-aliassen:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Andere hostnamen die dezelfde site aanbieden, samengevoegd tot één; één vermelding per regel (bijv. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
%s en alle onderliggende hosts spiegelen
Ignore %s and every host below it
%s en alle onderliggende hosts uitsluiten

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Vis dokumentasjonen
Go To HTTrack Website
Gå til HTTracks nettsted
Go To HTTrack Forum
Gå til HTTracks forum
View License
Vis lisensen
Beware: you local browser might be unable to browse files with embedded filenames
Merk: den lokale nettleseren din klarer kanskje ikke å åpne filer med innebygde filnavn
Recreated HTTrack internal cached resources
HTTracks interne bufrede ressurser er opprettet på nytt
Could not create internal cached resources
Klarte ikke å opprette de interne bufrede ressursene
Could not get the system external storage directory
Fant ikke systemets eksterne lagringsmappe
Could not write to:
Klarte ikke å skrive til:
Read-only media (SDCARD)
Skrivebeskyttet medium (SDCARD)
No storage media (SDCARD)
Intet lagringsmedium (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Så lenge dette problemet varer, klarer HTTrack kanskje ikke å laste ned nettsteder
HTTrack: mirror '%s' stopped!
HTTrack: kopien "%s" ble stoppet!
Click on this notification to restart the interrupted mirror
Trykk på dette varselet for å fortsette den avbrutte kopien
HTTrack: could not save profile for '%s'!
HTTrack: klarte ikke å lagre profilen for "%s"!
Proxy type:
Proxytype:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ St
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Begynn et nytt WARC-segment når det gjeldende overstiger dette antallet byte; la feltet stå tomt eller 0 for én fil.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Vertsaliaser:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Andre vertsnavn som betjener det samme nettstedet, slått sammen til ett, én regel per linje (f.eks. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Kopier %s og alle verter under den
Ignore %s and every host below it
Ignorer %s og alle verter under den

View File

@@ -946,6 +946,36 @@ Server terminated
Serwer zakonczyl prace
A fatal error has occurred during this mirror
Podczas tworzenia lustra wydarzyl sie fatalny blad.
View Documentation
Poka¿ dokumentacjê
Go To HTTrack Website
PrzejdŸ do witryny HTTrack
Go To HTTrack Forum
PrzejdŸ do forum HTTrack
View License
Poka¿ licencjê
Beware: you local browser might be unable to browse files with embedded filenames
Uwaga: lokalna przegl¹darka mo¿e nie otworzyæ plików z osadzonymi nazwami plików
Recreated HTTrack internal cached resources
Wewnêtrzne zasoby podrêczne HTTrack zosta³y utworzone ponownie
Could not create internal cached resources
Nie mo¿na utworzyæ wewnêtrznych zasobów podrêcznych
Could not get the system external storage directory
Nie mo¿na odnaleŸæ zewnêtrznej pamiêci systemu
Could not write to:
Nie mo¿na zapisaæ do:
Read-only media (SDCARD)
Noœnik tylko do odczytu (SDCARD)
No storage media (SDCARD)
Brak noœnika pamiêci (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Dopóki ten problem nie zostanie rozwi¹zany, HTTrack mo¿e nie pobieraæ witryn
HTTrack: mirror '%s' stopped!
HTTrack: lustro „%s” zosta³o zatrzymane!
Click on this notification to restart the interrupted mirror
Dotknij tego powiadomienia, aby wznowiæ przerwane tworzenie lustra
HTTrack: could not save profile for '%s'!
HTTrack: nie mo¿na zapisaæ profilu dla „%s”!
Proxy type:
Typ proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Maksymalny rozmiar segmentu WARC:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Rozpocznij nowy segment WARC, gdy bie¿¹cy przekroczy tê liczbê bajtów; pozostaw puste lub 0, aby zachowaæ jeden plik.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Aliasy hostów:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Inne nazwy hostów obs³uguj¹ce tê sam¹ witrynê, sprowadzone do jednej, jedna regu³a w wierszu (np. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Utwórz lustro %s i wszystkich hostów poni¿ej
Ignore %s and every host below it
Ignoruj %s i wszystkie hosty poni¿ej

View File

@@ -1045,6 +1045,10 @@ Tamanho m
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Iniciar um novo segmento WARC quando o atual passar deste número de bytes; deixe em branco ou 0 para um único arquivo.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Apelidos de host:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Outros nomes de host que servem este mesmo site, unificados em um só, uma regra por linha (ex.: www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Copiar %s e todos os hosts abaixo dele
Ignore %s and every host below it
Ignorar %s e todos os hosts abaixo dele

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Ver a documentação
Go To HTTrack Website
Ir para o sítio do HTTrack
Go To HTTrack Forum
Ir para o fórum do HTTrack
View License
Ver a licença
Beware: you local browser might be unable to browse files with embedded filenames
Atenção: o seu navegador local pode não conseguir abrir ficheiros com nomes de ficheiro incorporados
Recreated HTTrack internal cached resources
Recursos internos em cache do HTTrack recriados
Could not create internal cached resources
Não foi possível criar os recursos internos em cache
Could not get the system external storage directory
Não foi possível encontrar o armazenamento externo do sistema
Could not write to:
Não foi possível escrever em:
Read-only media (SDCARD)
Suporte só de leitura (SDCARD)
No storage media (SDCARD)
Sem suporte de armazenamento (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Enquanto este problema não for resolvido, o HTTrack pode não conseguir transferir sítios
HTTrack: mirror '%s' stopped!
HTTrack: a cópia "%s" foi interrompida!
Click on this notification to restart the interrupted mirror
Toque nesta notificação para retomar a cópia interrompida
HTTrack: could not save profile for '%s'!
HTTrack: não foi possível guardar o perfil de "%s"!
Proxy type:
Tipo de proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Tamanho m
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Iniciar um novo segmento WARC quando o atual ultrapassar este número de bytes; deixe em branco ou 0 para um único ficheiro.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Nomes alternativos do anfitrião:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Outros nomes de anfitrião que servem este mesmo sítio, reunidos num só, uma regra por linha (ex.: www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Copiar %s e todos os anfitriões abaixo
Ignore %s and every host below it
Ignorar %s e todos os anfitriões abaixo

View File

@@ -946,6 +946,36 @@ Server terminated
Server terminat
A fatal error has occurred during this mirror
A survenit o eroare fatală în timpul acestei clonări.
View Documentation
Vezi documentaţia
Go To HTTrack Website
Mergi la site-ul HTTrack
Go To HTTrack Forum
Mergi la forumul HTTrack
View License
Vezi licenţa
Beware: you local browser might be unable to browse files with embedded filenames
Atenţie: navigatorul local ar putea să nu poată deschide fişiere cu nume de fişier încorporate
Recreated HTTrack internal cached resources
Resursele interne din cache ale HTTrack au fost recreate
Could not create internal cached resources
Resursele interne din cache nu au putut fi create
Could not get the system external storage directory
Directorul de stocare externă al sistemului nu a putut fi găsit
Could not write to:
Nu se poate scrie în:
Read-only media (SDCARD)
Suport numai pentru citire (SDCARD)
No storage media (SDCARD)
Niciun suport de stocare (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Cât timp această problemă persistă, HTTrack ar putea să nu poată descărca site-uri
HTTrack: mirror '%s' stopped!
HTTrack: clona "%s" a fost oprită!
Click on this notification to restart the interrupted mirror
Atinge această notificare pentru a relua clona întreruptă
HTTrack: could not save profile for '%s'!
HTTrack: profilul pentru "%s" nu a putut fi salvat!
Proxy type:
Tip proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Dimensiunea maxima a unui segment WARC:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Incepe un nou segment WARC cand cel curent depaseste acest numar de octeti; lasati gol sau 0 pentru un singur fisier.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Alias-uri gazdă:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Alte nume de gazdă care servesc acelaşi site, reunite într-unul singur, o regulă pe linie (ex. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Clonează %s şi toate gazdele de sub el
Ignore %s and every host below it
Ignoră %s şi toate gazdele de sub el

View File

@@ -946,6 +946,36 @@ Server terminated
Server terminated
A fatal error has occurred during this mirror
Âî âðåìÿ òåêóùåé çàêà÷êè ïðîèçîøëà ôàòàëüíàÿ îøèáêà
View Documentation
Ïîêàçàòü äîêóìåíòàöèþ
Go To HTTrack Website
Ïåðåéòè íà ñàéò HTTrack
Go To HTTrack Forum
Ïåðåéòè íà ôîðóì HTTrack
View License
Ïîêàçàòü ëèöåíçèþ
Beware: you local browser might be unable to browse files with embedded filenames
Âíèìàíèå: ëîêàëüíûé áðàóçåð ìîæåò íå îòêðûòü ôàéëû ñî âñòðîåííûìè èìåíàìè ôàéëîâ
Recreated HTTrack internal cached resources
Âíóòðåííèå êýøèðîâàííûå ðåñóðñû HTTrack ñîçäàíû çàíîâî
Could not create internal cached resources
Íå óäàëîñü ñîçäàòü âíóòðåííèå êýøèðîâàííûå ðåñóðñû
Could not get the system external storage directory
Íå óäàëîñü íàéòè âíåøíåå õðàíèëèùå ñèñòåìû
Could not write to:
Íå óäàëîñü çàïèñàòü â:
Read-only media (SDCARD)
Íîñèòåëü òîëüêî äëÿ ÷òåíèÿ (SDCARD)
No storage media (SDCARD)
Íåò íîñèòåëÿ äëÿ õðàíåíèÿ (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Ïîêà ýòà ïðîáëåìà íå óñòðàíåíà, HTTrack ìîæåò íå çàãðóæàòü ñàéòû
HTTrack: mirror '%s' stopped!
HTTrack: çåðêàëî «%s» îñòàíîâëåíî!
Click on this notification to restart the interrupted mirror
Íàæìèòå íà ýòî óâåäîìëåíèå, ÷òîáû ïðîäîëæèòü ïðåðâàííîå ñîçäàíèå çåðêàëà
HTTrack: could not save profile for '%s'!
HTTrack: íå óäàëîñü ñîõðàíèòü ïðîôèëü äëÿ «%s»!
Proxy type:
Òèï ïðîêñè:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC segment size limit:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Íà÷èíàòü íîâûé ñåãìåíò WARC, êîãäà òåêóùèé ïðåâûñèò óêàçàííîå ÷èñëî áàéò; îñòàâüòå ïóñòûì èëè 0 äëÿ îäíîãî ôàéëà.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ïñåâäîíèìû õîñòà:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Äðóãèå èìåíà õîñòîâ, îáñëóæèâàþùèå ýòîò æå ñàéò, ñâåä¸ííûå ê îäíîìó, ïî îäíîìó ïðàâèëó â ñòðîêå (íàïð. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Ñäåëàòü çåðêàëî %s è âñåõ õîñòîâ ïîä íèì
Ignore %s and every host below it
Èãíîðèðîâàòü %s è âñå õîñòû ïîä íèì

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Zobrazi<EFBFBD> dokumentáciu
Go To HTTrack Website
Prejs<EFBFBD> na webovú stránku HTTrack
Go To HTTrack Forum
Prejs<EFBFBD> na fórum HTTrack
View License
Zobrazi<EFBFBD> licenciu
Beware: you local browser might be unable to browse files with embedded filenames
Pozor: váš miestny prehliadaè nemusí otvori<72> súbory s vloženými názvami súborov
Recreated HTTrack internal cached resources
Interné zdroje vo vyrovnávacej pamäti HTTrack boli znovu vytvorené
Could not create internal cached resources
Interné zdroje vo vyrovnávacej pamäti sa nepodarilo vytvori<72>
Could not get the system external storage directory
Externé úložisko systému sa nepodarilo nájs<6A>
Could not write to:
Nedá sa zapisova<76> do:
Read-only media (SDCARD)
Médium len na èítanie (SDCARD)
No storage media (SDCARD)
Žiadne úložné médium (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Kým sa tento problém nevyrieši, HTTrack nemusí s<>ahova<76> webové stránky
HTTrack: mirror '%s' stopped!
HTTrack: zrkadlenie „%s“ bolo zastavené!
Click on this notification to restart the interrupted mirror
<EFBFBD>uknutím na toto oznámenie obnovíte prerušené zrkadlenie
HTTrack: could not save profile for '%s'!
HTTrack: profil pre „%s“ sa nepodarilo uloži<C5BE>!
Proxy type:
Typ proxy:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Najv
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Zaèa<EFBFBD> nový segment WARC, keï aktuálny prekroèí tento poèet bajtov; ponechajte prázdne alebo 0 pre jeden súbor.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Aliasy hostite¾a:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ïalšie názvy hostite¾ov poskytujúce tú istú lokalitu, zlúèené do jedného, jedno pravidlo na riadok (napr. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Zrkadli<EFBFBD> %s a všetkých hostite¾ov pod ním
Ignore %s and every host below it
Ignorova<EFBFBD> %s a všetkých hostite¾ov pod ním

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Prikaži dokumentacijo
Go To HTTrack Website
Pojdi na spletno mesto HTTrack
Go To HTTrack Forum
Pojdi na forum HTTrack
View License
Prikaži licenco
Beware: you local browser might be unable to browse files with embedded filenames
Pozor: lokalni brskalnik morda ne bo mogel odpreti datotek z vdelanimi imeni datotek
Recreated HTTrack internal cached resources
Notranji predpomnjeni viri HTTracka so bili znova ustvarjeni
Could not create internal cached resources
Notranjih predpomnjenih virov ni bilo mogoèe ustvariti
Could not get the system external storage directory
Zunanje shrambe sistema ni bilo mogoèe najti
Could not write to:
Ni mogoèe pisati v:
Read-only media (SDCARD)
Nosilec samo za branje (SDCARD)
No storage media (SDCARD)
Ni nosilca za shranjevanje (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Dokler ta težava ni odpravljena, HTTrack morda ne bo mogel prenašati spletnih mest
HTTrack: mirror '%s' stopped!
HTTrack: zrcaljenje „%s“ je bilo ustavljeno!
Click on this notification to restart the interrupted mirror
Tapnite to obvestilo za nadaljevanje prekinjenega zrcaljenja
HTTrack: could not save profile for '%s'!
HTTrack: profila za „%s“ ni bilo mogoèe shraniti!
Proxy type:
Vrsta proxyja:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ Najvecja velikost segmenta WARC:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Zacni nov segment WARC, ko trenutni preseze to stevilo bajtov; pustite prazno ali 0 za eno datoteko.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Vzdevki gostitelja:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Druga imena gostiteljev, ki strežejo isto stran, združena v eno, eno pravilo na vrstico (npr. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Zrcali %s in vse gostitelje pod njim
Ignore %s and every host below it
Prezri %s in vse gostitelje pod njim

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Visa dokumentationen
Go To HTTrack Website
Gå till HTTracks webbplats
Go To HTTrack Forum
Gå till HTTracks forum
View License
Visa licensen
Beware: you local browser might be unable to browse files with embedded filenames
Obs: din lokala webbläsare kan kanske inte öppna filer med inbäddade filnamn
Recreated HTTrack internal cached resources
HTTracks interna cachade resurser har skapats om
Could not create internal cached resources
Det gick inte att skapa de interna cachade resurserna
Could not get the system external storage directory
Det gick inte att hitta systemets externa lagring
Could not write to:
Det gick inte att skriva till:
Read-only media (SDCARD)
Skrivskyddat medium (SDCARD)
No storage media (SDCARD)
Inget lagringsmedium (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Så länge problemet kvarstår kan HTTrack kanske inte hämta webbplatser
HTTrack: mirror '%s' stopped!
HTTrack: kopian "%s" stoppades!
Click on this notification to restart the interrupted mirror
Tryck på denna avisering för att återuppta den avbrutna kopian
HTTrack: could not save profile for '%s'!
HTTrack: det gick inte att spara profilen för "%s"!
Proxy type:
Proxytyp:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ St
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Börja ett nytt WARC-segment när det aktuella passerar detta antal byte; lämna tomt eller 0 för en enda fil.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Värdaliaser:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Andra värdnamn som betjänar samma webbplats, sammanslagna till ett, en regel per rad (t.ex. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Kopiera %s och alla värdar under den
Ignore %s and every host below it
Ignorera %s och alla värdar under den

View File

@@ -946,6 +946,36 @@ Server terminated
Sunucu sonlandýrýldý
A fatal error has occurred during this mirror
Bu yansýlama iþlemi sýrasýnda ölümcül bir hata oluþtu
View Documentation
Belgeleri görüntüle
Go To HTTrack Website
HTTrack web sitesine git
Go To HTTrack Forum
HTTrack forumuna git
View License
Lisansý görüntüle
Beware: you local browser might be unable to browse files with embedded filenames
Dikkat: yerel tarayýcýnýz gömülü dosya adlarý içeren dosyalarý açamayabilir
Recreated HTTrack internal cached resources
HTTrack iç önbellek kaynaklarý yeniden oluþturuldu
Could not create internal cached resources
Ýç önbellek kaynaklarý oluþturulamadý
Could not get the system external storage directory
Sistemin dýþ depolama dizini bulunamadý
Could not write to:
Þuraya yazýlamadý:
Read-only media (SDCARD)
Salt okunur ortam (SDCARD)
No storage media (SDCARD)
Depolama ortamý yok (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Bu sorun giderilene kadar HTTrack web sitelerini indiremeyebilir
HTTrack: mirror '%s' stopped!
HTTrack: "%s" yansýsý durduruldu!
Click on this notification to restart the interrupted mirror
Yarým kalan yansýyý sürdürmek için bu bildirime dokunun
HTTrack: could not save profile for '%s'!
HTTrack: "%s" için profil kaydedilemedi!
Proxy type:
Proxy türü:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC par
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Geçerli parça bu bayt sayýsýný aþtýðýnda yeni bir WARC parçasý baþlat; tek dosya için boþ býrakýn veya 0 girin.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Sunucu takma adlarý:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ayný siteyi sunan ve tek bir ada indirgenen diðer sunucu adlarý; her satýrda bir kural (örn. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
%s ve altýndaki tüm sunucularý yansýla
Ignore %s and every host below it
%s ve altýndaki tüm sunucularý yoksay

View File

@@ -946,6 +946,36 @@ Server terminated
A fatal error has occurred during this mirror
View Documentation
Ïîêàçàòè äîêóìåíòàö³þ
Go To HTTrack Website
Ïåðåéòè íà ñàéò HTTrack
Go To HTTrack Forum
Ïåðåéòè íà ôîðóì HTTrack
View License
Ïîêàçàòè ë³öåíç³þ
Beware: you local browser might be unable to browse files with embedded filenames
Óâàãà: ëîêàëüíèé áðàóçåð ìîæå íå â³äêðèòè ôàéëè ç âáóäîâàíèìè ³ìåíàìè ôàéë³â
Recreated HTTrack internal cached resources
Âíóòð³øí³ êåøîâàí³ ðåñóðñè HTTrack ñòâîðåíî çàíîâî
Could not create internal cached resources
Íå âäàëîñÿ ñòâîðèòè âíóòð³øí³ êåøîâàí³ ðåñóðñè
Could not get the system external storage directory
Íå âäàëîñÿ çíàéòè çîâí³øíº ñõîâèùå ñèñòåìè
Could not write to:
Íå âäàëîñÿ çàïèñàòè â:
Read-only media (SDCARD)
Íîñ³é ëèøå äëÿ ÷èòàííÿ (SDCARD)
No storage media (SDCARD)
Íåìຠíîñ³ÿ äëÿ çáåð³ãàííÿ (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Äîêè öþ ïðîáëåìó íå óñóíóòî, HTTrack ìîæå íå çàâàíòàæóâàòè ñàéòè
HTTrack: mirror '%s' stopped!
HTTrack: äçåðêàëî «%s» çóïèíåíî!
Click on this notification to restart the interrupted mirror
Íàòèñí³òü íà öå ñïîâ³ùåííÿ, ùîá ïðîäîâæèòè ïåðåðâàíå äçåðêàëî
HTTrack: could not save profile for '%s'!
HTTrack: íå âäàëîñÿ çáåðåãòè ïðîô³ëü äëÿ «%s»!
Proxy type:
Òèï ïðîêñ³:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC segment size limit:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Ïî÷èíàòè íîâèé ñåãìåíò WARC, êîëè ïîòî÷íèé ïåðåâèùèòü öþ ê³ëüê³ñòü áàéò³â; çàëèøòå ïîðîæí³ì àáî 0 äëÿ îäíîãî ôàéëó.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Ïñåâäîí³ìè õîñòà:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
²íø³ ³ìåíà õîñò³â, ùî îáñëóãîâóþòü öåé ñàìèé ñàéò, çâåäåí³ äî îäíîãî, ïî îäíîìó ïðàâèëó â ðÿäêó (íàïð. www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
Äçåðêàëþâàòè %s ³ âñ³ õîñòè ï³ä íèì
Ignore %s and every host below it
²ãíîðóâàòè %s ³ âñ³ õîñòè ï³ä íèì

View File

@@ -946,6 +946,36 @@ Server terminated
Server terminated
A fatal error has occurred during this mirror
Joriy kochirish vaqtida jiddiy xatolik yuz berdi
View Documentation
Hujjatlarni ko'rish
Go To HTTrack Website
HTTrack veb-saytiga o'tish
Go To HTTrack Forum
HTTrack forumiga o'tish
View License
Litsenziyani ko'rish
Beware: you local browser might be unable to browse files with embedded filenames
Diqqat: mahalliy brauzeringiz nomi ichiga joylangan fayllarni ocholmasligi mumkin
Recreated HTTrack internal cached resources
HTTrack ichki kesh resurslari qayta yaratildi
Could not create internal cached resources
Ichki kesh resurslarini yaratib bo'lmadi
Could not get the system external storage directory
Tizimning tashqi xotira jildini topib bo'lmadi
Could not write to:
Quyidagiga yozib bo'lmadi:
Read-only media (SDCARD)
Faqat o'qish uchun tashuvchi (SDCARD)
No storage media (SDCARD)
Xotira tashuvchisi yo'q (SDCARD)
HTTrack may not be able to download websites until this problem is fixed
Bu muammo bartaraf etilmaguncha HTTrack veb-saytlarni yuklab ololmasligi mumkin
HTTrack: mirror '%s' stopped!
HTTrack: "%s" nusxasi to'xtatildi!
Click on this notification to restart the interrupted mirror
Uzilgan nusxani davom ettirish uchun bu bildirishnomani bosing
HTTrack: could not save profile for '%s'!
HTTrack: "%s" uchun profilni saqlab bo'lmadi!
Proxy type:
Proksi turi:
Proxy protocol. HTTP: standard proxy. HTTP (CONNECT tunnel): sends every request through a CONNECT tunnel, for CONNECT-only proxies like Tor's HTTPTunnelPort. SOCKS5: default port 1080.
@@ -1015,6 +1045,10 @@ WARC segmentining eng katta hajmi:
Start a new WARC segment once the current one grows past this many bytes; leave blank or 0 to keep a single file.
Joriy segment shu baytlar sonidan oshganda yangi WARC segmentini boshlash; bitta fayl uchun bosh qoldiring yoki 0 kiriting.
Host aliases:
Host aliases:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Xost taxalluslari:
Other hostnames serving this same site, folded onto one, one rule per line (e.g. www2.example.com,m.example.com=example.com).
Shu saytga xizmat qiluvchi va bittaga birlashtiriladigan boshqa xost nomlari; har satrda bitta qoida (masalan www2.example.com,m.example.com=example.com).
Mirror %s and every host below it
%s va uning ostidagi barcha xostlardan nusxa olish
Ignore %s and every host below it
%s va uning ostidagi barcha xostlarni e'tiborsiz qoldirish

View File

@@ -33,6 +33,9 @@ Please visit our Website: http://www.httrack.com
#ifndef HTTRACK_DEFTMPL
#define HTTRACK_DEFTMPL
/* Generated data: clang-format rewrites the whole table on any edit. */
/* clang-format off */
/* Index for each project */
/*
regen:
@@ -168,7 +171,7 @@ regen:
" <BR>"LF\
" <BR>"LF\
" <H6 ALIGN=\"RIGHT\">"LF\
" <I>Mirror and index made by HTTrack Website Copier [XR&amp;CO'2014]</I>"LF\
" <I>Mirror and index made by HTTrack Website Copier [XR&amp;CO]</I>"LF\
" </H6>"LF\
" %s"LF\
" <!-- Thanks for using HTTrack Website Copier! -->"LF\
@@ -186,7 +189,7 @@ regen:
""LF\
"<table width=\"76%%\" border=\"0\" align=\"center\" valign=\"bottom\" cellspacing=\"0\" cellpadding=\"0\">"LF\
" <tr>"LF\
" <td id=\"footer\"><small>&copy; 2014 Xavier Roche & other contributors - Web Design: Kauler Leto.</small></td>"LF\
" <td id=\"footer\"><small>&copy; 1998 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>"LF\
" </tr>"LF\
"</table>"LF\
""LF\
@@ -317,7 +320,7 @@ regen:
" </TABLE>"LF\
" <BR>"LF\
" <H6 ALIGN=\"RIGHT\">"LF\
" <I>Mirror and index made by HTTrack Website Copier [XR&CO'2014]</I>"LF\
" <I>Mirror and index made by HTTrack Website Copier [XR&CO]</I>"LF\
" </H6>"LF\
" %s"LF\
" <!-- Thanks for using HTTrack Website Copier! -->"LF\
@@ -335,7 +338,7 @@ regen:
""LF\
"<table width=\"76%%\" border=\"0\" align=\"center\" valign=\"bottom\" cellspacing=\"0\" cellpadding=\"0\">"LF\
" <tr>"LF\
" <td id=\"footer\"><small>&copy; 2014 Xavier Roche & other contributors - Web Design: Kauler Leto.</small></td>"LF\
" <td id=\"footer\"><small>&copy; 1998 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>"LF\
" </tr>"LF\
"</table>"LF\
""LF\
@@ -475,7 +478,7 @@ regen:
""LF\
"<table width=\"76%%\" height=\"100%%\" border=\"0\" align=\"center\" valign=\"bottom\" cellspacing=\"0\" cellpadding=\"0\">"LF\
" <tr>"LF\
" <td id=\"footer\"><small>&copy; 2014 Xavier Roche & other contributors - Web Design: Kauler Leto.</small></td>"LF\
" <td id=\"footer\"><small>&copy; 1998 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>"LF\
" </tr>"LF\
"</table>"LF\
""LF\
@@ -785,4 +788,6 @@ regen:
"\x19\x0\xaf\x61\x13\x48\x10\xdb\xc0\x83\x4\xb\x16\x44\x88\x50\xe1\x41\x86\x9\x21\x1a\x74\x78\x2d\x20\x0\x3b\xff"
#define HTS_DATA_FADE_GIF_LEN 828
/* clang-format on */
#endif

View File

@@ -2022,8 +2022,10 @@ int httpmirror(char *url1, httrackp * opt) {
if ((HTS_STAT.stat_files <= 0)
&& (HTS_STAT.HTS_TOTAL_RECV < 32768) /* should be fine */
) {
hts_log_print(opt, LOG_NOTICE,
"No data seems to have been transferred during this session! : restoring previous one!");
/* not a notice: the session is rolled back and the WARC aborted */
hts_log_print(opt, LOG_WARNING,
"No data seems to have been transferred during this session! "
": restoring previous one!");
/* this run replaces nothing, so its archive must not be committed */
warc_abort_opt(opt);
opt->state.exit_xh = 2; /* interrupted (no connection detected) */
@@ -3413,17 +3415,17 @@ int check_sockdata(T_SOC s) {
// Attente de touche
int ask_continue(httrackp * opt) {
const char *s;
const char *s = RUN_CALLBACK1(opt, query2, opt->state.HTbuff);
int go = 1;
s = RUN_CALLBACK1(opt, query2, opt->state.HTbuff);
if (s) {
if (strnotempty(s)) {
if ((strfield2(s, "N")) || (strfield2(s, "NO")) || (strfield2(s, "NON")))
return 0;
}
return 1;
}
return 1;
if (s != NULL && strnotempty(s) &&
((strfield2(s, "N")) || (strfield2(s, "NO")) || (strfield2(s, "NON"))))
go = 0;
if (HAS_CALLBACK(opt, query2)) /* nobody was asked otherwise */
hts_log_print(opt, LOG_NOTICE, "(wizard) answer '%s' to \"%.*s\": %s",
s != NULL ? s : "", (int) strcspn(opt->state.HTbuff, "\r\n"),
opt->state.HTbuff, go ? "continue" : "abort");
return go;
}
// nombre de digits dans un nombre

View File

@@ -1565,7 +1565,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
return -1;
} else {
na++;
if (strlen(argv[na]) >= 254) {
if (strlen(argv[na]) >= HTS_FILELIST_MAXSIZE) {
HTS_PANIC_PRINTF("File list string too long");
htsmain_free();
return -1;
@@ -1582,7 +1582,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
return -1;
} else {
na++;
if (strlen(argv[na]) >= 254) {
if (strlen(argv[na]) >= HTS_BINDHOST_MAXSIZE) {
HTS_PANIC_PRINTF("Hostname string too long");
htsmain_free();
return -1;
@@ -1683,7 +1683,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
return -1;
} else {
na++;
if (strlen(argv[na]) >= 62) {
if (strlen(argv[na]) >= HTS_LANGISO_MAXSIZE) {
HTS_PANIC_PRINTF("Lang list string too long");
htsmain_free();
return -1;
@@ -1740,7 +1740,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
return -1;
} else {
na++;
if (strlen(argv[na]) >= 254) {
if (strlen(argv[na]) >= HTS_FOOTER_MAXSIZE) {
HTS_PANIC_PRINTF("Footer string too long");
htsmain_free();
return -1;
@@ -1841,7 +1841,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
return -1;
} else {
na++;
if (strlen(argv[na]) >= 254) {
if (strlen(argv[na]) >= HTS_REFERER_MAXSIZE) {
HTS_PANIC_PRINTF("Referer URL too long");
htsmain_free();
return -1;
@@ -1858,7 +1858,7 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
return -1;
} else {
na++;
if (strlen(argv[na]) >= 254) {
if (strlen(argv[na]) >= HTS_FROMEMAIL_MAXSIZE) {
HTS_PANIC_PRINTF("From email too long");
htsmain_free();
return -1;
@@ -2734,9 +2734,10 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
"wb");
if (fp) {
for(i = 0 + 1; i < argc; i++) {
if (((strchr(argv[i], ' ') != NULL)
|| (strchr(argv[i], '"') != NULL)
|| (strchr(argv[i], '\\') != NULL)) && (argv[i][0] != '"')) {
/* argv[] is already unquoted here, so a leading quote is data */
if ((strchr(argv[i], ' ') != NULL) ||
(strchr(argv[i], '"') != NULL) ||
(strchr(argv[i], '\\') != NULL)) {
size_t j;
fprintf(fp, "\"");
@@ -2749,9 +2750,9 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
fprintf(fp, "%c", argv[i][j]);
}
fprintf(fp, "\"");
} else if (strnotempty(argv[i]) == 0) { // ""
} else if (strnotempty(argv[i]) == 0) { // ""
fprintf(fp, "\"\"");
} else { // non critique
} else { // nothing to escape
fprintf(fp, "%s", argv[i]);
}
if (i < argc - 1)

View File

@@ -146,6 +146,13 @@ typedef const char *(*t_hts_htmlcheck_query3)(t_hts_callbackarg *carg,
httrackp *opt,
const char *question);
/* query3 answers HTS_WIZARD_SCOPE_INCLUDE+k and HTS_WIZARD_SCOPE_EXCLUDE+k take
or drop the k-th host scope hts_wizard_host_scope() enumerates. The stride
outruns any hostname's label count, so it cannot collide with the plain
single-digit answers. */
#define HTS_WIZARD_SCOPE_INCLUDE 1000
#define HTS_WIZARD_SCOPE_EXCLUDE 2000
/* Per-tick progress hook: 'back' is the transfer slot array of 'back_max'
entries, back_index the active one; lien_tot/lien_ntot and stats report
queue size and running totals, stat_time the elapsed time. */

View File

@@ -219,11 +219,23 @@ Please visit our Website: http://www.httrack.com
the longest registered MIME type, the Office OOXML ones reaching 73 chars */
#define HTS_MIMETYPE_SIZE 128
/* Caps on single option arguments, in bytes, exclusive like HTS_CDLMAXSIZE.
They bound the value, not a buffer: these option fields are dynamic Strings.
Named so the front ends can check against them instead of copying them. */
#define HTS_FOOTER_MAXSIZE 254 /* -%F */
#define HTS_LANGISO_MAXSIZE 62 /* -%l */
#define HTS_REFERER_MAXSIZE 254 /* -%R */
#define HTS_FILELIST_MAXSIZE 254 /* -%L */
#define HTS_BINDHOST_MAXSIZE 254 /* -%b */
#define HTS_FROMEMAIL_MAXSIZE 254 /* -%E */
/* Copyright (C) 1998 Xavier Roche and other contributors */
#define HTTRACK_AFF_AUTHORS "[XR&CO'2014]"
#define HTTRACK_AFF_AUTHORS "[XR&CO]"
/* Named fields (hts_footer_format); a "%s" anywhere would switch the template
back to the legacy positional model, a user's own additions included. */
#define HTS_DEFAULT_FOOTER \
"<!-- Mirrored from %s%s by HTTrack Website Copier/" HTTRACK_AFF_VERSION \
" " HTTRACK_AFF_AUTHORS ", %s -->"
"<!-- Mirrored from {url} by HTTrack Website Copier/" HTTRACK_AFF_VERSION \
" " HTTRACK_AFF_AUTHORS ", {date} -->"
/* Honest crawler User-Agent; no fake OS/browser to go stale. */
#define HTS_DEFAULT_USER_AGENT \
"Mozilla/5.0 (compatible; HTTrack/" HTTRACK_AFF_VERSION \

View File

@@ -4136,6 +4136,19 @@ hts_boolean hts_is_control_free(const char *str) {
return hts_is_control_free_sized(str, strlen(str));
}
hts_boolean hts_host_is_ipv4(const char *host, size_t len) {
size_t i;
int dots = 0;
for (i = 0; i < len; i++) {
if (host[i] == '.')
dots++;
else if (host[i] < '0' || host[i] > '9')
return HTS_FALSE;
}
return dots > 0 ? HTS_TRUE : HTS_FALSE;
}
hts_boolean hts_proxy_is_socks(const char *name) {
if (name == NULL)
return HTS_FALSE;
@@ -6218,9 +6231,9 @@ HTSEXT_API void hts_log_vprint(httrackp * opt, int type, const char *format, va_
s_type = "debug";
break;
case LOG_INFO:
case LOG_NOTICE: /* not a warning: counted in the footer's messages */
s_type = "info";
break;
case LOG_NOTICE:
case LOG_WARNING:
s_type = "warning";
break;

View File

@@ -255,6 +255,10 @@ hts_boolean hts_is_control_free_sized(const char *str, size_t len);
/* Same over a NUL-terminated string. */
hts_boolean hts_is_control_free(const char *str);
/* TRUE if host[0..len) is an IPv4 literal: digits and dots, at least one dot.
Such a host has no domain structure to reverse or to widen into. */
hts_boolean hts_host_is_ipv4(const char *host, size_t len);
/* TRUE if this -P proxy name (which keeps its scheme) is a SOCKS5 proxy. */
hts_boolean hts_proxy_is_socks(const char *name);
@@ -424,16 +428,15 @@ void *hts_get_callback(t_hts_htmlcheck_callbacks * callbacks,
#define CBSTRUCT(OPT) ((t_hts_htmlcheck_callbacks*) ((OPT)->callbacks_fun))
#define GET_USERCALLBACK(OPT, NAME) ( CBSTRUCT(OPT)-> NAME .fun )
#define GET_USERARG(OPT, NAME) ( CBSTRUCT(OPT)-> NAME .carg )
#define GET_USERDEF(OPT, NAME) ( \
(CBSTRUCT(OPT) != NULL && CBSTRUCT(OPT)-> NAME .fun != NULL) \
? ( GET_USERARG(OPT, NAME) ) \
: ( default_callbacks. NAME .carg ) \
)
#define GET_CALLBACK(OPT, NAME) ( \
(CBSTRUCT(OPT) != NULL && CBSTRUCT(OPT)-> NAME .fun != NULL) \
? ( GET_USERCALLBACK(OPT, NAME ) ) \
: ( default_callbacks. NAME .fun ) \
)
/* True when a front end registered its own NAME callback */
#define HAS_CALLBACK(OPT, NAME) \
(CBSTRUCT(OPT) != NULL && CBSTRUCT(OPT)->NAME.fun != NULL)
#define GET_USERDEF(OPT, NAME) \
(HAS_CALLBACK(OPT, NAME) ? (GET_USERARG(OPT, NAME)) \
: (default_callbacks.NAME.carg))
#define GET_CALLBACK(OPT, NAME) \
(HAS_CALLBACK(OPT, NAME) ? (GET_USERCALLBACK(OPT, NAME)) \
: (default_callbacks.NAME.fun))
/* Predefined macros */
#define RUN_CALLBACK_NOARG(OPT, NAME) GET_CALLBACK(OPT, NAME)(GET_USERARG(OPT, NAME))
@@ -447,19 +450,6 @@ void *hts_get_callback(t_hts_htmlcheck_callbacks * callbacks,
#define RUN_CALLBACK7(OPT, NAME, ARG1, ARG2, ARG3, ARG4, ARG5, ARG6, ARG7) GET_CALLBACK(OPT, NAME)(GET_USERARG(OPT, NAME), OPT, ARG1, ARG2, ARG3, ARG4, ARG5, ARG6, ARG7)
#define RUN_CALLBACK8(OPT, NAME, ARG1, ARG2, ARG3, ARG4, ARG5, ARG6, ARG7, ARG8) GET_CALLBACK(OPT, NAME)(GET_USERARG(OPT, NAME), OPT, ARG1, ARG2, ARG3, ARG4, ARG5, ARG6, ARG7, ARG8)
/*
#define GET_CALLBACK(OPT, NAME, ARG) ( \
( \
( ARG ) = GET_USERDEF(OPT, NAME), \
( \
(CBSTRUCT(OPT) != NULL && CBSTRUCT(OPT)-> NAME .fun != NULL) \
? ( GET_USERCALLBACK(OPT, NAME ) ) \
: ( default_callbacks. NAME .fun ) \
) \
) \
)
*/
/* UTF-8 aware FILE API */
#ifndef HTS_DEF_FILEAPI
#ifdef _WIN32

View File

@@ -855,33 +855,33 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
// footer sits inside an HTML comment, so a value holding
// "-->" would otherwise close it and inject markup (#165).
// status/size are formatted integers and need no escaping.
const hts_footer_field fields[] = {
{"addr", safe_adr},
{"path", safe_fil},
{"url", safe_url},
{"date", gmttime},
{"lastmodified",
html_inline_safe(r->lastmodified, safe_lastmod,
sizeof(safe_lastmod))},
{"version", HTTRACK_VERSIONID},
{"mime", html_inline_safe(r->contenttype, safe_ctype,
sizeof(safe_ctype))},
{"charset", html_inline_safe(r->charset, safe_charset,
sizeof(safe_charset))},
{"status", status_str},
{"size", size_str},
};
// Zeroed first: a field added to the enum expands empty
// here until this block fills it, never a stale pointer.
const char *values[HTS_FOOTER_FIELD_COUNT] = {NULL};
values[HTS_FOOTER_ADDR] = safe_adr;
values[HTS_FOOTER_PATH] = safe_fil;
values[HTS_FOOTER_URL] = safe_url;
values[HTS_FOOTER_DATE] = gmttime;
values[HTS_FOOTER_LASTMODIFIED] = html_inline_safe(
r->lastmodified, safe_lastmod, sizeof(safe_lastmod));
values[HTS_FOOTER_VERSION] = HTTRACK_VERSIONID;
values[HTS_FOOTER_MIME] = html_inline_safe(
r->contenttype, safe_ctype, sizeof(safe_ctype));
values[HTS_FOOTER_CHARSET] = html_inline_safe(
r->charset, safe_charset, sizeof(safe_charset));
values[HTS_FOOTER_STATUS] = status_str;
values[HTS_FOOTER_SIZE] = size_str;
tempo[0] = '\0';
strcatbuff(tempo, eol);
// hts_footer_format returns <0 on overflow, leaving tempo
// unterminated; emitting it would abort in strcatbuff
// below.
if (hts_footer_format(tempo + strlen(tempo),
sizeof(tempo) - strlen(tempo),
StringBuff(opt->footer), fields,
sizeof(fields) / sizeof(fields[0])) >=
0) {
// Overflow (<0) leaves tempo unterminated, and the closing
// eol is reserved below because strcatbuff aborts rather
// than clips: either one would kill the crawl.
if (hts_footer_format(
tempo + strlen(tempo),
sizeof(tempo) - strlen(tempo) - strlen(eol),
StringBuff(opt->footer), values) >= 0) {
strcatbuff(tempo, eol);
HT_ADD(tempo);
}

View File

@@ -1279,40 +1279,60 @@ static int st_entities(httrackp *opt, int argc, char **argv) {
return 0;
}
// -#test=footerfmt <template>: expand a -%F footer with fixed fields (drives
// tests/01_engine-footerfmt.test). Also asserts the overflow/zero-size returns
// the CLI cap keeps out of reach.
// -#test=footerfmt <template>: expand a -%F footer with fixed values (drives
// tests/01_engine-footerfmt.test). Also asserts the published field names and
// the overflow/zero-size returns the CLI cap keeps out of reach.
static int st_footerfmt(httrackp *opt, int argc, char **argv) {
static const hts_footer_field fields[] = {
{"addr", "host.example"},
{"path", "/dir/page.html"},
{"url", "http://host.example/dir/page.html"},
{"date", "DATE"},
{"lastmodified", "LASTMOD"},
{"version", "VER"},
{"mime", "text/html"},
{"charset", "utf-8"},
{"status", "200"},
{"size", "1234"},
};
const size_t nfields = sizeof(fields) / sizeof(fields[0]);
// Spelled out, not read back from the engine: these ten are the published
// contract front ends validate their templates against.
static const char *const names[] = {
"addr", "path", "url", "date", "lastmodified",
"version", "mime", "charset", "status", "size"};
// Filled by id, as htsparse.c does, so the .test's expected strings pin each
// name to its enum slot; a positional list would not see them drift apart.
const char *values[HTS_FOOTER_FIELD_COUNT] = {NULL};
size_t i;
char out[1024];
char tiny[4];
(void) opt;
values[HTS_FOOTER_ADDR] = "host.example";
values[HTS_FOOTER_PATH] = "/dir/page.html";
values[HTS_FOOTER_URL] = "http://host.example/dir/page.html";
values[HTS_FOOTER_DATE] = "DATE";
values[HTS_FOOTER_LASTMODIFIED] = "LASTMOD";
values[HTS_FOOTER_VERSION] = "VER";
values[HTS_FOOTER_MIME] = "text/html";
values[HTS_FOOTER_CHARSET] = "utf-8";
values[HTS_FOOTER_STATUS] = "200";
values[HTS_FOOTER_SIZE] = "1234";
for (i = 0; i < sizeof(names) / sizeof(names[0]); i++) {
assertf(hts_footer_field_ok(names[i]) == HTS_TRUE);
}
assertf(!hts_footer_field_ok(NULL));
assertf(!hts_footer_field_ok(""));
assertf(!hts_footer_field_ok("nosuchfield"));
// A prefix or a longer name must not match, or "{addr}" and "{addrx}" would
// validate alike.
assertf(!hts_footer_field_ok("add"));
assertf(!hts_footer_field_ok("addrx"));
// Matching is exact: the expander is case-sensitive, so a validator that
// accepted "{ADDR}" would green-light a template the crawl emits verbatim.
assertf(!hts_footer_field_ok("Addr"));
// Overflow (named and legacy) and a zero-size buffer must return <0, never
// truncate silently or write out of bounds.
assertf(hts_footer_format(tiny, sizeof(tiny), "{addr}", fields, nfields) < 0);
assertf(hts_footer_format(tiny, sizeof(tiny), "a %s b", fields, nfields) < 0);
assertf(hts_footer_format(out, 0, "", fields, nfields) < 0);
assertf(hts_footer_format(tiny, sizeof(tiny), "{addr}", values) < 0);
assertf(hts_footer_format(tiny, sizeof(tiny), "a %s b", values) < 0);
assertf(hts_footer_format(out, 0, "", values) < 0);
// An empty template yields an empty, terminated string.
assertf(hts_footer_format(out, sizeof(out), "", fields, nfields) == 1 &&
assertf(hts_footer_format(out, sizeof(out), "", values) == 1 &&
out[0] == '\0');
if (argc < 1) {
fprintf(stderr, "footerfmt: needs a template\n");
return 1;
}
if (hts_footer_format(out, sizeof(out), argv[0], fields, nfields) < 0) {
if (hts_footer_format(out, sizeof(out), argv[0], values) < 0) {
fprintf(stderr, "footerfmt: overflow\n");
return 1;
}
@@ -4112,25 +4132,27 @@ static int st_hashkey_bounds(httrackp *opt, int argc, char **argv) {
return 0;
}
/* Prints the filter answer <n> emits for (adr, fil) [up]; with no arguments,
asserts every answer against its expected pattern (#1119). */
/* Prints the filter answer <n> emits for (adr, fil) [up] in [slot]; with no
arguments, asserts every answer against its expected pattern (#1119). */
static int st_wizardfilter(httrackp *opt, int argc, char **argv) {
char pattern[HTS_URLMAXSIZE * 2];
htsbuff f = htsbuff_array(pattern);
(void) opt;
if (argc >= 3) {
hts_wizard_answer_filter(&f, atoi(argv[0]), argv[1], argv[2],
argc >= 4 && atoi(argv[3]) != 0 ? HTS_TRUE
: HTS_FALSE);
hts_wizard_answer_filter(
&f, argc >= 5 ? atoi(argv[4]) : 0, atoi(argv[0]), argv[1], argv[2],
argc >= 4 && atoi(argv[3]) != 0 ? HTS_TRUE : HTS_FALSE);
printf("%s\n", pattern);
return 0;
}
#define EMITS(n, adr, fil, up, expect) \
#define EMITS_SLOT(slot, n, adr, fil, up, expect) \
do { \
hts_wizard_answer_filter(&f, (n), (adr), (fil), (up)); \
hts_wizard_answer_filter(&f, (slot), (n), (adr), (fil), (up)); \
assertf(strcmp(pattern, (expect)) == 0); \
} while (0)
#define EMITS(n, adr, fil, up, expect) \
EMITS_SLOT(0, (n), (adr), (fil), (up), (expect))
/* the host-wide answers: 2 forbids, 5 (allowed to go up) and 6 authorize */
EMITS(2, "foo.com", "/index.html", HTS_FALSE, "-foo.com/*");
@@ -4171,11 +4193,267 @@ static int st_wizardfilter(httrackp *opt, int argc, char **argv) {
EMITS(3, "foo.com", "/x", HTS_FALSE, "");
EMITS(4, "foo.com", "/x", HTS_FALSE, "");
EMITS(50, "foo.com", "/x", HTS_FALSE, "");
/* only slot 0 is ever filled outside the host-scope answers */
EMITS_SLOT(1, 2, "foo.com", "/x", HTS_FALSE, "");
EMITS_SLOT(1, 6, "foo.com", "/x", HTS_FALSE, "");
/* the host-scope answers (#1117): both slots, the starred one missing the
apex is why the second exists */
#define SCOPE_IN HTS_WIZARD_SCOPE_INCLUDE
#define SCOPE_EX HTS_WIZARD_SCOPE_EXCLUDE
EMITS_SLOT(0, SCOPE_IN, "www.example.co.uk", "/x", HTS_FALSE,
"+*.www.example.co.uk/*");
EMITS_SLOT(1, SCOPE_IN, "www.example.co.uk", "/x", HTS_FALSE,
"+www.example.co.uk/*");
EMITS_SLOT(0, SCOPE_IN + 1, "www.example.co.uk", "/x", HTS_FALSE,
"+*.example.co.uk/*");
EMITS_SLOT(1, SCOPE_IN + 1, "www.example.co.uk", "/x", HTS_FALSE,
"+example.co.uk/*");
EMITS_SLOT(0, SCOPE_EX + 1, "www.example.co.uk", "/x", HTS_FALSE,
"-*.example.co.uk/*");
EMITS_SLOT(1, SCOPE_EX + 1, "www.example.co.uk", "/x", HTS_FALSE,
"-example.co.uk/*");
/* the port rides along, the credentials do not */
EMITS_SLOT(0, SCOPE_IN + 1, "www.foo.com:8080", "/x", HTS_FALSE,
"+*.foo.com:8080/*");
EMITS_SLOT(1, SCOPE_IN, "user:pass@www.foo.com", "/x", HTS_FALSE,
"+www.foo.com/*");
/* past the last scope, and slot 2, emit nothing */
EMITS_SLOT(0, SCOPE_IN + 2, "www.foo.com", "/x", HTS_FALSE, "");
EMITS_SLOT(2, SCOPE_IN, "www.foo.com", "/x", HTS_FALSE, "");
/* what the pair must and must not catch */
EMITS_SLOT(0, SCOPE_IN + 1, "www.example.co.uk", "/x", HTS_FALSE,
"+*.example.co.uk/*");
assertf(strjoker("a.b.example.co.uk/x", pattern + 1, NULL, NULL) != NULL);
assertf(strjoker("example.co.uk/x", pattern + 1, NULL, NULL) == NULL);
assertf(strjoker("notexample.co.uk/x", pattern + 1, NULL, NULL) == NULL);
assertf(strjoker("example.co.uk.evil.com/x", pattern + 1, NULL, NULL) ==
NULL);
EMITS_SLOT(1, SCOPE_IN + 1, "www.example.co.uk", "/x", HTS_FALSE,
"+example.co.uk/*");
assertf(strjoker("example.co.uk/x", pattern + 1, NULL, NULL) != NULL);
assertf(strjoker("notexample.co.uk/x", pattern + 1, NULL, NULL) == NULL);
#undef SCOPE_IN
#undef SCOPE_EX
#undef EMITS_SLOT
#undef EMITS
printf("wizardfilter self-test OK\n");
return 0;
}
/* Prints the domain scopes offered for <question>; with no argument, asserts
the enumeration (#1117). */
static int st_wizardscope(httrackp *opt, int argc, char **argv) {
char scope[HTS_URLMAXSIZE];
int k;
(void) opt;
if (argc >= 1) {
for (k = 0; hts_wizard_host_scope(argv[0], k, scope, sizeof(scope)); k++)
printf("%d %s\n", k, scope);
return 0;
}
#define SCOPE(question, k, expect) \
do { \
assertf(hts_wizard_host_scope((question), (k), scope, sizeof(scope))); \
assertf(strcmp(scope, (expect)) == 0); \
} while (0)
/* poisoned first: comparing against '\0' cannot see a clear that never ran */
#define NOSCOPE_SIZED(question, k, size) \
do { \
memset(scope, 'X', sizeof(scope)); \
assertf(!hts_wizard_host_scope((question), (k), scope, (size))); \
assertf(scope[0] == '\0'); \
} while (0)
#define NOSCOPE(question, k) NOSCOPE_SIZED((question), (k), sizeof(scope))
/* k widens by one label at a time, starting at the host itself */
SCOPE("download.example.co.uk/x", 0, "download.example.co.uk");
SCOPE("download.example.co.uk/x", 1, "example.co.uk");
SCOPE("download.example.co.uk/x", 2, "co.uk");
NOSCOPE("download.example.co.uk/x", 3); /* "uk" is a bare TLD */
SCOPE("example.com", 0, "example.com"); /* an adr with no fil works */
NOSCOPE("example.com", 1);
NOSCOPE("localhost/x", 0); /* nothing to widen into */
NOSCOPE("download.example.co.uk/x", -1);
/* protocol and credentials are stripped, the port kept */
SCOPE("ftp://user:pass@www.foo.com/x", 1, "foo.com");
SCOPE("www.foo.com:8080/x", 0, "www.foo.com:8080");
SCOPE("www.foo.com:8080/x", 1, "foo.com:8080");
NOSCOPE("www.foo.com:8080/x", 2);
/* a path that carries dots or a colon must not be read as host labels */
SCOPE("foo.com/a.b.c/d:e", 0, "foo.com");
NOSCOPE("foo.com/a.b.c/d:e", 1);
/* the shared predicate: a dotless run of digits is a hostname, not an IP */
assertf(hts_host_is_ipv4("1.2.3.4", 7));
assertf(!hts_host_is_ipv4("12345", 5));
assertf(!hts_host_is_ipv4("foo.com", 7));
/* an IP literal splits on dots without being a domain */
NOSCOPE("192.168.1.1/x", 0);
NOSCOPE("192.168.1.1:8080/x", 0);
NOSCOPE("[3ffe:b80:1234::1]/x", 0);
/* the dots inside this one reach the label walk unless brackets are refused
*/
NOSCOPE("[::ffff:1.2.3.4]/x", 0);
/* the root label of a fully-qualified host is not a label */
SCOPE("www.foo.com./x", 0, "www.foo.com.");
SCOPE("www.foo.com./x", 1, "foo.com.");
NOSCOPE("www.foo.com./x", 2); /* "com." is still a bare TLD */
NOSCOPE(".", 0);
/* the destination must fit the scope and its terminator, and never truncate
*/
{
const char *q = "www.example.com/x";
const size_t need = strlen("www.example.com");
memset(scope, 'X', sizeof(scope));
assertf(hts_wizard_host_scope(q, 0, scope, need + 1));
assertf(strcmp(scope, "www.example.com") == 0);
NOSCOPE_SIZED(q, 0, need);
NOSCOPE_SIZED(q, 0, 4);
NOSCOPE_SIZED(q, 0, 1);
}
#undef SCOPE
#undef NOSCOPE
#undef NOSCOPE_SIZED
printf("wizardscope self-test OK\n");
return 0;
}
/* #1117: which host-scope range an answer falls in. */
static int st_wizardscopeanswer(httrackp *opt, int argc, char **argv) {
(void) opt;
(void) argc;
(void) argv;
#define ANSWER(n, expect) assertf(hts_wizard_scope_answer(n) == (expect))
/* the plain answers, and the boundary just below the first range */
ANSWER(-999, HTS_DEFAULT);
ANSWER(-1, HTS_DEFAULT);
ANSWER(0, HTS_DEFAULT);
ANSWER(7, HTS_DEFAULT);
ANSWER(50, HTS_DEFAULT);
ANSWER(HTS_WIZARD_SCOPE_INCLUDE - 1, HTS_DEFAULT);
/* include runs up to the exclude base, and exclude has no upper end */
ANSWER(HTS_WIZARD_SCOPE_INCLUDE, HTS_FALSE);
ANSWER(HTS_WIZARD_SCOPE_EXCLUDE - 1, HTS_FALSE);
ANSWER(HTS_WIZARD_SCOPE_EXCLUDE, HTS_TRUE);
ANSWER(INT_MAX, HTS_TRUE);
#undef ANSWER
printf("wizardscopeanswer self-test OK\n");
return 0;
}
/* Poison: comparing the recursion cap against 0 would not see a stray write of
the level the crawl uses. */
#define PRIO_UNSET 42
/* Prints what answer `n` does to the crawl; with no arguments, asserts every
answer, on an undecided, an allowed and an already refused link. */
static int st_wizardverdict(httrackp *opt, int argc, char **argv) {
const hts_wizard asked = opt->wizard;
FILE *const projectlog = opt->log;
char line[HTS_URLMAXSIZE];
FILE *log;
int url, depth;
url = -1; /* the undecided verdict the wizard is asked about */
depth = PRIO_UNSET;
opt->wizard = HTS_WIZARD_ASK;
if (argc >= 1) {
hts_wizard_apply_verdict(opt, atoi(argv[0]), "foo.com", "/a/b.html", &url,
&depth);
printf("forbidden=%d stop=%d prio=%d\n", url,
opt->wizard == HTS_WIZARD_AUTO, depth);
opt->wizard = asked;
return 0;
}
opt->log = NULL; /* the battery walks the answers that warn */
/* answer `n` over a link the crawl had left at `in`: the verdict it must leave,
whether it stops the questions, and the recursion cap it must set. */
#define APPLIES(n, in, forbidden, stop, prio) \
do { \
url = (in); \
depth = PRIO_UNSET; \
opt->wizard = HTS_WIZARD_ASK; \
hts_wizard_apply_verdict(opt, (n), "foo.com", "/a/b.html", &url, &depth); \
assertf(url == (forbidden)); \
assertf(opt->wizard == ((stop) ? HTS_WIZARD_AUTO : HTS_WIZARD_ASK)); \
assertf(depth == (prio)); \
} while (0)
/* '*' refuses and stops the questions */
APPLIES(-1, 0, 1, 1, PRIO_UNSET);
APPLIES(-1, 1, 1, 1, PRIO_UNSET);
/* the refusing answers, 3 included although it emits no filter yet */
APPLIES(0, 0, 1, 0, PRIO_UNSET);
APPLIES(1, 0, 1, 0, PRIO_UNSET);
APPLIES(2, 0, 1, 0, PRIO_UNSET);
APPLIES(3, 0, 1, 0, PRIO_UNSET);
/* 4 caps the recursion, and takes the link like any accepting answer */
APPLIES(4, 0, 0, 0, 1);
APPLIES(4, 1, 1, 0, 1);
/* an accepting answer never clears a refusal the crawl already computed */
APPLIES(5, 1, 1, 0, PRIO_UNSET);
APPLIES(6, 1, 1, 0, PRIO_UNSET);
APPLIES(7, 1, 1, 0, PRIO_UNSET);
APPLIES(50, 1, 1, 0, PRIO_UNSET);
APPLIES(-999, 1, 1, 0, PRIO_UNSET);
APPLIES(6, 0, 0, 0, PRIO_UNSET);
/* both ends of each scope range: include allows, exclude forbids */
APPLIES(HTS_WIZARD_SCOPE_INCLUDE, 0, 0, 0, PRIO_UNSET);
APPLIES(HTS_WIZARD_SCOPE_EXCLUDE - 1, 0, 0, 0, PRIO_UNSET);
APPLIES(HTS_WIZARD_SCOPE_EXCLUDE, 0, 1, 0, PRIO_UNSET);
APPLIES(INT_MAX, 0, 1, 0, PRIO_UNSET);
/* an answer in no range never overturns an accept or a refusal */
APPLIES(8, 0, 0, 0, PRIO_UNSET);
APPLIES(8, 1, 1, 0, PRIO_UNSET);
APPLIES(999, 0, 0, 0, PRIO_UNSET);
APPLIES(-2, 0, 0, 0, PRIO_UNSET);
APPLIES(-1000, 0, 0, 0, PRIO_UNSET);
APPLIES(INT_MIN, 0, 0, 0, PRIO_UNSET);
APPLIES(HTS_WIZARD_SCOPE_INCLUDE - 1, 0, 0, 0, PRIO_UNSET);
/* an answer that does not refuse leaves the link allowed, never undecided */
APPLIES(4, -1, 0, 0, 1);
APPLIES(5, -1, 0, 0, PRIO_UNSET);
APPLIES(6, -1, 0, 0, PRIO_UNSET);
APPLIES(7, -1, 0, 0, PRIO_UNSET);
APPLIES(50, -1, 0, 0, PRIO_UNSET);
APPLIES(8, -1, 0, 0, PRIO_UNSET); /* -999 is #1259's, not asserted here */
APPLIES(HTS_WIZARD_SCOPE_INCLUDE, -1, 0, 0, PRIO_UNSET);
/* and one that does refuse must still refuse it */
APPLIES(-1, -1, 1, 1, PRIO_UNSET);
APPLIES(0, -1, 1, 0, PRIO_UNSET);
APPLIES(1, -1, 1, 0, PRIO_UNSET);
APPLIES(2, -1, 1, 0, PRIO_UNSET);
APPLIES(3, -1, 1, 0, PRIO_UNSET);
APPLIES(HTS_WIZARD_SCOPE_EXCLUDE, -1, 1, 0, PRIO_UNSET);
#undef APPLIES
/* the warning is all an unknown answer does, so a known one must be silent */
log = tmpfile();
assertf(log != NULL);
opt->log = log;
hts_wizard_apply_verdict(opt, 6, "foo.com", "/a/b.html", &url, &depth);
hts_wizard_apply_verdict(opt, 8, "foo.com", "/a/b.html", &url, &depth);
rewind(log);
assertf(fgets(line, (int) sizeof(line), log) != NULL);
assertf(strstr(line, "unknown answer 8") != NULL);
assertf(fgets(line, (int) sizeof(line), log) == NULL);
fclose(log);
opt->log = projectlog;
opt->wizard = asked;
printf("wizardverdict self-test OK\n");
return 0;
}
#undef PRIO_UNSET
/* #159: hts_redirect_same_savefile decides whether a redirect is a same-file
* alias. */
static int st_redirect_samefile(httrackp *opt, int argc, char **argv) {
@@ -9687,8 +9965,14 @@ static const struct selftest_entry {
st_hashkey_bounds},
{"redirect-samefile", "", "same-file redirect detection self-test (#159)",
st_redirect_samefile},
{"wizardfilter", "[<answer> <adr> <fil> [up]]",
{"wizardfilter", "[<answer> <adr> <fil> [up [slot]]]",
"filter emitted by a wizard answer", st_wizardfilter},
{"wizardscope", "[<question>]",
"domain scopes the wizard can offer for a host", st_wizardscope},
{"wizardscopeanswer", "", "host-scope range of a wizard answer",
st_wizardscopeanswer},
{"wizardverdict", "[<answer>]", "what a wizard answer applies",
st_wizardverdict},
{"mime", "<filename>", "MIME type for a filename", st_mime},
{"charset", "<charset> <hex:..|string>",
"convert a string to UTF-8 from a charset", st_charset},

View File

@@ -93,9 +93,16 @@ int commandReturnSet = 0;
httrackp *global_opt = NULL;
static void (*pingFun)(void*) = NULL;
static void (*pingFun)(void *, smallserver_client_event, const char *) = NULL;
static void* pingFunArg = NULL;
/* Report a client liveness event, if anybody is listening. */
static void client_event(smallserver_client_event ev, const char *window) {
if (pingFun != NULL) {
pingFun(pingFunArg, ev, window);
}
}
/* Extern */
extern void webhttrack_main(char *cmd);
extern void webhttrack_lock(void);
@@ -385,6 +392,43 @@ static void copy_header_value(char *dst, size_t size, const char *value) {
strlncatbuff(dst, value, size, size - 1);
}
/** Copy query parameter "name"'s alphanumeric value into dst; true when a
non-empty one fit, and dst is left empty otherwise. Query-string counterpart
to the POST-body checker below. */
static hts_boolean query_alnum_value(char *dst, size_t size, const char *query,
const char *name) {
const size_t namelen = strlen(name);
const char *s = query;
dst[0] = '\0';
while (*s != '\0') {
const char *const amp = strchr(s, '&');
if (strncmp(s, name, namelen) == 0 && s[namelen] == '=') {
const char *v = s + namelen + 1;
size_t n = 0;
while (*v != '\0' && *v != '&' && n + 1 < size &&
isalnum((unsigned char) *v)) {
dst[n++] = *v++;
}
dst[n] = '\0';
/* Truncated, or not alphanumeric to its end, is no value at all: it must
not reach a caller that trusted the return. */
if (n > 0 && (*v == '\0' || *v == '&')) {
return HTS_TRUE;
}
dst[0] = '\0';
return HTS_FALSE;
}
if (amp == NULL) {
break;
}
s = amp + 1;
}
return HTS_FALSE;
}
/** Does the urlencoded request body present the expected session id?
True only if at least one "sid" field is present and every occurrence
matches, so it holds whichever one a later last-write-wins parse keeps.
@@ -686,8 +730,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
};
initStrElt initStr[] = {
{"user", HTS_DEFAULT_USER_AGENT},
{"footer", "<!-- Mirrored from %s%s by HTTrack Website Copier/3.x "
"[XR&CO'2014], %s -->"},
{"footer", HTS_DEFAULT_FOOTER},
{"url2",
"+*.png +*.gif +*.jpg +*.jpeg +*.css +*.js -ad.doubleclick.net/*"},
{NULL, NULL}};
@@ -723,6 +766,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
LLint length = 0;
const char *error_redirect = NULL;
hts_boolean denied = HTS_FALSE;
/* The request proved it holds the session id. */
hts_boolean authed = HTS_FALSE;
char origin[256];
char host[256];
@@ -753,9 +798,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
while((soc_c = (T_SOC) accept(soc, NULL, NULL)) == INVALID_SOCKET) ;
/* Ping */
if (pingFun != NULL) {
pingFun(pingFunArg);
}
client_event(SMALLSERVER_CLIENT_REQUEST, NULL);
/* Lock */
webhttrack_lock();
@@ -865,6 +908,8 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
buffer[0] = '\0';
meth = 0;
denied = HTS_TRUE;
} else {
authed = HTS_TRUE;
}
}
@@ -1158,6 +1203,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
if (url && *++url == '/' && (pos = strchr(url, ' ')) && !(*pos = '\0')) {
char fsfile[1024];
const char *file;
const char *query = "";
FILE *fp;
char *qpos;
@@ -1166,6 +1212,7 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
if (error_redirect == NULL) {
if ((qpos = strchr(url, '?'))) {
*qpos = '\0';
query = qpos + 1;
}
if (strcmp(url, "/") == 0) {
file = "/server/index.html";
@@ -1790,13 +1837,32 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
}
}
fclose(fp);
} else if (strcmp(file, "/ping") == 0 ||
strncmp(file, "/ping?", 6) == 0) {
} else if (strcmp(file, "/ping") == 0) {
/* A cached heartbeat would never reach us again, and silence is
what the watchdog reads as a dead window. */
char error_hdr[] =
"HTTP/1.0 200 Pong\r\n" "Server: httrack small server\r\n"
"Content-type: text/html\r\n";
"HTTP/1.0 200 Pong\r\n"
"Server: httrack small server\r\n"
"Content-type: text/html\r\n"
"Cache-Control: no-cache, must-revalidate, private\r\n"
"Pragma: no-cache\r\n";
char window[SMALLSERVER_WINDOW_ID_MAX + 1];
StringCat(headers, error_hdr);
if (query_alnum_value(window, sizeof(window), query, "w")) {
char verb[SMALLSERVER_WINDOW_ID_MAX + 1];
/* Ending a session is a command, so it carries the session id
like every other one. A heartbeat can only extend a life, and
any local peer or visited page can send one of those. */
client_event(
authed && query_alnum_value(verb, sizeof(verb), query, "e") &&
strcmp(verb, "bye") == 0
? SMALLSERVER_CLIENT_LEAVING
: SMALLSERVER_CLIENT_PING,
window);
}
} else {
char error_hdr[] =
"HTTP/1.0 404 Not Found\r\n" "Server: httrack small server\r\n"
@@ -1873,6 +1939,10 @@ int smallserver(T_SOC soc, char *url, char *method, char *data, char *path) {
#endif
}
/* Only the UI asking to quit is a clean stop; losing the socket or the buffer
is what the caller reports as a failure. */
retour = willexit;
StringFree(headers);
StringFree(output);
StringFree(tmpbuff);
@@ -1918,7 +1988,9 @@ int htslang_uninit(void) {
return 1;
}
void smallserver_setpinghandler(void (*fun)(void*), void*arg) {
void smallserver_setpinghandler(void (*fun)(void *, smallserver_client_event,
const char *),
void *arg) {
pingFun = fun;
pingFunArg = arg;
}

View File

@@ -94,7 +94,20 @@ extern httrackp *global_opt;
#define min(a,b) ((a)>(b)?(b):(a))
#define max(a,b) ((a)>(b)?(a):(b))
extern void smallserver_setpinghandler(void (*fun)(void*), void*arg);
/* What the UI just told us about itself, reported to the ping handler. */
typedef enum {
SMALLSERVER_CLIENT_REQUEST, /* any request; claims no window */
SMALLSERVER_CLIENT_PING, /* one window's heartbeat */
SMALLSERVER_CLIENT_LEAVING /* that window is closing */
} smallserver_client_event;
/* Longest window id a page may claim; a longer one is ignored. */
#define SMALLSERVER_WINDOW_ID_MAX 32
/* fun() receives the id of the window the event came from, NULL when no window
claimed it. */
extern void smallserver_setpinghandler(
void (*fun)(void *, smallserver_client_event, const char *), void *arg);
extern int smallserver_setkey(const char *key, const char *value);
extern int smallserver_setkeyint(const char *key, LLint value);
extern int smallserver_setkeyarr(const char *key, int id, const char *key2, const char *value);

View File

@@ -517,11 +517,11 @@ static void sf_warn_oversize(sf_ctx *ctx, const char *path, LLint size,
if (*ctx->warn_budget <= 0)
return;
if (--(*ctx->warn_budget) == 0) {
hts_log_print(ctx->opt, LOG_NOTICE,
hts_log_print(ctx->opt, LOG_WARNING,
"single-file: further over-cap assets not reported");
return;
}
hts_log_print(ctx->opt, LOG_NOTICE,
hts_log_print(ctx->opt, LOG_WARNING,
"single-file: %s left as a link (" LLintP
" bytes, over the " LLintP "-byte cap)",
path, size, cap);

View File

@@ -890,20 +890,39 @@ int hts_template_format_str(char *buffer, size_t size, const char *format, ...)
return success;
}
// Value of the named field, or "" if absent (never NULL, so callers can pass it
// straight to a formatter).
static const char *footer_field_value(const hts_footer_field *fields,
size_t nfields, const char *name) {
size_t j;
for (j = 0; j < nfields; j++) {
if (strcmp(fields[j].name, name) == 0)
return fields[j].value != NULL ? fields[j].value : "";
// Indexed by hts_footer_field_id. Sized by the initializer, then pinned below:
// a new id without a name here is a compile error, not a NULL slot.
static const char *const footer_field_names[] = {
"addr", "path", "url", "date", "lastmodified",
"version", "mime", "charset", "status", "size"};
enum {
footer_field_names_complete =
1 / (int) (sizeof(footer_field_names) / sizeof(footer_field_names[0]) ==
HTS_FOOTER_FIELD_COUNT)
};
HTSEXT_API hts_boolean hts_footer_field_ok(const char *name) {
size_t i;
if (name == NULL)
return HTS_FALSE;
for (i = 0; i < HTS_FOOTER_FIELD_COUNT; i++) {
if (strcmp(footer_field_names[i], name) == 0)
return HTS_TRUE;
}
return "";
return HTS_FALSE;
}
// Value of a field, or "" (never NULL, so callers can pass it straight to a
// formatter).
static const char *footer_field_value(const char *const *values,
hts_footer_field_id id) {
return values[id] != NULL ? values[id] : "";
}
int hts_footer_format(char *buffer, size_t size, const char *footer,
const hts_footer_field *fields, size_t nfields) {
const char *const values[HTS_FOOTER_FIELD_COUNT]) {
hts_template_format_buf buf = {NULL, buffer, size, 0};
size_t i;
@@ -914,10 +933,10 @@ int hts_footer_format(char *buffer, size_t size, const char *footer,
// order-independent.
if (strstr(footer, "%s") != NULL)
return hts_template_format_str(
buffer, size, footer, footer_field_value(fields, nfields, "addr"),
footer_field_value(fields, nfields, "path"),
footer_field_value(fields, nfields, "date"),
footer_field_value(fields, nfields, "version"), /* EOF */ NULL);
buffer, size, footer, footer_field_value(values, HTS_FOOTER_ADDR),
footer_field_value(values, HTS_FOOTER_PATH),
footer_field_value(values, HTS_FOOTER_DATE),
footer_field_value(values, HTS_FOOTER_VERSION), /* EOF */ NULL);
// "{{"/"}}" emit a literal brace; an unknown "{...}" is left verbatim so
// typos stay visible.
for (i = 0; footer[i] != '\0'; i++) {
@@ -936,11 +955,11 @@ int hts_footer_format(char *buffer, size_t size, const char *footer,
if (end != NULL) {
const size_t namelen = (size_t) (end - (footer + i + 1));
size_t j;
for (j = 0; j < nfields; j++) {
if (strlen(fields[j].name) == namelen &&
strncmp(fields[j].name, footer + i + 1, namelen) == 0) {
if (htsfmt_puts(&buf,
fields[j].value != NULL ? fields[j].value : "") < 0)
for (j = 0; j < HTS_FOOTER_FIELD_COUNT; j++) {
if (strlen(footer_field_names[j]) == namelen &&
strncmp(footer_field_names[j], footer + i + 1, namelen) == 0) {
if (htsfmt_puts(&buf, footer_field_value(
values, (hts_footer_field_id) j)) < 0)
return -1;
i += namelen + 1; // consume the name and its closing '}'
matched = 1;

View File

@@ -78,19 +78,30 @@ HTS_INLINE int rech_tageq_all(const char *adr, const char *s);
int hts_template_format(FILE *const out, const char *format, ...);
int hts_template_format_str(char *buffer, size_t size, const char *format, ...);
// A footer named field and its already-context-escaped value.
typedef struct hts_footer_field {
const char *name;
const char *value;
} hts_footer_field;
// Index of a footer {field} in the engine's name table (htstools.c), and the
// slot its value occupies in the values[] handed to hts_footer_format().
typedef enum {
HTS_FOOTER_ADDR = 0,
HTS_FOOTER_PATH,
HTS_FOOTER_URL,
HTS_FOOTER_DATE,
HTS_FOOTER_LASTMODIFIED,
HTS_FOOTER_VERSION,
HTS_FOOTER_MIME,
HTS_FOOTER_CHARSET,
HTS_FOOTER_STATUS,
HTS_FOOTER_SIZE,
HTS_FOOTER_FIELD_COUNT
} hts_footer_field_id;
// Expand a footer template. A "%s" in it selects the legacy positional model,
// consuming the "addr"/"path"/"date"/"version" fields in that order; otherwise
// "{name}" is substituted from fields by name ("{{"/"}}" emit a literal brace,
// an unknown "{...}" is left verbatim). Values must already be escaped for the
// target context by the caller. Returns <0 on overflow.
// consuming addr, path, date and version in that order; otherwise "{name}" is
// substituted from values, indexed by hts_footer_field_id ("{{"/"}}" emit a
// literal brace, an unknown "{...}" is left verbatim). A NULL slot expands
// empty. Values must already be escaped for the target context by the caller.
// Returns <0 on overflow.
int hts_footer_format(char *buffer, size_t size, const char *footer,
const hts_footer_field *fields, size_t nfields);
const char *const values[HTS_FOOTER_FIELD_COUNT]);
#define rech_tageq(adr,s) \
( \

View File

@@ -557,19 +557,6 @@ static char *path_basename_dup(const char *path) {
return strdupt(b);
}
/* A host made only of digits and dots is an IPv4 literal (never reversed). */
static int surt_host_is_ip(const char *h, size_t n) {
size_t i;
int dots = 0;
for (i = 0; i < n; i++) {
if (h[i] == '.')
dots++;
else if (h[i] < '0' || h[i] > '9')
return 0;
}
return dots > 0;
}
/* SURT-canonicalize url into out (no newline): scheme and userinfo dropped,
host lowercased with a leading www[digits] label stripped and the scheme
default port removed, labels reversed and comma-joined then ')', path+query
@@ -637,7 +624,7 @@ static int surt_canon(const char *url, wbuf *out) {
hostbuf[hlen] = '\0';
if (!is_ipv6)
is_ip = surt_host_is_ip(hostbuf, hlen);
is_ip = hts_host_is_ipv4(hostbuf, hlen);
if (!is_ipv6 && !is_ip && hlen >= 4 && hostbuf[0] == 'w' &&
hostbuf[1] == 'w' && hostbuf[2] == 'w') {

View File

@@ -105,49 +105,133 @@ static void htsweb_sig_brpipe(int code) {
/* Threads that never return; no wait may count on them draining. */
static int nonjoinable_threads = 0;
/* Server/client ping handling */
/* Session lifetime: each window pings under its own id and drops it when it
closes, so an abandoned server stops instead of outliving the session and
holding its payload open (a mounted disk image, on macOS). Windows are
counted, not timed: closing one of several must not end the session. */
#define PING_PERIOD 5
/* Silence tolerated from one window. Generous: a hidden tab has its timers
throttled to as little as one wake-up a minute. */
static int pingTimeout = 120;
/* Once the last window leaves, only a page navigation can bring one back, and
that takes a fraction of a second over the loopback. */
#define LEAVE_GRACE max(2, min(5, pingTimeout / 4))
/* Windows tracked at once. A full table refuses newcomers rather than evicting:
dropping a live window is what would let a flood of ids end the session. */
#define MAX_WINDOWS 16
static htsmutex pingMutex = HTSMUTEX_INIT;
static unsigned int pingId = 0;
static unsigned int getPingId(void) {
unsigned int id;
hts_mutexlock(&pingMutex);
id = pingId;
hts_mutexrelease(&pingMutex);
return id;
/* Seconds the watchdog has been awake, not wall-clock: time(NULL) jumps across
a laptop suspend, and a suspended machine must not age a session. */
static int ticks = 0;
static struct {
char id[SMALLSERVER_WINDOW_ID_MAX + 1];
int last_seen;
} windows[MAX_WINDOWS];
static int windowCount = 0;
static int emptySince = 0; /* tick the last window left at */
static hts_boolean anyWindow = HTS_FALSE; /* a window has claimed an id */
static int lastSeen = 0; /* tick of the last request of any kind */
static hts_boolean anyRequest = HTS_FALSE; /* something has connected */
/* Drop windows[i], moving the last entry into its slot: a caller removing while
it iterates must walk backwards. Caller holds pingMutex. */
static void window_forget(int i) {
windows[i] = windows[--windowCount];
if (windowCount == 0) {
emptySince = ticks;
}
}
static void ping(void) {
static void pingHandler(void *arg, smallserver_client_event ev,
const char *window) {
int i = 0;
(void) arg;
hts_mutexlock(&pingMutex);
pingId++;
lastSeen = ticks;
anyRequest = HTS_TRUE;
/* A bare request names no window: any local peer can open a connection, but
none may cancel a real window's departure. */
if (window != NULL) {
while (i < windowCount && strcmp(windows[i].id, window) != 0) {
i++;
}
if (ev == SMALLSERVER_CLIENT_LEAVING) {
if (i < windowCount) {
window_forget(i);
}
} else if (i < windowCount) {
windows[i].last_seen = ticks;
} else if (windowCount < MAX_WINDOWS) {
windows[windowCount].id[0] = '\0';
strlncatbuff(windows[windowCount].id, window,
sizeof(windows[windowCount].id),
sizeof(windows[windowCount].id) - 1);
windows[windowCount++].last_seen = ticks;
anyWindow = HTS_TRUE;
}
}
hts_mutexrelease(&pingMutex);
}
static void client_ping(void *pP) {
#ifndef _WIN32
/* Timeout to 120s ; normally client pings every 30 second */
static int timeout = 120;
/* Wait for parent to die (legacy browser mode). */
const pid_t ppid = (pid_t) (uintptr_t) pP;
while (!kill(ppid, 0)) {
sleep(1);
}
/* Parent (webhttrack script) is dead: is client pinging ? */
for(;;) {
unsigned int id = getPingId();
sleep(timeout);
if (getPingId() == id) {
break;
}
}
/* Die! */
fprintf(stderr,
"Parent process %d died, and client did not ping for %ds: exiting!\n",
(int) ppid, timeout);
exit(EXIT_FAILURE);
/* True unless the launcher we were started from is known to be gone. */
static hts_boolean parent_is_alive(uintptr_t ppid) {
#ifdef _WIN32
(void) ppid;
return HTS_TRUE; /* no cheap probe; the heartbeat carries this */
#else
/* kill(0) would signal our own process group, never a parent. */
return ppid == 0 || kill((pid_t) ppid, 0) == 0 ? HTS_TRUE : HTS_FALSE;
#endif
}
static void pingHandler(void*arg) {
ping();
static void client_ping(void *pP) {
/* uintptr_t, not pid_t: MSVC has no such type, and this signature is not
inside a POSIX guard. */
const uintptr_t ppid = (uintptr_t) pP;
const char *why = NULL;
while (why == NULL) {
int i;
Sleep(1000);
/* A mirror in flight outranks every rule below: it may have hours of
crawling behind it, and the user can always come back to its page. */
if (commandRunning) {
continue;
}
hts_mutexlock(&pingMutex);
ticks++;
/* A window that stops pinging without a goodbye crashed with its browser.
*/
for (i = windowCount; i-- > 0;) {
if (ticks - windows[i].last_seen >= pingTimeout) {
window_forget(i);
}
}
if (anyWindow && windowCount == 0 && ticks - emptySince >= LEAVE_GRACE) {
why = "the interface was closed";
} else if (!anyWindow &&
ticks - lastSeen >= pingTimeout
/* No window ever pinged: a browser too old for it, or none
opened. Fall back to the launcher dying with that browser,
rather than to silence, which a reader also produces. */
&& (!anyRequest || !parent_is_alive(ppid))) {
why = "the interface went silent";
}
hts_mutexrelease(&pingMutex);
/* Re-read after the decision: a mirror may have started while it was made,
and exiting now would lose it. */
if (commandRunning) {
why = NULL;
}
}
fprintf(stderr, "Exiting: %s\n", why);
exit(EXIT_SUCCESS);
}
int main(int argc, char *argv[]) {
@@ -184,6 +268,7 @@ int main(int argc, char *argv[]) {
fprintf(stderr, "** Warning: use the webhttrack frontend if available\n");
fprintf(stderr,
"usage: %s [--port <port>] [--bind <address>] [--ppid parent-pid] "
"[--ping-timeout <seconds>] "
"<path-to-html-root-dir> [key value [key value]..]\n",
argv[0]);
fprintf(stderr, "example: %s /usr/share/httrack/\n", argv[0]);
@@ -288,6 +373,17 @@ int main(int argc, char *argv[]) {
fprintf(stderr, "couldn't set the parent PID to %s\n", argv[i + 1]);
return -1;
}
} else if (strcmp(argv[i], "--ping-timeout") == 0 && i + 1 < argc) {
/* Bounded, not just parsed: %d wrapping a huge value into a plausible one
is what #614 cost on --port, two cases above. */
char *end = NULL;
const long v = strtol(argv[i + 1], &end, 10);
if (end == argv[i + 1] || *end != '\0' || v < 1 || v > 86400) {
fprintf(stderr, "couldn't set the ping timeout to %s\n", argv[i + 1]);
return -1;
}
pingTimeout = (int) v;
} else if (i + 1 < argc) {
smallserver_setkey(argv[i], argv[i + 1]);
} else {
@@ -304,9 +400,7 @@ int main(int argc, char *argv[]) {
/* pinger */
if (parentPid > 0) {
if (hts_newthread(client_ping, (void *) (uintptr_t) parentPid) == 0) {
#ifndef _WIN32
nonjoinable_threads++; /* client_ping() only ever leaves through exit() */
#endif
}
smallserver_setpinghandler(pingHandler, NULL);
}
@@ -389,7 +483,13 @@ static void back_launch_cmd(void *pP) {
void webhttrack_main(char *cmd) {
commandRunning = 1;
DEBUG(fprintf(stderr, "commandRunning=1\n"));
hts_newthread(back_launch_cmd, (void *) strdup(cmd));
if (hts_newthread(back_launch_cmd, (void *) strdup(cmd)) != 0) {
/* Nothing else clears the flag, and while it is set the watchdog holds the
server open for a mirror that never started. */
commandRunning = 0;
commandEnd = 1;
commandReturn = -1;
}
}
void webhttrack_lock(void) {

View File

@@ -181,7 +181,84 @@ static void wizard_cat_path(htsbuff *f, const char *sign, const char *adr,
htsbuff_catn(f, fil, len);
}
void hts_wizard_answer_filter(htsbuff *f, int n, const char *adr,
HTSEXT_API hts_boolean hts_wizard_host_scope(const char *question, int k,
char *dst, size_t dstsize) {
const char *host, *port, *slash, *end, *scope;
size_t len;
if (dst == NULL || dstsize == 0)
return HTS_FALSE;
dst[0] = '\0';
if (question == NULL || k < 0)
return HTS_FALSE;
host = jump_identification_const(question);
port = jump_toport_const(question);
slash = strchr(host, '/');
end = host + strlen(host);
scope = host;
if (slash != NULL && slash < end)
end = slash;
/* the port belongs to the filter, so keep it and only bound the label walk */
if (port != NULL && port < end)
slash = port;
else
slash = end;
/* a fully-qualified "foo.com." ends on the root label, which is not one */
if (slash > host && slash[-1] == '.')
slash--;
if (slash == host || *host == '[') /* no host, or an IPv6 literal */
return HTS_FALSE;
if (hts_host_is_ipv4(host, (size_t) (slash - host)))
return HTS_FALSE;
/* widen by dropping one leading label per step, and never offer a bare TLD */
for (; k > 0; k--) {
const char *dot = memchr(scope, '.', (size_t) (slash - scope));
if (dot == NULL)
return HTS_FALSE;
scope = dot + 1;
}
if (memchr(scope, '.', (size_t) (slash - scope)) == NULL)
return HTS_FALSE;
len = (size_t) (end - scope);
if (len >= dstsize)
return HTS_FALSE;
memcpy(dst, scope, len);
dst[len] = '\0';
return HTS_TRUE;
}
hts_tristate hts_wizard_scope_answer(int n) {
if (n >= HTS_WIZARD_SCOPE_EXCLUDE)
return HTS_TRUE;
if (n >= HTS_WIZARD_SCOPE_INCLUDE)
return HTS_FALSE;
return HTS_DEFAULT;
}
/* The subdomain form of the scope in slot 0, its apex in slot 1: the starred
one does not match the apex, so a whole-domain answer needs both. */
static void wizard_cat_scope(htsbuff *f, const char *sign, const char *adr,
int n, int slot) {
char scope[HTS_URLMAXSIZE];
const int k =
n - (hts_wizard_scope_answer(n) == HTS_TRUE ? HTS_WIZARD_SCOPE_EXCLUDE
: HTS_WIZARD_SCOPE_INCLUDE);
if (slot >= HTS_WIZARD_MAX_FILTERS ||
!hts_wizard_host_scope(adr, k, scope, sizeof(scope)))
return;
htsbuff_cpy(f, sign);
if (slot == 0)
htsbuff_cat(f, "*.");
htsbuff_cat(f, scope);
htsbuff_cat(f, "/*");
}
void hts_wizard_answer_filter(htsbuff *f, int slot, int n, const char *adr,
const char *fil, hts_boolean seeker_up) {
size_t dir = hts_lastcharoffset(fil);
@@ -189,6 +266,13 @@ void hts_wizard_answer_filter(htsbuff *f, int n, const char *adr,
dir--;
htsbuff_cpy(f, "");
if (hts_wizard_scope_answer(n) != HTS_DEFAULT) {
wizard_cat_scope(f, hts_wizard_scope_answer(n) == HTS_TRUE ? "-" : "+", adr,
n, slot);
return;
}
if (slot != 0) /* every other answer emits a single filter */
return;
switch (n) {
case 0: /* this link only */
wizard_cat_path(f, "-", adr, fil, (size_t) -1);
@@ -238,6 +322,51 @@ void hts_wizard_answer_filter(htsbuff *f, int n, const char *adr,
}
}
void hts_wizard_apply_verdict(httrackp *opt, int n, const char *adr,
const char *fil, int *forbidden_url,
int *set_prio_to) {
switch (n) {
case -1: /* skip this link and every question after it */
*forbidden_url = 1;
opt->wizard = HTS_WIZARD_AUTO;
break;
case 0: /* this link */
case 1: /* this directory and below */
case 2: /* the whole host */
case 3: /* the parent directory, which emits no filter yet */
*forbidden_url = 1;
break;
case 4: /* wizard filters both allow and forbid, so an isolated link taken
with no depth limit would mirror the whole site */
*set_prio_to = 0 + 1; /* recursion level 0 */
break;
case 5: /* this directory and below, or the whole host */
case 6: /* the whole host */
case 7: /* this directory, files only */
case 50: /* nothing to do */
case -999: /* the "!" answer, and anything the front end could not parse */
break;
default: /* a scope answer forbids like 2 or allows like 6 */
if (hts_wizard_scope_answer(n) == HTS_TRUE)
*forbidden_url = 1;
else if (hts_wizard_scope_answer(n) == HTS_DEFAULT)
hts_log_print(opt, LOG_WARNING,
"(wizard) unknown answer %d at %s%s, keeping the computed "
"verdict",
n, adr, fil);
break;
}
/* the question is asked only while undecided; an answer that does not forbid
authorizes the link */
if (*forbidden_url == -1)
*forbidden_url = 0;
}
static int hts_acceptlink_(httrackp * opt, int ptr,
const char *adr, const char *fil, const char *tag,
const char *attribute, int *set_prio_to,
@@ -712,7 +841,7 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
/* en cas de question, ou lien primaire (enregistrer autorisations) */
if (question || (ptr == 0)) {
const char *s;
const char *s = NULL; /* the front end's raw reply, NULL if unasked */
int n = 0;
// si primaire (plus bas) alors ...
@@ -765,8 +894,9 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
n = force_mirror;
}
/* sanity check - reallocate filters HERE */
if ((*_FILTERS_PTR) + 1 >= opt->maxfilter) {
/* sanity check - reallocate filters HERE (a host-scope answer emits two)
*/
if ((*_FILTERS_PTR) + 2 >= opt->maxfilter) {
opt->maxfilter += HTS_FILTERSINC;
if (filters_init(&_FILTERS, opt->maxfilter, HTS_FILTERSINC) == 0) {
printf("PANIC! : Too many filters : >%d [%d]\n", (*_FILTERS_PTR),
@@ -780,66 +910,39 @@ static int hts_acceptlink_(httrackp * opt, int ptr,
}
}
// here we have enough room for a new filter if necessary
switch (n) {
case -1: // sauter tout le reste
forbidden_url = 1;
opt->wizard = HTS_WIZARD_AUTO; // sauter tout le reste
break;
case 0: // forbid the same link: adr/fil
case 1: // forbid the whole directory and subdirs: adr/path/*
case 2: // the whole address: adr/*
forbidden_url = 1;
break;
case 3: // ** A FAIRE
forbidden_url = 1;
/*
{
int i=strlen(adr)-1;
while((adr[i]!='/') && (i>0)) i--;
if (i>0) {
hts_wizard_apply_verdict(opt, n, adr, fil, &forbidden_url, set_prio_to);
}
} */
break;
//
case 4: // same link
// PAS BESOIN!!
/*HT_INSERT_FILTERS0; // insérer en 0
strcpybuff(_FILTERS[0],"+");
strcatbuff(_FILTERS[0],adr);
if (*fil!='/') strcatbuff(_FILTERS[0],"/");
strcatbuff(_FILTERS[0],fil); */
// étant donné le renversement wizard/primary filter (les primary autorisent up/down ET interdisent)
// il faut éviter d'un lien isolé effectue un miroir total..
*set_prio_to = 0 + 1; // niveau de récursion=0 (pas de miroir)
break;
case 5: // allow the whole directory and its children, or the domain
case 6: // same domain
case 7: // allow this directory
break;
case 50: // nothing to do
break;
} // switch
/* the pattern half of the answer: a new answer needs both switches */
/* the pattern half of the answer */
{
char BIGSTK pattern[HTS_FILTER_SLOT_SIZE];
/* the log echoes the inserted slots, so it cannot drift from them */
char BIGSTK list[HTS_WIZARD_MAX_FILTERS * (HTS_FILTER_SLOT_SIZE + 1)];
htsbuff f = htsbuff_array(pattern);
htsbuff added = htsbuff_array(list);
int slot;
hts_wizard_answer_filter(
&f, n, adr, fil,
(opt->seeker & HTS_SEEKER_UP) != 0 ? HTS_TRUE : HTS_FALSE);
if (f.len != 0) {
for (slot = 0; slot < HTS_WIZARD_MAX_FILTERS; slot++) {
hts_wizard_answer_filter(
&f, slot, n, adr, fil,
(opt->seeker & HTS_SEEKER_UP) != 0 ? HTS_TRUE : HTS_FALSE);
if (f.len == 0)
break;
HT_INSERT_FILTERS0; // insert at slot 0
strlcpybuff(_FILTERS[0], pattern, HTS_FILTER_SLOT_SIZE);
if (added.len != 0)
htsbuff_cat(&added, " ");
htsbuff_cat(&added, _FILTERS[0]);
}
/* the built-in query3 answers "" for nobody, so ask who replied */
if (s != NULL && HAS_CALLBACK(opt, query3)) {
hts_log_print(
opt, LOG_NOTICE, "(wizard) answer '%s' (n=%d) for %s%s: %s%s%s",
s, n, adr, fil,
forbidden_url == 1 ? "forbidden"
: forbidden_url == 0 ? "allowed"
: "no verdict",
added.len != 0 ? ", filters: " : "", htsbuff_str(&added));
}
}

View File

@@ -57,12 +57,30 @@ hts_boolean hts_robots_forbids(httrackp *opt, const char *adr, const char *fil,
hts_boolean filters_decided,
hts_boolean filters_refused);
/* Builds into `f` the filter answer `n` adds for the link (adr,fil), and leaves
`f` empty when the answer adds none. `seeker_up` is the HTS_SEEKER_UP bit of
opt->seeker, read by answer 5. */
void hts_wizard_answer_filter(htsbuff *f, int n, const char *adr,
/* Most filters one wizard answer can add. Slots must stay contiguous: the
caller stops at the first empty one. */
#define HTS_WIZARD_MAX_FILTERS 2
/* Builds into `f` the `slot`-th filter answer `n` adds for the link (adr,fil),
and leaves `f` empty past the last one. Only the host-scope answers emit a
second, because their starred form misses the apex. `seeker_up` is the
HTS_SEEKER_UP bit of opt->seeker, read by answer 5. */
void hts_wizard_answer_filter(htsbuff *f, int slot, int n, const char *adr,
const char *fil, hts_boolean seeker_up);
/* Which host-scope range answer `n` falls in: HTS_TRUE excludes the scope,
HTS_FALSE includes it, HTS_DEFAULT for any answer outside both ranges. */
hts_tristate hts_wizard_scope_answer(int n);
/* Applies the verdict half of answer `n` for the link (adr,fil): refuses it,
stops the questions, or bans recursion from it. It never overturns a verdict
already computed, but resolves an undecided (-1) link: refused, or
authorized. The filter half is hts_wizard_answer_filter(), and a new answer
needs both. */
void hts_wizard_apply_verdict(httrackp *opt, int n, const char *adr,
const char *fil, int *forbidden_url,
int *set_prio_to);
/* A (tag, attribute) pair naming a reference kind. */
#ifndef HTS_DEF_DEFSTRUCT_htspair_t
#define HTS_DEF_DEFSTRUCT_htspair_t

View File

@@ -361,6 +361,11 @@ HTSEXT_API int copy_htsopt(const httrackp *from, httrackp *to);
*/
HTSEXT_API hts_boolean hts_host_alias_rule_ok(const char *rule);
/** Whether @p name is a field the -%F footer expands as "{name}", so a front
end can validate a template against the engine's own list. Matching is
exact, as the expander's is. @return HTS_TRUE if known. */
HTSEXT_API hts_boolean hts_footer_field_ok(const char *name);
/** Return the engine's last error message, or NULL. The string is owned by
@p opt; do not free it, and use it only while @p opt lives. */
HTSEXT_API char *hts_errmsg(httrackp *opt);
@@ -452,6 +457,23 @@ HTSEXT_API char *jump_identification(char *);
HTSEXT_API const char *jump_identification_const(const char *);
/** Write into dst the k-th domain scope the wizard can offer for the string
query3 was handed (an "adr" or an "adr[/fil]"). Scopes widen as k grows: for
download.example.co.uk/x, "download.example.co.uk", then "example.co.uk",
then "co.uk". A front end enumerates its menu by looping until this returns
HTS_FALSE, and answers HTS_WIZARD_SCOPE_INCLUDE+k or
HTS_WIZARD_SCOPE_EXCLUDE+k.
A bare TLD is never offered, and an IP literal has no scopes, so an empty
menu is normal. Protocol and credentials are stripped and the port is kept,
so a caller must not split the host itself.
dst is emptied on every failure, and HTS_URLMAXSIZE bytes always suffice. A
shorter dst also returns HTS_FALSE, which the loop cannot tell from the end
of the list. */
HTSEXT_API hts_boolean hts_wizard_host_scope(const char *question, int k,
char *dst, size_t dstsize);
/** Like jump_identification() and also strip a leading "www." host prefix,
returning a pointer into the input to the normalized host. */
HTSEXT_API char *jump_normalized(char *);

View File

@@ -712,10 +712,28 @@ static const char *__cdecl htsshow_query3(t_hts_callbackarg * carg,
"5 Mirror this link (useful)\n"
"6 Mirror all links located on the same domain as this link\n" "\n",
question);
/* the domain scopes are host-dependent, so the engine enumerates them */
{
char scope[HTS_URLMAXSIZE];
int k;
for (k = 0; hts_wizard_host_scope(question, k, scope, sizeof(scope)); k++)
printf("%d Mirror %s and every host below it\n",
HTS_WIZARD_SCOPE_INCLUDE + k, scope);
for (k = 0; hts_wizard_host_scope(question, k, scope, sizeof(scope)); k++)
printf("%d Ignore %s and every host below it\n",
HTS_WIZARD_SCOPE_EXCLUDE + k, scope);
if (k != 0)
printf("\n");
}
do {
printf(">> ");
io_flush;
linput(stdin, line, 200);
/* linput() reports neither EOF nor a read error, so an unanswerable prompt
would spin here; a closed stdin sets only ferror */
if (!strnotempty(line) && (feof(stdin) || ferror(stdin)))
strcpybuff(line, "*"); /* refuse this link and ask nothing more */
} while(!strnotempty(line));
printf("ok..\n");
return line;

View File

@@ -59,6 +59,18 @@ function launch_browser {
log "Browser (or helper) exited"
}
# Wait until the server stops (the interface closed, so this process has nothing
# left to represent) or the browser exits, whichever comes first. An old browser
# only ever signals by exiting.
function wait_for_session {
local browserpid=$1 sessionpid=$2
while test -n "${sessionpid}${browserpid}"; do
test -n "${sessionpid}" && ! kill -0 "${sessionpid}" 2>/dev/null && break
test -n "${browserpid}" && ! kill -0 "${browserpid}" 2>/dev/null && break
sleep 1
done
}
# First ensure that we can launch the server
BINPATH=
for i in "${SRCHPATH[@]}"; do
@@ -147,8 +159,14 @@ function cleanup {
# Cleanup in case of emergency
trap "cleanup now; exit" HUP INT QUIT PIPE TERM
# Got SRVURL, launch browser
launch_browser "${BROWSEREXE}" "${SRVURL}"
# Got SRVURL, launch browser. Backgrounded so the wait below can watch the
# server too, rather than only the browser.
launch_browser "${BROWSEREXE}" "${SRVURL}" &
BROWSERPID=$!
# Deliberately not SRVPID, which cleanup would then kill, taking a running mirror
# with it.
SESSIONPID=$(grep -E PID= "${TMPSRVFILE}" | cut -f2- -d=)
wait_for_session "${BROWSERPID}" "${SESSIONPID}"
# That's all, folks!
trap "" HUP INT QUIT PIPE TERM

View File

@@ -3,7 +3,7 @@
<BR>
<BR>
<H6 ALIGN="RIGHT">
<I>Mirror and index made by HTTrack Website Copier [XR&amp;CO'2008]</I>
<I>Mirror and index made by HTTrack Website Copier [XR&amp;CO]</I>
</H6>
%s
<!-- Thanks for using HTTrack Website Copier! -->
@@ -22,7 +22,7 @@
<table width="76%%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 2008 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
<td id="footer"><small>&copy; 1998 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>

View File

@@ -1,7 +1,7 @@
</TABLE>
<BR>
<H6 ALIGN="RIGHT">
<I>Mirror and index made by HTTrack Website Copier [XR&CO'2008]</I>
<I>Mirror and index made by HTTrack Website Copier [XR&CO]</I>
</H6>
%s
<!-- Thanks for using HTTrack Website Copier! -->
@@ -19,7 +19,7 @@
<table width="76%%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small>&copy; 2008 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
<td id="footer"><small>&copy; 1998 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></td>
</tr>
</table>

View File

@@ -114,4 +114,49 @@ grep -q ' -%F "" -r2' "$out2/hts-cache/doit.log" || {
exit 1
}
# --- 4. a value whose first character is a quote survives the round-trip -----
# Two leading quotes, because the -O pass strips one pair: the footer keeps a
# leading quote, which reopens on reprise and swallows -r2 unless escaped.
out3="$tmp/out3"
site3="$tmp/site3"
footer='"<!-- MARK -->'
# the footer is injected into the html body, so this fixture needs real tags
mkdir -p "$site3"
printf '<html><body><a href="a.html">a</a></body></html>' >"$site3/index.html"
echo '<html><body>aaa</body></html>' >"$site3/a.html"
rc=0
"$bin" "file://$site3/index.html" -O "$out3" --quiet -n -%v0 \
-%F "\"$footer\"" -r2 >/dev/null 2>&1 || rc=$?
test "$rc" -eq 0 || {
echo "FAIL: initial mirror with a leading-quote footer exited $rc"
exit 1
}
page=$(find "$out3" -path '*/site3/index.html' -print -quit)
test -n "$page" || {
echo "FAIL: mirrored page missing after the leading-quote mirror"
exit 1
}
grep -qF "$footer" "$page" || {
echo "FAIL: run 1 did not inject the footer verbatim"
exit 1
}
rc=0
"$bin" -O "$out3" --quiet >/dev/null 2>&1 || rc=$?
test "$rc" -eq 0 || {
echo "FAIL: leading-quote reprise exited $rc"
exit 1
}
grep -qF "$footer" "$page" || {
echo "FAIL: the footer changed across the reprise (leading quote written raw?)"
grep -o '<!--[^>]*MARK[^>]*-->' "$page" | head -1
exit 1
}
# -r2 sits after the footer. A plain ' -r2' match would also hit it *inside* a
# swallowed footer, so pin the escaped token and its neighbour instead.
grep -qF -- '-%F "\"<!-- MARK -->" -r2' "$out3/hts-cache/doit.log" || {
echo "FAIL: footer and -r2 are not two tokens in the regenerated doit.log"
head -1 "$out3/hts-cache/doit.log"
exit 1
}
exit 0

View File

@@ -0,0 +1,104 @@
#!/bin/bash
#
# A -%F footer whose expansion exactly filled the on-page buffer aborted the
# crawl (SIGABRT): the emitter appends its closing newline with strcatbuff,
# which aborts rather than clips. #670 covered the overflow, not the exact fit.
set -eu
# shellcheck source=tests/testlib.sh
. "$(dirname "$0")/testlib.sh"
# The emitter's buffer, 1024 + 2 * HTS_URLMAXSIZE. The sweep window is derived
# from it; a wrong value makes the crossing check below fail loudly.
bufsize=3072
# A {path} of this length divides the buffer with room left for the literal
# padding that tunes the last few bytes, and keeps the mirror under MAX_PATH.
pathlen=120
mark=ZFOOTERZ
dir=$(mktemp -d)
cleanup_push rm -rf "$dir"
mir="$dir/mir"
page=
body=
make_page() { # make_page SEGLEN, leaving an LF page at $page
local seg
seg=$(printf 'a%.0s' $(seq 1 "$1"))
mkdir -p "$dir/$seg"
page="$dir/$seg/index.html"
write_page '<html><body>hi</body></html>'
}
# The emitter follows the page's own line ending, so the CR here is what picks
# "\r\n" and moves the boundary two bytes.
write_page() { printf '%s' "$1" >"$page"; }
crawl() { # crawl FOOTER [LABEL], leaving the mirrored page in $body
rm -rf "$mir"
httrack "file://$page" -O "$mir" -%F "$1" -q -s0 -%v0 >/dev/null 2>&1 ||
fail "crawl died (exit $?) on ${2:-a ${#1}-char footer}"
# Under file/, not $mir: the makeindex top index carries no -%F footer.
local found
found=$(find "$mir/file" -name index.html 2>/dev/null || true)
test -n "$found" || fail "page not mirrored; the footer path never ran"
body=$(cat "$found")
}
# Which line the footer lands on follows the page's own newlines, so find it by
# its marker rather than by position.
footer_line() { firstline "$(tr -d '\r' <<<"$body" | grep -F "$mark" || true)"; }
path_expansion() { # the length {path} expands to
local line
crawl "${mark}{path}"
line=$(footer_line)
test -n "$line" || fail "no footer emitted while measuring {path}"
printf '%s\n' "$((${#line} - ${#mark}))"
}
# -%F caps at 253 chars, so reaching a ~3 KB expansion takes repeats of the
# longest field a file:// crawl offers. Size the directory to make {path}
# exactly pathlen, rather than inherit whatever length mktemp handed out.
make_page 8
fixed=$(($(path_expansion) - 8))
# Loudly, not a skip: the Windows leg compares the skip set exactly, so a skip
# reds it there anyway and quietly drops the coverage everywhere else.
test "$fixed" -lt "$pathlen" ||
fail "temp path is ${fixed} chars, leaving no room for a ${pathlen}-char one"
make_page $((pathlen - fixed))
got=$(path_expansion)
test "$got" -eq "$pathlen" || fail "{path} is ${got} chars, wanted ${pathlen}"
reps=$(((bufsize - 72) / pathlen))
sweep() { # sweep LABEL: walk the expansion across the point where footers drop
local total pad footer line emitted=0 dropped=0
for total in $(seq $((bufsize - 6)) "$bufsize"); do
pad=$((total - reps * pathlen))
test "$pad" -gt ${#mark} || fail "padding is down to ${pad} chars"
footer="${mark}$(printf 'A%.0s' $(seq 1 $((pad - ${#mark}))))"
footer="${footer}$(printf '{path}%.0s' $(seq 1 "$reps"))"
test ${#footer} -lt 254 || fail "footer template is ${#footer} chars"
crawl "$footer" "a $1 page at expansion ${total}"
if grep -q "$mark" <<<"$body"; then
line=$(footer_line)
# Whole or not at all: a clipped footer runs into the page below it.
test "${#line}" -eq "$total" ||
fail "$1: ${total}-char footer emitted as ${#line} chars"
emitted=$((emitted + 1))
else
dropped=$((dropped + 1))
fi
done
# Both outcomes prove the window straddles the drop point, where the abort
# was. All-emitted or all-dropped means the window missed it.
test "$emitted" -gt 0 || fail "$1: no length in the sweep emitted a footer"
test "$dropped" -gt 0 || fail "$1: no length in the sweep was dropped"
}
sweep LF
# CRLF costs two bytes at each end, so the boundary sits two lengths lower: a
# fix reserving one byte rather than strlen(eol) still aborts here.
write_page $'<html>\r\n<body>hi</body></html>'
sweep CRLF

View File

@@ -3,8 +3,8 @@
set -euo pipefail
# -%F footer expansion (hts_footer_format via -#test=footerfmt). Fixed fields:
# addr=host.example path=/dir/page.html date=DATE version=VER.
# -%F footer expansion (hts_footer_format via -#test=footerfmt). The expected
# strings below also pin each field name to its engine enum slot.
ftr() {
out="$(httrack -O /dev/null -#test=footerfmt "$1")"
test "$out" == "$2" || {
@@ -36,7 +36,7 @@ ftr '100% {path}' '100% /dir/page.html'
# %s anywhere forces legacy mode, so a mixed template leaves {addr} literal.
ftr '{addr} %s' '{addr} host.example'
# The added named fields (url, lastmodified, mime, charset, status, size).
# Distinct values per field, so a slot transposition shows as swapped text.
ftr '{url}' 'http://host.example/dir/page.html'
ftr 'src {lastmodified}, got {date}' 'src LASTMOD, got DATE'
ftr '{mime}; {charset}; {status}; {size}' 'text/html; utf-8; 200; 1234'

View File

@@ -36,11 +36,43 @@ echo "marker: the wedged test started"
sleep 300
EOF
start=$SECONDS
# Time the guard to its DUMP announcement, not to the driver's exit: the dump that
# follows runs for minutes on an emulated host, and that is not what is under test.
rc=0
HTTRACK_PROGRESS_LOG="$tmp/progress" HTTRACK_TEST_TIMEOUT=5 \
bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 || rc=$?
elapsed=$((SECONDS - start))
fired=
marked=
total=
run_wedged() { # run_wedged <budget> [VAR=VAL...]
local budget=$1 start=$SECONDS pid
shift
: >"$tmp/progress"
rc=0
fired=
marked=
env "$@" HTTRACK_PROGRESS_LOG="$tmp/progress" HTTRACK_TEST_TIMEOUT="$budget" \
bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 &
pid=$!
while kill -0 "$pid" 2>/dev/null; do
# Read with the shell: a fork per poll would blur the latency being measured.
if read -r line <"$tmp/progress" 2>/dev/null && test "$line" = "DUMP 90_wedged.test"; then
fired=$((SECONDS - start))
marked=1
break
fi
poll_wait 0.1 || sleep 1
done
wait "$pid" || rc=$?
total=$((SECONDS - start))
# Missed between two polls: the whole run then bounds the latency from above.
test -n "$fired" || fired=$total
}
# Retried once, because a dump short enough to fall between two polls is a race, not a
# regression; announcing after the dump loses both attempts.
run_wedged 5
test -n "$marked" || run_wedged 5
test -n "$marked" ||
fail "the announcement was never seen while the guard ran, so its latency is unknown"
# Announced when the dump starts, which runs for minutes: a suite watchdog
# reading that log would take the silence for a wedge and kill the step.
@@ -49,8 +81,8 @@ grep -qx 'DUMP 90_wedged.test' "$tmp/progress" ||
test "$rc" -eq 124 || fail "wedged test reported $rc, want 124"
# Never before the budget, or a slow-but-healthy test would be killed too.
test "$elapsed" -ge 5 || fail "the guard fired early (${elapsed}s of a 5s budget)"
test "$elapsed" -lt 30 || fail "the guard fired late (${elapsed}s)"
test "$fired" -ge 5 || fail "the guard fired early (${fired}s of a 5s budget)"
test "$fired" -lt 30 || fail "the guard fired late (${fired}s)"
grep -q 'marker: the wedged test started' "$out" || fail "the test's own output was lost"
# The header, not a bare name: the process list quotes the test's path too, so a
# wrapper that named nothing would still match that.
@@ -58,6 +90,20 @@ grep -q '^===== TIMEOUT: 90_wedged.test exceeded' "$out" ||
fail "the diagnostics do not name the test"
grep -q "own process tree" "$out" || fail "no process list in the diagnostics"
# Announced BEFORE the dump, not merely at some point during it: the watchdog reading
# that log takes the silence of a dump for a wedge. Only a slow dump tells the two
# orderings apart, and the dump's own ps is what this makes slow.
if ! is_windows; then
printf '#!/bin/sh\nsleep 3\nexec %s "$@"\n' "$(command -v ps)" >"$shim/ps"
chmod +x "$shim/ps"
run_wedged 5 "PATH=$shim:$PATH"
rm -f "$shim/ps" # before the starve shim shares this directory
test "$rc" -eq 124 || fail "the guard reported $rc under a slow dump, want 124"
test -n "$marked" || fail "no announcement under a slow dump"
test "$((total - fired))" -ge 2 ||
fail "announced with the dump (${fired}s of ${total}s), not before it"
fi
# The budget is read, not hard-coded: well under it, the same shape survives.
printf 'sleep 3\necho "slow but healthy"\n' >"$tmp/92_slow.test"
rc=0
@@ -78,15 +124,11 @@ sleep 1
# the loop it stretches is the same one poll_wait's fd tick drives.
(
starve_sleep "$shim" 4 || fail "could not install the slow sleep"
start=$SECONDS
rc=0
HTTRACK_POLL_SLEEP=1 HTTRACK_TEST_TIMEOUT=1 \
bash "$driver" "$tmp/90_wedged.test" >"$out" 2>&1 || rc=$?
elapsed=$((SECONDS - start))
run_wedged 1 HTTRACK_POLL_SLEEP=1
test "$rc" -eq 124 || fail "starved guard reported $rc, want 124"
# Generous: the diagnostics dump runs inside this window too.
test "$elapsed" -lt 25 ||
fail "budget counted polls, not seconds: ${elapsed}s for a 1s budget"
# 10 stretched polls would be 40s; a handful of them is the whole margin here.
test "$fired" -lt 25 ||
fail "budget counted polls, not seconds: ${fired}s for a 1s budget"
)
# --- exit status and output of a healthy test pass straight through ----------
@@ -127,6 +169,84 @@ saw_budget 45 "an explicit budget"
HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/95_budget.test" >"$out" 2>&1
saw_budget 0 "a disabled guard"
# --- a test may raise the budget, never lower it ----------------------------
# 269's header sweep is n^2 compiles, real work that outlasts the wedge budget on an
# emulated host; anything else asking would be disarming the guard.
raiser() { # raiser <asked for>
# shellcheck disable=SC2016 # the fixture has to read the variable, not us
printf '# TEST_TIMEOUT_AT_LEAST: %s\necho "budget=${HTTRACK_TEST_TIMEOUT-unset}"\n' \
"$1" >"$tmp/97_raise.test"
}
# 0900 must not read as octal, and a value past intmax must not reach `test`, which
# errors on it rather than comparing and would leave the guard unarmed.
for want in 900:900 5:600 garbage:600 0900:900 99999999999999999999:600 ' 900':600; do
raiser "${want%%:*}"
HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1
saw_budget "${want##*:}" "a test asking for '${want%%:*}'"
done
raiser 900
HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1
saw_budget 0 "a raise under a disabled guard"
# Read from the header only, or a test's own data would be one: 151 writes this line.
window() { # window <lines of padding> <budget wanted>
{
i=0
while test "$i" -lt "$1"; do
i=$((i + 1))
echo "# pad $i"
done
cat "$tmp/97_raise.test"
} >"$tmp/97_deep.test"
HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/97_deep.test" >"$out" 2>&1
saw_budget "$2" "a raise on line $(($1 + 1))"
}
raiser 900
window 39 900 # the last line the header reaches
window 40 600
# Enforced, not merely exported: the number the guard uses is the one it kills on.
printf '# TEST_TIMEOUT_AT_LEAST: 900\nsleep 4\necho "outlived the default"\n' \
>"$tmp/97_raise.test"
rc=0
HTTRACK_TEST_TIMEOUT=2 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1 || rc=$?
test "$rc" -eq 0 || fail "a 4s test that raised the budget to 900 reported $rc"
grep -q 'outlived the default' "$out" || fail "the raised budget killed the test anyway"
printf '# TEST_TIMEOUT_AT_LEAST: 1\nsleep 3\necho "not shrunk"\n' >"$tmp/97_raise.test"
rc=0
HTTRACK_TEST_TIMEOUT=600 bash "$driver" "$tmp/97_raise.test" >"$out" 2>&1 || rc=$?
test "$rc" -eq 0 || fail "a 3s test asking for a 1s budget reported $rc"
grep -q 'not shrunk' "$out" || fail "the header shrank the budget and killed the test"
# --- one budget parser, and what is left of it ------------------------------
secs() { # secs <value in the environment> <seconds it must read as>
local got
got=$(HTTRACK_TEST_TIMEOUT=$1 bash -c '. "$1"; budget_secs' _ "${testdir}/testlib.sh" 2>&1)
test "$got" = "$2" || fail "budget_secs read '$1' as '$got', want $2"
}
secs 45 45
secs 0900 900 # decimal, or $((...)) and test disagree on the same string
secs garbage 600
secs 99999999999999999999 600 # past intmax, where test errors instead of comparing
secs 0 0
# budget_left hands a child what is left of it, and keeps 0 meaning "no guard".
# shellcheck disable=SC2016 # the fixture has to call the helper, not us
printf '. "%s"\nsleep 2\necho "left=$(budget_left) at=$SECONDS"\n' "${testdir}/testlib.sh" \
>"$tmp/98_left.test"
# Exact, against the clock the child itself read: a tolerance would pass a wrong epoch.
HTTRACK_TEST_TIMEOUT=60 bash "$driver" "$tmp/98_left.test" >"$out" 2>&1
got=$(sed -n 's/^left=\([0-9][0-9]*\) .*/\1/p' "$out")
at=$(sed -n 's/^left=[0-9][0-9]* at=\([0-9][0-9]*\)$/\1/p' "$out")
case "$got$at" in '' | *[!0-9]*) fail "a 60s budget printed '$(cat "$out")'" ;; esac
test "$got" -eq "$((60 - at))" ||
fail "a 60s budget left $got with $at gone, want $((60 - at))"
HTTRACK_TEST_TIMEOUT=0 bash "$driver" "$tmp/98_left.test" >"$out" 2>&1
grep -q '^left=0 ' "$out" || fail "a disabled guard left '$(cat "$out")', want 0"
# Never 0 on an exhausted budget: a child would read that as the guard being off.
HTTRACK_TEST_TIMEOUT=1 bash -c '. "$1"; sleep 2; echo "left=$(budget_left)"' \
_ "${testdir}/testlib.sh" >"$out" 2>&1
grep -qx 'left=1' "$out" || fail "an exhausted budget left '$(cat "$out")', want 1"
# --- a test too slow to finish skips instead of being killed ----------------
# hppa spends ~150s on one configure run, and 124 takes the build down where 77
# does not.

View File

@@ -8,6 +8,8 @@ set -euo pipefail
# shellcheck source=tests/testlib.sh
. "$(dirname "$0")/testlib.sh"
# shellcheck source=tests/proclib.sh
. "$(dirname "$0")/proclib.sh"
sh=${BASH_SHELL:-}
test -n "$sh" || {
@@ -51,11 +53,21 @@ chmod 755 "$tmp/fakebin/bash"
mkfifo "$tmp/fifo"
chmod 755 "$tmp/fifo"
# Sampled rather than polled per second: the size read is a fork, and an emulated
# host pays for it. SILENCE clears the slowest single configure probe there.
SAMPLE=5
SILENCE=${HTTRACK_CONFIGURE_SILENCE:-120}
RESERVE=15 # what killing the run and skipping still needs of the budget
n=0
cases=16 # reject/accept calls below; pinned again once they have all run
status=0
log=
rundir=
took=0
# What run() launches, so the checks below can hand it a child that hangs or one that
# only crawls; nothing else may override it.
configure_cmd=(bash "$tmp/src/configure" --disable-https)
run() { # run <label> <env argument>...
local label=$1 began=$SECONDS
shift
@@ -65,23 +77,56 @@ run() { # run <label> <env argument>...
status=0
# Capped: configure executes the candidate, and a hang wedges "make check" with no output
# at all. Polled, not a backgrounded "sleep" watchdog, which outlives the run it guards.
(cd "$rundir" && env "$@" bash "$tmp/src/configure" --disable-https) \
local had_m=
case "$-" in *m*) had_m=1 ;; esac
# Own process group, so the kills below reach what configure spawned: bash 3.2 keeps
# the subshell it runs in, and killing that alone leaves the child running (macOS).
set -m
(cd "$rundir" && env "$@" "${configure_cmd[@]}") \
>"$rundir/log" 2>&1 &
local pid=$! waited=0
while test "$waited" -lt 300 && kill -0 "$pid" 2>/dev/null; do
local pid=$! waited=0 quiet=0 size=0 now left
test -n "$had_m" || set +m
# A hang is silence, not slowness: configure writes a line per probe,
# but hppa's emulated run can take longer overall than a runner's whole budget (#1146).
while kill -0 "$pid" 2>/dev/null; do
sleep 1
waited=$((waited + 1))
test "$((waited % SAMPLE))" -eq 0 || continue
now=$(wc -c <"$rundir/log")
if test "$now" -gt "$size"; then
size=$now
quiet=0
else
quiet=$((quiet + SAMPLE))
fi
test "$quiet" -lt "$SILENCE" || {
kill_tree "$pid"
echo "configure wrote nothing for ${quiet}s of ${waited}s for $label" >&2
tail -5 "$rundir/log" >&2
exit 1
}
# Still writing but out of time: skip, where the harness would kill the whole test
# and take the build down with it. Only while writing, and only with a full silence
# window still affordable, or a hang would reach this before the check above fires
# and a wedge would report a skip. 0 is the guard off.
left=$(budget_left)
if test "$quiet" -eq 0 && test "$left" -ne 0 &&
test "$left" -le "$((RESERVE + SILENCE))"; then
kill_tree "$pid"
echo "$label was still configuring ${waited}s in and the budget is out; skipping" >&2
exit 77
fi
done
if kill -0 "$pid" 2>/dev/null; then
kill -9 "$pid" 2>/dev/null
echo "configure did not return within ${waited}s for $label" >&2
tail -5 "$rundir/log" >&2
exit 1
fi
wait "$pid" || status=$?
log=$(cat "$rundir/log")
took=$((SECONDS - began))
echo "run $n ($label): exit $status"
skip_if_out_of_budget "$((cases - n))" "$((SECONDS - began))"
}
# Pace here rather than in run(), which returns with the answer still unjudged: a
# skip between the two would bury a configure that answered wrongly.
paced() {
skip_if_out_of_budget "$((cases - n))" "$took"
}
reject() { # reject <label> <expected message> <env argument>...
@@ -97,6 +142,7 @@ reject() { # reject <label> <expected message> <env argument>...
tail -5 <<<"$log" >&2
exit 1
}
paced
}
# accept <label> <expected $(BASH_SHELL), "" for any> <expected message, "" for none> <env argument>...
@@ -125,8 +171,69 @@ accept() {
exit 1
}
fi
paced
}
# --- what run() does to a child that hangs, and to one that is merely slow -------
# Driven through configure_cmd, since the real configure can do neither on demand.
probe() { # probe <run number> <seconds of budget left> <command>...
local want_n=$1 left=$2 rc=0
shift 2
(
# shellcheck disable=SC2030 # the isolation is the point: the real count is next door
n=$want_n SAMPLE=1 SILENCE=2
# shellcheck disable=SC2030,SC2031 # likewise: the budget here is the probe's own
export HTTRACK_TEST_TIMEOUT=$((SECONDS + left))
configure_cmd=("$@")
run probe
) >"$tmp/probe.log" 2>&1 || rc=$?
echo "$rc"
}
# A wedge must fail even with the budget gone, or #922 comes back as a skip.
rc=$(probe 90 6 sleep 999)
test "$rc" -eq 1 || fail "a silent configure with 6s of budget reported $rc, want 1"
grep -q 'wrote nothing' "$tmp/probe.log" || fail "the hang was not named: $(cat "$tmp/probe.log")"
# Slow but talking is the emulated buildd, and a skip there beats the harness kill.
rc=$(probe 91 6 bash -c 'while :; do echo tick; sleep 1; done')
test "$rc" -eq 77 || fail "a slow but writing configure with 6s of budget reported $rc, want 77"
# The kill has to reach what configure spawned. bash 3.2 keeps the subshell around the
# child, so killing that alone leaves a live configure behind: it outlives "make check"
# and holds the CI step open to its own timeout, with the suite reporting no failure.
rc=$(probe 92 6 bash -c 'sleep 987 & wait')
test "$rc" -eq 1 || fail "a silent configure with a child of its own reported $rc, want 1"
sleep 1
! ps_snapshot | grep -q '[s]leep 987' || fail "the killed run left its child running"
# The pacer must not fire before the case is judged: run() returns with the verdict
# still unread, and a skip there would bury a configure that answered wrongly. Through
# the real run(), since a stub cannot see a pacer left inside the one it replaced.
verdict() { # verdict <accept|reject> <run number> <status the child exits with>
local rc=0
(
# shellcheck disable=SC2030,SC2031 # the isolation is the point: the real run is next door
# Cases still to come, or the pacer this is looking for would decline to fire.
n=$2 cases=$(($2 + 5)) SAMPLE=1
# Spent by the time the run ends, so a pacer anywhere after it would fire.
# shellcheck disable=SC2030,SC2031 # likewise: the budget here is the probe's own
export HTTRACK_TEST_TIMEOUT=$((SECONDS + 4))
configure_cmd=(bash -c "sleep 3; exit $3")
# Their arities differ, and an extra argument would reach run() as an env
# assignment: the child would then fail to exec and answer the wrong question.
case "$1" in
accept) accept probe-verdict '' '' ;;
*) reject probe-verdict '' ;;
esac
) >/dev/null 2>&1 || rc=$?
test "$rc" -eq 1 || fail "$1 of a wrong answer with the budget spent reported $rc, want 1"
}
verdict accept 80 1 # configure rejected what it must accept
verdict reject 81 0 # configure accepted what it must reject
# The probes ran in subshells, so the real cases below start from a clean count.
cases=16
n=0
status=0
log=
took=0
# The four that configure to completion run first. A reject stops at the
# BASH_SHELL check and costs a fraction of one, and the pacer projects the step it
# just timed: behind the cheap ones it read far too low and 151 met the harness

View File

@@ -188,20 +188,29 @@ rc=0
kill_pid() { echo "DIRECT $1" >>"$rec"; }
# shellcheck disable=SC2317
kill_tree() {
echo "TREE $1" >>"$rec"
echo "TREE $*" >>"$rec"
exit 9
}
# The suite's own pid has no /proc entry here, so the capture answers what a
# POSIX box answers: empty, and the kill goes on unguarded.
# shellcheck disable=SC2317
win_capture() {
echo "CAPTURE $1" >>"$rec"
WIN_PID=4242 WIN_IMAGE=bash.exe
}
hb_depth=$BASH_SUBSHELL
ci_suite_heartbeat 960 360 "$progress" 900 4242 >"$tmp/hedge" 2>&1
) || rc=$?
test "$rc" -eq 9 || fail "the tree kill never fired: watchdog returned $rc"
test "$(sed -n 1p "$rec")" = "DIRECT 777" ||
fail "the reporter was not killed ahead of the suite: $(tr '\n' '/' <"$rec")"
test "$(sed -n 2p "$rec")" = "DIRECT 4242" ||
test "$(sed -n 2p "$rec")" = "CAPTURE 4242" ||
fail "the winpid was not read before the target was signalled: $(tr '\n' '/' <"$rec")"
test "$(sed -n 3p "$rec")" = "DIRECT 4242" ||
fail "the target was not signalled directly ahead of the tree walk: $(tr '\n' '/' <"$rec")"
test "$(sed -n 3p "$rec")" = "TREE 4242" ||
fail "the tree was not killed after the direct signal: $(tr '\n' '/' <"$rec")"
test "$(sed -n '$=' "$rec")" -eq 3 || fail "extra kills: $(tr '\n' '/' <"$rec")"
test "$(sed -n 4p "$rec")" = "TREE 4242 4242 bash.exe" ||
fail "the tree kill did not carry what was captured: $(tr '\n' '/' <"$rec")"
test "$(sed -n '$=' "$rec")" -eq 4 || fail "extra kills: $(tr '\n' '/' <"$rec")"
test ! -e "$tmp/forked" || fail "the clock was read through a subshell, a fork a starved box cannot spare"

View File

@@ -34,7 +34,11 @@ cleanup_push rm -rf "$tmp"
"$nm" --defined-only "$lib" >"$tmp/all.raw" 2>/dev/null ||
skip "$nm cannot read the symbol table of $lib"
awk 'NF >= 3 { print $3 }' "$tmp/exported.raw" | sort -u >"$tmp/exported"
awk 'NF >= 3 { print $3 }' "$tmp/all.raw" | sort -u >"$tmp/all"
# LTO and the optimizer's clones rename a local to "name.lto_priv.0", which
# stops matching the header identifier and empties the candidate list.
clone_alt='lto_priv|constprop|isra|part|cold|llvm'
strip_clones="s/(\\.($clone_alt)(\\.[0-9]+)*)+\$//"
awk 'NF >= 3 { print $3 }' "$tmp/all.raw" | sed -E "$strip_clones" | sort -u >"$tmp/all"
comm -23 "$tmp/all" "$tmp/exported" >"$tmp/hidden"
n_exported=$(wc -l <"$tmp/exported")
n_hidden=$(wc -l <"$tmp/hidden")
@@ -89,7 +93,26 @@ env MAKEFLAGS= MFLAGS= "$make" -C "$abs_top_builddir/src" install-DevIncludesDAT
# A synthetic leak the loop must report: every real candidate is a static defined
# in the header, which links from the probe's own copy and so proves nothing.
canary=zzz-canary.h
printf 'extern void %s(void);\n' "$neg" >"$tmp/include/httrack/$canary"
canaries=("$neg")
# One name per clone suffix, taken from the raw table: a strip that drops a
# suffix fails here instead of quietly shedding the candidates carrying it.
# Listed again on purpose; sharing $clone_alt would let one edit disarm both.
awk 'NF >= 3 && $3 !~ /\./ { print $3 }' "$tmp/all.raw" | sort -u >"$tmp/plain"
for suffix in lto_priv constprop isra part cold llvm; do
awk -v s="$suffix" 'NF >= 3 && $3 ~ "\\." s "([.$]|$)" { print $3 }' "$tmp/all.raw" |
sed -E 's/(\.[A-Za-z_][A-Za-z0-9_]*(\.[0-9]+)*)+$//' | sort -u >"$tmp/cloned"
sym=$(comm -23 <(comm -23 "$tmp/cloned" "$tmp/plain") "$tmp/exported" |
awk '/^[A-Za-z_][A-Za-z0-9_]*$/ { print; exit }')
if [ -n "$sym" ]; then
canaries+=("$sym")
fi
done
mapfile -t canaries < <(printf '%s\n' "${canaries[@]}" | sort -u)
for sym in "${canaries[@]}"; do
printf 'extern void %s(void);\n' "$sym"
done >"$tmp/include/httrack/$canary"
headers=("$tmp/include/httrack"/*.h)
[ "${#headers[@]}" -ge 10 ] || fail "only ${#headers[@]} headers installed, the list cannot be right"
@@ -113,7 +136,7 @@ ours {
EOF
probed=0
leaks=""
leaks=()
for h in "${headers[@]}"; do
b=$(basename "$h")
# config.h first, as a consumer must: it is what turns HTS_USEOPENSSL on, and
@@ -136,16 +159,26 @@ for h in "${headers[@]}"; do
"$tmp/probe.c" 2>/dev/null || continue
probed=$((probed + 1))
"${cc_argv[@]}" -w -o "$tmp/probe" "$tmp/probe.o" "$lib" 2>/dev/null ||
leaks="$leaks $b:$sym"
leaks+=("$b:$sym")
done < <(comm -12 "$tmp/hidden" "$tmp/ids")
done
echo "linked $probed reachable symbol(s) from ${#headers[@]} installed headers" \
"($n_hidden hidden, $n_exported exported)"
[ "$probed" -ge 5 ] || fail "only $probed symbols reached the link probe, the candidate list is broken"
[ "${leaks#* "$canary":"$neg"}" != "$leaks" ] ||
fail "the synthetic $canary:$neg leak went unreported, the candidate list is broken"
leaks=${leaks/ "$canary":"$neg"/}
[ -z "$leaks" ] || fail "installed headers declare symbols $lib does not export:$leaks"
for sym in "${canaries[@]}"; do
case " ${leaks[*]} " in
*" $canary:$sym "*) ;;
*) fail "the synthetic $canary:$sym leak went unreported, the candidate list is broken" ;;
esac
done
real=()
for leak in "${leaks[@]}"; do
case $leak in
"$canary":*) ;;
*) real+=("$leak") ;;
esac
done
[ "${#real[@]}" -eq 0 ] || fail "installed headers declare symbols $lib does not export: ${real[*]}"
exit 0

View File

@@ -214,8 +214,9 @@ def perms_of(wf, job):
WANT_ENV = {
"WATCHDOG_TOKEN": "${{ secrets.GITHUB_TOKEN }}",
"WATCHDOG_REPO": "${{ github.repository }}",
# The merge commit, which no PR checks UI reads.
"WATCHDOG_SHA": "${{ github.sha }}",
# The PR head: statuses on the merge commit are GC'd, and they are the only
# trace a lost runner leaves (#1228).
"WATCHDOG_SHA": "${{ github.event.pull_request.head.sha || github.sha }}",
}
def audit(wf):
@@ -264,7 +265,7 @@ def mutate(wf, kind):
elif kind == "token":
suite_steps(wf)[0]["env"]["WATCHDOG_TOKEN"] = "${{ secrets.WATCHDOG_PAT }}"
elif kind == "sha":
suite_steps(wf)[0]["env"]["WATCHDOG_SHA"] = "${{ github.event.pull_request.head.sha }}"
suite_steps(wf)[0]["env"]["WATCHDOG_SHA"] = "${{ github.sha }}"
elif kind == "context":
suite_steps(wf)[0]["env"]["WATCHDOG_CONTEXT"] = "windows-suite"
elif kind == "url":
@@ -555,6 +556,12 @@ backoff_leg() {
echo "$calls calls and $lines log lines against an API rejecting every one: nothing backs off"
return 1
fi
# Every attempt here is refused, so each one after the first must say how many
# went missing: x= is what separates a stopped box from a network that healed.
grep -q ' x=[1-9]' "$posts" || {
echo "no posted status counted the failures before it: $(cat "$posts")"
return 1
}
}
fullstdout_leg() {
@@ -609,9 +616,18 @@ for psrun in "${psruns[@]}"; do
# shellcheck disable=SC2016
mutate tail-always-ok 's/\$r = @{ Ok = \$false;/$r = @{ Ok = $true;/' 'reads as one that was read'
mutate counters-empty "s/return (\$f -join ' ')/return ''/" 'counters are not key=value'
# shellcheck disable=SC2016
mutate lag-not-measured "s/'l={0}' -f \$LagMs/'l={0}' -f 0/" 'loop lag is not what the caller measured'
# shellcheck disable=SC2016
mutate failed-not-reported "s/'x={0}' -f \$Failed/'x={0}' -f 0/" 'failed-post count is not what the caller'
# shellcheck disable=SC2016
mutate tcp-total-not-delta 's/(\$Cur\[0\] - \$Prev\[0\])/($Cur[0])/' 'total was reported where the delta'
# shellcheck disable=SC2016
mutate lag-not-a-peak 's/if (\$lag -gt \$Peak) { return \$lag }/if ($false) { return $lag }/' \
'overshot by 200ms'
# The production cadence: no leg below runs without a schedule of its own.
# shellcheck disable=SC2016
mutate default-cadence 's/\[int\]\$IntervalSeconds = 30/[int]$IntervalSeconds = 3000/' \
mutate default-cadence 's/\[int\]\$IntervalSeconds = 15/[int]$IntervalSeconds = 1500/' \
'default status cadence'
# shellcheck disable=SC2016
mutate default-poll 's/\[int\]\$PollSeconds = 5/[int]$PollSeconds = 50/' 'default poll'
@@ -714,6 +730,10 @@ for psrun in "${psruns[@]}"; do
throttle 'a landed post throttles the next'
# shellcheck disable=SC2016
mutate_leg backoff-never-skips 's/if (\$skip -gt 0)/if ($false)/' backoff 'nothing backs off'
# shellcheck disable=SC2016
mutate_leg failures-not-counted \
's/if (\$ok) { \$failed = 0; \$lagMax = 0 } else { \$failed++ }/$failed = 0/' \
backoff 'counted the failures before it'
if test "$devfull" -eq 1; then
mutate_leg log-write-fatal 's/try { \(Write-Host .*\) } catch { }/\1/' \
fullstdout 'took the loop with it'

View File

@@ -72,6 +72,45 @@ kill_tree 99
test "$(cat "$tmp/killed")" = '/F /T /PID 4242' || fail "kill_tree with a winpid ran: $(cat "$tmp/killed")"
win_pid() { :; }
# The image read while the target was alive: a freed winpid can already be a
# stranger's, and /T would take its children too (#1228).
: >"$tmp/killed"
kill_tree 99 4242 PROXYTRACK.EXE
test "$(cat "$tmp/killed")" = '/F /T /PID 4242' ||
fail "a verified tree kill did not run: $(cat "$tmp/killed")"
: >"$tmp/killed"
out=$(kill_tree 99 4242 python.exe)
test ! -s "$tmp/killed" || fail "pid 4242 was killed as a python.exe: $(cat "$tmp/killed")"
grep -q '::warning::pid 4242 no longer runs python.exe' <<<"$out" ||
fail "the skipped kill was not reported: $out"
# Gone from the table entirely, which is what a freed winpid usually looks like.
: >"$tmp/killed"
kill_tree 99 4343 proxytrack.exe >/dev/null
test ! -s "$tmp/killed" || fail "a pid tasklist does not list was killed: $(cat "$tmp/killed")"
# A stranger's pid leaves the caller with no target, so the serial runner's last
# resort still applies: skipping it too would leave the engines running.
: >"$tmp/killed"
HTTRACK_EXCLUSIVE_HOST=1 kill_tree 99 4242 python.exe >/dev/null
got=$(sort "$tmp/killed")
test "$got" = "$want" || fail "an unverified pid skipped the last-resort sweep: $got"
# Graded on the order, since only a winpid read while the target lived names it.
: >"$tmp/killed"
: >"$tmp/order"
kill() { echo "kill $*" >>"$tmp/order"; }
win_capture() {
echo "capture $*" >>"$tmp/order"
WIN_PID=4242 WIN_IMAGE=proxytrack.exe
}
# The pid is fictional, and reap_bounded polls it through the kill stub above.
reap_bounded() { :; }
stop_server 99
got=$(tr '\n' ' ' <"$tmp/order")
test "$got" = 'capture 99 kill 99 ' || fail "stop_server did not capture before signalling: $got"
test "$(cat "$tmp/killed")" = '/F /T /PID 4242' ||
fail "stop_server did not tree-kill what it captured: $(cat "$tmp/killed")"
unset -f kill win_capture reap_bounded
: >"$tmp/killed"
out=$(reap_leftover_processes 99_probe.test)
grep -q '99_probe.test left processes behind' <<<"$out" || fail "the leak was not attributed: $out"

View File

@@ -6,6 +6,10 @@
# a break needing three headers, or a macro the consumer defined first, is out
# of reach here. The sweep is shared with the MSVC job, which has no automake to
# install with and so stages the same list out of DevIncludes_DATA (#1153).
#
# n^2 compiles is real work, not a wedge: emulated, it needs more than the suite's
# default budget, and the sweep paces itself against whatever is left of this one.
# TEST_TIMEOUT_AT_LEAST: 900
set -euo pipefail
@@ -69,7 +73,10 @@ done
sweep_argv=(--headers-dir "$tmp/include/httrack" --cc "${CC:-cc}" --cxx "$cxx")
[ "${#cpp_argv[@]}" -eq 0 ] || sweep_argv+=(-- "${cpp_argv[@]}")
bash "$testdir/install-headers-sweep.sh" "${sweep_argv[@]}" ||
fail "installed headers do not survive every include order"
rc=0
bash "$testdir/install-headers-sweep.sh" --budget "$(budget_left)" "${sweep_argv[@]}" || rc=$?
# 77 is the sweep giving up on a host too slow to finish it, not a broken header.
[ "$rc" -ne 77 ] || exit 77
[ "$rc" -eq 0 ] || fail "installed headers do not survive every include order"
exit 0

View File

@@ -0,0 +1,243 @@
#!/bin/bash
#
# An abandoned htsserver has to stop on its own: it holds its payload open, and
# on macOS that payload is a disk image the user then cannot eject.
set -euo pipefail
# shellcheck source=tests/webhttracklib.sh
. "$(dirname "$0")/webhttracklib.sh"
htsserver_require
# Short enough to sit in the suite, long enough that a loaded parallel run
# cannot starve a ping past it. The server derives the leave grace from it.
timeout=10
grace=2
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_life.XXXXXX") || fail "no tmpdir"
pinger=
csrv=
sleeper=
cleanup() {
htsserver_cleanup
for p in "${pinger}" "${csrv}" "${sleeper}"; do
test -z "${p}" || kill -9 "${p}" 2>/dev/null || true
done
wait "${pinger}" "${csrv}" "${sleeper}" 2>/dev/null || true
rm -rf "${work}"
}
cleanup_push cleanup
export HOME="${work}"
# Each case owns a server, so save its handle: the library keeps only the last.
# $1, when given, is the pid to name as the launcher instead of this shell.
start_case() {
htsserver_start --home "${work}" -- \
--ppid "${1:-$$}" --ping-timeout "${timeout}"
test -n "${HTS_PID}" || skip "this platform announces no server pid"
}
get() { "${HTS_PYTHON}" "${testdir}/httpclient.py" --port "$1" --path "$2"; }
ping() { get "$1" "/ping?w=$2&t=${RANDOM}${3:+&$3}" >/dev/null; }
# The session id every UI page carries, which the server demands of a farewell.
page_sid() {
firstline "$(get "$1" /server/index.html |
sed -n 's/.*name="sid" value="\([0-9a-f]*\)".*/\1/p')"
}
# A farewell as the page sends one: a POST holding that id.
bye() {
"${HTS_PYTHON}" "${testdir}/httpclient.py" --port "$1" \
--path "/ping?w=$2&t=${RANDOM}&e=bye" --field "sid=$3" >/dev/null
}
# Wait up to $2 seconds for $1 to go away.
died_within() {
local pid=$1 limit=$2 start=$SECONDS
while kill -0 "${pid}" 2>/dev/null; do
test "$((SECONDS - start))" -lt "${limit}" || return 1
poll_wait 0.2
done
}
printf '[an abandoned WebHTTrack server stops on its own] ..\t'
# The page and the server agree on one wire format, and only the page half runs
# in a browser: a client that stopped naming its window, or spelled the farewell
# differently, would leave every case below testing the server against itself.
js="${HTS_DISTDIR}/html/server/ping.js"
grep -qF '"/ping?w=" + PING_WINDOW' "${js}" || fail "ping.js stopped naming its window"
grep -qF 'ping_url("e=bye")' "${js}" || fail "ping.js stopped saying goodbye"
grep -qF '"sid=" + encodeURIComponent(sid)' "${js}" ||
fail "ping.js stopped signing its goodbye, which the server then ignores"
# 1. The heartbeat must never be answered from a cache: a reply the browser
# reuses is one the server never sees, and silence is what kills it.
start_case
bye_pid=${HTS_PID}
bye_port=${HTS_PORT}
sid=$(page_sid "${bye_port}")
test "${#sid}" -eq 32 || fail "no session id on the wizard's first page"
reply=$(get "${bye_port}" '/ping?w=w1')
grep -q '^HTTP/1\.0 200 ' <<<"${reply}" || fail "no pong: $(head -1 <<<"${reply}")"
grep -qi '^Cache-Control:.*no-cache' <<<"${reply}" ||
fail "the heartbeat is cacheable: ${reply}"
# 2. Closing one of two windows must not end a session the other is still using.
# A hidden tab has its timers throttled to as little as one wake-up a minute, so
# this cannot rest on the survivor answering inside the grace.
ping "${bye_port}" w2
bye "${bye_port}" w1 "${sid}"
sleep $((grace + 2))
kill -0 "${bye_pid}" 2>/dev/null ||
fail "closing one window ended a session another window still had open"
# 3. An unsigned farewell is not one: /ping is a GET, so it clears neither the
# session-id nor the Origin gate, and any local process or visited page can send
# it. Only a heartbeat may go unproven, and a heartbeat only extends a life.
ping "${bye_port}" w2 e=bye
sleep $((grace + 2))
kill -0 "${bye_pid}" 2>/dev/null ||
fail "an unauthenticated goodbye ended the session"
# 4. A flood of window ids must not push a real window out of the table: with
# the last real one evicted, saying goodbye to the flood would end the session.
# One more than the table holds, so the refusal itself is exercised.
for i in $(seq 0 16); do
ping "${bye_port}" "f${i}"
done
for i in $(seq 0 16); do
bye "${bye_port}" "f${i}" "${sid}"
done
sleep $((grace + 2))
kill -0 "${bye_pid}" 2>/dev/null ||
fail "a flood of window ids evicted the real window and ended the session"
# 5. The last window leaving takes the server with it, well inside the idle
# timeout that would otherwise apply.
bye "${bye_port}" w2 "${sid}"
bye_at=${SECONDS}
died_within "${bye_pid}" $((timeout - 1)) ||
fail "the server outlived its last window by $((SECONDS - bye_at))s"
test "$((SECONDS - bye_at))" -le $((grace + 3)) ||
fail "the last window took $((SECONDS - bye_at))s, past the ${grace}s grace"
# The four cases below share one wait, so they cost one timeout, not four.
# 6. A window that stops pinging without a goodbye has crashed with its browser.
start_case
lost_pid=${HTS_PID}
ping "${HTS_PORT}" w1
lost_at=${SECONDS}
# 7. A window that keeps pinging keeps its server, past that same deadline.
start_case
live_pid=${HTS_PID}
live_port=${HTS_PORT}
live_at=${SECONDS}
pingfail="${work}/pingfail"
(
while :; do
ping "${live_port}" w1 || echo failed >>"${pingfail}"
poll_wait 0.5
done
) &
pinger=$!
# 8. A client that never pings at all is a browser too old for the heartbeat, or
# one with scripting off, and may be a user reading the page: only the
# launcher's death may end that session, and nothing here kills this shell.
start_case
quiet_pid=${HTS_PID}
quiet_at=${SECONDS}
get "${HTS_PORT}" /server/index.html >/dev/null
# 9. ..and when that launcher does die, the same session ends: it is the only
# signal such a browser produces. A process of our own stands in for it.
sleep 600 &
sleeper=$!
start_case "${sleeper}"
legacy_pid=${HTS_PID}
get "${HTS_PORT}" /server/index.html >/dev/null
legacy_at=${SECONDS}
kill "${sleeper}" 2>/dev/null || true
wait "${sleeper}" 2>/dev/null || true # absorb bash's async "Terminated" notice
sleeper=
# The crashed window must outlive its setup, or the floor asserted below could
# be met by the time these four servers took to start.
kill -0 "${lost_pid}" 2>/dev/null ||
fail "the idle timeout fired during setup, $((SECONDS - lost_at))s in"
died_within "${lost_pid}" $((timeout * 3)) ||
fail "a server whose window stopped pinging survived $((SECONDS - lost_at))s"
test "$((SECONDS - lost_at))" -ge $((timeout - 2)) ||
fail "the idle timeout fired after $((SECONDS - lost_at))s, under ${timeout}s"
died_within "${legacy_pid}" $((timeout * 3)) ||
fail "a server outlived its launcher by $((SECONDS - legacy_at))s"
# Both survivors started after the crashed one, so its death is too early to
# judge them by. Two timeouts, not one: a window whose refresh stopped working
# would still be inside its first.
while test "$((SECONDS - live_at))" -le $((timeout * 2)) ||
test "$((SECONDS - quiet_at))" -le $((timeout * 2)); do
poll_wait 0.5
done
# Aliveness first: a server that dies also fails the pings aimed at it, and the
# probe's own failure would then be the only thing reported.
kill -0 "${live_pid}" 2>/dev/null || fail "a pinged server was killed anyway"
kill -0 "${quiet_pid}" 2>/dev/null ||
fail "a server whose client never pings was killed under a live launcher"
! test -f "${pingfail}" || fail "the probe's own pings failed; nothing was proven"
kill "${pinger}" 2>/dev/null || true
pinger=
# 10. Closing the window while a mirror runs must not take the mirror down: it
# may have hours of crawling behind it. The veto has to lift when the crawl
# ends, or the server it saved becomes immortal.
start_case
crawl_pid=${HTS_PID}
crawl_port=${HTS_PORT}
clog="${work}/content.log"
"${HTS_PYTHON}" "${testdir}/local-server.py" --root "${work}" >"${clog}" 2>&1 &
csrv=$!
cport=$(discover_server_port "${clog}" "${csrv}") || fail "no content server"
sid=$(page_sid "${crawl_port}")
test "${#sid}" -eq 32 || fail "no session id to start a mirror with"
# Its index links one page that sleeps 5s, so the crawl outlasts the grace and
# still ends inside the test.
"${HTS_PYTHON}" "${testdir}/httpclient.py" --port "${crawl_port}" \
--path /step4.html --field "sid=${sid}" --field "path=${work}" \
--field projname=crawl --field winprofile=x --field command_do=start \
--field "command=httrack --quiet --robots=0 http://127.0.0.1:${cport}/abortpurge/index.html -O ${work}/crawl" \
>/dev/null
ping "${crawl_port}" w1
bye "${crawl_port}" w1 "${sid}"
sleep $((grace + 2))
kill -0 "${crawl_pid}" 2>/dev/null ||
fail "the crawling server quit, taking its mirror with it"
# A crawl that never started would leave the veto unexercised, and the survival
# above would prove nothing.
test -d "${work}/crawl/hts-cache" || fail "no mirror ever started: $(cat "${clog}")"
died_within "${crawl_pid}" $((timeout * 3)) ||
fail "the server never left once its mirror had finished"
# 11. The wizard's own Quit button is the other way out, and the only one that
# leaves through smallserver(), which has to report the server it did create.
start_case
quit_pid=${HTS_PID}
sid=$(page_sid "${HTS_PORT}")
test "${#sid}" -eq 32 || fail "no session id to quit with"
"${HTS_PYTHON}" "${testdir}/httpclient.py" --port "${HTS_PORT}" \
--path /server/exit.html --field "sid=${sid}" --field command=quit >/dev/null
died_within "${quit_pid}" "${grace}" || fail "Quit did not stop the server"
! grep -q 'Unable to create the server' "${HTS_LOG}" ||
fail "a clean quit reported a server it could not create: $(cat "${HTS_LOG}")"
htsserver_stop
htsserver_assert_reaped
echo OK

View File

@@ -0,0 +1,16 @@
#!/bin/bash
#
# The default footer carries named fields, so a mirrored page must show the
# crawled URL and date, not a literal "{url}" (what the legacy positional model
# emits for a template it does not understand).
set -euo pipefail
: "${top_srcdir:=..}"
bash "$top_srcdir/tests/local-crawl.sh" \
--files 5 --errors 0 \
--file-matches 'simple/basic.html' \
"<!-- Mirrored from http://127\.0\.0\.1:[0-9]+/simple/basic\.html by HTTrack Website Copier/[^ ]+ \[XR&CO\], [A-Z][a-z]{2}, [0-9]{2} [A-Z][a-z]{2} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2} GMT -->" \
--file-not-matches 'simple/basic.html' '\{(url|date)\}' \
httrack 'BASEURL/simple/basic.html'

View File

@@ -0,0 +1,39 @@
#!/bin/bash
#
# WebHTTrack prefills the engine's default footer, and a new project must get
# the named fields (see HTS_DEFAULT_FOOTER on why a stray "%s" is fatal).
set -euo pipefail
# shellcheck source=tests/webhttracklib.sh
. "$(dirname "$0")/webhttracklib.sh"
htsserver_require
work=$(mktemp -d "${TMPDIR:-/tmp}/webhttrack_footer.XXXXXX") || fail "no tmpdir"
cleanup() {
htsserver_cleanup
rm -rf "${work}"
}
cleanup_push cleanup
printf '[a fresh profile prefills the named-field footer] ..\t'
# An isolated HOME: a stored ~/.httrack.ini would answer with its own footer.
mkdir -p "${work}/websites"
htsserver_start --home "${work}"
reply=$(htsserver_get /server/option6.html)
grep -q '^HTTP/1.[01] 200' <<<"${reply}" || fail "option6.html: ${reply}"
value=$(firstline "$(sed -n 's/.*name="footer" value="\([^"]*\)".*/\1/p' <<<"${reply}")")
test -n "${value}" || fail "option6.html serves no footer field: ${reply}"
case ${value} in
*'%s'*) fail "the prefilled footer is still positional: ${value}" ;;
esac
grep -q '{url}.*{date}' <<<"${value}" || fail "prefilled footer: ${value}"
htsserver_stop
# A leaked server wedges the parallel harness behind a green log.
htsserver_assert_reaped
echo OK

View File

@@ -0,0 +1,50 @@
#!/bin/bash
#
set -euo pipefail
# shellcheck source=tests/testlib.sh
. "$(dirname "$0")/testlib.sh"
# #1117: the self-test asserts the scopes and the patterns; this feeds the pair
# a scope answer emits back through the matcher that authorizes a link.
# The front end enumerates by looping until the engine stops answering. Strip
# the CRs: MSYS hands the engine a text-mode stdout and $(...) eats only the
# trailing newline, so an assertion spanning several lines keeps the interior
# ones. Every other assert_selftest here compares a single line and cannot see
# this.
want="0 download.example.co.uk
1 example.co.uk
2 co.uk"
got=$(httrack -O /dev/null -#test=wizardscope download.example.co.uk/x | tr -d '\r')
test "$got" = "$want" || fail "wizardscope enumeration: got [$got]"
# widening stops before a bare TLD, and an IP literal offers nothing at all
assert_selftest "0 example.com" wizardscope example.com/x
assert_selftest "" wizardscope 192.168.1.1/x
# answer 1000+1 is "example.co.uk and every host below it": two filters, since
# the starred one misses the apex
sub=$(httrack -O /dev/null -#test=wizardfilter 1001 www.example.co.uk /x)
apex=$(httrack -O /dev/null -#test=wizardfilter 1001 www.example.co.uk /x 0 1)
test "$sub" = "+*.example.co.uk/*" || fail "subdomain filter: got [$sub]"
test "$apex" = "+example.co.uk/*" || fail "apex filter: got [$apex]"
for host in www.example.co.uk a.b.example.co.uk; do
assert_selftest "$host/x does match ${sub#+}" filter "${sub#+}" "$host/x"
done
assert_selftest "example.co.uk/x does NOT match ${sub#+}" filter "${sub#+}" example.co.uk/x
assert_selftest "example.co.uk/x does match ${apex#+}" filter "${apex#+}" example.co.uk/x
# neither half may leak past the domain boundary
for bad in notexample.co.uk/x example.co.uk.evil.com/x; do
assert_selftest "$bad does NOT match ${sub#+}" filter "${sub#+}" "$bad"
assert_selftest "$bad does NOT match ${apex#+}" filter "${apex#+}" "$bad"
done
# the exclude range emits the same pair negated
assert_selftest "-*.example.co.uk/*" wizardfilter 2001 www.example.co.uk /x
assert_selftest "wizardscope self-test OK" wizardscope
assert_selftest "wizardfilter self-test OK" wizardfilter

View File

@@ -0,0 +1,16 @@
#!/bin/bash
#
set -euo pipefail
# shellcheck source=tests/testlib.sh
. "$(dirname "$0")/testlib.sh"
# The verdict half of a wizard answer, driven without the interactive callback.
# Both start from the undecided verdict the question is asked in.
assert_selftest "forbidden=1 stop=1 prio=42" wizardverdict -1
assert_selftest "forbidden=0 stop=0 prio=42" wizardverdict 6
# every answer, all three incoming verdicts, both scope ranges
assert_selftest "wizardverdict self-test OK" wizardverdict

View File

@@ -0,0 +1,57 @@
#!/bin/bash
# #1247: with no answer to be had, the wizard's answer loop re-prompted forever.
set -eu
# shellcheck source=tests/crawllib.sh
. "$(dirname "$0")/crawllib.sh"
httrack=$(command -v httrack) || ! echo "could not find httrack" >&2 || exit 1
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_1247.XXXXXX") || exit 1
cleanup() {
rm -rf "$tmpdir"
}
cleanup_push cleanup
# The out-of-scope host is a second local server, so a link wrongly accepted
# leaves a mirrored directory behind instead of failing to connect.
mkdir -p "${tmpdir}/foreign"
printf '<html><body>beyond</body></html>\n' >"${tmpdir}/foreign/a.html"
cp "${tmpdir}/foreign/a.html" "${tmpdir}/foreign/b.html"
local_server_start --root "${tmpdir}/foreign" --log "${tmpdir}/foreign.log"
foreign=$SRV_PORT
# Three links over two host names: an answer refusing only this link asks
# again for the second, and one refusing only this domain for the third.
mkdir -p "${tmpdir}/root/wizardeof"
cat >"${tmpdir}/root/wizardeof/index.html" <<EOF
<html><body>
<a href="http://127.0.0.1:${foreign}/a.html">a</a>
<a href="http://127.0.0.1:${foreign}/b.html">b</a>
<a href="http://localhost:${foreign}/b.html">c</a>
</body></html>
EOF
local_server_start --root "${tmpdir}/root"
# crawl LABEL; the caller redirects the stdin the wizard will read from
crawl() {
local label=$1 rc=0 asked
run_with_timeout 60 "$httrack" -O "${tmpdir}/$label" -W --robots=0 \
--retries=0 "http://127.0.0.1:${SRV_PORT}/wizardeof/index.html" \
>"${tmpdir}/$label.log" 2>&1 || rc=$?
test "$rc" -ne 124 || fail "$label: the wizard never returned"
test "$rc" -eq 0 || fail "$label: the crawl exited $rc"
# asked once and then stopped: refusing this one link would ask again
asked=$(grep -c "beyond this mirror scope" "${tmpdir}/$label.log" || true)
test "$asked" -eq 1 || fail "$label: asked $asked times, expected 1"
test -f "${tmpdir}/$label/127.0.0.1_${SRV_PORT}/wizardeof/index.html" ||
fail "$label: the start page was not mirrored"
test ! -d "${tmpdir}/$label/127.0.0.1_${foreign}" ||
fail "$label: the refused link was mirrored"
}
crawl eof </dev/null
# a closed descriptor sets only ferror, where EOF sets only feof
crawl closed <&-

65
tests/295_wizard-log.test Normal file
View File

@@ -0,0 +1,65 @@
#!/bin/bash
#
# A wizard session must be readable afterwards: one line per answer, carrying
# the raw reply and the filters it inserted, reaches hts-log.txt at the default
# verbosity, and a run where nobody was asked stays silent.
set -euo pipefail
# shellcheck source=tests/testlib.sh
. "$(dirname "$0")/testlib.sh"
bin=$(httrack_path)
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_wizardlog.XXXXXX") || exit 1
cleanup_push rm -rf "$tmp"
link=other.invalid/x.html
tag='(wizard) answer'
site="$tmp/site"
mkdir -p "$site"
printf '<a href="a.html">a</a> <a href="http://%s">e</a>' "$link" \
>"$site/index.html"
echo hello >"$site/a.html"
# the CLI re-prompts until the reply is not empty, so never run out of answers
answers() { printf '%s\n' "$1" "$1" "$1" "$1" "$1" >"$tmp/answers"; }
crawl() { # crawl MIRRORDIR [httrack args...], sets $log
local out=$1
shift
"$bin" "file://$site/index.html" -O "$out" --quiet "$@" \
<"$tmp/answers" >/dev/null 2>&1
log="$out/hts-log.txt"
}
# "ignore this link": the answer forbids it and inserts the single-link filter
answers 0
crawl "$tmp/m0" -W
grep -q "$tag '0' (n=0) for $link: forbidden, filters: -$link\$" "$log" ||
fail "the answer line is wrong in $log"
# the prefix fspc() tallies by: a notice is a message, never a warning
grep -q "Info:.*$tag '0'" "$log" ||
fail "the answer is not tallied as a message in $log"
# "ignore the whole host": same reply path, a different filter. The log echoes
# what was inserted, so a hardcoded pattern would still read -.../x.html here.
answers 2
crawl "$tmp/m2" -W
grep -q "$tag '2' (n=2) for $link: forbidden, filters: -other.invalid/[*]\$" "$log" ||
fail "the answer line does not echo the inserted filter in $log"
# a host-scope answer, which accepts and inserts two filters: the starred form
# misses the apex, so both slots must reach the log, in insertion order
answers 1000
crawl "$tmp/m1000" -W
grep -q "$tag '1000' (n=1000) for $link: allowed, filters: [+][*].other.invalid/[*] [+]other.invalid/[*]\$" "$log" ||
fail "a two-filter answer is not logged whole in $log"
# nobody was asked: the automatic resolution stays at the debug level. This run
# is silent because no question is put, not because the callback is missing;
# the CLI always registers query3, so no test here reaches that guard.
answers 0
crawl "$tmp/mauto"
! grep -q "$tag" "$log" ||
fail "a non-interactive run logged an answer in $log"

View File

@@ -0,0 +1,76 @@
#!/bin/bash
#
# An authorized wizard link was recorded before its type was resolved, so it
# reached the mirror under its <base>.<id>.delayed placeholder name.
set -euo pipefail
# shellcheck source=tests/crawllib.sh
. "$(dirname "$0")/crawllib.sh"
httrack=$(httrack_path)
# "mirror this link"
WIZARD_MIRROR=5
# spare answers: an empty one means "ignore this link", and EOF spins the
# prompt (#1247)
WIZARD_SPARES=40
# the body the placeholder file never carried
MARKER=beyond-the-scope
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/httrack_wizdelayed.XXXXXX") || exit 1
cleanup_push rm -rf "$tmpdir"
# a second port is a second address to the wizard, hence a question
mkdir -p "${tmpdir}/foreign"
printf '<html><body>%s</body></html>\n' "$MARKER" >"${tmpdir}/foreign/a.html"
local_server_start --root "${tmpdir}/foreign" --log "${tmpdir}/foreign.log"
foreign=$SRV_PORT
# the SAME link twice: the first reference records the placeholder name, the
# second one recalls it and can no longer resolve the type
mkdir -p "${tmpdir}/root/wizdelayed"
cat >"${tmpdir}/root/wizdelayed/index.html" <<EOF
<html><body>
<a href="http://127.0.0.1:${foreign}/a.html">a</a>
<a href="http://127.0.0.1:${foreign}/a.html">b</a>
</body></html>
EOF
local_server_start --root "${tmpdir}/root"
answers="${tmpdir}/answers"
for _ in $(seq "$WIZARD_SPARES"); do echo "$WIZARD_MIRROR"; done >"$answers"
# redirect here, not in run_with_timeout's args: it backgrounds the job with
# stdin closed (#1258)
crawl() { "$httrack" "$@" <"$answers"; }
rc=0
run_with_timeout "$(crawl_deadline)" crawl --max-time="$CRAWL_MAX_TIME" \
-O "${tmpdir}/mirror" -W --robots=0 --retries=0 \
"http://127.0.0.1:${SRV_PORT}/wizdelayed/index.html" \
>"${tmpdir}/crawl.log" 2>&1 || rc=$?
test "$rc" -ne 124 || fail "the crawl never returned"
test "$rc" -eq 0 || fail "the crawl exited $rc"
grep -q 'beyond this mirror scope' "${tmpdir}/crawl.log" ||
fail "the wizard never asked about the foreign link"
# what the mirror holds, which is what the user sees
page="${tmpdir}/mirror/127.0.0.1_${foreign}/a.html"
test -f "$page" || fail "the authorized link is missing from the mirror"
grep -q "$MARKER" "$page" || fail "the authorized link was saved without its body"
left=$(find "${tmpdir}/mirror" -name '*.delayed' -print)
test -z "$left" || fail "placeholder names left in the mirror: $left"
# both references, including the second one the stored placeholder came back for
index="${tmpdir}/mirror/127.0.0.1_${SRV_PORT}/wizdelayed/index.html"
local_refs=$(grep -o "127\.0\.0\.1_${foreign}/a\.html" "$index" | wc -l)
test "$local_refs" -eq 2 ||
fail "$local_refs of the 2 references were rewritten to the mirrored page"
# secondary: the engine's own account of the same failure
log="${tmpdir}/mirror/hts-log.txt"
! grep -q 'Duplicate entry in hts_wait_delayed' "$log" ||
fail "the link was recorded before its type was resolved"
! grep -q 'probably looping, type unknown' "$log" ||
fail "the authorized link was dropped as looping"

59
tests/299_option-arg-caps.test Executable file
View File

@@ -0,0 +1,59 @@
#!/bin/bash
#
# Pin each option's cap against the macro htsglobal.h declares for it, so the
# header and the engine cannot drift apart. Limits are read from the header
# rather than copied here, which is the duplication the macros exist to end.
set -euo pipefail
# shellcheck source=tests/testlib.sh
. "$(dirname "$0")/testlib.sh"
bin=$(httrack_path)
testdir=$(cd "$(dirname "$0")" && pwd)
hdr="$(cd "${top_srcdir:-${testdir}/..}" && pwd)/src/htsglobal.h"
assert_file "$hdr" "no htsglobal.h to read the caps from"
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_argcaps.XXXXXX") || exit 1
cleanup_push rm -rf "$tmp"
echo '<html><body>x</body></html>' >"$tmp/page.html"
# POSIX BRE only: BSD sed has no \+
cap() {
sed -n "s/^#define $1[[:space:]][[:space:]]*\([0-9][0-9]*\).*/\1/p" "$hdr"
}
run() { # option value tag
"$bin" "file://$tmp/page.html" -O "$tmp/out$3" --quiet -n -%v0 "$1" "$2" 2>&1
}
# option macro message
caps="-%F HTS_FOOTER_MAXSIZE Footer string too long
-%l HTS_LANGISO_MAXSIZE Lang list string too long
-%R HTS_REFERER_MAXSIZE Referer URL too long
-%L HTS_FILELIST_MAXSIZE File list string too long
-%b HTS_BINDHOST_MAXSIZE Hostname string too long
-%E HTS_FROMEMAIL_MAXSIZE From email too long"
n=0
while read -r opt macro msg; do
n=$((n + 1))
limit=$(cap "$macro")
test "${limit:-0}" -gt 1 || fail "$macro: no usable cap read from htsglobal.h"
val=$(printf "%*s" "$limit" "" | tr ' ' a)
rc=0
run "$opt" "${val%a}" "${n}a" >/dev/null || rc=$?
assert_eq 0 "$rc" "$opt rejected a value one byte under $macro ($limit)"
rc=0
out=$(run "$opt" "$val" "${n}b") || rc=$?
test "$rc" -ne 0 || fail "$opt accepted a value of exactly $macro ($limit)"
# the message pins which check fired, so an unrelated panic cannot pass
grep -qF "$msg" <<<"$out" || fail "$opt was not rejected by \"$msg\""
done <<<"$caps"
test "$n" -eq 6 || fail "expected 6 caps, checked $n"
exit 0

View File

@@ -39,15 +39,8 @@ test "$((SECONDS - start))" -lt 15 || fail "watchdog fired late"
rc=0
if is_windows; then
# Existence by exact Windows PID, not a global ping.exe count: the timing
# sub-test above leaves a still-dying ping that a count would race. Plain
# tasklist, no switches (the workflow's MSYS2_ARG_CONV_EXCL='*' mangles a
# //FI filter arg into a silent no-match); $2 is the PID, and $1 must be
# ping.exe too, since Windows hands a freed PID straight back out. Folded
# case, as the tasklist matchers in proclib.sh already are.
alive() {
tasklist 2>/dev/null |
awk -v p="$1" 'tolower($1) == "ping.exe" && $2 == p {f = 1} END {exit !f}'
}
# sub-test above leaves a still-dying ping that a count would race.
alive() { win_pid_runs "$1" ping.exe; }
# alive() is a conjunction now, and one that never matches would call every
# survivor reaped. Prove it fires on a live ping, reached the same way.
ping -n 20 127.0.0.1 >/dev/null 2>&1 &
@@ -55,6 +48,12 @@ if is_windows; then
cw=$(cat "/proc/$cpid/winpid" 2>/dev/null)
test -n "$cw" || fail "could not read a live ping's Windows PID"
alive "$cw" || fail "alive() cannot see a running ping.exe (pid $cw)"
# What kill_tree checks before firing (#1228), on a live process: /proc must
# name the image tasklist answers with, or the check passes nothing on.
win_capture "$cpid"
test "$WIN_PID" = "$cw" || fail "win_capture read winpid '$WIN_PID', /proc says $cw"
win_pid_runs "$cw" "$WIN_IMAGE" || fail "tasklist does not call pid $cw a '$WIN_IMAGE'"
! win_pid_runs "$cw" no-such-image.exe || fail "win_pid_runs accepts any image at all"
kill_tree "$cpid" "$cw"
wait "$cpid" 2>/dev/null || true
# The grandchild ping records its own Windows PID: non-empty proves it ran

View File

@@ -161,45 +161,27 @@ LC_ALL=C awk -v keys="$keys" '
}
' "$def" || fail=1
# A msgid absent from a language file silently falls back to English (#862).
# What is left waived is the Android UI, whose strings never reached lang/*.txt.
LC_ALL=C sort -u <"$keys" >"$tmp/english.sorted"
LC_ALL=C sort -u >"$tmp/waived" <<'UNTRANSLATED'
Beware: you local browser might be unable to browse files with embedded filenames
Click on this notification to restart the interrupted mirror
Could not create internal cached resources
Could not get the system external storage directory
Could not write to:
Go To HTTrack Forum
Go To HTTrack Website
HTTrack may not be able to download websites until this problem is fixed
HTTrack: could not save profile for '%s'!
HTTrack: mirror '%s' stopped!
No storage media (SDCARD)
Read-only media (SDCARD)
Recreated HTTrack internal cached resources
View Documentation
View License
UNTRANSLATED
# The CLI prints the wizard's scope answers itself while the GUI reads them from
# lang.def, so a reword on one side alone silently drifts the two apart (#1117).
for scope in "Mirror %s and every host below it" "Ignore %s and every host below it"; do
for f in "$eng" "$def" "$top_srcdir/src/httrack.c"; do
if ! LC_ALL=C grep -qF "$scope" "$f"; then
echo "${f##*/}: missing the wizard scope answer \"$scope\""
fail=1
fi
done
done
: >"$tmp/complete"
# A msgid absent from a language file silently falls back to English (#862).
LC_ALL=C sort -u <"$keys" >"$tmp/english.sorted"
for f in "$langdir"/*.txt; do
LC_ALL=C awk 'NR%2==1 { sub(/\r$/, ""); print }' "$f" | LC_ALL=C sort -u >"$tmp/have"
LC_ALL=C comm -23 "$tmp/english.sorted" "$tmp/have" >"$tmp/absent"
untranslated=$(LC_ALL=C comm -23 "$tmp/absent" "$tmp/waived")
untranslated=$(LC_ALL=C comm -23 "$tmp/english.sorted" "$tmp/have")
if [ -n "$untranslated" ]; then
printf '%s\n' "$untranslated" | awk -v n="${f##*/}" '{print n ": untranslated msgid: " $0}'
fail=1
fi
# A waiver nobody needs any more has to go, or the hole quietly reopens.
LC_ALL=C comm -12 "$tmp/absent" "$tmp/waived" >>"$tmp/complete"
done
LC_ALL=C sort -u -o "$tmp/complete" "$tmp/complete"
stale=$(LC_ALL=C comm -23 "$tmp/waived" "$tmp/complete")
if [ -n "$stale" ]; then
printf '%s\n' "$stale" | awk '{print "lang: translated everywhere now, drop the waiver: " $0}'
fail=1
fi
# An unknown ${LANG_*} renders as nothing rather than erroring, so check that
# every macro the templates interpolate, in any wrapper form, resolves.

View File

@@ -73,14 +73,18 @@ ci_start_native_watchdog() {
# End the step, announcing $2 first: the kill runs no EXIT trap, so an unexplained
# death is all the log would otherwise hold.
ci_heartbeat_kill() {
local main=$1
local main=$1 winpid winimage
ci_annotate error "suite watchdog" "$2"
# Ahead of the kill, which runs no EXIT trap: an orphan would outlive the
# step and overwrite its last status with a frozen tail.
test -z "${watchdog:-}" || kill_pid "$watchdog"
# Read before the two kills below, which would leave the winpid naming
# whoever Windows hands the number to next (#1228).
win_capture "$main"
winpid=$WIN_PID winimage=$WIN_IMAGE
# Direct first: kill_tree may reap this watchdog before its own root (#953).
kill_pid "$main"
kill_tree "$main"
kill_tree "$main" "$winpid" "$winimage"
}
ci_suite_heartbeat() {

View File

@@ -3,7 +3,8 @@
# channel that outlives a dead runner. Every status carries the same state.
param(
[string]$ProgressLog = '',
[int]$IntervalSeconds = 30,
# 15s, not 30: a lost runner dies inside a single status period (#1228).
[int]$IntervalSeconds = 15,
[int]$PollSeconds = 5,
# Cannot outlive the step, whatever the caller forgets to kill.
[int]$MaxSeconds = 2700,
@@ -48,19 +49,40 @@ function Format-WatchdogStatus {
$q = '?'
if ($Static -ge 0) { $q = [string]$Static }
$t = ($InFlight -replace '\s+', ' ').Trim()
if ($t.Length -gt 46) { $t = $t.Substring(0, 46) }
if ($t.Length -gt 30) { $t = $t.Substring(0, 30) }
$s = 't={0}s q={1}s {2} | {3}' -f $Elapsed, $q, $t, $Counters
if ($s.Length -gt 140) { $s = $s.Substring(0, 140) }
return $s
}
# The worse of the running peak and how much longer the last iteration took than
# the poll it asked for. Peak, not last: a status covers several iterations.
function Get-MaxLag {
param([int]$Peak, [double]$Elapsed, [double]$Since, [int]$Poll)
$lag = [int]($Elapsed - $Since - $Poll * 1000)
if ($lag -gt $Peak) { return $lag }
return $Peak
}
# The TCP counters are cumulative since boot, so only the change over a status
# period says what the suite did; $Prev is $null on the first sample.
function Get-TcpDelta {
param($Prev, $Cur)
if ($null -eq $Prev) { return 'n=? f=?' }
return 'n={0} f={1}' -f ($Cur[0] - $Prev[0]), ($Cur[1] - $Prev[1])
}
# --- probes ------------------------------------------------------------------
$script:LastTcp = $null
# One try/catch per counter: a probe that fails costs its own field, not the loop.
# In-process only. A CIM query is richer, but its connect to a wedged WMI service
# is unbounded, and would hang the one reporter still standing.
function Get-WatchdogCounters {
param([int]$LagMs = 0, [int]$Failed = 0)
$f = New-Object System.Collections.ArrayList
$ps = @()
try {
$ps = @(Get-Process)
[void]$f.Add('p={0}' -f $ps.Count)
@@ -68,8 +90,31 @@ function Get-WatchdogCounters {
} catch { [void]$f.Add('p=? h=?') }
try {
$drive = New-Object System.IO.DriveInfo($env:SystemDrive + '\')
[void]$f.Add('d={0}' -f [int]($drive.AvailableFreeSpace / 1MB))
[void]$f.Add('d={0}' -f [int]($drive.AvailableFreeSpace / 1GB))
} catch { [void]$f.Add('d=?') }
# The ramp detector: starvation is what makes a poll overshoot.
[void]$f.Add('l={0}' -f $LagMs)
# Box-stop against network-break: the status that lands after an outage says
# how many it swallowed, and a box that stopped never lands one.
[void]$f.Add('x={0}' -f $Failed)
try {
# One GetTcpStatisticsEx; GetActiveTcpConnections() would allocate per socket.
$t = [System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties().GetTcpIPv4Statistics()
$cur = @($t.ConnectionsInitiated, ($t.FailedConnectionAttempts + $t.ResetConnections))
[void]$f.Add((Get-TcpDelta $script:LastTcp $cur))
$script:LastTcp = $cur
[void]$f.Add('e={0}' -f $t.CurrentConnections)
} catch { [void]$f.Add('n=? f=? e=?') }
try {
if ($ps.Count -lt 1) { throw 'no process list' }
[void]$f.Add('m={0}' -f [int]((($ps | Measure-Object -Property WorkingSet64 -Sum).Sum) / 1MB))
[void]$f.Add('c={0}' -f [int]((($ps | Measure-Object -Property PagedMemorySize64 -Sum).Sum) / 1MB))
# Its own field: the agent is what stops reporting, and the box total hides it.
$agent = @($ps | Where-Object { $_.Name -eq 'Runner.Worker' })
$ws = 0
if ($agent.Count -gt 0) { $ws = [int]((($agent | Measure-Object -Property WorkingSet64 -Sum).Sum) / 1MB) }
[void]$f.Add('a={0}' -f $ws)
} catch { [void]$f.Add('m=? c=? a=?') }
return ($f -join ' ')
}
@@ -156,14 +201,15 @@ function Invoke-WatchdogSelfTest {
Assert-That ($ko[0] -eq 8 -and $ko[1] -eq 8) 'a repeat rejection does not widen the gap'
$long = '43_local-update-truncate-with-a-very-long-name-indeed.test'
$line = Format-WatchdogStatus 812 41 $long 'p=118 h=41230 d=13210'
# The widest real counter line, so a status that fits here fits on the runner.
$line = Format-WatchdogStatus 2700 2700 $long 'p=201 h=54598 d=85 l=120 x=0 n=412 f=0 e=180 m=3100 c=4200 a=210'
Assert-That ($line.Length -le 140) ('status description is {0} characters' -f $line.Length)
Assert-That ($line -like 't=812s q=41s 43_local-update-truncate*') ('status leads with the wrong fields: {0}' -f $line)
Assert-That ($line -like '*d=13210') 'the counters did not survive a long test name'
Assert-That ($line -like 't=2700s q=2700s 43_local-update-truncate*') ('status leads with the wrong fields: {0}' -f $line)
Assert-That ($line -like '*a=210') 'the counters did not survive a long test name'
# -match, not -like: '?' is a wildcard there, so q=0s would satisfy it too.
Assert-That ((Format-WatchdogStatus 8 -1 'x' 'y') -match '^t=8s q=\?s x \| y$') 'an unknown staticness reads as a number'
$clip = Format-WatchdogStatus 1 2 ('x' * 80) 'c'
Assert-That ($clip -match '^t=1s q=2s x{46} \| c$') ('the in-flight name was not clipped to 46: {0}' -f $clip)
Assert-That ($clip -match '^t=1s q=2s x{30} \| c$') ('the in-flight name was not clipped to 30: {0}' -f $clip)
$wide = Format-WatchdogStatus 1 2 ('x' * 300) ('y' * 300)
Assert-That ($wide.Length -le 140) ('an oversized status was not clipped: {0}' -f $wide.Length)
# Cut from the tail: the head carries the fields a wedge is read for.
@@ -181,15 +227,23 @@ function Invoke-WatchdogSelfTest {
# Space-separated key=value: the counters share the 140-char description with
# the fields a wedge is read for, and '?' from a failed probe is a value.
$c = Get-WatchdogCounters
$c = Get-WatchdogCounters 120 3
Assert-That ($c -match '^[a-z]+=\S+( [a-z]+=\S+)*$') ('the counters are not key=value pairs: {0}' -f $c)
foreach ($k in 'p', 'h', 'd') {
foreach ($k in 'p', 'h', 'd', 'l', 'x', 'n', 'f', 'e', 'm', 'c', 'a') {
Assert-That ($c -match ('(^| ){0}=' -f $k)) ('the counters dropped {0}=: {1}' -f $k, $c)
}
Assert-That ($c.Length -le 60) ('the counters take {0} of the 140 characters' -f $c.Length)
Assert-That ($c -match '(^| )l=120( |$)') ('the loop lag is not what the caller measured: {0}' -f $c)
Assert-That ($c -match '(^| )x=3( |$)') ('the failed-post count is not what the caller passed: {0}' -f $c)
Assert-That ((Get-MaxLag 0 6200 1000 5) -eq 200) 'a poll that overshot by 200ms was not measured'
Assert-That ((Get-MaxLag 500 6200 1000 5) -eq 500) 'a smaller lag replaced the peak'
Assert-That ((Get-MaxLag 0 5900 1000 5) -eq 0) 'a poll that returned early reported a lag'
Assert-That ((Get-TcpDelta $null @(70, 9)) -eq 'n=? f=?') 'a first sample with no predecessor reported a delta'
Assert-That ((Get-TcpDelta @(64, 7) @(70, 9)) -eq 'n=6 f=2') 'a total was reported where the delta was asked for'
# 140 less the 16 of t=/q= and the 33 a clipped test name and its separator take.
Assert-That ($c.Length -le 91) ('the counters take {0} of the 140 characters' -f $c.Length)
# Nothing else reads these: every other leg passes its own schedule.
Assert-That ($IntervalSeconds -eq 30) ('the default status cadence is {0}s' -f $IntervalSeconds)
Assert-That ($IntervalSeconds -eq 15) ('the default status cadence is {0}s' -f $IntervalSeconds)
Assert-That ($PollSeconds -eq 5) ('the default poll is {0}s' -f $PollSeconds)
Assert-That (-not (Send-WatchdogStatus 'self-test')) 'the self-test can reach the API'
@@ -216,6 +270,9 @@ $movedAt = 0
$postedAt = -$IntervalSeconds
$backoff = 0
$skip = 0
$lagMax = 0
$failed = 0
$tickAt = $sw.Elapsed.TotalMilliseconds
# Guarded like the rest; the launcher waits for this exact line.
try { Write-Host 'watchdog ready' } catch { }
@@ -223,7 +280,12 @@ Write-WatchdogLog ('watching {0} every {1}s' -f $ProgressLog, $IntervalSeconds)
while ($sw.Elapsed.TotalSeconds -lt $MaxSeconds) {
# Measured, never accumulated: starvation is what makes a sleep overshoot.
$now = [int]$sw.Elapsed.TotalSeconds
$ms = $sw.Elapsed.TotalMilliseconds
$now = [int]($ms / 1000)
# Measured at the top, so the lag covers the probes and the post as well as
# the sleep: starvation stretches all three.
$lagMax = Get-MaxLag $lagMax $ms $tickAt $PollSeconds
$tickAt = $ms
try {
$tail = Get-ProgressTail -Path $ProgressLog
if ($tail.Ok -and $tail.Signature -ne $lastSig) {
@@ -234,14 +296,18 @@ while ($sw.Elapsed.TotalSeconds -lt $MaxSeconds) {
$postedAt = $now
$static = -1
if ($tail.Ok) { $static = $now - $movedAt }
$desc = Format-WatchdogStatus $now $static $tail.Line (Get-WatchdogCounters)
$desc = Format-WatchdogStatus $now $static $tail.Line (Get-WatchdogCounters $lagMax $failed)
# Logged whatever the backoff decides: it throttles the API, not the
# artifact, which is all a run whose token cannot post will leave.
Write-WatchdogLog $desc
if ($skip -gt 0) {
$skip--
} else {
$next = Get-NextThrottle (Send-WatchdogStatus $desc) $backoff
$ok = Send-WatchdogStatus $desc
# Cleared together, and only by a status that landed: a peak reached
# while nothing was getting through is what the next one has to carry.
if ($ok) { $failed = 0; $lagMax = 0 } else { $failed++ }
$next = Get-NextThrottle $ok $backoff
$skip = $next[0]
$backoff = $next[1]
}

View File

@@ -14,7 +14,8 @@ set -euo pipefail
usage() {
echo "usage: ${0##*/} {--srcdir DIR [--builddir DIR] | --headers-dir DIR}" \
"[--backend cl|cc] [--cc CMD] [--cxx CMD] [--self-test] [-- CPPFLAGS...]" >&2
"[--backend cl|cc] [--cc CMD] [--cxx CMD] [--budget SECONDS] [--self-test]" \
"[-- CPPFLAGS...]" >&2
exit 1
}
@@ -26,6 +27,7 @@ cc_cmd=""
cxx_cmd=""
cxx_set=0
selftest=0
budget=
extra=()
while [ $# -gt 0 ]; do
case $1 in
@@ -33,6 +35,10 @@ while [ $# -gt 0 ]; do
selftest=1
shift
;;
--budget)
budget=${2-}
shift 2 || usage
;;
--srcdir)
srcdir=${2-}
shift 2 || usage
@@ -237,15 +243,45 @@ fi
began=$SECONDS
bad=0
# Sliced only for a caller that gave a budget: one call per batch cannot be given up on,
# and an emulated compiler needs more time for it than the harness allows a test (#1146).
# Unsliced elsewhere, so the Windows job keeps paying one compiler spawn per batch.
if [ -n "$budget" ] && [ "$budget" -gt 0 ]; then
export HTTRACK_TEST_TIMEOUT=$budget
slices=8
else
slices=1
fi
slice=$(((${#units[@]} + slices - 1) / slices))
# From the slice size, not from $slices: they differ whenever the units do not divide
# evenly, and a step count that outlives the loop leaves the pacer projecting forever.
per=$(((${#units[@]} + slice - 1) / slice))
left=$((${#langs[@]} * ${#modes[@]} * per))
swept=0
for lang in "${langs[@]}"; do
for mode in "${modes[@]}"; do
compile "$lang" "$mode" "${units[@]}" || {
head -40 "$sweep_log" >&2
echo "the headers do not compile as $lang standalone and pairwise ($mode)" >&2
bad=1
}
i=0
while [ "$i" -lt "${#units[@]}" ]; do
step=$SECONDS
chunk=("${units[@]:i:slice}")
swept=$((swept + ${#chunk[@]}))
compile "$lang" "$mode" "${chunk[@]}" || {
head -40 "$sweep_log" >&2
echo "the headers do not compile as $lang standalone and pairwise ($mode)" >&2
bad=1
}
i=$((i + slice))
left=$((left - 1))
# Only while nothing has failed: a skip past a real break would bury it.
[ "$bad" -ne 0 ] || [ -z "$budget" ] ||
skip_if_out_of_budget "$left" "$((SECONDS - step))"
done
done
done
# What reached the compiler, not what was generated: a slice loop that steps past a unit
# would otherwise report the full set and pass.
want=$((${#langs[@]} * ${#modes[@]} * ${#units[@]}))
[ "$swept" -eq "$want" ] || fail "compiled $swept units of $want, the slicing lost some"
echo "swept $n headers standalone and pairwise x ${#modes[@]} bytecode modes x ${langs[*]}" \
"= $((${#modes[@]} * ${#langs[@]} * ${#units[@]})) units in $((SECONDS - began))s with $backend"
[ "$bad" -eq 0 ] || exit 1

View File

@@ -22,19 +22,39 @@ testdir=$(cd "$(dirname "$0")" && pwd)
# (CRAWL_DEADLINE, 180s a pass) -- budget below that and a slow-but-legitimate
# run would be killed. The slowest healthy test measures 39s. A non-numeric or
# absurd value falls back; 0 disables the guard, for use under a debugger.
budget=${HTTRACK_TEST_TIMEOUT:-600}
case "$budget" in
'' | *[!0-9]*) budget=600 ;;
esac
budget=$(budget_secs)
# The test script is the last argument; automake passes no others today.
for path in "$@"; do :; done
name=$(basename "$path")
# A test whose work legitimately outlasts the wedge budget says so in its header
# (269 sweeps n^2 compiles and paces itself inside it). The name carries the rule the
# reader cannot see: it raises the budget, so no test can disarm the guard. Read with
# the shell to keep it off the per-test fork bill, and bounded, since bash's `test`
# errors rather than compares past intmax and would leave the guard unarmed.
if test "$budget" -gt 0 && test -r "$path"; then
read_lines=0
while test "$read_lines" -lt 40 && IFS= read -r line; do
read_lines=$((read_lines + 1))
case "$line" in
'# TEST_TIMEOUT_AT_LEAST: '*)
want=${line#'# TEST_TIMEOUT_AT_LEAST: '}
case "$want" in
'' | *[!0-9]* | ???????*) ;;
*) test "$((10#$want))" -le "$budget" || budget=$((10#$want)) ;;
esac
break
;;
esac
done <"$path"
fi
# Exported so a test can pace itself against the same number (skip_if_out_of_budget)
# instead of being killed halfway.
export HTTRACK_TEST_TIMEOUT="$budget"
test "$budget" -gt 0 || exec "$BASH" "$@"
# The test script is the last argument; automake passes no others today.
for name in "$@"; do :; done
name=$(basename "$name")
# Give the test its own TMPDIR, so the hang dump can salvage exactly this test's
# crawl logs instead of racing (and deleting) a sibling's under "make check -j".
tmproot=${TMPDIR:-/tmp}

View File

@@ -286,8 +286,12 @@ poll_wait() {
# trap, where a survivor would turn a passing test into a harness timeout.
stop_server() {
test -n "${1:-}" || return 0
local winpid winimage
# Before the signal: a winpid read after it can already name a stranger.
win_capture "$1"
winpid=$WIN_PID winimage=$WIN_IMAGE
kill "$1" 2>/dev/null || true
if is_windows; then kill_tree "$1"; fi
if is_windows; then kill_tree "$1" "$winpid" "$winimage"; fi
reap_bounded "$1" || true
return 0
}
@@ -454,6 +458,29 @@ win_pid() {
fi
}
# WIN_PID and WIN_IMAGE for MSYS pid $1, read while it is alive: /proc keeps the
# entry once the process is gone and Windows reissues the number at once, so a
# later read can name a stranger (#1228). Not for a job just backgrounded: until
# its exec lands, tens of milliseconds later, both still name the forking shell.
# Assigned rather than echoed, a command substitution being a fork (#795).
win_capture() { # win_capture <pid>
WIN_PID='' WIN_IMAGE=''
is_windows || return 0
# Unguarded reads: a missing file leaves the empty value set above, and read
# reports EOF on an unterminated line having already assigned it.
{ read -r WIN_PID <"/proc/$1/winpid"; } 2>/dev/null || true
{ read -r WIN_IMAGE <"/proc/$1/winexename"; } 2>/dev/null || true
WIN_IMAGE=${WIN_IMAGE##*[\\/]}
return 0
}
# Whether Windows PID $1 runs image $2. Both columns at once, since either alone
# answers for a recycled PID, and case-folded as the proclib.sh matchers are.
win_pid_runs() { # win_pid_runs <winpid> <image>
tasklist 2>/dev/null |
awk -v p="$1" -v i="$2" 'tolower($1) == tolower(i) && $2 == p { f = 1 } END { exit !f }'
}
# Signal one process, never its descendants: a caller inside the target's own
# tree cannot rely on kill_tree, whose taskkill is then a grandchild of it (#953).
kill_pid() {
@@ -477,11 +504,17 @@ kill_pid() {
# so args pass verbatim and a //T would reach taskkill unfolded and be rejected.
# $2 is that Windows PID when the caller read it while the job was certainly
# alive: /proc/<pid>/winpid is already gone for a job that has just died, and
# without it the only route left is the host-wide sweep below.
# without it the only route left is the host-wide sweep below. $3 is the image it
# ran then: a number that no longer runs it was reissued while we were not
# looking, and naming a stranger is as good as naming nobody (#1228).
kill_tree() {
local pid=$1 winpid=${2:-}
local pid=$1 winpid=${2:-} image=${3:-}
if is_windows; then
test -n "$winpid" || winpid=$(win_pid "$pid")
if test -n "$winpid" && test -n "$image" && ! win_pid_runs "$winpid" "$image"; then
printf '::warning::pid %s no longer runs %s, not killing it\n' "$winpid" "$image"
winpid=
fi
if test -n "$winpid"; then
taskkill /F /T /PID "$winpid" >/dev/null 2>&1 || true
# Last resort, so it is opt-in: it kills every engine and every python on
@@ -539,16 +572,39 @@ EOF
# one step is slower than its neighbours. It asks an ordering of the callers
# instead, expensive steps first, so no step left can outrun the reserve the one
# before it set (#1146).
skip_if_out_of_budget() { # skip_if_out_of_budget <steps left> <seconds the last took>
local budget=${HTTRACK_TEST_TIMEOUT:-600} need=$(($2 + $2 / 2))
# The budget test-timeout.sh enforces, in seconds, 0 being the guard off. The one
# parser: a value bash arithmetic or test would choke on falls back to the default,
# and a leading zero would otherwise read as octal in one place and decimal in the next.
budget_secs() {
local budget=${HTTRACK_TEST_TIMEOUT:-600}
case "$budget" in '' | *[!0-9]* | ???????*) budget=600 ;; esac
echo "$((10#$budget))"
}
case "$budget" in '' | *[!0-9]*) budget=600 ;; esac
skip_if_out_of_budget() { # skip_if_out_of_budget <steps left> <seconds the last took>
local budget need=$(($2 + $2 / 2))
budget=$(budget_secs)
test "$1" -gt 0 && test "$budget" -gt 0 || return 0
test "$((SECONDS + need))" -ge "$budget" || return 0
echo "$1 steps left, the last took ${2}s and the budget is ${budget}s; skipping" >&2
exit 77
}
# Seconds left of the budget, for a child pacing itself against it (269 hands it to
# the sweep). Never below 1 unless the guard is off, when it stays 0.
budget_left() {
local budget left
budget=$(budget_secs)
test "$budget" -gt 0 || {
echo 0
return 0
}
left=$((budget - SECONDS))
test "$left" -ge 1 || left=1
echo "$left"
}
# Collect a killed job, giving up after REAP_GRACE seconds. kill_tree can fail to
# reap a native Windows descendant -- the very case these watchdogs exist for --
# and a bare `wait` then blocks the watchdog itself forever, so the timeout it was

View File

@@ -48,7 +48,12 @@ cd /bld
bash "${GITHUB_WORKSPACE:-/src}/configure"
make -j"$(nproc)"
# The buildd's own invocation, so a failure here is the one it would report.
make check -j"$(nproc)"
rc=0
make check -j"$(nproc)" || rc=$?
# Always, not only where automake prints it: this leg exists to say what an emulated
# host does. A paced-out skip must not read as coverage with no reason given.
cat tests/test-suite.log || true
test "$rc" -eq 0 || exit "$rc"
# make check exits 0 for an all-SKIP run, and this leg skips a lot by design, so
# a container that quietly lost a dependency would report a green covering

View File

@@ -22,7 +22,8 @@
# -o, --outdir DIR output directory (default: <repo>/dist)
# --orig FILE reuse this upstream orig tarball instead of
# regenerating it (required for a Debian revision
# >= 2, whose orig is frozen in the archive)
# >= 2, whose orig is frozen in the archive, and
# whenever debian/patches carries a patch)
# -s, --source-only build only the source package
# -u, --unsigned do not sign anything (implies no release sigs)
# --no-release-artifacts skip the orig tarball .asc/.md5/.sha1
@@ -39,7 +40,9 @@
#
# The Debian revision in debian/changelog decides the orig: revision 1 builds a
# fresh upstream tarball; revision >= 2 must reuse the orig frozen at revision 1
# (the .dsc references it by checksum), so pass it with --orig.
# (the .dsc references it by checksum), so pass it with --orig. debian/patches
# needs the same tarball for a different reason: a patch backported from upstream
# no longer applies to a tree that has the fix, which is what HEAD would give.
#
# SOURCE_DATE_EPOCH is honored for reproducible output.
@@ -127,6 +130,10 @@ main() {
if [[ $unsigned -eq 0 ]]; then
need gpg
[[ -n $key ]] || die "no signing key (pass --key or set DEBSIGN_KEYID, or use --unsigned)"
# Here rather than at debsign, which runs once the tarball is built: a key id gpg
# cannot resolve to a secret key would otherwise cost the whole build first.
gpg --list-secret-keys -- "$key" >/dev/null 2>&1 ||
die "gpg has no secret key for '$key' (an 0x-prefixed full fingerprint is unambiguous)"
fi
local repo
@@ -168,6 +175,13 @@ main() {
die "Debian revision $rev needs --orig FILE (the orig is frozen from revision 1)"
fi
# A quilt patch is written against the orig it is applied to. Once the fix is
# upstream, HEAD already carries it, so a regenerated orig makes the patch fail
# or, worse, apply with fuzz. Unsigned too: this one breaks the build, not policy.
if [[ -z $orig_in && -s $export_dir/debian/patches/series ]]; then
die "debian/patches is not empty, so --orig FILE is required: the orig built from HEAD already carries the patches"
fi
if [[ -n $orig_in ]]; then
info "reusing upstream tarball $orig_in"
cp -- "$orig_in" "$scratch/$orig"