Compare commits

..

2 Commits

Author SHA1 Message Date
Xavier Roche
77cd8ea2c7 Range-check --warc-max-size and regenerate the man page
Replace the unchecked sscanf on the --warc-max-size argument with a
strtoll parse that rejects non-numeric, negative, and overflowing values,
leaving the default 0 (single archive) intact. The man page and its HTML
render were already regenerated for the option's help line in the feature
commit, so this only tightens the parse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 21:35:33 +02:00
Xavier Roche
8135969d79 Add WARC/1.1 v1.1: --warc-max-size rotation, WARC-Truncated, FTP resource records
Extends the merged WARC writer with the three v1.1 items. All logic stays in
htswarc.c; the engine hooks stay thin.

--warc-max-size N (-%rs) rotates the archive into NAME-00000.warc.gz, -00001,
... once a segment passes N bytes (wget naming), each segment led by its own
warcinfo and never splitting a record. N<=0 keeps the single-file behavior.

WARC-Truncated tags a body cut short by a cap. The mirror size (-M) and time
(-E) caps abort in-flight transfers and overwrite the slot's status to a
negative TIMEOUT, after which back_finalize and the WARC hook both bail; so the
partial is archived at the abort site, before the clobber, with WARC-Truncated:
length/time. HTTrack's own incomplete/retry bookkeeping is untouched, and a
genuine broken transfer (not a cap) is still not archived. The per-file cap (-m)
rejects on Content-Length rather than truncating, so it has no truncated body.
Standard ISO 28500 tokens only (length/time/disconnect); the task's non-standard
"disk" is left out.

FTP transfers have no HTTP envelope, so an ftp:// capture becomes one resource
record: WARC-Type: resource, the payload's own Content-Type, block = payload,
no request/response. FTP back_finalize runs on the main loop (the worker thread
only flips status), so the single-writer no-lock design holds.

Self-tests warc-trunc / warc-ftp / warc-rotate added to the -#test=warc family
and 01_zlib-warc.test; each was confirmed to fail without the corresponding
code. man/httrack.1 + html regenerated for the %r help line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-22 21:29:18 +02:00
35 changed files with 1 additions and 269 deletions

View File

@@ -103,17 +103,6 @@ ${do:end-if}
> ${LANG_I61}
<br><br>
<input type="checkbox" name="warc" ${checked:warc}
title='${html:LANG_WARCTIP}' onMouseOver="info('${html:LANG_WARCTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_WARC}
<br><br>
${LANG_WARCFILE}
<input name="warcfile" value="${warcfile}" size="40"
title='${html:LANG_WARCFILETIP}' onMouseOver="info('${html:LANG_WARCFILETIP}'); return true" onMouseOut="info('&nbsp;'); return true"
>
<br><br>
<input type="checkbox" name="norecatch" ${checked:norecatch}
title='${html:LANG_I5b}' onMouseOver="info('${html:LANG_I5b}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I34b}

View File

@@ -141,8 +141,6 @@ ${do:copy:KeepSlashes:keepslashes}
${do:copy:KeepQueryOrder:keepqueryorder}
${do:copy:StripQuery:stripquery}
${do:copy:StoreAllInCache:cache2}
${do:copy:Warc:warc}
${do:copy:WarcFile:warcfile}
${do:copy:LogType:logtype}
${do:copy:UseHTTPProxyForFTP:ftpprox}
${do:copy:ProxyType:proxytype}

View File

@@ -187,8 +187,6 @@ ${do:end-if}
${test:toler:--tolerant}
${test:http10:--http-10}
${test:cache2:--store-all-in-cache}
${test:warc:--warc}
${test:warcfile:--warc-file "}${html:warcfile}${test:warcfile:"}
${test:norecatch:--do-not-recatch}
${test:logf:--single-log}
${test:logtype:::--extra-log:--debug-log}
@@ -239,8 +237,6 @@ KeepSlashes=${ztest:keepslashes:0:1}
KeepQueryOrder=${ztest:keepqueryorder:0:1}
StripQuery=${stripquery}
StoreAllInCache=${ztest:cache2:0:1}
Warc=${ztest:warc:0:1}
WarcFile=${warcfile}
LogType=${logtype}
UseHTTPProxyForFTP=${ztest:ftpprox:0:1}
ProxyType=${proxytype}

View File

@@ -1034,11 +1034,3 @@ LANG_STRIPQUERY
Strip query keys:
LANG_STRIPQUERYTIP
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
LANG_WARC
Write a WARC archive of the crawl
LANG_WARCTIP
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
LANG_WARCFILE
WARC archive name:
LANG_WARCFILETIP
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Ïðåìàõâàíå íà êëþ÷îâå îò çàÿâêàòà:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Êëþ÷îâå îò çàÿâêàòà, ðàçäåëåíè ñúñ çàïåòàÿ, êîèòî äà ñå ïðåìàõíàò îò èìåòî íà çàïèñàíèÿ ôàéë (íàïðèìåð sid,utm_source).
Write a WARC archive of the crawl
Çàïèñâàíå íà WARC àðõèâ íà îáõîæäàíåòî
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Çàïèñâàíå íà âñåêè èçòåãëåí îòãîâîð è â WARC/1.1 àðõèâ ïî ISO-28500, äî îãëåäàëîòî.
WARC archive name:
Èìå íà WARC àðõèâà:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Íåçàäúëæèòåëíî áàçîâî èìå çà WARC àðõèâà; îñòàâåòå ïðàçíî çà àâòîìàòè÷íî èìåíóâàíå â èçõîäíàòà äèðåêòîðèÿ.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Eliminar claves de query string:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Claves de query string, separadas por comas, que se eliminarán del nombre de los archivos guardados (p. ej. sid,utm_source).
Write a WARC archive of the crawl
Escribir un archivo WARC del rastreo
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Guardar también cada respuesta descargada en un archivo WARC/1.1 ISO-28500, junto a la réplica.
WARC archive name:
Nombre del archivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nombre base opcional para el archivo WARC; déjelo en blanco para nombrarlo automáticamente en el directorio de salida.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Odebrat klíèe dotazu:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Klíèe dotazu oddìlené èárkami, které se vynechají z pojmenování ukládaných souborù (napø. sid,utm_source).
Write a WARC archive of the crawl
Zapsat archiv WARC z procházení
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Uložit také každou staženou odpovìï do archivu WARC/1.1 podle ISO-28500 vedle zrcadla.
WARC archive name:
Název archivu WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Volitelný základní název archivu WARC; ponechte prázdné pro automatické pojmenování ve výstupním adresáøi.

View File

@@ -956,11 +956,3 @@ Strip query keys:
移除查詢鍵:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
以逗號分隔的查詢鍵,將其從儲存檔案的命名中移除 (例如 sid,utm_source)。
Write a WARC archive of the crawl
寫入此次抓取的 WARC 封存檔
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
同時將每個已擷取的回應儲存為 ISO-28500 WARC/1.1 封存檔,置於鏡像網站旁。
WARC archive name:
WARC 封存檔名稱:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC 封存檔的選用基本名稱;留空則於輸出目錄中自動命名。

View File

@@ -956,11 +956,3 @@ Strip query keys:
剥离查询键:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
用逗号分隔的查询键,将其从保存文件的命名中删除 (例如 sid,utm_source)。
Write a WARC archive of the crawl
写入本次抓取的 WARC 归档
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
同时将每个已获取的响应保存为 ISO-28500 WARC/1.1 归档,置于镜像站点旁边。
WARC archive name:
WARC 归档名称:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC 归档的可选基本名称;留空则在输出目录中自动命名。

View File

@@ -958,11 +958,3 @@ Strip query keys:
Ukloniti kljuèeve upita:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Zarezom odvojeni kljuèevi upita koji se izostavljaju iz naziva spremljenih datoteka (npr. sid,utm_source).
Write a WARC archive of the crawl
Zapi¹i WARC arhivu obilaska
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Spremi i svaki preuzeti odgovor u ISO-28500 WARC/1.1 arhivu, uz zrcalo.
WARC archive name:
Naziv WARC arhive:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Neobavezni osnovni naziv WARC arhive; ostavite prazno za automatsko imenovanje u izlaznom direktoriju.

View File

@@ -1004,11 +1004,3 @@ Strip query keys:
Fjern forespørgselsnøgler:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kommaseparerede forespørgselsnøgler, der udelades i navngivningen af gemte filer (f.eks. sid,utm_source).
Write a WARC archive of the crawl
Skriv et WARC-arkiv af gennemsøgningen
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Gem også hvert hentet svar i et ISO-28500 WARC/1.1-arkiv ved siden af spejlet.
WARC archive name:
Navn på WARC-arkiv:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valgfrit basisnavn til WARC-arkivet; lad feltet stå tomt for automatisk navngivning i outputmappen.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Query-Schlüssel entfernen:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kommagetrennte Query-Schlüssel, die bei der Benennung gespeicherter Dateien entfallen (z. B. sid,utm_source).
Write a WARC archive of the crawl
WARC-Archiv des Crawls schreiben
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Jede heruntergeladene Antwort zusätzlich in einem ISO-28500-WARC/1.1-Archiv neben dem Spiegel speichern.
WARC archive name:
Name des WARC-Archivs:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optionaler Basisname für das WARC-Archiv; leer lassen, um es automatisch im Ausgabeverzeichnis zu benennen.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Eemalda päringuvõtmed:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Komadega eraldatud päringuvõtmed, mis jäetakse salvestatud faili nimest välja (nt sid,utm_source).
Write a WARC archive of the crawl
Kirjuta läbimise WARC-arhiiv
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Salvesta iga alla laaditud vastus ka ISO-28500 WARC/1.1 arhiivi peegli kõrvale.
WARC archive name:
WARC-arhiivi nimi:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC-arhiivi valikuline põhinimi; jäta tühjaks, et see väljundkataloogis automaatselt nimetada.

View File

@@ -1004,11 +1004,3 @@ Strip query keys:
Strip query keys:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Write a WARC archive of the crawl
Write a WARC archive of the crawl
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
WARC archive name:
WARC archive name:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.

View File

@@ -958,11 +958,3 @@ Strip query keys:
Poista kyselyavaimet:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Pilkuin erotellut kyselyavaimet, jotka jätetään pois tallennettujen tiedostojen nimeämisestä (esim. sid,utm_source).
Write a WARC archive of the crawl
Kirjoita imuroinnin WARC-arkisto
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Tallenna myös jokainen noudettu vastaus ISO-28500 WARC/1.1 -arkistoon peilin viereen.
WARC archive name:
WARC-arkiston nimi:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valinnainen WARC-arkiston perusnimi; jätä tyhjäksi, jotta se nimetään automaattisesti tulostehakemistoon.

View File

@@ -1004,11 +1004,3 @@ Strip query keys:
Supprimer les clés de query string :
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Clés de query string à retirer du nommage des fichiers enregistrés, séparées par des virgules (par ex. sid,utm_source).
Write a WARC archive of the crawl
Écrire une archive WARC du crawl
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Enregistrer aussi chaque réponse téléchargée dans une archive WARC/1.1 (ISO-28500), à côté du miroir.
WARC archive name:
Nom de l'archive WARC :
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nom de base optionnel pour l'archive WARC ; laissez vide pour le générer automatiquement dans le répertoire de sortie.

View File

@@ -958,11 +958,3 @@ Strip query keys:
Αφαίρεση κλειδιών ερωτήματος:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Κλειδιά ερωτήματος χωρισμένα με κόμμα, που θα αφαιρεθούν από την ονομασία των αποθηκευμένων αρχείων (π.χ. sid,utm_source).
Write a WARC archive of the crawl
Εγγραφή αρχείου WARC της ανίχνευσης
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Αποθήκευση κάθε ληφθείσας απόκρισης και σε αρχείο WARC/1.1 ISO-28500, δίπλα στο είδωλο.
WARC archive name:
Όνομα αρχείου WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Προαιρετικό βασικό όνομα για το αρχείο WARC. Αφήστε το κενό για αυτόματη ονομασία στον κατάλογο εξόδου.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Rimuovi chiavi della query string:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Chiavi della query string, separate da virgole, da rimuovere dai nomi dei file salvati (ad es. sid,utm_source).
Write a WARC archive of the crawl
Scrivi un archivio WARC della scansione
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Salva anche ogni risposta scaricata in un archivio WARC/1.1 ISO-28500, accanto al mirror.
WARC archive name:
Nome dell'archivio WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nome di base facoltativo per l'archivio WARC; lascia vuoto per assegnarlo automaticamente nella directory di output.

View File

@@ -956,11 +956,3 @@ Strip query keys:
削除するクエリキー:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
保存ファイル名の生成から除外するクエリキーをカンマ区切りで指定します (例: sid,utm_source)。
Write a WARC archive of the crawl
クロールの WARC アーカイブを書き出す
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
取得した各レスポンスを ISO-28500 WARC/1.1 アーカイブとしてミラーの隣にも保存します。
WARC archive name:
WARC アーカイブ名:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC アーカイブの任意のベース名。空欄にすると出力ディレクトリ内で自動的に名前が付けられます。

View File

@@ -956,11 +956,3 @@ Strip query keys:
¾âáâàÐÝØ ÚÛãçÕÒØ ÞÔ ÑÐàÐúÕâÞ:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
ºÛãçÕÒØ ÞÔ ÑÐàÐúÕâÞ, ÞÔÔÕÛÕÝØ áÞ ×ÐߨàÚÐ, èâÞ áÕ ÞâáâàÐÝãÒÐÐâ ÞÔ ØÜÕâÞ ÝÐ ×ÐçãÒÐÝÐâÐ ÔÐâÞâÕÚÐ (ÝÐ ßàØÜÕà sid,utm_source).
Write a WARC archive of the crawl
·ÐßØèØ WARC ÐàåØÒÐ ÝÐ ßàÕÑÐàãÒÐúÕâÞ
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
·ÐçãÒÐø ÓÞ áÕÚÞø ßàÕ×ÕÜÕÝ ÞÔÓÞÒÞà Ø ÒÞ ISO-28500 WARC/1.1 ÐàåØÒÐ, ßÞÚàÐø ÞÓÛÕÔÐÛÞâÞ.
WARC archive name:
¸ÜÕ ÝÐ WARC ÐàåØÒÐâÐ:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
¸×ÑÞàÝÞ ÞáÝÞÒÝÞ ØÜÕ ×Ð WARC ÐàåØÒÐâÐ; ÞáâÐÒÕâÕ ßàÐ×ÝÞ ×Ð ÐÒâÞÜÐâáÚÞ ØÜÕÝãÒÐúÕ ÒÞ Ø×ÛÕ×ÝØÞâ ÔØàÕÚâÞàØãÜ.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Lekérdezési kulcsok eltávolítása:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Vesszõvel elválasztott lekérdezési kulcsok, amelyeket el kell hagyni a mentett fájl elnevezésébõl (pl. sid,utm_source).
Write a WARC archive of the crawl
A bejárás WARC archívumának írása
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Minden letöltött válasz mentése ISO-28500 WARC/1.1 archívumba is, a tükör mellé.
WARC archive name:
WARC archívum neve:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
A WARC archívum opcionális alapneve; hagyja üresen az automatikus elnevezéshez a kimeneti könyvtárban.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Query-sleutels verwijderen:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Door komma's gescheiden query-sleutels die bij het benoemen van opgeslagen bestanden worden weggelaten (bijv. sid,utm_source).
Write a WARC archive of the crawl
Schrijf een WARC-archief van de crawl
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Sla ook elke opgehaalde respons op in een ISO-28500 WARC/1.1-archief, naast de mirror.
WARC archive name:
Naam van WARC-archief:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Optionele basisnaam voor het WARC-archief; laat leeg om het automatisch een naam te geven in de uitvoermap.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Fjern spørrenøkler:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kommaseparerte spørrenøkler som utelates i navngivingen av lagrede filer (f.eks. sid,utm_source).
Write a WARC archive of the crawl
Skriv et WARC-arkiv av gjennomgangen
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Lagre også hvert nedlastet svar i et ISO-28500 WARC/1.1-arkiv ved siden av speilet.
WARC archive name:
Navn på WARC-arkiv:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valgfritt basisnavn for WARC-arkivet; la feltet stå tomt for automatisk navngivning i utdatamappen.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Usuñ klucze zapytania:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Rozdzielone przecinkami klucze zapytania pomijane przy nazywaniu zapisanych plików (np. sid,utm_source).
Write a WARC archive of the crawl
Zapisz archiwum WARC z indeksowania
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Zapisz te¿ ka¿d± pobran± odpowied¼ do archiwum WARC/1.1 ISO-28500, obok kopii lustrzanej.
WARC archive name:
Nazwa archiwum WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Opcjonalna nazwa bazowa archiwum WARC; pozostaw puste, aby nazwaæ je automatycznie w katalogu wyj¶ciowym.

View File

@@ -1004,11 +1004,3 @@ Strip query keys:
Remover chaves da query string:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Chaves da query string, separadas por vírgulas, a serem removidas da nomeação dos arquivos salvos (ex.: sid,utm_source).
Write a WARC archive of the crawl
Gravar um arquivo WARC do rastreamento
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Salvar também cada resposta baixada em um arquivo WARC/1.1 ISO-28500, ao lado do espelho.
WARC archive name:
Nome do arquivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nome base opcional para o arquivo WARC; deixe em branco para nomeá-lo automaticamente no diretório de saída.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Remover chaves da query string:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Chaves da query string, separadas por vírgulas, a remover da nomeação dos ficheiros guardados (por ex. sid,utm_source).
Write a WARC archive of the crawl
Escrever um arquivo WARC do rastreio
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Guardar também cada resposta transferida num arquivo WARC/1.1 ISO-28500, ao lado do espelho.
WARC archive name:
Nome do arquivo WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nome base opcional para o arquivo WARC; deixe em branco para o nomear automaticamente no diretório de saída.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Elimina cheile din query string:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Chei din query string, separate prin virgula, de eliminat din denumirea fisierelor salvate (de ex. sid,utm_source).
Write a WARC archive of the crawl
Scrie o arhiva WARC a parcurgerii
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Salveaza si fiecare raspuns descarcat intr-o arhiva WARC/1.1 ISO-28500, langa oglinda.
WARC archive name:
Numele arhivei WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Nume de baza optional pentru arhiva WARC; lasati gol pentru a-l denumi automat in directorul de iesire.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Óäàëÿòü êëþ÷è çàïðîñà:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Êëþ÷è çàïðîñà ÷åðåç çàïÿòóþ, óäàëÿåìûå èç èìåíè ñîõðàíÿåìîãî ôàéëà (íàïðèìåð, sid,utm_source).
Write a WARC archive of the crawl
Çàïèñàòü WARC-àðõèâ îáõîäà
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Ñîõðàíÿòü êàæäûé çàãðóæåííûé îòâåò òàêæå â àðõèâ WARC/1.1 ISO-28500 ðÿäîì ñ çåðêàëîì.
WARC archive name:
Èìÿ WARC-àðõèâà:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Íåîáÿçàòåëüíîå áàçîâîå èìÿ WARC-àðõèâà; îñòàâüòå ïóñòûì äëÿ àâòîìàòè÷åñêîãî èìåíîâàíèÿ â âûõîäíîì êàòàëîãå.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Odstráni» kµúèe dotazu:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kµúèe dotazu oddelené èiarkami, ktoré sa vynechajú z pomenovania ukladaných súborov (napr. sid,utm_source).
Write a WARC archive of the crawl
Zapísa» archív WARC z prehµadávania
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Ulo¾i» aj ka¾dú stiahnutú odpoveï do archívu WARC/1.1 ISO-28500 vedµa zrkadla.
WARC archive name:
Názov archívu WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Voliteµný základný názov archívu WARC; ponechajte prázdne pre automatické pomenovanie vo výstupnom adresári.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Odstrani kljuce poizvedbe:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Z vejicami loceni kljuci poizvedbe, ki se izpustijo pri poimenovanju shranjenih datotek (npr. sid,utm_source).
Write a WARC archive of the crawl
Zapisi arhiv WARC iz pregledovanja
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Shrani tudi vsak preneseni odgovor v arhiv WARC/1.1 ISO-28500 poleg zrcala.
WARC archive name:
Ime arhiva WARC:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Neobvezno osnovno ime arhiva WARC; pustite prazno za samodejno poimenovanje v izhodni mapi.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Ta bort frågenycklar:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kommaseparerade frågenycklar som utelämnas vid namngivningen av sparade filer (t.ex. sid,utm_source).
Write a WARC archive of the crawl
Skriv ett WARC-arkiv av genomsökningen
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Spara även varje hämtat svar i ett ISO-28500 WARC/1.1-arkiv, bredvid spegeln.
WARC archive name:
WARC-arkivets namn:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Valfritt basnamn för WARC-arkivet; lämna tomt för att namnge det automatiskt i utdatakatalogen.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Sorgu anahtarlarýný çýkar:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Kaydedilen dosya adlandýrmasýndan çýkarýlacak, virgülle ayrýlmýþ sorgu anahtarlarý (örn. sid,utm_source).
Write a WARC archive of the crawl
Taramanýn WARC arþivini yaz
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Ýndirilen her yanýtý ayrýca aynanýn yanýna bir ISO-28500 WARC/1.1 arþivine kaydet.
WARC archive name:
WARC arþivi adý:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC arþivi için isteðe baðlý temel ad; çýktý dizininde otomatik adlandýrma için boþ býrakýn.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Âèëó÷àòè êëþ÷³ çàïèòó:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Êëþ÷³ çàïèòó ÷åðåç êîìó, ÿê³ âèëó÷àþòüñÿ ç ³ìåí³ çáåðåæåíîãî ôàéëó (íàïðèêëàä, sid,utm_source).
Write a WARC archive of the crawl
Çàïèñàòè WARC-àðõ³â îáõîäó
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Çáåð³ãàòè êîæíó çàâàíòàæåíó â³äïîâ³äü òàêîæ ó àðõ³â WARC/1.1 ISO-28500 ïîðÿä ³ç äçåðêàëîì.
WARC archive name:
²ì'ÿ WARC-àðõ³âó:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
Íåîáîâ'ÿçêîâà áàçîâà íàçâà WARC-àðõ³âó; çàëèøòå ïîðîæí³ì äëÿ àâòîìàòè÷íîãî íàéìåíóâàííÿ ó âèõ³äíîìó êàòàëîç³.

View File

@@ -956,11 +956,3 @@ Strip query keys:
Olib tashlanadigan sorov kalitlari:
Comma-separated query keys to drop from the saved-file naming (e.g. sid,utm_source).
Saqlangan fayl nomidan olib tashlanadigan, vergul bilan ajratilgan sorov kalitlari (masalan, sid,utm_source).
Write a WARC archive of the crawl
Qidiruvning WARC arxivini yozish
Also save every fetched response into an ISO-28500 WARC/1.1 archive, next to the mirror.
Har bir yuklab olingan javobni ISO-28500 WARC/1.1 arxiviga ham, ko'zgu yonida saqlash.
WARC archive name:
WARC arxivi nomi:
Optional base name for the WARC archive; leave blank to auto-name it under the output directory.
WARC arxivi uchun ixtiyoriy asosiy nom; chiqish katalogida avtomatik nomlash uchun bo'sh qoldiring.

View File

@@ -46,12 +46,9 @@ cat >"$stubdir/x-www-browser" <<EOF
echo "stub browser invoked with: \$1" >&2
# Also fetch an option page and require a rendered title='' tooltip: proves the
# option template expands and the \${html:} filter escapes into the attribute.
# option9 additionally proves the WARC control renders with its expanded label.
opturl="\${1%/}/server/option2.html"
warcurl="\${1%/}/server/option9.html"
if body="\$(curl -fsSL --max-time 20 "\$1")" && printf '%s' "\$body" | grep -qai httrack && printf '%s' "\$body" | grep -qaF step2.html &&
opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='" &&
warc="\$(curl -fsSL --max-time 20 "\$warcurl")" && printf '%s' "\$warc" | grep -qaF 'name="warcfile"' && printf '%s' "\$warc" | grep -qaF WARC; then
opt="\$(curl -fsSL --max-time 20 "\$opturl")" && printf '%s' "\$opt" | grep -qaF "title='"; then
echo PASS >"$marker"
else
echo "FAIL: unexpected response from \$1" >"$marker"