rms/httrack
1
0
Fork 0
mirror of https://github.com/xroche/httrack.git synced 2026-07-28 11:32:48 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
2fccb170583d9c36738aff0f106eeb6be71c19fa
httrack/html/server/option9.html
Xavier Roche 7e2209d4cc No way to see what changed between two mirrors (#721) 
* Report what a crawl changed against the previous mirror (--changes)

--update already knows which resources were new, which changed and which the
server called unchanged, and throws it away: the flags reach file_notify() and
go no further than a log line, while deletions exist only as a side effect of
purging. --changes (-%d) keeps all of it and writes hts-changes.json plus a
one-line summary in the log.

"Changed" means the bytes differ, not that the server re-sent the resource.
Comparing the mirrored files directly would not work: HTTrack stamps every
parsed page with the crawl date via the footer, so those bytes differ on every
run. Payloads are compared instead, the previous one coming from the cache for
parsed pages and from the local copy sampled just before it is overwritten for
everything else.

The mirror-relative path, not the URL, is the accumulator's key, so a redirect
and its target that share a save name are one entry; and only the first notify
for a file samples its pre-run state, so a retried transfer is not counted
twice. What counts as already mirrored comes from the previous run's file
index rather than from the file's presence on disk: a partial left by this
crawl's own failed attempt is on disk but was never part of the previous
mirror.

The deleted set is now computed whether or not purging is enabled; unlinking
still happens only under --purge-old.

Closes #714

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* changes: never leave a stale report behind

A crawl that mirrored nothing created no accumulator, so hts_changes_close_opt
returned without writing and the previous run's report stayed on disk as if it
described this one. Write it whenever --changes is on. The no-data rollback is
the deliberate exception, and is now documented: it restores the previous cache
generation, so leaving the matching report alone is the consistent behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* changes: drop em dashes from the format page

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* changes: fix the review's three findings

The size shortcut compared rendered on-disk sizes even for parsed pages, whose
payload digests describe something else entirely, so it decided the outcome
before the payload comparison could run: a page whose payload never changed but
whose rewritten links moved read as changed. It now only applies when both
digests describe the file on disk.

file_notify() reaches the accumulator from the FTP download thread as well as
the main one, and the lazy allocation, the coucal write and the entries realloc
were all unguarded. Every entry point now takes a mutex, and the HTML hook does
its cache read before taking it, since that read can itself re-enter
file_notify() and move the array.

Two fixtures cover what nothing did: a gzipped direct-to-disk body that changes
at constant length (without the pre-sample before the decoded temp is renamed,
it reads as unchanged), and a page with a fixed payload behind a redirect whose
target is renamed, so its file on disk changes length while its bytes do not.
Both were checked against builds with the respective fix reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* changes: document the renamed-file case in the format page

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* changes: refresh two stale test comments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* Merge origin/master into feat/change-report

Both sides appended to tests/Makefile.am's TESTS; kept master's
86_local-proxytrack-cache-longfields.test alongside 88_local-changes.test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>

* changes: translate the new GUI strings into the remaining 28 locales (#740)

The feature PR added the two LANG_CHANGES* entries to lang.def with English and
Francais only; every other locale fell back to English in the WebHTTrack form.
Each file is written in its own declared charset.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* changes: fix the review's blocking findings

Lock the report path against the FTP thread the crawl never joins, seal the
accumulator once the report is written, key entries off the project directory
so the report survives --cache=0, stop calling a file gone when the crawl only
failed to re-fetch it, and skip the hook's work entirely when --changes is off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* changes: prove the fixes, and give the report a cache-off mode

Adds a changes-race self-test (the FTP shape: notifier threads against the
report path), fixtures for a transfer the crawl never completes, for a leftover
file at a name the crawl mirrors fresh, and for a cache-off mirror, plus a pass
with purging on. Registers the web GUI's --changes box with the clearing
companion master's #725 now requires, and documents the degraded mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* changes: keep the failed-transfer case portable

A connection killed before the status line surfaces differently on macOS, where
it truncates the mirrored file to zero (#748). Assert only what holds on both:
it is never reported gone, and its file survives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 06:58:33 +02:00

202 lines
9.1 KiB
HTML
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" lang="${LANGUAGE_ISO}">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=${LANGUAGE_CHARSET}" />
<meta name="description" content="${LANG_METADESC}" />
<meta name="keywords" content="${LANG_METAKEYW}" />
<title>'${projname}' - HTTrack Website Copier</title>
<link rel="stylesheet" type="text/css" media="screen" href="style.css">
<script type="text/javascript" src="ping.js" defer></script>
<script language="javascript">
<!--
function do_load() {
window.status=' ';
${do:if-not-empty:closeme}
window.close();
${do:end-if}
${do:set:closeme:}
}
function do_unload() {
}
function info(str) {
window.status = str;
}
// -->
</script>
</head>
<body onLoad="do_load();" onUnload="do_unload();">
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr>
<td><img src="images/header_title_4.gif" width="400" height="34" alt="HTTrack Website Copier" title="" border="0" id="title" /></td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="3" class="tableWidth">
<tr>
<td id="subTitle">${LANG_OSFWB}</td>
</tr>
</table>
<table width="76%" border="0" align="center" cellspacing="0" cellpadding="0" class="tableWidth">
<tr class="blak">
<td>
<table width="100%" border="0" align="center" cellspacing="1" cellpadding="0">
<tr>
<td colspan="6">
<table width="100%" border="0" align="center" cellspacing="0" cellpadding="10">
<tr>
<td id="pageContent">
<!-- ==================== End prologue ==================== -->
<table border="0" width="100%">
<tr><td width="90%">
<h2 align="center"><em>${LANG_O2} - ${LANG_IOPT9}</em></h2>
</td>
${/* show help only if available */}
${do:if-file-exists:html/index.html}
<td>
<a href="/step9_opt9.html" target="_blank"
onClick="window.open('/step9_opt9.html', 'help', 'toolbar=no, location=no, directories=no, status=yes, menubar=no, scrollbars=yes, resizable=yes, width=640, height=480'); return false"
title='${html:LANG_TIPHELP}' onMouseOver="info('${html:LANG_TIPHELP}'); return true" onMouseOut="info('&nbsp;'); return true"
>${LANG_TIPHELP}</a>
</td>
${do:end-if}
</tr></table>
<br>
<table class="tableWidth" border="0" width="100%" cellspacing="0">
<tr>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option1.html" onClick="form.redirect.value='option1.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT1}' onMouseOver="info('${html:LANG_IOPT1}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT1}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option2.html" onClick="form.redirect.value='option2.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT2}' onMouseOver="info('${html:LANG_IOPT2}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT2}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option3.html" onClick="form.redirect.value='option3.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT3}' onMouseOver="info('${html:LANG_IOPT3}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT3}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option4.html" onClick="form.redirect.value='option4.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT4}' onMouseOver="info('${html:LANG_IOPT4}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT4}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option5.html" onClick="form.redirect.value='option5.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT5}' onMouseOver="info('${html:LANG_IOPT5}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT5}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option11.html" onClick="form.redirect.value='option11.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT11}' onMouseOver="info('${html:LANG_IOPT11}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT11}</a></td>
</tr><tr>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option6.html" onClick="form.redirect.value='option6.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT6}' onMouseOver="info('${html:LANG_IOPT6}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT6}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option7.html" onClick="form.redirect.value='option7.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT7}' onMouseOver="info('${html:LANG_IOPT7}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT7}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option8.html" onClick="form.redirect.value='option8.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT8}' onMouseOver="info('${html:LANG_IOPT8}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT8}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option9.html" onClick="form.redirect.value='option9.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT9}' onMouseOver="info('${html:LANG_IOPT9}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT9}</a></td>
<td class="tabCtrl" align="center"><a style="background:black;color: white" href="option10.html" onClick="form.redirect.value='option10.html'; form.submit(); return false;" align="center" title='${html:LANG_IOPT10}' onMouseOver="info('${html:LANG_IOPT10}'); return true" onMouseOut="info('&nbsp;'); return true">${LANG_IOPT10}</a></td>
<td class="tabCtrl" align="center">&nbsp;</td>
</tr>
</table>
<br><br>
<table>
<tr><td>
<form method="POST" action="${thisfile}" name="form">
<input type="hidden" name="sid" value="${sid}">
<input type="hidden" name="redirect" value="">
<input type="hidden" name="closeme" value="">
<!-- clear if not checked -->
<input type="hidden" name="warc" value="">
<input type="hidden" name="changes" value="">
<input type="hidden" name="norecatch" value="">
<input type="hidden" name="logf" value="">
<input type="hidden" name="index" value="">
<input type="hidden" name="index2" value="">
<input type="checkbox" name="cache2" ${checked:cache2}
title='${html:LANG_I1e}' onMouseOver="info('${html:LANG_I1e}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I61}
<br><br>
<input type="checkbox" name="warc" ${checked:warc}
title='${html:LANG_WARCTIP}' onMouseOver="info('${html:LANG_WARCTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_WARC}
<br><br>
${LANG_WARCFILE}
<input name="warcfile" value="${warcfile}" size="40"
title='${html:LANG_WARCFILETIP}' onMouseOver="info('${html:LANG_WARCFILETIP}'); return true" onMouseOut="info('&nbsp;'); return true"
>
<br><br>
<input type="checkbox" name="changes" ${checked:changes}
title='${html:LANG_CHANGESTIP}' onMouseOver="info('${html:LANG_CHANGESTIP}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_CHANGES}
<br><br>
<input type="checkbox" name="norecatch" ${checked:norecatch}
title='${html:LANG_I5b}' onMouseOver="info('${html:LANG_I5b}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I34b}
<br><br>
<input type="checkbox" name="logf" ${checked:logf}
title='${html:LANG_I7}' onMouseOver="info('${html:LANG_I7}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I36}
<select name="logtype"
title='${html:LANG_I1f}' onMouseOver="info('${html:LANG_I1f}'); return true" onMouseOut="info('&nbsp;'); return true"
>
${listid:logtype:LISTDEF_9}
</select>
<br><br>
<input type="checkbox" name="index" ${checked:index}
title='${html:LANG_I6}' onMouseOver="info('${html:LANG_I6}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I35}
<br><br>
<input type="checkbox" name="index2" ${checked:index2}
title='${html:LANG_I6b}' onMouseOver="info('${html:LANG_I6b}'); return true" onMouseOut="info('&nbsp;'); return true"
> ${LANG_I35b}
<br><br>
<tr><td>
<table width="100%">
<tr><td align="left">
<input type="submit" value="${LANG_OK}"
onClick="form.closeme.value=1; form.submit(); return false;"
title='${html:LANG_TIPOK}' onMouseOver="info('${html:LANG_TIPOK}'); return true" onMouseOut="info('&nbsp;'); return true"
>
</td><td align="right">
<input type="button" value="${LANG_CANCEL}"
onClick="window.close();"
title='${html:LANG_TIPCANCEL}' onMouseOver="info('${html:LANG_TIPCANCEL}'); return true" onMouseOut="info('&nbsp;'); return true"
>
</td></tr>
</table>
</td></tr>
</form>
</td></tr>
</table>
<!-- ==================== Start epilogue ==================== -->
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
<table width="76%" border="0" align="center" valign="bottom" cellspacing="0" cellpadding="0">
<tr>
<td id="footer"><small><small>&copy; 1998-2026 Xavier Roche & other contributors - Web Design: Leto Kauler.</small></small></td>
</tr>
</table>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink