Change report format specification
Run with --changes (-%d), HTTrack writes hts-changes.json
in the project directory, next to hts-log.txt, describing what the crawl
left new, changed, unchanged and gone compared to the previous mirror. The file is
rewritten from scratch at the end of every run, and the log carries a one-line
summary of the same counts.
What "changed" means
A resource is changed when its bytes differ, not when the server merely re-sent
it. HTTrack compares the payload it just received against the copy the previous
run left behind: for pages it parses, the previous payload comes from the cache
(the file on disk carries the mirror footer and its crawl date, so its bytes
differ on every run); for everything else, the mirrored file is the payload
verbatim and is compared directly.
Where no digest can be taken on either side, because the cache is disabled or
the previous copy is gone, the report falls back to the transfer signal, and a
server that answers 200 rather than 304 reads as changed. Keeping the cache on
(the default) is what makes the report precise.
Fields
- schema: format version, currently 1. It is bumped only
on an incompatible change; new fields may appear without one.
- generator: the HTTrack build that wrote the file.
- date: when the report was written, UTC, YYYY-MM-DDThh:mm:ssZ.
- first_crawl: true when no index of a previous mirror
(hts-cache/old.lst) was found, so there was nothing to compare against and
everything is listed as new.
- partial: true when the report ran out of memory and lists only
part of the mirror.
- purged: true when --purge-old was in effect, so the
files under gone were also deleted from disk.
- counts: the size of each of the four lists.
- new, changed, unchanged, gone: the
lists themselves. Every mirrored file appears in exactly one of them.
Each entry is an object:
- url: the absolute URL the file came from. Empty under
gone: deletions are computed from the mirror's file index, which records
paths, not URLs.
- file: the path relative to the mirror root, with forward
slashes. This is the entry's identity: a URL and a redirect that resolve to the
same local file are one entry, not two.
- size: the mirrored file's size in bytes, absent when the file
is not on disk.
- previous_size: under changed only, the size of the
copy the previous run left.
Encoding
The file is JSON, UTF-8. URLs and local paths reach HTTrack as raw bytes and are
not guaranteed to be valid UTF-8; any byte sequence that is not becomes
U+FFFD (\ufffd), so the file always parses. Compare on file
rather than on url when a mirror is known to carry legacy-charset URLs.
Example
{
"schema": 1,
"generator": "HTTrack Website Copier/3.49-14",
"date": "2026-07-26T15:29:03Z",
"first_crawl": false,
"partial": false,
"purged": true,
"counts": { "new": 1, "changed": 1, "unchanged": 1, "gone": 1 },
"new": [
{ "url": "http://example.com/d.html", "file": "example.com/d.html", "size": 280 }
],
"changed": [
{ "url": "http://example.com/a.html", "file": "example.com/a.html", "size": 281, "previous_size": 273 }
],
"unchanged": [
{ "url": "http://example.com/b.html", "file": "example.com/b.html", "size": 277 }
],
"gone": [
{ "url": "", "file": "example.com/c.html" }
]
}
Notes
- A file listed under gone is only deleted when --purge-old is
on. Left in place it drops out of the mirror's index, so it is reported once and
not again.
- A resource whose local file name changed since the previous mirror (a new
MIME type, say) is reported as new under its new name; the old name is
reported as gone only if the file is still on disk. The two entries are
not paired.
- A run that transfers no data at all is rolled back: HTTrack restores the
previous cache generation and leaves the previous report in place, so a lost
connection does not overwrite a good report with an empty one.
- Content diffs, and keeping the previous copy of a changed page, are out of
scope: both change what a mirror directory contains.
|
