Compare commits

...

6 Commits

Author SHA1 Message Date
Xavier Roche
43478895df htsparse: extract HT_ADD_END into hts_finish_html_file, drop dead macros
The final HTML-flush step lived in the ~60-line HT_ADD_END macro, captured by
lexical scope in htsparse(). Move it to a real function in htscore.c (next to
its sibling hts_finish_makeindex), passing the output buffer as a plain
pointer+length since the TypedArray is an anonymous struct type that can't cross
a function boundary; the caller keeps the size guard and the trailing
TypedArrayFree. Behavior-preserving: a two-crawl mirror produces byte-identical
output against master.

Also delete the _FILTERS/_FILTERS_PTR macros, dead in htsparse.c (htscore.c and
htswizard.c keep their own local copies); _ROBOTS stays.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-01 20:22:27 +02:00
Xavier Roche
4958bb8666 htsparse: fix HTML-escape truncation, cache-buffer leak, and stats-throttle reset (#465)
* htsparse: reserve 6x room for full HTML-escaping, not 5x

HT_ADD_HTMLESCAPED_ANY reserved strlen*5+1024 on the assumption that
"&amp;" (5 bytes) is the worst-case expansion. That holds for
escape_for_html_print, but escape_for_html_print_full turns a high byte
into "&#xHH;" (6 bytes). Past ~1023 high bytes the reservation is short,
so the escaper hits its internal cap: it truncates the string mid-run and
its overflow return counts the terminating NUL, which then lands inside
the mirrored HTML file. The only _full call site rewrites a link into a
2KB buffer, so a long non-ASCII local path triggers it.

Give the macro a per-function expansion factor (HTS_HTMLESCAPE_MAXEXP=5,
HTS_HTMLESCAPE_FULL_MAXEXP=6) and pass 6 for the _full variant. A new
escape-room self-test pins each function's real worst-case expansion
against the constant the macro reserves, so the two can't drift again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsparse: free the cache buffer in HT_ADD_END

The not-modified fast path reads the stored //[HTML-MD5]// digest via
cache_readdata, which malloc's the buffer, but never freed it. Every page
whose on-disk size already matches the freshly rewritten one leaks that
buffer. Free it after the compare.

Wrapped the macro in clang-format off/on: it is hand-aligned and
clang-format realigns every backslash on any edit, churning untouched
lines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

* htsparse: keep makestat_time out of ENGINE_SET_CONTEXT

makestat_time throttles the makestat/maketrack stats to once per minute:
the wait loop compares time_local() against it and, when it fires, writes
it back to the local. But the field is by-value in the extended context,
so it can't round-trip through ENGINE_SAVE_CONTEXT, while ENGINE_SET_CONTEXT
re-read it from the load-once baseline on every loop iteration. That reset
the local before the next compare, so under -%v / maketrack the throttle
never held and the stats line plus the full back-stack dump were emitted
every iteration.

Drop makestat_time (and the never-changing makestat_fp) from SET_CONTEXT;
they belong to the load-once set. Wrapped the macro in clang-format off/on
for the same backslash-realignment reason as HT_ADD_END.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>

---------

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 19:49:14 +02:00
Xavier Roche
07da404cb8 debian: lintian cleanup (metadata, compat 14, priority, NMU override) (#466)
Add debian/upstream/metadata (DEP-12), bump debhelper-compat to 14, drop the
redundant Priority: optional source field, and rename the stale NMU override
tag changelog-should-mention-nmu to no-nmu-in-changelog with a note on why the
address mismatch trips it.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 19:42:34 +02:00
Xavier Roche
694e45c698 Collapse the 5 inplace_escape_* bodies into one shared helper (#464)
DECLARE_INPLACE_ESCAPE_VERSION stamped five byte-identical function
bodies, one per escaper. Move the body into a single static
inplace_escape() helper parameterized by a function pointer to the
underlying escape_*(); the five HTSEXT_API inplace_escape_* symbols
stay as thin wrappers, so the exported ABI is unchanged.

A new -#test=inplace-escape self-test asserts each inplace_escape_*()
equals its escape_*() applied to a copy across several samples
(including a >255-byte one to hit the helper's malloct path), guarding
the refactor.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 09:14:15 +02:00
Xavier Roche
db9ec2cc3b Replace duplicated HT_INDEX_END macro with a shared function (#463)
The macro that closes the makeindex index.html (footer, optional refresh meta, then the user "primary" command) was copy-pasted into htsparse.c and htscore.c, flagged by a `// COPY IN HTSCORE.C` comment and drifting in whitespace. Collapse both into hts_finish_makeindex() in htscore.c, declared in htscore.h.

The two copies were not byte-identical: the final usercommand() call passed "primary","primary" from htsparse.c but "","" from htscore.c. The helper takes those as adr/fil parameters so each call site keeps its exact behavior.

Add a -#test=makeindex self-test (driven by 01_engine-makeindex.test) that drives the function offline and asserts the footer is written, the refresh meta appears only for a single first link, and *fp/*done are updated.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 08:49:34 +02:00
Xavier Roche
6a9ab2a11f Fix macro-hygiene defects in htsstrings.h (dup define, double-eval) (#462)
StringSubRW was defined twice (the second under a redundant comment); drop
the duplicate. StringCatN and StringSetLength each evaluated their SIZE
argument twice, so a side-effecting argument would run twice and a wrapping
expression could clamp inconsistently. Capture SIZE once into a local, the
way StringCopyN already does. StringSetLength keeps a signed local so the
"negative means strlen(buffer_)" contract is preserved.

No current call site passes a side-effecting SIZE, so the change is
behavior-preserving; the strsafe self-test now passes a (counter++, V)
argument and asserts a single evaluation, which fails on the old macros.

Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 08:49:01 +02:00
13 changed files with 383 additions and 151 deletions

3
debian/control vendored
View File

@@ -1,9 +1,8 @@
Source: httrack
Section: web
Priority: optional
Maintainer: Xavier Roche <roche@httrack.com>
Standards-Version: 4.7.4
Build-Depends: debhelper-compat (= 13), autoconf, autoconf-archive, automake, libtool, zlib1g-dev, libssl-dev
Build-Depends: debhelper-compat (= 14), autoconf, autoconf-archive, automake, libtool, zlib1g-dev, libssl-dev
Rules-Requires-Root: no
Homepage: http://www.httrack.com
Vcs-Git: https://github.com/xroche/httrack.git

View File

@@ -1,4 +1,6 @@
httrack source: changelog-should-mention-nmu
# Maintainer uploads sign the changelog as xavier@debian.org while the control
# Maintainer is roche@httrack.com; lintian reads the address mismatch as an NMU.
httrack source: no-nmu-in-changelog
httrack source: source-nmu-has-incorrect-version-number
# The bundled HTML pages are the genuine upstream documentation taken from

6
debian/upstream/metadata vendored Normal file
View File

@@ -0,0 +1,6 @@
---
Repository: https://github.com/xroche/httrack.git
Repository-Browse: https://github.com/xroche/httrack
Bug-Database: https://github.com/xroche/httrack/issues
Bug-Submit: https://github.com/xroche/httrack/issues/new
Contact: Xavier Roche <roche@httrack.com>

View File

@@ -406,29 +406,106 @@ void hts_invalidate_link(httrackp * opt, int lpos) {
opt->liens[lpos]->pass2 = -1;
}
// Write the makeindex footer (refresh meta when makeindex_links==1), close
// the file, then run usercommand.
void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
FILE **makeindex_fp, int makeindex_links,
const char *makeindex_firstlink,
const char *template_footer, const char *adr,
const char *fil) {
if (!*makeindex_done) {
if (*makeindex_fp) {
char BIGSTK tempo[1024];
if (makeindex_links == 1) {
char BIGSTK link_escaped[HTS_URLMAXSIZE * 2];
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped));
snprintf(tempo, sizeof(tempo),
"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">" CRLF,
link_escaped);
} else
tempo[0] = '\0';
hts_template_format(*makeindex_fp, template_footer,
"<!-- Mirror and index made by HTTrack Website "
"Copier/" HTTRACK_VERSION " " HTTRACK_AFF_AUTHORS
" -->",
tempo, /* EOF */ NULL);
fflush(*makeindex_fp);
fclose(*makeindex_fp);
*makeindex_fp = NULL;
usercommand(opt, 0, NULL,
fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_html_utf8), "index.html"),
adr, fil);
}
}
*makeindex_done = 1;
}
#define HT_INDEX_END do { \
if (!makeindex_done) { \
if (makeindex_fp) { \
char BIGSTK tempo[1024]; \
if (makeindex_links == 1) { \
char BIGSTK link_escaped[HTS_URLMAXSIZE*2]; \
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped)); \
snprintf(tempo,sizeof(tempo),"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">"CRLF, link_escaped); \
} else \
tempo[0]='\0'; \
hts_template_format(makeindex_fp,template_footer, \
"<!-- Mirror and index made by HTTrack Website Copier/"HTTRACK_VERSION" "HTTRACK_AFF_AUTHORS" -->", \
tempo, /* EOF */ NULL \
); \
fflush(makeindex_fp); \
fclose(makeindex_fp); /* à ne pas oublier sinon on passe une nuit blanche */ \
makeindex_fp=NULL; \
usercommand(opt,0,NULL,fconcat(OPT_GET_BUFF(opt),OPT_GET_BUFF_SIZE(opt),StringBuff(opt->path_html_utf8),"index.html"),"",""); \
} \
} \
makeindex_done=1; /* ok c'est fait */ \
} while(0)
/* Flush the parsed HTML output buffer to disk, skipping the rewrite when the
* on-disk MD5 is unchanged. */
void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r,
FILE **fp, const char *ht_buff, size_t ht_len,
const char *adr, const char *fil, const char *save) {
char digest[32 + 2];
off_t fsize_old =
fsize(fconv(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), save));
int ok = 0;
digest[0] = '\0';
domd5mem(ht_buff, ht_len, digest, 1);
if (fsize_old == (off_t) ht_len) {
int mlen = 0;
char *mbuff;
cache_readdata(cache, "//[HTML-MD5]//", save, &mbuff, &mlen);
if (mlen)
mbuff[mlen] = '\0';
if ((mlen == 32) && (strcmp(((mbuff != NULL) ? mbuff : ""), digest) == 0)) {
ok = 1;
hts_log_print(opt, LOG_DEBUG, "File not re-written (md5): %s", save);
}
freet(mbuff);
}
if (!ok) {
file_notify(opt, adr, fil, save, 1, 1, r->notmodified);
*fp = filecreate(&opt->state.strc, save);
if (*fp) {
if (ht_len > 0 && fwrite(ht_buff, 1, ht_len, *fp) != ht_len) {
int fcheck = check_fatal_io_errno();
if (fcheck)
opt->state.exit_xh = -1;
if (opt->log) {
hts_log_print(opt, LOG_ERROR | LOG_ERRNO,
"Unable to write HTML file %s", save);
if (fcheck)
hts_log_print(opt, LOG_ERROR, "* * Fatal write error, giving up");
}
}
fclose(*fp);
*fp = NULL;
if (strnotempty(r->lastmodified))
set_filetime_rfc822(save, r->lastmodified);
} else {
int fcheck = check_fatal_io_errno();
if (fcheck) {
hts_log_print(opt, LOG_ERROR,
"Mirror aborted: disk full or filesystem problems");
opt->state.exit_xh = -1;
}
hts_log_print(opt, LOG_ERROR | LOG_ERRNO, "Unable to save file %s", save);
if (fcheck)
hts_log_print(opt, LOG_ERROR, "* * Fatal write error, giving up");
}
} else {
file_notify(opt, adr, fil, save, 0, 0, r->notmodified);
filenote(&opt->state.strc, save, NULL);
}
if (cache->ndx)
cache_writedata(cache->ndx, cache->dat, "//[HTML-MD5]//", save, digest,
(int) strlen(digest));
}
/* does it look like XML ? (SVG et al.) */
static int look_like_xml(const char *s) {
@@ -2044,7 +2121,8 @@ int httpmirror(char *url1, httrackp * opt) {
/*
Ensure the index is being closed
*/
HT_INDEX_END;
hts_finish_makeindex(opt, &makeindex_done, &makeindex_fp, makeindex_links,
makeindex_firstlink, template_footer, "", "");
/*
updating-a-remotely-deteted-website hack

View File

@@ -362,6 +362,20 @@ void usercommand(httrackp * opt, int exe, const char *cmd, const char *file,
void usercommand_exe(const char *cmd, const char *file);
// Finish the makeindex index.html (footer + refresh meta), run usercommand.
// Updates *makeindex_done/*makeindex_fp in place; adr/fil are the mode strings.
void hts_finish_makeindex(httrackp *opt, int *makeindex_done,
FILE **makeindex_fp, int makeindex_links,
const char *makeindex_firstlink,
const char *template_footer, const char *adr,
const char *fil);
// Flush ht_buff[0..ht_len] to save on disk (skip if MD5 unchanged); *fp
// closed+NULLed on write. Precondition: ht_len>0.
void hts_finish_html_file(httrackp *opt, cache_back *cache, htsblk *r,
FILE **fp, const char *ht_buff, size_t ht_len,
const char *adr, const char *fil, const char *save);
int filters_init(char ***ptrfilters, int maxfilter, int filterinc);
int fspc(httrackp * opt, FILE * fp, const char *type);
@@ -470,4 +484,8 @@ void voidf(void);
/* HTML marker comment marking where the top index is spliced. */
#define HTS_TOPINDEX "TOP_INDEX_HTTRACK"
/* Worst-case byte expansion HT_ADD_HTMLESCAPED* must reserve per escaper. */
#define HTS_HTMLESCAPE_MAXEXP 5 /* escape_for_html_print: '&'->"&amp;" */
#define HTS_HTMLESCAPE_FULL_MAXEXP 6 /* _full: high byte->"&#xHH;" */
#endif

View File

@@ -4131,25 +4131,33 @@ DECLARE_APPEND_ESCAPE_VERSION(escape_uri)
#undef DECLARE_APPEND_ESCAPE_VERSION
// Same as above, but in-place
#undef DECLARE_INPLACE_ESCAPE_VERSION
#define DECLARE_INPLACE_ESCAPE_VERSION(NAME) \
HTSEXT_API size_t inplace_ ##NAME(char *const dest, const size_t size) { \
char buffer[256]; \
const size_t len = strnlen(dest, size); \
const int in_buffer = len + 1 < sizeof(buffer); \
char *src = in_buffer ? buffer : malloct(len + 1); \
size_t ret; \
assertf(src != NULL); \
assertf(len < size); \
memcpy(src, dest, len + 1); \
ret = NAME(src, dest, size); \
if (!in_buffer) { \
freet(src); \
} \
return ret; \
// In-place escaping: copy dest aside, then escape that copy back into dest.
typedef size_t (*escape_fn_t)(const char *src, char *dest, size_t size);
static size_t inplace_escape(char *const dest, const size_t size,
escape_fn_t escape) {
char buffer[256];
const size_t len = strnlen(dest, size);
const int in_buffer = len + 1 < sizeof(buffer);
char *src = in_buffer ? buffer : malloct(len + 1);
size_t ret;
assertf(src != NULL);
assertf(len < size);
memcpy(src, dest, len + 1);
ret = escape(src, dest, size);
if (!in_buffer) {
freet(src);
}
return ret;
}
// Thin exported wrappers binding inplace_escape() to each escaper (ABI).
#undef DECLARE_INPLACE_ESCAPE_VERSION
#define DECLARE_INPLACE_ESCAPE_VERSION(NAME) \
HTSEXT_API size_t inplace_##NAME(char *const dest, const size_t size) { \
return inplace_escape(dest, size, NAME); \
}
DECLARE_INPLACE_ESCAPE_VERSION(escape_in_url)
DECLARE_INPLACE_ESCAPE_VERSION(escape_spc_url)
DECLARE_INPLACE_ESCAPE_VERSION(escape_uri_utf)

View File

@@ -77,13 +77,14 @@ Please visit our Website: http://www.httrack.com
/** Append to the output buffer the string 'A'. **/
#define HT_ADD(A) TypedArrayAppend(output_buffer, A, strlen(A))
/** Append to the output buffer the string 'A', html-escaped. **/
#define HT_ADD_HTMLESCAPED_ANY(A, FUNCTION) do { \
/* clang-format off: an edit realigns all backslashes, churning the macro. */
/* clang-format off */
/** Append 'A' to the output buffer, html-escaped; FACTOR = max byte expansion. **/
#define HT_ADD_HTMLESCAPED_ANY(A, FUNCTION, FACTOR) do { \
if ((opt->getmode & 1) != 0 && ptr>0) { \
const char *const str_ = (A); \
size_t size_; \
/* &amp; is the maximum expansion */ \
TypedArrayEnsureRoom(output_buffer, strlen(str_) * 5 + 1024); \
TypedArrayEnsureRoom(output_buffer, strlen(str_) * (FACTOR) + 1024); \
size_ = FUNCTION(str_, &TypedArrayTail(output_buffer), \
TypedArrayRoom(output_buffer)); \
TypedArraySize(output_buffer) += size_; \
@@ -91,106 +92,22 @@ Please visit our Website: http://www.httrack.com
} while(0)
/** Append to the output buffer the string 'A', html-escaped for &. **/
#define HT_ADD_HTMLESCAPED(A) HT_ADD_HTMLESCAPED_ANY(A, escape_for_html_print)
#define HT_ADD_HTMLESCAPED(A) \
HT_ADD_HTMLESCAPED_ANY(A, escape_for_html_print, HTS_HTMLESCAPE_MAXEXP)
/**
* Append to the output buffer the string 'A', html-escaped for & and
* Append to the output buffer the string 'A', html-escaped for & and
* high chars.
**/
#define HT_ADD_HTMLESCAPED_FULL(A) HT_ADD_HTMLESCAPED_ANY(A, escape_for_html_print_full)
#define HT_ADD_HTMLESCAPED_FULL(A) \
HT_ADD_HTMLESCAPED_ANY(A, escape_for_html_print_full, HTS_HTMLESCAPE_FULL_MAXEXP)
/* clang-format on */
// does nothing
#define XH_uninit do {} while(0)
#define HT_ADD_END { \
int ok=0;\
if (TypedArraySize(output_buffer) != 0) { \
const size_t ht_len = TypedArraySize(output_buffer); \
const char *const ht_buff = TypedArrayElts(output_buffer); \
char digest[32+2];\
off_t fsize_old = fsize(fconv(OPT_GET_BUFF(opt),OPT_GET_BUFF_SIZE(opt),savename()));\
digest[0] = '\0';\
domd5mem(TypedArrayElts(output_buffer), ht_len, digest, 1);\
if (fsize_old == (off_t) ht_len) { \
int mlen = 0;\
char* mbuff;\
cache_readdata(cache,"//[HTML-MD5]//",savename(),&mbuff,&mlen);\
if (mlen) \
mbuff[mlen]='\0';\
if ((mlen == 32) && (strcmp(((mbuff!=NULL)?mbuff:""),digest)==0)) {\
ok=1;\
hts_log_print(opt, LOG_DEBUG, "File not re-written (md5): %s",savename());\
} else {\
ok=0;\
} \
}\
if (!ok) { \
file_notify(opt,urladr(), urlfil(), savename(), 1, 1, r->notmodified); \
fp=filecreate(&opt->state.strc, savename()); \
if (fp) { \
if (ht_len>0) {\
if (fwrite(ht_buff,1,ht_len,fp) != ht_len) { \
int fcheck;\
if ((fcheck=check_fatal_io_errno())) {\
opt->state.exit_xh=-1;\
}\
if (opt->log) { \
hts_log_print(opt, LOG_ERROR | LOG_ERRNO, "Unable to write HTML file %s", savename());\
if (fcheck) {\
hts_log_print(opt, LOG_ERROR, "* * Fatal write error, giving up");\
}\
}\
}\
}\
fclose(fp); fp=NULL; \
if (strnotempty(r->lastmodified)) \
set_filetime_rfc822(savename(),r->lastmodified); \
} else {\
int fcheck;\
if ((fcheck=check_fatal_io_errno())) {\
hts_log_print(opt, LOG_ERROR, "Mirror aborted: disk full or filesystem problems"); \
opt->state.exit_xh=-1;\
}\
hts_log_print(opt, LOG_ERROR | LOG_ERRNO, "Unable to save file %s", savename());\
if (fcheck) {\
hts_log_print(opt, LOG_ERROR, "* * Fatal write error, giving up");\
}\
}\
} else {\
file_notify(opt,urladr(), urlfil(), savename(), 0, 0, r->notmodified); \
filenote(&opt->state.strc, savename(),NULL); \
}\
if (cache->ndx)\
cache_writedata(cache->ndx,cache->dat,"//[HTML-MD5]//",savename(),digest,(int)strlen(digest));\
} \
TypedArrayFree(output_buffer); \
}
#define HT_ADD_FOP
// COPY IN HTSCORE.C
#define HT_INDEX_END do { \
if (!makeindex_done) { \
if (makeindex_fp) { \
char BIGSTK tempo[1024]; \
if (makeindex_links == 1) { \
char BIGSTK link_escaped[HTS_URLMAXSIZE*2]; \
escape_uri_utf(makeindex_firstlink, link_escaped, sizeof(link_escaped)); \
snprintf(tempo,sizeof(tempo),"<meta HTTP-EQUIV=\"Refresh\" CONTENT=\"0; URL=%s\">"CRLF,link_escaped); \
} else \
tempo[0]='\0'; \
hts_template_format(makeindex_fp,template_footer, \
"<!-- Mirror and index made by HTTrack Website Copier/"HTTRACK_VERSION" "HTTRACK_AFF_AUTHORS" -->", \
tempo, /* EOF */ NULL \
); \
fflush(makeindex_fp); \
fclose(makeindex_fp); /* à ne pas oublier sinon on passe une nuit blanche */ \
makeindex_fp=NULL; \
usercommand(opt,0,NULL,fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_html_utf8),"index.html"),"primary","primary"); \
} \
} \
makeindex_done=1; /* ok c'est fait */ \
} while(0)
#define ENGINE_DEFINE_CONTEXT() \
ENGINE_DEFINE_CONTEXT_BASE(); \
/* */ \
@@ -217,6 +134,9 @@ Please visit our Website: http://www.httrack.com
HTS_UNUSED TStamp makestat_time = stre->makestat_time; \
HTS_UNUSED FILE* makestat_fp = stre->makestat_fp
/* clang-format off: an edit realigns all backslashes, churning the macro. */
/* clang-format off */
/* Load-once: re-reading resets makestat_time (mutated locally, never SAVEd). */
#define ENGINE_SET_CONTEXT() \
ENGINE_SET_CONTEXT_BASE(); \
/* */ \
@@ -227,9 +147,8 @@ Please visit our Website: http://www.httrack.com
makeindex_fp = *stre->makeindex_fp_; \
makeindex_links = *stre->makeindex_links_; \
/* */ \
stat_fragment = *stre->stat_fragment_; \
makestat_time = stre->makestat_time; \
makestat_fp = stre->makestat_fp
stat_fragment = *stre->stat_fragment_
/* clang-format on */
#define ENGINE_LOAD_CONTEXT() \
ENGINE_DEFINE_CONTEXT()
@@ -246,8 +165,6 @@ Please visit our Website: http://www.httrack.com
/* */ \
*stre->stat_fragment_ = stat_fragment
#define _FILTERS (*opt->filters.filters)
#define _FILTERS_PTR (opt->filters.filptr)
#define _ROBOTS ((robots_wizard*)opt->robotsptr)
/* Apply current *adr character for the script automate */
@@ -709,7 +626,9 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
}
} else if (heap(ptr)->depth < opt->depth) { // on a sauté level1+1 et level1
HT_INDEX_END;
hts_finish_makeindex(opt, &makeindex_done, &makeindex_fp,
makeindex_links, makeindex_firstlink,
template_footer, "primary", "primary");
}
} // if (opt->makeindex)
}
@@ -3540,7 +3459,12 @@ int htsparse(htsmoduleStruct * str, htsmoduleStructExtended * stre) {
}
/* Flush and save to disk */
HT_ADD_END; // achever
if (TypedArraySize(output_buffer) != 0) {
hts_finish_html_file(
opt, cache, r, &fp, TypedArrayElts(output_buffer),
TypedArraySize(output_buffer), urladr(), urlfil(), savename());
}
TypedArrayFree(output_buffer);
}
//
//

View File

@@ -524,6 +524,41 @@ static int string_safety_selftests(void) {
return 1;
}
/* StringCatN/StringSetLength must eval SIZE once: (n_eval++, V) leaves
n_eval == 2 on a double-eval macro. */
{
String s = STRING_EMPTY;
int n_eval = 0;
StringCat(s, "hello");
StringCatN(s, "world", (n_eval++, 3)); /* strlen>SIZE so the clamp runs */
if (n_eval != 1 || strcmp(StringBuff(s), "hellowor") != 0) {
StringFree(s);
return 1;
}
n_eval = 0;
StringSetLength(s, (n_eval++, 5));
if (n_eval != 1 || StringLength(s) != 5) {
StringFree(s);
return 1;
}
StringFree(s);
}
/* StringSubRW still reads/writes after dropping its duplicate definition. */
{
String s = STRING_EMPTY;
StringCat(s, "abc");
StringSubRW(s, 1) = 'X';
if (StringSub(s, 1) != 'X' || strcmp(StringBuff(s), "aXc") != 0) {
StringFree(s);
return 1;
}
StringFree(s);
}
return 0;
}
@@ -1305,6 +1340,134 @@ static int st_urlhack(httrackp *opt, int argc, char **argv) {
return 0;
}
// hts_finish_makeindex writes the footer, emits the refresh meta only when
// makeindex_links==1, and clears *fp / sets *done. argv[0] is a writable dir.
static int st_makeindex(httrackp *opt, int argc, char **argv) {
char path[HTS_URLMAXSIZE];
char buf[4096];
FILE *fp;
size_t n;
int done;
assertf(argc >= 1);
snprintf(path, sizeof(path), "%s/index.html", argv[0]);
/* single first link: footer + a refresh meta carrying the escaped URL */
done = 0;
fp = fopen(path, "wb");
assertf(fp != NULL);
hts_finish_makeindex(opt, &done, &fp, 1, "http://example.com/a b", "%s%s", "",
"");
assertf(fp == NULL); /* the function closed and cleared it */
assertf(done != 0);
fp = fopen(path, "rb");
assertf(fp != NULL);
n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
buf[n] = '\0';
assertf(strstr(buf, "Mirror and index made by HTTrack") != NULL);
assertf(strstr(buf, "Refresh") != NULL);
assertf(strstr(buf, "example.com") != NULL);
/* no single link: footer only, no refresh meta */
done = 0;
fp = fopen(path, "wb");
assertf(fp != NULL);
hts_finish_makeindex(opt, &done, &fp, 0, NULL, "%s%s", "", "");
assertf(fp == NULL);
assertf(done != 0);
fp = fopen(path, "rb");
assertf(fp != NULL);
n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
buf[n] = '\0';
assertf(strstr(buf, "Mirror and index made by HTTrack") != NULL);
assertf(strstr(buf, "Refresh") == NULL);
UNLINK(path);
printf("makeindex self-test OK\n");
return 0;
}
/* Each inplace_escape_*() must equal escape_*() on a copy. */
static int st_inplace_escape(httrackp *opt, int argc, char **argv) {
/* >255 bytes forces the helper's malloct path, not the stack buffer */
static char longstr[600];
static const char *const samples[] = {
"", "abc", "a b/c?d=e&f", "h\x8ello w\x94rld",
"a%b\"c<d>", "/path to/file", longstr};
static size_t (*const inplace[])(char *, size_t) = {
inplace_escape_in_url, inplace_escape_spc_url, inplace_escape_uri_utf,
inplace_escape_check_url, inplace_escape_uri};
static size_t (*const plain[])(const char *, char *, size_t) = {
escape_in_url, escape_spc_url, escape_uri_utf, escape_check_url,
escape_uri};
size_t i, f;
(void) opt;
(void) argc;
(void) argv;
memset(longstr, 'a', sizeof(longstr) - 1);
for (f = 0; f < sizeof(inplace) / sizeof(inplace[0]); f++) {
for (i = 0; i < sizeof(samples) / sizeof(samples[0]); i++) {
char ref[4096], work[4096];
size_t rret, iret;
rret = plain[f](samples[i], ref, sizeof(ref));
strcpybuff(work, samples[i]);
iret = inplace[f](work, sizeof(work));
assertf(iret == rret);
assertf(strcmp(work, ref) == 0);
}
}
printf("inplace-escape self-test OK\n");
return 0;
}
/* Pin HTS_HTMLESCAPE*_MAXEXP to each escaper's true max byte expansion. */
static int st_escape_room(httrackp *opt, int argc, char **argv) {
/* N > 1023: where 6n outgrows the old 5n+1024 reservation */
enum { N = 2000 };
char *src = malloct(N + 1);
char *dst;
size_t room, got;
(void) opt;
(void) argc;
(void) argv;
/* _full worst case: a high byte expands to "&#xHH;" (6 bytes) */
memset(src, 0xE9, N);
src[N] = '\0';
room = (size_t) N * HTS_HTMLESCAPE_FULL_MAXEXP + 1024;
dst = malloct(room);
got = escape_for_html_print_full(src, dst, room);
assertf(got == (size_t) N * HTS_HTMLESCAPE_FULL_MAXEXP);
assertf(strlen(dst) == got);
freet(dst);
/* one factor short overflows (returns size), truncating the page: the bug */
room = (size_t) N * (HTS_HTMLESCAPE_FULL_MAXEXP - 1) + 1024;
dst = malloct(room);
got = escape_for_html_print_full(src, dst, room);
assertf(got == room);
freet(dst);
/* plain escaper worst case: '&' -> "&amp;" (5); high bytes stay verbatim */
memset(src, '&', N);
src[N] = '\0';
room = (size_t) N * HTS_HTMLESCAPE_MAXEXP + 1024;
dst = malloct(room);
got = escape_for_html_print(src, dst, room);
assertf(got == (size_t) N * HTS_HTMLESCAPE_MAXEXP);
assertf(strlen(dst) == got);
freet(dst);
freet(src);
printf("escape-room self-test OK\n");
return 0;
}
/* Default User-Agent: honest HTTrack token, no resurrected Windows 98. */
static int st_useragent(httrackp *opt, int argc, char **argv) {
const char *ua = StringBuff(opt->user_agent);
@@ -1621,6 +1784,12 @@ static const struct selftest_entry {
{"dns", "", "DNS resolver/cache self-test", st_dns},
{"cookies", "", "cookie request-header self-test", st_cookies},
{"useragent", "", "default User-Agent self-test", st_useragent},
{"makeindex", "[dir]", "hts_finish_makeindex footer/refresh self-test",
st_makeindex},
{"inplace-escape", "", "inplace_escape_* vs escape_* equivalence self-test",
st_inplace_escape},
{"escape-room", "", "HT_ADD_HTMLESCAPED* reservation-factor self-test",
st_escape_room},
{"status", "", "HTTP status code -> reason phrase self-test", st_status},
{"acceptencoding", "[dir]",
"Accept-Encoding advertises gzip+deflate, both decode", st_acceptencoding},

View File

@@ -121,9 +121,6 @@ struct String {
/** Byte at POS (read/write). No bounds check; POS must be < StringLength. **/
#define StringSubRW(BLK, POS) (StringBuffRW(BLK)[POS])
/** Subcharacter (read/write) **/
#define StringSubRW(BLK, POS) (StringBuffRW(BLK)[POS])
/** Byte POS positions from the end (read). POS==1 is the last byte. **/
#define StringRight(BLK, POS) (StringBuff(BLK)[StringLength(BLK) - POS])
@@ -191,8 +188,9 @@ HTS_STATIC char *StringBuffN_(String *blk, int size) {
asserts SIZE fits the existing content; does not (re)allocate. **/
#define StringSetLength(BLK, SIZE) \
do { \
if (SIZE >= 0) { \
(BLK).length_ = SIZE; \
const int len__ = (SIZE); /* signed: negative means strlen(buffer_) */ \
if (len__ >= 0) { \
(BLK).length_ = len__; \
} else { \
(BLK).length_ = strlen((BLK).buffer_); \
} \
@@ -308,10 +306,11 @@ HTS_STATIC void StringAttach(String *blk, char **str) {
#define StringCatN(BLK, STR, SIZE) \
do { \
const char *str__ = (STR); \
const size_t usize__ = (SIZE); \
if (str__ != NULL) { \
size_t size__ = strlen(str__); \
if (size__ > (SIZE)) { \
size__ = (SIZE); \
if (size__ > usize__) { \
size__ = usize__; \
} \
StringMemcat(BLK, str__, size__); \
} \

View File

@@ -0,0 +1,7 @@
#!/bin/bash
#
set -euo pipefail
# HT_ADD_HTMLESCAPED* must reserve the escaper's worst case (6 for _full).
httrack -O /dev/null -#test=escape-room run | grep -q "escape-room self-test OK"

View File

@@ -0,0 +1,7 @@
#!/bin/bash
#
set -euo pipefail
# inplace_escape_*() must match escape_*() on a copy: guards the shared helper.
httrack -O /dev/null -#test=inplace-escape run | grep -q "inplace-escape self-test OK"

12
tests/01_engine-makeindex.test Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/bash
#
set -euo pipefail
# hts_finish_makeindex writes the footer and gates the refresh meta on a single
# first link (guards the macro->function extraction).
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=makeindex "$dir" run |
grep -q "makeindex self-test OK"

View File

@@ -36,6 +36,9 @@ TESTS = \
01_engine-filter.test \
01_engine-hashtable.test \
01_engine-idna.test \
01_engine-escape-room.test \
01_engine-inplace-escape.test \
01_engine-makeindex.test \
01_engine-mime.test \
01_engine-parse.test \
01_engine-pause.test \