Compare commits

..

3 Commits

Author SHA1 Message Date
Xavier Roche
84197bf158 Cover the SOCKS5 origin port in the self-test, and trim comments
Reverting htsproxy.c alone to the wrap-then-check version passed the whole
suite, so nothing pinned that half of the fix and a later refactor could have
dropped it silently. The scripted SOCKS5 harness now asserts a bad origin port
is refused with no byte sent.

65616 would not have tested anything: it fits an int, so the old range check
already caught it. The case the old check could not see is 4294967376, which
overflowed the sscanf("%d") into a plausible 80 and passed; the trailing-junk
and signed spellings sscanf also accepted are covered alongside it. Against
master's htsproxy.c the new assertion fails.

Also note at the SOCKS5 call site that an empty "host:" stays refused there
while the direct path accepts it, so the asymmetry reads as intended rather
than as an oversight, and halve two comments.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-17 17:04:52 +02:00
Xavier Roche
03ce63ccf9 Keep accepting an empty "host:" port, which means the default
WHATWG and curl both treat an empty port as valid and meaning the scheme
default, and httrack's URL parser hands "http://host:/x" to newhttp_addr() as
adr="host:", so refusing it as an unparseable port would drop a legal link that
master crawls today.

Guard the parse on a non-empty port text, and move the case to the accepted
side of the self-test. Removing the guard fails that test, so the case is
covered rather than merely asserted.

The SOCKS5 origin path already refused an empty port before this series and
still does; that predates #614 and is left alone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-17 17:04:52 +02:00
Xavier Roche
b2f55c67b4 Refuse a URL whose port is outside 1..65535
newhttp_addr() parsed a crawled URL's port with sscanf("%d") and validated it
with "i != -1", then cast it to unsigned short. An out-of-range port therefore
folded into 0..65535 and httrack connected there: http://host:105579/ mirrored
off port 40043, storing the result under host_105579 while the bytes came from
somewhere else. A value past INT_MAX is also UB, and wraps on glibc.

What carries the weight is that a user's port exclusion filter stops describing
the port httrack opens, since the filter sees the literal spelling and the
connect uses the folded one.

Parse it with hts_parse_url_port() instead, factored out of #610's
parse_proxy_port(), and refuse the link through newhttp_addr()'s existing
"msg + INVALID_SOCKET" path. A proxy port can fall back to the scheme default;
a URL port cannot, because the URL named one and named a nonsense one.

socks5_handshake_stream() had the same shape, range-checking only after the
sscanf had already wrapped a huge value into a plausible port; it now shares
the parser and its existing socks5_fail() rejection.

Covered by the dns self-test, which drives newhttp_addr() through the mocked
resolver: *addr_count separates a refused port (0, no resolve) from one merely
truncated or defaulted (2), so a "fall back to 80" fix would not pass.

Closes #614

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
2026-07-17 17:04:52 +02:00
27 changed files with 76 additions and 511 deletions

View File

@@ -1,6 +1,6 @@
AC_PREREQ([2.71])
AC_INIT([httrack], [3.49.13], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_INIT([httrack], [3.49.12], [roche+packaging@httrack.com], [httrack], [http://www.httrack.com/])
AC_COPYRIGHT([
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998-2015 Xavier Roche and other contributors
@@ -29,13 +29,11 @@ AC_CONFIG_SRCDIR(src/httrack.c)
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_HEADERS(config.h)
AM_INIT_AUTOMAKE([subdir-objects])
# 3:5:0: 3.49.13 leaves every installed struct layout and the exported symbol set
# untouched vs 3.49.12; the 2 GB (_stat64) and UTF-8 export widenings are _WIN32-only,
# and INTsys widened on POSIX (#600) but sits in no installed struct. Stays soname
# .so.3; bump revision. x32 alone changes layout (LLint was 4 bytes there, #524), and
# is not a release architecture.
# 3:4:0: 3.49.12 tail-appends one field (why_url) to httrackp and otherwise only
# deletes dead comments; no struct layout or exported signature broke vs 3.49.11,
# so it stays soname .so.3; bump revision.
# (3:0:0 was the htsblk mime-buffer widening, the ABI break that moved .so.2 -> .so.3.)
VERSION_INFO="3:5:0"
VERSION_INFO="3:4:0"
AM_MAINTAINER_MODE
AC_USE_SYSTEM_EXTENSIONS

15
debian/changelog vendored
View File

@@ -1,18 +1,3 @@
httrack (3.49.13-1) unstable; urgency=medium
* New upstream release: SOCKS5 and CONNECT proxy support, brotli and zstd
content decoding, --update data-loss fixes, and hardening of several
network-facing parsers; full list in history.txt.
* Add libbrotli-dev and libzstd-dev to Build-Depends for the new content
codings.
* libhttrack3 and libhttrack-dev no longer ship libhtsjava.so.3*: the
obsolete Java-applet parser was removed upstream. It was a dlopen plugin
with no reverse dependencies, so this is a plain file drop on upgrade;
libhttrack.so.3 itself is unchanged and needs no rename.
* Drop the htsjava attribution from debian/copyright.
-- Xavier Roche <xavier@debian.org> Fri, 17 Jul 2026 10:49:52 +0200
httrack (3.49.12-1) unstable; urgency=medium
* New upstream release: security and crawl-correctness fixes (remote stack

View File

@@ -4,31 +4,6 @@ HTTrack Website Copier release history:
This file lists all changes and fixes that have been made for HTTrack
3.49-13
+ New: SOCKS5 proxy support, with scheme-aware -P URLs (socks5://, socks5h://, connect://) and plain HTTP tunneled through a CONNECT-only proxy (#563, #564)
+ New: decode brotli and zstd content codings, advertised over TLS only as browsers do (#556)
+ New: webhttrack exposes the engine options added since 3.49-2, among them --cookies-file, --pause and --strip-query (#587)
+ Fixed: files of 2 GB or more were mishandled on Windows and on every 32-bit build (#569)
+ Fixed: --update destroyed a good local copy when the re-fetch returned an HTTP error, was aborted by -M/-E, failed to decode, or came in short (#176, #521, #557, #562)
+ Fixed: a self-redirect cookie wall was dropped instead of being re-fetched with the cookie (#15)
+ Fixed: a stalled TLS handshake ignored --timeout, and synchronous DNS resolution could wedge a crawl past --max-time (#607, #613)
+ Fixed: -M metered saved bytes rather than received volume, and overshot its cap under a slow server (#77, #520)
+ Fixed: several network-facing overflows and denial-of-service paths in the Content-Range, chunked-transfer, cookie, filter and ProxyTrack cache parsers
+ Fixed: a failed connect did not fall back to the next address on Windows (#579)
+ Fixed: -P took an out-of-range port as a garbage port, and scanned past an IPv6 literal's closing bracket (#598, #602)
+ Fixed: reject a port outside 1..65535 wherever one is parsed (a crawled URL, the htsserver and proxytrack listen arguments, an ftp:// URL), instead of letting a bare sscanf wrap a huge value into a plausible port and silently use it (#614, #626)
+ Fixed: a configured proxy still resolved and dialed the origin itself (#592)
+ Fixed: ~/ in the -O base path was never expanded (#270)
+ Fixed: a non-ASCII -O output path was double-encoded on Windows once argv became UTF-8 (#621)
+ Fixed: files under a non-ASCII project path were saved to a mangled directory on Windows (#217)
+ Fixed: --build-top-index (-%i) and --protocol (-@i) were taken for the -i continue flag, wiping the URL list and exiting on the usage screen (#615)
+ Fixed: webhttrack ignored LC_ALL/LC_MESSAGES and picked the wrong Chinese and Portuguese (#95)
+ Fixed: webhttrack wrote its base path and httrack.ini to the filesystem root when $HOME was empty (#625)
+ Fixed: crawls on a non-default port were slowed by a per-request pre-resolve (#181)
+ Changed: Windows builds moved to Visual Studio 2022 and OpenSSL 3.x, the VS2008 project files are retired, and the binaries carry a version resource
+ Changed: removed the obsolete Java-applet (.class) parser and the dead SWF module remnants
+ Changed: multiple internal hardening, test and CI improvements (Windows and macOS crawl suites, HTML-parser fuzzing, parallel make check)
3.49-12
+ New: --why explains which filter rule accepts or rejects a given URL, then exits (#505)
+ Fixed: links carrying raw UTF-8 bytes were fetched double-encoded and 404'd (#516)

View File

@@ -46,11 +46,9 @@ htsserver_LDFLAGS = $(AM_LDFLAGS) $(LDFLAGS_PIE)
lib_LTLIBRARIES = libhttrack.la
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h \
htsurlport.c htsurlport.h
htsserver_SOURCES = htsserver.c htsserver.h htsweb.c htsweb.h
proxytrack_SOURCES = proxy/main.c \
proxy/proxytrack.c proxy/store.c \
htsurlport.c htsurlport.h \
coucal/coucal.c htsmd5.c md5.c \
minizip/ioapi.c minizip/mztools.c minizip/unzip.c minizip/zip.c
@@ -60,7 +58,7 @@ whttrackrun_SCRIPTS = webhttrack
libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htscache_selftest.c htsdns_selftest.c htsselftest.c \
htscatchurl.c htsfilters.c htsftp.c htshash.c coucal/coucal.c \
htshelp.c htslib.c htsurlport.c htscoremain.c \
htshelp.c htslib.c htscoremain.c \
htsname.c htsrobots.c htstools.c htswizard.c \
htsalias.c htsthread.c htsindex.c htsbauth.c \
htsmd5.c htscodec.c htsproxy.c htszlib.c htswrap.c htsconcat.c \
@@ -71,7 +69,7 @@ libhttrack_la_SOURCES = htscore.c htsparse.c htsback.c htscache.c \
htsbasenet.h htsbauth.h htscache.h htscache_selftest.h htsdns_selftest.h htsselftest.h htscatchurl.h \
htsconfig.h htscore.h htsparse.h htscoremain.h htsdefines.h \
htsfilters.h htsftp.h htsglobal.h htshash.h coucal/coucal.h \
htshelp.h htsindex.h htslib.h htsurlport.h htsmd5.h \
htshelp.h htsindex.h htslib.h htsmd5.h \
htsmodules.h htsname.h htsnet.h htssniff.h \
htsopt.h htsrobots.h htsthread.h \
htstools.h htswizard.h htswrap.h htscodec.h htsproxy.h htszlib.h \

View File

@@ -425,10 +425,25 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
} // for
// path_html is already UTF-8 (argv is UTF-8 on Windows via
// hts_argv_utf8), so no re-encoding.
// Convert path to UTF-8
#ifdef _WIN32
{
char *const path =
hts_convertStringSystemToUTF8(StringBuff(opt->path_html),
(int) StringLength(opt->path_html));
if (path != NULL) {
StringCopy(opt->path_html_utf8, path);
free(path);
} else {
StringCopyN(opt->path_html_utf8, StringBuff(opt->path_html),
StringLength(opt->path_html));
}
}
#else
// Assume UTF-8 filesystem.
StringCopyN(opt->path_html_utf8, StringBuff(opt->path_html),
StringLength(opt->path_html));
#endif
/* if doit.log exists, or if new URL(s) defined,
then DO NOT load standard config files */
@@ -2471,12 +2486,9 @@ static int hts_main_internal(int argc, char **argv, httrackp * opt) {
}
} */
// vérifier existence de la structure (path_html/path_log are UTF-8, use
// the UTF-8 mkdir path)
structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_html), "/"));
structcheck_utf8(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt),
StringBuff(opt->path_log), "/"));
// vérifier existence de la structure
structcheck(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_html), "/"));
structcheck(fconcat(OPT_GET_BUFF(opt), OPT_GET_BUFF_SIZE(opt), StringBuff(opt->path_log), "/"));
// reprise/update
if (opt->cache) {

View File

@@ -248,13 +248,7 @@ int run_launch_ftp(FTPDownloadStruct * pStruct) {
// port
a = strchr(adr, ':'); // port
if (a) {
// folding a nonsense port into 1..65535 fetches one the link never named;
// an empty "host:" just means the default (#614)
if (a[1] != '\0' && !hts_parse_url_port(a + 1, &port)) {
snprintf(back->r.msg, sizeof(back->r.msg), "Invalid port: %s", a + 1);
back->r.statuscode = STATUSCODE_INVALID; // permanent, unlike a DNS miss
_HALT_FTP return 0;
}
sscanf(a + 1, "%d", &port);
strncatbuff(_adr, adr, (int) (a - adr));
} else
strcpybuff(_adr, adr);

View File

@@ -43,8 +43,8 @@ Please visit our Website: http://www.httrack.com
configure.ac, decoupled from these). VERSION is the display form, VERSIONID
the dotted numeric form, AFF_VERSION the short form shown in footers,
LIB_VERSION the data/cache format generation. */
#define HTTRACK_VERSION "3.49-13"
#define HTTRACK_VERSIONID "3.49.13"
#define HTTRACK_VERSION "3.49-12"
#define HTTRACK_VERSIONID "3.49.12"
#define HTTRACK_AFF_VERSION "3.x"
#define HTTRACK_LIB_VERSION "2.0"

View File

@@ -3751,6 +3751,19 @@ static int proxy_default_port(const char *arg) {
return hts_proxy_is_socks(arg) ? 1080 : 8080;
}
hts_boolean hts_parse_url_port(const char *a, int *port) {
char *end;
long p;
if (!isdigit((unsigned char) *a)) // strtol would eat a sign or leading space
return HTS_FALSE;
p = strtol(a, &end, 10);
if (*end != '\0' || p < 1 || p > 65535) // ERANGE lands out of range too
return HTS_FALSE;
*port = (int) p;
return HTS_TRUE;
}
// port "a" of -P argument "arg": digits fitting TCP's 1..65535, else the scheme
// default. Not sscanf("%d"): past INT_MAX it wraps to a garbage port (#602)
static int parse_proxy_port(const char *a, const char *arg) {

View File

@@ -61,7 +61,6 @@ typedef struct lien_adrfilsave lien_adrfilsave;
/* définitions globales */
#include "htsglobal.h"
#include "htsurlport.h"
/* basic net definitions */
#include "htsbase.h"
@@ -289,6 +288,12 @@ const char *strrchr_limit(const char *s, char c, const char *limit);
char *jump_protocol(char *source);
const char *jump_protocol_const(const char *source);
/* Parse a URL's port text "a" (after the ':', up to the end of the string):
TRUE and *port set for a bare decimal in 1..65535, else FALSE and *port left
alone. Not sscanf("%d"), which range-checks nothing and wraps past INT_MAX.
*/
hts_boolean hts_parse_url_port(const char *a, int *port);
/* Split a -P proxy argument "[scheme://][user:pass@]host[:port]" into the proxy
host string (scheme and any user:pass kept, for later stripping and auth),
written NUL-terminated into name[name_size] (truncated to fit), and the port

View File

@@ -2477,45 +2477,6 @@ static int st_makeindex(httrackp *opt, int argc, char **argv) {
return 0;
}
// hts_buildtopindex takes a system-charset path but verif_backblue() below it
// expects utf-8, so on Windows a non-ASCII project dir gets the gifs written to
// a mangled twin (issues #216/#217). argv[0] is a writable dir.
static int st_topindex(httrackp *opt, int argc, char **argv) {
char projdir[HTS_URLMAXSIZE];
char path[HTS_URLMAXSIZE + 16]; /* projdir plus a basename */
#ifdef _WIN32
/* the GUI hands hts_buildtopindex an ANSI path; mimic it. CP1252 'cafe' */
static const char *const projName = "caf\xE9";
#else
/* POSIX system charset is UTF-8 */
static const char *const projName = "caf\xC3\xA9";
#endif
assertf(argc >= 1);
snprintf(projdir, sizeof(projdir), "%s/%s", argv[0], projName);
/* structcheck(), not the utf-8 MKDIR family: same charset as buildtopindex */
snprintf(path, sizeof(path), "%s/", projdir);
assertf(structcheck(path) == 0);
/* returns 0 here: the dir holds no sub-project, only the gifs matter */
(void) hts_buildtopindex(opt, projdir, "");
/* the gifs must land in the project dir itself, not in a mangled sibling */
snprintf(path, sizeof(path), "%s/backblue.gif", projdir);
assertf(fexist(path));
/* raw unlink/rmdir: UNLINK is utf-8 on Windows, these paths aren't */
unlink(path);
snprintf(path, sizeof(path), "%s/fade.gif", projdir);
unlink(path);
snprintf(path, sizeof(path), "%s/index.html", projdir);
unlink(path);
rmdir(projdir);
printf("topindex self-test OK\n");
return 0;
}
/* Each inplace_escape_*() must equal escape_*() on a copy. */
static int st_inplace_escape(httrackp *opt, int argc, char **argv) {
/* >255 bytes forces the helper's malloct path, not the stack buffer */
@@ -3223,9 +3184,6 @@ static const struct selftest_entry {
{"useragent", "", "default User-Agent self-test", st_useragent},
{"makeindex", "[dir]", "hts_finish_makeindex footer/refresh self-test",
st_makeindex},
{"topindex", "[dir]",
"hts_buildtopindex charset handling of a non-ASCII project dir",
st_topindex},
{"inplace-escape", "", "inplace_escape_* vs escape_* equivalence self-test",
st_inplace_escape},
{"escape-room", "", "HT_ADD_HTMLESCAPED* reservation-factor self-test",

View File

@@ -164,12 +164,13 @@ HTS_UNUSED static int linputsoc_t(T_SOC soc, char *s, int max, int timeout) {
return -1;
}
/* Same contract as hts_gethome(), which is hidden and out of reach from here */
static const char *gethomedir(void) {
const char *home = getenv("HOME");
/* An empty $HOME would put the base path and httrack.ini at the root */
return strnotempty(home) ? home : ".";
if (home)
return home;
else
return ".";
}
static int linput_cpp(FILE * fp, char *s, int max) {
int rlen = 0;

View File

@@ -896,20 +896,7 @@ HTSEXT_API int hts_buildtopindex(httrackp * opt, const char *path,
if (fpo) {
find_handle h;
// générer gif. verif_backblue() is utf-8, but our path is the system
// charset, so on Windows convert it or the gifs land in a mangled twin
// dir (#217). Elsewhere the system charset is already utf-8.
#ifdef _WIN32
{
const char *const base = concat(catbuff, sizeof(catbuff), rpath, "/");
char *const base_utf8 =
hts_convertStringSystemToUTF8(base, strlen(base));
verif_backblue(opt, base_utf8 != NULL ? base_utf8 : base);
free(base_utf8);
}
#else
verif_backblue(opt, concat(catbuff, sizeof(catbuff), rpath, "/"));
#endif
verif_backblue(opt, concat(catbuff, sizeof(catbuff), rpath, "/")); // générer gif
// Header
hts_template_format(fpo, toptemplate_header,
"<!-- Mirror and index made by HTTrack Website Copier/"

View File

@@ -1,50 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: TCP port parser, shared by the engine, htsserver and */
/* proxytrack */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#include "htsurlport.h"
#include <ctype.h>
#include <stdlib.h>
hts_boolean hts_parse_url_port(const char *a, int *port) {
char *end;
long p;
if (!isdigit((unsigned char) *a)) // strtol would eat a sign or leading space
return HTS_FALSE;
p = strtol(a, &end, 10);
if (*end != '\0' || p < 1 || p > 65535) // ERANGE lands out of range too
return HTS_FALSE;
*port = (int) p;
return HTS_TRUE;
}

View File

@@ -1,45 +0,0 @@
/* ------------------------------------------------------------ */
/*
HTTrack Website Copier, Offline Browser for Windows and Unix
Copyright (C) 1998 Xavier Roche and other contributors
SPDX-License-Identifier: GPL-3.0-or-later
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Ethical use: we kindly ask that you NOT use this software to harvest email
addresses or to collect any other private information about people. Doing so
would dishonor our work and waste the many hours we have spent on it.
Please visit our Website: http://www.httrack.com
*/
/* ------------------------------------------------------------ */
/* File: TCP port parser, shared by the engine, htsserver and */
/* proxytrack */
/* Author: Xavier Roche */
/* ------------------------------------------------------------ */
#ifndef HTSURLPORT_DEFH
#define HTSURLPORT_DEFH
#include "htsglobal.h"
/* Parse the port text "a" (after the ':', up to the end of the string): TRUE
and *port set for a bare decimal in 1..65535, else FALSE and *port left
alone. Not sscanf("%d"), which range-checks nothing and wraps past INT_MAX.
Its own file so proxytrack, which does not link the library, can share it. */
hts_boolean hts_parse_url_port(const char *a, int *port);
#endif

View File

@@ -63,7 +63,6 @@ Please visit our Website: http://www.httrack.com
#include "md5.c"
#include "htsserver.h"
#include "htsurlport.h"
#include "htsweb.h"
#if USE_BEGINTHREAD==0
@@ -258,10 +257,8 @@ int main(int argc, char *argv[]) {
/* set commandline keys */
for(i = 2; i < argc; i += 2) {
if (strcmp(argv[i], "--port") == 0 && i + 1 < argc) {
// the range check ran after sscanf("%d") had wrapped a huge value into a
// plausible port, and listened there (#614). 0 (was the auto-pick) and
// 65535 (was refused, off by one) now both mean what they say.
if (!hts_parse_url_port(argv[i + 1], &defaultPort)) {
if (sscanf(argv[i + 1], "%d", &defaultPort) != 1 || defaultPort < 0
|| defaultPort >= 65535) {
fprintf(stderr, "couldn't set the port number to %s\n", argv[i + 1]);
return -1;
}

View File

@@ -121,7 +121,6 @@
<ClCompile Include="htshelp.c" />
<ClCompile Include="htsindex.c" />
<ClCompile Include="htslib.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="htsmodules.c" />
<ClCompile Include="htsname.c" />

View File

@@ -47,9 +47,7 @@ static void sig_brpipe(int code) {
}
#endif
// split a "host:port" listen argument; FALSE sends the caller to the usage
// screen. The port was unchecked, so a huge one wrapped into range (#614).
static hts_boolean scanHostPort(const char *str, char *host, int *port) {
static int scanHostPort(const char *str, char *host, int *port) {
char *pos = strrchr(str, ':');
if (pos != NULL) {
@@ -58,10 +56,12 @@ static hts_boolean scanHostPort(const char *str, char *host, int *port) {
if (n < 256) {
host[0] = '\0';
strncat(host, str, n);
return hts_parse_url_port(pos + 1, port);
if (sscanf(pos + 1, "%d", port) == 1) {
return 1;
}
}
}
return HTS_FALSE;
return 0;
}
int main(int argc, char *argv[]) {

View File

@@ -107,12 +107,13 @@ HTS_UNUSED static void proxytrack_print_log(const char *severity, const char *fo
"<!-- _-._.--._._-._.--._._-._.--._._-._.--._._-._.--._. -->\r\n" \
"<!-- End Disable IE Friendly HTTP Error Messages -->\r\n"
/* Same contract as hts_gethome(): proxytrack does not link libhttrack */
HTS_UNUSED static const char *gethomedir(void) {
const char *home = getenv("HOME");
/* An empty $HOME would resolve a relative path against the root */
return strnotempty(home) ? home : ".";
if (home)
return home;
else
return ".";
}
HTS_UNUSED static int linput(FILE * fp, char *s, int max) {

View File

@@ -99,7 +99,6 @@
<ClCompile Include="proxy\main.c" />
<ClCompile Include="proxy\proxytrack.c" />
<ClCompile Include="proxy\store.c" />
<ClCompile Include="htsurlport.c" />
<ClCompile Include="coucal\coucal.c" />
<ClCompile Include="htsmd5.c" />
<ClCompile Include="md5.c" />

View File

@@ -17,8 +17,8 @@
#endif
VS_VERSION_INFO VERSIONINFO
FILEVERSION 3, 49, 13, 0
PRODUCTVERSION 3, 49, 13, 0
FILEVERSION 3, 49, 12, 0
PRODUCTVERSION 3, 49, 12, 0
FILEFLAGSMASK VS_FFI_FILEFLAGSMASK
#ifdef _DEBUG
FILEFLAGS VS_FF_DEBUG
@@ -35,12 +35,12 @@ BEGIN
BEGIN
VALUE "CompanyName", "Xavier Roche"
VALUE "FileDescription", VER_FILE_DESCRIPTION
VALUE "FileVersion", "3.49.13"
VALUE "FileVersion", "3.49.12"
VALUE "InternalName", VER_ORIGINAL_FILENAME
VALUE "LegalCopyright", "Copyright (C) 1998-2026 Xavier Roche and other contributors. GNU GPL v3 or later."
VALUE "OriginalFilename", VER_ORIGINAL_FILENAME
VALUE "ProductName", "HTTrack Website Copier"
VALUE "ProductVersion", "3.49-13"
VALUE "ProductVersion", "3.49-12"
END
END
BLOCK "VarFileInfo"

View File

@@ -97,7 +97,6 @@
<ItemGroup>
<ClCompile Include="htsserver.c" />
<ClCompile Include="htsweb.c" />
<ClCompile Include="htsurlport.c" />
</ItemGroup>
<ItemGroup>

View File

@@ -1,12 +0,0 @@
#!/bin/bash
#
set -euo pipefail
# hts_buildtopindex takes a system-charset path but verif_backblue below it
# expects utf-8, mangling a non-ASCII project dir on Windows (#216, #217).
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
httrack -O /dev/null -#test=topindex "$dir" run |
grep -q "topindex self-test OK"

View File

@@ -1,88 +0,0 @@
#!/bin/bash
#
# htsserver's $HOME lookup, through the web UI it actually renders. An exported
# but empty $HOME must not resolve the default base path against the root.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
distdir=${top_srcdir:-$(cd "${testdir}/.." && pwd)}
distdir=$(cd "${distdir}" && pwd)
fail() {
echo "FAIL: $*" >&2
exit 1
}
command -v htsserver >/dev/null || fail "no htsserver in PATH"
# Mirror local-crawl.sh's skip-with-77: the Debian buildd chroot has no python3.
command -v python3 >/dev/null || {
echo "python3 not found; skipping" >&2
exit 77
}
srv=
cleanup() {
test -z "${srv}" || kill -9 "${srv}" 2>/dev/null || true
}
trap cleanup EXIT HUP INT QUIT PIPE TERM
log=$(mktemp)
trap 'cleanup; rm -f "${log}"' EXIT
# Ask htsserver for its rendered default base path, under the given $HOME
# ("-" = leave $HOME unset).
pathfor() {
local homeval=$1 port url got
port=$(python3 -c 'import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()')
: >"${log}"
(
# htsserver ignores SIGTERM/SIGTTOU handling from the harness shell
trap '' TERM TTOU
if test "${homeval}" = -; then unset HOME; else export HOME="${homeval}"; fi
exec htsserver "${distdir}/" --port "${port}" >"${log}" 2>&1
) &
srv=$!
for _ in $(seq 1 40); do
url=$(sed -n 's/^URL=//p' "${log}" 2>/dev/null) && test -n "${url}" && break
kill -0 "${srv}" 2>/dev/null || break
sleep 0.25
done
test -n "${url:-}" || fail "htsserver did not come up: $(cat "${log}")"
# The UI is served ISO-8859-1, so keep python out of text mode.
got=$(python3 -c 'import re, sys, urllib.request
body = urllib.request.urlopen(sys.argv[1] + "server/step2.html", timeout=20).read()
m = re.search(br"name=\"path\" value=\"([^\"]*)\"", body)
print(m.group(1).decode("latin-1") if m else "")' "${url}")
kill -9 "${srv}" 2>/dev/null || true
wait "${srv}" 2>/dev/null || true
srv=
test -n "${got}" || fail "no path field in the rendered step2.html"
echo "${got}"
}
check() {
local desc=$1 homeval=$2 want=$3 got
got=$(pathfor "${homeval}")
test "${got}" = "${want}" || fail "${desc}: want ${want}, got ${got}"
echo "ok: ${desc} -> ${got}"
}
# A real HOME must reach the page, or the two cases below are vacuous: they
# also pass when gethomedir() ignores the environment and always answers ".".
check 'a set HOME is used' /nonexistent/hts-home /nonexistent/hts-home/websites
# Exported but empty is the regression: getenv() returns "", not NULL, so a
# bare NULL check let the base path collapse onto /websites.
check 'an empty HOME' '' ./websites
check 'an unset HOME' - ./websites
echo "PASS"

View File

@@ -1,13 +0,0 @@
#!/bin/bash
#
# A non-ASCII -O output dir must receive the mirror, not a double-encoded twin
# (#621). argv is UTF-8 on Windows, so path_html was re-encoded as if it were
# the ANSI codepage, prefixing every saved file with a mangled directory. Only
# path_html is the non-ASCII "café"; path_log stays ASCII so the harness reads
# hts-log.txt/hts-cache normally and this isolates the -O encoding path.
: "${top_srcdir:=..}"
bash "$top_srcdir/tests/local-crawl.sh" --html-subdir 'café' --errors 0 --files 5 \
--found 'simple/basic.html' \
httrack 'BASEURL/simple/basic.html'

View File

@@ -1,127 +0,0 @@
#!/bin/bash
#
# Port parsing outside the crawled-URL path (#614): the htsserver --port listen
# port, proxytrack's host:port listen arguments, and an ftp:// URL port. All
# three took the port from sscanf("%d"), which range-checks nothing and wraps
# past INT_MAX, so a value naming no real port silently became a plausible one.
set -euo pipefail
testdir=$(cd "$(dirname "$0")" && pwd)
# run_with_timeout: timeout(1) is absent on macOS, and the listen servers block.
# shellcheck source=tests/testlib.sh
. "${testdir}/testlib.sh"
tmp=$(mktemp -d "${TMPDIR:-/tmp}/httrack_portsib.XXXXXX") || exit 1
trap 'rm -rf "$tmp"' EXIT HUP INT QUIT PIPE TERM
# A port the old code accepted only by wrapping: 4295009395 = 2^32 + 42099 and
# 4294967376 = 2^32 + 80 both fold to a plausible listen port. They are the
# load-bearing cases, since a value that merely exceeds 65535 while still
# fitting an int (65616) was already caught by the htsserver range check.
overflowing="4295009395"
# Values sscanf("%d") took as a port that no operator meant: a trailing
# character, a sign, leading whitespace, a decimal point. Each silently listened
# on the prefix it managed to scan.
malformed=("80x" "+80" " 80" "8.0" "")
# --- htsserver --port ------------------------------------------------------
# Its argv[1] is the html root; the option pairs follow. It blocks serving once
# a port is accepted, so bound it; we assert on the output, not the exit status.
websrv() {
run_with_timeout 3 htsserver "$tmp" --port "$1" >"$tmp/web.log" 2>&1 || true
}
web_refused() {
websrv "$1"
grep -q "couldn't set the port number" "$tmp/web.log" ||
! echo "FAIL: #614: htsserver --port '$1' not refused" || exit 1
# it must not have reached the listen socket on some other port
! grep -q "^URL=" "$tmp/web.log" ||
! echo "FAIL: #614: htsserver --port '$1' listened anyway" || exit 1
}
web_accepted() {
local port=$1
: >"$tmp/web.log"
# A bound server blocks, so stop it once it announces, not on a fixed
# deadline: the announce trails a local-hostname resolve that stalls on macOS,
# where a 3s kill landed mid-resolve, before the URL= line ever printed.
local had_m=
case "$-" in *m*) had_m=1 ;; esac
is_windows || set -m
htsserver "$tmp" --port "$port" >"$tmp/web.log" 2>&1 &
local pid=$!
test -n "$had_m" || is_windows || set +m
local waited=0
until grep -qE "^URL=http://.*:$port/|couldn't set the port number|Unable to initialize a temporary server" "$tmp/web.log"; do
test "$waited" -lt 20 || break
sleep 1
waited=$((waited + 1))
done
kill_tree "$pid"
wait "$pid" 2>/dev/null || true
! grep -q "couldn't set the port number" "$tmp/web.log" ||
! echo "FAIL: #614: htsserver --port '$port' wrongly refused" || exit 1
grep -qE "^URL=http://.*:$port/|Unable to initialize a temporary server" "$tmp/web.log" ||
! echo "FAIL: #614: htsserver --port '$port' never reached the listener" || exit 1
}
# this used to listen on 42099
web_refused "$overflowing"
for p in "${malformed[@]}" 0 65536 99999 -1; do
web_refused "$p"
done
# 65535 is a valid port the old "< 65535" bound refused
web_accepted 65535
# --- proxytrack <proxy-addr:port> <ICP-addr:port> --------------------------
# A bad argument falls through to the usage screen; it had no range check at
# all, so 65616 quietly listened on port 80. A valid one binds and blocks.
proxy_refused() {
run_with_timeout 3 proxytrack "127.0.0.1:$1" 127.0.0.1:3130 >"$tmp/pt.log" 2>&1 || true
grep -q "^usage:" "$tmp/pt.log" ||
! echo "FAIL: #614: proxytrack port '$1' not refused" || exit 1
}
proxy_accepted() {
run_with_timeout 3 proxytrack "127.0.0.1:$1" 127.0.0.1:3130 >"$tmp/pt.log" 2>&1 || true
! grep -q "^usage:" "$tmp/pt.log" ||
! echo "FAIL: #614: proxytrack port '$1' wrongly refused" || exit 1
}
for p in "${malformed[@]}" 65616 4294967376 "$overflowing" 0 -1; do
proxy_refused "$p"
done
proxy_accepted 45678
# --- ftp:// URL port -------------------------------------------------------
# A URL port, so it follows the crawled-URL policy: refuse the link rather than
# fold it into range. An ftp:// link in scanned HTML is not followed, so the
# seed has to come from the command line.
ftp_port() {
local out="$tmp/ftp"
rm -rf "$out"
run_with_timeout 30 httrack "ftp://127.0.0.1:$1/x" -O "$out" --quiet -n \
>/dev/null 2>&1 || true
grep -c "Invalid port: $1" "$out/hts-log.txt" 2>/dev/null || true
}
# 65616 truncated to 80 and really did connect there
for p in 65616 99999; do
test "$(ftp_port "$p")" -gt 0 ||
! echo "FAIL: #614: ftp:// port $p not refused" || exit 1
done
# a valid port still reaches the connect attempt
test "$(ftp_port 65535)" -eq 0 ||
! echo "FAIL: #614: ftp:// port 65535 wrongly refused" || exit 1
exit 0

View File

@@ -71,7 +71,6 @@ TESTS = \
01_engine-status.test \
01_engine-stripquery.test \
01_engine-strsafe.test \
01_engine-topindex.test \
01_engine-urlhack.test \
01_engine-unescape-bounds.test \
01_engine-useragent.test \
@@ -143,9 +142,6 @@ TESTS = \
59_local-tls-stall.test \
60_crawl-log-salvage.test \
61_webhttrack-locale.test \
62_lang-integrity.test \
63_webhttrack-home.test \
64_local-intl-outdir.test \
65_port-siblings.test
62_lang-integrity.test
CLEANFILES = check-network_sh.cache

View File

@@ -47,7 +47,6 @@ key="${testdir}/server.key"
tls=
verbose=
html_subdir=
rerun=
rerun_args=
rerun_dead=
@@ -133,13 +132,6 @@ while test "$pos" -lt "$nargs"; do
pos=$((pos + 1))
rerun_args="${args[$pos]}"
;;
--html-subdir)
# Mirror into "$out/NAME" (path_html) but keep logs/cache in "$out"
# (path_log): a non-ASCII NAME exercises the -O path encoding (#621)
# without routing the harness-read hts-log.txt through a non-ASCII path.
pos=$((pos + 1))
html_subdir="${args[$pos]}"
;;
--errors | --errors-content | --files)
audit+=("${args[$pos]}" "${args[$((pos + 1))]}")
pos=$((pos + 1))
@@ -215,21 +207,12 @@ test -n "$ver" || die "could not run httrack"
out="${tmpdir}/crawl"
mkdir "$out" || die "could not create $out"
# path_html holds the mirror + index; path_log holds hts-cache/hts-log.txt.
# Default: both are "$out". With --html-subdir, path_html becomes "$out/NAME"
# (the mirror root the audits inspect) while path_log stays "$out".
mirrorroot="$out"
odir="$out"
if test -n "$html_subdir"; then
mirrorroot="${out}/${html_subdir}"
odir="${mirrorroot},${out}"
fi
# Localhost is fast; disable the rate/bandwidth safety limits but keep a
# max-time backstop so a hang cannot wedge the suite.
declare -a moreargs=(--quiet --max-time=120 --timeout=30 --disable-security-limits --robots=0)
log="${tmpdir}/log"
info "running httrack ${hts[*]}"
httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" "${moreargs[@]}" "${hts[@]}" >"$log" 2>&1 &
httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" "${moreargs[@]}" "${hts[@]}" >"$log" 2>&1 &
crawlpid=$!
wait "$crawlpid"
crawlres=$?
@@ -246,7 +229,7 @@ grep -iE "^[0-9:]*[[:space:]]Error:" "${out}/hts-log.txt" >&2
# --- optional second pass: re-mirror into the same dir (cache/update path) ----
if test -n "$rerun"; then
info "re-running httrack (update pass)"
httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" \
httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" \
"${moreargs[@]}" "${hts[@]}" >"${log}.2" 2>&1 &
crawlpid=$!
wait "$crawlpid"
@@ -274,7 +257,7 @@ fi
if test -n "$rerun_args"; then
read -ra extra <<<"$rerun_args"
info "re-running httrack with ${rerun_args}"
httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" \
httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" \
"${moreargs[@]}" "${hts[@]}" "${extra[@]}" >"${log}.2" 2>&1 &
crawlpid=$!
wait "$crawlpid"
@@ -296,7 +279,7 @@ if test -n "$rerun_dead"; then
stop_server "$serverpid"
serverpid=
info "re-running httrack against the stopped server"
httrack -O "$odir" --user-agent="httrack $ver local ($(uname -mrs))" \
httrack -O "$out" --user-agent="httrack $ver local ($(uname -mrs))" \
"${moreargs[@]}" "${hts[@]}" >"${log}.dead" 2>&1 &
crawlpid=$!
wait "$crawlpid" || true
@@ -325,7 +308,7 @@ fi
# --- discover the single host root (127.0.0.1_<port> or 127.0.0.1) -----------
hostroot=
for cand in "${mirrorroot}/127.0.0.1_${port}" "${mirrorroot}/127.0.0.1"; do
for cand in "${out}/127.0.0.1_${port}" "${out}/127.0.0.1"; do
if test -d "$cand"; then
hostroot="$cand"
break