mirror of
https://github.com/xroche/httrack.git
synced 2026-07-23 01:01:41 +03:00
Compare commits
1 Commits
feat/webht
...
prose-typo
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0f7b4ed66c |
@@ -217,7 +217,7 @@ school or
|
||||
shows. They might do that because they are connected through expensive modem connection,
|
||||
or because they would like to consult pages while travelling, or archive sites that may be
|
||||
removed
|
||||
one day, make some data mining, comiling information ("if only I could find this
|
||||
one day, make some data mining, compiling information ("if only I could find this
|
||||
website I saw one day.."). <br>
|
||||
There are many good reasons to mirror websites, and this helps many good people.<br>
|
||||
As a webmaster, you might be interested to use such tools, too: test broken links, move a
|
||||
@@ -229,7 +229,7 @@ test the webserver response and performances, index it..<br>
|
||||
Anyway, bandwidth abuse can be a problem. If your site is regularly "clobbered"
|
||||
by evil downloaders, you have <br>
|
||||
various solutions. You have radical solutions, and intermediate solutions. I strongly
|
||||
recomment not to use<br>
|
||||
recommend not to use<br>
|
||||
radical solutions, because of the previous remarks (good people often mirror websites).<br>
|
||||
<br>
|
||||
In general, for all solutions,<br>
|
||||
@@ -244,7 +244,7 @@ or, to be extreme: if you unplug the wire, there will be no bandwidth abuse<br>
|
||||
Good: Will work with good people. Many good people just don't KNOW that they can slow down
|
||||
a network.<br>
|
||||
Bad: Will **only** work with good people<br>
|
||||
How to do: Obvious - place a note, a warning, an article, a draw, a poeme or whatever you
|
||||
How to do: Obvious - place a note, a warning, an article, a draw, a poem or whatever you
|
||||
want<br>
|
||||
<br>
|
||||
</li><li>Use "robots.txt" file<br>
|
||||
@@ -266,7 +266,7 @@ Good: Efficient<br>
|
||||
Bad: Multiple users behind proxies will be slow down, not really easy to setup<br>
|
||||
How to do: Depends on webserver. Might be done with low-level IP rules (QoS)<br>
|
||||
<br>
|
||||
</li><li>Priorize small files, against large files<br>
|
||||
</li><li>Prioritize small files, against large files<br>
|
||||
Good: Efficient if large files are the cause of abuse<br>
|
||||
Bad: Not always efficient<br>
|
||||
How to do: Depends on the webserver<br>
|
||||
@@ -283,7 +283,7 @@ How to do: Use routine QoS (fair queuing), or webserver options<br>
|
||||
<br>
|
||||
</li><li>Use technical tricks (like javascript) to hide URLs<br>
|
||||
Good: Efficient<br>
|
||||
Bad: The most efficient tricks will also cause your website to he heavy, and not
|
||||
Bad: The most efficient tricks will also cause your website to be heavy, and not
|
||||
user-friendly (and therefore less attractive, even for surfing users). Remember: clients
|
||||
or visitors might want to consult offline your website. Advanced users will also be still
|
||||
able to note the URLs and catch them. Will not work on non-javascript browsers. It will
|
||||
@@ -335,7 +335,7 @@ Example: Use things like
|
||||
</li><li>Use technical tricks to temporarily ban IPs<br>
|
||||
Good: Efficient<br>
|
||||
Bad: Radical (your site will only be available online for all users), not easy to setup<br>
|
||||
How to to: Create fake links with "killing" targets<br>
|
||||
How to do: Create fake links with "killing" targets<br>
|
||||
Example: Use things like <a href="killme.cgi"><nothing></a>
|
||||
(again an example in php:)<br>
|
||||
<tt>
|
||||
@@ -417,7 +417,7 @@ Example:<br>
|
||||
</li><li>Another one is to create images of emails<br>
|
||||
Good: Efficient, does not require javascript<br>
|
||||
Bad: There is still the problem of the link (mailto:), images are bigger than text, and it can cause problems for blind people (a good solution is use an ALT attribute with the email written like "smith at mycompany dot com")<br>
|
||||
How to do: Not so obvious of you do not want to create images by yourself<br>
|
||||
How to do: Not so obvious if you do not want to create images by yourself<br>
|
||||
Example: (php, Unix)<br>
|
||||
|
||||
<tt>
|
||||
@@ -491,7 +491,7 @@ echo <br>
|
||||
</li><li>You can also create temporary email aliases, each week, for all users<br>
|
||||
Good: Efficient, and you can give your real email in your reply-to address<br>
|
||||
Bad: Temporary emails<br>
|
||||
How to do: Not so hard todo<br>
|
||||
How to do: Not so hard to do<br>
|
||||
Example: (script & php, Unix)<br>
|
||||
|
||||
<tt>
|
||||
|
||||
@@ -111,7 +111,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
starts links, the default mode is to mirror these links - i.e. if one of your start page is
|
||||
www.example.com/test/index.html, all links starting with www.example.com/test/ will be
|
||||
accepted. But links directly in www.example.com/.. will not be accepted, however, because
|
||||
they are in a higher strcuture. This prevent HTTrack from mirroring the whole site. (All
|
||||
they are in a higher structure. This prevent HTTrack from mirroring the whole site. (All
|
||||
files in structure levels equal or lower than the primary links will be retrieved.)<br>
|
||||
</i>
|
||||
<br>
|
||||
@@ -127,13 +127,15 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
an authorization filter, like <b><tt>+*.gif</tt></b>. The pattern is a plus (this one: <b><tt>+</tt></b>),
|
||||
followed by a pattern composed of letters and wildcards (this one: <b><tt>*</tt></b>).
|
||||
<br><br>
|
||||
To forbide a family of links, define
|
||||
To forbid a family of links, define
|
||||
an authorization filter, like <b><tt>-*.gif</tt></b>. The pattern is a dash (this one: <b><tt>-</tt></b>),
|
||||
followed by a the same kind of pattern as for the authorization filter.
|
||||
<br><br>
|
||||
Example: +*.gif will accept all files finished by .gif<br>
|
||||
Example: -*.gif will refuse all files finished by .gif<br>
|
||||
<br>
|
||||
To see which rule accepted or blocked a given URL, run HTTrack with the <b><tt>--why</tt></b> (<b><tt>-%Y</tt></b>) option, described in <a href="httrack.man.html#OPTIONS">the manual page</a>.<br>
|
||||
<br>
|
||||
|
||||
<p>
|
||||
<h4>Scan rules based on size (e.g. accept or refuse files bigger/smaller than a certain size)</h4>
|
||||
@@ -143,7 +145,7 @@ See also: The <a href="faq.html#VF1">FAQ</a><br>
|
||||
size to ensure that you won't reach a defined limit.
|
||||
|
||||
Example: You may want to accept all files on the domain www.example.com, using '+www.example.com/*',
|
||||
including gif files inside this domain and outside (eternal images), but not take to large images,
|
||||
including gif files inside this domain and outside (external images), but not take to large images,
|
||||
or too small ones (thumbnails)<br>
|
||||
Excluding gif images smaller than 5KB and images larger than 100KB is therefore a good option;
|
||||
+www.example.com +*.gif -*.gif*[<5] -*.gif*[>100]
|
||||
|
||||
Reference in New Issue
Block a user