crawler.request_delay (collection.cfg setting)
This setting specifies the number of milliseconds to wait before making another request to a single host.
Note: The default behaviour of the webcrawler is to dynamically calculate how long to delay based on the time taken for the previous request to a given web server. This is done so that fast web servers can be crawled rapidly while slow servers are given more time to respond (e.g. 10 x
This means that the actual request delay may sometimes be less than this fixed setting. If you wish to ensure that this fixed setting is always used you will need to set crawler.monitor_delay_type to "fixed".
If you have an internal server and are crawling it out of hours, then you can reduce the request delay to speed up the crawl:
This parameter is similar to the
crawler.monitor.* parameters in that it will be checked by the crawler monitor during the crawl. This means that if you modify this value during a running crawl the new value will be used for subsequent requests.