Skip to content

crawler.max_files_stored

Specifies the maximum number of files to download.

Key: crawler.max_files_stored
Type: Integer
Can be set in: collection.cfg

Description

This parameter is used to specify an optional value for the maximum number of files the webcrawler should download during the crawl. If this limit is specified and reached then the crawler will halt cleanly at that point, even if there are more URLs in the frontier.

Alternatively, if this value has been configured for a site or domain using the site profiles mechanism then the crawler will not download any more content from the relevant site or domain.

By default the value for this is empty i.e. no limit specified.

Default Value

crawler.max_files_stored= 

Examples

Limit crawl to 50,000 files:

crawler.max_files_stored=50000

See Also

top

Funnelback logo
v15.22.0