crawler.max_individual_frontier_size
Specifies the maximum size of an individual frontier.
Key: crawler.max_individual_frontier_size
Type: Integer
Can be set in: collection.cfg
Description
This option limits the number of URLs stored in a frontier. If new URLs are discovered during a crawl but can not be queued because the frontier is full, they will be dropped (though they may be found again later in the crawl).
Default Value
If not defined, it is unlimited.
Examples
crawler.max_individual_frontier_size=1000