Skip to content

crawler.max_individual_frontier_size (collection.cfg setting)

Description

This option limits the number of URLs stored in a frontier. If new URLs are discovered during a crawl but can not be queued because the frontier is full, they will be dropped (though they may be found again later in the crawl).

Default value

If not defined, it is unlimited.

Examples

crawler.max_individual_frontier_size=1000

See also

top

Funnelback logo
v15.16.0