Skip to content

crawler.frontier_num_top_level_dirs

Specifies the number of top level directories to store disk based frontier files in.

Key: crawler.frontier_num_top_level_dirs
Type: Integer
Can be set in: collection.cfg

Description

This optional parameter can be used to specify the number of top level directories to use when storing files that are required for a disk based frontier.

Default Value

(empty)

Examples

If you are performing a very large crawl you may run into filesystem limits on the number of files allowed in a single directory. To get around this you might split the frontier files among 100 top level directories by setting:

crawler.frontier_num_top_level_dirs=100

See Also

top

Funnelback logo
v15.24.0