Skip to content

crawler.frontier_hosts

Lists of hosts running crawlers if performing a distributed web crawl.

Key: crawler.frontier_hosts
Type: List<String>
Can be set in: collection.cfg

Description

This option allows you to specify which hosts crawlers are running on when using a DistributedFrontier (distributed crawling). Each host will be responsible for its own subset of the web servers being crawled.

Default Value

(empty)

Examples

Specify the IP addresses of the hosts that the distributed crawl is running on.

crawler.frontier_hosts=12.67.89.8,12,67.89.9

See Also

top

Funnelback logo
v15.22.0