Skip to content

crawler.frontier_use_ip_mapping

Whether to map hosts to frontiers based on IP address.

Key: crawler.frontier_use_ip_mapping
Type: Boolean
Can be set in: collection.cfg

Description

This flag controls whether hosts are mapped to frontiers using IP addresses during a web crawl.

You would set this to true if you know you might be crawling a server which uses a single IP address to serve a large number of web sites. When set to true this will trigger the use of an internal DNS cache to cache the IP address information.

Default Value

crawler.frontier_use_ip_mapping=false

See Also

top

Funnelback logo
v15.22.0