Skip to content

crawler.frontier_use_ip_mapping (collection.cfg setting)

Description

This flag controls whether hosts are mapped to frontiers using IP addresses during a web crawl.

You would set this to true if you know you might be crawling a server which uses a single IP address to serve a large number of web sites. When set to true this will trigger the use of an internal DNS cache to cache the IP address information.

Default value

crawler.frontier_use_ip_mapping=false

See also

top

Funnelback logo
v15.16.0