Skip to content

crawler.robotAgent (collection.cfg setting)

Description

This is a string that is used to examine any robots.txt files that are found to see if there are Funnelback specific restrictions.

The string is case-insensitive, so it will match

User-Agent: funnelback
User-Agent: FunnelBACK

The user-agent check is equals, not a substring so it will not match

User-Agent: funnelback crawler

Note: The webcrawler will still obey all rules in the robots.txt file which are meant to apply to all webcrawlers regardless of name.

Default value

crawler.robotAgent=FunnelBack

See also

top

Funnelback logo
v15.16.0