Skip to content

crawler.robotAgent

Matching is case-insensitive over the length of the name in a robots.txt file

Key: crawler.robotAgent
Type: String
Can be set in: collection.cfg

Description

This is a string that is used to examine any robots.txt files that are found to see if there are Funnelback specific restrictions.

The string is case-insensitive, so it will match

User-Agent: funnelback
User-Agent: FunnelBACK

The user-agent check is equals, not a substring so it will not match

User-Agent: funnelback crawler

Note: The webcrawler will still obey all rules in the robots.txt file which are meant to apply to all webcrawlers regardless of name.

Default Value

crawler.robotAgent=Funnelback

See Also

top

Funnelback logo
v15.22.0