crawler.user_agent (collection.cfg setting)
This parameter specifies the user agent string used by the web crawler when making HTTP(S) requests.
This default browser-based user-agent is used to maximise the chances that we will get content from websites which return different content depending on browser type.
Some sites will return_"Your browser doesn't support frames_ as a response if their code doesn't see a specific user-agent like
Mozilla/5.0, and the Funnelback web crawler would then get no content from the site.
If you are crawling other people's web sites, then it is proper "netiquette" to identify yourself:
crawler.user_agent=Mozilla/5.0 (compatible; FunnelBack)
You may also wish to use this more specific string to identify the Funnelback webcrawler in your web server access logs.