Skip to content

crawler.user_agent (collection.cfg setting)

Description

This parameter specifies the user agent string used by the web crawler when making HTTP(S) requests.

Default value

crawler.user_agent=Mozilla/5.0

This default browser-based user-agent is used to maximise the chances that we will get content from websites which return different content depending on browser type.

Some sites will return_"Your browser doesn't support frames_ as a response if their code doesn't see a specific user-agent like Mozilla/5.0, and the Funnelback web crawler would then get no content from the site.

Example

If you are crawling other people's web sites, then it is proper "netiquette" to identify yourself:

crawler.user_agent=Mozilla/5.0 (compatible; FunnelBack)

You may also wish to use this more specific string to identify the Funnelback webcrawler in your web server access logs.

See also

top

Funnelback logo
v15.18.0