Skip to content

crawler.max_url_repeating_elements

A URL with more than this many repeating elements (directories) will be ignored.

Key: crawler.max_url_repeating_elements
Type: Integer
Can be set in: collection.cfg

Description

The crawler will ignore all URLs that contain more than this number of repeating elements. For example, the following url:

http://example.com/a/a/a/a/a/a/

will be ignored if the default limit of 5 is being used, as it has 6 repeating "a" elements or directories. This check is used to guard against crawler traps and badly configured web servers.

Default Value

crawler.max_url_repeating_elements=5

See Also

top

Funnelback logo
v15.24.0