crawler.max_url_length
Specifies the maximum length a URL can be in order for it to be crawled.
Key: crawler.max_url_length
Type: Integer
Can be set in: collection.cfg
Description
The crawler will ignore all URLs that are larger than this option's value. The check is based on the number of characters in the URL e.g. if the limit is 256 then any URL with more than 256 characters will be ignored. Typically if the URL is too long, it is likely to be a crawler trap.
Default Value
crawler.max_url_length=256