Specifies the maximum length a URL can be in order for it to be crawled.
Can be set in: collection.cfg
The crawler will ignore all URLs that are larger than this option's value. The check is based on the number of characters in the URL e.g. if the limit is 256 then any URL with more than 256 characters will be ignored. Typically if the URL is too long, it is likely to be a crawler trap.