Skip to content

crawler.verbosity

Verbosity level (0-6) of crawler logs. Higher number results in more messages.

Key: crawler.verbosity
Type: Integer
Can be set in: collection.cfg

Description

This parameter specifies the verbosity setting for the webcrawler when it writes to its log files. The higher the number the higher the verbosity and the more detailed messages it will write out.

Verbosity levels

The verbosity is set to a value between 0 and 7. Each level includes messages from the lower numbered levels.

You may set this value higher if you are trying to debug a crawling issue, but normally it should be left at the default as otherwise it may generate very large log files.

  • Level 0: no output, apart from error messages.
  • Level 1: addresses of pages downloaded, redirects, aliases.
  • Level 2: crawler status and memory messages, pages added to frontier, sleep messages, file types, storage, cache, frontier, level messages.
  • Level 3: loading policy messages.
  • Level 4: HTTP header, network byte counts, unacceptable (rejected) files, robots.txt, parameter removal.
  • Level 5: absolute URLs, rule matches.
  • Level 6: extracted text from each HTML document parsed.
  • Level 7: regular expression patterns.

Default Value

crawler.verbosity=4

See Also

top

Funnelback logo
v15.22.0