Number of crawler threads which simultaneously crawl different hosts.
Can be set in: collection.cfg
This option specifies how many crawler threads will be created to download pages from a set of web sites. Note that with the default frontier, only one thread will access a given host at a time (i.e. if you have 10 threads but are only crawling one site then 9 threads will be idle).
If you have a small number of distinct web sites to crawl, then you might decide to reduce the number of threads:
Note: Having the extra threads should not have any performance impact on the system (as they will be idle if there is no site for them to crawl).