Skip to content

crawler.allow_concurrent_in_crawl_form_interaction (collection.cfg setting)

Description

This experimental option allows concurrent form interaction processes to be prevented, which may work around problems in some web single sign-on environments.

When using in-crawl form interaction, the web crawler normally processes required form interaction while fetching the page which requires it, which may mean that multiple in-crawl form interaction processes occur concurrently.

In some web authentication environments, this may pose a problem because the web crawler shares cookies between all crawler threads. For example, if a separate SSO authentication server relies on cookies to control the post-authentication return page, two parallel logins may not behave as expected.

Please note that the locking necessary to prevent concurrent form interaction may have a detrimental effect on web crawler scalability/performance.

Note: this option is experimental it may be subject to major changes without backwards compatibility in future versions of Funnelback.

Default value

crawler.allow_concurrent_in_crawl_form_interaction=true

Examples

Prevent concurrent form interaction processing

crawler.allow_concurrent_in_crawl_form_interaction=false

See also

top

Funnelback logo
v15.16.0