A web collection is a collection of documents obtained from one or more web sites. Web collections contain HTML, PDF and Microsoft Office files that are gathered using a web crawler which discovers content by following the links it finds.
In order to avoid crawling the entire Internet the crawler uses a number of configuration options to determine which links it will follow and what web sites or domains it should limit its crawl to.
Web collection basics
The web crawler works by accessing a number of defined URLs in the seed list and extracting the links from these pages. A number of checks are performed on each link as it is extracted to determine if the link should be crawled. The link is compared against a set of include/exclude rules and a number of other settings (such as acceptable file types) determining if the link is suitable for inclusion in the index. A link that is deemed suitable is added to a list of uncrawled URLs called the crawl frontier.
The web crawler will continue run taking URLs off the crawl frontier, extracting links in the pages and checking the links against the include/exclude rules until it runs out of links in the crawl frontier, or an overall timeout is reached.
The crawler processes each URL it encounters against the various options in collection.cfg to determine if the URL will be included
+ or excluded
− from further processing:
Assuming you had the following options:
Then the following URLs will be included or excluded...
|/green/emerald||PASS||passes include, passes exclude|
|/green/olive||FAIL||passes include, fails exclude|
Regular expressions in include/exclude patterns
To express more advanced include or exclude patterns you can use regular expressions for the
exclude_patterns configuration options.
Regular expressions follow Perl 5 syntax and start with
regexp: followed by a compound regular expression in which each distinct include/exclude pattern is separated by the
| character. Regex and simple include/exclude patterns cannot be mixed within a single configuration option.
An example of the more advanced
regexp: form is:
which combines five alternative patterns into one overall pattern expression to match:
search?date=for example, to exclude calendars.
- HTTPS urls
- Dynamic content generated by URLs containing
- Dynamic content from CGI scripts.
- PDF files.
Note: regex special characters that appear in patterns must be escaped (e.g.
Excluding URLs during a running crawl
The crawler supports exclusion of URLs during a running crawl. The crawler.monitor_url_reject_list
collection.cfg parameter allows an administrator to specify additional URLs patterns to exclude while the crawler is running. These URL patterns will apply from the next crawler checkpoint and should be converted to a regular exclude pattern once the crawl completes.
Creating a web collection
Web collections require at a minimum a set of seed URLs and include/exclude patterns to be defined when setting up the collection.
To create a new web collection:
- Select web from the create collection menu in the administration interface.
- Enter a unique collection id to identify the collection. (eg.
- Enter a collection title which is a human readable identifier for the collection. (e.g. Example Organisation: public websites)
- Enter a set of start URLs.
- Enter include and exclude patterns.
- Save and update the collection.
Web crawler configuration
Web crawler logs
The web crawler writes a number of log files detailing various aspects of the crawl. See: web crawler logs