Warc storage
Introduction
Funnelback can store gathered content in a single WARC file for indexing, and this is the default storage option for most collection types. By compressing the content and avoiding the creation of a large number of files and directories for storage this approach saves on storage space. It also simplifies the transfer of gathered data to another machine in a multi-server setup.
Configuring WARC storage
You can configure the use of a WARC store by specifying the appropriate Java class in your collection.cfg file:
- crawler.classes.URLStore for web collections
- store.raw-bytes.class for filecopy collections
- store.xml.class for database, connector and directory collections