crawler.monitor_checkpoint_interval
Time interval at which to checkpoint (seconds).
Key: crawler.monitor_checkpoint_interval
Type: Integer
Can be set in: collection.cfg
Description
This parameter specifies the time interval at which to checkpoint, in seconds. For example, a value of 1800 would mean the webcrawler would checkpoint every 30 minutes. Checkpointing involves writing critical data structures out to disk so that the webcrawler can be restarted from that point if required.
Default Value
crawler.monitor_checkpoint_interval=1800