Implementing Pause and Resume Functionality with Checkpoints in Scrapling Spiders

Scrapling enables graceful pause and resume functionality by writing periodic checkpoints to disk, allowing spiders to recover their exact state—including the pending request queue and seen URL set—after an interruption.

Implementing pause and resume functionality with checkpoints in Scrapling spiders requires only enabling the built-in checkpoint system by providing a crawldir directory. The Scrapling repository handles all state serialization automatically, making your crawls resilient to crashes, network interruptions, or intentional pauses.

How Scrapling's Checkpoint System Works

The checkpoint system in scrapling/spiders/engine.py operates as an opt-in persistence layer. When enabled, the CrawlerEngine periodically serializes the scheduler's state—including all pending Request objects and the deduplication set—to an atomic pickle file. On startup, the engine detects existing checkpoint data in the specified directory and restores the spider to its exact pre-interruption state, ensuring no URLs are re-crawled and no requests are lost.

Enabling Checkpoints in Your Spider

To activate pause and resume functionality, instantiate your spider with the crawldir parameter:

from scrapling.spiders.spider import Spider

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

# Enable checkpoints by providing a directory path

spider = MySpider(crawldir="./crawl_state")

The crawldir argument triggers CrawlerEngine to enable its checkpoint system (self._checkpoint_system_enabled = True) and prepares the directory to store checkpoint.pkl files.

Core Components of the Pause and Resume System

Spider.pause() and Signal Handling

The user-facing pause mechanism resides in scrapling/spiders/spider.py. The Spider.pause() method (lines 18-24) delegates to engine.request_pause(), setting an internal flag for graceful shutdown.

For handling external interruptions, the spider installs a custom SIGINT handler (lines 25-34) that translates the first Ctrl+C into a pause request. A second rapid Ctrl+C forces immediate termination by setting _force_stop in the engine.

CrawlerEngine Pause Logic

The CrawlerEngine.request_pause() method in scrapling/spiders/engine.py (lines 66-83) manages the pause state machine. The first invocation sets _pause_requested = True, triggering the main loop to enter shutdown sequence. A second call sets _force_stop, causing the task group to cancel immediately without waiting for active downloads to complete.

Checkpoint Serialization

The CheckpointManager class in scrapling/spiders/checkpoint.py (lines 28-38) handles atomic disk writes. It serializes a CheckpointData tuple containing the pending request queue and seen URL set using Python's pickle module. To prevent corruption, it writes to a temporary file (checkpoint.tmp) before atomically renaming it to checkpoint.pkl.

Scheduler State Management

The scheduler in scrapling/spiders/scheduler.py provides snapshot() and restore() methods (lines 60-78) that capture the priority queue and deduplication set without draining pending requests. During restoration, the engine reattaches callbacks to each request using request._restore_callback(self.spider) before injecting them back into the scheduler.

Step-by-Step Implementation Guide

Basic Spider with Checkpoint Support

Create a spider that can pause and resume:

# example_spider.py

from scrapling.spiders.spider import Spider
from scrapling.spiders.request import Request

class ResumableSpider(Spider):
    name = "resumable"
    start_urls = ["https://quotes.toscrape.com/"]
    allowed_domains = {"quotes.toscrape.com"}

    async def parse(self, response):
        # Extract quotes

        for quote in response.html.select("div.quote"):
            yield {
                "text": quote.select_one("span.text").text,
                "author": quote.select_one("small.author").text
            }
        
        # Follow pagination

        next_page = response.html.select_one("li.next a")
        if next_page:
            yield Request(next_page.attrs["href"])

Running with Checkpoint Directory

Execute the spider with a persistent directory:

# First run - creates checkpoint every 5 minutes (default)

python -m scrapling example_spider.ResumableSpider --crawldir ./crawl_data

Press Ctrl+C once to trigger a graceful pause. The engine saves the checkpoint to ./crawl_data/checkpoint.pkl.

Resuming from Interruption

Restart the spider using the same directory:

# Resume from checkpoint

python -m scrapling example_spider.ResumableSpider --crawldir ./crawl_data

The CrawlerEngine._restore_from_checkpoint() method loads the pending queue and seen set, passing resuming=True to spider.on_start().

Explicit Pause from Spider Code

Trigger a pause programmatically within your parsing logic:

class ConditionalSpider(Spider):
    # ... configuration ...

    
    async def parse(self, response):
        if response.url.endswith("stop-here"):
            self.pause()  # Calls engine.request_pause()

            return
        
        # Normal processing continues...

Custom Checkpoint Intervals

Adjust how frequently the engine writes state to disk:

# Checkpoints every 60 seconds instead of default 300

spider = ResumableSpider(
    crawldir="./crawl_state",
    interval=60.0
)

The interval parameter controls the _is_checkpoint_time() check in the engine's main loop.

Summary

• Enable checkpoints by providing a crawldir directory when instantiating your Scrapling spider.
• Graceful pauses are triggered via Ctrl+C (handled by the SIGINT handler in spider.py) or programmatically via self.pause().
• Atomic serialization uses checkpoint.pkl written via temporary files to prevent corruption during unexpected shutdowns.
• State restoration automatically occurs on startup when CrawlerEngine detects an existing checkpoint, restoring the scheduler queue and deduplication set without reprocessing seen URLs.
• Lifecycle hooks allow spiders to detect resumes via the on_start(resuming=True) callback in spider.py.

Frequently Asked Questions

How does Scrapling prevent duplicate URL crawling after resuming from a checkpoint?

Scrapling stores the deduplication set (seen URLs) alongside the pending request queue in checkpoint.pkl. When CrawlerEngine._restore_from_checkpoint() loads the checkpoint in engine.py (lines 2-20), it restores both the scheduler's priority queue and the seen set via Scheduler.restore() in scheduler.py (lines 60-78). This ensures URLs processed before the pause remain filtered out after resumption.

What happens if I press Ctrl+C multiple times during a crawl?

The first Ctrl+C triggers the custom SIGINT handler in spider.py (lines 25-34), which calls engine.request_pause() to initiate a graceful pause. The engine waits for active downloads to complete, saves a checkpoint, and exits cleanly. If you press Ctrl+C again before the graceful shutdown completes, request_pause() detects the second call (lines 66-83 in engine.py) and sets _force_stop, causing the task group to cancel immediately without saving a checkpoint.

Can I adjust how frequently Scrapling saves checkpoints during a long crawl?

Yes, you can customize the checkpoint interval by passing the interval parameter when instantiating your spider. The default is 300 seconds (5 minutes), but you can reduce this to 60 seconds or any float value: spider = MySpider(crawldir="./state", interval=60.0). This value controls the _is_checkpoint_time() check in the engine's main loop (engine.py lines 84-90), triggering _save_checkpoint() when the elapsed time exceeds your specified interval.

Does the checkpoint system preserve callback functions for pending requests?

Yes, Scrapling preserves callbacks through a rehydration process during restoration. When CrawlerEngine restores requests from the checkpoint in engine.py (lines 2-20), it calls request._restore_callback(self.spider) on each pending request. This method reattaches the original parsing method to the request object, ensuring that when the scheduler later processes that URL, it invokes the correct callback with the spider instance bound correctly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}