# Implementing Pause and Resume Functionality with Checkpoints in Scrapling Spiders

> Implement pause and resume in Scrapling spiders using checkpoints. Save and recover spider state, including request queues and seen URLs, for seamless interruptions.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Scrapling enables graceful pause and resume functionality by writing periodic checkpoints to disk, allowing spiders to recover their exact state—including the pending request queue and seen URL set—after an interruption.**

Implementing pause and resume functionality with checkpoints in Scrapling spiders requires only enabling the built-in checkpoint system by providing a `crawldir` directory. The Scrapling repository handles all state serialization automatically, making your crawls resilient to crashes, network interruptions, or intentional pauses.

## How Scrapling's Checkpoint System Works

The checkpoint system in [`scrapling/spiders/engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py) operates as an opt-in persistence layer. When enabled, the `CrawlerEngine` periodically serializes the scheduler's state—including all pending `Request` objects and the deduplication set—to an atomic pickle file. On startup, the engine detects existing checkpoint data in the specified directory and restores the spider to its exact pre-interruption state, ensuring no URLs are re-crawled and no requests are lost.

## Enabling Checkpoints in Your Spider

To activate pause and resume functionality, instantiate your spider with the `crawldir` parameter:

```python
from scrapling.spiders.spider import Spider

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

# Enable checkpoints by providing a directory path

spider = MySpider(crawldir="./crawl_state")

```

The `crawldir` argument triggers `CrawlerEngine` to enable its checkpoint system (`self._checkpoint_system_enabled = True`) and prepares the directory to store `checkpoint.pkl` files.

## Core Components of the Pause and Resume System

### Spider.pause() and Signal Handling

The user-facing pause mechanism resides in [`scrapling/spiders/spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py). The `Spider.pause()` method (lines 18-24) delegates to `engine.request_pause()`, setting an internal flag for graceful shutdown.

For handling external interruptions, the spider installs a custom SIGINT handler (lines 25-34) that translates the first `Ctrl+C` into a pause request. A second rapid `Ctrl+C` forces immediate termination by setting `_force_stop` in the engine.

### CrawlerEngine Pause Logic

The `CrawlerEngine.request_pause()` method in [`scrapling/spiders/engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py) (lines 66-83) manages the pause state machine. The first invocation sets `_pause_requested = True`, triggering the main loop to enter shutdown sequence. A second call sets `_force_stop`, causing the task group to cancel immediately without waiting for active downloads to complete.

### Checkpoint Serialization

The `CheckpointManager` class in [`scrapling/spiders/checkpoint.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/checkpoint.py) (lines 28-38) handles atomic disk writes. It serializes a `CheckpointData` tuple containing the pending request queue and seen URL set using Python's `pickle` module. To prevent corruption, it writes to a temporary file (`checkpoint.tmp`) before atomically renaming it to `checkpoint.pkl`.

### Scheduler State Management

The scheduler in [`scrapling/spiders/scheduler.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/scheduler.py) provides `snapshot()` and `restore()` methods (lines 60-78) that capture the priority queue and deduplication set without draining pending requests. During restoration, the engine reattaches callbacks to each request using `request._restore_callback(self.spider)` before injecting them back into the scheduler.

## Step-by-Step Implementation Guide

### Basic Spider with Checkpoint Support

Create a spider that can pause and resume:

```python

# example_spider.py

from scrapling.spiders.spider import Spider
from scrapling.spiders.request import Request

class ResumableSpider(Spider):
    name = "resumable"
    start_urls = ["https://quotes.toscrape.com/"]
    allowed_domains = {"quotes.toscrape.com"}

    async def parse(self, response):
        # Extract quotes

        for quote in response.html.select("div.quote"):
            yield {
                "text": quote.select_one("span.text").text,
                "author": quote.select_one("small.author").text
            }
        
        # Follow pagination

        next_page = response.html.select_one("li.next a")
        if next_page:
            yield Request(next_page.attrs["href"])

```

### Running with Checkpoint Directory

Execute the spider with a persistent directory:

```bash

# First run - creates checkpoint every 5 minutes (default)

python -m scrapling example_spider.ResumableSpider --crawldir ./crawl_data

```

Press `Ctrl+C` once to trigger a graceful pause. The engine saves the checkpoint to `./crawl_data/checkpoint.pkl`.

### Resuming from Interruption

Restart the spider using the same directory:

```bash

# Resume from checkpoint

python -m scrapling example_spider.ResumableSpider --crawldir ./crawl_data

```

The `CrawlerEngine._restore_from_checkpoint()` method loads the pending queue and seen set, passing `resuming=True` to `spider.on_start()`.

### Explicit Pause from Spider Code

Trigger a pause programmatically within your parsing logic:

```python
class ConditionalSpider(Spider):
    # ... configuration ...

    
    async def parse(self, response):
        if response.url.endswith("stop-here"):
            self.pause()  # Calls engine.request_pause()

            return
        
        # Normal processing continues...

```

### Custom Checkpoint Intervals

Adjust how frequently the engine writes state to disk:

```python

# Checkpoints every 60 seconds instead of default 300

spider = ResumableSpider(
    crawldir="./crawl_state",
    interval=60.0
)

```

The `interval` parameter controls the `_is_checkpoint_time()` check in the engine's main loop.

## Summary

- **Enable checkpoints** by providing a `crawldir` directory when instantiating your Scrapling spider.
- **Graceful pauses** are triggered via `Ctrl+C` (handled by the SIGINT handler in [`spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/spider.py)) or programmatically via `self.pause()`.
- **Atomic serialization** uses `checkpoint.pkl` written via temporary files to prevent corruption during unexpected shutdowns.
- **State restoration** automatically occurs on startup when `CrawlerEngine` detects an existing checkpoint, restoring the scheduler queue and deduplication set without reprocessing seen URLs.
- **Lifecycle hooks** allow spiders to detect resumes via the `on_start(resuming=True)` callback in [`spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/spider.py).

## Frequently Asked Questions

### How does Scrapling prevent duplicate URL crawling after resuming from a checkpoint?

Scrapling stores the **deduplication set** (seen URLs) alongside the pending request queue in `checkpoint.pkl`. When `CrawlerEngine._restore_from_checkpoint()` loads the checkpoint in [`engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/engine.py) (lines 2-20), it restores both the scheduler's priority queue and the seen set via `Scheduler.restore()` in [`scheduler.py`](https://github.com/D4Vinci/Scrapling/blob/main/scheduler.py) (lines 60-78). This ensures URLs processed before the pause remain filtered out after resumption.

### What happens if I press Ctrl+C multiple times during a crawl?

The first `Ctrl+C` triggers the custom SIGINT handler in [`spider.py`](https://github.com/D4Vinci/Scrapling/blob/main/spider.py) (lines 25-34), which calls `engine.request_pause()` to initiate a **graceful pause**. The engine waits for active downloads to complete, saves a checkpoint, and exits cleanly. If you press `Ctrl+C` again before the graceful shutdown completes, `request_pause()` detects the second call (lines 66-83 in [`engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/engine.py)) and sets `_force_stop`, causing the task group to cancel immediately without saving a checkpoint.

### Can I adjust how frequently Scrapling saves checkpoints during a long crawl?

Yes, you can customize the checkpoint interval by passing the `interval` parameter when instantiating your spider. The default is **300 seconds** (5 minutes), but you can reduce this to 60 seconds or any float value: `spider = MySpider(crawldir="./state", interval=60.0)`. This value controls the `_is_checkpoint_time()` check in the engine's main loop ([`engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/engine.py) lines 84-90), triggering `_save_checkpoint()` when the elapsed time exceeds your specified interval.

### Does the checkpoint system preserve callback functions for pending requests?

Yes, Scrapling preserves callbacks through a **rehydration process** during restoration. When `CrawlerEngine` restores requests from the checkpoint in [`engine.py`](https://github.com/D4Vinci/Scrapling/blob/main/engine.py) (lines 2-20), it calls `request._restore_callback(self.spider)` on each pending request. This method reattaches the original parsing method to the request object, ensuring that when the scheduler later processes that URL, it invokes the correct callback with the spider instance bound correctly.