# Crawl Hooks vs Snapshot Hooks in abx-dl: Understanding the Plugin Execution Pipeline

> Understand Crawl hooks vs Snapshot hooks in abx-dl. Learn how these plugin hooks prepare your crawl environment and archive individual URLs for effective content extraction.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: internals
- Published: 2026-02-25

---

**Crawl hooks run once per crawl session to prepare the environment, while snapshot hooks execute for every URL to extract and archive content.**

The `archivebox/abx-dl` repository implements a two-phase plugin execution pipeline that separates environment setup from content extraction. Understanding the distinction between these hook types is essential for developing plugins that integrate correctly with the download lifecycle.

## What Are Crawl Hooks?

Crawl hooks handle one-time setup operations that prepare the system before any URLs are processed.

### Execution Timing and Purpose

Crawl hooks execute **once per crawl session** and run **before any snapshot is created**. According to the source code in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), these hooks are collected and sorted first, then executed prior to snapshot processing.

These hooks are designed for:
- Installing headless browsers or binary dependencies
- Generating configuration files
- Setting up temporary directories
- Registering shared resources via JSONL records

### File Naming Convention

The plugin system identifies crawl hooks by scanning for the substring **`Crawl`** in the hook filename. As implemented in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) (lines 59-64), the `Plugin.get_crawl_hooks()` method returns all hooks matching this pattern.

Example filename: [`on_Crawl__00_install_browser.py`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__00_install_browser.py)

### Dependency Handling

Plugins providing only crawl hooks are **assumed to self-install** their binaries. The global dependency checker in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) (lines 70-76) skips verification for these plugins, allowing them to handle their own setup during execution.

## What Are Snapshot Hooks?

Snapshot hooks perform the actual work of extracting and archiving content for individual URLs.

### Execution Timing and Purpose

Snapshot hooks execute **once per snapshot** (for every URL being archived). They run **after all crawl hooks have finished**, as shown in the executor logic that appends snapshot hooks to the execution list after processing crawl hooks.

These hooks handle:
- Taking screenshots of web pages
- Fetching page titles and metadata
- Saving PDFs or DOM snapshots
- Extracting specific content types

### File Naming Convention

Snapshot hooks are identified by the substring **`Snapshot`** in their filenames. The `Plugin.get_snapshot_hooks()` method in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) (lines 52-57) filters for these patterns.

Example filename: [`on_Snapshot__10_capture_title.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__10_capture_title.py)

### Result Handling

Unlike crawl hooks that typically produce side effects or JSONL records, snapshot hooks return **`ArchiveResult`** objects. These results are written to `index.jsonl` and become part of the final archive, as defined in [`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py).

## Key Differences Between Crawl and Snapshot Hooks

| Aspect | Crawl Hooks | Snapshot Hooks |
|--------|-------------|----------------|
| **Execution frequency** | Once per crawl session | Once per URL (snapshot) |
| **Pipeline position** | Before any snapshots | After crawl hooks complete |
| **Filename identifier** | Contains `Crawl` | Contains `Snapshot` |
| **Primary purpose** | Environment setup and installation | Content extraction and archiving |
| **Return type** | JSONL records or side effects | `ArchiveResult` objects |
| **Dependency checking** | Skipped (self-installing) | Verified before execution |

## How the Plugin Execution Pipeline Works

The executor in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) orchestrates the two-phase pipeline by collecting and sorting hooks before execution.

```python

# Simplified excerpt from abx_dl/executor.py (lines 31-42)

crawl_hooks: list[tuple[Plugin, Hook]] = []
snapshot_hooks: list[tuple[Plugin, Hook]] = []

for plugin in available_plugins.values():
    for hook in plugin.get_crawl_hooks():
        crawl_hooks.append((plugin, hook))
    for hook in plugin.get_snapshot_hooks():
        snapshot_hooks.append((plugin, hook))

# Order matters: crawl first, then snapshot

all_hooks = sorted(crawl_hooks, key=lambda x: x[1].sort_key) + \
            sorted(snapshot_hooks, key=lambda x: x[1].sort_key)

```

The `sort_key` (derived from the numeric prefix in filenames like `00_` or `10_`) determines execution order within each phase.

## Practical Examples

### Creating a Crawl Hook for Environment Setup

This example installs a headless browser before any URLs are processed:

```python

# plugins/my_plugin/on_Crawl__00_install_browser.py

#!/usr/bin/env python3
import subprocess
import json

# Self-installing binary (dependency checker skips this plugin)

subprocess.run(["playwright", "install", "chromium"], check=True)

# Register the binary path for snapshot hooks

print(json.dumps({
    "type": "Binary",
    "name": "CHROMIUM",
    "abspath": "/path/to/chromium"
}))

```

### Creating a Snapshot Hook for Content Extraction

This hook captures the page title for each archived URL:

```python

# plugins/my_plugin/on_Snapshot__10_capture_title.py

#!/usr/bin/env python3
import sys
import json
import requests
from bs4 import BeautifulSoup

url = sys.argv[1]          # Target URL passed by abx-dl

snapshot_id = sys.argv[2]  # Unique snapshot identifier

resp = requests.get(url, timeout=30)
title = BeautifulSoup(resp.text, "html.parser").title.string.strip()

# Return ArchiveResult for the index

print(json.dumps({
    "type": "ArchiveResult",
    "snapshot_id": snapshot_id,
    "plugin": "my_plugin",
    "hook_name": "on_Snapshot__10_capture_title",
    "status": "succeeded",
    "data": {"title": title}
}))

```

## Summary

- **Crawl hooks** execute once per session in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) to handle setup tasks like binary installation, identified by `Crawl` in their filenames and returned by `Plugin.get_crawl_hooks()`.
- **Snapshot hooks** run for every URL via `Plugin.get_snapshot_hooks()`, performing extraction work like screenshots or title capture, and producing `ArchiveResult` objects stored in `index.jsonl`.
- The pipeline explicitly sorts crawl hooks before snapshot hooks using filename prefixes (e.g., `00_`, `10_`) as sort keys.
- Dependency checking skips plugins with only crawl hooks, assuming they self-install during execution.

## Frequently Asked Questions

### Can a single plugin provide both crawl and snapshot hooks?

Yes. A plugin can define multiple hook files in its directory, mixing both patterns (e.g., [`on_Crawl__00_setup.py`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__00_setup.py) and [`on_Snapshot__10_extract.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__10_extract.py)). The `Plugin` class in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) will correctly categorize each hook based on its filename and return it via the appropriate getter method.

### How does abx-dl determine the execution order of hooks?

The executor extracts a sort key from the numeric prefix in the hook filename (e.g., `00` from [`on_Crawl__00_install.py`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__00_install.py)). Hooks are sorted by this key within their respective phases—crawl hooks execute first in ascending order, followed by snapshot hooks. This allows plugins to declare dependencies on other plugins having run first.

### What happens if a crawl hook fails?

Since crawl hooks run before any snapshots are created, a failure here halts the entire crawl session. The executor in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) processes hooks sequentially, and an uncaught exception or non-zero exit code from a crawl hook will prevent snapshot hooks from running, ensuring the environment is properly prepared before extraction begins.

### Are snapshot hooks executed in parallel?

The provided source analysis shows sequential execution in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), where hooks are sorted and run one after another. While the core pipeline processes hooks sequentially to maintain deterministic ordering, individual snapshot hooks may internally implement parallel processing (e.g., concurrent network requests). However, the hook invocation loop itself iterates through the sorted list without explicit parallelism in the executor code shown.