Crawl Hooks vs Snapshot Hooks in abx-dl: Understanding the Plugin Execution Pipeline

Crawl hooks run once per crawl session to prepare the environment, while snapshot hooks execute for every URL to extract and archive content.

The archivebox/abx-dl repository implements a two-phase plugin execution pipeline that separates environment setup from content extraction. Understanding the distinction between these hook types is essential for developing plugins that integrate correctly with the download lifecycle.

What Are Crawl Hooks?

Crawl hooks handle one-time setup operations that prepare the system before any URLs are processed.

Execution Timing and Purpose

Crawl hooks execute once per crawl session and run before any snapshot is created. According to the source code in abx_dl/executor.py, these hooks are collected and sorted first, then executed prior to snapshot processing.

These hooks are designed for:

• Installing headless browsers or binary dependencies
• Generating configuration files
• Setting up temporary directories
• Registering shared resources via JSONL records

File Naming Convention

The plugin system identifies crawl hooks by scanning for the substring Crawl in the hook filename. As implemented in abx_dl/plugins.py (lines 59-64), the Plugin.get_crawl_hooks() method returns all hooks matching this pattern.

Example filename: on_Crawl__00_install_browser.py

Dependency Handling

Plugins providing only crawl hooks are assumed to self-install their binaries. The global dependency checker in abx_dl/executor.py (lines 70-76) skips verification for these plugins, allowing them to handle their own setup during execution.

What Are Snapshot Hooks?

Snapshot hooks perform the actual work of extracting and archiving content for individual URLs.

Execution Timing and Purpose

Snapshot hooks execute once per snapshot (for every URL being archived). They run after all crawl hooks have finished, as shown in the executor logic that appends snapshot hooks to the execution list after processing crawl hooks.

These hooks handle:

• Taking screenshots of web pages
• Fetching page titles and metadata
• Saving PDFs or DOM snapshots
• Extracting specific content types

File Naming Convention

Snapshot hooks are identified by the substring Snapshot in their filenames. The Plugin.get_snapshot_hooks() method in abx_dl/plugins.py (lines 52-57) filters for these patterns.

Example filename: on_Snapshot__10_capture_title.py

Result Handling

Unlike crawl hooks that typically produce side effects or JSONL records, snapshot hooks return ArchiveResult objects. These results are written to index.jsonl and become part of the final archive, as defined in abx_dl/models.py.

Key Differences Between Crawl and Snapshot Hooks

Aspect	Crawl Hooks	Snapshot Hooks
Execution frequency	Once per crawl session	Once per URL (snapshot)
Pipeline position	Before any snapshots	After crawl hooks complete
Filename identifier	Contains Crawl	Contains Snapshot
Primary purpose	Environment setup and installation	Content extraction and archiving
Return type	JSONL records or side effects	ArchiveResult objects
Dependency checking	Skipped (self-installing)	Verified before execution

How the Plugin Execution Pipeline Works

The executor in abx_dl/executor.py orchestrates the two-phase pipeline by collecting and sorting hooks before execution.

# Simplified excerpt from abx_dl/executor.py (lines 31-42)

crawl_hooks: list[tuple[Plugin, Hook]] = []
snapshot_hooks: list[tuple[Plugin, Hook]] = []

for plugin in available_plugins.values():
    for hook in plugin.get_crawl_hooks():
        crawl_hooks.append((plugin, hook))
    for hook in plugin.get_snapshot_hooks():
        snapshot_hooks.append((plugin, hook))

# Order matters: crawl first, then snapshot

all_hooks = sorted(crawl_hooks, key=lambda x: x[1].sort_key) + \
            sorted(snapshot_hooks, key=lambda x: x[1].sort_key)

The sort_key (derived from the numeric prefix in filenames like 00_ or 10_) determines execution order within each phase.

Practical Examples

Creating a Crawl Hook for Environment Setup

This example installs a headless browser before any URLs are processed:

# plugins/my_plugin/on_Crawl__00_install_browser.py

#!/usr/bin/env python3
import subprocess
import json

# Self-installing binary (dependency checker skips this plugin)

subprocess.run(["playwright", "install", "chromium"], check=True)

# Register the binary path for snapshot hooks

print(json.dumps({
    "type": "Binary",
    "name": "CHROMIUM",
    "abspath": "/path/to/chromium"
}))

Creating a Snapshot Hook for Content Extraction

This hook captures the page title for each archived URL:

# plugins/my_plugin/on_Snapshot__10_capture_title.py

#!/usr/bin/env python3
import sys
import json
import requests
from bs4 import BeautifulSoup

url = sys.argv[1]          # Target URL passed by abx-dl

snapshot_id = sys.argv[2]  # Unique snapshot identifier

resp = requests.get(url, timeout=30)
title = BeautifulSoup(resp.text, "html.parser").title.string.strip()

# Return ArchiveResult for the index

print(json.dumps({
    "type": "ArchiveResult",
    "snapshot_id": snapshot_id,
    "plugin": "my_plugin",
    "hook_name": "on_Snapshot__10_capture_title",
    "status": "succeeded",
    "data": {"title": title}
}))

Summary

• Crawl hooks execute once per session in abx_dl/executor.py to handle setup tasks like binary installation, identified by Crawl in their filenames and returned by Plugin.get_crawl_hooks().
• Snapshot hooks run for every URL via Plugin.get_snapshot_hooks(), performing extraction work like screenshots or title capture, and producing ArchiveResult objects stored in index.jsonl.
• The pipeline explicitly sorts crawl hooks before snapshot hooks using filename prefixes (e.g., 00_, 10_) as sort keys.
• Dependency checking skips plugins with only crawl hooks, assuming they self-install during execution.

Frequently Asked Questions

Can a single plugin provide both crawl and snapshot hooks?

Yes. A plugin can define multiple hook files in its directory, mixing both patterns (e.g., on_Crawl__00_setup.py and on_Snapshot__10_extract.py). The Plugin class in abx_dl/plugins.py will correctly categorize each hook based on its filename and return it via the appropriate getter method.

How does abx-dl determine the execution order of hooks?

The executor extracts a sort key from the numeric prefix in the hook filename (e.g., 00 from on_Crawl__00_install.py). Hooks are sorted by this key within their respective phases—crawl hooks execute first in ascending order, followed by snapshot hooks. This allows plugins to declare dependencies on other plugins having run first.

What happens if a crawl hook fails?

Since crawl hooks run before any snapshots are created, a failure here halts the entire crawl session. The executor in abx_dl/executor.py processes hooks sequentially, and an uncaught exception or non-zero exit code from a crawl hook will prevent snapshot hooks from running, ensuring the environment is properly prepared before extraction begins.

Are snapshot hooks executed in parallel?

The provided source analysis shows sequential execution in abx_dl/executor.py, where hooks are sorted and run one after another. While the core pipeline processes hooks sequentially to maintain deterministic ordering, individual snapshot hooks may internally implement parallel processing (e.g., concurrent network requests). However, the hook invocation loop itself iterates through the sorted list without explicit parallelism in the executor code shown.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}