# How to Add a New Plugin to the abx-dl Plugin System: A Complete Developer Guide

> Learn to add a new plugin to the abx-dl plugin system. Follow our developer guide to create hook scripts and output JSON-Lines with ArchiveResult records.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: how-to-guide
- Published: 2026-02-25

---

**To add a new plugin to abx-dl, create a subdirectory in `abx_dl/plugins/`, add hook scripts following the strict naming convention `on_{Event}__{step}{priority}_{description}[.bg].{ext}`, and ensure each script outputs valid JSON-Lines containing `ArchiveResult` records to stdout.**

The `abx-dl` download manager from the ArchiveBox ecosystem discovers plugins dynamically at runtime from the filesystem. The discovery mechanism in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) scans the plugins directory, parses optional configuration manifests, and instantiates hook objects that the executor in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) runs in a deterministic order during the download pipeline.

## How Plugin Discovery Works

The `load_plugin()` function in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) (lines 92-101) implements a five-stage discovery process that activates whenever the application starts:

1. **Directory Loading** – The system scans `abx_dl/plugins/` and loads any folder that does not start with `.` or `_`, skipping hidden directories automatically.

2. **Configuration Parsing** – If present, [`config.json`](https://github.com/archivebox/abx-dl/blob/main/config.json) is parsed to expose a JSON Schema for environment-variable configuration (lines 105-112).

3. **Binary Dependency Resolution** – If present, `binaries.jsonl` describes external binaries required by the plugin, including download URLs and installation commands (lines 114-122).

4. **Hook Script Discovery** – The system identifies files matching the pattern `on_{Event}__{step}{priority}_{description}[.bg].{ext}` and extracts metadata via `parse_hook_filename()` (lines 67-90).

5. **Hook Instantiation** – Each matching file generates a `Hook` object attached to the `Plugin` instance (lines 124-145).

## The Hook Filename Convention

Hook scripts must follow a precise naming scheme parsed by `parse_hook_filename()` in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py):

```

on_{Event}__{step}{priority}_{description}[.bg].{ext}

```

- **Event**: Either `Crawl` (runs once per download batch) or `Snapshot` (runs for each individual URL).
- **step**: A single digit (0-9) representing the execution phase.
- **priority**: A single digit (0-9) controlling order within the phase; lower numbers execute first.
- **description**: Arbitrary descriptive text.
- **.bg**: Optional suffix indicating a background daemon hook that runs asynchronously.
- **ext**: The interpreter type—`py` for Python, `js` for Node.js, or `sh` for shell scripts.

For example, [`on_Snapshot__20_fetch_page.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__20_fetch_page.py) runs during the Snapshot event at step 2, priority 0, while [`on_Crawl__01_init.bg.sh`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__01_init.bg.sh) runs as a background daemon at step 0, priority 1.

## Implementing a Hook Script

When the user executes `abx-dl download`, the `download()` function in the CLI (wired through [`cli.py`](https://github.com/archivebox/abx-dl/blob/main/cli.py)) builds ordered lists of `crawl_hooks` and `snapshot_hooks` from all enabled plugins (executor.py lines 32-42). The `run_hook()` function (executor.py lines 28-31) then executes each hook by:

- Selecting the appropriate interpreter based on the file extension.
- Building a per-plugin environment via `build_env_for_plugin()` (executor.py lines 44-51).
- Passing `--url` and `--snapshot-id` arguments to the script.
- Capturing stdout and stderr to `.stdout.log` and `.stderr.log` files.
- Creating `.pid` files for background hooks (executor.py lines 75-82).

### Required Output Format

Hooks must emit **JSON-Lines** (`.jsonl`) to stdout. The executor parses each line searching for records with specific types:

- **`ArchiveResult`**: The primary output containing the archived data.
- **`Binary`** or **`Machine`**: Used for configuration propagation.

A valid success record looks like:

```json
{"type": "ArchiveResult", "snapshot_id": "uuid-here", "plugin": "my_plugin", "hook_name": "on_Snapshot__20_fetch_page", "status": "succeeded", "output_str": "Saved content to file.html"}

```

The executor aggregates these results into the snapshot's `index.jsonl` (executor.py lines 62-66).

## Step-by-Step Plugin Creation Process

Follow these steps to implement a functional plugin:

1. **Create the plugin directory** at `abx_dl/plugins/<your_plugin>/`, ensuring the name contains no leading dots or underscores.

2. **Define optional configuration** by adding [`config.json`](https://github.com/archivebox/abx-dl/blob/main/config.json) with a JSON Schema under the `"properties"` key to map environment variables to defaults and descriptions.

3. **Declare binary dependencies** by adding `binaries.jsonl` with lines containing `{"name": "BINARY_NAME", "url": "...", "install_cmd": "..."}`.

4. **Write hook scripts** following the naming convention, choosing Python, JavaScript, or Shell based on your requirements.

5. **Implement the hook logic** to read `--url` and `--snapshot-id` arguments, perform the archival work, write artifacts to the plugin's output subdirectory, and print the required JSON-Lines to stdout.

6. **Test the plugin** by running `abx-dl dl --plugins=<your_plugin> <url>` and inspecting the generated `index.jsonl` and plugin output directory.

## Code Examples

### Directory Structure

```

abx_dl/
└─ plugins/
   └─ my_plugin/
      ├─ config.json          # optional

      ├─ binaries.jsonl       # optional

      └─ on_Snapshot__20_fetch_page.py

```

### Configuration Schema (config.json)

```json
{
  "type": "object",
  "properties": {
    "MY_PLUGIN_TIMEOUT": {
      "type": "integer",
      "default": 30,
      "description": "Seconds before the hook times out"
    }
  }
}

```

### Binary Dependencies (binaries.jsonl)

```jsonl
{"name":"CHROME","url":"https://dl.google.com/chrome/chrome.tar.gz","install_cmd":"tar -xzf chrome.tar.gz -C $LIB_DIR"}

```

### Example Hook Script (on_Snapshot__20_fetch_page.py)

```python
#!/usr/bin/env python3
import argparse
import json
import pathlib
import requests
import sys

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--url', required=True)
    parser.add_argument('--snapshot-id', required=True)
    args = parser.parse_args()

    try:
        resp = requests.get(args.url, timeout=30)
        resp.raise_for_status()
        html = resp.text
    except Exception as e:
        result = {
            "type": "ArchiveResult",
            "snapshot_id": args.snapshot_id,
            "plugin": "my_plugin",
            "hook_name": "on_Snapshot__20_fetch_page",
            "status": "failed",
            "error": str(e)
        }
        print(json.dumps(result))
        sys.exit(1)

    out_path = pathlib.Path.cwd() / "page.html"
    out_path.write_text(html, encoding="utf-8")

    result = {
        "type": "ArchiveResult",
        "snapshot_id": args.snapshot_id,
        "plugin": "my_plugin",
        "hook_name": "on_Snapshot__20_fetch_page",
        "status": "succeeded",
        "output_str": f"Saved page to {out_path}"
    }
    print(json.dumps(result))

if __name__ == '__main__':
    main()

```

Make the script executable with `chmod +x on_Snapshot__20_fetch_page.py`. The script receives the target URL and snapshot identifier, persists the artifact to [`page.html`](https://github.com/archivebox/abx-dl/blob/main/page.html), and emits the required JSON-Line for the executor to capture.

## Summary

- **abx-dl** discovers plugins automatically from the `abx_dl/plugins/` directory at runtime.
- Hook filenames must follow `on_{Event}__{step}{priority}_{description}[.bg].{ext}` to be recognized by `parse_hook_filename()`.
- Scripts receive `--url` and `--snapshot-id` arguments and must output JSON-Lines with `type: ArchiveResult` to stdout.
- The executor in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) handles interpreter selection, environment setup, and result aggregation into `index.jsonl`.
- Optional [`config.json`](https://github.com/archivebox/abx-dl/blob/main/config.json) and `binaries.jsonl` files provide schema validation and dependency management.

## Frequently Asked Questions

### What programming languages can I use to write abx-dl plugins?

You can write hook scripts in **Python** (`.py`), **JavaScript/Node.js** (`.js`), or **Shell** (`.sh`). The `run_hook()` function in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) automatically selects the appropriate interpreter based on the file extension. Ensure the interpreter is available in the system PATH when the hook executes.

### How do I control the execution order of my plugin hooks?

Execution order is determined by the **step** and **priority** digits in the hook filename. The format `on_Event__{step}{priority}_{desc}.ext` uses single digits (0-9) where lower numbers run first. For example, [`on_Snapshot__01_early_hook.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__01_early_hook.py) runs before [`on_Snapshot__20_later_hook.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__20_later_hook.py) because step 0 executes before step 2.

### What is the difference between Crawl and Snapshot events?

**Crawl** hooks run once per download invocation (per batch), making them suitable for setup or cleanup operations that should execute only once regardless of URL count. **Snapshot** hooks run once for every individual URL being archived. According to the executor logic in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), the system collects all crawl hooks first, executes them, then iterates through URLs executing applicable snapshot hooks for each.

### How do background hooks (.bg) work in abx-dl?

Hooks with the `.bg` suffix in their filename (e.g., [`on_Crawl__00_server.bg.py`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__00_server.bg.py)) execute as background daemons. The `run_hook()` function creates a `.pid` file to track these processes and does not block the pipeline waiting for completion. Background hooks are ideal for long-running services like browser instances or proxy servers that need to stay alive across multiple snapshots.