How to Add a New Plugin to the abx-dl Plugin System: A Complete Developer Guide

To add a new plugin to abx-dl, create a subdirectory in abx_dl/plugins/, add hook scripts following the strict naming convention on_{Event}__{step}{priority}_{description}[.bg].{ext}, and ensure each script outputs valid JSON-Lines containing ArchiveResult records to stdout.

The abx-dl download manager from the ArchiveBox ecosystem discovers plugins dynamically at runtime from the filesystem. The discovery mechanism in abx_dl/plugins.py scans the plugins directory, parses optional configuration manifests, and instantiates hook objects that the executor in abx_dl/executor.py runs in a deterministic order during the download pipeline.

How Plugin Discovery Works

The load_plugin() function in abx_dl/plugins.py (lines 92-101) implements a five-stage discovery process that activates whenever the application starts:

1. Directory Loading – The system scans abx_dl/plugins/ and loads any folder that does not start with . or _, skipping hidden directories automatically.

2. Configuration Parsing – If present, config.json is parsed to expose a JSON Schema for environment-variable configuration (lines 105-112).

3. Binary Dependency Resolution – If present, binaries.jsonl describes external binaries required by the plugin, including download URLs and installation commands (lines 114-122).

4. Hook Script Discovery – The system identifies files matching the pattern on_{Event}__{step}{priority}_{description}[.bg].{ext} and extracts metadata via parse_hook_filename() (lines 67-90).

5. Hook Instantiation – Each matching file generates a Hook object attached to the Plugin instance (lines 124-145).

The Hook Filename Convention

Hook scripts must follow a precise naming scheme parsed by parse_hook_filename() in abx_dl/plugins.py:

on_{Event}__{step}{priority}_{description}[.bg].{ext}

• Event: Either Crawl (runs once per download batch) or Snapshot (runs for each individual URL).
• step: A single digit (0-9) representing the execution phase.
• priority: A single digit (0-9) controlling order within the phase; lower numbers execute first.
• description: Arbitrary descriptive text.
• .bg: Optional suffix indicating a background daemon hook that runs asynchronously.
• ext: The interpreter type—py for Python, js for Node.js, or sh for shell scripts.

For example, on_Snapshot__20_fetch_page.py runs during the Snapshot event at step 2, priority 0, while on_Crawl__01_init.bg.sh runs as a background daemon at step 0, priority 1.

Implementing a Hook Script

When the user executes abx-dl download, the download() function in the CLI (wired through cli.py) builds ordered lists of crawl_hooks and snapshot_hooks from all enabled plugins (executor.py lines 32-42). The run_hook() function (executor.py lines 28-31) then executes each hook by:

• Selecting the appropriate interpreter based on the file extension.
• Building a per-plugin environment via build_env_for_plugin() (executor.py lines 44-51).
• Passing --url and --snapshot-id arguments to the script.
• Capturing stdout and stderr to .stdout.log and .stderr.log files.
• Creating .pid files for background hooks (executor.py lines 75-82).

Required Output Format

Hooks must emit JSON-Lines (.jsonl) to stdout. The executor parses each line searching for records with specific types:

• ArchiveResult: The primary output containing the archived data.
• Binary or Machine: Used for configuration propagation.

A valid success record looks like:

{"type": "ArchiveResult", "snapshot_id": "uuid-here", "plugin": "my_plugin", "hook_name": "on_Snapshot__20_fetch_page", "status": "succeeded", "output_str": "Saved content to file.html"}

The executor aggregates these results into the snapshot's index.jsonl (executor.py lines 62-66).

Step-by-Step Plugin Creation Process

Follow these steps to implement a functional plugin:

1. Create the plugin directory at abx_dl/plugins/<your_plugin>/, ensuring the name contains no leading dots or underscores.

2. Define optional configuration by adding config.json with a JSON Schema under the "properties" key to map environment variables to defaults and descriptions.

3. Declare binary dependencies by adding binaries.jsonl with lines containing {"name": "BINARY_NAME", "url": "...", "install_cmd": "..."}.

4. Write hook scripts following the naming convention, choosing Python, JavaScript, or Shell based on your requirements.

5. Implement the hook logic to read --url and --snapshot-id arguments, perform the archival work, write artifacts to the plugin's output subdirectory, and print the required JSON-Lines to stdout.

6. Test the plugin by running abx-dl dl --plugins=<your_plugin> <url> and inspecting the generated index.jsonl and plugin output directory.

Code Examples

Directory Structure

abx_dl/
└─ plugins/
   └─ my_plugin/
      ├─ config.json          # optional

      ├─ binaries.jsonl       # optional

      └─ on_Snapshot__20_fetch_page.py

Configuration Schema (config.json)

{
  "type": "object",
  "properties": {
    "MY_PLUGIN_TIMEOUT": {
      "type": "integer",
      "default": 30,
      "description": "Seconds before the hook times out"
    }
  }
}

Binary Dependencies (binaries.jsonl)

{"name":"CHROME","url":"https://dl.google.com/chrome/chrome.tar.gz","install_cmd":"tar -xzf chrome.tar.gz -C $LIB_DIR"}

Example Hook Script (on_Snapshot__20_fetch_page.py)

#!/usr/bin/env python3
import argparse
import json
import pathlib
import requests
import sys

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--url', required=True)
    parser.add_argument('--snapshot-id', required=True)
    args = parser.parse_args()

    try:
        resp = requests.get(args.url, timeout=30)
        resp.raise_for_status()
        html = resp.text
    except Exception as e:
        result = {
            "type": "ArchiveResult",
            "snapshot_id": args.snapshot_id,
            "plugin": "my_plugin",
            "hook_name": "on_Snapshot__20_fetch_page",
            "status": "failed",
            "error": str(e)
        }
        print(json.dumps(result))
        sys.exit(1)

    out_path = pathlib.Path.cwd() / "page.html"
    out_path.write_text(html, encoding="utf-8")

    result = {
        "type": "ArchiveResult",
        "snapshot_id": args.snapshot_id,
        "plugin": "my_plugin",
        "hook_name": "on_Snapshot__20_fetch_page",
        "status": "succeeded",
        "output_str": f"Saved page to {out_path}"
    }
    print(json.dumps(result))

if __name__ == '__main__':
    main()

Make the script executable with chmod +x on_Snapshot__20_fetch_page.py. The script receives the target URL and snapshot identifier, persists the artifact to page.html, and emits the required JSON-Line for the executor to capture.

Summary

• abx-dl discovers plugins automatically from the abx_dl/plugins/ directory at runtime.
• Hook filenames must follow on_{Event}__{step}{priority}_{description}[.bg].{ext} to be recognized by parse_hook_filename().
• Scripts receive --url and --snapshot-id arguments and must output JSON-Lines with type: ArchiveResult to stdout.
• The executor in abx_dl/executor.py handles interpreter selection, environment setup, and result aggregation into index.jsonl.
• Optional config.json and binaries.jsonl files provide schema validation and dependency management.

Frequently Asked Questions

What programming languages can I use to write abx-dl plugins?

You can write hook scripts in Python (.py), JavaScript/Node.js (.js), or Shell (.sh). The run_hook() function in abx_dl/executor.py automatically selects the appropriate interpreter based on the file extension. Ensure the interpreter is available in the system PATH when the hook executes.

How do I control the execution order of my plugin hooks?

Execution order is determined by the step and priority digits in the hook filename. The format on_Event__{step}{priority}_{desc}.ext uses single digits (0-9) where lower numbers run first. For example, on_Snapshot__01_early_hook.py runs before on_Snapshot__20_later_hook.py because step 0 executes before step 2.

What is the difference between Crawl and Snapshot events?

Crawl hooks run once per download invocation (per batch), making them suitable for setup or cleanup operations that should execute only once regardless of URL count. Snapshot hooks run once for every individual URL being archived. According to the executor logic in abx_dl/executor.py, the system collects all crawl hooks first, executes them, then iterates through URLs executing applicable snapshot hooks for each.

How do background hooks (.bg) work in abx-dl?

Hooks with the .bg suffix in their filename (e.g., on_Crawl__00_server.bg.py) execute as background daemons. The run_hook() function creates a .pid file to track these processes and does not block the pipeline waiting for completion. Background hooks are ideal for long-running services like browser instances or proxy servers that need to stay alive across multiple snapshots.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}