# How abx-dl Discovers and Loads Plugins from the Plugins Directory

> Learn how abx-dl discovers and loads plugins from its plugins directory by treating subdirectories as individual plugins, configuring their hooks automatically.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: internals
- Published: 2026-02-25

---

**abx-dl automatically discovers plugins by treating each sub-directory in `abx_dl/plugins` as a separate plugin, loading their configurations and hooks via the `discover_plugins()` function in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py).**

The **archivebox/abx-dl** repository implements a modular architecture that enables dynamic extension through automatic plugin discovery. Understanding how abx-dl discovers and loads plugins from the plugins directory allows developers to add custom archiving backends without modifying core code.

## Plugin Discovery Architecture

abx-dl treats every sub-directory inside **`abx_dl/plugins`** as an independent plugin module. This design enables the system to dynamically load new archiving backends simply by placing them in the designated folder, eliminating the need for manual registration or configuration file updates.

The discovery mechanism relies on three core components defined in **[`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py)**: the root directory constant, the individual plugin loader, and the directory scanner.

## The Discovery Pipeline in abx_dl/plugins.py

### Locating the Plugins Root

The system defines a constant that points to the plugins directory relative to the source file location. This path is typically symlinked to the ArchiveBox plugins repository in production environments.

```python
from pathlib import Path

PLUGINS_DIR = Path(__file__).parent / 'plugins'   # → abx_dl/plugins.py L14-L16

```

### Loading Individual Plugins with load_plugin()

The **`load_plugin(plugin_dir)`** function inspects a candidate sub-directory and constructs a **Plugin** object. It reads optional metadata files including [`config.json`](https://github.com/archivebox/abx-dl/blob/main/config.json) for configuration schemas and `binaries.jsonl` for external dependency specifications.

The function scans for hook files matching the pattern `on_{Event}__{step}{priority}_{desc}[.bg].{ext}`, creating **Hook** objects that define when and how plugin code executes during the download pipeline. This implementation is found in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py).

### Aggregating Plugins with discover_plugins()

The **`discover_plugins(plugins_dir=PLUGINS_DIR)`** function iterates over every entry in the plugins folder, invoking `load_plugin()` for each valid directory and aggregating results into a dictionary keyed by plugin name.

```python
def discover_plugins(plugins_dir=PLUGINS_DIR):
    plugins = {}
    for plugin_dir in sorted(plugins_dir.iterdir()):
        plugin = load_plugin(plugin_dir)
        if plugin:
            plugins[plugin.name] = plugin   # → abx_dl/plugins.py L56-L60

    return plugins

```

## CLI Integration and Runtime Registration

When the CLI initializes in **[`abx_dl/cli.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/cli.py)**, it triggers the discovery process and stores the resulting plugin catalog in the Click context object. This makes the full plugin registry available to all subcommands and the download executor.

```python
@click.group()
@click.pass_context
def cli(ctx):
    ctx.obj['plugins'] = discover_plugins()   # → abx_dl/cli.py L58-L60

```

The **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** module subsequently consumes these discovered hooks, sorting them by step and priority to orchestrate the download pipeline. The **Plugin** and **Hook** data structures are defined in **[`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py)**.

## Working with Discovered Plugins

You can interact with the plugin system programmatically to inspect available capabilities and hook configurations.

List all discovered plugin names:

```python
from abx_dl.plugins import discover_plugins

plugins = discover_plugins()
print("Available plugins:", list(plugins.keys()))

# Output: ['chrome', 'git', 'wget', ...]

```

Inspect a specific plugin's snapshot hooks:

```python
plugins = discover_plugins()
chrome = plugins.get('chrome')
if chrome:
    for hook in chrome.get_snapshot_hooks():
        print(f"{hook.full_name} – step {hook.step}.{hook.priority} – bg={hook.is_background}")

```

## Summary

- **abx-dl** discovers plugins by scanning sub-directories in `abx_dl/plugins` via the **`discover_plugins()`** function.
- Each plugin directory is processed by **`load_plugin()`**, which parses [`config.json`](https://github.com/archivebox/abx-dl/blob/main/config.json), `binaries.jsonl`, and hook files matching the pattern `on_{Event}__{step}{priority}_{desc}[.bg].{ext}`.
- The resulting **Plugin** objects contain **config_schema**, **binaries**, and **hooks** attributes used for validation, dependency management, and execution.
- The CLI stores the plugin dictionary in **`ctx.obj['plugins']`** during startup, making the catalog available to **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** and other components.

## Frequently Asked Questions

### What directory structure does abx-dl expect for plugins?

abx-dl expects each plugin to reside in its own sub-directory under `abx_dl/plugins/`. The directory name becomes the plugin identifier. According to the source code in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py), the system iterates over these directories using `sorted(plugins_dir.iterdir())`, treating each as a separate plugin module.

### How does abx-dl identify hook files within a plugin?

The system scans for files matching the pattern `on_{Event}__{step}{priority}_{desc}[.bg].{ext}` within each plugin directory. These filenames encode the event type, execution step, priority level, and whether the hook runs in the background. The `load_plugin()` function parses these patterns to create **Hook** objects that the executor later sorts and invokes.

### Where does abx-dl store the plugin catalog during runtime?

The plugin catalog is stored in the Click context object's **`plugins`** key. In [`abx_dl/cli.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/cli.py), the CLI entry point calls `ctx.obj['plugins'] = discover_plugins()`, making the dictionary of **Plugin** instances available throughout the application lifecycle, including to the executor module that orchestrates download tasks.

### What metadata files can a plugin include?

A plugin can include **[`config.json`](https://github.com/archivebox/abx-dl/blob/main/config.json)** to define its configuration schema for runtime validation, and **`binaries.jsonl`** to specify external dependencies that the dependency manager must install. The `load_plugin()` function reads these files when constructing the **Plugin** object, populating the `config_schema` and `binaries` attributes respectively.