# How the abx-dl Binary Discovery System Works: Complete Guide to Custom Binary Paths

> Learn how abx-dl's binary discovery system finds executables and customizes paths. Explore environment variables, pip, npm, brew, and apt providers for seamless integration.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: deep-dive
- Published: 2026-02-25

---

**abx-dl’s binary discovery system locates required executables by reading plugin specifications from `binaries.jsonl`, querying a chain of providers (environment variables, pip, npm, brew, apt), and exposing discovered paths as `<NAME>_BINARY` environment variables that hooks can consume.**

The **abx-dl** binary discovery system provides a deterministic, extensible mechanism for plugins to declare and locate external dependencies. According to the archivebox/abx-dl source code, this system operates through a four-stage pipeline that transforms static specifications into runtime environment variables. Whether you need to override Chrome’s path or point to a custom FFmpeg installation, understanding this architecture lets you control exactly how binaries are resolved across your data extraction pipeline.

## How the Binary Discovery System Works

### Stage 1: Load the Binary Spec from the Plugin

Each plugin ships a `binaries.jsonl` file listing the external binaries it requires. In **[`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py)**, the `load_plugin()` function (lines 15-22) reads this file and stores the specifications in `plugin.binaries`.

A typical `binaries.jsonl` entry declares the binary name, eligible providers, and optional overrides:

```json
{"name":"ffmpeg","binproviders":"env,pip","overrides":{}}

```

### Stage 2: Resolve the Binary via Provider Chain

In **[`abx_dl/dependencies.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/dependencies.py)**, the `load_binary()` function (lines 35-44) constructs a **Binary** object from the spec and queries the provider chain. The `DEFAULT_PROVIDERS` list (lines 11-31) defines the resolution order:

- **`EnvProvider`** – checks for `<NAME>_BINARY` environment variables
- **`PipProvider`** – searches via pip when available
- **`NpmProvider`**, **`BrewProvider`**, **`AptProvider`** – added when their respective package managers exist on the host

If no provider locates the binary and `auto_install=True` (the default), `install_binary()` (lines 49-57) invokes `binary.load_or_install()`, which may trigger `pip install`, `npm install`, or equivalent commands. Otherwise, `check_plugin_dependencies()` in **[`executor.py`](https://github.com/archivebox/abx-dl/blob/main/executor.py)** (lines 64-86) marks the plugin as *missing* and skips execution.

### Stage 3: Publish the Discovered Path

When a plugin runs, it can emit a JSON line on stdout to communicate binary locations dynamically:

```json
{"type":"Binary","name":"chrome","abspath":"/usr/local/bin/chrome"}

```

In **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)**, the parser (lines 66-74) captures these records and updates the shared configuration:

```python
if record.get('type') == 'Binary':
    name = record.get('name', '')
    abspath = record.get('abspath', '')
    if name and abspath:
        shared_config[f'{name.upper()}_BINARY'] = abspath

```

### Stage 4: Inject into Hook Environment

The `build_env_for_plugin()` function in **[`abx_dl/config.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/config.py)** (lines 91-115) copies every `*_BINARY` entry from the shared configuration into the environment dictionary passed to each hook. Hooks then read these variables directly via `os.getenv('CHROME_BINARY')`.

## How to Specify Custom Binary Paths

You can override the discovery system using three methods, all of which ultimately result in a `<NAME>_BINARY` environment variable accessible to downstream hooks.

### Method 1: Environment Variables

Set a variable matching the binary name in uppercase with `_BINARY` suffix:

```bash
export CHROME_BINARY="/opt/custom/chrome"
export FFMPEG_BINARY="/usr/local/ffmpeg/bin/ffmpeg"

```

The `EnvProvider` checks these first, bypassing all other discovery logic.

### Method 2: Override in the binaries.jsonl Spec

Add an `overrides` dictionary to force a specific absolute path regardless of provider chain results:

```json
{"name":"chrome","binproviders":"env","overrides":{"abspath":"/opt/custom/chrome"}}

```

The **Binary** object treats the supplied path as valid without querying providers.

### Method 3: Emit a Binary JSON Line from a Hook

Inside a hook script, print a JSON record to stdout:

```python
#!/usr/bin/env python3
import json
import sys

# After verifying custom binary location

print(json.dumps({
    "type": "Binary",
    "name": "chrome",
    "abspath": "/opt/custom/chrome"
}))
sys.exit(0)

```

The executor captures this output and updates `shared_config`, making the path available to subsequent hooks via the standard environment variable.

## Code Examples

### Loading and Resolving a Binary

```python

# abx_dl/dependencies.py

from abx_pkg import Binary, EnvProvider, PipProvider

def load_binary(spec: dict) -> Binary:
    providers_str = spec.get('binproviders', 'env')
    providers = [p for p in DEFAULT_PROVIDERS 
                 if p.name in providers_str.split(',')]
    overrides = spec.get('overrides', {})
    
    binary = Binary(
        name=spec['name'], 
        binproviders=providers, 
        overrides=overrides
    )
    return binary.load()  # Returns Binary with resolved abspath

```

### Auto-Installing Missing Dependencies

```python

# abx_dl/dependencies.py

def install_binary(spec: dict) -> Binary:
    providers_str = spec.get('binproviders', 'env')
    providers = [p for p in DEFAULT_PROVIDERS 
                 if p.name in providers_str.split(',')]
    overrides = spec.get('overrides', {})
    
    binary = Binary(
        name=spec['name'],
        binproviders=providers,
        overrides=overrides
    )
    # Attempts load() first, then install if missing

    return binary.load_or_install()

```

### Consuming Binary Paths in Hooks

```python
#!/usr/bin/env python3
import os
import subprocess

chrome_path = os.getenv('CHROME_BINARY')
if not chrome_path:
    raise RuntimeError("CHROME_BINARY not set")

subprocess.run([chrome_path, "--headless", "--dump-dom", "https://example.com"])

```

## Summary

- **abx-dl** discovers binaries via a provider chain defined in `DEFAULT_PROVIDERS` (**[`dependencies.py`](https://github.com/archivebox/abx-dl/blob/main/dependencies.py)** lines 11-31), prioritized as environment variables → pip → npm → brew → apt.
- Plugins declare requirements in `binaries.jsonl`, loaded by **[`plugins.py`](https://github.com/archivebox/abx-dl/blob/main/plugins.py)** (lines 15-22) and resolved by **[`dependencies.py`](https://github.com/archivebox/abx-dl/blob/main/dependencies.py)**`load_binary()` (lines 35-44).
- Custom paths can be supplied via environment variables, JSON overrides in the spec, or runtime JSON emission from hooks.
- Discovered paths propagate through **[`executor.py`](https://github.com/archivebox/abx-dl/blob/main/executor.py)** (lines 66-74) and **[`config.py`](https://github.com/archivebox/abx-dl/blob/main/config.py)**`build_env_for_plugin()` (lines 91-115) as `<NAME>_BINARY` environment variables.
- With `auto_install=True`, missing binaries trigger automatic installation via the appropriate package manager; otherwise, plugins are skipped.

## Frequently Asked Questions

### What file format does abx-dl use to declare plugin binary dependencies?

abx-dl uses **JSON Lines** format in a file named `binaries.jsonl`. Each line is a JSON object containing keys like `name`, `binproviders` (comma-separated provider list), and optional `overrides`. The plugin loader in **[`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py)** reads this file during initialization.

### How does abx-dl determine which environment variable to check for a binary?

The system converts the binary name to uppercase and appends `_BINARY`. For a binary named `chrome`, abx-dl checks the `CHROME_BINARY` environment variable. This convention is enforced by the `EnvProvider` class and the shared config logic in **[`executor.py`](https://github.com/archivebox/abx-dl/blob/main/executor.py)**.

### Can I prevent abx-dl from attempting to auto-install missing binaries?

Yes. Set `auto_install=False` when calling `install_binary()`, or ensure your binary is discoverable by one of the providers in `DEFAULT_PROVIDERS` before execution. If `auto_install` is disabled and the binary is not found, `check_plugin_dependencies()` in **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** marks the plugin as missing and skips it rather than invoking package managers.

### Why would I emit a Binary JSON line from a hook instead of using environment variables?

Emitting a JSON line allows **dynamic discovery** during execution. For example, a hook that compiles a tool from source or downloads a version-specific binary can communicate the exact path to subsequent hooks without needing to know the path before the pipeline starts. This method updates the shared configuration in real-time via **[`executor.py`](https://github.com/archivebox/abx-dl/blob/main/executor.py)** (lines 66-74).