How the abx-dl Binary Discovery System Works: Complete Guide to Custom Binary Paths

abx-dl’s binary discovery system locates required executables by reading plugin specifications from binaries.jsonl, querying a chain of providers (environment variables, pip, npm, brew, apt), and exposing discovered paths as <NAME>_BINARY environment variables that hooks can consume.

The abx-dl binary discovery system provides a deterministic, extensible mechanism for plugins to declare and locate external dependencies. According to the archivebox/abx-dl source code, this system operates through a four-stage pipeline that transforms static specifications into runtime environment variables. Whether you need to override Chrome’s path or point to a custom FFmpeg installation, understanding this architecture lets you control exactly how binaries are resolved across your data extraction pipeline.

How the Binary Discovery System Works

Stage 1: Load the Binary Spec from the Plugin

Each plugin ships a binaries.jsonl file listing the external binaries it requires. In abx_dl/plugins.py, the load_plugin() function (lines 15-22) reads this file and stores the specifications in plugin.binaries.

A typical binaries.jsonl entry declares the binary name, eligible providers, and optional overrides:

{"name":"ffmpeg","binproviders":"env,pip","overrides":{}}

Stage 2: Resolve the Binary via Provider Chain

In abx_dl/dependencies.py, the load_binary() function (lines 35-44) constructs a Binary object from the spec and queries the provider chain. The DEFAULT_PROVIDERS list (lines 11-31) defines the resolution order:

• EnvProvider – checks for <NAME>_BINARY environment variables
• PipProvider – searches via pip when available
• NpmProvider, BrewProvider, AptProvider – added when their respective package managers exist on the host

If no provider locates the binary and auto_install=True (the default), install_binary() (lines 49-57) invokes binary.load_or_install(), which may trigger pip install, npm install, or equivalent commands. Otherwise, check_plugin_dependencies() in executor.py (lines 64-86) marks the plugin as missing and skips execution.

Stage 3: Publish the Discovered Path

When a plugin runs, it can emit a JSON line on stdout to communicate binary locations dynamically:

{"type":"Binary","name":"chrome","abspath":"/usr/local/bin/chrome"}

In abx_dl/executor.py, the parser (lines 66-74) captures these records and updates the shared configuration:

if record.get('type') == 'Binary':
    name = record.get('name', '')
    abspath = record.get('abspath', '')
    if name and abspath:
        shared_config[f'{name.upper()}_BINARY'] = abspath

Stage 4: Inject into Hook Environment

The build_env_for_plugin() function in abx_dl/config.py (lines 91-115) copies every *_BINARY entry from the shared configuration into the environment dictionary passed to each hook. Hooks then read these variables directly via os.getenv('CHROME_BINARY').

How to Specify Custom Binary Paths

You can override the discovery system using three methods, all of which ultimately result in a <NAME>_BINARY environment variable accessible to downstream hooks.

Method 1: Environment Variables

Set a variable matching the binary name in uppercase with _BINARY suffix:

export CHROME_BINARY="/opt/custom/chrome"
export FFMPEG_BINARY="/usr/local/ffmpeg/bin/ffmpeg"

The EnvProvider checks these first, bypassing all other discovery logic.

Method 2: Override in the binaries.jsonl Spec

Add an overrides dictionary to force a specific absolute path regardless of provider chain results:

{"name":"chrome","binproviders":"env","overrides":{"abspath":"/opt/custom/chrome"}}

The Binary object treats the supplied path as valid without querying providers.

Method 3: Emit a Binary JSON Line from a Hook

Inside a hook script, print a JSON record to stdout:

#!/usr/bin/env python3
import json
import sys

# After verifying custom binary location

print(json.dumps({
    "type": "Binary",
    "name": "chrome",
    "abspath": "/opt/custom/chrome"
}))
sys.exit(0)

The executor captures this output and updates shared_config, making the path available to subsequent hooks via the standard environment variable.

Code Examples

Loading and Resolving a Binary

# abx_dl/dependencies.py

from abx_pkg import Binary, EnvProvider, PipProvider

def load_binary(spec: dict) -> Binary:
    providers_str = spec.get('binproviders', 'env')
    providers = [p for p in DEFAULT_PROVIDERS 
                 if p.name in providers_str.split(',')]
    overrides = spec.get('overrides', {})
    
    binary = Binary(
        name=spec['name'], 
        binproviders=providers, 
        overrides=overrides
    )
    return binary.load()  # Returns Binary with resolved abspath

Auto-Installing Missing Dependencies

# abx_dl/dependencies.py

def install_binary(spec: dict) -> Binary:
    providers_str = spec.get('binproviders', 'env')
    providers = [p for p in DEFAULT_PROVIDERS 
                 if p.name in providers_str.split(',')]
    overrides = spec.get('overrides', {})
    
    binary = Binary(
        name=spec['name'],
        binproviders=providers,
        overrides=overrides
    )
    # Attempts load() first, then install if missing

    return binary.load_or_install()

Consuming Binary Paths in Hooks

#!/usr/bin/env python3
import os
import subprocess

chrome_path = os.getenv('CHROME_BINARY')
if not chrome_path:
    raise RuntimeError("CHROME_BINARY not set")

subprocess.run([chrome_path, "--headless", "--dump-dom", "https://example.com"])

Summary

• abx-dl discovers binaries via a provider chain defined in DEFAULT_PROVIDERS (dependencies.py lines 11-31), prioritized as environment variables → pip → npm → brew → apt.
• Plugins declare requirements in binaries.jsonl, loaded by plugins.py (lines 15-22) and resolved by dependencies.pyload_binary() (lines 35-44).
• Custom paths can be supplied via environment variables, JSON overrides in the spec, or runtime JSON emission from hooks.
• Discovered paths propagate through executor.py (lines 66-74) and config.pybuild_env_for_plugin() (lines 91-115) as <NAME>_BINARY environment variables.
• With auto_install=True, missing binaries trigger automatic installation via the appropriate package manager; otherwise, plugins are skipped.

Frequently Asked Questions

What file format does abx-dl use to declare plugin binary dependencies?

abx-dl uses JSON Lines format in a file named binaries.jsonl. Each line is a JSON object containing keys like name, binproviders (comma-separated provider list), and optional overrides. The plugin loader in abx_dl/plugins.py reads this file during initialization.

How does abx-dl determine which environment variable to check for a binary?

The system converts the binary name to uppercase and appends _BINARY. For a binary named chrome, abx-dl checks the CHROME_BINARY environment variable. This convention is enforced by the EnvProvider class and the shared config logic in executor.py.

Can I prevent abx-dl from attempting to auto-install missing binaries?

Yes. Set auto_install=False when calling install_binary(), or ensure your binary is discoverable by one of the providers in DEFAULT_PROVIDERS before execution. If auto_install is disabled and the binary is not found, check_plugin_dependencies() in abx_dl/executor.py marks the plugin as missing and skips it rather than invoking package managers.

Why would I emit a Binary JSON line from a hook instead of using environment variables?

Emitting a JSON line allows dynamic discovery during execution. For example, a hook that compiles a tool from source or downloads a version-specific binary can communicate the exact path to subsequent hooks without needing to know the path before the pipeline starts. This method updates the shared configuration in real-time via executor.py (lines 66-74).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}