How Plugins Output Binary Records to Register Custom Binary Paths in abx-dl

Plugins register custom binary paths by printing a JSON line to STDOUT containing {"type": "Binary", "name": "<NAME>", "abspath": "<PATH>"}, which the executor parses and injects as environment variables for subsequent hooks.

The abx-dl framework from the ArchiveBox ecosystem provides a standardized mechanism for plugins to dynamically discover and register executable binaries. When a plugin locates a required binary—such as Chrome, ffmpeg, or a custom tool—it can communicate that path to the executor, ensuring downstream hooks automatically receive the correct location via environment variables without hard-coding paths.

The Binary Record Protocol

Plugins emit Binary records as newline-delimited JSON objects written directly to standard output. The structure follows a strict schema:

{"type": "Binary", "name": "<BINARY_NAME>", "abspath": "<ABSOLUTE_PATH>"}

• type: Must be the string "Binary" to trigger the registration logic.
• name: The canonical name of the binary (e.g., "chrome", "ffmpeg"). The executor normalizes this to uppercase when creating environment variables.
• abspath: The absolute filesystem path to the executable.

This design mirrors ArchiveBox’s binary discovery convention, allowing any plugin to act as a binary provider without modifying global configuration files.

How the Executor Processes Binary Records

The registration mechanism operates within the hook execution loop defined in abx_dl/executor.py. When the executor runs a plugin hook, it captures the process’s STDOUT and scans each line for JSON records.

Parsing STDOUT in abx_dl/executor.py

During snapshot execution, the executor iterates over the captured output (lines 66‑79 of abx_dl/executor.py), attempting to parse each non-empty line as JSON:

for line in proc.stdout.split('\n'):
    if line.strip():
        try:
            record = json.loads(line)
            if record.get('type') == 'Binary':
                name = record.get('name', '')
                abspath = record.get('abspath', '')
                if name and abspath:
                    # Make the path available to later plugins

                    shared_config[f'{name.upper()}_BINARY'] = abspath
        except json.JSONDecodeError:
            pass

Storing Paths in Shared Configuration

When the executor identifies a valid Binary record, it extracts the name and abspath fields. The executor then updates a shared configuration dictionary, mapping the normalized binary name to its absolute path using the key pattern {NAME}_BINARY.

This shared dictionary persists across the lifecycle of a snapshot, acting as a transient registry for binary locations discovered by earlier hooks.

Propagating Binary Paths to Subsequent Hooks

After a hook completes, the executor must ensure that discovered binary paths are accessible to downstream plugins. This occurs through environment variable injection handled in abx_dl/config.py.

The function build_env_for_plugin merges the shared configuration into the environment for the next plugin. Specifically, it injects any keys ending with _BINARY as environment variables. For example, if a previous hook registered a Chrome binary at /usr/bin/google-chrome, subsequent hooks receive:

CHROME_BINARY=/usr/bin/google-chrome

Plugins can then locate the binary using standard environment variable lookups, eliminating the need for redundant discovery logic or hard-coded paths.

Practical Implementation Examples

Emitting a Binary Record from a Plugin

The following Python snippet demonstrates how a plugin locates a custom Chrome installation and registers it for downstream use:

import json
import sys
import shutil

def register_chrome():
    # Custom logic to locate the binary

    chrome_path = "/opt/custom/google-chrome"
    
    # Verify the binary exists

    if not shutil.which(chrome_path):
        raise FileNotFoundError(f"Chrome not found at {chrome_path}")
    
    # Emit the Binary record to STDOUT

    record = {
        "type": "Binary",
        "name": "chrome",
        "abspath": chrome_path
    }
    print(json.dumps(record))
    sys.stdout.flush()

def main():
    register_chrome()
    # Continue with normal plugin execution...

    print("Chrome registered successfully")

if __name__ == "__main__":
    main()

When executed by the abx-dl executor, this plugin outputs {"type": "Binary", "name": "chrome", "abspath": "/opt/custom/google-chrome"}, making the path available to all subsequent hooks.

Consuming the Binary Path in a Later Hook

Downstream plugins retrieve the registered binary path through environment variables. The following example shows how a later hook accesses the Chrome binary registered by an earlier plugin:

import os
import subprocess

def capture_screenshot(url):
    # Retrieve the binary path from the environment

    chrome_bin = os.getenv("CHROME_BINARY")
    
    if not chrome_bin:
        raise RuntimeError("CHROME_BINARY not set. Ensure a previous plugin registered the Chrome binary.")
    
    # Use the binary in a subprocess

    cmd = [
        chrome_bin,
        "--headless",
        "--screenshot=output.png",
        url
    ]
    
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.returncode == 0

# Example usage

if __name__ == "__main__":
    success = capture_screenshot("https://example.com")
    print(f"Screenshot captured: {success}")

This pattern ensures that binary discovery happens once—typically in an early setup hook—and all subsequent operations reuse the validated path without redundant filesystem checks.

Summary

• Binary records are JSON lines emitted to STDOUT with the schema {"type": "Binary", "name": "...", "abspath": "..."}.
• The executor in abx_dl/executor.py (lines 66‑79) parses these records during hook execution and stores paths in a shared configuration dictionary using the {NAME}_BINARY key pattern.
• abx_dl/config.py propagates these values to subsequent hooks via environment variables, enabling downstream plugins to locate binaries through standard os.getenv() calls.
• This mechanism allows plugins to dynamically register custom binary paths without modifying global configuration files or requiring redundant discovery logic in every hook.

Frequently Asked Questions

What happens if a plugin outputs invalid JSON to STDOUT?

The executor wraps each JSON parsing attempt in a try-except block that catches json.JSONDecodeError. If a line fails to parse, the executor silently ignores that line and continues processing subsequent lines. Only valid JSON objects with "type": "Binary" trigger the registration logic.

Can multiple plugins register different binaries in the same snapshot?

Yes. The shared configuration dictionary accumulates binary registrations throughout the snapshot lifecycle. Each plugin can emit multiple Binary records, and subsequent hooks receive all registered paths as environment variables. If two plugins register the same binary name, the later registration overwrites the earlier one in the shared configuration.

How does a plugin access binaries registered by previous hooks?

Plugins retrieve registered binary paths through standard environment variable lookups. The build_env_for_plugin function in abx_dl/config.py injects each binary registration as an environment variable using the uppercase binary name followed by _BINARY. For example, a registration with "name": "chrome" becomes accessible via os.getenv("CHROME_BINARY").

Is there a specific order in which hooks must run to ensure binaries are available?

While there is no enforced global order, plugin authors should design their hook stages so that binary discovery occurs in early stages (such as setup or init) before execution stages that consume those binaries. The executor processes hooks sequentially within a snapshot, so as long as a binary-emitting hook runs before a consuming hook, the environment variable will be populated and available.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}