# How abx-dl Lazy Dependency Auto-Installation Works: A Technical Deep Dive

> Explore how abx-dl achieves lazy dependency auto-installation automatically detecting and installing missing binaries like yt-dlp before execution.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: deep-dive
- Published: 2026-02-25

---

**When you run `abx-dl`, the tool automatically detects missing external binaries (like `wget`, `yt-dlp`, or Chrome) and installs them on-the-fly before executing the requested plugin, unless you disable the feature with `--no-install`.**

The `abx-dl` command-line tool from the `archivebox/abx-dl` repository streamlines media downloads by abstracting complex plugin dependencies. Its **lazy dependency auto-installation** system ensures that required external binaries are present only when needed, eliminating manual setup while giving power users full control over their environment.

## The Three-Stage Lazy Installation Pipeline

The lazy installation mechanism operates through three distinct stages that trigger when `abx-dl` prepares to run a download task.

### Stage 1: Binary Discovery via Plugin Metadata

Each plugin declares its external binary requirements in the `plugin.binaries` attribute. This metadata specifies which system binaries (e.g., `wget`, `yt-dlp`, `chrome`) the plugin needs to function. The executor retrieves these specs before invoking any plugin hooks.

### Stage 2: Pre-Hook Dependency Validation

Before a plugin hook executes, `executor.check_plugin_dependencies()` (lines 64‑77 in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)) iterates over the binary specs. For each binary, it attempts to load the binary using `dependencies.load_binary()`.

If the binary is missing and `auto_install=True` (the default), the system calls `dependencies.install_binary()` (lines 49‑60 in [`abx_dl/dependencies.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/dependencies.py)). This function wraps `abx_pkg.Binary` and invokes `binary.load_or_install()`, which delegates to the appropriate package manager (pip, npm, brew, apt, or environment-specific providers) to fetch and install the binary.

### Stage 3: Self-Managing Crawl Hooks

Plugins that implement a *Crawl* hook (such as the Chrome plugin with [`on_Crawl__00_install_puppeteer_chromium.py`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__00_install_puppeteer_chromium.py)) follow a different path. When `check_plugin_dependencies()` detects a Crawl hook (lines 70‑77), it returns early and skips the pre-check. These plugins are assumed to handle their own dependency installation within the Crawl hook itself, typically using package managers like npm or brew directly.

## Core Implementation: Files and Functions

The lazy installation logic is distributed across three primary modules in the `archivebox/abx-dl` repository:

- **[`abx_dl/dependencies.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/dependencies.py)** (lines 35‑60): Contains `load_binary()` and `install_binary()`. These functions interface with `abx_pkg.Binary` to validate binary presence or trigger installation through the appropriate provider.

- **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** (lines 64‑77): Houses `check_plugin_dependencies()`, which orchestrates the validation flow. It checks for Crawl hooks to skip pre-validation, then iterates through binary specs to ensure availability, triggering lazy installation when configured.

- **[`abx_dl/cli.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/cli.py)** (lines 67‑74): Defines the `--no-install` flag. This CLI option inverts the default behavior by setting `auto_install=not no_install`, allowing users to disable lazy installation and skip plugins with missing dependencies rather than attempting to install them.

## Controlling Lazy Installation Behavior

While lazy installation is enabled by default, `abx-dl` provides multiple mechanisms to control or override this behavior based on your environment and security requirements.

### Default Behavior (Auto-Install Enabled)

By default, `download()` in [`executor.py`](https://github.com/archivebox/abx-dl/blob/main/executor.py) (documented at lines 94‑96) passes `auto_install=True` to `check_plugin_dependencies()`. This means missing binaries are automatically fetched and installed without user intervention:

```bash

# Missing wget, yt-dlp, or Chrome will be installed automatically

abx-dl 'https://example.com/video'

```

### Disabling Auto-Install with `--no-install`

For air-gapped environments, security-hardened systems, or when you prefer manual dependency management, use the `--no-install` flag:

```bash

# Skip plugins that require missing binaries; do not attempt installation

abx-dl --no-install 'https://example.com/video'

```

This sets `auto_install=False`, causing `check_plugin_dependencies()` to record missing binaries as warnings and skip the affected plugins rather than invoking `install_binary()`.

### Pre-Installing Dependencies

To avoid runtime installation overhead, you can pre-install all dependencies for discovered plugins:

```bash

# Install all required binaries for all plugins upfront

abx-dl plugins --install

# Subsequent runs use existing binaries without lazy installation

abx-dl 'https://example.com/video'

```

This approach is useful for Docker images or CI/CD pipelines where you want to resolve dependencies during the build phase rather than at runtime.

## Summary

- **abx-dl** implements **lazy dependency auto-installation** to ensure required external binaries are available only when needed, fetching them automatically during plugin execution.
- The system discovers binary requirements via `plugin.binaries`, validates them in `executor.check_plugin_dependencies()`, and installs missing binaries via `dependencies.install_binary()` when `auto_install=True`.
- Plugins with **Crawl hooks** (like the Chrome plugin) bypass pre-checks and handle their own dependency installation internally.
- Users control this behavior through the `--no-install` flag (disabling auto-install) or by pre-installing dependencies with `abx-dl plugins --install`.

## Frequently Asked Questions

### What happens if a required binary is missing and I don't have internet access?

If you run `abx-dl` without the `--no-install` flag and a required binary is missing, the tool attempts to download and install it automatically. Without internet access, this installation fails, and the plugin requiring that binary is skipped with a warning. To avoid this, use `abx-dl --no-install` to skip plugins with missing dependencies rather than attempting installation.

### How does abx-dl decide which package manager to use for installing binaries?

The `install_binary()` function in [`abx_dl/dependencies.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/dependencies.py) delegates to `abx_pkg.Binary.load_or_install()`, which automatically selects the appropriate provider based on your operating system and environment. It can use **pip** for Python packages, **npm** for Node.js tools, **brew** for macOS, **apt** for Debian/Ubuntu, or environment-specific providers, handling the detection and installation transparently.

### Why does the Chrome plugin skip the standard dependency check?

The Chrome plugin (and any plugin implementing a Crawl hook) follows a **self-installing** pattern. When `check_plugin_dependencies()` in [`executor.py`](https://github.com/archivebox/abx-dl/blob/main/executor.py) detects that a plugin has Crawl hooks (lines 70‑77), it returns early and skips the pre-validation. This is because the plugin's Crawl hook (such as [`on_Crawl__00_install_puppeteer_chromium.py`](https://github.com/archivebox/abx-dl/blob/main/on_Crawl__00_install_puppeteer_chromium.py)) handles its own dependency installation internally, typically using npm or brew commands directly within the hook execution.

### Can I disable lazy installation for specific plugins only?

Currently, `abx-dl` does not support granular per-plugin auto-install controls via CLI flags. The `--no-install` flag applies globally to the entire execution, causing all plugins with missing dependencies to be skipped rather than installed. However, you can achieve similar control by pre-installing specific binaries manually before running `abx-dl`, or by temporarily removing plugins from the configuration that you don't want to trigger installations.