# How the abx-dl Executor Handles Hook Timeouts and Process Termination

> Learn how the abx-dl executor enforces hook timeouts using subprocess Popen wait. Discover its SIGKILL termination and how it logs timeout errors for failed archives.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: internals
- Published: 2026-02-25

---

**The abx-dl executor enforces hook timeouts by passing a configurable limit to `subprocess.Popen.wait()`, immediately terminating processes that exceed their deadline with SIGKILL, and persisting the failure as an `ArchiveResult` with exit code -1 and a descriptive timeout error message.**

The `archivebox/abx-dl` download orchestration system manages plugin-based hooks through a centralized execution pipeline. When external archival processes run indefinitely, the executor's hook timeout mechanism prevents workflow stagnation by enforcing strict time limits. This system ensures that slow or unresponsive plugins are terminated cleanly, preserving system resources and providing clear failure diagnostics to users.

## Timeout Configuration and Environment Variables

In [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), the timeout value originates in the `download()` function where environment variables are assembled. The executor checks for a plugin-specific timeout first, falling back to a global default:

```python
timeout = int(env.get(f"{plugin.name.upper()}_TIMEOUT", env.get('TIMEOUT', '60')))

```

This allows per-plugin customization (e.g., `CHROME_TIMEOUT=120`) while maintaining a 60-second default defined by the generic `TIMEOUT` variable.

## The Hook Execution Flow in executor.py

The core timeout enforcement logic resides in the `run_hook()` function within [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py). When a hook executes, the executor spawns a subprocess and monitors its duration against the configured limit.

### Subprocess Initialization

The executor initializes the hook process using `subprocess.Popen` with the calculated working directory and environment:

```python
process = subprocess.Popen(
    cmd,
    cwd=str(output_dir),
    stdout=out,
    stderr=err,
    env=env,
)

```

### Timeout Enforcement Mechanism

After spawning the process, the executor blocks on `process.wait()` with the specified timeout parameter. If the process completes within the limit, execution proceeds normally. However, if the operation exceeds the timeout, the executor triggers a `subprocess.TimeoutExpired` exception handler:

```python
try:
    returncode = process.wait(timeout=timeout)
except subprocess.TimeoutExpired:
    process.kill()
    process.wait()
    proc.exit_code = -1
    proc.stderr = f'Hook timed out after {timeout} seconds'
    # ArchiveResult creation follows...

```

## Consequences of Exceeding a Hook Timeout

When a hook exceeds its allotted execution time, the executor performs a specific sequence of cleanup and logging actions to ensure system stability and auditability.

### Immediate Process Termination

Upon timeout detection, the executor sends **SIGKILL** via `process.kill()` to force-terminate the hanging process immediately. A subsequent `process.wait()` call reaps the child process to prevent zombie processes from accumulating.

### Failure State Recording

The executor updates the internal `Process` model with diagnostic metadata:
- **Exit code**: Set to `-1` to indicate abnormal termination
- **Standard error**: Populated with the message `"Hook timed out after X seconds"`
- **End timestamp**: Captured via `now_iso()` to record the exact failure moment

### ArchiveResult Creation and Propagation

The system generates an `ArchiveResult` object with `status='failed'` and the timeout error message. This result is yielded back to the caller and subsequently written to `index.jsonl`, providing a permanent record of the timeout event:

```python
ar = ArchiveResult(
    snapshot_id=snapshot_id,
    plugin=hook.plugin_name,
    hook_name=hook.name,
    status='failed',
    process_id=proc.id,
    start_ts=proc.started_at,
    end_ts=proc.ended_at,
    error=proc.stderr,
)
return proc, ar, False

```

### Output Handling

Because the hook process never completed successfully, no additional output files are generated for that specific operation. However, temporary log files created during execution are retained for debugging purposes, allowing administrators to inspect partial output leading up to the timeout.

## Configuring Hook Timeouts: Practical Examples

You can customize timeout behavior programmatically or via the command line using environment variable overrides.

### Python API Configuration

```python

# Example: Run a download with a custom timeout for the "chrome" plugin

import subprocess
from abx_dl import download, plugins
from pathlib import Path

# Load plugins (normally done via `abx_dl.plugins.load_all()`)

all_plugins = plugins.load_all()

# Override the chrome timeout to 10 seconds

config_overrides = {"CHROME_TIMEOUT": "10"}

# Perform the download – the timeout is enforced inside `run_hook`

for result in download(
    url="https://example.com",
    plugins=all_plugins,
    output_dir=Path("/tmp/abx-dl-output"),
    config_overrides=config_overrides,
):
    if result.status == "failed" and "timed out" in (result.error or ""):
        print(f"⚠️ Hook {result.hook_name} timed out!")

```

### CLI Configuration

```bash

# Direct CLI equivalent (uses the same env-variable logic)

abx-dl dl --plugins=chrome \
    --config CHROME_TIMEOUT=10 \
    "https://example.com"

```

In both examples, if the Chrome-related hook does not finish within **10 seconds**, the executor kills the process, marks the result as failed, and includes a timeout message in the logs.

## Key Components of the Timeout System

The timeout enforcement pipeline spans several modules within the `archivebox/abx-dl` repository:

- **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)**: Implements `run_hook()`, enforces timeouts via `subprocess.Popen.wait()`, handles `TimeoutExpired` exceptions, and constructs failure results.
- **[`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py)**: Defines the `Process` and `ArchiveResult` data models that capture execution metadata, exit codes, and error strings.
- **[`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py)**: Describes `Hook` objects and their configuration, including the `is_background` flag that determines timeout applicability (foreground hooks only).
- **[`abx_dl/cli.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/cli.py)**: Connects command-line arguments to the `download()` function, processing environment variable overrides for timeout configuration.

## Summary

- **Timeout enforcement**: The executor passes a configurable timeout to `process.wait()`, defaulting to 60 seconds unless overridden by plugin-specific environment variables.
- **Immediate termination**: Expired hooks receive SIGKILL via `process.kill()`, followed by process reaping to prevent zombies.
- **Failure recording**: Timed-out hooks receive exit code `-1`, a descriptive stderr message, and generate an `ArchiveResult` with `status='failed'`.
- **Configuration**: Set `TIMEOUT` for global limits or `{PLUGIN}_TIMEOUT` for plugin-specific boundaries (e.g., `CHROME_TIMEOUT`).
- **No partial output**: Incomplete hooks do not generate final output files, though temporary logs persist for debugging.

## Frequently Asked Questions

### What is the default hook timeout in abx-dl?

The default hook timeout is **60 seconds**, defined in the `download()` function within [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py). This value applies when neither the plugin-specific `{PLUGIN}_TIMEOUT` nor the global `TIMEOUT` environment variable is set.

### How can I configure a custom timeout for a specific plugin?

Set an environment variable using the pattern `{PLUGIN_NAME}_TIMEOUT` where the plugin name is uppercase. For example, set `CHROME_TIMEOUT=120` to allow the Chrome plugin 120 seconds, or `WGET_TIMEOUT=300` for wget operations. Pass these via `--config` flags in the CLI or the `config_overrides` dictionary in Python.

### Does abx-dl send SIGTERM before SIGKILL when timing out a hook?

**No.** According to the source code in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), the executor calls `process.kill()` immediately upon catching `subprocess.TimeoutExpired`. This sends SIGKILL directly without a preceding SIGTERM or grace period, ensuring immediate termination of unresponsive processes.

### Where is the timeout error information stored when a hook fails?

The timeout message is stored in three locations: the `Process.stderr` field receives the string `"Hook timed out after X seconds"`; the `Process.exit_code` is set to `-1`; and the `ArchiveResult.error` field contains the same descriptive message. These records are persisted to `index.jsonl` for later analysis.