How the abx-dl Executor Handles Hook Timeouts and Process Termination

The abx-dl executor enforces hook timeouts by passing a configurable limit to subprocess.Popen.wait(), immediately terminating processes that exceed their deadline with SIGKILL, and persisting the failure as an ArchiveResult with exit code -1 and a descriptive timeout error message.

The archivebox/abx-dl download orchestration system manages plugin-based hooks through a centralized execution pipeline. When external archival processes run indefinitely, the executor's hook timeout mechanism prevents workflow stagnation by enforcing strict time limits. This system ensures that slow or unresponsive plugins are terminated cleanly, preserving system resources and providing clear failure diagnostics to users.

Timeout Configuration and Environment Variables

In abx_dl/executor.py, the timeout value originates in the download() function where environment variables are assembled. The executor checks for a plugin-specific timeout first, falling back to a global default:

timeout = int(env.get(f"{plugin.name.upper()}_TIMEOUT", env.get('TIMEOUT', '60')))

This allows per-plugin customization (e.g., CHROME_TIMEOUT=120) while maintaining a 60-second default defined by the generic TIMEOUT variable.

The Hook Execution Flow in executor.py

The core timeout enforcement logic resides in the run_hook() function within abx_dl/executor.py. When a hook executes, the executor spawns a subprocess and monitors its duration against the configured limit.

Subprocess Initialization

The executor initializes the hook process using subprocess.Popen with the calculated working directory and environment:

process = subprocess.Popen(
    cmd,
    cwd=str(output_dir),
    stdout=out,
    stderr=err,
    env=env,
)

Timeout Enforcement Mechanism

After spawning the process, the executor blocks on process.wait() with the specified timeout parameter. If the process completes within the limit, execution proceeds normally. However, if the operation exceeds the timeout, the executor triggers a subprocess.TimeoutExpired exception handler:

try:
    returncode = process.wait(timeout=timeout)
except subprocess.TimeoutExpired:
    process.kill()
    process.wait()
    proc.exit_code = -1
    proc.stderr = f'Hook timed out after {timeout} seconds'
    # ArchiveResult creation follows...

Consequences of Exceeding a Hook Timeout

When a hook exceeds its allotted execution time, the executor performs a specific sequence of cleanup and logging actions to ensure system stability and auditability.

Immediate Process Termination

Upon timeout detection, the executor sends SIGKILL via process.kill() to force-terminate the hanging process immediately. A subsequent process.wait() call reaps the child process to prevent zombie processes from accumulating.

Failure State Recording

The executor updates the internal Process model with diagnostic metadata:

• Exit code: Set to -1 to indicate abnormal termination
• Standard error: Populated with the message "Hook timed out after X seconds"
• End timestamp: Captured via now_iso() to record the exact failure moment

ArchiveResult Creation and Propagation

The system generates an ArchiveResult object with status='failed' and the timeout error message. This result is yielded back to the caller and subsequently written to index.jsonl, providing a permanent record of the timeout event:

ar = ArchiveResult(
    snapshot_id=snapshot_id,
    plugin=hook.plugin_name,
    hook_name=hook.name,
    status='failed',
    process_id=proc.id,
    start_ts=proc.started_at,
    end_ts=proc.ended_at,
    error=proc.stderr,
)
return proc, ar, False

Output Handling

Because the hook process never completed successfully, no additional output files are generated for that specific operation. However, temporary log files created during execution are retained for debugging purposes, allowing administrators to inspect partial output leading up to the timeout.

Configuring Hook Timeouts: Practical Examples

You can customize timeout behavior programmatically or via the command line using environment variable overrides.

Python API Configuration

# Example: Run a download with a custom timeout for the "chrome" plugin

import subprocess
from abx_dl import download, plugins
from pathlib import Path

# Load plugins (normally done via `abx_dl.plugins.load_all()`)

all_plugins = plugins.load_all()

# Override the chrome timeout to 10 seconds

config_overrides = {"CHROME_TIMEOUT": "10"}

# Perform the download – the timeout is enforced inside `run_hook`

for result in download(
    url="https://example.com",
    plugins=all_plugins,
    output_dir=Path("/tmp/abx-dl-output"),
    config_overrides=config_overrides,
):
    if result.status == "failed" and "timed out" in (result.error or ""):
        print(f"⚠️ Hook {result.hook_name} timed out!")

CLI Configuration

# Direct CLI equivalent (uses the same env-variable logic)

abx-dl dl --plugins=chrome \
    --config CHROME_TIMEOUT=10 \
    "https://example.com"

In both examples, if the Chrome-related hook does not finish within 10 seconds, the executor kills the process, marks the result as failed, and includes a timeout message in the logs.

Key Components of the Timeout System

The timeout enforcement pipeline spans several modules within the archivebox/abx-dl repository:

• abx_dl/executor.py: Implements run_hook(), enforces timeouts via subprocess.Popen.wait(), handles TimeoutExpired exceptions, and constructs failure results.
• abx_dl/models.py: Defines the Process and ArchiveResult data models that capture execution metadata, exit codes, and error strings.
• abx_dl/plugins.py: Describes Hook objects and their configuration, including the is_background flag that determines timeout applicability (foreground hooks only).
• abx_dl/cli.py: Connects command-line arguments to the download() function, processing environment variable overrides for timeout configuration.

Summary

• Timeout enforcement: The executor passes a configurable timeout to process.wait(), defaulting to 60 seconds unless overridden by plugin-specific environment variables.
• Immediate termination: Expired hooks receive SIGKILL via process.kill(), followed by process reaping to prevent zombies.
• Failure recording: Timed-out hooks receive exit code -1, a descriptive stderr message, and generate an ArchiveResult with status='failed'.
• Configuration: Set TIMEOUT for global limits or {PLUGIN}_TIMEOUT for plugin-specific boundaries (e.g., CHROME_TIMEOUT).
• No partial output: Incomplete hooks do not generate final output files, though temporary logs persist for debugging.

Frequently Asked Questions

What is the default hook timeout in abx-dl?

The default hook timeout is 60 seconds, defined in the download() function within abx_dl/executor.py. This value applies when neither the plugin-specific {PLUGIN}_TIMEOUT nor the global TIMEOUT environment variable is set.

How can I configure a custom timeout for a specific plugin?

Set an environment variable using the pattern {PLUGIN_NAME}_TIMEOUT where the plugin name is uppercase. For example, set CHROME_TIMEOUT=120 to allow the Chrome plugin 120 seconds, or WGET_TIMEOUT=300 for wget operations. Pass these via --config flags in the CLI or the config_overrides dictionary in Python.

Does abx-dl send SIGTERM before SIGKILL when timing out a hook?

No. According to the source code in abx_dl/executor.py, the executor calls process.kill() immediately upon catching subprocess.TimeoutExpired. This sends SIGKILL directly without a preceding SIGTERM or grace period, ensuring immediate termination of unresponsive processes.

Where is the timeout error information stored when a hook fails?

The timeout message is stored in three locations: the Process.stderr field receives the string "Hook timed out after X seconds"; the Process.exit_code is set to -1; and the ArchiveResult.error field contains the same descriptive message. These records are persisted to index.jsonl for later analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}