How abx-dl Background Hooks (Daemon Processes) Work and Get Cleaned Up

abx-dl runs background hooks as daemon processes by detecting the .bg suffix in hook filenames, launching them with subprocess.Popen without blocking, and later cleaning them up via a graceful SIGTERM-to-SIGKILL sequence that validates PID files and finalizes results to index.jsonl.

abx-dl is an extensible download framework that supports long-running daemon processes through background hooks. These hooks allow plugins to spawn persistent services—such as monitoring tasks or headless browser instances—that outlive the initial execution phase. Understanding how abx-dl declares, tracks, and terminates these background processes is essential for building reliable plugins and preventing zombie processes.

Declaring Background Hooks via Filename Conventions

The .bg Suffix Detection

A hook becomes a background daemon simply by adding the .bg suffix to its filename before the extension. For example, on_Snapshot__21_consolelog.bg.js signals to the plugin loader that this script should run as a persistent background process.

In abx_dl/plugins.py, the function parse_hook_filename (lines 71‑85) parses the filename using a regex that captures the optional .bg segment. When detected, it sets Hook.is_background = True, flagging the hook for non-blocking execution later in the pipeline.

Running Background Hooks in abx-dl

Spawning Daemon Processes with run_hook()

When the main download() function iterates over snapshot hooks, it delegates execution to run_hook() in abx_dl/executor.py (lines 71‑112). This function inspects the Hook.is_background flag to determine execution behavior.

For background hooks, run_hook() creates isolated log files (.stdout.log, .stderr.log), a PID file (.pid), and a shell command script (.sh) using unique per-hook basenames to avoid collisions across plugins (lines 77‑81). It then launches the process via subprocess.Popen and immediately writes the PID file.

Crucially, when is_background is True, run_hook() returns immediately without waiting for the process to complete, marking the initial ArchiveResult status as started (lines 104‑112). The daemon continues writing to its isolated log files while the main abx-dl command proceeds to the next hook.

Per-Hook PID and Log File Isolation

Each background hook receives its own set of tracking files to prevent interference between plugins:

• <hook_basename>.pid: Stores the process ID for later cleanup
• <hook_basename>.stdout.log: Captures standard output
• <hook_basename>.stderr.log: Captures error streams
• <hook_basename>.sh: The generated shell command for debugging

This isolation ensures that concurrent background hooks do not corrupt each other's state or output.

Tracking Daemon Processes

PID File Validation and Modification Times

To prevent PID reuse attacks and stale process detection, abx-dl employs careful PID file management. The function write_pid_file_with_mtime in abx_dl/process_utils.py sets the PID file’s modification time to the exact process start time (referenced in abx_dl/executor.py lines 100‑103).

During cleanup, validate_pid_file checks whether the PID file’s modification time matches the actual process start time. If the PID has been reused by a different process or the original process died, validation fails and the stale PID file is removed safely.

Cleaning Up abx-dl Background Hooks

When the overall download finishes or the user aborts, abx-dl calls cleanup_background_hooks() in abx_dl/executor.py (lines 15‑88). This function implements a robust termination sequence:

Discovery and Validation of PID Files

The cleanup process begins by globbing for all on_*.pid files under the output directory, explicitly ignoring plugin-specific PID files such as chrome.pid (lines 15‑18). Each discovered PID file undergoes validation via validate_pid_file to ensure the process is still the original daemon and not a recycled PID (lines 25‑33).

Graceful Termination with SIGTERM

For valid daemons, abx-dl first attempts graceful shutdown by sending SIGTERM to the entire process group using os.killpg (lines 37‑45). If the process group kill fails (e.g., the process detached), it falls back to a direct SIGTERM to the specific PID.

Waiting and Force Kill with SIGKILL

After sending SIGTERM, the code pauses for 2 seconds then checks is_process_alive (lines 50‑58). If the process exited cleanly, the PID file is removed and the hook is finalized as successful.

If the daemon persists, abx-dl escalates to SIGKILL sent to the entire process group (lines 60‑68). This handles detached child processes like Chrome that ignore SIGTERM.

Final Verification and Unkillable Processes

Following the force kill, another verification check occurs. If the process is still alive (e.g., due to a macOS kernel-crashed Chrome), it is deemed unkillable (lines 77‑84). A warning is printed, the hook is finalized with success=False, and the PID file remains for manual inspection. Otherwise, the PID file is removed and the hook marked successful (lines 85‑88).

Finalizing Results to index.jsonl

The _finalize_background_hook() function (lines 94‑114) reads the hook’s stdout/stderr logs, parses any ArchiveResult JSON lines, creates a Process object and final ArchiveResult, and appends both to index.jsonl. This ensures background hooks appear in the final result set exactly like synchronous hooks, after which the temporary log files are deleted.

Code Examples

Creating a Background Hook Plugin

Here is a complete example of a background hook that emits periodic status updates:

# plugins/monitor/on_Snapshot__20_monitor.bg.py

#!/usr/bin/env python3
import time
import json
import sys

def get_snapshot_id():
    for arg in sys.argv[1:]:
        if arg.startswith("snapshot_id="):
            return arg.split("=", 1)[1]
    return "unknown"

snapshot_id = get_snapshot_id()

while True:
    result = {
        "type": "ArchiveResult",
        "snapshot_id": snapshot_id,
        "plugin": "monitor",
        "hook_name": "on_Snapshot__20_monitor.bg",
        "status": "running",
        "output_str": "Monitoring active"
    }
    print(json.dumps(result))
    sys.stdout.flush()
    time.sleep(5)

The .bg suffix ensures abx-dl treats this script as a daemon, returning control immediately while the loop continues running.

Running abx-dl with Background Hooks

Execute a download with your background hook enabled:

# Run download with the monitor plugin active

abx-dl dl --plugins=monitor 'https://example.com' --output-dir=./output

During execution, the CLI indicates background status. When the main download completes, cleanup_background_hooks() automatically terminates the monitor daemon and appends its final results to ./output/index.jsonl.

Manual Cleanup Invocation

If you interrupt a download with Ctrl-C, background hooks may persist. Clean them up manually:

from pathlib import Path
from abx_dl.executor import cleanup_background_hooks

output_dir = Path("/tmp/abx-dl-output")
index_path = output_dir / "index.jsonl"

# Execute cleanup outside normal workflow

cleanup_background_hooks(output_dir, index_path, is_tty=False)

This performs the same validation, SIGTERM/SIGKILL sequence, and result finalization as the normal workflow.

Key Files in abx-dl Background Hook Handling

File	Role in Background Hook Lifecycle
abx_dl/plugins.py	Parses hook filenames via parse_hook_filename (lines 71‑85) to detect the .bg suffix and set Hook.is_background.
abx_dl/executor.py	Core execution engine containing run_hook() (lines 71‑112) to spawn daemons, cleanup_background_hooks() (lines 15‑88) for termination, and _finalize_background_hook() (lines 94‑114) for result integration.
abx_dl/process_utils.py	Utility functions including write_pid_file_with_mtime, validate_pid_file, and is_process_alive used for PID management and process validation.
abx_dl/models.py	Defines Process and ArchiveResult dataclasses that background hooks populate and that get serialized to index.jsonl during finalization.

These files work together to enable abx-dl background hooks to run as independent processes, write isolated logs, and shut down safely without leaving zombie processes or orphaned data.

Summary

• Declaration: Background hooks are declared by adding the .bg suffix to the hook filename, which parse_hook_filename in abx_dl/plugins.py detects to set is_background = True.
• Execution: The run_hook() function in abx_dl/executor.py spawns the process via subprocess.Popen, writes isolated PID and log files, and returns immediately for background hooks with status started.
• Tracking: Each daemon is tracked via a dedicated PID file with modification time set to process start time, enabling validation against PID reuse via validate_pid_file in abx_dl/process_utils.py.
• Cleanup: The cleanup_background_hooks() function implements a graceful-to-force termination sequence: SIGTERM to process group, 2-second wait, SIGKILL if necessary, and final verification to handle unkillable processes.
• Finalization: Results are integrated into the main output via _finalize_background_hook(), which parses logs, creates ArchiveResult entries, and appends them to index.jsonl before deleting temporary files.

Frequently Asked Questions

How does abx-dl know which hooks to run as background daemons?

abx-dl uses a filename convention: hooks intended to run as daemons must include .bg before the file extension (e.g., on_Snapshot__10_watch.bg.py). The parse_hook_filename function in abx_dl/plugins.py (lines 71‑85) detects this suffix and sets Hook.is_background = True, causing the executor to treat the hook as a non-blocking daemon.

What prevents PID reuse from corrupting the cleanup process?

abx-dl mitigates PID reuse through modification time validation. When spawning a daemon, write_pid_file_with_mtime in abx_dl/process_utils.py sets the PID file’s mtime to the exact process start time. During cleanup, validate_pid_file compares this mtime against the actual process start time; if they differ, the PID file is considered stale and removed safely without touching the recycled process.

How does abx-dl handle background hooks that spawn child processes like Chrome?

The cleanup routine targets entire process groups to catch detached children. The cleanup_background_hooks() function in abx_dl/executor.py first attempts os.killpg to send SIGTERM to the process group (lines 37‑45). If the daemon persists after the grace period, it sends SIGKILL to the process group (lines 60‑68). This approach handles browsers and other tools that spawn independent child processes that would otherwise survive the parent’s termination.

Where can I find the final output from a background hook after cleanup completes?

Background hook results are integrated into the main archive index. The _finalize_background_hook() function in abx_dl/executor.py (lines 94‑114) reads the hook’s .stdout.log and .stderr.log, parses any JSON ArchiveResult lines, creates a final ArchiveResult object, and appends it to index.jsonl in the output directory. Temporary log files are then deleted, leaving the permanent record in the JSONL index.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}