# How abx-dl Background Hooks (Daemon Processes) Work and Get Cleaned Up

> Learn how abx-dl background hooks run as daemon processes using subprocess.Popen and are cleaned up with a graceful SIGTERM-to-SIGKILL sequence, validating PIDs and finalizing results.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: internals
- Published: 2026-02-25

---

**abx-dl runs background hooks as daemon processes by detecting the `.bg` suffix in hook filenames, launching them with `subprocess.Popen` without blocking, and later cleaning them up via a graceful SIGTERM-to-SIGKILL sequence that validates PID files and finalizes results to `index.jsonl`.**

abx-dl is an extensible download framework that supports long-running daemon processes through **background hooks**. These hooks allow plugins to spawn persistent services—such as monitoring tasks or headless browser instances—that outlive the initial execution phase. Understanding how abx-dl declares, tracks, and terminates these background processes is essential for building reliable plugins and preventing zombie processes.

## Declaring Background Hooks via Filename Conventions

### The .bg Suffix Detection

A hook becomes a background daemon simply by adding the **`.bg`** suffix to its filename before the extension. For example, [`on_Snapshot__21_consolelog.bg.js`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__21_consolelog.bg.js) signals to the plugin loader that this script should run as a persistent background process.

In **[`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py)**, the function `parse_hook_filename` (lines 71‑85) parses the filename using a regex that captures the optional `.bg` segment. When detected, it sets `Hook.is_background = True`, flagging the hook for non-blocking execution later in the pipeline.

## Running Background Hooks in abx-dl

### Spawning Daemon Processes with run_hook()

When the main `download()` function iterates over snapshot hooks, it delegates execution to `run_hook()` in **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** (lines 71‑112). This function inspects the `Hook.is_background` flag to determine execution behavior.

For background hooks, `run_hook()` creates isolated log files (`.stdout.log`, `.stderr.log`), a PID file (`.pid`), and a shell command script (`.sh`) using unique per-hook basenames to avoid collisions across plugins (lines 77‑81). It then launches the process via `subprocess.Popen` and immediately writes the PID file.

Crucially, when `is_background` is `True`, `run_hook()` returns immediately without waiting for the process to complete, marking the initial `ArchiveResult` status as **`started`** (lines 104‑112). The daemon continues writing to its isolated log files while the main `abx-dl` command proceeds to the next hook.

### Per-Hook PID and Log File Isolation

Each background hook receives its own set of tracking files to prevent interference between plugins:

- **`<hook_basename>.pid`**: Stores the process ID for later cleanup
- **`<hook_basename>.stdout.log`**: Captures standard output
- **`<hook_basename>.stderr.log`**: Captures error streams
- **`<hook_basename>.sh`**: The generated shell command for debugging

This isolation ensures that concurrent background hooks do not corrupt each other's state or output.

## Tracking Daemon Processes

### PID File Validation and Modification Times

To prevent PID reuse attacks and stale process detection, abx-dl employs careful PID file management. The function `write_pid_file_with_mtime` in **[`abx_dl/process_utils.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/process_utils.py)** sets the PID file’s modification time to the exact process start time (referenced in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) lines 100‑103).

During cleanup, `validate_pid_file` checks whether the PID file’s modification time matches the actual process start time. If the PID has been reused by a different process or the original process died, validation fails and the stale PID file is removed safely.

## Cleaning Up abx-dl Background Hooks

When the overall download finishes or the user aborts, abx-dl calls `cleanup_background_hooks()` in **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** (lines 15‑88). This function implements a robust termination sequence:

### Discovery and Validation of PID Files

The cleanup process begins by globbing for all `on_*.pid` files under the output directory, explicitly ignoring plugin-specific PID files such as `chrome.pid` (lines 15‑18). Each discovered PID file undergoes validation via `validate_pid_file` to ensure the process is still the original daemon and not a recycled PID (lines 25‑33).

### Graceful Termination with SIGTERM

For valid daemons, abx-dl first attempts graceful shutdown by sending **SIGTERM** to the entire process group using `os.killpg` (lines 37‑45). If the process group kill fails (e.g., the process detached), it falls back to a direct SIGTERM to the specific PID.

### Waiting and Force Kill with SIGKILL

After sending SIGTERM, the code pauses for 2 seconds then checks `is_process_alive` (lines 50‑58). If the process exited cleanly, the PID file is removed and the hook is finalized as successful.

If the daemon persists, abx-dl escalates to **SIGKILL** sent to the entire process group (lines 60‑68). This handles detached child processes like Chrome that ignore SIGTERM.

### Final Verification and Unkillable Processes

Following the force kill, another verification check occurs. If the process is *still* alive (e.g., due to a macOS kernel-crashed Chrome), it is deemed **unkillable** (lines 77‑84). A warning is printed, the hook is finalized with `success=False`, and the PID file remains for manual inspection. Otherwise, the PID file is removed and the hook marked successful (lines 85‑88).

### Finalizing Results to index.jsonl

The `_finalize_background_hook()` function (lines 94‑114) reads the hook’s stdout/stderr logs, parses any `ArchiveResult` JSON lines, creates a `Process` object and final `ArchiveResult`, and appends both to `index.jsonl`. This ensures background hooks appear in the final result set exactly like synchronous hooks, after which the temporary log files are deleted.

## Code Examples

### Creating a Background Hook Plugin

Here is a complete example of a background hook that emits periodic status updates:

```python

# plugins/monitor/on_Snapshot__20_monitor.bg.py

#!/usr/bin/env python3
import time
import json
import sys

def get_snapshot_id():
    for arg in sys.argv[1:]:
        if arg.startswith("snapshot_id="):
            return arg.split("=", 1)[1]
    return "unknown"

snapshot_id = get_snapshot_id()

while True:
    result = {
        "type": "ArchiveResult",
        "snapshot_id": snapshot_id,
        "plugin": "monitor",
        "hook_name": "on_Snapshot__20_monitor.bg",
        "status": "running",
        "output_str": "Monitoring active"
    }
    print(json.dumps(result))
    sys.stdout.flush()
    time.sleep(5)

```

The `.bg` suffix ensures abx-dl treats this script as a daemon, returning control immediately while the loop continues running.

### Running abx-dl with Background Hooks

Execute a download with your background hook enabled:

```bash

# Run download with the monitor plugin active

abx-dl dl --plugins=monitor 'https://example.com' --output-dir=./output

```

During execution, the CLI indicates background status. When the main download completes, `cleanup_background_hooks()` automatically terminates the monitor daemon and appends its final results to `./output/index.jsonl`.

### Manual Cleanup Invocation

If you interrupt a download with Ctrl-C, background hooks may persist. Clean them up manually:

```python
from pathlib import Path
from abx_dl.executor import cleanup_background_hooks

output_dir = Path("/tmp/abx-dl-output")
index_path = output_dir / "index.jsonl"

# Execute cleanup outside normal workflow

cleanup_background_hooks(output_dir, index_path, is_tty=False)

```

This performs the same validation, SIGTERM/SIGKILL sequence, and result finalization as the normal workflow.

## Key Files in abx-dl Background Hook Handling

| File | Role in Background Hook Lifecycle |
|------|-----------------------------------|
| **[`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py)** | Parses hook filenames via `parse_hook_filename` (lines 71‑85) to detect the `.bg` suffix and set `Hook.is_background`. |
| **[`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py)** | Core execution engine containing `run_hook()` (lines 71‑112) to spawn daemons, `cleanup_background_hooks()` (lines 15‑88) for termination, and `_finalize_background_hook()` (lines 94‑114) for result integration. |
| **[`abx_dl/process_utils.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/process_utils.py)** | Utility functions including `write_pid_file_with_mtime`, `validate_pid_file`, and `is_process_alive` used for PID management and process validation. |
| **[`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py)** | Defines `Process` and `ArchiveResult` dataclasses that background hooks populate and that get serialized to `index.jsonl` during finalization. |

These files work together to enable **abx-dl background hooks** to run as independent processes, write isolated logs, and shut down safely without leaving zombie processes or orphaned data.

## Summary

- **Declaration**: Background hooks are declared by adding the `.bg` suffix to the hook filename, which `parse_hook_filename` in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) detects to set `is_background = True`.
- **Execution**: The `run_hook()` function in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) spawns the process via `subprocess.Popen`, writes isolated PID and log files, and returns immediately for background hooks with status `started`.
- **Tracking**: Each daemon is tracked via a dedicated PID file with modification time set to process start time, enabling validation against PID reuse via `validate_pid_file` in [`abx_dl/process_utils.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/process_utils.py).
- **Cleanup**: The `cleanup_background_hooks()` function implements a graceful-to-force termination sequence: SIGTERM to process group, 2-second wait, SIGKILL if necessary, and final verification to handle unkillable processes.
- **Finalization**: Results are integrated into the main output via `_finalize_background_hook()`, which parses logs, creates `ArchiveResult` entries, and appends them to `index.jsonl` before deleting temporary files.

## Frequently Asked Questions

### How does abx-dl know which hooks to run as background daemons?

abx-dl uses a filename convention: hooks intended to run as daemons must include **`.bg`** before the file extension (e.g., [`on_Snapshot__10_watch.bg.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__10_watch.bg.py)). The `parse_hook_filename` function in [`abx_dl/plugins.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/plugins.py) (lines 71‑85) detects this suffix and sets `Hook.is_background = True`, causing the executor to treat the hook as a non-blocking daemon.

### What prevents PID reuse from corrupting the cleanup process?

abx-dl mitigates PID reuse through modification time validation. When spawning a daemon, `write_pid_file_with_mtime` in [`abx_dl/process_utils.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/process_utils.py) sets the PID file’s mtime to the exact process start time. During cleanup, `validate_pid_file` compares this mtime against the actual process start time; if they differ, the PID file is considered stale and removed safely without touching the recycled process.

### How does abx-dl handle background hooks that spawn child processes like Chrome?

The cleanup routine targets entire process groups to catch detached children. The `cleanup_background_hooks()` function in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) first attempts `os.killpg` to send SIGTERM to the process group (lines 37‑45). If the daemon persists after the grace period, it sends SIGKILL to the process group (lines 60‑68). This approach handles browsers and other tools that spawn independent child processes that would otherwise survive the parent’s termination.

### Where can I find the final output from a background hook after cleanup completes?

Background hook results are integrated into the main archive index. The `_finalize_background_hook()` function in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) (lines 94‑114) reads the hook’s `.stdout.log` and `.stderr.log`, parses any JSON `ArchiveResult` lines, creates a final `ArchiveResult` object, and appends it to `index.jsonl` in the output directory. Temporary log files are then deleted, leaving the permanent record in the JSONL index.