How PID Files Are Created and Validated for Background Hooks in abx-dl

The abx-dl executor writes PID files with modification times set to the exact process start time, then validates them during cleanup by comparing the file mtime against the running process's create_time() to detect PID reuse before termination.

The abx-dl download manager tracks long-running daemon processes through a robust PID file mechanism implemented in the executor. When handling background hooks—identified by the .bg. suffix—the system must reliably distinguish between active processes and stale PID files to prevent accidentally killing unrelated processes. This implementation relies on kernel-level process metadata and filesystem timestamps rather than simple PID storage.

Creating PID Files for Background Hooks

When run_hook in abx_dl/executor.py detects a background hook (lines 100-103), it launches the subprocess via subprocess.Popen and immediately records the process identity. The executor generates a unique basename from the hook name and creates four associated files: <basename>.stdout.log, <basename>.stderr.log, <basename>.pid, and <basename>.sh.

The Background Hook Identification Pattern

Background hooks are distinguished by the .bg. suffix in their name, which sets the is_background flag on the Hook object. Unlike regular hooks that block until completion, these daemons require lifecycle management across the duration of a download task. The executor prepares a command script at <basename>.sh via write_cmd_file for later validation, then captures the wall-clock time immediately after process creation.

Writing Start Time Metadata to PID Files

Immediately after spawning the subprocess, the executor calls write_pid_file_with_mtime() from abx_dl/process_utils.py:

process = subprocess.Popen(cmd, cwd=str(output_dir), stdout=out, stderr=err, env=env)
process_start_time = time.time()
write_pid_file_with_mtime(pid_file, process.pid, process_start_time)

This function stores the numeric PID in the file while setting the file's modification time (mtime) to the exact process_start_time using os.utime(). By embedding the start time in the filesystem metadata rather than file content, the system creates an immutable validation token that survives process termination.

Validating PID Files to Prevent Reuse Errors

During cleanup, validate_pid_file() in abx_dl/process_utils.py (lines 28-46) performs multi-layer validation to ensure the PID still refers to the original process. This prevents the executor from sending signals to wrong processes when the operating system reassigns PIDs—a common occurrence on long-running systems.

The 5-Second Tolerance Check

The validation logic uses psutil to obtain the create_time() of the running process and compares it against the PID file's st_mtime:

def validate_pid_file(pid_file: Path, cmd_file: Optional[Path] = None, tolerance: float = 5.0) -> bool:
    if not pid_file.exists():
        return False
    pid = int(pid_file.read_text().strip())
    proc = psutil.Process(pid)
    if abs(pid_file.stat().st_mtime - proc.create_time()) > tolerance:
        return False  # PID reused or stale

    return True

If the difference between the file's modification time and the process creation time exceeds 5 seconds, validation fails. This tolerance accounts for filesystem timestamp granularity while reliably detecting PID reuse scenarios where a new process inherits an old PID number.

Optional Command Line Verification

For Chrome and Chromium-related hooks, the validation includes an additional safety check. If a .sh command file exists (passed as cmd_file), the function verifies that the running process's command line and executable name match the recorded script. This prevents edge cases where a process might have the same PID and creation time window but represents a different execution context.

Cleanup and Safe Termination

The cleanup_background_hooks function in abx_dl/executor.py (lines 21-33) orchestrates the shutdown sequence. It recursively scans for all files matching the on_*.pid pattern—the naming convention used exclusively by the executor for background hook tracking.

Graceful Versus Forced Termination

For each discovered PID file, the executor first extracts the hook basename and attempts validation:

pid_files = list(output_dir.glob('**/on_*.pid'))
for pid_file in pid_files:
    hook_basename = pid_file.stem
    cmd_file = pid_file.parent / f'{hook_basename}.sh'
    if not validate_pid_file(pid_file, cmd_file):
        pid_file.unlink(missing_ok=True)
        _finalize_background_hook(pid_file.parent, hook_basename, index_path, is_tty, success=True)
        continue

If validation fails, the PID file is removed and the hook is finalized without kill attempts. For validated processes, the executor sends SIGTERM to the entire process group using os.killpg(), falling back to the individual PID if the group signal fails. After a brief wait, it checks is_process_alive(); if the process persists, it escalates to SIGKILL (again targeting the group first) to guarantee termination. Finally, _finalize_background_hook() collects logs, writes the ArchiveResult, and cleans up all temporary files.

Summary

• PID file creation occurs immediately after subprocess.Popen returns, using write_pid_file_with_mtime() to store the PID and set the file mtime to the exact nanosecond-precision start time
• Validation in validate_pid_file() compares the file's mtime against proc.create_time() from psutil, rejecting PIDs that differ by more than 5 seconds to prevent accidental termination of reused process IDs
• Cleanup scans for on_*.pid files, validates each against its corresponding .sh command file when applicable, then sends SIGTERM to the process group followed by SIGKILL if necessary
• Finalization occurs via _finalize_background_hook(), which writes the final ArchiveResult and removes temporary log and PID files regardless of termination success

Frequently Asked Questions

What distinguishes a background hook from a regular hook in abx-dl?

Background hooks contain the .bg. suffix in their name and are treated as long-running daemons rather than blocking commands. The executor identifies these through the is_background flag on Hook objects and manages their lifecycle through PID files in the output directory instead of waiting for immediate process completion.

How does abx-dl prevent killing the wrong process during cleanup?

The system validates PID files by comparing the file's modification time—set to the original start time—against the current process's creation time using psutil. If these timestamps differ by more than 5 seconds, the executor considers the PID stale and removes the file without sending termination signals, preventing accidental kills of processes that have inherited reused PID numbers.

Why does the executor store start time as file modification time rather than file content?

File modification times are kernel-maintained metadata that remain constant unless explicitly changed, whereas file contents can be corrupted or tampered with. By using os.utime() to set the mtime to the exact process start time, the system creates an immutable validation token that survives process termination and requires no additional parsing overhead during the critical cleanup phase.

What happens if a background process ignores the initial SIGTERM signal?

The executor implements a two-phase termination sequence. It first sends SIGTERM to the process group via os.killpg(), waits briefly, then checks if the process remains alive. If the daemon is still running, it escalates to SIGKILL (first targeting the group, then falling back to the individual PID) to force immediate termination before finalizing the ArchiveResult and cleaning up temporary files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}