# How abx-dl Detects New Files Created by Hooks and Excludes Log/PID Files from Output

> Learn how abx-dl detects new files from hooks by comparing directory snapshots and excludes log pid and sh files with suffix rules preventing junk in your output.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: internals
- Published: 2026-02-25

---

**abx-dl detects new files created by hooks by taking directory snapshots before and after execution, then filters out bookkeeping artifacts like `.stdout.log`, `.stderr.log`, `.pid`, and `.sh` files using suffix-based exclusion rules.**

abx-dl is a Python-based archival tool that executes plugin hooks to capture web content. When hooks run, they generate output files, but the system also creates temporary logs, PID files, and shell scripts for process management. The challenge lies in distinguishing genuine hook output from these operational artifacts to ensure only relevant files appear in the final `output_files` list.

## The Challenge: Distinguishing Hook Output from Operational Artifacts

When `abx-dl` executes a hook via the `run_hook()` function in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), it spawns a subprocess and redirects stdout and stderr to log files, writes a PID file for process tracking, and generates a temporary shell script to wrap the command. These bookkeeping files coexist in the same output directory as the actual files the hook intends to create (such as PDFs, screenshots, or HTML archives). Without explicit filtering, these temporary artifacts would pollute the snapshot's `output_files` metadata.

## How abx-dl Detects New Files Created by Hooks

The detection mechanism relies on a before-and-after directory comparison strategy implemented in the `run_hook()` function.

### Capturing the Pre-Execution State

Before spawning the hook process, `abx-dl` captures a complete snapshot of the output directory using `pathlib.Path.rglob()`. This creates a set of all existing file paths:

```python

# Capture files before execution to detect new output

files_before = set(output_dir.rglob('*')) if output_dir.exists() else set()

```

This set serves as the baseline for comparison, ensuring that any files existing prior to the hook's execution are not mistakenly flagged as new output.

### Executing the Hook and Generating Bookkeeping Files

During execution, the system writes four types of temporary files to the output directory:

- `*.stdout.log`: Captures standard output
- `*.stderr.log`: Captures standard error
- `*.pid`: Stores the process ID for background process management
- `*.sh`: The generated shell script wrapping the hook command

These files are necessary for process management and debugging but must be excluded from the final output list.

### Identifying New Files via Set Difference

After the hook process terminates, `abx-dl` captures a second snapshot and computes the difference between the two sets:

```python
files_after = set(output_dir.rglob('*')) if output_dir.exists() else set()
new_files = [str(f.relative_to(output_dir))
             for f in (files_after - files_before) if f.is_file()]

```

This set subtraction (`files_after - files_before`) yields only the paths created during the hook's execution, including both genuine output and the bookkeeping files.

## Filtering Out Log, PID, and Script Files

Once the new files are identified, `abx-dl` applies a suffix-based filter to remove the operational artifacts.

### The Excluded Suffixes Tuple

The system defines a tuple of suffixes that correspond to the bookkeeping files generated during execution:

```python
excluded_suffixes = ('.stdout.log', '.stderr.log', '.pid', '.sh')

```

This explicit list ensures that even if a hook legitimately creates files with similar names, only these specific patterns generated by the executor are removed.

### Suffix-Based Filtering Implementation

The filtering logic uses a list comprehension with `str.endswith()` to strip out the unwanted paths:

```python
new_files = [f for f in new_files
             if not any(f.endswith(suffix) for suffix in excluded_suffixes)]

```

After this operation, `new_files` contains only the genuine output produced by the hook, free from stdout logs, stderr logs, PID files, and temporary shell scripts.

## Cleaning Up Temporary Bookkeeping Files

After successfully filtering the output list, `abx-dl` performs cleanup to remove the temporary files from the filesystem. This occurs when the hook exits with a return code of zero:

```python
if returncode == 0:
    stdout_file.unlink(missing_ok=True)
    stderr_file.unlink(missing_ok=True)
    pid_file.unlink(missing_ok=True)

```

This ensures that successful hook executions leave behind only the intended output files, maintaining a clean snapshot directory. If the hook fails, these files are retained for debugging purposes.

## Storing the Filtered Results in ArchiveResult

The final filtered list is persisted in the `ArchiveResult` object, which tracks the outcome of each hook execution. In [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), the `ArchiveResult` is instantiated with the cleaned `new_files` list:

```python
ar = ArchiveResult(
    snapshot_id=snapshot_id,
    plugin=hook.plugin_name,
    hook_name=hook.name,
    status='succeeded' if returncode == 0 else 'failed',
    output_files=new_files,
    # ... additional fields ...

)

```

This structured result is then used by the rest of the `abx-dl` pipeline to index and serve the captured content, ensuring that only relevant files are exposed to the user interface and search indices.

## Summary

- **abx-dl** detects hook-generated files by comparing directory snapshots taken before and after hook execution using `rglob()` and set difference operations.
- The system explicitly filters out operational artifacts—`.stdout.log`, `.stderr.log`, `.pid`, and `.sh` files—using a suffix-based exclusion tuple.
- Temporary bookkeeping files are unlinked after successful execution to maintain clean output directories.
- Only filtered, relevant files are stored in the `ArchiveResult.output_files` field for downstream processing.

## Frequently Asked Questions

### How does abx-dl distinguish between existing files and new files created by hooks?

abx-dl captures a set of all files in the output directory before the hook runs using `output_dir.rglob('*')`, then captures another set after execution. It computes the difference between these sets (`files_after - files_before`) to identify only the paths created during the hook's runtime.

### Why does abx-dl exclude .stdout.log, .stderr.log, .pid, and .sh files from output?

These files are generated by abx-dl's own executor to manage the hook process—capturing logs, tracking PIDs, and wrapping commands in shell scripts. They are operational artifacts rather than intentional hook output, so the system filters them using an `excluded_suffixes` tuple to ensure only relevant content appears in the snapshot.

### What happens to the temporary log and PID files after a hook succeeds?

When a hook exits with a return code of zero, abx-dl immediately deletes the temporary bookkeeping files using `unlink(missing_ok=True)`. This cleanup removes the `.stdout.log`, `.stderr.log`, and `.pid` files from the filesystem, leaving only the genuine output files produced by the hook.

### Where does abx-dl store the final list of detected files?

The filtered list of new files is stored in the `output_files` field of an `ArchiveResult` object instantiated in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py). This structured result object tracks the hook's status, metadata, and the cleaned file list, which downstream components use for indexing and serving the archived content.