How abx-dl Detects New Files Created by Hooks and Excludes Log/PID Files from Output

abx-dl detects new files created by hooks by taking directory snapshots before and after execution, then filters out bookkeeping artifacts like .stdout.log, .stderr.log, .pid, and .sh files using suffix-based exclusion rules.

abx-dl is a Python-based archival tool that executes plugin hooks to capture web content. When hooks run, they generate output files, but the system also creates temporary logs, PID files, and shell scripts for process management. The challenge lies in distinguishing genuine hook output from these operational artifacts to ensure only relevant files appear in the final output_files list.

The Challenge: Distinguishing Hook Output from Operational Artifacts

When abx-dl executes a hook via the run_hook() function in abx_dl/executor.py, it spawns a subprocess and redirects stdout and stderr to log files, writes a PID file for process tracking, and generates a temporary shell script to wrap the command. These bookkeeping files coexist in the same output directory as the actual files the hook intends to create (such as PDFs, screenshots, or HTML archives). Without explicit filtering, these temporary artifacts would pollute the snapshot's output_files metadata.

How abx-dl Detects New Files Created by Hooks

The detection mechanism relies on a before-and-after directory comparison strategy implemented in the run_hook() function.

Capturing the Pre-Execution State

Before spawning the hook process, abx-dl captures a complete snapshot of the output directory using pathlib.Path.rglob(). This creates a set of all existing file paths:

# Capture files before execution to detect new output

files_before = set(output_dir.rglob('*')) if output_dir.exists() else set()

This set serves as the baseline for comparison, ensuring that any files existing prior to the hook's execution are not mistakenly flagged as new output.

Executing the Hook and Generating Bookkeeping Files

During execution, the system writes four types of temporary files to the output directory:

• *.stdout.log: Captures standard output
• *.stderr.log: Captures standard error
• *.pid: Stores the process ID for background process management
• *.sh: The generated shell script wrapping the hook command

These files are necessary for process management and debugging but must be excluded from the final output list.

Identifying New Files via Set Difference

After the hook process terminates, abx-dl captures a second snapshot and computes the difference between the two sets:

files_after = set(output_dir.rglob('*')) if output_dir.exists() else set()
new_files = [str(f.relative_to(output_dir))
             for f in (files_after - files_before) if f.is_file()]

This set subtraction (files_after - files_before) yields only the paths created during the hook's execution, including both genuine output and the bookkeeping files.

Filtering Out Log, PID, and Script Files

Once the new files are identified, abx-dl applies a suffix-based filter to remove the operational artifacts.

The Excluded Suffixes Tuple

The system defines a tuple of suffixes that correspond to the bookkeeping files generated during execution:

excluded_suffixes = ('.stdout.log', '.stderr.log', '.pid', '.sh')

This explicit list ensures that even if a hook legitimately creates files with similar names, only these specific patterns generated by the executor are removed.

Suffix-Based Filtering Implementation

The filtering logic uses a list comprehension with str.endswith() to strip out the unwanted paths:

new_files = [f for f in new_files
             if not any(f.endswith(suffix) for suffix in excluded_suffixes)]

After this operation, new_files contains only the genuine output produced by the hook, free from stdout logs, stderr logs, PID files, and temporary shell scripts.

Cleaning Up Temporary Bookkeeping Files

After successfully filtering the output list, abx-dl performs cleanup to remove the temporary files from the filesystem. This occurs when the hook exits with a return code of zero:

if returncode == 0:
    stdout_file.unlink(missing_ok=True)
    stderr_file.unlink(missing_ok=True)
    pid_file.unlink(missing_ok=True)

This ensures that successful hook executions leave behind only the intended output files, maintaining a clean snapshot directory. If the hook fails, these files are retained for debugging purposes.

Storing the Filtered Results in ArchiveResult

The final filtered list is persisted in the ArchiveResult object, which tracks the outcome of each hook execution. In abx_dl/executor.py, the ArchiveResult is instantiated with the cleaned new_files list:

ar = ArchiveResult(
    snapshot_id=snapshot_id,
    plugin=hook.plugin_name,
    hook_name=hook.name,
    status='succeeded' if returncode == 0 else 'failed',
    output_files=new_files,
    # ... additional fields ...

)

This structured result is then used by the rest of the abx-dl pipeline to index and serve the captured content, ensuring that only relevant files are exposed to the user interface and search indices.

Summary

• abx-dl detects hook-generated files by comparing directory snapshots taken before and after hook execution using rglob() and set difference operations.
• The system explicitly filters out operational artifacts—.stdout.log, .stderr.log, .pid, and .sh files—using a suffix-based exclusion tuple.
• Temporary bookkeeping files are unlinked after successful execution to maintain clean output directories.
• Only filtered, relevant files are stored in the ArchiveResult.output_files field for downstream processing.

Frequently Asked Questions

How does abx-dl distinguish between existing files and new files created by hooks?

abx-dl captures a set of all files in the output directory before the hook runs using output_dir.rglob('*'), then captures another set after execution. It computes the difference between these sets (files_after - files_before) to identify only the paths created during the hook's runtime.

Why does abx-dl exclude .stdout.log, .stderr.log, .pid, and .sh files from output?

These files are generated by abx-dl's own executor to manage the hook process—capturing logs, tracking PIDs, and wrapping commands in shell scripts. They are operational artifacts rather than intentional hook output, so the system filters them using an excluded_suffixes tuple to ensure only relevant content appears in the snapshot.

What happens to the temporary log and PID files after a hook succeeds?

When a hook exits with a return code of zero, abx-dl immediately deletes the temporary bookkeeping files using unlink(missing_ok=True). This cleanup removes the .stdout.log, .stderr.log, and .pid files from the filesystem, leaving only the genuine output files produced by the hook.

Where does abx-dl store the final list of detected files?

The filtered list of new files is stored in the output_files field of an ArchiveResult object instantiated in abx_dl/executor.py. This structured result object tracks the hook's status, metadata, and the cleaned file list, which downstream components use for indexing and serving the archived content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}