How to Debug Failed Hooks in abx-dl Using stdout.log and stderr.log Files

To debug failed hooks in abx-dl, examine the *.stdout.log and *.stderr.log files retained in each plugin's output subdirectory; these files capture the complete subprocess output and persist only when hooks fail, allowing you to diagnose external binary crashes, permission errors, or hook logic issues.

abx-dl is an extensible download automation framework that executes plugin hooks as subprocesses to archive content. When a hook fails, the framework writes detailed execution logs to help you debug failed hooks using stdout.log and stderr.log files located in each plugin directory under the main output folder.

Where abx-dl Writes Hook Debug Logs

When a hook runs, abx-dl creates a dedicated subdirectory for the plugin (e.g., output_dir/<plugin_name>/) and writes all hook-specific output to log files named after the hook itself:

• \<hook_name>.stdout.log – captured STDOUT of the hook
• \<hook_name>.stderr.log – captured STDERR of the hook
• \<hook_name>.pid – PID file for background (daemon) hooks
• \<hook_name>.sh – a tiny script containing the exact command used to launch the hook (helpful for reproducing the run)

These files are created in run_hook inside abx_dl/executor.py at lines 75-80. The function opens the log files before spawning the subprocess at lines 90-97, ensuring every line the hook writes to stdout or stderr ends up in the corresponding log file.

Log File Lifecycle and Retention Policies

Understanding when abx-dl deletes or retains these files is crucial for debugging:

Successful foreground hooks – After a hook finishes with exit code 0, the code cleans up the logs automatically (along with the PID file) at lines 62-66 in abx_dl/executor.py.

Failed foreground hooks – The logs are kept so you can inspect why the hook failed.

Background hooks – The logs are retained until the final cleanup phase runs. The cleanup logic (cleanup_background_hooks → _finalize_background_hook) reads the logs, writes the final ArchiveResult entry, and then removes the logs only if the hook succeeded at lines 58-63 and 95-101.

Therefore, any hook that reports a status other than succeeded will leave its *.stdout.log and *.stderr.log files on disk for investigation.

Locating and Reading Failed Hook Logs

Follow this workflow to find and analyze failing hook output:

1. Identify the plugin directory – The plugin name is the immediate sub-directory under the output directory (e.g., chrome/, wget/).
2. Find the hook files – Look for files matching the hook name (e.g., on_Snapshot__20_chrome_tab.bg.stdout.log).
3. Examine the content – Open the files with any text viewer (cat, less, or an IDE).

Example directory layout after a failed run:

output/
└─ chrome/
   ├─ on_Snapshot__20_chrome_tab.bg.stdout.log
   ├─ on_Snapshot__20_chrome_tab.bg.stderr.log
   ├─ on_Snapshot__20_chrome_tab.bg.pid
   └─ on_Snapshot__20_chrome_tab.bg.sh

What to look for inside the logs:

Log File	Typical Content	Diagnostic Value
*.stderr.log	Python tracebacks, subprocess.CalledProcessError messages, Chrome crash logs	External failures, permission problems, missing binaries
*.stdout.log	JSONL lines such as {"type":"ArchiveResult",...} or {"type":"Binary",...}	Hook-level status, custom error messages in the error field, and any output_str emitted
*.sh	Exact command array used to launch the hook	Allows manual reproduction outside of abx-dl

If a background hook never writes a final ArchiveResult line, the cleanup routine surfaces the last 500 bytes of stderr as the error field in the final result at lines 78-80 in abx_dl/executor.py.

Diagnostic Workflow and Manual Reproduction

Use this command sequence to diagnose and reproduce hook failures:

# 1️⃣ Find the failing plugin directory

cd /path/to/output/<plugin_name>

# 2️⃣ List the hook-specific log files

ls *.stdout.log *.stderr.log

# 3️⃣ Inspect STDERR first – most failures surface here

cat <hook_name>.stderr.log

# 4️⃣ Look at STDOUT for JSONL status messages

cat <hook_name>.stdout.log | grep ArchiveResult

# 5️⃣ (Optional) Re-run the exact command manually

sh <hook_name>.sh

The <hook_name>.sh file contains the precise command array used to launch the hook, allowing you to reproduce the environment and test fixes outside of the abx-dl execution context.

Summary

• abx-dl creates *.stdout.log and *.stderr.log files in each plugin's output subdirectory to capture hook output.
• Log files are automatically deleted only when foreground hooks succeed; failed hooks and background hooks retain logs for debugging.
• The *.stderr.log file typically contains Python tracebacks and external binary errors, while *.stdout.log contains JSONL status messages and ArchiveResult entries.
• Each hook directory includes a *.sh script containing the exact launch command, enabling manual reproduction of failures.

Frequently Asked Questions

Where are the stdout.log and stderr.log files located in abx-dl?

The log files are located in plugin-specific subdirectories under your main output directory. For a plugin named chrome, the logs appear at output/chrome/<hook_name>.stdout.log and output/chrome/<hook_name>.stderr.log. These paths are determined by the output_dir parameter passed from abx_dl/cli.py to the executor.

Why are log files missing for some hooks but present for others?

abx-dl automatically cleans up log files for successful foreground hooks to save disk space, removing them immediately after exit code 0. Logs are retained only when hooks fail (non-zero exit code) or when running as background hooks, which keep logs until the final cleanup phase confirms success. This retention policy ensures you always have diagnostic data available for investigations.

How can I manually rerun a failed hook to test fixes?

Each hook execution generates a <hook_name>.sh script in the plugin directory containing the exact command array used for launch. Navigate to the plugin directory and execute sh <hook_name>.sh to reproduce the failure outside of abx-dl's execution context. This allows you to modify environment variables or arguments directly to test fixes before the next automated run.

What is the difference between foreground and background hook logging?

Foreground hooks run synchronously and delete their logs immediately upon success, retaining them only on failure. Background hooks run asynchronously and maintain their logs throughout their lifetime; the cleanup routine reads these logs to construct the final ArchiveResult, then deletes them only if the hook succeeded. If a background hook fails, both the stdout and stderr logs persist for debugging, with the last 500 bytes of stderr surfaced in the error report.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}