# How to Debug Failed Hooks in abx-dl Using stdout.log and stderr.log Files

> Debug failed abx-dl hooks by reviewing stdout.log and stderr.log files in plugin directories. Diagnose crashes, permissions, or logic issues effectively.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: how-to-guide
- Published: 2026-02-25

---

**To debug failed hooks in abx-dl, examine the `*.stdout.log` and `*.stderr.log` files retained in each plugin's output subdirectory; these files capture the complete subprocess output and persist only when hooks fail, allowing you to diagnose external binary crashes, permission errors, or hook logic issues.**

abx-dl is an extensible download automation framework that executes plugin hooks as subprocesses to archive content. When a hook fails, the framework writes detailed execution logs to help you debug failed hooks using stdout.log and stderr.log files located in each plugin directory under the main output folder.

## Where abx-dl Writes Hook Debug Logs

When a hook runs, abx-dl creates a dedicated subdirectory for the plugin (e.g., `output_dir/<plugin_name>/`) and writes all hook-specific output to log files named after the hook itself:

- `\<hook_name>.stdout.log` – captured **STDOUT** of the hook
- `\<hook_name>.stderr.log` – captured **STDERR** of the hook
- `\<hook_name>.pid` – PID file for background (daemon) hooks
- `\<hook_name>.sh` – a tiny script containing the exact command used to launch the hook (helpful for reproducing the run)

These files are created in `run_hook` inside [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) at lines 75-80. The function opens the log files before spawning the subprocess at lines 90-97, ensuring every line the hook writes to stdout or stderr ends up in the corresponding log file.

## Log File Lifecycle and Retention Policies

Understanding when abx-dl deletes or retains these files is crucial for debugging:

**Successful foreground hooks** – After a hook finishes with exit code 0, the code cleans up the logs automatically (along with the PID file) at lines 62-66 in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py).

**Failed foreground hooks** – The logs are **kept** so you can inspect why the hook failed.

**Background hooks** – The logs are retained until the final cleanup phase runs. The cleanup logic (`cleanup_background_hooks` → `_finalize_background_hook`) reads the logs, writes the final `ArchiveResult` entry, and then removes the logs **only if the hook succeeded** at lines 58-63 and 95-101.

Therefore, any hook that reports a status other than `succeeded` will leave its `*.stdout.log` and `*.stderr.log` files on disk for investigation.

## Locating and Reading Failed Hook Logs

Follow this workflow to find and analyze failing hook output:

1. **Identify the plugin directory** – The plugin name is the immediate sub-directory under the output directory (e.g., `chrome/`, `wget/`).
2. **Find the hook files** – Look for files matching the hook name (e.g., `on_Snapshot__20_chrome_tab.bg.stdout.log`).
3. **Examine the content** – Open the files with any text viewer (`cat`, `less`, or an IDE).

**Example directory layout after a failed run:**

```text
output/
└─ chrome/
   ├─ on_Snapshot__20_chrome_tab.bg.stdout.log
   ├─ on_Snapshot__20_chrome_tab.bg.stderr.log
   ├─ on_Snapshot__20_chrome_tab.bg.pid
   └─ on_Snapshot__20_chrome_tab.bg.sh

```

**What to look for inside the logs:**

| Log File | Typical Content | Diagnostic Value |
|----------|----------------|------------------|
| `*.stderr.log` | Python tracebacks, `subprocess.CalledProcessError` messages, Chrome crash logs | External failures, permission problems, missing binaries |
| `*.stdout.log` | JSONL lines such as `{"type":"ArchiveResult",...}` or `{"type":"Binary",...}` | Hook-level status, custom error messages in the `error` field, and any `output_str` emitted |
| `*.sh` | Exact command array used to launch the hook | Allows manual reproduction outside of abx-dl |

If a background hook never writes a final `ArchiveResult` line, the cleanup routine surfaces the last 500 bytes of `stderr` as the `error` field in the final result at lines 78-80 in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py).

## Diagnostic Workflow and Manual Reproduction

Use this command sequence to diagnose and reproduce hook failures:

```bash

# 1️⃣ Find the failing plugin directory

cd /path/to/output/<plugin_name>

# 2️⃣ List the hook-specific log files

ls *.stdout.log *.stderr.log

# 3️⃣ Inspect STDERR first – most failures surface here

cat <hook_name>.stderr.log

# 4️⃣ Look at STDOUT for JSONL status messages

cat <hook_name>.stdout.log | grep ArchiveResult

# 5️⃣ (Optional) Re-run the exact command manually

sh <hook_name>.sh

```

The `<hook_name>.sh` file contains the precise command array used to launch the hook, allowing you to reproduce the environment and test fixes outside of the abx-dl execution context.

## Summary

- **abx-dl** creates `*.stdout.log` and `*.stderr.log` files in each plugin's output subdirectory to capture hook output.
- Log files are **automatically deleted** only when foreground hooks succeed; failed hooks and background hooks retain logs for debugging.
- The `*.stderr.log` file typically contains Python tracebacks and external binary errors, while `*.stdout.log` contains JSONL status messages and `ArchiveResult` entries.
- Each hook directory includes a `*.sh` script containing the exact launch command, enabling manual reproduction of failures.

## Frequently Asked Questions

### Where are the stdout.log and stderr.log files located in abx-dl?

The log files are located in plugin-specific subdirectories under your main output directory. For a plugin named `chrome`, the logs appear at `output/chrome/<hook_name>.stdout.log` and `output/chrome/<hook_name>.stderr.log`. These paths are determined by the `output_dir` parameter passed from [`abx_dl/cli.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/cli.py) to the executor.

### Why are log files missing for some hooks but present for others?

abx-dl automatically cleans up log files for **successful foreground hooks** to save disk space, removing them immediately after exit code 0. Logs are retained only when hooks fail (non-zero exit code) or when running as **background hooks**, which keep logs until the final cleanup phase confirms success. This retention policy ensures you always have diagnostic data available for investigations.

### How can I manually rerun a failed hook to test fixes?

Each hook execution generates a `<hook_name>.sh` script in the plugin directory containing the exact command array used for launch. Navigate to the plugin directory and execute `sh <hook_name>.sh` to reproduce the failure outside of abx-dl's execution context. This allows you to modify environment variables or arguments directly to test fixes before the next automated run.

### What is the difference between foreground and background hook logging?

**Foreground hooks** run synchronously and delete their logs immediately upon success, retaining them only on failure. **Background hooks** run asynchronously and maintain their logs throughout their lifetime; the cleanup routine reads these logs to construct the final `ArchiveResult`, then deletes them only if the hook succeeded. If a background hook fails, both the stdout and stderr logs persist for debugging, with the last 500 bytes of stderr surfaced in the error report.