# Understanding the index.jsonl Output Format and ArchiveResult Records in abx-dl

> Explore the index.jsonl output format in abx-dl. Understand the structure of Snapshot, Process, and ArchiveResult records for efficient data management.

- Repository: [ArchiveBox/abx-dl](https://github.com/archivebox/abx-dl)
- Tags: api-reference
- Published: 2026-02-25

---

**The `index.jsonl` file in abx-dl uses a JSON Lines format to store three distinct record types—Snapshot, Process, and ArchiveResult—with each line representing a self-contained JSON object that includes a mandatory `"type"` field for identification.**

The `archivebox/abx-dl` repository generates an `index.jsonl` file in the output directory to capture a complete, chronological trace of every archiving operation. This append-only JSON Lines format enables streaming processing and ensures that each record—whether describing the source URL, a subprocess execution, or a plugin result—can be parsed independently without loading the entire file into memory.

## Structure of the index.jsonl File

The `index.jsonl` format relies on three dataclasses defined in [`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py). Each record type serves a specific purpose in the archiving pipeline and shares a common serialization pattern via the `to_jsonl()` method.

### Record Types Overview

Every line in `index.jsonl` contains one of the following record types, distinguished by the `"type"` field:

- **Snapshot**: Represents the URL being archived. Key fields include `url`, `id`, `title`, `timestamp`, `bookmarked_at`, `created_at`, and `tags`.
- **Process**: Describes a subprocess that executed a plugin hook. Captures `cmd`, `binary_id`, `pwd`, `env`, `timeout`, `started_at`, `ended_at`, `exit_code`, `stdout`, `stderr`, `machine_hostname`, and `machine_os`.
- **ArchiveResult**: The high-level outcome of a single plugin hook execution. Contains `snapshot_id`, `plugin`, `id`, `hook_name`, `status`, `process_id`, `output_str`, `output_files`, `start_ts`, `end_ts`, and `error`.

### JSON Lines Serialization

In [`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py), each dataclass implements a `to_jsonl()` method that standardizes the output format. The serialization process follows four steps:

1. Converts the dataclass to a dictionary using `asdict(self)`.
2. Strips keys with `None` values to keep records concise.
3. Injects the `"type"` field to identify the record class.
4. Serializes to JSON using `json.dumps(..., default=str)` to handle non-standard types.

```python

# From abx_dl/models.py

def to_jsonl(self) -> str:
    d = {k: v for k, v in asdict(self).items() if v is not None}
    d['type'] = 'ArchiveResult'  # or 'Process' / 'Snapshot'

    return json.dumps(d, default=str)

```

## ArchiveResult Record Format

The **ArchiveResult** record serves as the primary output for plugin hook executions, linking the snapshot being processed to the specific files and status generated by the plugin.

### Core Fields and Metadata

Each ArchiveResult record contains identifiers that connect it to other records in the pipeline:

- **snapshot_id**: Links to the parent Snapshot record's `id`.
- **plugin**: The name of the plugin that generated this result (e.g., `"chrome"`, `"wget"`).
- **hook_name**: The specific hook function executed (e.g., [`on_Snapshot__10_capture_html.py`](https://github.com/archivebox/abx-dl/blob/main/on_Snapshot__10_capture_html.py)).
- **process_id**: References the Process record that executed the underlying subprocess, if applicable.

### Status Tracking and Output Files

The ArchiveResult captures execution outcomes through several status and output fields:

- **status**: String indicating success or failure (e.g., `"succeeded"`, `"failed"`, `"started"`).
- **output_str**: Textual summary of the result, often containing error messages or success confirmations.
- **output_files**: List of file paths generated by the plugin relative to the output directory.
- **start_ts** and **end_ts**: ISO-format timestamps marking the hook execution window.

## Writing and Reading index.jsonl

The [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py) module orchestrates the archiving pipeline and persists records using the `write_jsonl()` helper function.

### Appending Records During Execution

As the executor runs plugin hooks, it appends records chronologically to `index.jsonl`:

```python

# Called throughout abx_dl/executor.py

write_jsonl(index_path, proc, also_print=not is_tty)   # Process line

write_jsonl(index_path, ar, also_print=not is_tty)    # ArchiveResult line

write_jsonl(index_path, snapshot, also_print=not is_tty)  # Snapshot line

```

This append-only approach ensures that even if the process crashes, all completed work is preserved in the index.

### Parsing index.jsonl

Downstream tools can stream-process the file by filtering on the `"type"` field:

```python
import json
from pathlib import Path

def load_archive_results(index_path: Path):
    """Yield only ArchiveResult records from index.jsonl."""
    with open(index_path) as f:
        for line in f:
            record = json.loads(line)
            if record.get("type") == "ArchiveResult":
                yield record

# Usage

for result in load_archive_results(Path("output/index.jsonl")):
    print(f"Plugin {result['plugin']} finished with status {result['status']}")

```

## Execution Flow and Record Ordering

The chronological order of records in `index.jsonl` reflects the actual execution sequence of the archiving pipeline.

### Standard Record Sequence

A typical archiving run produces records in this order:

1. **Snapshot**: The first line always contains the Snapshot record defining the target URL and metadata.
2. **Process**: For each plugin hook that executes a subprocess, a Process record logs the command, environment, and system details.
3. **ArchiveResult**: Immediately following its associated Process (if any), the ArchiveResult record indicates the hook's completion status, output files, and timing.

Background hooks that run asynchronously emit an initial ArchiveResult with status `"started"` when they begin, then append a final ArchiveResult when they complete, allowing consumers to track long-running operations.

## Summary

- **index.jsonl** is a JSON Lines file located in the output directory that stores the complete execution trace of an abx-dl archiving run.
- Three record types populate the file: **Snapshot** (URL metadata), **Process** (subprocess details), and **ArchiveResult** (plugin hook outcomes).
- Each record serializes via `to_jsonl()` in [`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py), which strips null values, injects a `"type"` field, and uses `json.dumps(default=str)`.
- Records append chronologically during execution in [`abx_dl/executor.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/executor.py), with Snapshot first, followed by Process/ArchiveResult pairs for each plugin hook.
- The format supports streaming consumption—tools can filter by `"type"` to extract only ArchiveResult records without loading the entire file.

## Frequently Asked Questions

### What is the difference between a Process record and an ArchiveResult record in index.jsonl?

A **Process** record captures low-level subprocess execution details including the command array, working directory, environment variables, exit code, and captured stdout/stderr. An **ArchiveResult** record represents the high-level outcome of a plugin hook, linking to the parent Snapshot, referencing the Process ID that executed it (if applicable), and containing the final status, output files, and error messages. While every subprocess generates a Process record, every plugin hook generates an ArchiveResult regardless of whether it spawned a subprocess.

### How does abx-dl handle null or empty values when writing to index.jsonl?

The `to_jsonl()` method defined in [`abx_dl/models.py`](https://github.com/archivebox/abx-dl/blob/main/abx_dl/models.py) explicitly removes keys with `None` values before serialization. It constructs the dictionary using a dictionary comprehension that filters out nulls: `{k: v for k, v in asdict(self).items() if v is not None}`. This ensures that the resulting JSON Lines file remains compact and only contains meaningful data, making it easier to parse and reducing file size.

### Can I parse index.jsonl incrementally without loading the entire file into memory?

Yes, the JSON Lines format is specifically designed for streaming and incremental processing. Because each line is a valid, self-contained JSON object, you can open the file and iterate line by line, parsing only the current record into memory. This approach allows you to filter for specific `"type"` values—such as extracting only `ArchiveResult` records from a large archiving run—using minimal memory regardless of file size.

### What does the status field indicate in an ArchiveResult record?

The `status` field in an ArchiveResult record indicates the completion state of the plugin hook execution. Common values include `"succeeded"` when the hook completes successfully and generates output files, `"failed"` when the hook encounters an error (with details typically stored in the `error` field), and `"started"` for background hooks that emit an initial record when they begin execution before later appending a final record upon completion. This field allows downstream tools to quickly identify successful archives versus failures without parsing additional metadata.