Understanding the index.jsonl Output Format and ArchiveResult Records in abx-dl

The index.jsonl file in abx-dl uses a JSON Lines format to store three distinct record types—Snapshot, Process, and ArchiveResult—with each line representing a self-contained JSON object that includes a mandatory "type" field for identification.

The archivebox/abx-dl repository generates an index.jsonl file in the output directory to capture a complete, chronological trace of every archiving operation. This append-only JSON Lines format enables streaming processing and ensures that each record—whether describing the source URL, a subprocess execution, or a plugin result—can be parsed independently without loading the entire file into memory.

Structure of the index.jsonl File

The index.jsonl format relies on three dataclasses defined in abx_dl/models.py. Each record type serves a specific purpose in the archiving pipeline and shares a common serialization pattern via the to_jsonl() method.

Record Types Overview

Every line in index.jsonl contains one of the following record types, distinguished by the "type" field:

• Snapshot: Represents the URL being archived. Key fields include url, id, title, timestamp, bookmarked_at, created_at, and tags.
• Process: Describes a subprocess that executed a plugin hook. Captures cmd, binary_id, pwd, env, timeout, started_at, ended_at, exit_code, stdout, stderr, machine_hostname, and machine_os.
• ArchiveResult: The high-level outcome of a single plugin hook execution. Contains snapshot_id, plugin, id, hook_name, status, process_id, output_str, output_files, start_ts, end_ts, and error.

JSON Lines Serialization

In abx_dl/models.py, each dataclass implements a to_jsonl() method that standardizes the output format. The serialization process follows four steps:

1. Converts the dataclass to a dictionary using asdict(self).
2. Strips keys with None values to keep records concise.
3. Injects the "type" field to identify the record class.
4. Serializes to JSON using json.dumps(..., default=str) to handle non-standard types.

# From abx_dl/models.py

def to_jsonl(self) -> str:
    d = {k: v for k, v in asdict(self).items() if v is not None}
    d['type'] = 'ArchiveResult'  # or 'Process' / 'Snapshot'

    return json.dumps(d, default=str)

ArchiveResult Record Format

The ArchiveResult record serves as the primary output for plugin hook executions, linking the snapshot being processed to the specific files and status generated by the plugin.

Core Fields and Metadata

Each ArchiveResult record contains identifiers that connect it to other records in the pipeline:

• snapshot_id: Links to the parent Snapshot record's id.
• plugin: The name of the plugin that generated this result (e.g., "chrome", "wget").
• hook_name: The specific hook function executed (e.g., on_Snapshot__10_capture_html.py).
• process_id: References the Process record that executed the underlying subprocess, if applicable.

Status Tracking and Output Files

The ArchiveResult captures execution outcomes through several status and output fields:

• status: String indicating success or failure (e.g., "succeeded", "failed", "started").
• output_str: Textual summary of the result, often containing error messages or success confirmations.
• output_files: List of file paths generated by the plugin relative to the output directory.
• start_ts and end_ts: ISO-format timestamps marking the hook execution window.

Writing and Reading index.jsonl

The abx_dl/executor.py module orchestrates the archiving pipeline and persists records using the write_jsonl() helper function.

Appending Records During Execution

As the executor runs plugin hooks, it appends records chronologically to index.jsonl:

# Called throughout abx_dl/executor.py

write_jsonl(index_path, proc, also_print=not is_tty)   # Process line

write_jsonl(index_path, ar, also_print=not is_tty)    # ArchiveResult line

write_jsonl(index_path, snapshot, also_print=not is_tty)  # Snapshot line

This append-only approach ensures that even if the process crashes, all completed work is preserved in the index.

Parsing index.jsonl

Downstream tools can stream-process the file by filtering on the "type" field:

import json
from pathlib import Path

def load_archive_results(index_path: Path):
    """Yield only ArchiveResult records from index.jsonl."""
    with open(index_path) as f:
        for line in f:
            record = json.loads(line)
            if record.get("type") == "ArchiveResult":
                yield record

# Usage

for result in load_archive_results(Path("output/index.jsonl")):
    print(f"Plugin {result['plugin']} finished with status {result['status']}")

Execution Flow and Record Ordering

The chronological order of records in index.jsonl reflects the actual execution sequence of the archiving pipeline.

Standard Record Sequence

A typical archiving run produces records in this order:

1. Snapshot: The first line always contains the Snapshot record defining the target URL and metadata.
2. Process: For each plugin hook that executes a subprocess, a Process record logs the command, environment, and system details.
3. ArchiveResult: Immediately following its associated Process (if any), the ArchiveResult record indicates the hook's completion status, output files, and timing.

Background hooks that run asynchronously emit an initial ArchiveResult with status "started" when they begin, then append a final ArchiveResult when they complete, allowing consumers to track long-running operations.

Summary

• index.jsonl is a JSON Lines file located in the output directory that stores the complete execution trace of an abx-dl archiving run.
• Three record types populate the file: Snapshot (URL metadata), Process (subprocess details), and ArchiveResult (plugin hook outcomes).
• Each record serializes via to_jsonl() in abx_dl/models.py, which strips null values, injects a "type" field, and uses json.dumps(default=str).
• Records append chronologically during execution in abx_dl/executor.py, with Snapshot first, followed by Process/ArchiveResult pairs for each plugin hook.
• The format supports streaming consumption—tools can filter by "type" to extract only ArchiveResult records without loading the entire file.

Frequently Asked Questions

What is the difference between a Process record and an ArchiveResult record in index.jsonl?

A Process record captures low-level subprocess execution details including the command array, working directory, environment variables, exit code, and captured stdout/stderr. An ArchiveResult record represents the high-level outcome of a plugin hook, linking to the parent Snapshot, referencing the Process ID that executed it (if applicable), and containing the final status, output files, and error messages. While every subprocess generates a Process record, every plugin hook generates an ArchiveResult regardless of whether it spawned a subprocess.

How does abx-dl handle null or empty values when writing to index.jsonl?

The to_jsonl() method defined in abx_dl/models.py explicitly removes keys with None values before serialization. It constructs the dictionary using a dictionary comprehension that filters out nulls: {k: v for k, v in asdict(self).items() if v is not None}. This ensures that the resulting JSON Lines file remains compact and only contains meaningful data, making it easier to parse and reducing file size.

Can I parse index.jsonl incrementally without loading the entire file into memory?

Yes, the JSON Lines format is specifically designed for streaming and incremental processing. Because each line is a valid, self-contained JSON object, you can open the file and iterate line by line, parsing only the current record into memory. This approach allows you to filter for specific "type" values—such as extracting only ArchiveResult records from a large archiving run—using minimal memory regardless of file size.

What does the status field indicate in an ArchiveResult record?

The status field in an ArchiveResult record indicates the completion state of the plugin hook execution. Common values include "succeeded" when the hook completes successfully and generates output files, "failed" when the hook encounters an error (with details typically stored in the error field), and "started" for background hooks that emit an initial record when they begin execution before later appending a final record upon completion. This field allows downstream tools to quickly identify successful archives versus failures without parsing additional metadata.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}