What Information Is Contained in the Book-to-Skill metadata.json File?

The metadata.json file contains aggregated extraction statistics, per-source processing details, and global run metadata, serving as the atomic completion signal for Book-to-Skill pipeline runs.

When the Book-to-Skill extractor finishes processing documents, it writes a comprehensive metadata.json file that serves as the single source of truth for the extraction run. This JSON artefact captures everything from total word counts to per-source diagnostics, enabling downstream agents to validate inputs and locate intermediate files. Understanding the structure of this file is essential for anyone building automation pipelines with the virgiliojr94/book-to-skill repository.

Aggregated Statistics Section

The root level of metadata.json provides high-level counters that summarize the entire extraction batch. These fields aggregate data across all processed sources:

• pages: Total page count across all documents
• words: Cumulative word count from all sources
• tokens: Aggregate token count for downstream generation sizing

These totals allow quick validation of extraction scope without parsing individual source entries.

Per-Source Array Structure

The sources array contains one entry for every input document (PDF, EPUB, HTML, etc.). Each object in this array includes specific metadata about that source:

• source_file: Absolute path to the original input file
• filename: Base name displayed to users
• type: Detected format (e.g., pdf, epub, html)
• pages, words, tokens: Individual statistics for that specific source
• dropped_images: Number of EPUB images ignored during extraction (see tests/test_epub_image_reporting.py)
• workdir: Temporary directory holding intermediate artefacts for that specific source

This granularity enables precise troubleshooting when specific documents fail validation or produce unexpected token counts.

Global Metadata and File Location

According to book_to_skill/config.py, the constant OUTPUT_META defines the filename for this artefact, which is written to the run's work directory. The file is always encoded in UTF-8, as enforced by tests/test_metadata_encoding.py, ensuring cross-platform compatibility regardless of host locale settings.

In addition to the aggregated and per-source data, the root object includes convenience fields duplicated from the first source:

• workdir: Absolute path to the run's temporary directory (used by agents to locate full_text.txt)
• filename and source_file: Top-level references for quick access without iterating the sources array

Atomic Completion Guarantee

The metadata.json file is deliberately written only after all extraction steps have succeeded. As documented in docs/how-it-works.md, any pipeline failure aborts before the file is created. This design guarantees that the mere presence of metadata.json signals a complete, successful extraction—making it a reliable checkpoint for downstream automation.

How to Parse metadata.json in Python

Consume the metadata file to validate extraction results before generation steps. The following pattern mirrors the example from SKILL.md:

import json
from pathlib import Path

# Path to the metadata file printed by the extraction run

meta_path = Path("/tmp/book_skill_work/metadata.json")

# Load the JSON document (UTF-8 is guaranteed)

with meta_path.open(encoding="utf-8") as fp:
    meta = json.load(fp)

# Quick overview

print("Total pages:", meta["pages"])
print("Total words:", meta["words"])
print("Total tokens:", meta["tokens"])

# Inspect each source

for src in meta["sources"]:
    print(f"→ {src['filename']} ({src['type']}) – {src['pages']} pages")

This snippet demonstrates accessing both aggregated totals and per-source diagnostics, confirming the correct document was processed before initiating downstream generation tasks.

Summary

• Aggregated statistics (pages, words, tokens) provide extraction batch totals at the root level.
• The sources array contains detailed per-document metadata including file paths, detected types, and processing statistics.
• dropped_images specifically tracks ignored EPUB image assets for format-specific debugging.
• The file is written to the work directory defined by OUTPUT_META in book_to_skill/config.py using guaranteed UTF-8 encoding.
• Presence of metadata.json acts as an atomic completion signal indicating successful extraction.

Frequently Asked Questions

What encoding does the Book-to-Skill metadata.json file use?

The file is always encoded in UTF-8, as validated by tests/test_metadata_encoding.py. This ensures the JSON can be parsed identically across different operating systems and locale configurations without encoding errors.

Where is the metadata.json file located after extraction?

According to book_to_skill/config.py, the file is written to the extraction run's work directory using the filename defined by the OUTPUT_META constant. The absolute path is also stored in the workdir field within the JSON itself.

What per-source information is available for EPUB files specifically?

In addition to standard fields like pages and words, EPUB entries include dropped_images, which reports the count of images ignored during extraction. This field is tested in tests/test_epub_image_reporting.py and helps diagnose content loss in ebook processing pipelines.

How can I verify that a Book-to-Skill extraction completed successfully?

Check for the existence of metadata.json in the work directory. Because the file is only written after all extraction steps succeed (as documented in docs/how-it-works.md), its presence guarantees that the pipeline finished without errors and that the aggregated statistics and per-source arrays contain valid, complete data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}