What Information Is Contained in the book-to-skill metadata.json File?

The metadata.json file generated by the book-to-skill extractor stores high-level document statistics (pages, words, tokens, chapters) and a per-source breakdown for tracing extraction provenance.

The metadata.json file serves as the consolidated metadata output for the virgiliojr94/book-to-skill repository. After processing books or documents through the extraction pipeline, this JSON file captures document structure, token counts for cost estimation, and source tracking. Located within the temporary working directory specified during extraction, it acts as the single source of truth for downstream tooling that needs to verify extraction completeness or calculate generation costs.

Core Structure and Data Fields

The metadata follows a hierarchical structure that separates global document statistics from individual source contributions.

Document-Level Metadata

At the root level, metadata.json contains aggregate statistics describing the entire extraction run:

• workdir – The temporary working directory path (e.g., /tmp/book_skill_work/) that held intermediate files during processing.
• filename – The name of the original book or document being processed.
• source_file – The absolute path to the source file that was fed into the extractor.
• pages – Total page count across all merged sources.
• words – Total word count for the entire document.
• tokens – Language-model token count, essential for estimating API costs in downstream generation tasks.
• chapters – Number of detected chapters derived from the Table of Contents.
• toc – The merged Table of Contents as an ordered list of chapter titles and starting page numbers.
• dropped_epub_images – Count of images stripped during EPUB processing (these images are omitted from final text output).

Per-Source Metadata Arrays

The sources field contains an array of objects, where each entry mirrors the document-level statistics but applies to individual input files:

{
  "sources": [
    {
      "source_file": "/path/to/chapter1.pdf",
      "pages": 45,
      "words": 12000,
      "tokens": 15000,
      "chapters": 3,
      "toc": [...]
    }
  ]
}

This structure enables precise tracing of which content originated from which source file, critical when merging multiple books or chapters into a single skill output.

Encoding and File Location

According to the test suite in tests/test_metadata_encoding.py, the file is written with UTF-8 encoding to ensure proper handling of international characters in titles and filenames. By default, the file is generated at {workdir}/metadata.json, with the path defined by the OUTPUT_META constant in book_to_skill/config.py.

Working with metadata.json in Python

Downstream tools can load and analyze the metadata to verify extraction results or estimate costs.

Loading and Inspecting Metadata

import json
from pathlib import Path

# Load the metadata for a completed extraction run

meta_path = Path("/tmp/book_skill_work/metadata.json")
with meta_path.open(encoding="utf-8") as f:
    meta = json.load(f)

# Display high-level document statistics

print(f"Document: {meta['filename']}")
print(f"Pages: {meta['pages']:,} | Words: {meta['words']:,} | Tokens: {meta['tokens']:,}")
print(f"Chapters detected: {meta['chapters']}")

# Display per-source breakdown

for src in meta["sources"]:
    print(f"- {src['source_file']}: {src['pages']} pages, {src['words']} words")

Estimating Generation Costs

Use the token count to calculate approximate API costs before sending content to language models:

# Example: estimating generation cost at $0.00002 per token

TOKEN_COST = 0.00002

meta_path = Path("/tmp/book_skill_work/metadata.json")
with meta_path.open(encoding="utf-8") as f:
    meta = json.load(f)

estimated_cost = meta["tokens"] * TOKEN_COST
print(f"Estimated generation cost: ${estimated_cost:,.2f}")

Source Code References

The metadata generation logic and field definitions are distributed across these key files:

• book_to_skill/config.py – Defines OUTPUT_META, the configuration constant pointing to the metadata file path.
• docs/architecture.md – Documents the data flow showing where metadata.json sits in the extraction pipeline.
• docs/how-it-works.md – Explains the extraction stages that populate the metadata fields.
• SKILL.md – Provides the authoritative field reference for consumers of metadata.json.
• tests/test_metadata_encoding.py – Validates UTF-8 encoding requirements for the output file.

Summary

• The metadata.json file stores aggregate statistics (pages, words, tokens, chapters) and Table of Contents data for extracted documents.
• Per-source arrays allow tracing which statistics belong to which input file when processing multiple sources.
• Token counts enable accurate cost estimation for downstream language model generation.
• UTF-8 encoding ensures proper handling of international text, as verified by the test suite.
• Configuration is centralized in book_to_skill/config.py via the OUTPUT_META constant.

Frequently Asked Questions

What is the exact file path for metadata.json?

The file is located in the temporary working directory specified during the extraction run, typically at /tmp/book_skill_work/metadata.json. The precise path is controlled by the OUTPUT_META setting in book_to_skill/config.py and follows the pattern {workdir}/metadata.json.

Why does the metadata.json file include token counts?

Token counts are included to enable cost estimation for downstream language model operations. By knowing the exact token count upfront, applications can calculate API costs (e.g., multiplying tokens by $0.00002) before submitting content for generation, preventing unexpected charges.

What does the dropped_epub_images field indicate?

This field tracks how many images were stripped out during EPUB processing. Since the book-to-skill extractor focuses on text content, images embedded in EPUB files are omitted from the final output, and this counter helps diagnose why visual content is missing from the extracted text.

How can I verify the metadata.json encoding is correct?

The repository includes a dedicated test in tests/test_metadata_encoding.py that validates the file is written with UTF-8 encoding. You can run this test to ensure the metadata properly handles international characters and non-ASCII symbols in filenames and chapter titles.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}