# What Information Is Contained in the book-to-skill metadata.json File?

> Discover what information the metadata.json file contains, including document stats like pages and words, and source breakdowns for extraction provenance. Learn more about book-to-skill metadata.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: api-reference
- Published: 2026-08-24

---

**The [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file generated by the book-to-skill extractor stores high-level document statistics (pages, words, tokens, chapters) and a per-source breakdown for tracing extraction provenance.**

The [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file serves as the **consolidated metadata output** for the virgiliojr94/book-to-skill repository. After processing books or documents through the extraction pipeline, this JSON file captures document structure, token counts for cost estimation, and source tracking. Located within the temporary working directory specified during extraction, it acts as the single source of truth for downstream tooling that needs to verify extraction completeness or calculate generation costs.

## Core Structure and Data Fields

The metadata follows a hierarchical structure that separates global document statistics from individual source contributions.

### Document-Level Metadata

At the root level, [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) contains aggregate statistics describing the entire extraction run:

- **workdir** – The temporary working directory path (e.g., `/tmp/book_skill_work/`) that held intermediate files during processing.
- **filename** – The name of the original book or document being processed.
- **source_file** – The absolute path to the source file that was fed into the extractor.
- **pages** – Total page count across all merged sources.
- **words** – Total word count for the entire document.
- **tokens** – Language-model token count, essential for estimating API costs in downstream generation tasks.
- **chapters** – Number of detected chapters derived from the Table of Contents.
- **toc** – The merged Table of Contents as an ordered list of chapter titles and starting page numbers.
- **dropped_epub_images** – Count of images stripped during EPUB processing (these images are omitted from final text output).

### Per-Source Metadata Arrays

The **sources** field contains an array of objects, where each entry mirrors the document-level statistics but applies to individual input files:

```json
{
  "sources": [
    {
      "source_file": "/path/to/chapter1.pdf",
      "pages": 45,
      "words": 12000,
      "tokens": 15000,
      "chapters": 3,
      "toc": [...]
    }
  ]
}

```

This structure enables precise tracing of which content originated from which source file, critical when merging multiple books or chapters into a single skill output.

### Encoding and File Location

According to the test suite in [`tests/test_metadata_encoding.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_metadata_encoding.py), the file is written with **UTF-8 encoding** to ensure proper handling of international characters in titles and filenames. By default, the file is generated at `{workdir}/metadata.json`, with the path defined by the `OUTPUT_META` constant in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py).

## Working with metadata.json in Python

Downstream tools can load and analyze the metadata to verify extraction results or estimate costs.

### Loading and Inspecting Metadata

```python
import json
from pathlib import Path

# Load the metadata for a completed extraction run

meta_path = Path("/tmp/book_skill_work/metadata.json")
with meta_path.open(encoding="utf-8") as f:
    meta = json.load(f)

# Display high-level document statistics

print(f"Document: {meta['filename']}")
print(f"Pages: {meta['pages']:,} | Words: {meta['words']:,} | Tokens: {meta['tokens']:,}")
print(f"Chapters detected: {meta['chapters']}")

# Display per-source breakdown

for src in meta["sources"]:
    print(f"- {src['source_file']}: {src['pages']} pages, {src['words']} words")

```

### Estimating Generation Costs

Use the token count to calculate approximate API costs before sending content to language models:

```python

# Example: estimating generation cost at $0.00002 per token

TOKEN_COST = 0.00002

meta_path = Path("/tmp/book_skill_work/metadata.json")
with meta_path.open(encoding="utf-8") as f:
    meta = json.load(f)

estimated_cost = meta["tokens"] * TOKEN_COST
print(f"Estimated generation cost: ${estimated_cost:,.2f}")

```

## Source Code References

The metadata generation logic and field definitions are distributed across these key files:

- **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)** – Defines `OUTPUT_META`, the configuration constant pointing to the metadata file path.
- **[`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md)** – Documents the data flow showing where [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) sits in the extraction pipeline.
- **[`docs/how-it-works.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/how-it-works.md)** – Explains the extraction stages that populate the metadata fields.
- **[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)** – Provides the authoritative field reference for consumers of [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json).
- **[`tests/test_metadata_encoding.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_metadata_encoding.py)** – Validates UTF-8 encoding requirements for the output file.

## Summary

- **The [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file** stores aggregate statistics (pages, words, tokens, chapters) and Table of Contents data for extracted documents.
- **Per-source arrays** allow tracing which statistics belong to which input file when processing multiple sources.
- **Token counts** enable accurate cost estimation for downstream language model generation.
- **UTF-8 encoding** ensures proper handling of international text, as verified by the test suite.
- **Configuration** is centralized in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) via the `OUTPUT_META` constant.

## Frequently Asked Questions

### What is the exact file path for metadata.json?

The file is located in the temporary working directory specified during the extraction run, typically at [`/tmp/book_skill_work/metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main//tmp/book_skill_work/metadata.json). The precise path is controlled by the `OUTPUT_META` setting in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) and follows the pattern `{workdir}/metadata.json`.

### Why does the metadata.json file include token counts?

Token counts are included to enable **cost estimation** for downstream language model operations. By knowing the exact token count upfront, applications can calculate API costs (e.g., multiplying tokens by $0.00002) before submitting content for generation, preventing unexpected charges.

### What does the dropped_epub_images field indicate?

This field tracks how many images were **stripped out** during EPUB processing. Since the book-to-skill extractor focuses on text content, images embedded in EPUB files are omitted from the final output, and this counter helps diagnose why visual content is missing from the extracted text.

### How can I verify the metadata.json encoding is correct?

The repository includes a dedicated test in [`tests/test_metadata_encoding.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_metadata_encoding.py) that validates the file is written with UTF-8 encoding. You can run this test to ensure the metadata properly handles international characters and non-ASCII symbols in filenames and chapter titles.