# What Information Is Contained in the Book-to-Skill metadata.json File?

> Discover what information the metadata.json file holds for Book-to-Skill, including extraction statistics, processing details, and run metadata. Understand pipeline completion signals.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: api-reference
- Published: 2026-08-29

---

**The metadata.json file contains aggregated extraction statistics, per-source processing details, and global run metadata, serving as the atomic completion signal for Book-to-Skill pipeline runs.**

When the Book-to-Skill extractor finishes processing documents, it writes a comprehensive [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file that serves as the single source of truth for the extraction run. This JSON artefact captures everything from total word counts to per-source diagnostics, enabling downstream agents to validate inputs and locate intermediate files. Understanding the structure of this file is essential for anyone building automation pipelines with the virgiliojr94/book-to-skill repository.

## Aggregated Statistics Section

The root level of [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) provides high-level counters that summarize the entire extraction batch. These fields aggregate data across all processed sources:

- **`pages`**: Total page count across all documents
- **`words`**: Cumulative word count from all sources  
- **`tokens`**: Aggregate token count for downstream generation sizing

These totals allow quick validation of extraction scope without parsing individual source entries.

## Per-Source Array Structure

The `sources` array contains one entry for every input document (PDF, EPUB, HTML, etc.). Each object in this array includes specific metadata about that source:

- **`source_file`**: Absolute path to the original input file
- **`filename`**: Base name displayed to users
- **`type`**: Detected format (e.g., `pdf`, `epub`, `html`)
- **`pages`**, **`words`**, **`tokens`**: Individual statistics for that specific source
- **`dropped_images`**: Number of EPUB images ignored during extraction (see [`tests/test_epub_image_reporting.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_epub_image_reporting.py))
- **`workdir`**: Temporary directory holding intermediate artefacts for that specific source

This granularity enables precise troubleshooting when specific documents fail validation or produce unexpected token counts.

## Global Metadata and File Location

According to [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), the constant `OUTPUT_META` defines the filename for this artefact, which is written to the run's work directory. The file is always encoded in UTF-8, as enforced by [`tests/test_metadata_encoding.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_metadata_encoding.py), ensuring cross-platform compatibility regardless of host locale settings.

In addition to the aggregated and per-source data, the root object includes convenience fields duplicated from the first source:

- **`workdir`**: Absolute path to the run's temporary directory (used by agents to locate [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt))
- **`filename`** and **`source_file`**: Top-level references for quick access without iterating the sources array

## Atomic Completion Guarantee

The [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file is deliberately written only after all extraction steps have succeeded. As documented in [`docs/how-it-works.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/how-it-works.md), any pipeline failure aborts before the file is created. This design guarantees that the mere presence of [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) signals a complete, successful extraction—making it a reliable checkpoint for downstream automation.

## How to Parse metadata.json in Python

Consume the metadata file to validate extraction results before generation steps. The following pattern mirrors the example from [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md):

```python
import json
from pathlib import Path

# Path to the metadata file printed by the extraction run

meta_path = Path("/tmp/book_skill_work/metadata.json")

# Load the JSON document (UTF-8 is guaranteed)

with meta_path.open(encoding="utf-8") as fp:
    meta = json.load(fp)

# Quick overview

print("Total pages:", meta["pages"])
print("Total words:", meta["words"])
print("Total tokens:", meta["tokens"])

# Inspect each source

for src in meta["sources"]:
    print(f"→ {src['filename']} ({src['type']}) – {src['pages']} pages")

```

This snippet demonstrates accessing both aggregated totals and per-source diagnostics, confirming the correct document was processed before initiating downstream generation tasks.

## Summary

- **Aggregated statistics** (`pages`, `words`, `tokens`) provide extraction batch totals at the root level.
- The **`sources`** array contains detailed per-document metadata including file paths, detected types, and processing statistics.
- **`dropped_images`** specifically tracks ignored EPUB image assets for format-specific debugging.
- The file is written to the work directory defined by `OUTPUT_META` in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) using guaranteed UTF-8 encoding.
- Presence of [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) acts as an atomic completion signal indicating successful extraction.

## Frequently Asked Questions

### What encoding does the Book-to-Skill metadata.json file use?

The file is always encoded in UTF-8, as validated by [`tests/test_metadata_encoding.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_metadata_encoding.py). This ensures the JSON can be parsed identically across different operating systems and locale configurations without encoding errors.

### Where is the metadata.json file located after extraction?

According to [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), the file is written to the extraction run's work directory using the filename defined by the `OUTPUT_META` constant. The absolute path is also stored in the `workdir` field within the JSON itself.

### What per-source information is available for EPUB files specifically?

In addition to standard fields like `pages` and `words`, EPUB entries include `dropped_images`, which reports the count of images ignored during extraction. This field is tested in [`tests/test_epub_image_reporting.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_epub_image_reporting.py) and helps diagnose content loss in ebook processing pipelines.

### How can I verify that a Book-to-Skill extraction completed successfully?

Check for the existence of [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) in the work directory. Because the file is only written after all extraction steps succeed (as documented in [`docs/how-it-works.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/how-it-works.md)), its presence guarantees that the pipeline finished without errors and that the aggregated statistics and per-source arrays contain valid, complete data.