# How Extracted Text and Metadata Flow Into Agent Skill Generation in book-to-skill

> Discover how book-to-skill uses extracted text and metadata to generate agent skills. Learn about the two-phase architecture for efficient chapter creation, token budgeting, and cost estimation.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-24

---

**The book-to-skill pipeline operates through a strict two-phase architecture where the extraction phase generates [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) artefacts, and the skill-generation phase consumes these files to drive chapter creation, token budgeting, and cost estimation.**

The virgiliojr94/book-to-skill repository converts technical books into structured agent skills through a deterministic data flow. Understanding how extracted text and metadata flow into the agent skill generation process enables precise control over token budgets, source attribution, and incremental skill updates across the conversion toolchain.

## The Two-Phase Architecture

The converter separates concerns into distinct extraction and generation phases. This separation ensures that text parsing happens once, creating immutable artefacts that downstream steps reference repeatedly without re-parsing source documents.

- **Phase 1: Extraction** – `extract_single_file` in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) processes PDF, EPUB, and DOCX inputs to produce raw text and metadata dictionaries.
- **Phase 2: Skill Generation** – Steps 3 through 9 documented in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) consume the extraction artefacts to produce chapter files, glossaries, and pattern definitions.

## Phase 1: Text and Metadata Extraction

The extraction phase transforms source documents into portable artefacts stored in a temporary work directory defined by `config.OUTPUT_DIR`.

### The extract_single_file Function

Located in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 77‑120), the `extract_single_file` function serves as the primary extraction entry point. It dispatches format-specific parsers from `book_to_skill/parsers/*` and returns a standardized dictionary:

```python
{
    "text": "<concatenated plain text>",
    "metadata": {
        "source_file": "<original path>",
        "pages": <int>,
        "estimated_tokens": <int>,
        "images_dropped": <int>
    }
}

```

The function aggregates these results across all resolved inputs, summing page counts and token estimates while preserving individual source paths.

### Persisting Extraction Artefacts

Immediately following extraction, the CLI in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) writes two critical files into the temporary work directory:

- **[`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt)** – UTF-8 encoded concatenation of all resolved input text.
- **[`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json)** – Structured JSON containing combined size metrics, word counts, token counts, the complete list of individual sources, and the resolved `workdir` path.

The CLI prints three essential paths to stdout (as documented in Step 2 of [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)):

```bash

# Workdir -> /tmp/book_skill_work-12345

# Text    -> /tmp/book_skill_work-12345/full_text.txt

# Meta    -> /tmp/book_skill_work-12345/metadata.json

```

These paths constitute the only interface between the extraction and generation phases.

## Phase 2: Skill Generation Consumption

Subsequent pipeline steps (Steps 3 → 9) read the paths printed by the extraction step, or access [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) directly via the `config.OUTPUT_META` constant. The generation phase never reopens original source files; it operates exclusively on the extraction artefacts.

### Metadata-Driven Budgeting and Cost Estimation

The JSON payload drives critical generation decisions:

- **`sources`** – Powers the Chapter Index and enables cost estimation calculations in Step 2.5.
- **`pages`** and **`spine_items`** – Inform per-chapter token budget matrices (Step 7), determining content depth and summarization intensity.
- **`images_dropped`** – Triggers warnings about lost visual information that may impact skill completeness.

Step 2.5 specifically leverages these values to provide upfront cost estimates before invoking expensive LLM operations:

```python
import json
from book_to_skill.config import OUTPUT_META

meta = json.loads(OUTPUT_META.read_text(encoding="utf-8"))
print(f"📄 Combined Pages: {meta['pages']}")
print(f"📊 Estimated tokens: {meta['tokens']:,}")

```

### Text Slicing for Chapter Generation

With raw text consolidated in [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt), generators slice the file on-demand using standard POSIX tools (grep/sed) or the `Read` tool to feed chapter-specific summarizers. This approach eliminates redundant file I/O while maintaining random access to any section of the source material:

```python
from pathlib import Path
from book_to_skill.config import OUTPUT_DIR

full_text_path = Path(OUTPUT_DIR) / "full_text.txt"

# Extract Chapter 3 content for isolated processing

chapter_content = subprocess.check_output(
    ["sed", "-n", "/^Chapter 3/,/^Chapter 4/p", str(full_text_path)]
).decode()

```

The metadata-provided per-source statistics shape token budgets and depth parameters (`DEPTH`) for each chapter generation call.

## Data Flow and File Locations

The complete data flow traverses six distinct stages:

1. `resolve_input_files` resolves globs and paths into concrete `Path` objects.
2. `extract_single_file` generates text and metadata dictionaries.
3. The CLI accumulates results and writes [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) to `<tempdir>/book_skill_work-<pid>/`.
4. The CLI prints the Workdir, Text, and Meta paths to stdout.
5. All later steps read **only** these printed paths or the JSON file directly.
6. Chapter-generation functions pull required slices from [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) while using JSON-provided stats for cost warnings and budget allocation.

Key implementation files include:

- **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** – Core extraction logic and metadata assembly.
- **[`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py)** – Orchestration that bridges extraction and generation phases.
- **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)** – Defines `OUTPUT_DIR` and `OUTPUT_META` constants.
- **[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)** – Final security scanner that validates the generated skill directory (operating on final outputs, not extraction artefacts).

## Summary

- The pipeline enforces strict separation between text extraction and skill generation phases.
- **Two artefacts** ([`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json)) serve as the sole interface between phases.
- **Metadata.json** acts as the single source of truth for page counts, token budgets, source tracking, and cost warnings.
- The CLI in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) orchestrates the handoff while [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) handles parsing details.
- Downstream steps consume extraction outputs via `config.OUTPUT_META` and filesystem slicing of [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt).

## Frequently Asked Questions

### What specific files does the extraction phase create?

The extraction phase creates exactly two files in the temporary work directory: [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) containing the concatenated plain text of all inputs, and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) containing the aggregated statistics, source list, and path information. These files are defined by the `OUTPUT_DIR` and `OUTPUT_META` constants in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py).

### How does the skill generator access the extracted content?

Skill generation steps access content exclusively through the three paths printed by the CLI (Workdir, Text, and Meta) or by reading [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) directly via `config.OUTPUT_META`. The pipeline never reopens original PDF or EPUB files during the generation phase; all content is pulled from the temporary [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) artefact using text slicing tools.

### What metadata is available for cost estimation?

The [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file provides `pages` (total page count), `tokens` (estimated token count), `sources` (list of processed files), and `images_dropped` (count of non-extracted images). Step 2.5 of the pipeline specifically consumes these values to calculate API costs and warn about potential information loss from dropped images.

### How does the pipeline handle multiple input files?

The `resolve_input_files` function aggregates all user-supplied paths, then `extract_single_file` processes each individually. The CLI accumulates text segments and metadata dictionaries, summing page counts and token estimates while preserving the complete source file list. This aggregated data is then written to the shared [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) files for unified downstream processing.