How Extracted Text and Metadata Flow Into Agent Skill Generation in book-to-skill

The book-to-skill pipeline operates through a strict two-phase architecture where the extraction phase generates full_text.txt and metadata.json artefacts, and the skill-generation phase consumes these files to drive chapter creation, token budgeting, and cost estimation.

The virgiliojr94/book-to-skill repository converts technical books into structured agent skills through a deterministic data flow. Understanding how extracted text and metadata flow into the agent skill generation process enables precise control over token budgets, source attribution, and incremental skill updates across the conversion toolchain.

The Two-Phase Architecture

The converter separates concerns into distinct extraction and generation phases. This separation ensures that text parsing happens once, creating immutable artefacts that downstream steps reference repeatedly without re-parsing source documents.

• Phase 1: Extraction – extract_single_file in book_to_skill/utils.py processes PDF, EPUB, and DOCX inputs to produce raw text and metadata dictionaries.
• Phase 2: Skill Generation – Steps 3 through 9 documented in SKILL.md consume the extraction artefacts to produce chapter files, glossaries, and pattern definitions.

Phase 1: Text and Metadata Extraction

The extraction phase transforms source documents into portable artefacts stored in a temporary work directory defined by config.OUTPUT_DIR.

The extract_single_file Function

Located in book_to_skill/utils.py (lines 77‑120), the extract_single_file function serves as the primary extraction entry point. It dispatches format-specific parsers from book_to_skill/parsers/* and returns a standardized dictionary:

{
    "text": "<concatenated plain text>",
    "metadata": {
        "source_file": "<original path>",
        "pages": <int>,
        "estimated_tokens": <int>,
        "images_dropped": <int>
    }
}

The function aggregates these results across all resolved inputs, summing page counts and token estimates while preserving individual source paths.

Persisting Extraction Artefacts

Immediately following extraction, the CLI in book_to_skill/cli.py writes two critical files into the temporary work directory:

• full_text.txt – UTF-8 encoded concatenation of all resolved input text.
• metadata.json – Structured JSON containing combined size metrics, word counts, token counts, the complete list of individual sources, and the resolved workdir path.

The CLI prints three essential paths to stdout (as documented in Step 2 of SKILL.md):

# Workdir -> /tmp/book_skill_work-12345

# Text    -> /tmp/book_skill_work-12345/full_text.txt

# Meta    -> /tmp/book_skill_work-12345/metadata.json

These paths constitute the only interface between the extraction and generation phases.

Phase 2: Skill Generation Consumption

Subsequent pipeline steps (Steps 3 → 9) read the paths printed by the extraction step, or access metadata.json directly via the config.OUTPUT_META constant. The generation phase never reopens original source files; it operates exclusively on the extraction artefacts.

Metadata-Driven Budgeting and Cost Estimation

The JSON payload drives critical generation decisions:

• sources – Powers the Chapter Index and enables cost estimation calculations in Step 2.5.
• pages and spine_items – Inform per-chapter token budget matrices (Step 7), determining content depth and summarization intensity.
• images_dropped – Triggers warnings about lost visual information that may impact skill completeness.

Step 2.5 specifically leverages these values to provide upfront cost estimates before invoking expensive LLM operations:

import json
from book_to_skill.config import OUTPUT_META

meta = json.loads(OUTPUT_META.read_text(encoding="utf-8"))
print(f"📄 Combined Pages: {meta['pages']}")
print(f"📊 Estimated tokens: {meta['tokens']:,}")

Text Slicing for Chapter Generation

With raw text consolidated in full_text.txt, generators slice the file on-demand using standard POSIX tools (grep/sed) or the Read tool to feed chapter-specific summarizers. This approach eliminates redundant file I/O while maintaining random access to any section of the source material:

from pathlib import Path
from book_to_skill.config import OUTPUT_DIR

full_text_path = Path(OUTPUT_DIR) / "full_text.txt"

# Extract Chapter 3 content for isolated processing

chapter_content = subprocess.check_output(
    ["sed", "-n", "/^Chapter 3/,/^Chapter 4/p", str(full_text_path)]
).decode()

The metadata-provided per-source statistics shape token budgets and depth parameters (DEPTH) for each chapter generation call.

Data Flow and File Locations

The complete data flow traverses six distinct stages:

1. resolve_input_files resolves globs and paths into concrete Path objects.
2. extract_single_file generates text and metadata dictionaries.
3. The CLI accumulates results and writes full_text.txt and metadata.json to <tempdir>/book_skill_work-<pid>/.
4. The CLI prints the Workdir, Text, and Meta paths to stdout.
5. All later steps read only these printed paths or the JSON file directly.
6. Chapter-generation functions pull required slices from full_text.txt while using JSON-provided stats for cost warnings and budget allocation.

Key implementation files include:

• book_to_skill/utils.py – Core extraction logic and metadata assembly.
• book_to_skill/cli.py – Orchestration that bridges extraction and generation phases.
• book_to_skill/config.py – Defines OUTPUT_DIR and OUTPUT_META constants.
• tools/scan_generated_skill.py – Final security scanner that validates the generated skill directory (operating on final outputs, not extraction artefacts).

Summary

• The pipeline enforces strict separation between text extraction and skill generation phases.
• Two artefacts (full_text.txt and metadata.json) serve as the sole interface between phases.
• Metadata.json acts as the single source of truth for page counts, token budgets, source tracking, and cost warnings.
• The CLI in book_to_skill/cli.py orchestrates the handoff while book_to_skill/utils.py handles parsing details.
• Downstream steps consume extraction outputs via config.OUTPUT_META and filesystem slicing of full_text.txt.

Frequently Asked Questions

What specific files does the extraction phase create?

The extraction phase creates exactly two files in the temporary work directory: full_text.txt containing the concatenated plain text of all inputs, and metadata.json containing the aggregated statistics, source list, and path information. These files are defined by the OUTPUT_DIR and OUTPUT_META constants in book_to_skill/config.py.

How does the skill generator access the extracted content?

Skill generation steps access content exclusively through the three paths printed by the CLI (Workdir, Text, and Meta) or by reading metadata.json directly via config.OUTPUT_META. The pipeline never reopens original PDF or EPUB files during the generation phase; all content is pulled from the temporary full_text.txt artefact using text slicing tools.

What metadata is available for cost estimation?

The metadata.json file provides pages (total page count), tokens (estimated token count), sources (list of processed files), and images_dropped (count of non-extracted images). Step 2.5 of the pipeline specifically consumes these values to calculate API costs and warn about potential information loss from dropped images.

How does the pipeline handle multiple input files?

The resolve_input_files function aggregates all user-supplied paths, then extract_single_file processes each individually. The CLI accumulates text segments and metadata dictionaries, summing page counts and token estimates while preserving the complete source file list. This aggregated data is then written to the shared full_text.txt and metadata.json files for unified downstream processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}