# Understanding Book-to-Skill's Two-Half Pipeline: Extractor and Generator Architecture

> Discover the Book-to-Skill two-half pipeline. Learn how its Python extractor and spec-driven generator create structured skills from source files and SKILL.md.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: architecture
- Published: 2026-08-29

---

**Book-to-Skill uses a deterministic Python extractor to parse source files into clean text and metadata, followed by a spec-driven generator that consumes [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) to build structured, on-demand skills.**

Book-to-Skill is an open-source tool that converts books and technical documents into AI-ready skills through a unique dual-stage process. The **Book-to-Skill pipeline** deliberately separates content extraction from skill generation, ensuring deterministic, reproducible outputs while maintaining security and cross-platform compatibility for Copilot, Claude Code, and Amp.

## The Two-Half Architecture Overview

The pipeline architecture implements a strict separation between data preparation and content generation. The first half operates as a deterministic Python-based **extractor** that transforms raw files into sanitized intermediate artefacts. The second half functions as a **spec-driven generator** (agent) that interprets the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification to compile the final skill structure. This design enforces the principle of "extract structure, not summaries" and ensures that given identical inputs, the extractor always produces identical [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) files.

## First Half: The Deterministic Extractor

The extractor phase is implemented entirely in Python and handles all file parsing, content sanitization, and metadata generation. Located in the `virgiliojr94/book-to-skill` repository, this half processes multiple source formats and produces standardized intermediate outputs before any AI generation occurs.

### Entry Point and CLI Orchestration

Processing begins at [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), which serves as a backwards-compatible shim that forwards to the modern implementation. The core orchestration happens in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py), which exposes the `extract()` function used by both the command-line interface and programmatic API. This module coordinates multi-source input resolution, chapter discovery, and table-of-contents generation, while [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) provides supporting utilities for path handling and output management.

```python
from book_to_skill.cli import extract

# Run the extractor on a list of sources

extract(
    sources=["chapter1.pdf", "appendix.epub"],
    output_dir="my_skill_tmp",
    skill_name="my_book_skill"
)

```

The `extract()` function returns the path to a temporary work directory containing the intermediate artefacts, using the same implementation path as the CLI shim.

### Configuration and Dependency Management

The extractor's capabilities are centralized in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), which declares supported file extensions, default output locations, and dependency mappings. Optional external tools like `pdftotext` or `docling` are detected by [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py), which provides a `--check` flag to print availability reports. This modular approach allows the extractor to gracefully degrade to standard-library implementations when external parsers are unavailable.

### Content Parsing and Sanitization

Format-specific parsing logic resides in `book_to_skill/parsers/`, with dedicated modules for **PDF**, **EPUB**, **DOCX**, **HTML**, **RTF**, **Calibre**, and plain text. Each extractor module handles raw text extraction and structural hint detection for its respective format.

Security-critical text processing occurs in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), which strips zero-width Unicode characters and tag-block characters from all parser outputs. This sanitization prevents hidden instructions or injection patterns from surviving the extraction phase and compromising downstream AI systems.

### Intermediate Outputs

Upon completion, the extractor writes two deterministic artefacts to the temporary work directory:

- **[`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt)** — Merged, source-marked text containing the entire document content
- **[`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json)** — Structured data including page counts, word counts, token estimates, and the chapter map

These files serve as the immutable input for the second half of the pipeline, ensuring that the generator phase operates on controlled, pre-validated content.

## Second Half: The Spec-Driven Generator

The generator half operates as an agent that consumes the extractor's outputs and the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification to produce the final skill directory. Unlike the deterministic extractor, this phase involves AI-driven content organization following strict architectural rules defined in the specification document.

### The SKILL.md Specification

The generator follows the workflow defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md), which describes a step-by-step process (numbered 0-10) for transforming extracted artefacts into a functional skill. The specification also defines the "fold-in" update process, allowing incremental skill refinement without full regeneration. This document effectively programs the agent's behavior, ensuring consistent output structure across different books and document types.

### Validation and Security Tooling

Quality assurance is enforced through dedicated tooling in the `tools/` directory. The [`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py) script checks generated skills against host-specific rules for Copilot, Amp, and Claude compatibility. Security verification is handled by [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py), which scans output files for hidden instructions or injection patterns that might have bypassed initial sanitization.

### Output Structure and Cost Analysis

The generator creates a standardized directory structure at `<SKILLS_HOME>/<slug>/` containing:

- [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) — Core framework and chapter index
- `chapters/*.md` — On-demand chapter files loaded individually
- [`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md) — Extracted terminology definitions
- [`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md) — Reusable code patterns and examples
- [`cheatsheet.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/cheatsheet.md) — Quick reference material

Host-specific installation locations vary between `~/.copilot/skills/`, `~/.agents/skills/`, and `~/.claude/skills/`. The economic benefit of this architecture is quantified by [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), which calculates the token cost savings achieved by the pipeline's selective chapter loading versus naively dumping entire books into context windows.

## Practical Usage Examples

### Extracting Content from the Command Line

Process a PDF through the complete first half using the historic CLI entry point:

```bash
book-to-skill ./my-book.pdf my-book-skill

```

This command invokes [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), which delegates to [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) to select the appropriate parser (such as `docling` for technical PDFs) and write the intermediate artefacts to a temporary directory.

### Validating Generated Skills

Ensure compliance with host-specific standards before installation:

```bash
python -m tools.validate_skill ~/.copilot/skills/my-book-skill --lens copilot

```

The validator reads the generated [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) and auxiliary files, verifying they obey the target platform's content rules and structural requirements.

## Summary

- **Book-to-Skill** implements a two-half pipeline separating deterministic extraction from AI-driven generation.
- The **extractor** ([`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py), `book_to_skill/parsers/`) produces sanitized [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) from multiple input formats.
- **Security** is enforced at multiple layers through [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) and [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py).
- The **generator** follows the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification to create on-demand skill structures with individual chapter files.
- **Cost efficiency** is measured by [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), quantifying token savings over full-document context loading.

## Frequently Asked Questions

### What makes the Book-to-Skill extractor deterministic?

The extractor guarantees identical outputs for identical inputs through its pure Python implementation in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) and format-specific parsers. It bypasses stochastic processes, producing immutable [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) files that serve as reliable inputs for the generator phase. This determinism ensures reproducible builds and simplifies debugging across different environments.

### How does the generator use SKILL.md to create skills?

The generator treats [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) as an executable specification that defines a 10-step workflow for transforming extracted artefacts into a skill directory. The document also specifies the "fold-in" update mechanism for incremental refinement. By following this specification, the agent produces consistent output structures including chapter files, glossaries, and cheatsheets regardless of the input document's original format.

### What security measures are implemented in the pipeline?

Security is enforced at extraction time by [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), which removes zero-width Unicode characters and tag blocks that could conceal malicious instructions. Post-generation validation is performed by [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py), which audits final outputs for hidden injection patterns. Additionally, [`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py) ensures generated content complies with host platform safety standards before installation.

### Which file formats does Book-to-Skill support?

According to [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) and the parsers directory, the extractor handles **PDF**, **EPUB**, **DOCX**, **HTML**, **RTF**, **Calibre** exports, and plain text files. The dependency probe in [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) detects optional external tools like `pdftotext` or `docling`, falling back to standard-library implementations when these are unavailable.