# Understanding the Book-to-Skill Architecture: A Two-Stage Processing Pipeline

> Discover the book-to-skill architecture, a two-stage pipeline that extracts structured text from documents and generates modular skills. Learn how this system works.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: architecture
- Published: 2026-08-26

---

**The book-to-skill architecture is a loosely coupled two-stage system comprising a deterministic extractor that converts documents into structured text and metadata, followed by a spec-driven generator that transforms that raw material into a modular skill adhering to the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification.**

The `virgiliojr94/book-to-skill` repository implements this architecture to convert static books and documents into queryable, on-demand AI skills. According to the [Architecture documentation](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md), the system separates document ingestion from skill generation, allowing the extractor to run independently while the generator consumes its output via a standardized directory layout.

## Core Components

The book-to-skill architecture consists of two complementary halves that communicate through a temporary working directory.

### Deterministic Extractor

The **deterministic extractor** is a pure-Python pipeline responsible for ingesting documents and producing clean, structured artifacts. Located primarily in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) and [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py), this component handles multi-source resolution, chapter/ToC detection, and normalization. It outputs two critical files to `<tempdir>/book_skill_work/`:

- [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) – merged source-marked text with invisible Unicode stripped
- [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) – structural data including pages, words, tokens, chapters, and table of contents

The extractor supports PDF, EPUB, DOCX, HTML, RTF, and plain text through dedicated modules in `book_to_skill/parsers/`, with each format having a standard-library fallback for graceful degradation.

### Spec-Driven Generator

The **spec-driven generator** is an agentic component that consumes the extractor's output and produces the final skill. Following the specification defined in the repository's root [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md), the generator executes a ten-step workflow including content-type detection (Step 1.5), cost estimation (Steps 2/2.5), REPL-style probing for large books (Step 2.6), and per-chapter summarization (Step 7). The final output consists of a core [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) file plus ancillary markdown files stored in a host-specific skills home such as `~/.copilot/skills/`, `~/.claude/skills/`, or `.github/skills/`.

## End-to-End Pipeline Flow

The architecture processes documents through distinct extraction and generation phases.

### Phase 1: Extraction

Documents enter the pipeline through [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), a thin shim that forwards to the main CLI in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py). The flow proceeds through:

```text
documents ──► scripts/extract.py ──► book_to_skill/
                 │
                 ├─ cli.py & utils.py   – CLI handling and chapter detection
                 ├─ config.py           – Supported extensions and paths
                 ├─ dependencies.py     – Optional-dependency probing
                 ├─ sanitize.py         – Unicode sanitization
                 └─ parsers/            – Format-specific extraction
                 │
                 ▼
          <tempdir>/book_skill_work/
                 ├─ full_text.txt
                 └─ metadata.json

```

The [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) module strips zero-width characters and Unicode tag blocks before parsing, while [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) maintains the dependency map and supported extensions.

### Phase 2: Generation

The generator consumes the temporary directory and executes the skill-creation workflow:

```text
<tempdir>/book_skill_work/  ►  Generator
      │
      ├─ Step 1.5  – Content-type detection (technical vs. plain)
      ├─ Steps 2/2.5 – Cost estimation and confirmation
      ├─ Step 2.6  – REPL-style probing (grep/sed) for large books
      ├─ Step 3    – Structure analysis (title, author, ToC)
      ├─ Step 4    – Purpose determination (reference vs. study)
      ├─ Step 7    – Per-chapter summaries (budget = BOOK_TYPE × DEPTH)
      ├─ Step 8    – Glossary, patterns, and cheatsheet construction
      └─ Steps 9/9.5 – Final SKILL.md assembly and indexing

```

## Design Principles

The book-to-skill architecture adheres to five core design principles detailed in the [Architecture documentation](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md):

- **Extract structure, not summaries** – Focus on reusable frameworks, decision rules, and anti-patterns rather than raw passages.
- **Compile-time over runtime** – Heavy parsing and token cost estimation occur once during extraction; queries load only needed chapters.
- **On-demand chapters** – The core [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) remains compact, with individual chapter files fetched only when requested.
- **Front-loaded SKILL.md** – Critical content appears first, allowing safe truncation from the end if token limits are reached.
- **Graceful degradation** – Each format implements standard-library fallbacks; a single failing source does not abort the entire run.

## Security Hardening Implementation

The architecture implements defense-in-depth security across the extraction and generation boundary:

**Sanitization Layer** – The [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) module removes invisible Unicode characters and tag blocks before processing.

**DOCX XML Guard** – The [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py) module validates XML structure and rejects documents containing dangerous DTDs or entities.

**Subprocess Safety** – Before invoking external tools like `pdftotext`, `pdfinfo`, or `ebook-convert`, the system resolves all file paths to absolute paths to prevent injection attacks.

**Generated-Skill Scanning** – The [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) utility performs advisory checks on the final output for prompt-injection patterns, hidden Unicode payloads, and unauthorized front-matter.

## Key Source Files and Module Structure

| File | Role |
|------|------|
| [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) | Thin shim forwarding to the main CLI for legacy compatibility |
| [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) | Main command-line interface orchestrating extraction |
| [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) | Helpers for source detection and chapter/ToC discovery |
| [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) | Extension declarations and dependency mapping |
| [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) | Optional tool probing with `--check` reporting |
| [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) | Unicode sanitization and invisible character removal |
| `book_to_skill/parsers/` | Per-format extraction modules (`pdf`, `epub`, `docx`, `html`, `rtf`, `calibre`, `text`) |
| [`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py) | Host-specific validation of generated skills |
| [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) | Security scanning for injection patterns |

## Quick Start: Running the Pipeline

Execute the two-stage pipeline using the module's CLI commands:

```bash

# Phase 1: Extract document structure

python -m book_to_skill extract my_technical_book.pdf

# Phase 2: Generate the skill from extracted data

python -m book_to_skill generate ./book_skill_work/

```

The `extract` command invokes [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) to produce [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) in a temporary working directory. The `generate` command triggers the spec-driven agent to analyze this data and output the structured skill files to your configured skills home directory.

## Summary

- The book-to-skill architecture separates **deterministic extraction** from **spec-driven generation**, enabling independent operation of each stage.
- The extractor outputs standardized [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) to a temporary working directory consumed by the generator.
- The generator follows a ten-step workflow defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) to produce on-demand chapter files and a compact core specification.
- Security hardening includes Unicode sanitization in [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py), XML validation in [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py), and post-generation scanning via [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py).
- Design prioritizes compile-time processing, graceful degradation, and front-loaded content to optimize token usage and reliability.

## Frequently Asked Questions

### What is the deterministic extractor in book-to-skill?

The deterministic extractor is the pure-Python ingestion pipeline located in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) and [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) that converts documents into structured text and metadata. It handles multi-format parsing through `book_to_skill/parsers/` and produces [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) without applying AI summarization, ensuring reproducible, deterministic output.

### How does the spec-driven generator create skills?

The generator is an agentic component that reads the extractor's output and follows the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification through ten defined steps. It performs content-type detection, cost estimation, structural analysis, and per-chapter summarization (budgeted by `BOOK_TYPE × DEPTH`), ultimately producing a core [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) file plus on-demand chapter markdown files in host-specific directories like `~/.copilot/skills/`.

### What security measures protect against prompt injection?

The architecture implements four security layers: [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) removes zero-width Unicode characters; [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py) rejects dangerous XML entities; subprocess calls use absolute path resolution before invoking external tools; and [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) performs advisory scans on final outputs for hidden payloads and injection patterns.

### Which document formats does the extractor support?

According to [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), the extractor supports PDF, EPUB, DOCX, HTML, RTF, and plain text files. Each format has a dedicated parser module under `book_to_skill/parsers/` with standard-library fallbacks to ensure graceful degradation when optional dependencies like `pdftotext` or `ebook-convert` are unavailable.