# How the Book-to-Skill Generation Pipeline Works: From Document to Agent Skill > Discover how the book-to-skill generation pipeline converts documents into agent skills. Learn about Python extraction and Claude spec generation for on-demand skills. - Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill) - Tags: internals - Published: 2026-08-28 --- **The book-to-skill generation pipeline transforms PDF, EPUB, and DOCX documents into on-demand Agent Skills through a two-stage process: deterministic Python extraction followed by Claude-driven spec generation.** The `virgiliojr94/book-to-skill` repository implements an end-to-end conversion system that ingests books in multiple formats and outputs lightweight, queryable Agent Skills. Understanding this book-to-skill generation pipeline requires examining its strict separation between deterministic text extraction and generative AI assembly. ## Pipeline Architecture Overview The architecture enforces a clean boundary between data preparation and content generation. Raw documents enter through [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), flow through specialized parsers in `book_to_skill/parsers/`, and culminate in a temporary workspace that feeds a Claude agent governed by [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md). ``` documents ──► scripts/extract.py ──► book_to_skill/ (extractor) │ │ │ └─► full_text.txt (merged, source-marked) │ └─► metadata.json (pages, tokens, chapters) │ ▼ Claude agent (generator) ──► SKILL.md + chapter files ──► $SKILLS_HOME ``` This two-stage design ensures reproducible text extraction while allowing flexible, context-aware skill generation. ## Stage 1: Deterministic Extraction (Python) The extraction phase is entirely deterministic, converting heterogeneous document formats into sanitized plain text with metadata. ### Entry Point and CLI Resolution The pipeline begins at [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), a thin shim that delegates to [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py). The CLI module resolves input paths, detects document types via file signatures, and orchestrates the appropriate parser from `book_to_skill/parsers/`. It also handles multi-source resolution, allowing you to merge a PDF with supplementary EPUB chapters in a single command. ```bash # Extract mixed sources into temporary work directory scripts/extract.py \ path/to/book.pdf \ path/to/appendix.epub \ --mode technical ``` ### Format-Specific Parsers Each supported format has a dedicated parser module under `book_to_skill/parsers/`: - **PDF** – [`pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/pdf.py) uses `pdftotext` for standard documents or *Docling* for code-heavy PDFs requiring layout preservation - **EPUB** – [`epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/epub.py) extracts via `ebooklib` while preserving structural navigation - **DOCX** – [`docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/docx.py) parses XML with active XXE protection to prevent entity injection attacks - **HTML/RTF/Text** – corresponding lightweight parsers handle markup stripping and encoding normalization ### Sanitization and Security Raw parser output passes through [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), which removes zero-width Unicode characters, bidirectional text overrides, and other invisible payloads that could manipulate downstream LLM behavior. This step is critical for security, as books sourced from untrusted origins may contain instruction-override attempts hidden in metadata fields. ### Output Structure The extractor writes results to `/tmp/book_skill_work/`: - [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) – Concatenated text from all sources with source markers indicating origin file - [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) – Statistics including token count, detected chapters, table of contents, and page mappings These files form the immutable input for the generative stage. ## Stage 2: Spec-Driven Generation (Claude Agent) The generative phase interprets the extracted content through a Claude-based agent that strictly follows the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification. ### SKILL.md Specification [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) functions as the executable blueprint for the agent. It defines the generation steps, output schema, and depth budgeting constraints. The agent reads this specification first, then processes [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) to produce the final skill artifacts. ### Generation Steps The Claude agent executes a scripted workflow defined in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md): 1. **Content-type classification** – Determines if the book is technical (containing code) or text-only, affecting summarization depth 2. **Structure analysis** – Extracts title, author, chapter boundaries, and hierarchical TOC from metadata 3. **Depth budgeting** – Calculates token allocation per chapter based on total length and complexity 4. **Per-chunk summarization** – Processes content in 800–1200 token segments; technical books retain code examples and reference tables 5. **Ancillary content generation** – Creates glossary entries, design patterns, and cheatsheets derived from the text 6. **Assembly** – Compiles a compact core [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) (~4K tokens) with lazy-loading indexes to per-chapter detail files The generated skill is written to host-specific locations such as `~/.copilot/skills//`, `~/.agents/skills//`, or `~/.claude/skills//`. After successful generation, the temporary work directory is automatically cleaned up. ```bash # Query the generated skill (GitHub Copilot CLI example) copilot skill list copilot skill exec # Loads relevant chapter on-demand ``` ## Security and Quality Controls Beyond the sanitization layer, the pipeline implements defense-in-depth: - **DOCX XML guard** – [`docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/docx.py) disables external entity resolution to block XXE attacks - **Subprocess hardening** – File paths are passed as positional arguments with `--` delimiters to prevent flag injection when calling `pdftotext` or external converters - **Post-generation scanning** – [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) performs an advisory scan over output files, detecting residual instruction-override phrases or hidden Unicode - **Validation** – [`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py) enforces host-specific rules for [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) structure and cross-references - **CI enforcement** – CodeQL, Bandit, and CVE scanning run against the extraction codebase ## Practical Usage Examples For standard conversion workflows, invoke the extraction shim with the appropriate mode flag: ```bash # Technical book with code examples scripts/extract.py ./machine-learning-handbook.pdf --mode technical # Text-only narrative scripts/extract.py ./novel.epub --mode text ``` To fold new content into an existing skill (incremental updates), use `--mode fold` during extraction, then manually invoke the generator against the existing [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) as documented in the specification file. ## Summary - The **book-to-skill generation pipeline** separates deterministic extraction ([`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py)) from generative assembly (Claude agent driven by [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)) - **Extraction** produces [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) in `/tmp/book_skill_work/` via format-specific parsers with Unicode sanitization - **Generation** creates a compact core skill (~4K tokens) with lazy-loaded chapter files, hosted in directories like `~/.copilot/skills/` - **Security layers** include XXE protection in [`docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/docx.py), subprocess argument hardening, and post-generation scanning via [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) - The architecture supports **mixed-source ingestion** and **incremental folding** for evolving skill maintenance ## Frequently Asked Questions ### What document formats does the book-to-skill pipeline support? The pipeline supports PDF, EPUB, DOCX, HTML, RTF, and plain text via dedicated parsers in `book_to_skill/parsers/`. PDF handling intelligently selects between `pdftotext` for standard documents and Docling for layout-complex technical books containing code blocks. ### How does the pipeline prevent malicious content in source documents? The [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) module strips invisible Unicode characters and bidirectional overrides from extracted text. Additionally, [`docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/docx.py) implements XXE protection when parsing XML, and [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) scans final outputs for instruction-override attempts before installation. ### What is the difference between --mode technical and --mode text? The `--mode` flag signals the Claude agent to adjust depth budgeting and summarization strategy. **Technical** mode preserves code examples, syntax, and reference tables in 800–1200 token chunks, while **text** mode optimizes for narrative flow and conceptual summarization without code retention. ### Where are the generated skills stored after conversion? Generated skills are written to host-specific locations within the user's home directory, including `~/.copilot/skills//` for GitHub Copilot, `~/.agents/skills//`, or `~/.claude/skills//` depending on the target agent platform.