# How to Run the book-to-skill Extractor Pipeline End-to-End > Learn how to run the book-to-skill extractor pipeline end-to-end. Convert books into queryable Agent Skills with this guide. - Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill) - Tags: how-to-guide - Published: 2026-08-30 --- **The book-to-skill extractor pipeline converts books and documents into queryable Agent Skills by running a deterministic extraction phase followed by a spec-driven generation phase that outputs a complete [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) bundle.** The `virgiliojr94/book-to-skill` project transforms any collection of text-based sources into an Agent Skill that can be queried on demand. The pipeline is implemented in pure Python and handles everything from parsing PDFs and EPUBs to sanitizing text and generating the final skill bundle. Understanding the end-to-end flow ensures you can process single files or batch-process entire libraries efficiently. ## Pipeline Architecture Overview The extractor pipeline divides work into two logical halves: 1. **Deterministic extractor** – Reads raw files, cleans text, and produces a uniform JSON-like representation containing chapter titles, cleaned body content, and metadata. 2. **Spec-driven generator** – Consumes the extraction output and writes the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) bundle plus per-chapter markdown files that conform to the Agent Skills specification. The pipeline is **batch-safe**: if one source fails, the extractor records the error, continues processing remaining files, and exits with a non-zero status only after attempting all inputs. ## Step-by-Step Execution Flow The pipeline orchestrates eight distinct stages from CLI invocation to final skill generation. ### 0. CLI Entry Point Execution begins in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py), which parses command-line arguments, resolves file globs, and loads configuration. This module determines whether you are creating a new skill, updating an existing one, or running a dependency check. ### 1. Dependency Probe Before extraction begins, [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) verifies that required external tools are installed. This includes checking for `pdftotext`, `docling`, `ebook-convert`, and other format-specific parsers. Run the check independently with the `--check` flag to validate your environment. ### 2. Parser Dispatch Based on file extension and detected content type, [`book_to_skill/parsers/__init__.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/__init__.py) selects the appropriate format-specific parser. The dispatch system supports PDF, EPUB, DOCX, HTML, RTF, and plain text inputs. ### 3. Format Parsers Specialized parsers extract clean text and structural metadata: - **PDF** → [`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py) - **EPUB** → [`book_to_skill/parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/epub.py) - **DOCX** → [`book_to_skill/parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/docx.py) Each parser handles format-specific quirks while输出ing a standardized intermediate representation. ### 4. Sanitisation [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) removes invisible Unicode control characters, normalizes line endings, and protects against malformed markup. This step ensures downstream generators receive clean, consistent text. ### 5. Extraction Orchestration [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) coordinates multi-source batches, detects chapter breaks using heuristics, and builds a unified extraction document. This module handles the complexities of merging multiple files into a coherent skill structure. ### 6. Skill Generation The pipeline writes the canonical [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) plus per-chapter files (`chapters/chXX-*.md`), a glossary, patterns, and cheatsheet. All outputs conform to the Agent Skills specification defined in the repository's [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md). ### 7. Optional Publishing The CLI offers an interactive prompt to push the generated skill to a private GitHub repository using `gh repo create`, enabling installation on any host that supports Agent Skills. ## Prerequisites and Installation Before running the pipeline, verify that your system has the required external tools installed: ```bash python3 scripts/extract.py --check ``` This command executes the dependency probe in [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) and reports any missing format-specific converters. Install missing tools (such as `pdftotext` from poppler-utils or `ebook-convert` from Calibre) before proceeding with extraction. ## Running the Pipeline: Common Commands The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) file serves as the thin entry-point wrapper that wires the CLI to the core extractor. Use these patterns to process your documents: ### Process a Single File Process one PDF with interactive mode selection (technical vs. text-heavy): ```bash python3 scripts/extract.py ./my-book.pdf ``` ### Batch Process a Directory Convert an entire folder of mixed sources into a unified skill named "project-knowledge": ```bash python3 scripts/extract.py ./docs/ project-knowledge ``` ### Use Glob Patterns Process files matching a pattern by quoting the glob to prevent shell expansion: ```bash python3 scripts/extract.py "./books/*.epub" my-library ``` ### Update Existing Skills Fold new material into an existing skill directory: ```bash python3 scripts/extract.py ./new-paper.pdf ~/.claude/skills/project-knowledge ``` ## Querying Generated Skills After the skill directory is created (e.g., `~/.claude/skills/my-library`), query it with any Agent-Skills-compatible host: ```bash /my-library replication /my-library ch05 # Open chapter 5 /my-library "what chapters do you have?" ``` ## Key Implementation Files Understanding these core modules helps when debugging or extending the pipeline: - **[`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py)** – Entry-point wrapper that initializes the CLI and orchestrates extraction. - **`book_to_skill/parsers/`** – Directory containing format-specific parsers for PDF, EPUB, DOCX, HTML, and RTF. - **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** – Helper functions for multi-source resolution, chapter detection, and runner logic. - **[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)** – Text sanitization utilities that protect against hidden Unicode control characters. - **[`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py)** – Probe logic for external tool validation and the `--check` implementation. - **[`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py)** – Central CLI implementation handling argument parsing and mode selection. - **[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)** – The spec-driven generator template that produces the final Agent Skill bundle. ## Summary - The **book-to-skill extractor pipeline** consists of deterministic extraction followed by spec-driven generation to create queryable Agent Skills. - The pipeline is **batch-safe**, continuing processing even when individual files fail. - **Eight distinct stages** handle everything from CLI parsing in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) to final skill generation conforming to the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification. - Use `python3 scripts/extract.py --check` to verify dependencies before running extraction. - The system supports **multiple input formats** including PDF, EPUB, DOCX, HTML, and RTF through specialized parsers in `book_to_skill/parsers/`. ## Frequently Asked Questions ### What external tools does the book-to-skill pipeline require? The pipeline requires format-specific external tools such as `pdftotext` (from poppler-utils), `ebook-convert` (from Calibre), and optionally `docling` for certain document types. The dependency probe in [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) checks for these automatically when you run `python3 scripts/extract.py --check`, reporting exactly which tools are missing for your target input formats. ### Can I process multiple books into a single skill? Yes. Pass a directory path or glob pattern as the first argument to [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py). The extraction orchestrator in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) handles multi-source batches, detects chapter breaks across files, and merges them into a unified skill structure with consistent chapter numbering and metadata. ### How does the pipeline handle corrupted or unreadable files? The pipeline implements **batch-safe** error handling. If a single source file fails parsing or sanitization, the extractor records the specific error, continues processing remaining files, and only exits with a non-zero status after attempting all inputs. This ensures that one bad PDF or malformed EPUB does not abort an entire library conversion job. ### Where does the pipeline output the generated skill files? By default, the pipeline creates skill directories in the Agent Skills installation path (typically `~/.claude/skills/`). The generator writes [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) at the root, creates a `chapters/` subdirectory containing individual markdown files (e.g., [`ch01-intro.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/ch01-intro.md)), and optionally generates supplementary files like glossaries and cheatsheets depending on the detected content structure.