How to Run the book-to-skill Extractor Pipeline End-to-End

The book-to-skill extractor pipeline converts books and documents into queryable Agent Skills by running a deterministic extraction phase followed by a spec-driven generation phase that outputs a complete SKILL.md bundle.

The virgiliojr94/book-to-skill project transforms any collection of text-based sources into an Agent Skill that can be queried on demand. The pipeline is implemented in pure Python and handles everything from parsing PDFs and EPUBs to sanitizing text and generating the final skill bundle. Understanding the end-to-end flow ensures you can process single files or batch-process entire libraries efficiently.

Pipeline Architecture Overview

The extractor pipeline divides work into two logical halves:

1. Deterministic extractor – Reads raw files, cleans text, and produces a uniform JSON-like representation containing chapter titles, cleaned body content, and metadata.
2. Spec-driven generator – Consumes the extraction output and writes the SKILL.md bundle plus per-chapter markdown files that conform to the Agent Skills specification.

The pipeline is batch-safe: if one source fails, the extractor records the error, continues processing remaining files, and exits with a non-zero status only after attempting all inputs.

Step-by-Step Execution Flow

The pipeline orchestrates eight distinct stages from CLI invocation to final skill generation.

0. CLI Entry Point

Execution begins in book_to_skill/cli.py, which parses command-line arguments, resolves file globs, and loads configuration. This module determines whether you are creating a new skill, updating an existing one, or running a dependency check.

1. Dependency Probe

Before extraction begins, book_to_skill/dependencies.py verifies that required external tools are installed. This includes checking for pdftotext, docling, ebook-convert, and other format-specific parsers. Run the check independently with the --check flag to validate your environment.

2. Parser Dispatch

Based on file extension and detected content type, book_to_skill/parsers/__init__.py selects the appropriate format-specific parser. The dispatch system supports PDF, EPUB, DOCX, HTML, RTF, and plain text inputs.

3. Format Parsers

Specialized parsers extract clean text and structural metadata:

• PDF → book_to_skill/parsers/pdf.py
• EPUB → book_to_skill/parsers/epub.py
• DOCX → book_to_skill/parsers/docx.py

Each parser handles format-specific quirks while输出ing a standardized intermediate representation.

4. Sanitisation

book_to_skill/sanitize.py removes invisible Unicode control characters, normalizes line endings, and protects against malformed markup. This step ensures downstream generators receive clean, consistent text.

5. Extraction Orchestration

book_to_skill/utils.py coordinates multi-source batches, detects chapter breaks using heuristics, and builds a unified extraction document. This module handles the complexities of merging multiple files into a coherent skill structure.

6. Skill Generation

The pipeline writes the canonical SKILL.md plus per-chapter files (chapters/chXX-*.md), a glossary, patterns, and cheatsheet. All outputs conform to the Agent Skills specification defined in the repository's SKILL.md.

7. Optional Publishing

The CLI offers an interactive prompt to push the generated skill to a private GitHub repository using gh repo create, enabling installation on any host that supports Agent Skills.

Prerequisites and Installation

Before running the pipeline, verify that your system has the required external tools installed:

python3 scripts/extract.py --check

This command executes the dependency probe in book_to_skill/dependencies.py and reports any missing format-specific converters. Install missing tools (such as pdftotext from poppler-utils or ebook-convert from Calibre) before proceeding with extraction.

Running the Pipeline: Common Commands

The scripts/extract.py file serves as the thin entry-point wrapper that wires the CLI to the core extractor. Use these patterns to process your documents:

Process a Single File

Process one PDF with interactive mode selection (technical vs. text-heavy):

python3 scripts/extract.py ./my-book.pdf

Batch Process a Directory

Convert an entire folder of mixed sources into a unified skill named "project-knowledge":

python3 scripts/extract.py ./docs/ project-knowledge

Use Glob Patterns

Process files matching a pattern by quoting the glob to prevent shell expansion:

python3 scripts/extract.py "./books/*.epub" my-library

Update Existing Skills

Fold new material into an existing skill directory:

python3 scripts/extract.py ./new-paper.pdf ~/.claude/skills/project-knowledge

Querying Generated Skills

After the skill directory is created (e.g., ~/.claude/skills/my-library), query it with any Agent-Skills-compatible host:

/my-library replication
/my-library ch05          # Open chapter 5

/my-library "what chapters do you have?"

Key Implementation Files

Understanding these core modules helps when debugging or extending the pipeline:

• scripts/extract.py – Entry-point wrapper that initializes the CLI and orchestrates extraction.
• book_to_skill/parsers/ – Directory containing format-specific parsers for PDF, EPUB, DOCX, HTML, and RTF.
• book_to_skill/utils.py – Helper functions for multi-source resolution, chapter detection, and runner logic.
• book_to_skill/sanitize.py – Text sanitization utilities that protect against hidden Unicode control characters.
• book_to_skill/dependencies.py – Probe logic for external tool validation and the --check implementation.
• book_to_skill/cli.py – Central CLI implementation handling argument parsing and mode selection.
• SKILL.md – The spec-driven generator template that produces the final Agent Skill bundle.

Summary

• The book-to-skill extractor pipeline consists of deterministic extraction followed by spec-driven generation to create queryable Agent Skills.
• The pipeline is batch-safe, continuing processing even when individual files fail.
• Eight distinct stages handle everything from CLI parsing in book_to_skill/cli.py to final skill generation conforming to the SKILL.md specification.
• Use python3 scripts/extract.py --check to verify dependencies before running extraction.
• The system supports multiple input formats including PDF, EPUB, DOCX, HTML, and RTF through specialized parsers in book_to_skill/parsers/.

Frequently Asked Questions

What external tools does the book-to-skill pipeline require?

The pipeline requires format-specific external tools such as pdftotext (from poppler-utils), ebook-convert (from Calibre), and optionally docling for certain document types. The dependency probe in book_to_skill/dependencies.py checks for these automatically when you run python3 scripts/extract.py --check, reporting exactly which tools are missing for your target input formats.

Can I process multiple books into a single skill?

Yes. Pass a directory path or glob pattern as the first argument to scripts/extract.py. The extraction orchestrator in book_to_skill/utils.py handles multi-source batches, detects chapter breaks across files, and merges them into a unified skill structure with consistent chapter numbering and metadata.

How does the pipeline handle corrupted or unreadable files?

The pipeline implements batch-safe error handling. If a single source file fails parsing or sanitization, the extractor records the specific error, continues processing remaining files, and only exits with a non-zero status after attempting all inputs. This ensures that one bad PDF or malformed EPUB does not abort an entire library conversion job.

Where does the pipeline output the generated skill files?

By default, the pipeline creates skill directories in the Agent Skills installation path (typically ~/.claude/skills/). The generator writes SKILL.md at the root, creates a chapters/ subdirectory containing individual markdown files (e.g., ch01-intro.md), and optionally generates supplementary files like glossaries and cheatsheets depending on the detected content structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}