Where to Find book-to-skill Pipeline Documentation: Complete Guide

The official book-to-skill pipeline documentation is located in three core markdown files within the repository's docs/ directory: architecture.md for system design, how-it-works.md for execution workflows, and performance.md for benchmarking metrics.

The book-to-skill pipeline converts books into AI-ready skills through a deterministic extraction phase followed by an agent-driven generation phase. This guide maps the complete documentation structure for the virgiliojr94/book-to-skill repository, providing direct paths to architectural diagrams, source code references, and practical usage examples.

Core Documentation Files

The project maintains three primary documentation files that collectively explain the pipeline's operation:

• docs/architecture.md – Contains the high-level pipeline diagram, component responsibilities, design principles, security hardening measures, and extension points. This file defines how the deterministic extractor and spec-driven generator interact.

• docs/how-it-works.md – Provides a step-by-step walkthrough of the extractor (handling PDF, EPUB, DOCX, HTML, and RTF formats) and the generator (using Claude-driven skill creation). Includes benchmark tables and design rationale.

• docs/performance.md – Details benchmarking of extraction methods (comparing pdftotext versus Docling) and provides cost estimates for skill generation.

Pipeline Architecture Overview

According to the source code in docs/architecture.md, the pipeline operates through two distinct stages:

The Extractor (Deterministic Python)

The extractor is invoked via scripts/extract.py, which serves as a thin shim that forwards arguments to book_to_skill.cli (implemented in book_to_skill/cli.py). As documented in lines 12-24 of the architecture file, this stage parses each source format through dedicated modules in book_to_skill/parsers/ and produces a temporary working directory containing:

• full_text.txt – The extracted text content
• metadata.json – Structured metadata about the source

The parser modules handle specific formats including PDF, EPUB, DOCX, HTML, RTF, Calibre, and plain text. Security hardening is implemented in book_to_skill/sanitize.py, which strips invisible Unicode characters and protects against XXE attacks.

The Generator (Spec-Driven Agent)

The generator stage reads the extracted text and analyzes its structure according to the specification defined in SKILL.md (located at the repository root). As described in lines 27-38 of docs/architecture.md, this Claude-driven agent follows the steps outlined in the specification to produce:

• Per-chapter summaries
• A comprehensive glossary
• Pattern definitions
• A compact SKILL.md file

Output Locations

The final generated skill is written to host-specific folders as detailed in lines 40-45 of the architecture documentation. Default locations include:

• ~/.copilot/skills/ – For GitHub Copilot integration
• ~/.agents/skills/ – For general AI agent frameworks
• ~/.claude/skills/ – For Claude-specific implementations

Key Source Files and Entry Points

Understanding the pipeline requires familiarity with these critical files:

File	Role
scripts/extract.py	Entry-point shim that forwards command-line arguments to the CLI extractor
book_to_skill/cli.py	Core CLI implementation that parses arguments and orchestrates multi-source extraction
book_to_skill/parsers/	Directory containing one module per format (pdf, epub, docx, html, rtf, calibre, text) handling the actual conversion logic
book_to_skill/sanitize.py	Security module that sanitizes input by removing invisible Unicode characters and preventing XXE attacks
SKILL.md	Specification document that drives the agent generator, defining steps, indexes, and output structure
tools/validate_skill.py	CLI utility that validates generated SKILL.md files against host-specific rules

Running the Pipeline

The documentation references several standard command patterns for executing the pipeline. All commands use repository-relative paths; the scripts resolve absolute paths internally.

# Basic extraction of a PDF in technical mode with skill generation

scripts/extract.py my_book.pdf --mode technical

# Extraction of multiple sources with automatic mode detection

scripts/extract.py chapter1.pdf chapter2.epub

# Run extractor only (no skill generation) for debugging purposes

python -m book_to_skill.cli --extract-only path/to/file.pdf

# Validate a generated SKILL.md against host standards

python tools/validate_skill.py SKILL.md --lens claude

Summary

• Three core documents comprise the official book-to-skill pipeline documentation: architecture.md (design), how-it-works.md (execution), and performance.md (benchmarks).
• Two-stage architecture: A deterministic Python extractor (book_to_skill/parsers/) followed by a Claude-driven generator guided by SKILL.md.
• Entry points: Use scripts/extract.py for full pipeline execution or python -m book_to_skill.cli for extraction-only workflows.
• Security: Input sanitization occurs in book_to_skill/sanitize.py to prevent XXE attacks and remove invisible Unicode characters.
• Validation: Use tools/validate_skill.py to verify generated skills against host-specific requirements before deployment.

Frequently Asked Questions

Where is the book-to-skill pipeline architecture explained?

The complete architecture is documented in /docs/architecture.md at the repository root. This file contains the high-level pipeline diagram, explains the separation between the deterministic extractor and spec-driven generator, and details security hardening measures. Lines 12-45 specifically cover the extractor workflow, generator logic, and output directory structures.

How do I run the book-to-skill pipeline without generating a skill?

To run only the extraction phase without invoking the skill generator, use the --extract-only flag with the CLI module: python -m book_to_skill.cli --extract-only path/to/file.pdf. This creates the temporary working directory with full_text.txt and metadata.json without proceeding to the Claude-driven generation stage, making it useful for debugging parser behavior or verifying text extraction quality.

What determines where the generated skill files are saved?

Output locations are host-specific and configured in docs/architecture.md (lines 40-45). The generator automatically writes skills to framework-specific directories: ~/.copilot/skills/ for GitHub Copilot, ~/.agents/skills/ for general agent frameworks, or ~/.claude/skills/ for Claude implementations. The tools/validate_skill.py utility can verify that generated files meet each host's structural requirements.

Which file formats does the book-to-skill extractor support?

The extractor supports PDF, EPUB, DOCX, HTML, and RTF through dedicated parser modules located in book_to_skill/parsers/. Each format has its own implementation file (e.g., pdf.py, epub.py) that handles format-specific conversion to the standard full_text.txt and metadata.json interim format used by the generator stage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}