How the Book-to-Skill Generation Pipeline Works: From Document to Agent Skill

The book-to-skill generation pipeline transforms PDF, EPUB, and DOCX documents into on-demand Agent Skills through a two-stage process: deterministic Python extraction followed by Claude-driven spec generation.

The virgiliojr94/book-to-skill repository implements an end-to-end conversion system that ingests books in multiple formats and outputs lightweight, queryable Agent Skills. Understanding this book-to-skill generation pipeline requires examining its strict separation between deterministic text extraction and generative AI assembly.

Pipeline Architecture Overview

The architecture enforces a clean boundary between data preparation and content generation. Raw documents enter through scripts/extract.py, flow through specialized parsers in book_to_skill/parsers/, and culminate in a temporary workspace that feeds a Claude agent governed by SKILL.md.

documents ──► scripts/extract.py ──► book_to_skill/ (extractor)
   │                         │
   │                         └─► full_text.txt  (merged, source-marked)
   │                         └─► metadata.json  (pages, tokens, chapters)
   │
   ▼
Claude agent (generator) ──► SKILL.md + chapter files ──► $SKILLS_HOME

This two-stage design ensures reproducible text extraction while allowing flexible, context-aware skill generation.

Stage 1: Deterministic Extraction (Python)

The extraction phase is entirely deterministic, converting heterogeneous document formats into sanitized plain text with metadata.

Entry Point and CLI Resolution

The pipeline begins at scripts/extract.py, a thin shim that delegates to book_to_skill/cli.py. The CLI module resolves input paths, detects document types via file signatures, and orchestrates the appropriate parser from book_to_skill/parsers/. It also handles multi-source resolution, allowing you to merge a PDF with supplementary EPUB chapters in a single command.

# Extract mixed sources into temporary work directory

scripts/extract.py \
    path/to/book.pdf \
    path/to/appendix.epub \
    --mode technical

Format-Specific Parsers

Each supported format has a dedicated parser module under book_to_skill/parsers/:

• PDF – pdf.py uses pdftotext for standard documents or Docling for code-heavy PDFs requiring layout preservation
• EPUB – epub.py extracts via ebooklib while preserving structural navigation
• DOCX – docx.py parses XML with active XXE protection to prevent entity injection attacks
• HTML/RTF/Text – corresponding lightweight parsers handle markup stripping and encoding normalization

Sanitization and Security

Raw parser output passes through book_to_skill/sanitize.py, which removes zero-width Unicode characters, bidirectional text overrides, and other invisible payloads that could manipulate downstream LLM behavior. This step is critical for security, as books sourced from untrusted origins may contain instruction-override attempts hidden in metadata fields.

Output Structure

The extractor writes results to /tmp/book_skill_work/:

• full_text.txt – Concatenated text from all sources with source markers indicating origin file
• metadata.json – Statistics including token count, detected chapters, table of contents, and page mappings

These files form the immutable input for the generative stage.

Stage 2: Spec-Driven Generation (Claude Agent)

The generative phase interprets the extracted content through a Claude-based agent that strictly follows the SKILL.md specification.

SKILL.md Specification

SKILL.md functions as the executable blueprint for the agent. It defines the generation steps, output schema, and depth budgeting constraints. The agent reads this specification first, then processes full_text.txt and metadata.json to produce the final skill artifacts.

Generation Steps

The Claude agent executes a scripted workflow defined in SKILL.md:

1. Content-type classification – Determines if the book is technical (containing code) or text-only, affecting summarization depth
2. Structure analysis – Extracts title, author, chapter boundaries, and hierarchical TOC from metadata
3. Depth budgeting – Calculates token allocation per chapter based on total length and complexity
4. Per-chunk summarization – Processes content in 800–1200 token segments; technical books retain code examples and reference tables
5. Ancillary content generation – Creates glossary entries, design patterns, and cheatsheets derived from the text
6. Assembly – Compiles a compact core SKILL.md (~4K tokens) with lazy-loading indexes to per-chapter detail files

The generated skill is written to host-specific locations such as ~/.copilot/skills/<slug>/, ~/.agents/skills/<slug>/, or ~/.claude/skills/<slug>/. After successful generation, the temporary work directory is automatically cleaned up.

# Query the generated skill (GitHub Copilot CLI example)

copilot skill list
copilot skill exec <slug>  # Loads relevant chapter on-demand

Security and Quality Controls

Beyond the sanitization layer, the pipeline implements defense-in-depth:

• DOCX XML guard – docx.py disables external entity resolution to block XXE attacks
• Subprocess hardening – File paths are passed as positional arguments with -- delimiters to prevent flag injection when calling pdftotext or external converters
• Post-generation scanning – tools/scan_generated_skill.py performs an advisory scan over output files, detecting residual instruction-override phrases or hidden Unicode
• Validation – tools/validate_skill.py enforces host-specific rules for SKILL.md structure and cross-references
• CI enforcement – CodeQL, Bandit, and CVE scanning run against the extraction codebase

Practical Usage Examples

For standard conversion workflows, invoke the extraction shim with the appropriate mode flag:

# Technical book with code examples

scripts/extract.py ./machine-learning-handbook.pdf --mode technical

# Text-only narrative

scripts/extract.py ./novel.epub --mode text

To fold new content into an existing skill (incremental updates), use --mode fold during extraction, then manually invoke the generator against the existing SKILL.md as documented in the specification file.

Summary

• The book-to-skill generation pipeline separates deterministic extraction (scripts/extract.py, book_to_skill/cli.py) from generative assembly (Claude agent driven by SKILL.md)
• Extraction produces full_text.txt and metadata.json in /tmp/book_skill_work/ via format-specific parsers with Unicode sanitization
• Generation creates a compact core skill (~4K tokens) with lazy-loaded chapter files, hosted in directories like ~/.copilot/skills/
• Security layers include XXE protection in docx.py, subprocess argument hardening, and post-generation scanning via tools/scan_generated_skill.py
• The architecture supports mixed-source ingestion and incremental folding for evolving skill maintenance

Frequently Asked Questions

What document formats does the book-to-skill pipeline support?

The pipeline supports PDF, EPUB, DOCX, HTML, RTF, and plain text via dedicated parsers in book_to_skill/parsers/. PDF handling intelligently selects between pdftotext for standard documents and Docling for layout-complex technical books containing code blocks.

How does the pipeline prevent malicious content in source documents?

The book_to_skill/sanitize.py module strips invisible Unicode characters and bidirectional overrides from extracted text. Additionally, docx.py implements XXE protection when parsing XML, and tools/scan_generated_skill.py scans final outputs for instruction-override attempts before installation.

What is the difference between --mode technical and --mode text?

The --mode flag signals the Claude agent to adjust depth budgeting and summarization strategy. Technical mode preserves code examples, syntax, and reference tables in 800–1200 token chunks, while text mode optimizes for narrative flow and conceptual summarization without code retention.

Where are the generated skills stored after conversion?

Generated skills are written to host-specific locations within the user's home directory, including ~/.copilot/skills/<slug>/ for GitHub Copilot, ~/.agents/skills/<slug>/, or ~/.claude/skills/<slug>/ depending on the target agent platform.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}