Understanding the Book-to-Skill Architecture: A Two-Stage Processing Pipeline

The book-to-skill architecture is a loosely coupled two-stage system comprising a deterministic extractor that converts documents into structured text and metadata, followed by a spec-driven generator that transforms that raw material into a modular skill adhering to the SKILL.md specification.

The virgiliojr94/book-to-skill repository implements this architecture to convert static books and documents into queryable, on-demand AI skills. According to the Architecture documentation, the system separates document ingestion from skill generation, allowing the extractor to run independently while the generator consumes its output via a standardized directory layout.

Core Components

The book-to-skill architecture consists of two complementary halves that communicate through a temporary working directory.

Deterministic Extractor

The deterministic extractor is a pure-Python pipeline responsible for ingesting documents and producing clean, structured artifacts. Located primarily in book_to_skill/cli.py and book_to_skill/utils.py, this component handles multi-source resolution, chapter/ToC detection, and normalization. It outputs two critical files to <tempdir>/book_skill_work/:

• full_text.txt – merged source-marked text with invisible Unicode stripped
• metadata.json – structural data including pages, words, tokens, chapters, and table of contents

The extractor supports PDF, EPUB, DOCX, HTML, RTF, and plain text through dedicated modules in book_to_skill/parsers/, with each format having a standard-library fallback for graceful degradation.

Spec-Driven Generator

The spec-driven generator is an agentic component that consumes the extractor's output and produces the final skill. Following the specification defined in the repository's root SKILL.md, the generator executes a ten-step workflow including content-type detection (Step 1.5), cost estimation (Steps 2/2.5), REPL-style probing for large books (Step 2.6), and per-chapter summarization (Step 7). The final output consists of a core SKILL.md file plus ancillary markdown files stored in a host-specific skills home such as ~/.copilot/skills/, ~/.claude/skills/, or .github/skills/.

End-to-End Pipeline Flow

The architecture processes documents through distinct extraction and generation phases.

Phase 1: Extraction

Documents enter the pipeline through scripts/extract.py, a thin shim that forwards to the main CLI in book_to_skill/cli.py. The flow proceeds through:

documents ──► scripts/extract.py ──► book_to_skill/
                 │
                 ├─ cli.py & utils.py   – CLI handling and chapter detection
                 ├─ config.py           – Supported extensions and paths
                 ├─ dependencies.py     – Optional-dependency probing
                 ├─ sanitize.py         – Unicode sanitization
                 └─ parsers/            – Format-specific extraction
                 │
                 ▼
          <tempdir>/book_skill_work/
                 ├─ full_text.txt
                 └─ metadata.json

The book_to_skill/sanitize.py module strips zero-width characters and Unicode tag blocks before parsing, while book_to_skill/config.py maintains the dependency map and supported extensions.

Phase 2: Generation

The generator consumes the temporary directory and executes the skill-creation workflow:

<tempdir>/book_skill_work/  ►  Generator
      │
      ├─ Step 1.5  – Content-type detection (technical vs. plain)
      ├─ Steps 2/2.5 – Cost estimation and confirmation
      ├─ Step 2.6  – REPL-style probing (grep/sed) for large books
      ├─ Step 3    – Structure analysis (title, author, ToC)
      ├─ Step 4    – Purpose determination (reference vs. study)
      ├─ Step 7    – Per-chapter summaries (budget = BOOK_TYPE × DEPTH)
      ├─ Step 8    – Glossary, patterns, and cheatsheet construction
      └─ Steps 9/9.5 – Final SKILL.md assembly and indexing

Design Principles

The book-to-skill architecture adheres to five core design principles detailed in the Architecture documentation:

• Extract structure, not summaries – Focus on reusable frameworks, decision rules, and anti-patterns rather than raw passages.
• Compile-time over runtime – Heavy parsing and token cost estimation occur once during extraction; queries load only needed chapters.
• On-demand chapters – The core SKILL.md remains compact, with individual chapter files fetched only when requested.
• Front-loaded SKILL.md – Critical content appears first, allowing safe truncation from the end if token limits are reached.
• Graceful degradation – Each format implements standard-library fallbacks; a single failing source does not abort the entire run.

Security Hardening Implementation

The architecture implements defense-in-depth security across the extraction and generation boundary:

Sanitization Layer – The book_to_skill/sanitize.py module removes invisible Unicode characters and tag blocks before processing.

DOCX XML Guard – The parsers/docx.py module validates XML structure and rejects documents containing dangerous DTDs or entities.

Subprocess Safety – Before invoking external tools like pdftotext, pdfinfo, or ebook-convert, the system resolves all file paths to absolute paths to prevent injection attacks.

Generated-Skill Scanning – The tools/scan_generated_skill.py utility performs advisory checks on the final output for prompt-injection patterns, hidden Unicode payloads, and unauthorized front-matter.

Key Source Files and Module Structure

File	Role
scripts/extract.py	Thin shim forwarding to the main CLI for legacy compatibility
book_to_skill/cli.py	Main command-line interface orchestrating extraction
book_to_skill/utils.py	Helpers for source detection and chapter/ToC discovery
book_to_skill/config.py	Extension declarations and dependency mapping
book_to_skill/dependencies.py	Optional tool probing with --check reporting
book_to_skill/sanitize.py	Unicode sanitization and invisible character removal
book_to_skill/parsers/	Per-format extraction modules (pdf, epub, docx, html, rtf, calibre, text)
tools/validate_skill.py	Host-specific validation of generated skills
tools/scan_generated_skill.py	Security scanning for injection patterns

Quick Start: Running the Pipeline

Execute the two-stage pipeline using the module's CLI commands:

# Phase 1: Extract document structure

python -m book_to_skill extract my_technical_book.pdf

# Phase 2: Generate the skill from extracted data

python -m book_to_skill generate ./book_skill_work/

The extract command invokes book_to_skill/cli.py to produce full_text.txt and metadata.json in a temporary working directory. The generate command triggers the spec-driven agent to analyze this data and output the structured skill files to your configured skills home directory.

Summary

• The book-to-skill architecture separates deterministic extraction from spec-driven generation, enabling independent operation of each stage.
• The extractor outputs standardized full_text.txt and metadata.json to a temporary working directory consumed by the generator.
• The generator follows a ten-step workflow defined in SKILL.md to produce on-demand chapter files and a compact core specification.
• Security hardening includes Unicode sanitization in sanitize.py, XML validation in parsers/docx.py, and post-generation scanning via tools/scan_generated_skill.py.
• Design prioritizes compile-time processing, graceful degradation, and front-loaded content to optimize token usage and reliability.

Frequently Asked Questions

What is the deterministic extractor in book-to-skill?

The deterministic extractor is the pure-Python ingestion pipeline located in book_to_skill/cli.py and book_to_skill/utils.py that converts documents into structured text and metadata. It handles multi-format parsing through book_to_skill/parsers/ and produces full_text.txt and metadata.json without applying AI summarization, ensuring reproducible, deterministic output.

How does the spec-driven generator create skills?

The generator is an agentic component that reads the extractor's output and follows the SKILL.md specification through ten defined steps. It performs content-type detection, cost estimation, structural analysis, and per-chapter summarization (budgeted by BOOK_TYPE × DEPTH), ultimately producing a core SKILL.md file plus on-demand chapter markdown files in host-specific directories like ~/.copilot/skills/.

What security measures protect against prompt injection?

The architecture implements four security layers: book_to_skill/sanitize.py removes zero-width Unicode characters; parsers/docx.py rejects dangerous XML entities; subprocess calls use absolute path resolution before invoking external tools; and tools/scan_generated_skill.py performs advisory scans on final outputs for hidden payloads and injection patterns.

Which document formats does the extractor support?

According to book_to_skill/config.py, the extractor supports PDF, EPUB, DOCX, HTML, RTF, and plain text files. Each format has a dedicated parser module under book_to_skill/parsers/ with standard-library fallbacks to ensure graceful degradation when optional dependencies like pdftotext or ebook-convert are unavailable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}