Understanding Book-to-Skill's Two-Half Pipeline: Extractor and Generator Architecture

Book-to-Skill uses a deterministic Python extractor to parse source files into clean text and metadata, followed by a spec-driven generator that consumes SKILL.md to build structured, on-demand skills.

Book-to-Skill is an open-source tool that converts books and technical documents into AI-ready skills through a unique dual-stage process. The Book-to-Skill pipeline deliberately separates content extraction from skill generation, ensuring deterministic, reproducible outputs while maintaining security and cross-platform compatibility for Copilot, Claude Code, and Amp.

The Two-Half Architecture Overview

The pipeline architecture implements a strict separation between data preparation and content generation. The first half operates as a deterministic Python-based extractor that transforms raw files into sanitized intermediate artefacts. The second half functions as a spec-driven generator (agent) that interprets the SKILL.md specification to compile the final skill structure. This design enforces the principle of "extract structure, not summaries" and ensures that given identical inputs, the extractor always produces identical full_text.txt and metadata.json files.

First Half: The Deterministic Extractor

The extractor phase is implemented entirely in Python and handles all file parsing, content sanitization, and metadata generation. Located in the virgiliojr94/book-to-skill repository, this half processes multiple source formats and produces standardized intermediate outputs before any AI generation occurs.

Entry Point and CLI Orchestration

Processing begins at scripts/extract.py, which serves as a backwards-compatible shim that forwards to the modern implementation. The core orchestration happens in book_to_skill/cli.py, which exposes the extract() function used by both the command-line interface and programmatic API. This module coordinates multi-source input resolution, chapter discovery, and table-of-contents generation, while book_to_skill/utils.py provides supporting utilities for path handling and output management.

from book_to_skill.cli import extract

# Run the extractor on a list of sources

extract(
    sources=["chapter1.pdf", "appendix.epub"],
    output_dir="my_skill_tmp",
    skill_name="my_book_skill"
)

The extract() function returns the path to a temporary work directory containing the intermediate artefacts, using the same implementation path as the CLI shim.

Configuration and Dependency Management

The extractor's capabilities are centralized in book_to_skill/config.py, which declares supported file extensions, default output locations, and dependency mappings. Optional external tools like pdftotext or docling are detected by book_to_skill/dependencies.py, which provides a --check flag to print availability reports. This modular approach allows the extractor to gracefully degrade to standard-library implementations when external parsers are unavailable.

Content Parsing and Sanitization

Format-specific parsing logic resides in book_to_skill/parsers/, with dedicated modules for PDF, EPUB, DOCX, HTML, RTF, Calibre, and plain text. Each extractor module handles raw text extraction and structural hint detection for its respective format.

Security-critical text processing occurs in book_to_skill/sanitize.py, which strips zero-width Unicode characters and tag-block characters from all parser outputs. This sanitization prevents hidden instructions or injection patterns from surviving the extraction phase and compromising downstream AI systems.

Intermediate Outputs

Upon completion, the extractor writes two deterministic artefacts to the temporary work directory:

• full_text.txt — Merged, source-marked text containing the entire document content
• metadata.json — Structured data including page counts, word counts, token estimates, and the chapter map

These files serve as the immutable input for the second half of the pipeline, ensuring that the generator phase operates on controlled, pre-validated content.

Second Half: The Spec-Driven Generator

The generator half operates as an agent that consumes the extractor's outputs and the SKILL.md specification to produce the final skill directory. Unlike the deterministic extractor, this phase involves AI-driven content organization following strict architectural rules defined in the specification document.

The SKILL.md Specification

The generator follows the workflow defined in SKILL.md, which describes a step-by-step process (numbered 0-10) for transforming extracted artefacts into a functional skill. The specification also defines the "fold-in" update process, allowing incremental skill refinement without full regeneration. This document effectively programs the agent's behavior, ensuring consistent output structure across different books and document types.

Validation and Security Tooling

Quality assurance is enforced through dedicated tooling in the tools/ directory. The tools/validate_skill.py script checks generated skills against host-specific rules for Copilot, Amp, and Claude compatibility. Security verification is handled by tools/scan_generated_skill.py, which scans output files for hidden instructions or injection patterns that might have bypassed initial sanitization.

Output Structure and Cost Analysis

The generator creates a standardized directory structure at <SKILLS_HOME>/<slug>/ containing:

• SKILL.md — Core framework and chapter index
• chapters/*.md — On-demand chapter files loaded individually
• glossary.md — Extracted terminology definitions
• patterns.md — Reusable code patterns and examples
• cheatsheet.md — Quick reference material

Host-specific installation locations vary between ~/.copilot/skills/, ~/.agents/skills/, and ~/.claude/skills/. The economic benefit of this architecture is quantified by tools/discovery_tax.py, which calculates the token cost savings achieved by the pipeline's selective chapter loading versus naively dumping entire books into context windows.

Practical Usage Examples

Extracting Content from the Command Line

Process a PDF through the complete first half using the historic CLI entry point:

book-to-skill ./my-book.pdf my-book-skill

This command invokes scripts/extract.py, which delegates to book_to_skill/cli.py to select the appropriate parser (such as docling for technical PDFs) and write the intermediate artefacts to a temporary directory.

Validating Generated Skills

Ensure compliance with host-specific standards before installation:

python -m tools.validate_skill ~/.copilot/skills/my-book-skill --lens copilot

The validator reads the generated SKILL.md and auxiliary files, verifying they obey the target platform's content rules and structural requirements.

Summary

• Book-to-Skill implements a two-half pipeline separating deterministic extraction from AI-driven generation.
• The extractor (book_to_skill/cli.py, book_to_skill/parsers/) produces sanitized full_text.txt and metadata.json from multiple input formats.
• Security is enforced at multiple layers through sanitize.py and tools/scan_generated_skill.py.
• The generator follows the SKILL.md specification to create on-demand skill structures with individual chapter files.
• Cost efficiency is measured by tools/discovery_tax.py, quantifying token savings over full-document context loading.

Frequently Asked Questions

What makes the Book-to-Skill extractor deterministic?

The extractor guarantees identical outputs for identical inputs through its pure Python implementation in book_to_skill/cli.py and format-specific parsers. It bypasses stochastic processes, producing immutable full_text.txt and metadata.json files that serve as reliable inputs for the generator phase. This determinism ensures reproducible builds and simplifies debugging across different environments.

How does the generator use SKILL.md to create skills?

The generator treats SKILL.md as an executable specification that defines a 10-step workflow for transforming extracted artefacts into a skill directory. The document also specifies the "fold-in" update mechanism for incremental refinement. By following this specification, the agent produces consistent output structures including chapter files, glossaries, and cheatsheets regardless of the input document's original format.

What security measures are implemented in the pipeline?

Security is enforced at extraction time by book_to_skill/sanitize.py, which removes zero-width Unicode characters and tag blocks that could conceal malicious instructions. Post-generation validation is performed by tools/scan_generated_skill.py, which audits final outputs for hidden injection patterns. Additionally, tools/validate_skill.py ensures generated content complies with host platform safety standards before installation.

Which file formats does Book-to-Skill support?

According to book_to_skill/config.py and the parsers directory, the extractor handles PDF, EPUB, DOCX, HTML, RTF, Calibre exports, and plain text files. The dependency probe in book_to_skill/dependencies.py detects optional external tools like pdftotext or docling, falling back to standard-library implementations when these are unavailable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}