What Are the Main Components of the book-to-skill Architecture?

The book-to-skill architecture consists of two distinct layers—a deterministic Python-based extractor that parses documents into clean text and metadata, and a spec-driven LLM agent that compiles that output into a loadable skill package.

The book-to-skill project transforms static documents (PDFs, EPUBs, DOCX, and more) into interactive, queryable skills for AI assistants like Copilot or Claude. Understanding the book-to-skill architecture reveals how the system separates deterministic text processing from generative AI workflows to ensure reliability, security, and cost efficiency.

Core Components of the book-to-skill Architecture

The repository is explicitly split into two halves that communicate through a well-defined interface: full_text.txt and metadata.json.

Deterministic Extractor (Python Layer)

The Deterministic Extractor is a fully Python-based pipeline that operates without LLM dependencies. Its sole responsibility is ingesting documents and emitting sanitized, structured data.

• scripts/extract.py – The entry-point shim that forwards arguments to the CLI.
• book_to_skill/cli.py – Implements command-line parsing, multi-source handling, and the main runner orchestration. This module coordinates chapter detection and table-of-contents extraction.
• book_to_skill/parsers/ – Contains one module per supported format (pdf, epub, docx, html, rtf, calibre, text). Each parser handles format-specific quirks and extracts raw content.
• book_to_skill/config.py – Declares supported extensions, default output locations, and dependency mappings required by the pipeline.
• book_to_skill/dependencies.py – Probes for optional external tools like pdftotext or ebook-convert and implements the --check report for environment validation.
• book_to_skill/sanitize.py – Strips invisible Unicode characters and tag-block artifacts to harden the extracted text against downstream parsing errors.

This layer produces two artifacts in a working directory (typically book_skill_work/): full_text.txt containing merged, source-marked content, and metadata.json storing page counts, word counts, token estimates, and chapter boundaries.

Spec-Driven Generator (Agent Layer)

The Spec-Driven Generator consumes the extractor's output and executes a sequence of LLM-driven steps defined in SKILL.md.

• SKILL.md – The master specification located in the repository root that drives the agent through steps 0-10, including content-type detection, cost estimation, per-chapter summarization, glossary creation, and final skill assembly.
• chapters/*.md – On-demand chapter files generated by the agent and loaded lazily only when specific queries target that content.
• glossary.md, patterns.md, cheatsheet.md – Supporting artifacts containing extracted terms, reusable techniques, and decision trees compiled from the source material.
• tools/validate_skill.py – Validates a generated skill package against host-specific rules (e.g., Copilot or Claude compatibility).
• tools/scan_generated_skill.py – Performs an advisory scan for prompt-injection vectors or unsafe content in the generated skill.

How the Extraction Pipeline Works

The deterministic layer follows a strict pipeline from raw document to sanitized output.

Input Parsing and Sanitization

When you run the extraction command, book_to_skill/cli.py dispatches to the appropriate parser in book_to_skill/parsers/ based on file extension. The parser extracts raw text, then passes it through book_to_skill/sanitize.py to remove zero-width characters, control sequences, and malformed tags that could disrupt LLM tokenization.

# From the repository root

python -m scripts.extract path/to/book.pdf

This command creates book_skill_work/full_text.txt and book_skill_work/metadata.json, with the latter containing structured metadata including detected chapter boundaries and table-of-contents data.

Metadata Generation

The book_to_skill/utils.py module handles chapter and ToC detection, calculating token counts using the tokenizer specified in book_to_skill/config.py. The metadata JSON includes page ranges, word counts, and content-type hints that inform the generator's cost estimation phase.

The Generation Workflow

Once extraction completes, the agent layer takes over to compile the skill package.

SKILL.md Specification

The SKILL.md file acts as the control program for the LLM agent. It defines a 10-step workflow: content classification, cost estimation, chunking strategy selection, per-chapter summarization, glossary extraction, pattern identification, and final assembly. According to the source code in docs/architecture.md, this specification implements the "compile-time over runtime" principle by doing heavy processing once rather than at query time.

On-Demand Chapter Loading

Rather than embedding all content into SKILL.md, the architecture stores individual chapters as separate files in chapters/*.md. The main specification file remains small (under typical context window limits), while individual chapter files load lazily when the agent detects a query relevant to that specific content. This design keeps the working memory footprint low and query costs predictable.

# Assuming extraction already produced the work dir

copilot skill generate \
  --input book_skill_work/full_text.txt \
  --metadata book_skill_work/metadata.json \
  --output ~/.copilot/skills/my-book-skill

Security and Validation Layer

The architecture includes explicit safeguards before deployment.

tools/validate_skill.py checks the generated skill against host-specific constraints:

python tools/validate_skill.py ~/.copilot/skills/my-book-skill/SKILL.md --lens copilot

tools/scan_generated_skill.py provides an additional security pass:

python tools/scan_generated_skill.py ~/.copilot/skills/my-book-skill

These utilities ensure that prompt-injection attempts, unsafe code blocks, or malformed directives do not propagate to end users.

Design Principles

As documented in docs/architecture.md, three principles shape the book-to-skill architecture:

• Extract structure, not summaries – The deterministic layer focuses on frameworks, decision rules, and anti-patterns rather than generating abstractions.
• Compile-time over runtime – Heavy lifting occurs during the extraction and generation phases; runtime queries only load necessary chapter slices.
• Graceful degradation – Each format parser includes fallback logic in book_to_skill/parsers/; a single corrupted source does not abort the entire pipeline.

Summary

• The Deterministic Extractor (scripts/extract.py, book_to_skill/cli.py, and format-specific parsers) handles document ingestion and sanitization without LLM dependencies.
• The Spec-Driven Generator uses SKILL.md to orchestrate LLM steps that create on-demand chapter files (chapters/*.md) and reference artifacts (glossary.md, patterns.md).
• Validation tools (tools/validate_skill.py, tools/scan_generated_skill.py) enforce security and host compatibility before deployment.
• The architecture separates extraction from generation to ensure deterministic, reproducible outputs while allowing flexible, cost-effective AI-driven compilation.

Frequently Asked Questions

What document formats does the deterministic extractor support?

According to book_to_skill/config.py, the extractor supports PDF, EPUB, DOCX, HTML, RTF, and plain text files. The system probes for optional external dependencies like pdftotext (Poppler) or ebook-convert (Calibre) via book_to_skill/dependencies.py to handle proprietary formats gracefully.

How does the architecture prevent prompt injection in generated skills?

The book_to_skill/sanitize.py module strips invisible Unicode and control characters during extraction, while tools/scan_generated_skill.py performs an advisory scan on the final output for known prompt-injection patterns and unsafe content blocks before the skill is loaded by downstream agents.

Why are chapters stored as separate files rather than in a single document?

The architecture follows an "on-demand chapters" design principle from docs/architecture.md. Keeping chapters in chapters/*.md allows SKILL.md to remain small and fit within context window limits, while individual chapters load only when queried, reducing token costs and latency during runtime usage.

Can the extraction pipeline run without installing optional external tools?

Yes. While book_to_skill/dependencies.py checks for optional tools like pdftotext or ebook-convert, each parser in book_to_skill/parsers/ includes fallback logic. Run python -m scripts.extract --check to see which formats your current environment supports without external dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}