How book-to-skill's Architecture Differs from Traditional RAG Pipelines

Book-to-skill replaces the classic retrieve-then-generate loop with a deterministic extraction stage followed by a spec-driven generation stage, producing reusable, version-controlled skills instead of per-query responses.

Traditional retrieval-augmented generation (RAG) pipelines dominate modern AI applications, but they carry inherent costs: query-time latency, repeated token expenditure, and opaque vector indexes. The virgiliojr94/book-to-skill repository reimagines this workflow entirely. Instead of fetching and synthesizing on every request, book-to-skill front-loads the heavy work into a one-time extraction phase, then uses a declarative specification to generate structured, reusable knowledge artifacts. This article examines exactly how book-to-skill's architecture differs from traditional RAG pipelines, with direct references to the source code implementation.

Core Architectural Split: Extractor vs. Generator

The fundamental departure from RAG appears in the pipeline's bisected design, documented in docs/architecture.md. Where RAG couples retrieval and generation into a single runtime operation, book-to-skill isolates them into sequential, independent stages.

Stage 1: Deterministic Extractor

The extractor is pure Python—no LLM calls, no non-determinism. Running python -m book_to_skill extract invokes scripts/extract.py, which delegates to book_to_skill/cli.py for orchestration:

• Parser selection happens via extension mapping in book_to_skill/config.py (PDF → parsers/pdf.py, EPUB → parsers/epub.py, DOCX → parsers/docx.py, etc.)
• Sanitization runs through book_to_skill/sanitize.py, stripping invisible Unicode and blocking XML entity attacks
• Output lands as full_text.txt plus metadata.json in a temporary work directory

This stage executes once per source document. The result is a canonical, cleaned representation stored as plain files—not vectors, not embeddings, not opaque indexes.

Stage 2: Spec-Driven Generator

The generator is an agentic process governed by SKILL.md, a declarative specification defining steps 1-9:

1. Content-type detection
2. Cost estimation
3. Structural analysis
4. Depth selection
5. Per-chapter summaries
6. Skill assembly

Running python -m book_to_skill generate --skill-dir ./my_skill produces:

• SKILL.md — the core skill specification
• chapters/*.md — on-demand chapter files
• glossary.md, patterns.md, cheatsheet.md — supplemental artifacts

Crucially, this generated skill is human-readable, version-controllable, and reusable. Subsequent queries load only the relevant chapter files without re-running extraction or regeneration.

How book-to-skill's Architecture Differs from Traditional RAG: A Direct Comparison

Dimension	Traditional RAG	book-to-skill
Pipeline structure	Monolithic retrieve-then-generate loop	Two-stage: deterministic extractor → spec-driven generator
When heavy work happens	Query time (every request)	Extraction: offline, once; Generation: once per skill version
Query-time operations	Vector search + full LLM synthesis	File system load of pre-generated markdown
Output format	Ephemeral text response	Reusable SKILL.md + structured markdown files
Update mechanism	Re-index entire corpus	Commit new markdown file to version control
Token efficiency	Entire retrieved passages in prompt	Compact prompts via front-loaded SKILL.md + selective chapter loading
Extensibility	New retrievers or prompt templates	New parser modules + declarative SKILL.md edits

Token Efficiency Through Front-Loaded Design

Traditional RAG suffers from prompt bloat: retrieved passages consume tokens on every query. Book-to-skill inverts this through intentional information architecture, as detailed in the design principles of docs/architecture.md.

The SKILL.md file stores the most important content first—core concepts, structural overview, usage patterns. When an agent loads a skill, it reads SKILL.md entirely (small, fixed cost) then selectively loads only required chapters from chapters/*.md. This keeps working prompts compact and predictable in cost.

from pathlib import Path

skill_path = Path.home() / ".agents" / "skills" / "my_skill"

# Agent imports skill files directly—no re-extraction, no retrieval API calls

core_spec = (skill_path / "SKILL.md").read_text()
chapter_3 = (skill_path / "chapters" / "03_advanced_patterns.md").read_text()

Security: Layered Sanitization vs. Generic Defenses

RAG pipelines typically rely on post-retrieval text sanitization—often insufficient against creative injection attacks. Book-to-skill implements defense in depth across the pipeline:

Layer	File	Protection
Input sanitization	book_to_skill/sanitize.py	Strips invisible Unicode, normalizes whitespace
Format-specific hardening	parsers/docx.py	Blocks XML entity expansion attacks
Output auditing	tools/scan_generated_skill.py	Scans for prompt injection patterns and hidden instructions
Validation	tools/validate_skill.py	Enforces host-specific constraints (size limits, prohibited constructs)

Running python -m tools.validate_skill ./my_skill --lens claude ensures deployment readiness against target platform rules.

Extensibility Without Monolithic Growth

Extending traditional RAG means adding retrievers, retraining embeddings, or prompt engineering. Book-to-skill offers two clean extension vectors:

New source formats — Create a parser module in book_to_skill/parsers/, register its extension in config.py, and optionally declare dependencies in dependencies.py for probing support.

Modified generation behavior — Edit the declarative steps in SKILL.md. No code changes required. The agent follows the specification literally, making behavior transparent and auditable.

This component map appears in book_to_skill/parsers/ (PDF, EPUB, DOCX, HTML, RTF handlers) and book_to_skill/dependencies.py (optional-dependency resolution logic).

Practical Workflow: From Book to Reusable Skill

# 1. Extract once (deterministic, no LLM)

python -m book_to_skill extract path/to/my_book.pdf

# 2. Generate skill (spec-driven agent)

python -m book_to_skill generate --skill-dir ./my_skill

# 3. Validate for target platform

python -m tools.validate_skill ./my_skill --lens claude

# 4. Scan for security issues

python -m tools.scan_generated_skill ./my_skill

# 5. Deploy to agent runtime—subsequent queries load files, not APIs

The extraction stage in scripts/extract.py → book_to_skill/cli.py → parser-specific modules produces intermediate artifacts. The generation stage consumes these to produce the final skill structure. Neither stage repeats unless the source changes.

Summary

• Two-stage architecture separates deterministic extraction from spec-driven generation, unlike RAG's fused retrieve-generate loop
• Offline extraction happens once per source; query-time operations reduce to file system reads
• Front-loaded SKILL.md design minimizes token costs through selective chapter loading
• Human-readable, version-controlled outputs replace opaque vector indexes
• Layered security spans input sanitization, format hardening, and output auditing across sanitize.py, parsers/docx.py, and tools/scan_generated_skill.py
• Declarative extensibility via SKILL.md edits and modular parser registration in config.py

Frequently Asked Questions

What makes book-to-skill's extraction "deterministic"?

The extractor contains zero LLM calls—only Python code in book_to_skill/parsers/* and sanitize.py. Given the same input file, it produces identical full_text.txt and metadata.json every time. This predictability enables caching, versioning, and reproducible builds impossible with stochastic retrieval systems.

How does the SKILL.md specification control generation?

SKILL.md is a declarative document listing numbered steps (1-9) that the generator agent follows literally. Steps include content-type detection, cost estimation, structural analysis, depth selection, per-chapter summarization, and final assembly. Editing these steps changes agent behavior without touching Python code—making the generation process transparent and auditable.

Can I use book-to-skill outputs with existing RAG systems?

Yes. The generated skill files are plain markdown. A RAG system could index chapters/*.md as documents, though this sacrifices book-to-skill's efficiency advantages. Better integration: load SKILL.md as system context and retrieve relevant chapters as needed, bypassing vector search entirely.

What file types does the extractor support?

According to book_to_skill/parsers/ and config.py: PDF, EPUB, DOCX, HTML, and RTF. New formats require implementing a parser class and registering its extension mapping. The dependencies.py module probes for optional dependencies (like pypdf, ebooklib) and provides graceful degradation when unavailable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}