# How book-to-skill's Architecture Differs from Traditional RAG Pipelines

> Discover how book-to-skill's architecture revolutionizes RAG by replacing retrieve-generate with deterministic extraction and spec-driven generation for reusable skills.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: architecture
- Published: 2026-08-25

---

**Book-to-skill replaces the classic retrieve-then-generate loop with a deterministic extraction stage followed by a spec-driven generation stage, producing reusable, version-controlled skills instead of per-query responses.**

Traditional **retrieval-augmented generation (RAG)** pipelines dominate modern AI applications, but they carry inherent costs: query-time latency, repeated token expenditure, and opaque vector indexes. The [virgiliojr94/book-to-skill](https://github.com/virgiliojr94/book-to-skill) repository reimagines this workflow entirely. Instead of fetching and synthesizing on every request, book-to-skill front-loads the heavy work into a **one-time extraction phase**, then uses a **declarative specification** to generate structured, reusable knowledge artifacts. This article examines exactly how book-to-skill's architecture differs from traditional RAG pipelines, with direct references to the source code implementation.

## Core Architectural Split: Extractor vs. Generator

The fundamental departure from RAG appears in the pipeline's bisected design, documented in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md). Where RAG couples retrieval and generation into a single runtime operation, book-to-skill isolates them into sequential, independent stages.

### Stage 1: Deterministic Extractor

The **extractor** is pure Python—no LLM calls, no non-determinism. Running `python -m book_to_skill extract` invokes [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), which delegates to [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) for orchestration:

- **Parser selection** happens via extension mapping in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) (PDF → [`parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/pdf.py), EPUB → [`parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/epub.py), DOCX → [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py), etc.)
- **Sanitization** runs through [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), stripping invisible Unicode and blocking XML entity attacks
- **Output** lands as [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) plus [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) in a temporary work directory

This stage executes **once per source document**. The result is a canonical, cleaned representation stored as plain files—not vectors, not embeddings, not opaque indexes.

### Stage 2: Spec-Driven Generator

The **generator** is an agentic process governed by [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md), a declarative specification defining steps 1-9:

1. Content-type detection
2. Cost estimation
3. Structural analysis
4. Depth selection
5. Per-chapter summaries
6. Skill assembly

Running `python -m book_to_skill generate --skill-dir ./my_skill` produces:
- [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) — the core skill specification
- `chapters/*.md` — on-demand chapter files
- [`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md), [`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md), [`cheatsheet.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/cheatsheet.md) — supplemental artifacts

Crucially, this generated skill is **human-readable, version-controllable, and reusable**. Subsequent queries load only the relevant chapter files without re-running extraction or regeneration.

## How book-to-skill's Architecture Differs from Traditional RAG: A Direct Comparison

| Dimension | Traditional RAG | book-to-skill |
|-----------|----------------|---------------|
| **Pipeline structure** | Monolithic retrieve-then-generate loop | Two-stage: deterministic extractor → spec-driven generator |
| **When heavy work happens** | Query time (every request) | Extraction: offline, once; Generation: once per skill version |
| **Query-time operations** | Vector search + full LLM synthesis | File system load of pre-generated markdown |
| **Output format** | Ephemeral text response | Reusable [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) + structured markdown files |
| **Update mechanism** | Re-index entire corpus | Commit new markdown file to version control |
| **Token efficiency** | Entire retrieved passages in prompt | Compact prompts via front-loaded [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) + selective chapter loading |
| **Extensibility** | New retrievers or prompt templates | New parser modules + declarative [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) edits |

## Token Efficiency Through Front-Loaded Design

Traditional RAG suffers from **prompt bloat**: retrieved passages consume tokens on every query. Book-to-skill inverts this through intentional information architecture, as detailed in the design principles of [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md).

The [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) file stores the **most important content first**—core concepts, structural overview, usage patterns. When an agent loads a skill, it reads [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) entirely (small, fixed cost) then **selectively loads only required chapters** from `chapters/*.md`. This keeps working prompts compact and predictable in cost.

```python
from pathlib import Path

skill_path = Path.home() / ".agents" / "skills" / "my_skill"

# Agent imports skill files directly—no re-extraction, no retrieval API calls

core_spec = (skill_path / "SKILL.md").read_text()
chapter_3 = (skill_path / "chapters" / "03_advanced_patterns.md").read_text()

```

## Security: Layered Sanitization vs. Generic Defenses

RAG pipelines typically rely on post-retrieval text sanitization—often insufficient against creative injection attacks. Book-to-skill implements **defense in depth** across the pipeline:

| Layer | File | Protection |
|-------|------|------------|
| Input sanitization | [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) | Strips invisible Unicode, normalizes whitespace |
| Format-specific hardening | [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py) | Blocks XML entity expansion attacks |
| Output auditing | [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) | Scans for prompt injection patterns and hidden instructions |
| Validation | [`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py) | Enforces host-specific constraints (size limits, prohibited constructs) |

Running `python -m tools.validate_skill ./my_skill --lens claude` ensures deployment readiness against target platform rules.

## Extensibility Without Monolithic Growth

Extending traditional RAG means adding retrievers, retraining embeddings, or prompt engineering. Book-to-skill offers two clean extension vectors:

**New source formats** — Create a parser module in `book_to_skill/parsers/`, register its extension in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py), and optionally declare dependencies in [`dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/dependencies.py) for probing support.

**Modified generation behavior** — Edit the declarative steps in [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md). No code changes required. The agent follows the specification literally, making behavior transparent and auditable.

This component map appears in `book_to_skill/parsers/` (PDF, EPUB, DOCX, HTML, RTF handlers) and [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) (optional-dependency resolution logic).

## Practical Workflow: From Book to Reusable Skill

```bash

# 1. Extract once (deterministic, no LLM)

python -m book_to_skill extract path/to/my_book.pdf

# 2. Generate skill (spec-driven agent)

python -m book_to_skill generate --skill-dir ./my_skill

# 3. Validate for target platform

python -m tools.validate_skill ./my_skill --lens claude

# 4. Scan for security issues

python -m tools.scan_generated_skill ./my_skill

# 5. Deploy to agent runtime—subsequent queries load files, not APIs

```

The extraction stage in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) → [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) → parser-specific modules produces intermediate artifacts. The generation stage consumes these to produce the final skill structure. Neither stage repeats unless the source changes.

## Summary

- **Two-stage architecture** separates deterministic extraction from spec-driven generation, unlike RAG's fused retrieve-generate loop
- **Offline extraction** happens once per source; query-time operations reduce to file system reads
- **Front-loaded [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)** design minimizes token costs through selective chapter loading
- **Human-readable, version-controlled outputs** replace opaque vector indexes
- **Layered security** spans input sanitization, format hardening, and output auditing across [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py), [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py), and [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)
- **Declarative extensibility** via [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) edits and modular parser registration in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py)

## Frequently Asked Questions

### What makes book-to-skill's extraction "deterministic"?

The extractor contains zero LLM calls—only Python code in `book_to_skill/parsers/*` and [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py). Given the same input file, it produces identical [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) every time. This predictability enables caching, versioning, and reproducible builds impossible with stochastic retrieval systems.

### How does the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification control generation?

[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) is a declarative document listing numbered steps (1-9) that the generator agent follows literally. Steps include content-type detection, cost estimation, structural analysis, depth selection, per-chapter summarization, and final assembly. Editing these steps changes agent behavior without touching Python code—making the generation process transparent and auditable.

### Can I use book-to-skill outputs with existing RAG systems?

Yes. The generated skill files are plain markdown. A RAG system could index `chapters/*.md` as documents, though this sacrifices book-to-skill's efficiency advantages. Better integration: load [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) as system context and retrieve relevant chapters as needed, bypassing vector search entirely.

### What file types does the extractor support?

According to `book_to_skill/parsers/` and [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py): PDF, EPUB, DOCX, HTML, and RTF. New formats require implementing a parser class and registering its extension mapping. The [`dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/dependencies.py) module probes for optional dependencies (like `pypdf`, `ebooklib`) and provides graceful degradation when unavailable.