# Design Principles Behind book-to-skill: A Two-Stage Pipeline for Converting Books to LLM Skills

> Discover the book-to-skill design principles: a two-stage pipeline for efficient book to LLM skill conversion. Learn about compile-time processing, on-demand loading, and token efficiency.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: architecture
- Published: 2026-09-01

---

**The key design principles behind [book-to-skill](https://github.com/virgiliojr94/book-to-skill) center on a two-stage architecture that separates deterministic text extraction from generative skill synthesis, prioritizing compile-time processing, on-demand chapter loading, and front-loaded metadata for optimal token efficiency.**

The [book-to-skill](https://github.com/virgiliojr94/book-to-skill) repository provides a complete system for transforming books into structured "skills" that LLM agents can query efficiently. These design principles are grounded in production constraints: minimizing token costs, ensuring reproducibility, and maintaining security when processing untrusted documents.

## Two-Stage Architecture: Extractor and Generator

The pipeline splits cleanly between **deterministic extraction** and **spec-driven generation**. This separation allows each stage to be optimized independently and makes the system more maintainable.

| Stage | Responsibility | Entry Point |
|-------|----------------|-------------|
| **Extractor** | Parse multiple document formats, sanitize content, detect structure | [[`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/cli.py) |
| **Generator** | Follow [[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md) workflow to produce structured skill files | Agent driven by specification |

The extraction flow works as follows:

```python

# Example: programmatic extraction with structure detection

from book_to_skill.utils import extract_single_file

text, metadata = extract_single_file(
    path="design_patterns.epub",
    output_dir="./workdir",
    force=True  # bypass cache

)

print(f"Detected {len(metadata['chapters'])} chapters")
print(f"Estimated tokens: {metadata['approx_tokens']}")

```

The extractor produces two artifacts consumed by the generator:

- [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) — sanitized, concatenated content
- [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) — structural information including chapters, headings, and token estimates

## Core Design Principles

### Extract Structure, Not Summaries

The extractor focuses on **frameworks, headings, and organizational patterns** rather than semantic summaries. This principle appears in the architecture documentation ([[`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md)):

> The process should extract frameworks and patterns from the source material rather than just summarizing content.

Raw extraction preserves the book's native organization. The generator later transforms this into **decision rules, patterns, and cheatsheets** — actionable structures that agents can apply, not reference.

### Compile-Time Over Runtime

All expensive operations happen once during extraction:

- Format conversion and parsing
- Unicode sanitization ([[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/sanitize.py))
- Chapter boundary detection
- Token cost estimation

At runtime, the agent loads only required chapter files. This principle directly reduces latency and API costs for end users.

### On-Demand Chapters

Each chapter resides in its own file under `chapters/`. The main [[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md) contains only:

- Overview and scope
- Target audiences
- Command reference table
- Chapter index

When a user query requires specific content, the agent loads exactly one chapter file. This design principle prevents token waste from loading irrelevant material.

### Front-Loaded SKILL.md

Critical information appears at the **beginning of [[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md)**, ensuring it survives truncation:

```

SKILL.md structure (truncation-safe ordering):
├── Overview (essential context)
├── Scope boundaries (what the skill covers/doesn't cover)
├── Target audiences (who should use this)
├── Table: Commands/Patterns (quick reference)
├── Chapter index (file references only)
└── Extended examples (least critical, most truncatable)

```

### Graceful Degradation

Every format includes a pure-Python fallback. If a binary dependency fails — `pdftotext`, `ebook-convert`, `pandoc` — the pipeline attempts alternative parsers. Multi-source documents continue processing even when individual sources fail.

## Security-by-Design

Security considerations are embedded throughout the extractor:

| Layer | Implementation | File |
|-------|----------------|------|
| **Input sanitization** | Strip zero-width characters, Unicode tags, bidirectional overrides | [[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/sanitize.py) |
| **XML hardening** | Disable external entities in DOCX parsing | [[`book_to_skill/parsers/docx_parser.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/docx_parser.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/parsers/docx_parser.py) |
| **Subprocess isolation** | Absolute paths only, no shell interpolation | [[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/utils.py) |
| **Output validation** | Scan generated skills for prompt injection patterns | [[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/tools/scan_generated_skill.py) |

The architecture document ([[`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md)) dedicates a full section to security, emphasizing that untrusted documents must be treated as potentially malicious.

## Extensibility Patterns

### Adding a New Document Format

1. Create `book_to_skill/parsers/<format>.py` with a `parse()` function
2. Register the extension in [[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/config.py)
3. Add dependency metadata to [[`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/dependencies.py)
4. Update utility registration in [[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/utils.py)

All parsers follow a consistent interface defined in [[`book_to_skill/parsers/__init__.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/__init__.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/parsers/__init__.py).

### Modifying Generation Behavior

The agent workflow is controlled entirely by [[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md). To change how skills are structured:

- Edit the specification steps directly
- Maintain evidence-backed constraints per [[`CONTRIBUTING.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/CONTRIBUTING.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/CONTRIBUTING.md)
- Validate outputs with [[`tools/validate_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/validate_skill.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/tools/validate_skill.py)

## Validation and Quality Assurance

The repository includes dedicated tools for verifying generated skills:

```bash

# Validate against a specific language model's constraints

python -m tools.validate_skill ./my_skill --lens hermes

# Scan for potential prompt injection vulnerabilities

python -m tools.scan_generated_skill ./my_skill --strict

```

These tools implement the principle that **generated artifacts must be auditable** before deployment.

## Summary

- **Two-stage separation** enables independent optimization of extraction and generation
- **Structure-first extraction** preserves organizational patterns over raw content
- **Compile-time processing** minimizes runtime costs through front-loaded computation
- **On-demand chapter loading** reduces token consumption per query
- **Front-loaded metadata** ensures critical context survives truncation
- **Defense in depth** protects against malicious inputs at every pipeline stage

## Frequently Asked Questions

### How does book-to-skill handle unsupported document formats?

When encountering an unknown extension, the pipeline falls back to **plain text detection** using charset detection libraries. If that fails, the file is skipped with a logged warning, allowing multi-source documents to continue processing. Users can extend support by implementing the parser interface in [`book_to_skill/parsers/`](https://github.com/virgiliojr94/book-to-skill/tree/master/book_to_skill/parsers).

### What makes the SKILL.md specification different from a typical Prompt text file?

[[`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md) functions as an **executable specification** rather than static instructions. It contains structured metadata, explicit scope boundaries, and file paths that drive agent behavior. Most importantly, it separates concise reference material from extended examples, enabling **intelligent truncation** when token limits are encountered.

### Why separate extraction and generation into distinct stages?

Separating deterministic extraction from probabilistic generation provides **three advantages**: (1) extraction can be cached and rerun cheaply, (2) generation can be tuned without re-parsing source documents, and (3) the deterministic stage can be audited for security while the generative stage focuses on quality. This architecture is explicitly documented in [[`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md)](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md).

### How does the system prevent prompt injection from malicious source documents?

Multiple layers defend against injection: **input sanitization** strips invisible Unicode characters, **XML hardening** prevents entity expansion attacks in DOCX files, **subprocess isolation** uses absolute paths, and **output scanning** with [[`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py)](https://github.com/virgiliojr94/book-to-skill/blob/master/tools/scan_generated_skill.py) detects suspicious patterns in generated skills.