Design Principles Behind book-to-skill: A Two-Stage Pipeline for Converting Books to LLM Skills

The key design principles behind book-to-skill center on a two-stage architecture that separates deterministic text extraction from generative skill synthesis, prioritizing compile-time processing, on-demand chapter loading, and front-loaded metadata for optimal token efficiency.

The book-to-skill repository provides a complete system for transforming books into structured "skills" that LLM agents can query efficiently. These design principles are grounded in production constraints: minimizing token costs, ensuring reproducibility, and maintaining security when processing untrusted documents.

Two-Stage Architecture: Extractor and Generator

The pipeline splits cleanly between deterministic extraction and spec-driven generation. This separation allows each stage to be optimized independently and makes the system more maintainable.

Stage	Responsibility	Entry Point
Extractor	Parse multiple document formats, sanitize content, detect structure	[book_to_skill/cli.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/cli.py)
Generator	Follow [SKILL.md](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md) workflow to produce structured skill files	Agent driven by specification

The extraction flow works as follows:

# Example: programmatic extraction with structure detection

from book_to_skill.utils import extract_single_file

text, metadata = extract_single_file(
    path="design_patterns.epub",
    output_dir="./workdir",
    force=True  # bypass cache

)

print(f"Detected {len(metadata['chapters'])} chapters")
print(f"Estimated tokens: {metadata['approx_tokens']}")

The extractor produces two artifacts consumed by the generator:

• full_text.txt — sanitized, concatenated content
• metadata.json — structural information including chapters, headings, and token estimates

Core Design Principles

Extract Structure, Not Summaries

The extractor focuses on frameworks, headings, and organizational patterns rather than semantic summaries. This principle appears in the architecture documentation ([docs/architecture.md](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md)):

The process should extract frameworks and patterns from the source material rather than just summarizing content.

Raw extraction preserves the book's native organization. The generator later transforms this into decision rules, patterns, and cheatsheets — actionable structures that agents can apply, not reference.

Compile-Time Over Runtime

All expensive operations happen once during extraction:

• Format conversion and parsing
• Unicode sanitization ([book_to_skill/sanitize.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/sanitize.py))
• Chapter boundary detection
• Token cost estimation

At runtime, the agent loads only required chapter files. This principle directly reduces latency and API costs for end users.

On-Demand Chapters

Each chapter resides in its own file under chapters/. The main [SKILL.md](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md) contains only:

• Overview and scope
• Target audiences
• Command reference table
• Chapter index

When a user query requires specific content, the agent loads exactly one chapter file. This design principle prevents token waste from loading irrelevant material.

Front-Loaded SKILL.md

Critical information appears at the beginning of [SKILL.md](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md), ensuring it survives truncation:

SKILL.md structure (truncation-safe ordering):
├── Overview (essential context)
├── Scope boundaries (what the skill covers/doesn't cover)
├── Target audiences (who should use this)
├── Table: Commands/Patterns (quick reference)
├── Chapter index (file references only)
└── Extended examples (least critical, most truncatable)

Graceful Degradation

Every format includes a pure-Python fallback. If a binary dependency fails — pdftotext, ebook-convert, pandoc — the pipeline attempts alternative parsers. Multi-source documents continue processing even when individual sources fail.

Security-by-Design

Security considerations are embedded throughout the extractor:

Layer	Implementation	File
Input sanitization	Strip zero-width characters, Unicode tags, bidirectional overrides	[book_to_skill/sanitize.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/sanitize.py)
XML hardening	Disable external entities in DOCX parsing	[book_to_skill/parsers/docx_parser.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/parsers/docx_parser.py)
Subprocess isolation	Absolute paths only, no shell interpolation	[book_to_skill/utils.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/utils.py)
Output validation	Scan generated skills for prompt injection patterns	[tools/scan_generated_skill.py](https://github.com/virgiliojr94/book-to-skill/blob/master/tools/scan_generated_skill.py)

The architecture document ([docs/architecture.md](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md)) dedicates a full section to security, emphasizing that untrusted documents must be treated as potentially malicious.

Extensibility Patterns

Adding a New Document Format

1. Create book_to_skill/parsers/<format>.py with a parse() function
2. Register the extension in [book_to_skill/config.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/config.py)
3. Add dependency metadata to [book_to_skill/dependencies.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/dependencies.py)
4. Update utility registration in [book_to_skill/utils.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/utils.py)

All parsers follow a consistent interface defined in [book_to_skill/parsers/__init__.py](https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/parsers/__init__.py).

Modifying Generation Behavior

The agent workflow is controlled entirely by [SKILL.md](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md). To change how skills are structured:

• Edit the specification steps directly
• Maintain evidence-backed constraints per [CONTRIBUTING.md](https://github.com/virgiliojr94/book-to-skill/blob/master/CONTRIBUTING.md)
• Validate outputs with [tools/validate_skill.py](https://github.com/virgiliojr94/book-to-skill/blob/master/tools/validate_skill.py)

Validation and Quality Assurance

The repository includes dedicated tools for verifying generated skills:

# Validate against a specific language model's constraints

python -m tools.validate_skill ./my_skill --lens hermes

# Scan for potential prompt injection vulnerabilities

python -m tools.scan_generated_skill ./my_skill --strict

These tools implement the principle that generated artifacts must be auditable before deployment.

Summary

• Two-stage separation enables independent optimization of extraction and generation
• Structure-first extraction preserves organizational patterns over raw content
• Compile-time processing minimizes runtime costs through front-loaded computation
• On-demand chapter loading reduces token consumption per query
• Front-loaded metadata ensures critical context survives truncation
• Defense in depth protects against malicious inputs at every pipeline stage

Frequently Asked Questions

How does book-to-skill handle unsupported document formats?

When encountering an unknown extension, the pipeline falls back to plain text detection using charset detection libraries. If that fails, the file is skipped with a logged warning, allowing multi-source documents to continue processing. Users can extend support by implementing the parser interface in book_to_skill/parsers/.

What makes the SKILL.md specification different from a typical Prompt text file?

[SKILL.md](https://github.com/virgiliojr94/book-to-skill/blob/master/SKILL.md) functions as an executable specification rather than static instructions. It contains structured metadata, explicit scope boundaries, and file paths that drive agent behavior. Most importantly, it separates concise reference material from extended examples, enabling intelligent truncation when token limits are encountered.

Why separate extraction and generation into distinct stages?

Separating deterministic extraction from probabilistic generation provides three advantages: (1) extraction can be cached and rerun cheaply, (2) generation can be tuned without re-parsing source documents, and (3) the deterministic stage can be audited for security while the generative stage focuses on quality. This architecture is explicitly documented in [docs/architecture.md](https://github.com/virgiliojr94/book-to-skill/blob/master/docs/architecture.md).

How does the system prevent prompt injection from malicious source documents?

Multiple layers defend against injection: input sanitization strips invisible Unicode characters, XML hardening prevents entity expansion attacks in DOCX files, subprocess isolation uses absolute paths, and output scanning with [tools/scan_generated_skill.py](https://github.com/virgiliojr94/book-to-skill/blob/master/tools/scan_generated_skill.py) detects suspicious patterns in generated skills.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}