What Is book-to-skill and How Does It Reduce Token Costs by 24×–51×?

The book-to-skill system converts books and documents into compact, structured "skills" that LLM agents can load on demand, cutting token costs by 24× to 51× compared to full-context approaches.

book-to-skill is an open-source tool from virgiliojr94/book-to-skill that solves a critical problem for AI agents: processing large books without blowing context windows or API budgets. Instead of feeding entire documents to an LLM repeatedly, this system front-loads extraction work and lets agents retrieve only the relevant sections. The architecture delivers dramatic cost savings—measured at 24× to 51× token reduction—while making book knowledge searchable and reusable.

How book-to-skill Works: The Two-Part Architecture

The system separates extraction from generation, creating a clean pipeline that runs expensive operations once and reaps savings forever after.

Part 1: Deterministic Extractor

The extractor is a pure-Python pipeline that ingests PDFs, EPUBs, DOCX, HTML, RTF, and other formats. Entry points are scripts/extract.py (thin wrapper) and book_to_skill/cli.py (core driver).

# Convert any supported document into a skill

$ book-to-skill my-book.pdf

# Or directly:

$ ./scripts/extract.py my-book.epub

The extractor produces two key outputs:

• full_text.txt — merged raw text of the entire document
• metadata.json — page counts, token counts, chapter boundaries, and table of contents

This step runs deterministically and once per book. Per the architecture documentation, the component diagram in docs/architecture.md shows how parsers feed into a unified extraction flow that handles multiple formats through modular plugins like book_to_skill/parsers/pdf.py.

Part 2: Spec-Driven Generator

After extraction, a generator follows the SKILL.md specification to create the final skill directory. The structure is designed for lazy loading:

Component	Typical Size	Purpose
SKILL.md	~4,000 tokens	Core skill manifest—loaded on every query
chapters/01.md, etc.	~1,000 tokens each	Individual chapters—loaded only when queried
glossary.md, patterns.md, cheatsheet.md	Variable	Reference materials—loaded on demand

As documented in docs/architecture.md, this design means agents never pay the cost of unread content.

The 24×–51× Token Cost Reduction Explained

The dramatic savings come from eliminating the "Discovery Loop Tax"—the waste of including irrelevant book content in every LLM call.

The Problem: Full-Book Context Bloat

Without book-to-skill, agents typically dump entire books into context. For a substantial technical book, this reaches ~120,000 tokens per query. Repeated across many questions, costs multiply rapidly.

The Solution: Lazy Chapter Loading

With book-to-skill, a typical query loads:

• Core SKILL.md: ~5,000 tokens
• Relevant single chapter: ~1,000 tokens
• Total: ~5,000–6,000 tokens

The 24×–51× reduction is measured in docs/performance.md, comparing full-context baselines against skill-based queries targeting specific chapters.

Up-Front Cost vs. Ongoing Savings

Extraction costs ~$1 per book (one-time). After that:

• Queries scale with chapter size, not book size
• Re-reading the same chapter adds zero re-extraction cost
• Additional books add parallel, non-compounding costs

# Verify savings with the included benchmark tool

$ python3 tools/discovery_tax.py \
    --full-text /tmp/book_skill_work/full_text.txt \
    --target-chapter 5

# Output: Full context ≈ 120,000 tokens vs. Skill approach ≈ 5,000 tokens

Using book-to-skill Programmatically

Beyond CLI usage, integrate directly into Python applications:

from book_to_skill.cli import main as book_to_skill

# Process a document programmatically

book_to_skill(["my-book.epub", "--output", "my-book-skill"])

# The skill directory is now ready for agent consumption

# ~/.copilot/skills/my-book-skill/SKILL.md

# ~/.copilot/skills/my-book-skill/chapters/

The book_to_skill/cli.py module handles argument parsing, format detection, and optional PDF inspector hooks for debugging extraction issues.

Key Source Files and Their Roles

File	Role
scripts/extract.py	Thin entry-point launching the extractor CLI
book_to_skill/cli.py	Core extraction pipeline driver
book_to_skill/parsers/pdf.py	PDF parsing implementation (one of multiple format modules)
SKILL.md	Specification defining skill output structure
docs/architecture.md	Component diagrams and design rationale
docs/performance.md	Benchmark data validating token savings
tools/discovery_tax.py	Utility for measuring Discovery Loop Tax

Summary

• book-to-skill converts books into structured, queryable skills through a two-stage extraction and generation pipeline
• Deterministic extraction runs once per book, producing metadata and segmented chapters
• Lazy loading ensures agents retrieve only ~5K tokens (core + relevant chapter) versus ~120K tokens for full-context approaches
• Measured savings of 24×–51× are documented in docs/performance.md with reproducible benchmarks
• ~$1 up-front cost per book enables cheap, linearly-scaling queries thereafter

Frequently Asked Questions

What document formats does book-to-skill support?

The extractor handles PDF, EPUB, DOCX, HTML, RTF, and other common formats through modular parsers in book_to_skill/parsers/. Each format has a dedicated parser module; pdf.py demonstrates the pattern. The CLI auto-detects format from file extension.

Where are the generated skills stored?

By default, skills land in ~/.copilot/skills/<book-name>/ with SKILL.md at the root and chapters/ subdirectory. Use --output to override. The structure follows the SKILL.md specification exactly, making skills portable across agent frameworks.

How does the 24×–51× reduction vary by book?

Savings depend on book length and query specificity. Longer books with many chapters yield higher multiples (toward 51×) when querying single chapters. Shorter books or queries spanning multiple chapters land closer to 24×. The discovery_tax.py tool calculates exact ratios for your document.

Can I modify extracted skills after generation?

Yes—skills are plain markdown and JSON. Edit SKILL.md to adjust metadata or regenerate chapters, then update metadata.json if chapter boundaries change. However, re-running book-to-skill on the source document overwrites manual changes; version control your modifications separately.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}