# What Is book-to-skill and How Does It Reduce Token Costs by 24×–51×?

> Discover book-to-skill, a system that transforms documents into LLM-loadable skills. Slash token costs by up to 51×, making LLMs more efficient and affordable.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-09-01

---

**The book-to-skill system converts books and documents into compact, structured "skills" that LLM agents can load on demand, cutting token costs by 24× to 51× compared to full-context approaches.**

book-to-skill is an open-source tool from `virgiliojr94/book-to-skill` that solves a critical problem for AI agents: processing large books without blowing context windows or API budgets. Instead of feeding entire documents to an LLM repeatedly, this system front-loads extraction work and lets agents retrieve only the relevant sections. The architecture delivers dramatic cost savings—measured at 24× to 51× token reduction—while making book knowledge searchable and reusable.

## How book-to-skill Works: The Two-Part Architecture

The system separates **extraction** from **generation**, creating a clean pipeline that runs expensive operations once and reaps savings forever after.

### Part 1: Deterministic Extractor

The extractor is a pure-Python pipeline that ingests PDFs, EPUBs, DOCX, HTML, RTF, and other formats. Entry points are [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) (thin wrapper) and [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) (core driver).

```bash

# Convert any supported document into a skill

$ book-to-skill my-book.pdf

# Or directly:

$ ./scripts/extract.py my-book.epub

```

The extractor produces two key outputs:

- [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) — merged raw text of the entire document
- [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) — page counts, token counts, chapter boundaries, and table of contents

This step runs **deterministically and once per book**. Per the architecture documentation, the component diagram in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md) shows how parsers feed into a unified extraction flow that handles multiple formats through modular plugins like [`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py).

### Part 2: Spec-Driven Generator

After extraction, a generator follows the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification to create the final skill directory. The structure is designed for **lazy loading**:

| Component | Typical Size | Purpose |
|-----------|-----------|---------|
| [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) | ~4,000 tokens | Core skill manifest—loaded on every query |
| [`chapters/01.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/chapters/01.md), etc. | ~1,000 tokens each | Individual chapters—loaded only when queried |
| [`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md), [`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md), [`cheatsheet.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/cheatsheet.md) | Variable | Reference materials—loaded on demand |

As documented in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md), this design means agents never pay the cost of unread content.

## The 24×–51× Token Cost Reduction Explained

The dramatic savings come from eliminating the "Discovery Loop Tax"—the waste of including irrelevant book content in every LLM call.

### The Problem: Full-Book Context Bloat

Without book-to-skill, agents typically dump entire books into context. For a substantial technical book, this reaches ~120,000 tokens per query. Repeated across many questions, costs multiply rapidly.

### The Solution: Lazy Chapter Loading

With book-to-skill, a typical query loads:

- Core [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md): ~5,000 tokens
- Relevant single chapter: ~1,000 tokens
- **Total: ~5,000–6,000 tokens**

The **24×–51× reduction** is measured in [`docs/performance.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/performance.md), comparing full-context baselines against skill-based queries targeting specific chapters.

### Up-Front Cost vs. Ongoing Savings

Extraction costs ~$1 per book (one-time). After that:

- Queries scale with **chapter size**, not **book size**
- Re-reading the same chapter adds zero re-extraction cost
- Additional books add parallel, non-compounding costs

```bash

# Verify savings with the included benchmark tool

$ python3 tools/discovery_tax.py \
    --full-text /tmp/book_skill_work/full_text.txt \
    --target-chapter 5

# Output: Full context ≈ 120,000 tokens vs. Skill approach ≈ 5,000 tokens

```

## Using book-to-skill Programmatically

Beyond CLI usage, integrate directly into Python applications:

```python
from book_to_skill.cli import main as book_to_skill

# Process a document programmatically

book_to_skill(["my-book.epub", "--output", "my-book-skill"])

# The skill directory is now ready for agent consumption

# ~/.copilot/skills/my-book-skill/SKILL.md

# ~/.copilot/skills/my-book-skill/chapters/

```

The [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) module handles argument parsing, format detection, and optional PDF inspector hooks for debugging extraction issues.

## Key Source Files and Their Roles

| File | Role |
|------|------|
| [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) | Thin entry-point launching the extractor CLI |
| [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) | Core extraction pipeline driver |
| [`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py) | PDF parsing implementation (one of multiple format modules) |
| [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) | Specification defining skill output structure |
| [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md) | Component diagrams and design rationale |
| [`docs/performance.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/performance.md) | Benchmark data validating token savings |
| [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) | Utility for measuring Discovery Loop Tax |

## Summary

- **book-to-skill** converts books into structured, queryable skills through a two-stage extraction and generation pipeline
- **Deterministic extraction** runs once per book, producing metadata and segmented chapters
- **Lazy loading** ensures agents retrieve only ~5K tokens (core + relevant chapter) versus ~120K tokens for full-context approaches
- **Measured savings** of 24×–51× are documented in [`docs/performance.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/performance.md) with reproducible benchmarks
- **~$1 up-front cost** per book enables cheap, linearly-scaling queries thereafter

## Frequently Asked Questions

### What document formats does book-to-skill support?

The extractor handles PDF, EPUB, DOCX, HTML, RTF, and other common formats through modular parsers in `book_to_skill/parsers/`. Each format has a dedicated parser module; [`pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/pdf.py) demonstrates the pattern. The CLI auto-detects format from file extension.

### Where are the generated skills stored?

By default, skills land in `~/.copilot/skills/<book-name>/` with [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) at the root and `chapters/` subdirectory. Use `--output` to override. The structure follows the [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) specification exactly, making skills portable across agent frameworks.

### How does the 24×–51× reduction vary by book?

Savings depend on book length and query specificity. Longer books with many chapters yield higher multiples (toward 51×) when querying single chapters. Shorter books or queries spanning multiple chapters land closer to 24×. The [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) tool calculates exact ratios for your document.

### Can I modify extracted skills after generation?

Yes—skills are plain markdown and JSON. Edit [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) to adjust metadata or regenerate chapters, then update [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) if chapter boundaries change. However, re-running `book-to-skill` on the source document overwrites manual changes; version control your modifications separately.