# How Book-to-Skill Reduces Token Usage by Up to 51× Compared to Context Dumping

> Discover how Book-to-Skill cuts token usage by 51x vs context dumping. Learn how offline extraction and on-demand skill loading minimize your context window.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: performance
- Published: 2026-08-29

---

**Book-to-Skill achieves massive token savings by splitting processing into an offline extraction phase and an on-demand skill-loading phase, retaining only a 4K-token core and relevant chapter instead of dumping entire books into the context window.**

Book-to-Skill, developed by virgiliojr94, is an open-source toolchain that transforms technical books into AI-optimized skill files. Unlike naive approaches that stream whole documents into language models, this system reduces token usage through a deterministic pipeline that front-loads heavy processing. The result is a per-query token cost of roughly 5,000 tokens regardless of whether the source material contains 100 or 1,000 pages.

## The Two-Phase Architecture for Token Efficiency

The dramatic reduction in token consumption stems from a strict separation between one-time preprocessing and runtime context management.

### Phase 1: Deterministic Document Extraction (Offline)

The extraction process runs once and consumes zero model tokens. The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) entry point processes PDF, EPUB, DOCX, and HTML sources to produce a clean [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) file and a compact [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) file. As documented in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md), this step uses deterministic parsers rather than language models, meaning no inference costs are incurred during this initial processing.

### Phase 2: Spec-Driven Skill Generation (On-Demand)

After extraction, the generator creates a tiny [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) core (approximately 4,000 tokens) plus modular chapter files (approximately 1,000 tokens each). At query time, the agent loads only the resident core and the specific chapter required to answer the user’s question. According to the design principles in [`docs/architecture.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/architecture.md), no full-book text ever enters the model’s context window, maintaining a steady ~5,000 token footprint per turn.

## Measured Token Savings: 24× to 51× Reduction

The [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) script provides reproducible benchmarks comparing three strategies. While a context-dump streams the entire book into the model every turn, Book-to-Skill maintains a constant cost:

- **Think Python 2**: 119,264 tokens (dump) versus ~5,000 tokens (skill) yields a **24× reduction**
- **Working Backwards**: 175,000 tokens versus ~5,000 tokens yields a **35× reduction**
- **AI Engineering** (large): 256,000 tokens versus ~5,000 tokens yields a **51× reduction**

These measurements come from [`docs/performance.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/performance.md) and are generated by the `main()` function in [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), which calculates exact token counts using the `tiktoken` cl100k_base encoder.

## Reproducing the Token Reduction Locally

You can verify these savings using the provided CLI tools and measurement scripts.

Extract the source document once:

```bash
python3 scripts/extract.py my_book.pdf

```

Generate the optimized skill structure:

```bash
book-to-skill generate --input book_skill_work/full_text.txt

```

Measure the exact token tax for a specific chapter:

```bash
python3 tools/discovery_tax.py \
    --full-text book_skill_work/full_text.txt \
    --skill-dir ~/.agents/skills/my_book \
    --target-chapter 5

```

The output compares four approaches: context-dump (full book), discovery-best (TOC plus raw chapter), discovery-loop (multiple chapters), and Book-to-Skill (core plus compiled chapter). Typical results show Book-to-Skill using ~5,000 tokens compared to 119,264 tokens for a full context dump of a technical book like *Think Python 2*.

## Summary

- **Two-phase processing** separates offline extraction from lightweight runtime loading, eliminating the need to process full texts repeatedly
- **Constant token footprint** of approximately 5,000 tokens per query regardless of original book size, compared to 100,000+ tokens for context dumping
- **Measured efficiency gains** range from 24× to 51× fewer tokens depending on source material length
- **Verification tooling** via [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) provides reproducible benchmarks against baseline strategies
- **Modular architecture** keeps a ~4K token [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) core resident while loading only relevant ~1K token chapter files on demand

## Frequently Asked Questions

### What is the exact token cost per query with Book-to-Skill?

Each query loads the resident core skill file plus one relevant chapter, totaling approximately 5,000 tokens. This remains constant whether the source book is 100 pages or 1,000 pages, unlike context dumping where costs scale linearly with document length.

### How does the discovery tax tool calculate token savings?

The [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) script uses the `tiktoken` cl100k_base encoder to count tokens in three scenarios: full context-dump, best-case discovery (TOC plus raw target chapter), and worst-case discovery (multiple chapters). It compares these against the Book-to-Skill baseline to generate reduction ratios like 24× or 51×.

### Why doesn't the extraction phase consume model tokens?

The extraction phase in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) relies on deterministic parsers for PDF, EPUB, and other formats to generate [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) and [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json). This is pure data transformation without LLM inference, meaning no tokens are consumed during this one-time preprocessing step handled by [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) and [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py).

### Can I apply Book-to-Skill to documents other than books?

Yes. While optimized for technical books, the toolchain processes any PDF, EPUB, DOCX, or HTML file. The multi-source resolution capabilities in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) extend the token reduction benefits to any large document requiring efficient querying.