# Performance Benefits of book-to-skill vs Context Dumping: 24×-51× Token Reduction Explained

> Discover book-to-skill benefits: achieve 24x-51x token reduction vs context dumping by converting books into efficient, queryable skills for massive LLM cost savings.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: performance
- Published: 2026-08-27

---

**Book-to-skill reduces recurring LLM token costs by up to 51× compared to context dumping by converting entire books into compact, queryable skills that load only ~5,000 tokens per interaction instead of 100,000-300,000 tokens.**

The `book-to-skill` project transforms static books into dynamic, reusable AI skills. Unlike naive context dumping—which resends the full text on every query—this approach pre-processes content into chunked, summarized chapters that minimize token consumption and latency.

## What Makes book-to-skill Faster

Three architectural decisions drive the performance gains:

- **Chunked, pre-summarized chapters**: The extraction pipeline in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) merges source files into [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt) plus metadata, then generates ~800-1,200 token chapter summaries stored as separate markdown files.

- **Discovery loop architecture**: Each query loads only the resident model core plus one relevant chapter (~5,000 tokens total), regardless of original book size.

- **Selective content rendering**: Technical books use Docling for tables and code blocks; prose-only books use fast `pdftotext` extraction.

## Quantified Performance Comparison

| Book | Raw dump tokens | Discovery loop tokens | B2S tokens | Dump/B2S ratio | Loop/B2S ratio |
|------|---------------|----------------------|-----------|----------------|----------------|
| Think Python 2 (small) | 119,264 | 12,152 | ~5,000 | **24×** | **2.4×** |
| Working Backwards (medium) | 175,253 | 33,444 | ~5,000 | **35×** | **6.7×** |
| AI Engineering (large) | 256,287 | 77,866 | ~5,000 | **51×** | **15.6×** |

*Data sourced from [`docs/performance.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/performance.md) in the virgiliojr94/book-to-skill repository.*

The **dump/B2S ratio** (24-51×) represents recurring savings per conversation turn. The **loop/B2S ratio** (2.4-15.6×) reflects the one-time cost to discover and compile a chapter.

## Why Context Dumping Fails at Scale

Raw context dumping suffers from linear cost growth. Every interaction resends the complete book—120,000-260,000 tokens for typical technical books. At Claude Sonnet 4.5 pricing (~$3/$15 per MTok), this becomes prohibitive quickly.

Book-to-skill's discovery loop fixes this with fixed-cost queries:

```bash

# Measure the discovery tax for chapter 5

python3 tools/discovery_tax.py \
    --full-text /tmp/book_skill_work/full_text.txt \
    --target-chapter 5

```

The [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) script quantifies this overhead, consistently showing ~5,000 token costs regardless of source material size.

## Cost Economics: $1 Conversion vs Unlimited Queries

A complete book-to-skill conversion costs approximately **$1 per book** in Claude Sonnet 4.5 tokens. This one-time investment replaces:

- Repeated full-text context injection
- Per-turn token costs that scale with book size
- Latency penalties from oversized prompts

The extraction pipeline handles multiple formats through `book_to_skill/parsers/*`:

```bash

# Extract from PDF, EPUB, DOCX, or mixed sources

python3 scripts/extract.py path/to/book.pdf path/to/extra.epub --mode text

```

The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) file orchestrates source-type detection, merges extracted content, and produces the metadata JSON that enables efficient chapter routing.

## Using a Compiled Skill

Once converted, skills load instantly with minimal token overhead:

```python
from book_to_skill.cli import load_skill

skill = load_skill("~/.copilot/skills/think-python-2")
answer = skill.ask("How does recursion work in Python?")
print(answer)

```

The [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) module provides the `load_skill()` function and query interface. Each `ask()` call incurs only the ~5,000 token core cost—not the full 119,000+ tokens of the source book.

## Technical Content Preservation

For technical books, book-to-skill avoids the "garbage in, garbage out" problem of raw PDF extraction:

- **Docling pipeline**: ~1.5s/page, preserves tables and code blocks as structured markdown
- **pdftotext pipeline**: ~0.1s/page for prose-only content

This selective approach—documented in [`docs/how-it-works.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/how-it-works.md)—maintains technical fidelity while eliminating layout noise that would otherwise inflate token counts.

## Summary

- **Token reduction**: 24-51× lower recurring costs vs context dumping
- **Fixed query cost**: ~5,000 tokens per interaction, independent of book size
- **One-time conversion**: ~$1 per book using standard LLM APIs
- **Multi-format support**: PDF, EPUB, DOCX, HTML, RTF through unified parser interface
- **Purpose-built pipelines**: Docling for technical content, pdftotext for speed

## Frequently Asked Questions

### How does book-to-skill handle books with complex formatting?

For technical books containing tables and code, `book_to_skill/parsers/*` invokes Docling to preserve structural elements as markdown. For prose-only works, pdftotext extraction runs at ~0.1s/page. This selective rendering—detailed in [`docs/how-it-works.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/docs/how-it-works.md)—ensures important content survives while unnecessary layout data gets stripped.

### What is the discovery loop tax?

The discovery loop tax refers to the fixed ~5,000 token cost of loading the resident model core plus one compiled chapter. The [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) script measures this precisely. Unlike context dumping, this cost stays constant whether the book has 100 pages or 1,000 pages.

### Can I convert multiple source formats in one command?

Yes. The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) CLI accepts mixed source types: `python3 scripts/extract.py book.pdf supplement.epub notes.docx --mode text`. The script automatically detects formats, merges extracted content into [`full_text.txt`](https://github.com/virgiliojr94/book-to-skill/blob/main/full_text.txt), and generates the metadata JSON required for chapter summarization.

### Is the $1 conversion cost accurate for all book sizes?

The ~$1 estimate applies to typical 300-500 page technical books using Claude Sonnet 4.5 at current pricing (~$3 input/$15 output per MTok). Larger books cost proportionally more to convert, but the per-query savings scale even higher—AI Engineering's 256,000 tokens yield 51× reduction, making conversion economical for any book queried more than once.