# How `discovery_tax.py` Measures Token Savings Compared to Context‑Dumping

> Discover how discovery_tax.py measures token savings using actual token counts for full context-dump, discovery-loop, and Book-to-Skill strategies. Optimize your AI.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-25

---

**[`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) quantifies the "Discovery Loop Tax" by comparing three answering strategies—full context-dump, discovery-loop navigation, and the Book-to-Skill approach—using actual token counts derived from source text, Table of Contents, and compiled skill files.**

The [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) script in the [virgiliojr94/book-to-skill](https://github.com/virgiliojr94/book-to-skill) repository solves a critical problem for LLM-powered book Q&A systems: measuring exactly how many tokens each retrieval strategy consumes when answering a single targeted question. This article breaks down the methodology, the three strategies compared, and how token savings are calculated.

---

## What the Discovery Loop Tax Measures

The **Discovery Loop Tax** refers to the token overhead users pay when navigating a book via LLM interactions. Unlike simple context-dumping, which keeps the entire book resident in the model's context window, discovery loops require iterative searches through Table of Contents and chapters to locate relevant information.

[`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) in [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) operationalizes this by:

- Simulating three realistic answering scenarios
- Counting tokens using `tiktoken` (BPE) or a words-per-token heuristic
- Reporting compression ratios showing how much each strategy reduces token usage

---

## The Three Strategies Compared

### Context-Dump: Maximum Token Cost

The **context-dump** strategy represents the naive baseline: the entire extracted book remains in context for every turn.

In [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) lines 15–16, the script reads the full-text file via `--full-text` and passes the complete string to `count_tokens()`:

```python

# Lines 15-16: context-dump token calculation

full_text = Path(args.full_text).read_text(encoding="utf-8")
full_tokens = count_tokens(full_text)

```

This figure—often hundreds of thousands of tokens—serves as the denominator against which all savings are measured. The cost is **resident and re-billed on every turn**, making it prohibitively expensive for multi-turn conversations.

---

### Discovery-Loop: ToC Plus Target Navigation

The **discovery-loop** strategy simulates a user who must first consult the Table of Contents, then retrieve the target chapter, and potentially backtrack one chapter for missing definitions.

**Components counted:**

| Component | Source Function | Lines |
|-----------|-----------------|-------|
| ToC block | `extract_toc()` | 94–101 |
| Chapter splitting | `split_chapters()` | 69–84 |
| Target chapter selection | `best_chapter()` | 86–92 |
| Token aggregation | `count_tokens()` | 66–68 |

The script models two discovery-loop variants:

- **Best case**: ToC + target chapter only
- **Loop case**: ToC + target chapter + one prior chapter (when a definition is missing)

---

### Book-to-Skill: Core SKILL + Compiled Chapter

The **Book-to-Skill** approach leverages the repository's structured output: a resident **SKILL.md** core plus **on-demand compiled chapters**.

**Token derivation depends on input:**

| Input Available | Token Source | Lines |
|-----------------|--------------|-------|
| `--skill-dir` provided | [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) file + `chapters/*.md` target | 44–48, 49–57 |
| No skill directory | `--core-tokens` default (4000) + 1000-token chapter fallback | 60–63 |

This design captures the actual runtime cost of a deployed Book-to-Skill system: a small, evergreen skill core plus single-chapter retrieval.

---

## Token Counting Implementation

The `count_tokens()` function (lines 51–59) implements a two-tier counting strategy:

```python

# Lines 51-59: token counting with tiktoken fallback

def count_tokens(text: str) -> int:
    try:
        import tiktoken
        enc = tiktoken.get_encoding("cl100k_base")
        return len(enc.encode(text))
    except Exception:
        return int(len(text.split()) / 0.75)  # heuristic fallback

```

The **cl100k_base** encoder matches OpenAI's GPT-4/ChatGPT models, ensuring realistic BPE token counts. The words÷0.75 heuristic provides portability when `tiktoken` is unavailable.

The active method is surfaced via `token_method()` (lines 61–66) for auditability in output reports.

---

## Savings Calculation: Ratio Arithmetic

After gathering raw token figures, [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) computes compression ratios using a simple utility (lines 85–89):

```python
def ratio(a: int, b: int) -> str:
    return f"{a / b:.1f}x" if b else "n/a"

```

This yields statements like **"101.0x fewer tokens"** when comparing Book-to-Skill against context-dumping—directly quantifying the efficiency gain.

---

## Practical Usage

Run [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) against any full-text extraction to benchmark your own content:

```bash

# Basic measurement on full text only

python3 tools/discovery_tax.py --full-text data/full_text.txt

# Full comparison with generated skill artifacts

python3 tools/discovery_tax.py \
    --full-text data/full_text.txt \
    --skill-dir output/skill \
    --target-chapter 7

```

**Sample output structure:**

```

  Cost to answer ONE targeted question (tokens entering context):

    context-dump      :   1,254,321   (resident, re-billed EVERY turn)
    discovery (best)  :      84,652   ToC (12,345) + raw target chapter (72,307)
    discovery (loop)  :      97,198   + 1 prior chapter for missing definition
    book-to-skill     :      12,400   core [measured SKILL.md] (4,000) + compiled chapter (8,400)

  book-to-skill advantage:
    vs context-dump   : 101.0x fewer tokens
    vs discovery best :   6.8x fewer tokens
    vs discovery loop :   7.8x fewer tokens

```

---

## Supporting Infrastructure

| File | Role in Measurement |
|------|---------------------|
| [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) | Main script implementing token counting, chapter/ToC extraction, and comparative reporting |
| [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) | Reusable `_chapter_number()` detection and `_TOC_PATTERN` regex for consistent parsing |
| [`tests/test_discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_discovery_tax.py) | Unit tests validating token counts and ratios on sample book structures |

The [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) dependency ensures that [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) uses identical parsing logic to the main `book-to-skill` pipeline, making measurements representative of actual runtime behavior.

---

## Summary

- **[`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) measures token savings** by implementing three answering strategies with identical token-counting machinery
- **Context-dump** provides the expensive baseline: full book, every turn
- **Discovery-loop** models realistic ToC navigation with optional backtracking
- **Book-to-Skill** measures the optimized architecture: small resident core + single compiled chapter
- **Ratios are reported as "X.x fewer tokens"** for immediate interpretability
- **`tiktoken` cl100k_base** provides accurate BPE counts; a words÷0.75 fallback ensures portability

---

## Frequently Asked Questions

### How accurate is the token counting in [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py)?

When `tiktoken` is available, the script uses the **cl100k_base** BPE encoder—identical to OpenAI's GPT-4 tokenizer—yielding production-accurate counts. The words÷0.75 heuristic (used when `tiktoken` is missing) typically underestimates by 10–20% for English prose but preserves comparability across strategies since all three use the same counter.

### Can I run [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) without generating a full skill first?

Yes. The `--core-tokens` and internal defaults allow benchmarking against hypothetical Book-to-Skill deployments. Omitting `--skill-dir` uses 4000 tokens for the skill core and 1000 tokens per chapter as conservative estimates.

### What does the "loop case" in discovery-loop represent?

The loop case (ToC + target + prior chapter) models a realistic failure mode: the user finds the target chapter but encounters an undefined term that requires backtracking to the previous chapter. This adds approximately one extra chapter's worth of tokens to the discovery cost.

### Why compare against context-dumping if it's clearly inefficient?

Context-dumping remains the **implicit baseline** for many naive RAG implementations and "upload your document" UIs. Quantifying the 100×+ token reduction provides concrete justification for adopting structured skill architectures in production LLM applications.