# How book-to-skill Implements On-Demand Chapter Loading to Manage SKILL.md Token Count

> Discover how book-to-skill uses on-demand chapter loading to manage SKILL.md token count. Learn about its static core and runtime chapter loading for efficient token budgeting.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-25

---

**`book-to-skill` splits generated skills into a static core SKILL.md and individual chapter files, loading only the specific chapter needed at runtime while enforcing token budgets through its Discovery-Tax utility.**

The `virgiliojr94/book-to-skill` project solves a critical problem for AI agent skills: PDF books often contain tens of thousands of tokens, far exceeding the context windows of most language models. Rather than compressing or truncating content, the library implements a **lazy-loading architecture** that separates metadata from content and fetches chapters only when required.

## The Two-Part Skill Structure

`book-to-skill` generates skills with a deliberate split between core and variable content:

| Component | Location | Contents |
|-----------|----------|----------|
| **Core SKILL.md** | [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) (root) | Skill metadata, description, agent logic |
| **Chapter files** | [`chapters/ch01.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/chapters/ch01.md), [`chapters/ch02.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/chapters/ch02.md), etc. | Full text of individual book chapters |

This separation happens during extraction. The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) module handles the primary conversion, parsing the source PDF and writing each detected chapter to its own markdown file.

```python

# Generate a skill structure from a PDF

$ python -m book_to_skill.cli extract my_book.pdf --out-dir skill

```

After extraction, the `skill/` directory contains the lightweight core plus the `chapters/` folder with individual files that can be loaded independently.

## Lazy Loading Implementation

The runtime follows a strict **core-first, chapters-on-demand** pattern. The agent always receives the core SKILL.md, which contains sufficient information to determine whether additional context is needed.

```python
from pathlib import Path

# Core SKILL.md is always loaded first

core = Path("skill/SKILL.md").read_text(encoding="utf-8")

# Load a specific chapter only when it is needed

def load_chapter(chapter_number: int) -> str:
    chapter_path = Path(f"skill/chapters/ch{chapter_number:02d}.md")
    if not chapter_path.is_file():
        raise FileNotFoundError(f"Chapter {chapter_number} not found")
    return chapter_path.read_text(encoding="utf-8")

# Example: a user asks about chapter 3

chapter_text = load_chapter(3)

```

The [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) utility discovers available chapters by scanning the `chapters/` directory, enabling dynamic enumeration without hardcoding filenames.

## Token Budget Enforcement with Discovery-Tax

Token management is handled by [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), which implements the **Discovery-Tax** mechanism. This utility calculates cumulative token usage and prevents budget overruns.

The process works in three stages:

1. **Measure core tokens** — call `count_tokens` on SKILL.md contents
2. **Estimate or measure chapter tokens** — use typical compiled size or read the exact target chapter
3. **Validate against cap** — default threshold is approximately 4000 tokens

```python
from tools.discovery_tax import count_tokens

# Token-budget check (used internally by discovery_tax)

core_tokens = count_tokens(core)
chapter_tokens = count_tokens(chapter_text)

if core_tokens + chapter_tokens > 4000:
    raise RuntimeError("Token budget exceeded")

```

If adding a chapter would exceed the cap, the system either halts further loading or substitutes a smaller fallback chapter. This ensures the agent never receives more tokens than the model can process.

## Supporting Infrastructure

Several modules enable this on-demand chapter loading architecture:

| File | Function |
|------|----------|
| [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) | Generates SKILL.md and populates `chapters/` folder |
| [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) | Calculates token usage and enforces budget caps |
| [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) | Discovers and lists available chapter files |
| [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) | Provides `count_tokens` and text processing utilities |
| [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) | Defines runtime constants and supported extensions |

The `count_tokens` implementation in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) provides the underlying estimation that [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py) relies upon for its calculations.

## How This Compares to Alternative Approaches

Other PDF-to-skill tools often use:

- **Full ingestion** — load entire books, causing immediate token overflow
- **Aggressive summarization** — lose specific details that users may query
- **Fixed chunking** — break text arbitrarily, disrupting chapter coherence

The `book-to-skill` approach preserves complete chapter text while keeping runtime token usage bounded, trading storage footprint (multiple small files) for runtime flexibility.

## Summary

- `book-to-skill` generates **two-part skills**: static SKILL.md plus dynamic chapter files in `chapters/`
- **Lazy loading** fetches individual chapters only when specific context is required
- **Discovery-Tax** ([`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py)) enforces token budgets by summing core and chapter estimates against a ~4000 token cap
- Extraction in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), discovery in [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py), and token counting in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) together enable this architecture

## Frequently Asked Questions

### What happens if a requested chapter exceeds the remaining token budget?

The Discovery-Tax utility either raises a `RuntimeError` when the combined core-plus-chapter tokens exceed the threshold, or loads a smaller fallback chapter if one is configured. This prevents the agent from receiving oversized contexts.

### How does book-to-skill determine where chapters begin and end in a PDF?

The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) module handles chapter detection during the initial extraction phase, using heuristics based on document structure to identify chapter boundaries and write each to a separate [`chapters/chNN.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/chapters/chNN.md) file.

### Can the token budget be adjusted for different models?

Yes. The default cap of approximately 4000 tokens is defined in the configuration, and [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) checks against this configurable threshold. Users with larger context windows can increase this value, while those with stricter limits can decrease it.

### Does splitting into separate files affect search or retrieval performance?

The [`tools/scan_generated_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/scan_generated_skill.py) utility enables fast enumeration of available chapters without loading their contents. Since files are loaded by path and read only when needed, the overhead is minimal compared to the token savings achieved.