How book-to-skill Implements On-Demand Chapter Loading to Manage SKILL.md Token Count

book-to-skill splits generated skills into a static core SKILL.md and individual chapter files, loading only the specific chapter needed at runtime while enforcing token budgets through its Discovery-Tax utility.

The virgiliojr94/book-to-skill project solves a critical problem for AI agent skills: PDF books often contain tens of thousands of tokens, far exceeding the context windows of most language models. Rather than compressing or truncating content, the library implements a lazy-loading architecture that separates metadata from content and fetches chapters only when required.

The Two-Part Skill Structure

book-to-skill generates skills with a deliberate split between core and variable content:

Component	Location	Contents
Core SKILL.md	SKILL.md (root)	Skill metadata, description, agent logic
Chapter files	chapters/ch01.md, chapters/ch02.md, etc.	Full text of individual book chapters

This separation happens during extraction. The scripts/extract.py module handles the primary conversion, parsing the source PDF and writing each detected chapter to its own markdown file.

# Generate a skill structure from a PDF

$ python -m book_to_skill.cli extract my_book.pdf --out-dir skill

After extraction, the skill/ directory contains the lightweight core plus the chapters/ folder with individual files that can be loaded independently.

Lazy Loading Implementation

The runtime follows a strict core-first, chapters-on-demand pattern. The agent always receives the core SKILL.md, which contains sufficient information to determine whether additional context is needed.

from pathlib import Path

# Core SKILL.md is always loaded first

core = Path("skill/SKILL.md").read_text(encoding="utf-8")

# Load a specific chapter only when it is needed

def load_chapter(chapter_number: int) -> str:
    chapter_path = Path(f"skill/chapters/ch{chapter_number:02d}.md")
    if not chapter_path.is_file():
        raise FileNotFoundError(f"Chapter {chapter_number} not found")
    return chapter_path.read_text(encoding="utf-8")

# Example: a user asks about chapter 3

chapter_text = load_chapter(3)

The tools/scan_generated_skill.py utility discovers available chapters by scanning the chapters/ directory, enabling dynamic enumeration without hardcoding filenames.

Token Budget Enforcement with Discovery-Tax

Token management is handled by tools/discovery_tax.py, which implements the Discovery-Tax mechanism. This utility calculates cumulative token usage and prevents budget overruns.

The process works in three stages:

1. Measure core tokens — call count_tokens on SKILL.md contents
2. Estimate or measure chapter tokens — use typical compiled size or read the exact target chapter
3. Validate against cap — default threshold is approximately 4000 tokens

from tools.discovery_tax import count_tokens

# Token-budget check (used internally by discovery_tax)

core_tokens = count_tokens(core)
chapter_tokens = count_tokens(chapter_text)

if core_tokens + chapter_tokens > 4000:
    raise RuntimeError("Token budget exceeded")

If adding a chapter would exceed the cap, the system either halts further loading or substitutes a smaller fallback chapter. This ensures the agent never receives more tokens than the model can process.

Supporting Infrastructure

Several modules enable this on-demand chapter loading architecture:

File	Function
scripts/extract.py	Generates SKILL.md and populates chapters/ folder
tools/discovery_tax.py	Calculates token usage and enforces budget caps
tools/scan_generated_skill.py	Discovers and lists available chapter files
book_to_skill/utils.py	Provides count_tokens and text processing utilities
book_to_skill/config.py	Defines runtime constants and supported extensions

The count_tokens implementation in book_to_skill/utils.py provides the underlying estimation that discovery_tax.py relies upon for its calculations.

How This Compares to Alternative Approaches

Other PDF-to-skill tools often use:

• Full ingestion — load entire books, causing immediate token overflow
• Aggressive summarization — lose specific details that users may query
• Fixed chunking — break text arbitrarily, disrupting chapter coherence

The book-to-skill approach preserves complete chapter text while keeping runtime token usage bounded, trading storage footprint (multiple small files) for runtime flexibility.

Summary

• book-to-skill generates two-part skills: static SKILL.md plus dynamic chapter files in chapters/
• Lazy loading fetches individual chapters only when specific context is required
• Discovery-Tax (tools/discovery_tax.py) enforces token budgets by summing core and chapter estimates against a ~4000 token cap
• Extraction in scripts/extract.py, discovery in tools/scan_generated_skill.py, and token counting in book_to_skill/utils.py together enable this architecture

Frequently Asked Questions

What happens if a requested chapter exceeds the remaining token budget?

The Discovery-Tax utility either raises a RuntimeError when the combined core-plus-chapter tokens exceed the threshold, or loads a smaller fallback chapter if one is configured. This prevents the agent from receiving oversized contexts.

How does book-to-skill determine where chapters begin and end in a PDF?

The scripts/extract.py module handles chapter detection during the initial extraction phase, using heuristics based on document structure to identify chapter boundaries and write each to a separate chapters/chNN.md file.

Can the token budget be adjusted for different models?

Yes. The default cap of approximately 4000 tokens is defined in the configuration, and tools/discovery_tax.py checks against this configurable threshold. Users with larger context windows can increase this value, while those with stricter limits can decrease it.

Does splitting into separate files affect search or retrieval performance?

The tools/scan_generated_skill.py utility enables fast enumeration of available chapters without loading their contents. Since files are loaded by path and read only when needed, the overhead is minimal compared to the token savings achieved.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}