Book-to-Skill Strategies for Estimating Tokens in Different Text Types

Book-to-Skill employs a dual heuristic approach that divides word counts by 0.75 for Latin text and CJK character counts by 1.5 for Asian scripts, with an optional fallback to the exact tiktoken BPE tokenizer when available.

The virgiliojr94/book-to-skill repository needs deterministic token counts to forecast API costs and manage context window limits when processing books and documents. It implements language-aware strategies for estimating tokens that avoid heavy dependencies while maintaining accuracy across different writing systems.

Heuristic-Based Estimation Strategy

Book-to-Skill categorizes text into two processing paths based on the presence of CJK (Chinese, Japanese, Korean) characters. This approach accounts for the fundamental difference in tokenization density between whitespace-delimited languages and logographic scripts.

Latin and Whitespace-Delimited Text

For text containing only Latin characters and whitespace, the system applies a deterministic word-to-token ratio. The implementation in book_to_skill/utils.py splits the text on whitespace and divides the word count by the WORDS_PER_TOKEN constant defined in book_to_skill/config.py.

Key implementation details:

• The constant WORDS_PER_TOKEN = 0.75 represents the approximate inverse of the average tokens per word in English prose
• This ratio aligns with OpenAI's cl100k_base tokenizer behavior for common Latin-script content
• The function len(text.split()) provides the word count basis

CJK Character Handling

When the input contains CJK code points, Book-to-Skill switches to a character-based estimation model. The repository uses a compiled regex _CJK_RE to identify characters in the CJK Unicode blocks.

The calculation strategy:

• Each CJK character counts as approximately 1 / 1.5 tokens based on the CJK_CHARS_PER_TOKEN = 1.5 constant
• For mixed content, the system separates Latin words (with CJK characters substituted by spaces) and adds the CJK character count separately
• This handles supplementary plane characters correctly, as validated in tests/test_cjk_supplementary_plane.py

Optional Exact Tokenization with Tiktoken

For tooling that requires precise token counts rather than estimates, tools/discovery_tax.py provides a real BPE tokenizer implementation.

The count_tokens function attempts to import tiktoken and encode text using the cl100k_base encoding:

def count_tokens(text: str) -> int:
    """Real BPE count via tiktoken if available; else words/0.75 heuristic."""
    try:
        import tiktoken
        enc = tiktoken.get_encoding("cl100k_base")
        return len(enc.encode(text))
    except Exception:
        # Fallback to the deterministic heuristic

        return int(len(text.split()) / WORDS_PER_TOKEN)

If the tiktoken library is not installed or the import fails, the function falls back to the standard word-count heuristic, ensuring the tool remains functional without optional dependencies.

Configuration Constants and Sources

The estimation ratios live in book_to_skill/config.py and serve as the single source of truth for both the core utilities and discovery tools:

• WORDS_PER_TOKEN = 0.75 — Calibrated for Latin-based text
• CJK_CHARS_PER_TOKEN = 1.5 — Calibrated for CJK ideographic content

These constants allow users to adjust estimation aggressiveness globally without modifying the calculation logic.

Core Implementation Files

book_to_skill/utils.py

The primary estimate_tokens function implements the dual-path logic. It first checks for CJK characters using the compiled regex _CJK_RE. If none exist, it returns len(text.split()) / WORDS_PER_TOKEN. For mixed or pure CJK content, it calculates:

def estimate_tokens(text: str) -> int:
    """Estimate the token count of ``text`` with a deterministic heuristic."""
    if not text:
        return 0
    cjk = len(_CJK_RE.findall(text))
    if not cjk:
        return int(len(text.split()) / WORDS_PER_TOKEN)
    latin_words = len(_CJK_RE.sub(" ", text).split())
    return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)

This implementation ensures that a sentence like "Hello 世界" correctly separates the Latin word ("Hello") from the CJK characters ("世界") and applies the appropriate ratio to each segment.

tools/discovery_tax.py

This analysis script exposes the count_tokens wrapper function. It prioritizes accuracy over speed by attempting to use the actual OpenAI tokenizer when available, making it suitable for auditing the heuristic's accuracy or processing small text samples where exact counts matter.

Practical Usage Examples

Estimating Tokens for English Prose

For pure Latin text, the heuristic provides fast, battery-friendly estimates without importing heavy tokenization libraries:

from book_to_skill.utils import estimate_tokens

english = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(english))   # ≈ 13 tokens (13 words / 0.75)

Calculating CJK Content Costs

For Chinese, Japanese, or Korean content where characters dominate over whitespace, the character-based ratio prevents underestimation:

from book_to_skill.utils import estimate_tokens

chinese = "中" * 1500   # 1500 Chinese characters, no spaces

print(estimate_tokens(chinese))  # ≈ 1000 tokens (1500 / 1.5)

Using Real BPE Tokenization

When tiktoken is installed, developers can obtain exact token counts for billing reconciliation:

from tools.discovery_tax import count_tokens

mixed_text = "Hello, 世界!"
print(count_tokens(mixed_text))  # Accurate token count from tiktoken encoder

Summary

• Book-to-Skill uses a dual heuristic that applies 0.75 words-per-token for Latin text and 1.5 characters-per-token for CJK content
• Configuration constants reside in book_to_skill/config.py and control both estimation paths
• The estimate_tokens function in book_to_skill/utils.py handles mixed-language content by separating Latin words from CJK characters
• Optional exact counting is available through tools/discovery_tax.py when the tiktoken library is present
• Test coverage in tests/test_cjk_supplementary_plane.py ensures accurate handling of extended Unicode planes

Frequently Asked Questions

How accurate is the heuristic compared to OpenAI's tokenizer?

The heuristic provides estimates within 10-20% of actual cl100k_base token counts for most content. The constant values (0.75 for words, 1.5 for CJK) were chosen to match the average density observed in OpenAI's tokenizer, though exact counts require the optional tiktoken dependency.

Why does Book-to-Skill use different ratios for CJK and Latin text?

CJK characters typically consume fewer bytes per token in modern BPE tokenizers compared to Latin words. While a Latin word averages 1.3 tokens, a CJK character averages approximately 0.67 tokens. The ratios 0.75 and 1.5 represent the inverse of these densities to convert character/word counts into estimated token quantities.

What happens if tiktoken is not installed?

The system gracefully degrades to the heuristic estimator. In tools/discovery_tax.py, the count_tokens function catches import exceptions and falls back to int(len(text.split()) / WORDS_PER_TOKEN), ensuring all functionality remains available without the heavy tokenization library.

Where are the token estimation constants defined?

The constants WORDS_PER_TOKEN and CJK_CHARS_PER_TOKEN are defined in book_to_skill/config.py. Centralizing these values allows users to adjust estimation parameters globally without modifying the calculation logic in utils.py or discovery_tax.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}