How Book-to-Skill Handles Different Character Sets for Token Estimation

Book-to-Skill estimates tokens across Latin, CJK, and other Unicode character sets using a layered approach: preferring exact tiktoken BPE counts when available, otherwise falling back to a deterministic heuristic that treats CJK characters as individual tokens and non-CJK text with a words-to-tokens ratio.

Deterministic token estimation is critical for cost-aware LLM workflows. The virgiliojr94/book-to-skill library implements estimate_tokens() in book_to_skill/utils.py to predict token consumption before sending text to an API. The design prioritizes accuracy when dependencies permit, and falls back to a fast, language-agnostic heuristic that handles East Asian scripts correctly.

The Two-Layer Estimation Strategy

Layer 1: Exact BPE Tokenization with tiktoken

When the optional tiktoken package is installed, estimate_tokens() calls tiktoken.get_encoding("cl100k_base") to compute the precise token count that OpenAI's cl100k encoder would produce. This approach works correctly for any Unicode input without special-casing character sets.

Layer 2: Deterministic Heuristic Without Dependencies

If tiktoken is unavailable (the default in CI environments), the function switches to a custom heuristic that distinguishes CJK characters from non-CJK text:

• CJK characters (Chinese, Japanese, Korean) — Each individual CJK code point counts as one token, whether it appears in the Basic Multilingual Plane (BMP) or a supplementary plane. The implementation caps the ratio at 0.66 tokens per character, so 1,500 Chinese characters estimates to roughly 1,000 tokens.

• Non-CJK text — Uses the classic words-to-tokens conversion:

tokens ≈ int(len(text.split()) / 0.75)  # ≈ 4/3 tokens per word

This yields the historically-pinned 100 words → 133 tokens mapping verified in tests/test_book_to_skill.py.

The final estimate returns the maximum of the CJK character count and the word-based heuristic. This ensures documents heavy in CJK characters do not receive underestimated token counts.

Code Examples

English-Only Text (Word-Based Heuristic)

from book_to_skill.utils import estimate_tokens

english = " ".join(["word"] * 100)  # 100 words

print(estimate_tokens(english))     # → 133

Pure Chinese Text (Character-Based Counting)

chinese = "中" * 1500
print(estimate_tokens(chinese))     # → ~1000 (exactly 1000 in test suite)

Mixed English and Chinese (Maximum of Both Estimates)

mixed = ("hello 世界 " * 100).strip()
print(estimate_tokens(mixed))       # → > 100 (CJK part counted directly)

Automatic tiktoken Detection

No code change is required. The same estimate_tokens() function automatically switches to tiktoken when installed:

# With tiktoken installed: returns exact BPE count

# Without tiktoken installed: returns heuristic estimate

estimate_tokens("任何文本 works identically")  # Works for any Unicode input

Test Coverage for Character Set Handling

The token estimation logic is validated across multiple test files:

Test File	Coverage
tests/test_cjk_supplementary_plane.py	BMP and supplementary-plane CJK characters
tests/test_book_to_skill.py	100-word → 133-token mapping verification
tests/test_discovery_tax.py	Fallback behavior when tiktoken is absent
tools/discovery_tax.py	Demonstrates heuristic in standalone scripts

Implementation Details in Source Files

• book_to_skill/utils.py — Contains the core estimate_tokens() function implementing both layers
• tests/test_cjk_supplementary_plane.py — Validates that characters like 𠀀 (U+20000, outside BMP) are handled correctly
• tools/discovery_tax.py — Shows practical usage of the fallback heuristic

Summary

Book-to-Skill handles different character sets for token estimation through:

• Primary path: Exact BPE tokenization via tiktoken.get_encoding("cl100k_base") when available
• Fallback path: Deterministic heuristic counting CJK characters individually and non-CJK text at ~1.33 tokens per word
• Safety mechanism: max(CJK_count, word_estimate) prevents underestimation for mixed or CJK-heavy documents
• Unicode coverage: Correct handling of BMP and supplementary plane characters verified in dedicated tests

Frequently Asked Questions

What happens if tiktoken is not installed?

The estimate_tokens() function in book_to_skill/utils.py automatically detects the absence of tiktoken and switches to the deterministic heuristic. No exception is raised, and estimation continues with CJK-aware character counting for accurate results across languages.

Why does the heuristic treat CJK characters differently?

BPE tokenizers typically encode CJK text with one token per character or less, unlike English where one word often splits into multiple tokens. The 0.66 tokens-per-character cap and character-level counting align estimates with observed behavior in production tokenizers like cl100k_base.

Does the supplementary plane CJK handling actually work?

Yes. The test suite in tests/test_cjk_supplementary_plane.py explicitly validates characters outside the Basic Multilingual Plane (code points above U+FFFF). The implementation correctly identifies these as CJK and counts them as individual tokens, matching the behavior for BMP CJK characters.

Can I force the heuristic even with tiktoken installed?

The current implementation in book_to_skill/utils.py does not expose a flag to bypass tiktoken. To force heuristic usage, you would need to temporarily uninstall or shadow the tiktoken module, or modify the source to add a parameter controlling layer selection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}