How book-to-skill Handles Token Estimation for CJK Languages

book-to-skill estimates CJK token counts by detecting CJK code points with a compiled regex and dividing by 1.5 characters per token, while Latin text uses a whitespace-based word count divided by 0.75.

The book-to-skill library provides deterministic token estimation for language model applications without external dependencies. Unlike Latin scripts, Chinese, Japanese, and Korean (CJK) texts lack whitespace delimiters, requiring specialized handling to predict API token consumption accurately. This article explores how the virgiliojr94/book-to-skill repository implements CJK-aware token counting in its core utilities.

The Challenge of CJK Token Estimation

Standard word-count heuristics fail for CJK languages because these scripts do not separate words with spaces. A naive whitespace split would treat an entire Chinese paragraph as a single token, drastically underestimating costs. The library addresses this by treating individual CJK characters as distinct units, approximating the behavior of OpenAI's cl100k_base tokenizer.

CJK Detection and Token Calculation

The implementation relies on two complementary strategies: regex-based character detection and language-specific constants.

Detecting CJK Code Points with _CJK_RE

In book_to_skill/utils.py, the compiled regular expression _CJK_RE matches CJK ideographs, kana, hangul, full-width punctuation, and supplementary plane characters (U+20000-U+3FFFF). This ensures rare Han ideographs outside the Basic Multilingual Plane receive accurate counts alongside common BMP characters.

The estimate_tokens Function Logic

The estimate_tokens function applies a hybrid approach:

1. CJK character counting – Matches against _CJK_RE and divides the count by CJK_CHARS_PER_TOKEN (1.5) defined in book_to_skill/config.py
2. Latin word processing – Replaces CJK characters with spaces, splits on whitespace, and divides the word count by WORDS_PER_TOKEN (0.75)
3. Aggregation – Sums both estimates for mixed-language documents

Practical Code Examples

# Example 1 – Pure Latin text

from book_to_skill.utils import estimate_tokens
print(estimate_tokens("word " * 100))          # → 133 tokens (100 words / 0.75)

# Example 2 – Pure CJK text (no spaces)

cjk = "中" * 1500                               # 1,500 Chinese characters

print(estimate_tokens(cjk))                     # → 1000 tokens (1500 / 1.5)

# Example 3 – Mixed CJK + Latin

mixed = "hello 世界 " * 100
print(estimate_tokens(mixed))                   # Combines both heuristics

Configuration and Testing

The constants driving these calculations reside in book_to_skill/config.py, where CJK_CHARS_PER_TOKEN approximates OpenAI's tokenizer density for ideographic scripts. WORDS_PER_TOKEN handles space-delimited languages.

Validation occurs in tests/test_cjk_supplementary_plane.py, which verifies that supplementary-plane characters carry the same token weight as BMP characters. Integration tests in tests/test_book_to_skill.py confirm that 1,500 Chinese characters yield approximately 1,000 tokens, validating the 1.5 ratio.

Summary

• book-to-skill uses a compiled regex _CJK_RE to identify CJK code points including supplementary plane characters (U+20000-U+3FFFF)
• CJK characters are divided by 1.5 to estimate tokens, while Latin words are divided by 0.75
• The estimate_tokens function in book_to_skill/utils.py handles mixed-language text by processing CJK and Latin segments separately
• Comprehensive tests in tests/test_cjk_supplementary_plane.py ensure accurate counting across Unicode planes

Frequently Asked Questions

Why does book-to-skill use 1.5 characters per token for CJK?

This ratio approximates the cl100k_base tokenizer used by OpenAI models, where CJK characters typically consume between 1 and 2 tokens each. The constant CJK_CHARS_PER_TOKEN in config.py provides a deterministic average without requiring the tiktoken dependency.

How does book-to-skill handle mixed CJK and English text?

The estimate_tokens function replaces all CJK matches with spaces, then counts remaining Latin words. It calculates separate estimates for each script using their respective ratios and returns the sum, ensuring accurate predictions for bilingual documents.

Does book-to-skill support rare CJK characters outside the Basic Multilingual Plane?

Yes. The _CJK_RE regex in utils.py explicitly includes the supplementary planes range U+20000-U+3FFFF, ensuring rare Han ideographs receive proper token weight. Tests in test_cjk_supplementary_plane.py verify this behavior.

Where are the token estimation tests located?

Unit tests for CJK handling reside in tests/test_cjk_supplementary_plane.py, while broader integration tests covering multiple languages are in tests/test_book_to_skill.py. These files validate the 1.5 character ratio and supplementary plane support.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}