How CJK Characters Are Tokenized Differently in book-to-skill: Latin vs. CJK Token Estimation Explained

CJK characters are tokenized as individual code-points rather than whitespace-delimited words, using a separate ratio of 1.5 characters per token instead of 0.75 words per token.

The book-to-skill repository implements a deterministic token estimation heuristic that fundamentally distinguishes between Latin-script and CJK (Chinese-Japanese-Korean) scripts. This dual-path approach ensures accurate token budgeting for multilingual ebook processing, preventing massive under-estimation that would otherwise occur with spaceless East Asian languages.

Why CJK Requires Special Tokenization Handling

Standard LLM tokenizers split Latin text on whitespace boundaries, making word count a reliable proxy for token count. CJK languages present a critical challenge: they use little or no whitespace, so word-based counting would underestimate tokens by multiple orders of magnitude. A Chinese novel with 10,000 characters might register as only a few "words," leading to catastrophic under-allocation of API resources.

The estimate_tokens function in book_to_skill/utils.py solves this by detecting CJK code-points and applying a character-based ratio instead.

The Two-Branch Token Estimation Logic

Latin-Script Branch: Word-Based Estimation

For text without CJK characters, tokens are approximated by splitting on whitespace and applying the WORDS_PER_TOKEN ratio:

# From book_to_skill/utils.py lines 92-95

non_cjk_tokens = len(text.split()) / WORDS_PER_TOKEN

This yields roughly 0.75 words per token, matching the compression ratio observed in cl100k and similar tokenizers.

CJK Branch: Character-Based Estimation

When CJK characters are detected, each code-point counts directly against CJK_CHARS_PER_TOKEN:

# From book_to_skill/utils.py lines 73-77 and 92-96

_CJK_RE = re.compile(
    r'[\u4E00-\u9FFF\u3400-\u4DBF\u3040-\u309F\u30A0-\u30FF'
    r'\uAC00-\uD7AF\uF900-\uFAFF\u3000-\u303F\uFF00-\uFFEF'
    r'\U00020000-\U0003FFFF]+'
)
cjk_chars = len(_CJK_RE.findall(text))
cjk_tokens = cjk_chars / CJK_CHARS_PER_TOKEN

The regular expression captures:

• CJK Unified Ideographs (U+4E00–U+9FFF)
• CJK Extension A (U+3400–U+4DBF)
• Hiragana and Katakana (U+3040–U+30FF)
• Hangul syllables (U+AC00–U+D7AF)
• CJK punctuation and full-width forms (U+3000–U+303F, U+FF00–U+FFEF)
• Supplementary planes (U+20000–U+3FFFF) — rare but critical for historical and variant characters

This produces approximately 1.5 characters per token, reflecting the denser encoding of CJK in modern tokenizers.

Configuration Constants

Both ratios are centralized in book_to_skill/config.py (lines 14–18):

# Token estimation ratios

WORDS_PER_TOKEN = 0.75      # Latin-script languages

CJK_CHARS_PER_TOKEN = 1.5   # Chinese, Japanese, Korean

These values mirror the empirical behavior of OpenAI's cl100k_base tokenizer and can be adjusted for other tokenization schemes.

Practical Code Examples

from book_to_skill.utils import estimate_tokens

# Latin text — word-based estimate

latin = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(latin))

# Output: ~10.7 tokens (9 words / 0.75)

# Pure CJK — each ideograph counted directly

chinese = "人工智能是未来的关键技术"
print(estimate_tokens(chinese))

# Output: ~6.7 tokens (10 characters / 1.5)

# Mixed content — both branches combined

mixed = "Chapter 1: 人工智能概述"
print(estimate_tokens(mixed))

# Output: ~6.0 tokens (3 words / 0.75 + 4 chars / 1.5)

Testing and Validation

The CJK handling is explicitly validated by dedicated tests:

• tests/test_cjk_supplementary_plane.py — Verifies correct counting of supplementary-plane characters (U+20000+), ensuring the regex spans full Unicode range
• tests/test_metadata_encoding.py — Confirms CJK headings survive round-trip encoding without corruption

This test coverage was added following issue #103, which identified the original under-counting bug for CJK content.

Summary

• Latin text uses word-splitting with WORDS_PER_TOKEN = 0.75
• CJK text uses character-counting with CJK_CHARS_PER_TOKEN = 1.5
• _CJK_RE in utils.py matches all CJK blocks including supplementary planes
• Mixed text combines both estimates for accurate totals
• Configuration lives in config.py for easy ratio adjustment

Frequently Asked Questions

How does book-to-skill detect CJK characters?

The estimate_tokens function uses the pre-compiled _CJK_RE regular expression defined at module level in book_to_skill/utils.py. This regex matches Unicode blocks for Han ideographs, kana, Hangul, and related punctuation across both Basic Multilingual Plane and supplementary planes.

What happens if text mixes Latin and CJK scripts?

Both branches execute and sum their results. Whitespace-delimited Latin words divide by WORDS_PER_TOKEN, while matched CJK characters divide by CJK_CHARS_PER_TOKEN. The function returns the ceiling of the combined total.

Why 1.5 characters per token for CJK?

This ratio approximates the empirical behavior of OpenAI's cl100k tokenizer, where CJK characters typically consume 1–2 tokens each depending on frequency in the training corpus. The value is configurable in config.py for compatibility with other tokenizers.

Does the CJK regex include emoji or other symbols?

No. The _CJK_RE pattern is tightly scoped to CJK-specific Unicode blocks. Emoji and general symbols fall through to the Latin branch, where they are counted as whitespace-delimited tokens—usually accurate since emoji often appear space-separated in practice.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}