# How CJK Characters Are Tokenized Differently in book-to-skill: Latin vs. CJK Token Estimation Explained

> Understand how CJK characters tokenize differently in book-to-skill. Explore code-point tokenization vs. whitespace-delimited words and discover the impact on token estimation.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-23

---

**CJK characters are tokenized as individual code-points rather than whitespace-delimited words, using a separate ratio of 1.5 characters per token instead of 0.75 words per token.**

The `book-to-skill` repository implements a deterministic token estimation heuristic that fundamentally distinguishes between Latin-script and CJK (Chinese-Japanese-Korean) scripts. This dual-path approach ensures accurate token budgeting for multilingual ebook processing, preventing massive under-estimation that would otherwise occur with spaceless East Asian languages.

## Why CJK Requires Special Tokenization Handling

Standard LLM tokenizers split Latin text on whitespace boundaries, making word count a reliable proxy for token count. CJK languages present a critical challenge: they use **little or no whitespace**, so word-based counting would underestimate tokens by multiple orders of magnitude. A Chinese novel with 10,000 characters might register as only a few "words," leading to catastrophic under-allocation of API resources.

The `estimate_tokens` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) solves this by detecting CJK code-points and applying a character-based ratio instead.

## The Two-Branch Token Estimation Logic

### Latin-Script Branch: Word-Based Estimation

For text without CJK characters, tokens are approximated by splitting on whitespace and applying the `WORDS_PER_TOKEN` ratio:

```python

# From book_to_skill/utils.py lines 92-95

non_cjk_tokens = len(text.split()) / WORDS_PER_TOKEN

```

This yields roughly **0.75 words per token**, matching the compression ratio observed in cl100k and similar tokenizers.

### CJK Branch: Character-Based Estimation

When CJK characters are detected, each code-point counts directly against `CJK_CHARS_PER_TOKEN`:

```python

# From book_to_skill/utils.py lines 73-77 and 92-96

_CJK_RE = re.compile(
    r'[\u4E00-\u9FFF\u3400-\u4DBF\u3040-\u309F\u30A0-\u30FF'
    r'\uAC00-\uD7AF\uF900-\uFAFF\u3000-\u303F\uFF00-\uFFEF'
    r'\U00020000-\U0003FFFF]+'
)
cjk_chars = len(_CJK_RE.findall(text))
cjk_tokens = cjk_chars / CJK_CHARS_PER_TOKEN

```

The regular expression captures:
- **CJK Unified Ideographs** (U+4E00–U+9FFF)
- **CJK Extension A** (U+3400–U+4DBF)
- **Hiragana and Katakana** (U+3040–U+30FF)
- **Hangul syllables** (U+AC00–U+D7AF)
- **CJK punctuation and full-width forms** (U+3000–U+303F, U+FF00–U+FFEF)
- **Supplementary planes** (U+20000–U+3FFFF) — rare but critical for historical and variant characters

This produces approximately **1.5 characters per token**, reflecting the denser encoding of CJK in modern tokenizers.

## Configuration Constants

Both ratios are centralized in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) (lines 14–18):

```python

# Token estimation ratios

WORDS_PER_TOKEN = 0.75      # Latin-script languages

CJK_CHARS_PER_TOKEN = 1.5   # Chinese, Japanese, Korean

```

These values mirror the empirical behavior of OpenAI's cl100k_base tokenizer and can be adjusted for other tokenization schemes.

## Practical Code Examples

```python
from book_to_skill.utils import estimate_tokens

# Latin text — word-based estimate

latin = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(latin))

# Output: ~10.7 tokens (9 words / 0.75)

# Pure CJK — each ideograph counted directly

chinese = "人工智能是未来的关键技术"
print(estimate_tokens(chinese))

# Output: ~6.7 tokens (10 characters / 1.5)

# Mixed content — both branches combined

mixed = "Chapter 1: 人工智能概述"
print(estimate_tokens(mixed))

# Output: ~6.0 tokens (3 words / 0.75 + 4 chars / 1.5)

```

## Testing and Validation

The CJK handling is explicitly validated by dedicated tests:

- **[`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py)** — Verifies correct counting of supplementary-plane characters (U+20000+), ensuring the regex spans full Unicode range
- **[`tests/test_metadata_encoding.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_metadata_encoding.py)** — Confirms CJK headings survive round-trip encoding without corruption

This test coverage was added following issue #103, which identified the original under-counting bug for CJK content.

## Summary

- **Latin text** uses word-splitting with `WORDS_PER_TOKEN = 0.75`
- **CJK text** uses character-counting with `CJK_CHARS_PER_TOKEN = 1.5`
- **`_CJK_RE`** in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) matches all CJK blocks including supplementary planes
- **Mixed text** combines both estimates for accurate totals
- **Configuration** lives in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py) for easy ratio adjustment

## Frequently Asked Questions

### How does book-to-skill detect CJK characters?

The `estimate_tokens` function uses the pre-compiled `_CJK_RE` regular expression defined at module level in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py). This regex matches Unicode blocks for Han ideographs, kana, Hangul, and related punctuation across both Basic Multilingual Plane and supplementary planes.

### What happens if text mixes Latin and CJK scripts?

Both branches execute and sum their results. Whitespace-delimited Latin words divide by `WORDS_PER_TOKEN`, while matched CJK characters divide by `CJK_CHARS_PER_TOKEN`. The function returns the ceiling of the combined total.

### Why 1.5 characters per token for CJK?

This ratio approximates the empirical behavior of OpenAI's cl100k tokenizer, where CJK characters typically consume 1–2 tokens each depending on frequency in the training corpus. The value is configurable in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py) for compatibility with other tokenizers.

### Does the CJK regex include emoji or other symbols?

No. The `_CJK_RE` pattern is tightly scoped to CJK-specific Unicode blocks. Emoji and general symbols fall through to the Latin branch, where they are counted as whitespace-delimited tokens—usually accurate since emoji often appear space-separated in practice.