# How book-to-skill Handles Token Estimation for CJK Languages

> Discover how book-to-skill estimates CJK token counts using regex and a character division for efficient processing. Learn about its unique approach compared to Latin text.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-28

---

**book-to-skill estimates CJK token counts by detecting CJK code points with a compiled regex and dividing by 1.5 characters per token, while Latin text uses a whitespace-based word count divided by 0.75.**

The `book-to-skill` library provides deterministic token estimation for language model applications without external dependencies. Unlike Latin scripts, Chinese, Japanese, and Korean (CJK) texts lack whitespace delimiters, requiring specialized handling to predict API token consumption accurately. This article explores how the virgiliojr94/book-to-skill repository implements CJK-aware token counting in its core utilities.

## The Challenge of CJK Token Estimation

Standard word-count heuristics fail for CJK languages because these scripts do not separate words with spaces. A naive whitespace split would treat an entire Chinese paragraph as a single token, drastically underestimating costs. The library addresses this by treating individual CJK characters as distinct units, approximating the behavior of OpenAI's `cl100k_base` tokenizer.

## CJK Detection and Token Calculation

The implementation relies on two complementary strategies: regex-based character detection and language-specific constants.

### Detecting CJK Code Points with _CJK_RE

In [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py), the compiled regular expression `_CJK_RE` matches CJK ideographs, kana, hangul, full-width punctuation, and supplementary plane characters (U+20000-U+3FFFF). This ensures rare Han ideographs outside the Basic Multilingual Plane receive accurate counts alongside common BMP characters.

### The estimate_tokens Function Logic

The `estimate_tokens` function applies a hybrid approach:

1. **CJK character counting** – Matches against `_CJK_RE` and divides the count by `CJK_CHARS_PER_TOKEN` (1.5) defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)
2. **Latin word processing** – Replaces CJK characters with spaces, splits on whitespace, and divides the word count by `WORDS_PER_TOKEN` (0.75)
3. **Aggregation** – Sums both estimates for mixed-language documents

## Practical Code Examples

```python

# Example 1 – Pure Latin text

from book_to_skill.utils import estimate_tokens
print(estimate_tokens("word " * 100))          # → 133 tokens (100 words / 0.75)

```

```python

# Example 2 – Pure CJK text (no spaces)

cjk = "中" * 1500                               # 1,500 Chinese characters

print(estimate_tokens(cjk))                     # → 1000 tokens (1500 / 1.5)

```

```python

# Example 3 – Mixed CJK + Latin

mixed = "hello 世界 " * 100
print(estimate_tokens(mixed))                   # Combines both heuristics

```

## Configuration and Testing

The constants driving these calculations reside in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), where `CJK_CHARS_PER_TOKEN` approximates OpenAI's tokenizer density for ideographic scripts. `WORDS_PER_TOKEN` handles space-delimited languages.

Validation occurs in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py), which verifies that supplementary-plane characters carry the same token weight as BMP characters. Integration tests in [`tests/test_book_to_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_book_to_skill.py) confirm that 1,500 Chinese characters yield approximately 1,000 tokens, validating the 1.5 ratio.

## Summary

- book-to-skill uses a **compiled regex `_CJK_RE`** to identify CJK code points including supplementary plane characters (U+20000-U+3FFFF)
- **CJK characters** are divided by 1.5 to estimate tokens, while **Latin words** are divided by 0.75
- The **`estimate_tokens`** function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) handles mixed-language text by processing CJK and Latin segments separately
- **Comprehensive tests** in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py) ensure accurate counting across Unicode planes

## Frequently Asked Questions

### Why does book-to-skill use 1.5 characters per token for CJK?

This ratio approximates the `cl100k_base` tokenizer used by OpenAI models, where CJK characters typically consume between 1 and 2 tokens each. The constant `CJK_CHARS_PER_TOKEN` in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py) provides a deterministic average without requiring the `tiktoken` dependency.

### How does book-to-skill handle mixed CJK and English text?

The `estimate_tokens` function replaces all CJK matches with spaces, then counts remaining Latin words. It calculates separate estimates for each script using their respective ratios and returns the sum, ensuring accurate predictions for bilingual documents.

### Does book-to-skill support rare CJK characters outside the Basic Multilingual Plane?

Yes. The `_CJK_RE` regex in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) explicitly includes the supplementary planes range U+20000-U+3FFFF, ensuring rare Han ideographs receive proper token weight. Tests in [`test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/test_cjk_supplementary_plane.py) verify this behavior.

### Where are the token estimation tests located?

Unit tests for CJK handling reside in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py), while broader integration tests covering multiple languages are in [`tests/test_book_to_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_book_to_skill.py). These files validate the 1.5 character ratio and supplementary plane support.