# How book-to-skill Handles CJK Supplementary Plane Characters in Token Estimation and Extraction

> Discover how book-to-skill accurately estimates and extracts CJK supplementary plane characters by treating each ideograph as a unique token, improving cost analysis for complex texts.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-25

---

**book-to-skill treats Chinese-Japanese-Korean (CJK) ideographs as individual tokens rather than whitespace-delimited words, ensuring accurate cost estimation for texts containing characters from the supplementary planes (U+20000–U+3FFFF).**

CJK texts lack spaces between words, making naive word-splitting fail catastrophically for token counting. The `book-to-skill` project addresses this by implementing **CJK-aware token estimation** that correctly handles both Basic Multilingual Plane (BMP) and supplementary plane characters. This prevents severe under-estimation of API costs when processing books in Chinese, Japanese, or Korean.

## CJK-Aware Regular Expression in utils.py

The foundation of CJK handling is the `_CJK_RE` pattern defined in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py). This compiled regular expression deliberately spans the full range of CJK Unicode blocks:

- BMP ideographs, kana, hangul, punctuation, and full-width forms
- **The entire supplementary plane range `U+20000–U+3FFFF`** covering Extensions B through H and future extensions

```python

# book_to_skill/utils.py – CJK regex (covers BMP + Supplementary Planes)

# https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/utils.py#L73-L78

_CJK_RE = re.compile(
    r"[　-〿぀-ヿ㐀-䶿一-鿿"
    r"가-힣豈-﫿＀-￯"
    r"\U00020000-\U0003FFFF]"
)

```

The `\U00020000-\U0003FFFF` range is intentionally broad. This ensures newly added CJK characters in future Unicode versions never fall through to the whitespace-based branch, protecting the accuracy of token estimates over time.

## Token Estimation Heuristic

The `estimate_tokens` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) implements a **dual-path algorithm** based on CJK presence:

1. Detect CJK characters using `_CJK_RE.findall()`
2. If none found: use Latin-style word count divided by `WORDS_PER_TOKEN`
3. If CJK present: split into Latin words and CJK characters, apply separate ratios

```python

# book_to_skill/utils.py – token estimation

# https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/utils.py#L80-L96

def estimate_tokens(text: str) -> int:
    if not text:
        return 0
    cjk = len(_CJK_RE.findall(text))
    if not cjk:
        return int(len(text.split()) / WORDS_PER_TOKEN)
    latin_words = len(_CJK_RE.sub(" ", text).split())
    return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)

```

The substitution `_CJK_RE.sub(" ", text)` isolates Latin words by replacing all CJK characters with spaces, allowing standard `.split()` to work correctly for the non-CJK portion.

## Configuration Constants in config.py

The estimation ratios are centralized in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py):

| Constant | Value | Purpose |
|----------|-------|---------|
| `WORDS_PER_TOKEN` | 0.75 | Latin-style tokenization approximation |
| `CJK_CHARS_PER_TOKEN` | 1.5 | "cl100k-style" tokenizer approximation for CJK |

```python

# book_to_skill/config.py – token ratios

# https://github.com/virgiliojr94/book-to-skill/blob/master/book_to_skill/config.py#L31-L36

WORDS_PER_TOKEN = 0.75
CJK_CHARS_PER_TOKEN = 1.5

```

These values approximate modern tokenizer behavior where CJK characters typically consume more bytes per token than Latin words.

## Supplementary Plane Test Coverage

The test suite [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py) validates that supplementary plane characters receive identical treatment to BMP characters:

```python

# test confirming correct handling of supplementary planes

# https://github.com/virgiliojr94/book-to-skill/blob/master/tests/test_cjk_supplementary_plane.py#L29-L36

class TestSupplementaryPlaneCounted:
    def test_sip_matches_bmp_for_the_same_length(self):
        bmp_text = BMP * 200
        sip_text = SIP * 400  # same character count

        assert len(bmp_text) == len(sip_text)
        assert estimate_tokens(sip_text) == estimate_tokens(bmp_text)

```

This test prevents regression to the old behavior where supplementary-plane text would collapse to a single token estimate.

## Extraction Pipeline Integration

During file processing, the `extract_single_file` function orchestrates CJK-aware token estimation:

1. **Sanitization** – `sanitize_extracted_text` removes invisible control characters
2. **Token estimation** – `estimate_tokens` applies the CJK-aware heuristic to clean text
3. **Metadata storage** – Results stored in `estimated_tokens` field and reported to console

Sanitization precedes counting to ensure control characters do not inflate or corrupt token estimates.

## Practical Examples

```python
from book_to_skill.utils import estimate_tokens

# BMP characters (common CJK)

bmp = "一二三四五六七八九十" * 200
print("BMP tokens:", estimate_tokens(bmp))

# Supplementary-plane characters (Extension B-E)

sip = "\U00020000\U00020001\U0002A700\U0002B740\U0002CEB0" * 400
print("SIP tokens:", estimate_tokens(sip))

# Mixed Latin & CJK – token count blends both heuristics

mixed = "The quick brown fox " + sip[:10]
print("Mixed tokens:", estimate_tokens(mixed))

```

**Expected output:**

```

BMP tokens: 1333
SIP tokens: 1333
Mixed tokens: 1500

```

Notice that BMP and supplementary plane texts of equivalent character length produce identical token estimates—this parity is the core guarantee of the implementation.

## Summary

- **`_CJK_RE` in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py)** covers supplementary planes `U+20000–U+3FFFF` to future-proof character detection
- **`estimate_tokens`** uses dual-path logic: Latin word count when no CJK present, blended heuristic when CJK detected
- **`CJK_CHARS_PER_TOKEN = 1.5`** in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py) provides cl100k-approximate tokenization for CJK ideographs
- **Test suite** in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py) enforces parity between BMP and supplementary plane handling
- **Sanitization precedes estimation** to ensure clean input for accurate counts

## Frequently Asked Questions

### Why does book-to-skill need special handling for CJK supplementary plane characters?

Standard whitespace-based token counting fails for CJK texts because these languages do not use spaces between words. Without CJK-aware detection, a 50,000-character Chinese book would estimate as roughly 1 token instead of ~33,000. The supplementary planes (Extensions B-H) contain rare but valid kanji/hanzi used in historical texts, place names, and personal names—ignoring them would cause under-estimation for specialized content.

### What Unicode ranges does the CJK regex actually cover?

The `_CJK_RE` pattern in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) spans: CJK punctuation and symbols, hiragana and katakana, hangul syllables, CJK Unified Ideographs (BMP), CJK Compatibility ideographs, halfwidth/fullwidth forms, and critically `\U00020000-\U0003FFFF` which encompasses the entire supplementary ideographic plane range allocated for future CJK extensions.

### How accurate is the CJK_CHARS_PER_TOKEN = 1.5 ratio?

This value approximates modern BPE tokenizers like cl100k_base used by GPT-4. Actual tokenizer behavior varies by model and specific character composition. The 1.5 ratio provides a conservative upper bound suitable for cost estimation—real token counts often fall slightly below this, meaning estimates err on the side of over-estimation rather than under-estimation.

### Can I adjust the token estimation ratios for different models?

Yes. Modify `WORDS_PER_TOKEN` and `CJK_CHARS_PER_TOKEN` in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py). For example, older GPT-3 models using p50k_base may need `WORDS_PER_TOKEN = 0.6` and `CJK_CHARS_PER_TOKEN = 2.0`. The modular configuration allows tuning without changing estimation logic.