# How Book-to-Skill Handles Different Character Sets for Token Estimation

> Discover how Book-to-Skill expertly estimates tokens across Latin, CJK, and Unicode character sets with its adaptable layered approach, ensuring accurate tokenization for all your text.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-31

---

**Book-to-Skill estimates tokens across Latin, CJK, and other Unicode character sets using a layered approach: preferring exact `tiktoken` BPE counts when available, otherwise falling back to a deterministic heuristic that treats CJK characters as individual tokens and non-CJK text with a words-to-tokens ratio.**

Deterministic token estimation is critical for cost-aware LLM workflows. The virgiliojr94/book-to-skill library implements `estimate_tokens()` in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) to predict token consumption before sending text to an API. The design prioritizes accuracy when dependencies permit, and falls back to a fast, language-agnostic heuristic that handles East Asian scripts correctly.

## The Two-Layer Estimation Strategy

### Layer 1: Exact BPE Tokenization with tiktoken

When the optional `tiktoken` package is installed, `estimate_tokens()` calls `tiktoken.get_encoding("cl100k_base")` to compute the precise token count that OpenAI's cl100k encoder would produce. This approach works correctly for any Unicode input without special-casing character sets.

### Layer 2: Deterministic Heuristic Without Dependencies

If `tiktoken` is unavailable (the default in CI environments), the function switches to a custom heuristic that distinguishes **CJK characters** from **non-CJK text**:

- **CJK characters (Chinese, Japanese, Korean)** — Each individual CJK code point counts as **one token**, whether it appears in the Basic Multilingual Plane (BMP) or a supplementary plane. The implementation caps the ratio at 0.66 tokens per character, so 1,500 Chinese characters estimates to roughly 1,000 tokens.

- **Non-CJK text** — Uses the classic words-to-tokens conversion:

```python
tokens ≈ int(len(text.split()) / 0.75)  # ≈ 4/3 tokens per word

```

This yields the historically-pinned **100 words → 133 tokens** mapping verified in [`tests/test_book_to_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_book_to_skill.py).

The final estimate returns the **maximum** of the CJK character count and the word-based heuristic. This ensures documents heavy in CJK characters do not receive underestimated token counts.

## Code Examples

### English-Only Text (Word-Based Heuristic)

```python
from book_to_skill.utils import estimate_tokens

english = " ".join(["word"] * 100)  # 100 words

print(estimate_tokens(english))     # → 133

```

### Pure Chinese Text (Character-Based Counting)

```python
chinese = "中" * 1500
print(estimate_tokens(chinese))     # → ~1000 (exactly 1000 in test suite)

```

### Mixed English and Chinese (Maximum of Both Estimates)

```python
mixed = ("hello 世界 " * 100).strip()
print(estimate_tokens(mixed))       # → > 100 (CJK part counted directly)

```

### Automatic tiktoken Detection

No code change is required. The same `estimate_tokens()` function automatically switches to `tiktoken` when installed:

```python

# With tiktoken installed: returns exact BPE count

# Without tiktoken installed: returns heuristic estimate

estimate_tokens("任何文本 works identically")  # Works for any Unicode input

```

## Test Coverage for Character Set Handling

The token estimation logic is validated across multiple test files:

| Test File | Coverage |
|-----------|----------|
| [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py) | BMP and supplementary-plane CJK characters |
| [`tests/test_book_to_skill.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_book_to_skill.py) | 100-word → 133-token mapping verification |
| [`tests/test_discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_discovery_tax.py) | Fallback behavior when `tiktoken` is absent |
| [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) | Demonstrates heuristic in standalone scripts |

## Implementation Details in Source Files

- **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** — Contains the core `estimate_tokens()` function implementing both layers
- **[`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py)** — Validates that characters like 𠀀 (U+20000, outside BMP) are handled correctly
- **[`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py)** — Shows practical usage of the fallback heuristic

## Summary

Book-to-Skill handles different character sets for token estimation through:

- **Primary path**: Exact BPE tokenization via `tiktoken.get_encoding("cl100k_base")` when available
- **Fallback path**: Deterministic heuristic counting CJK characters individually and non-CJK text at ~1.33 tokens per word
- **Safety mechanism**: `max(CJK_count, word_estimate)` prevents underestimation for mixed or CJK-heavy documents
- **Unicode coverage**: Correct handling of BMP and supplementary plane characters verified in dedicated tests

## Frequently Asked Questions

### What happens if tiktoken is not installed?

The `estimate_tokens()` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) automatically detects the absence of `tiktoken` and switches to the deterministic heuristic. No exception is raised, and estimation continues with CJK-aware character counting for accurate results across languages.

### Why does the heuristic treat CJK characters differently?

BPE tokenizers typically encode CJK text with one token per character or less, unlike English where one word often splits into multiple tokens. The 0.66 tokens-per-character cap and character-level counting align estimates with observed behavior in production tokenizers like cl100k_base.

### Does the supplementary plane CJK handling actually work?

Yes. The test suite in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py) explicitly validates characters outside the Basic Multilingual Plane (code points above U+FFFF). The implementation correctly identifies these as CJK and counts them as individual tokens, matching the behavior for BMP CJK characters.

### Can I force the heuristic even with tiktoken installed?

The current implementation in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) does not expose a flag to bypass tiktoken. To force heuristic usage, you would need to temporarily uninstall or shadow the `tiktoken` module, or modify the source to add a parameter controlling layer selection.