# How book-to-skill Estimates Tokens for Different Book Types and CJK Characters

> Discover how book-to-skill calculates token estimates using distinct ratios for Latin text and CJK characters with a Unicode regex pattern.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: how-to-guide
- Published: 2026-08-25

---

**The `book-to-skill` project uses a deterministic heuristic that applies different ratios for Latin-script text (`WORDS_PER_TOKEN = 0.75`) and CJK characters (`CJK_CHARS_PER_TOKEN = 1.5`), detecting CJK code points with a comprehensive Unicode regex pattern.**

Token estimation is critical for pre-flight cost calculations when processing books with large language models. The `virgiliojr94/book-to-skill` repository implements a fast, dependency-free approach that handles English PDFs, Chinese novels, and mixed-language documents without requiring the `tiktoken` library or API calls.

## Core Configuration Constants

Two ratios defined in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)** drive the entire calculation:

| Constant | Value | Use Case |
|----------|-------|----------|
| `WORDS_PER_TOKEN` | `0.75` | Latin-script, whitespace-delimited text (≈133 tokens per 100 words)【[`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py)†L31-L33】 |
| `CJK_CHARS_PER_TOKEN` | `1.5` | CJK code points where whitespace is absent or minimal【[`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py)†L33-L35】 |

These values approximate the behavior of OpenAI's `cl100k_base` tokenizer without the overhead of actual tokenization.

## CJK Detection with Unicode Regex

The `estimate_tokens` function relies on `_CJK_RE`, a compiled regular expression at the top of **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)**【[`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py)†L59-L77】. This pattern matches:

- Basic Multilingual Plane (BMP) CJK ranges
- Full-width forms
- Supplementary-plane range U+20000–U+3FFFF

The supplementary-plane coverage ensures rare historical and variant CJK characters are counted correctly—verified by dedicated tests in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py).

## The estimate_tokens Algorithm

The `estimate_tokens` function in **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** implements a branching heuristic【[`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py)†L80-L96】:

1. **Empty input** → returns `0`

2. **CJK count extraction** → `cjk = len(_CJK_RE.findall(text))`

3. **Latin-only path** (when `cjk == 0`):

```python
return int(len(text.split()) / WORDS_PER_TOKEN)

```

4. **Mixed or CJK-heavy path**:

```python
latin_words = len(_CJK_RE.sub(" ", text).split())
return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)

```

The substitution step (`_CJK_RE.sub(" ", text)`) isolates Latin words by replacing CJK characters with spaces, then both components are summed with their respective ratios.

## Working Code Examples

```python
from book_to_skill.utils import estimate_tokens

# Pure English: 8 words → ~10.6 → 10 tokens

english = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(english))

# Output: 10

# Pure Chinese: 1500 ideographs

chinese = "中" * 1500
print(estimate_tokens(chinese))

# Output: 1000 (1500 / 1.5)

# Mixed: 100 English words + 200 Chinese characters

mixed = "Hello 世界 " * 100
print(estimate_tokens(mixed))

# Output: 200 (133 + 67)

```

## Why Separate Handling Matters

**Latin-only books** benefit from the classic word-to-token ratio without per-character processing overhead.

**CJK-only books** avoid catastrophic undercounting. A whitespace-split on a Chinese novel would yield ~1 token for the entire text; the CJK-specific ratio provides realistic estimates aligned with actual tokenizer behavior.

**Mixed-language books** receive accurate accounting for both scripts simultaneously, essential for academic texts, translated works, and technical documentation with embedded terminology.

## Summary

- Token estimation lives in **[`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)** with configuration in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)**
- Two ratios handle fundamentally different writing systems: `0.75` for words, `1.5` for CJK characters
- CJK detection uses a comprehensive regex covering supplementary-plane characters
- The algorithm treats mixed text as separate Latin and CJK components
- No external tokenization dependencies required

## Frequently Asked Questions

### How accurate is the CJK token estimation compared to tiktoken?

The `CJK_CHARS_PER_TOKEN = 1.5` ratio approximates OpenAI's `cl100k_base` tokenizer behavior without the library dependency. Actual token counts may vary by ±10-15% depending on character frequency and specific Unicode blocks, but the estimate is sufficient for cost pre-flight calculations.

### What happens with languages that use both scripts, like Japanese with Latin loanwords?

The mixed-language path in `estimate_tokens` handles this automatically. CJK characters are counted directly, Latin/whitespace segments are word-split, and both contributions are summed with their respective ratios.

### Does the supplementary-plane regex affect performance?

Minimal impact. The compiled `_CJK_RE` pattern uses efficient Unicode range matching, and the function operates in O(n) time relative to text length. For typical book-sized inputs, estimation completes in milliseconds.

### Can I adjust the ratios for different tokenizers?

Yes—modify `WORDS_PER_TOKEN` and `CJK_CHARS_PER_TOKEN` in **[`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)**. For example, some GPT-4 deployments with different tokenizers may benefit from `CJK_CHARS_PER_TOKEN = 2.0` for conservative estimates.