# How book-to-skill Estimates Token Count for Different Character Sets: Implementation Guide

> Discover how book-to-skill estimates token count for Latin and CJK text using a deterministic heuristic mimicking the cl100k_base tokenizer. Implement token counting efficiently.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: implementation-guide
- Published: 2026-08-30

---

**book-to-skill estimates token counts using a deterministic heuristic that applies a ratio of 0.75 words per token for Latin-script text and 1.5 characters per token for CJK (Chinese, Japanese, Korean) text, mimicking the behavior of the cl100k_base tokenizer.**

The `book-to-skill` repository implements a lightweight token estimation system designed to predict usage costs and processing requirements without requiring heavy dependencies. Understanding how it differentiates between character sets is essential for developers working with multilingual book processing pipelines or LLM context window management.

## Configuration Constants for Token Ratios

The estimation logic relies on two key configuration constants defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) at lines 31-35. These values approximate the compression ratios observed in modern BPE tokenizers:

- **`WORDS_PER_TOKEN = 0.75`** — Assumes approximately three-quarters of a token per word for English and other Latin-script languages.
- **`CJK_CHARS_PER_TOKEN = 1.5`** — Assumes roughly 1.5 CJK characters consume one token, reflecting the denser encoding requirements for logographic scripts.

These ratios align with the empirical behavior of OpenAI's `cl100k_base` encoding, allowing the library to provide consistent estimates across diverse linguistic content.

## Detecting CJK Character Sets

Before applying ratios, the system must classify the input text. In [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 59-81), a compiled regular expression named `_CJK_RE` identifies CJK code-points. The pattern covers the Unified Ideograph blocks including supplementary planes (U+20000–U+3FFFF) and Kangxi radicals, ensuring accurate detection of Chinese, Japanese, and Korean characters regardless of Unicode plane.

This regex is pre-compiled for performance, as the estimation function may process large book extracts repeatedly during skill extraction workflows.

## The Token Estimation Algorithm

The core function `estimate_tokens` in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 84-100) implements a branching logic that handles empty inputs, pure Latin text, pure CJK text, and mixed content.

### Handling Latin-Script Text

When `_CJK_RE` finds no matches, the algorithm treats the entire string as Latin-script. It splits the text on whitespace and applies the word-to-token ratio:

```python
int(len(text.split()) / WORDS_PER_TOKEN)

```

This approach assumes average English word lengths and punctuation density typical of prose content.

### Handling CJK Text

For strings containing CJK code-points, the function counts the total CJK characters and divides by `CJK_CHARS_PER_TOKEN`. This direct character-to-token mapping avoids word-boundary detection complexities inherent to CJK scripts, which often lack spaces between words.

### Mixed Character Set Processing

For documents combining both scripts—such as technical books with English terminology and Chinese annotations—the algorithm hybridizes both approaches. It substitutes CJK characters with spaces to isolate Latin words, then combines both estimates:

```python
latin_words = len(_CJK_RE.sub(" ", text).split())
return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)

```

This ensures that code-switching content receives accurate predictions without double-counting or omitting either character set.

## Fallback to BPE-Based Counting

While the heuristic provides speed, some tooling in the repository prioritizes precision. In [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) (lines 51-58), the helper function `count_tokens` attempts to use the actual **byte-pair encoding (BPE)** tokenizer via the optional `tiktoken` library.

The function first queries `tiktoken.get_encoding("cl100k_base")`. If the import fails—indicating `tiktoken` is not installed—it seamlessly falls back to the same `estimate_tokens` heuristic described above. This dual-mode operation ensures consistent token estimates across the project regardless of dependency availability.

## Practical Code Examples

The following implementations demonstrate the algorithm's behavior across different character sets:

```python
from book_to_skill.utils import estimate_tokens

# Latin-script example (16 words)

latin = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(latin))      # → 12 (16 / 0.75)

# Pure CJK example (1500 Chinese characters)

cjk = "中" * 1500
print(estimate_tokens(cjk))        # → 1000 (1500 / 1.5)

# Mixed example combining both character sets

mixed = "Hello 世界 " * 100
print(estimate_tokens(mixed))      # → 267 (100 Latin words / 0.75 + 100 CJK chars / 1.5)

```

These examples illustrate how the system proportionally weights each character set based on the configured ratios.

## Summary

- **book-to-skill** estimates token counts using heuristic ratios stored in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py), specifically `WORDS_PER_TOKEN = 0.75` and `CJK_CHARS_PER_TOKEN = 1.5`.
- The `_CJK_RE` regex in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) detects CJK code-points across all Unicode planes to trigger character-based counting.
- The `estimate_tokens` function handles pure Latin text, pure CJK text, and mixed content by applying the appropriate ratio to each segment and summing the results.
- An optional **BPE-based fallback** in [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) uses `tiktoken` when available, falling back to the heuristic when the library is absent.

## Frequently Asked Questions

### Why does book-to-skill use different ratios for CJK and Latin text?

CJK logographic scripts encode more information per character than Latin alphabetic scripts, causing tokenizers like `cl100k_base` to consume them at different rates. The 1.5:1 ratio for CJK versus the 0.75:1 ratio for Latin reflects that CJK characters typically require more token space per semantic unit than English words.

### How accurate is the heuristic compared to real tokenization?

The heuristic provides estimates within 10-15% of actual `cl100k_base` counts for standard prose, making it suitable for cost estimation and context window planning. However, code, mathematical notation, or repetitive patterns may diverge significantly from these averages, which is why the repository provides the optional `tiktoken` fallback for precision-critical applications.

### Can I adjust the token ratios for my specific use case?

Yes. Since `WORDS_PER_TOKEN` and `CJK_CHARS_PER_TOKEN` are defined as module-level constants in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), you can modify these values before importing the estimation functions. Adjusting these ratios allows calibration against specific tokenizer behaviors or proprietary encoding schemes.

### What happens if tiktoken is not installed?

The library operates without errors. Functions like `count_tokens` in the tools directory catch import failures and automatically route requests to `estimate_tokens`. This ensures the repository remains functional in lightweight environments where installing the full `tiktoken` dependency is undesirable or impossible.