# Book-to-Skill Strategies for Estimating Tokens in Different Text Types

> Discover Book-to-Skill's token estimation strategies for diverse text types. Learn about its dual heuristic approach and fallback options for accurate language processing.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-29

---

**Book-to-Skill employs a dual heuristic approach that divides word counts by 0.75 for Latin text and CJK character counts by 1.5 for Asian scripts, with an optional fallback to the exact `tiktoken` BPE tokenizer when available.**

The `virgiliojr94/book-to-skill` repository needs deterministic token counts to forecast API costs and manage context window limits when processing books and documents. It implements language-aware **strategies for estimating tokens** that avoid heavy dependencies while maintaining accuracy across different writing systems.

## Heuristic-Based Estimation Strategy

Book-to-Skill categorizes text into two processing paths based on the presence of CJK (Chinese, Japanese, Korean) characters. This approach accounts for the fundamental difference in tokenization density between whitespace-delimited languages and logographic scripts.

### Latin and Whitespace-Delimited Text

For text containing only Latin characters and whitespace, the system applies a deterministic word-to-token ratio. The implementation in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) splits the text on whitespace and divides the word count by the `WORDS_PER_TOKEN` constant defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py).

**Key implementation details**:
- The constant `WORDS_PER_TOKEN = 0.75` represents the approximate inverse of the average tokens per word in English prose
- This ratio aligns with OpenAI's `cl100k_base` tokenizer behavior for common Latin-script content
- The function `len(text.split())` provides the word count basis

### CJK Character Handling

When the input contains CJK code points, Book-to-Skill switches to a character-based estimation model. The repository uses a compiled regex `_CJK_RE` to identify characters in the CJK Unicode blocks.

**The calculation strategy**:
- Each CJK character counts as approximately `1 / 1.5` tokens based on the `CJK_CHARS_PER_TOKEN = 1.5` constant
- For mixed content, the system separates Latin words (with CJK characters substituted by spaces) and adds the CJK character count separately
- This handles supplementary plane characters correctly, as validated in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py)

## Optional Exact Tokenization with Tiktoken

For tooling that requires precise token counts rather than estimates, [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) provides a real BPE tokenizer implementation.

**The `count_tokens` function** attempts to import `tiktoken` and encode text using the `cl100k_base` encoding:

```python
def count_tokens(text: str) -> int:
    """Real BPE count via tiktoken if available; else words/0.75 heuristic."""
    try:
        import tiktoken
        enc = tiktoken.get_encoding("cl100k_base")
        return len(enc.encode(text))
    except Exception:
        # Fallback to the deterministic heuristic

        return int(len(text.split()) / WORDS_PER_TOKEN)

```

If the `tiktoken` library is not installed or the import fails, the function falls back to the standard word-count heuristic, ensuring the tool remains functional without optional dependencies.

## Configuration Constants and Sources

The estimation ratios live in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) and serve as the single source of truth for both the core utilities and discovery tools:

- `WORDS_PER_TOKEN = 0.75` — Calibrated for Latin-based text
- `CJK_CHARS_PER_TOKEN = 1.5` — Calibrated for CJK ideographic content

These constants allow users to adjust estimation aggressiveness globally without modifying the calculation logic.

## Core Implementation Files

### book_to_skill/utils.py

The primary `estimate_tokens` function implements the dual-path logic. It first checks for CJK characters using the compiled regex `_CJK_RE`. If none exist, it returns `len(text.split()) / WORDS_PER_TOKEN`. For mixed or pure CJK content, it calculates:

```python
def estimate_tokens(text: str) -> int:
    """Estimate the token count of ``text`` with a deterministic heuristic."""
    if not text:
        return 0
    cjk = len(_CJK_RE.findall(text))
    if not cjk:
        return int(len(text.split()) / WORDS_PER_TOKEN)
    latin_words = len(_CJK_RE.sub(" ", text).split())
    return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)

```

This implementation ensures that a sentence like `"Hello 世界"` correctly separates the Latin word ("Hello") from the CJK characters ("世界") and applies the appropriate ratio to each segment.

### tools/discovery_tax.py

This analysis script exposes the `count_tokens` wrapper function. It prioritizes accuracy over speed by attempting to use the actual OpenAI tokenizer when available, making it suitable for auditing the heuristic's accuracy or processing small text samples where exact counts matter.

## Practical Usage Examples

### Estimating Tokens for English Prose

For pure Latin text, the heuristic provides fast, battery-friendly estimates without importing heavy tokenization libraries:

```python
from book_to_skill.utils import estimate_tokens

english = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(english))   # ≈ 13 tokens (13 words / 0.75)

```

### Calculating CJK Content Costs

For Chinese, Japanese, or Korean content where characters dominate over whitespace, the character-based ratio prevents underestimation:

```python
from book_to_skill.utils import estimate_tokens

chinese = "中" * 1500   # 1500 Chinese characters, no spaces

print(estimate_tokens(chinese))  # ≈ 1000 tokens (1500 / 1.5)

```

### Using Real BPE Tokenization

When `tiktoken` is installed, developers can obtain exact token counts for billing reconciliation:

```python
from tools.discovery_tax import count_tokens

mixed_text = "Hello, 世界!"
print(count_tokens(mixed_text))  # Accurate token count from tiktoken encoder

```

## Summary

- **Book-to-Skill uses a dual heuristic** that applies `0.75` words-per-token for Latin text and `1.5` characters-per-token for CJK content
- **Configuration constants** reside in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) and control both estimation paths
- **The `estimate_tokens` function** in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) handles mixed-language content by separating Latin words from CJK characters
- **Optional exact counting** is available through [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) when the `tiktoken` library is present
- **Test coverage** in [`tests/test_cjk_supplementary_plane.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tests/test_cjk_supplementary_plane.py) ensures accurate handling of extended Unicode planes

## Frequently Asked Questions

### How accurate is the heuristic compared to OpenAI's tokenizer?

The heuristic provides estimates within 10-20% of actual `cl100k_base` token counts for most content. The constant values (`0.75` for words, `1.5` for CJK) were chosen to match the average density observed in OpenAI's tokenizer, though exact counts require the optional `tiktoken` dependency.

### Why does Book-to-Skill use different ratios for CJK and Latin text?

CJK characters typically consume fewer bytes per token in modern BPE tokenizers compared to Latin words. While a Latin word averages 1.3 tokens, a CJK character averages approximately 0.67 tokens. The ratios `0.75` and `1.5` represent the inverse of these densities to convert character/word counts into estimated token quantities.

### What happens if tiktoken is not installed?

The system gracefully degrades to the heuristic estimator. In [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), the `count_tokens` function catches import exceptions and falls back to `int(len(text.split()) / WORDS_PER_TOKEN)`, ensuring all functionality remains available without the heavy tokenization library.

### Where are the token estimation constants defined?

The constants `WORDS_PER_TOKEN` and `CJK_CHARS_PER_TOKEN` are defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py). Centralizing these values allows users to adjust estimation parameters globally without modifying the calculation logic in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) or [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py).