How book-to-skill Estimates Token Count for Different Character Sets: Implementation Guide

book-to-skill estimates token counts using a deterministic heuristic that applies a ratio of 0.75 words per token for Latin-script text and 1.5 characters per token for CJK (Chinese, Japanese, Korean) text, mimicking the behavior of the cl100k_base tokenizer.

The book-to-skill repository implements a lightweight token estimation system designed to predict usage costs and processing requirements without requiring heavy dependencies. Understanding how it differentiates between character sets is essential for developers working with multilingual book processing pipelines or LLM context window management.

Configuration Constants for Token Ratios

The estimation logic relies on two key configuration constants defined in book_to_skill/config.py at lines 31-35. These values approximate the compression ratios observed in modern BPE tokenizers:

• WORDS_PER_TOKEN = 0.75 — Assumes approximately three-quarters of a token per word for English and other Latin-script languages.
• CJK_CHARS_PER_TOKEN = 1.5 — Assumes roughly 1.5 CJK characters consume one token, reflecting the denser encoding requirements for logographic scripts.

These ratios align with the empirical behavior of OpenAI's cl100k_base encoding, allowing the library to provide consistent estimates across diverse linguistic content.

Detecting CJK Character Sets

Before applying ratios, the system must classify the input text. In book_to_skill/utils.py (lines 59-81), a compiled regular expression named _CJK_RE identifies CJK code-points. The pattern covers the Unified Ideograph blocks including supplementary planes (U+20000–U+3FFFF) and Kangxi radicals, ensuring accurate detection of Chinese, Japanese, and Korean characters regardless of Unicode plane.

This regex is pre-compiled for performance, as the estimation function may process large book extracts repeatedly during skill extraction workflows.

The Token Estimation Algorithm

The core function estimate_tokens in book_to_skill/utils.py (lines 84-100) implements a branching logic that handles empty inputs, pure Latin text, pure CJK text, and mixed content.

Handling Latin-Script Text

When _CJK_RE finds no matches, the algorithm treats the entire string as Latin-script. It splits the text on whitespace and applies the word-to-token ratio:

int(len(text.split()) / WORDS_PER_TOKEN)

This approach assumes average English word lengths and punctuation density typical of prose content.

Handling CJK Text

For strings containing CJK code-points, the function counts the total CJK characters and divides by CJK_CHARS_PER_TOKEN. This direct character-to-token mapping avoids word-boundary detection complexities inherent to CJK scripts, which often lack spaces between words.

Mixed Character Set Processing

For documents combining both scripts—such as technical books with English terminology and Chinese annotations—the algorithm hybridizes both approaches. It substitutes CJK characters with spaces to isolate Latin words, then combines both estimates:

latin_words = len(_CJK_RE.sub(" ", text).split())
return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)

This ensures that code-switching content receives accurate predictions without double-counting or omitting either character set.

Fallback to BPE-Based Counting

While the heuristic provides speed, some tooling in the repository prioritizes precision. In tools/discovery_tax.py (lines 51-58), the helper function count_tokens attempts to use the actual byte-pair encoding (BPE) tokenizer via the optional tiktoken library.

The function first queries tiktoken.get_encoding("cl100k_base"). If the import fails—indicating tiktoken is not installed—it seamlessly falls back to the same estimate_tokens heuristic described above. This dual-mode operation ensures consistent token estimates across the project regardless of dependency availability.

Practical Code Examples

The following implementations demonstrate the algorithm's behavior across different character sets:

from book_to_skill.utils import estimate_tokens

# Latin-script example (16 words)

latin = "The quick brown fox jumps over the lazy dog."
print(estimate_tokens(latin))      # → 12 (16 / 0.75)

# Pure CJK example (1500 Chinese characters)

cjk = "中" * 1500
print(estimate_tokens(cjk))        # → 1000 (1500 / 1.5)

# Mixed example combining both character sets

mixed = "Hello 世界 " * 100
print(estimate_tokens(mixed))      # → 267 (100 Latin words / 0.75 + 100 CJK chars / 1.5)

These examples illustrate how the system proportionally weights each character set based on the configured ratios.

Summary

• book-to-skill estimates token counts using heuristic ratios stored in config.py, specifically WORDS_PER_TOKEN = 0.75 and CJK_CHARS_PER_TOKEN = 1.5.
• The _CJK_RE regex in utils.py detects CJK code-points across all Unicode planes to trigger character-based counting.
• The estimate_tokens function handles pure Latin text, pure CJK text, and mixed content by applying the appropriate ratio to each segment and summing the results.
• An optional BPE-based fallback in tools/discovery_tax.py uses tiktoken when available, falling back to the heuristic when the library is absent.

Frequently Asked Questions

Why does book-to-skill use different ratios for CJK and Latin text?

CJK logographic scripts encode more information per character than Latin alphabetic scripts, causing tokenizers like cl100k_base to consume them at different rates. The 1.5:1 ratio for CJK versus the 0.75:1 ratio for Latin reflects that CJK characters typically require more token space per semantic unit than English words.

How accurate is the heuristic compared to real tokenization?

The heuristic provides estimates within 10-15% of actual cl100k_base counts for standard prose, making it suitable for cost estimation and context window planning. However, code, mathematical notation, or repetitive patterns may diverge significantly from these averages, which is why the repository provides the optional tiktoken fallback for precision-critical applications.

Can I adjust the token ratios for my specific use case?

Yes. Since WORDS_PER_TOKEN and CJK_CHARS_PER_TOKEN are defined as module-level constants in book_to_skill/config.py, you can modify these values before importing the estimation functions. Adjusting these ratios allows calibration against specific tokenizer behaviors or proprietary encoding schemes.

What happens if tiktoken is not installed?

The library operates without errors. Functions like count_tokens in the tools directory catch import failures and automatically route requests to estimate_tokens. This ensures the repository remains functional in lightweight environments where installing the full tiktoken dependency is undesirable or impossible.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}