# How Accurate Is Token Savings Estimation and How Is It Calculated?

> Discover the accuracy of token savings estimation and get the calculation details. Learn how code-review-graph uses a conservative 4 characters per token rule or precise tiktoken verification.

- Repository: [Tirth Kanani/code-review-graph](https://github.com/tirth8205/code-review-graph)
- Tags: performance
- Published: 2026-07-30

---

**The `code-review-graph` library estimates token savings using a conservative 4‑characters‑per‑token rule that guarantees an upper bound on usage, with optional verification via `tiktoken` for precise measurements.**

The `tirth8205/code-review-graph` repository implements a lightweight **token savings estimation** system designed to measure context reduction when converting raw code files into graph-based representations for LLM code review. Understanding how this estimation works—and its accuracy compared to real tokenizers—is essential for evaluating cost savings and context window management.

## How Token Savings Estimation Is Calculated

The estimation logic resides in [`code_review_graph/context_savings.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/context_savings.py) and follows a three-tier approach: character-based approximation for general content, byte-based sizing for files, and differential calculation for final savings metrics.

### The 4-Characters-Per-Token Rule

At the heart of the system is the `estimate_tokens()` function (lines 13‑33), which applies a simple **conservative heuristic**: `CHARS_PER_TOKEN = 4`. 

Any input value—whether a string or JSON‑serialized object—is converted to text, measured for length, divided by 4, and rounded up with `math.ceil()`. This guarantees at least one token for any non‑empty input, ensuring the estimate never under‑reports actual token consumption.

```python
from code_review_graph.context_savings import estimate_tokens

# String input

print(estimate_tokens("hello world"))  # → 3 tokens (11 chars / 4 = 2.75 → ceil → 3)

# JSON-serializable object

print(estimate_tokens({"key": "value", "list": [1, 2, 3]}))  # → 18 tokens

```

### Estimating File Size via Bytes

For file‑level calculations, `estimate_file_tokens()` (lines 35‑49) optimizes for speed by using **`os.stat().st_size`** rather than reading file contents into memory. The byte count is divided by the same 4‑character constant, treating one byte as approximately one character for ASCII‑heavy source code.

This approach avoids I/O overhead when processing large repositories while maintaining the conservative upper‑bound property.

### Core Savings Calculation Logic

The `estimate_context_savings()` function (lines 54‑82) computes savings by comparing baseline against returned context:

1. **Baseline tokens**: Either supplied directly or calculated via `estimate_tokens()` on the original context
2. **Returned tokens**: Calculated similarly on the graph‑generated response
3. **Savings**: `baseline - returned`
4. **Percentage**: `saved / baseline * 100`

```python
from code_review_graph.context_savings import estimate_context_savings

original = {"files": ["src/app.py", "src/models.py"], "content": "..."}
returned = {"changed_functions": ["app.main"], "summary": "..."}

result = estimate_context_savings(
    original_context=original,
    returned_context=returned
)

# Returns dict with tokens_baseline, tokens_returned, tokens_saved, percent_saved

```

## Accuracy and Verification with TikToken

While the 4‑character rule provides fast estimates, the library offers **optional verification** against real tokenizers to validate accuracy.

### Optional TikToken Verification

The `verify_with_tiktoken()` function (lines 85‑98) performs exact token counting by:

- Reading actual file contents from disk
- Encoding with the specified `tiktoken` encoding (default: **`cl100k_base`**)
- Computing exact counts for both raw files and the JSON‑serialized response
- Returning verified metrics: `verified_baseline`, `verified_returned`, `verified_saved`, and `verified_percent`

```python
from pathlib import Path
from code_review_graph.context_savings import verify_with_tiktoken

repo = Path("/path/to/repo")
changed = ["src/module.py", "README.md"]
response = {"changed_functions": ["func1"], "review": "..."}

verified = verify_with_tiktoken(repo, changed, response, encoding="cl100k_base")
print(verified)

# {'verified_baseline': 1240, 'verified_returned': 560, 

#  'verified_saved': 680, 'verified_percent': 55}

```

### Conservative Upper Bound Guarantee

Analysis of the verification step confirms that the 4‑character estimate is **deliberately conservative**. It will never under‑estimate the true token count, but may over‑estimate by a modest margin—typically a few percent for English code and comments. When `tiktoken` is unavailable, the estimate functions as a reliable **upper bound** for API cost projections.

## Integration Points

The estimation system integrates with the evaluation pipeline through specific components:

- **[`code_review_graph/eval/runner.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/runner.py)**: Attaches token savings metadata to tool responses during benchmark execution
- **[`code_review_graph/eval/reporter.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/reporter.py)**: Renders formatted token‑savings panels for CLI output, displaying both estimated and verified metrics when available

## Summary

- **Token savings estimation** in `code-review-graph` uses a **`CHARS_PER_TOKEN = 4`** rule implemented in `estimate_tokens()` and `estimate_file_tokens()`
- The 4‑character heuristic provides a **fast, conservative upper bound** that never under‑reports actual token usage
- **`estimate_context_savings()`** calculates percentage savings by comparing baseline and returned token counts
- **`verify_with_tiktoken()`** offers **exact measurement** using the `cl100k_base` encoding (or other specified encodings) when precision is required
- File size estimation uses **`st_size`** bytes for performance, avoiding file reads while maintaining accuracy for ASCII source code

## Frequently Asked Questions

### Is the 4-character-per-token estimate reliable for cost calculations?

Yes, according to the source code in [`code_review_graph/context_savings.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/context_savings.py), the 4‑character rule is specifically designed as a **conservative upper bound**. It will always estimate equal to or higher than the actual token count, making it safe for budget planning. Real‑world measurements with `tiktoken` typically show the estimate exceeds actual usage by only a few percent for standard codebases.

### Why does the library use file size instead of reading content for token estimation?

The `estimate_file_tokens()` function uses **`os.path.getsize()`** (returning `st_size`) rather than reading files to minimize I/O overhead when processing large repositories. Since source code is predominantly ASCII, bytes roughly equal characters, and the 4‑character rule already builds in conservatism, this approach balances speed with acceptable accuracy.

### How much can the estimate differ from actual tiktoken counts?

The difference is typically modest—usually within **5‑10%** for English language code and comments. The estimate always errs on the side of over‑estimation, meaning it functions as a worst‑case scenario. When precise accounting is required, the `verify_with_tiktoken()` function provides exact counts using the actual tokenizer.

### What happens if tiktoken is not installed?

The library gracefully handles missing dependencies. If `tiktoken` is unavailable, the system falls back entirely to the 4‑character estimation method. All core functionality—including savings percentage calculation—continues to work, providing a reliable (if slightly less precise) upper bound on token consumption without requiring additional dependencies.