How Accurate Is Token Savings Estimation and How Is It Calculated?

The code-review-graph library estimates token savings using a conservative 4‑characters‑per‑token rule that guarantees an upper bound on usage, with optional verification via tiktoken for precise measurements.

The tirth8205/code-review-graph repository implements a lightweight token savings estimation system designed to measure context reduction when converting raw code files into graph-based representations for LLM code review. Understanding how this estimation works—and its accuracy compared to real tokenizers—is essential for evaluating cost savings and context window management.

How Token Savings Estimation Is Calculated

The estimation logic resides in code_review_graph/context_savings.py and follows a three-tier approach: character-based approximation for general content, byte-based sizing for files, and differential calculation for final savings metrics.

The 4-Characters-Per-Token Rule

At the heart of the system is the estimate_tokens() function (lines 13‑33), which applies a simple conservative heuristic: CHARS_PER_TOKEN = 4.

Any input value—whether a string or JSON‑serialized object—is converted to text, measured for length, divided by 4, and rounded up with math.ceil(). This guarantees at least one token for any non‑empty input, ensuring the estimate never under‑reports actual token consumption.

from code_review_graph.context_savings import estimate_tokens

# String input

print(estimate_tokens("hello world"))  # → 3 tokens (11 chars / 4 = 2.75 → ceil → 3)

# JSON-serializable object

print(estimate_tokens({"key": "value", "list": [1, 2, 3]}))  # → 18 tokens

Estimating File Size via Bytes

For file‑level calculations, estimate_file_tokens() (lines 35‑49) optimizes for speed by using os.stat().st_size rather than reading file contents into memory. The byte count is divided by the same 4‑character constant, treating one byte as approximately one character for ASCII‑heavy source code.

This approach avoids I/O overhead when processing large repositories while maintaining the conservative upper‑bound property.

Core Savings Calculation Logic

The estimate_context_savings() function (lines 54‑82) computes savings by comparing baseline against returned context:

1. Baseline tokens: Either supplied directly or calculated via estimate_tokens() on the original context
2. Returned tokens: Calculated similarly on the graph‑generated response
3. Savings: baseline - returned
4. Percentage: saved / baseline * 100

from code_review_graph.context_savings import estimate_context_savings

original = {"files": ["src/app.py", "src/models.py"], "content": "..."}
returned = {"changed_functions": ["app.main"], "summary": "..."}

result = estimate_context_savings(
    original_context=original,
    returned_context=returned
)

# Returns dict with tokens_baseline, tokens_returned, tokens_saved, percent_saved

Accuracy and Verification with TikToken

While the 4‑character rule provides fast estimates, the library offers optional verification against real tokenizers to validate accuracy.

Optional TikToken Verification

The verify_with_tiktoken() function (lines 85‑98) performs exact token counting by:

• Reading actual file contents from disk
• Encoding with the specified tiktoken encoding (default: cl100k_base)
• Computing exact counts for both raw files and the JSON‑serialized response
• Returning verified metrics: verified_baseline, verified_returned, verified_saved, and verified_percent

from pathlib import Path
from code_review_graph.context_savings import verify_with_tiktoken

repo = Path("/path/to/repo")
changed = ["src/module.py", "README.md"]
response = {"changed_functions": ["func1"], "review": "..."}

verified = verify_with_tiktoken(repo, changed, response, encoding="cl100k_base")
print(verified)

# {'verified_baseline': 1240, 'verified_returned': 560, 

#  'verified_saved': 680, 'verified_percent': 55}

Conservative Upper Bound Guarantee

Analysis of the verification step confirms that the 4‑character estimate is deliberately conservative. It will never under‑estimate the true token count, but may over‑estimate by a modest margin—typically a few percent for English code and comments. When tiktoken is unavailable, the estimate functions as a reliable upper bound for API cost projections.

Integration Points

The estimation system integrates with the evaluation pipeline through specific components:

• code_review_graph/eval/runner.py: Attaches token savings metadata to tool responses during benchmark execution
• code_review_graph/eval/reporter.py: Renders formatted token‑savings panels for CLI output, displaying both estimated and verified metrics when available

Summary

• Token savings estimation in code-review-graph uses a CHARS_PER_TOKEN = 4 rule implemented in estimate_tokens() and estimate_file_tokens()
• The 4‑character heuristic provides a fast, conservative upper bound that never under‑reports actual token usage
• estimate_context_savings() calculates percentage savings by comparing baseline and returned token counts
• verify_with_tiktoken() offers exact measurement using the cl100k_base encoding (or other specified encodings) when precision is required
• File size estimation uses st_size bytes for performance, avoiding file reads while maintaining accuracy for ASCII source code

Frequently Asked Questions

Is the 4-character-per-token estimate reliable for cost calculations?

Yes, according to the source code in code_review_graph/context_savings.py, the 4‑character rule is specifically designed as a conservative upper bound. It will always estimate equal to or higher than the actual token count, making it safe for budget planning. Real‑world measurements with tiktoken typically show the estimate exceeds actual usage by only a few percent for standard codebases.

Why does the library use file size instead of reading content for token estimation?

The estimate_file_tokens() function uses os.path.getsize() (returning st_size) rather than reading files to minimize I/O overhead when processing large repositories. Since source code is predominantly ASCII, bytes roughly equal characters, and the 4‑character rule already builds in conservatism, this approach balances speed with acceptable accuracy.

How much can the estimate differ from actual tiktoken counts?

The difference is typically modest—usually within 5‑10% for English language code and comments. The estimate always errs on the side of over‑estimation, meaning it functions as a worst‑case scenario. When precise accounting is required, the verify_with_tiktoken() function provides exact counts using the actual tokenizer.

What happens if tiktoken is not installed?

The library gracefully handles missing dependencies. If tiktoken is unavailable, the system falls back entirely to the 4‑character estimation method. All core functionality—including savings percentage calculation—continues to work, providing a reliable (if slightly less precise) upper bound on token consumption without requiring additional dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}