Token Savings Estimation in Code-Review-Graph: How It Works and Its Accuracy

The code-review-graph repository estimates token savings by comparing raw prompt token counts against graph-optimized counts using a character-count heuristic, providing approximate savings percentages suitable for workflow previews but not precise billing calculations.

The tirth8205/code-review-graph project implements a token savings estimation system that quantifies how much LLM context is preserved when using graph-based code review workflows instead of naive full-context approaches. This estimation powers the "Real-time token savings" metrics shown in the CLI and helps developers understand the efficiency gains of the repository's change-detection algorithms.

How Token Savings Estimation Works

The estimation pipeline follows a three-stage process that converts arbitrary data into comparable token metrics.

The Core Heuristic in estimate_tokens

At the foundation lies the estimate_tokens function in code_review_graph/eval/token_benchmark.py. This utility serializes any JSON-compatible object to a string, counts the resulting characters, and divides by four—a standard approximation for English text and code tokenization.

from code_review_graph.eval.token_benchmark import estimate_tokens

# Estimate tokens for a minimal context payload

context = {"file": "src/app.py", "changes": ["line 42", "line 43"]}
token_count = estimate_tokens(context)  # Returns character_length // 4

The implementation assumes roughly four characters per token, which aligns with general-purpose LLM tokenizers but does not account for code-specific patterns or punctuation density.

Aggregating Workflow Token Counts

Higher-level benchmark wrappers such as benchmark_review_workflow, benchmark_architecture_workflow, and benchmark_debug_workflow instrument the actual review tools (get_minimal_context, detect_changes_func, etc.). These helpers feed every intermediate result through estimate_tokens and accumulate a running total representing the graph-optimized token spend.

Raw token counts are captured by simulating the same workflows without the graph pruning—essentially passing full file contents to the estimation function. This establishes the baseline against which savings are calculated.

Calculating Efficiency Metrics

The final computation occurs in code_review_graph/eval/scorer.py within the compute_token_efficiency function. This method accepts raw_tokens and graph_tokens integers and returns a structured dictionary:

from code_review_graph.eval.scorer import compute_token_efficiency

efficiency = compute_token_efficiency(raw_tokens=10000, graph_tokens=3000)
print(efficiency)

The returned payload includes:

• raw_tokens: Total tokens in the unoptimized workflow
• graph_tokens: Total tokens after graph-based minimization
• savings_ratio: (raw - graph) / raw as a float
• savings_percent: The same ratio expressed as a percentage

As validated in tests/test_eval.py, the function correctly handles the translation from absolute counts to relative savings metrics.

Accuracy and Limitations of the Estimation

Heuristic Approximation vs. Precise Tokenization

The character-count-divided-by-four method provides order-of-magnitude accuracy for English prose but deviates when processing code-heavy content. Actual LLM tokenizers (such as GPT-4's BPE or Claude's tokenizer) split on subword units and treat whitespace and punctuation differently than the simple character heuristic. Consequently, the estimation may undercount or overcount by 10–20% depending on the programming language and formatting style.

Empirical Validation Coverage

The test suite in tests/test_eval.py verifies that compute_token_efficiency returns the expected dictionary keys and that estimate_tokens produces integers. However, it does not enforce quantitative error bounds against a reference tokenizer, nor does it validate multi-byte Unicode handling. The tests confirm the calculation logic (e.g., that 10000 raw vs 3000 graph yields 70% savings) but not the absolute token accuracy.

Intended Use Case: Human-Readable Previews

According to the source, the token savings figure appears primarily in the CLI's --brief output flag for the detect-changes command. Because the metric is used for human-readable reporting rather than billing or strict rate-limit enforcement, the coarse granularity is acceptable. The repository treats these estimates as directional indicators—confirming that the graph optimization reduced context by roughly 70%—rather than as accounting data.

Summary

• Token estimation in tirth8205/code-review-graph relies on estimate_tokens, which calculates len(json.dumps(obj)) // 4 as a proxy for LLM tokens.
• Savings calculation is performed by compute_token_efficiency in scorer.py, returning ratios and percentages comparing raw vs. optimized workflows.
• Accuracy is approximate: The character-based heuristic ignores language-specific tokenization, making it suitable for relative comparisons but not precise budgeting.
• Validation is functional: Tests verify calculation correctness but do not bound estimation error against production tokenizers.
• Purpose is.preview: The metrics support the CLI's "Real-time token savings" display, helping developers visualize efficiency gains before submitting prompts.

Frequently Asked Questions

How does code-review-graph calculate token savings?

The repository calculates token savings by first estimating token counts for both full-context (raw) and graph-optimized workflows using estimate_tokens, then passing these integers to compute_token_efficiency. This function returns a dictionary containing savings_ratio and savings_percent derived from (raw_tokens - graph_tokens) / raw_tokens.

Is the token savings estimation accurate enough for billing?

No. The character-count-divided-by-four heuristic used in code_review_graph/eval/token_benchmark.py provides only rough approximations. Actual LLM billing relies on specific tokenizers that handle code punctuation and subword units differently, so this estimation should not be used for cost accounting or strict rate-limit checks.

Where does the character-count heuristic come from?

The division by four approximates the average characters-per-token ratio observed in English text and general code with OpenAI's tokenizer families. While this rule-of-thumb works for quick mental math, it is implemented in estimate_tokens purely for convenience and lacks calibration against specific model vocabularies.

Can I use the token estimation for non-English code?

You can use it, but accuracy degrades. The estimate_tokens function does not account for multi-byte Unicode characters or languages with different character-to-token density ratios. For repositories containing significant non-ASCII content, the estimation may substantially deviate from actual LLM token counts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}