Code Review Graph Benchmark Results for Token Efficiency vs Whole-Corpus Reading

The code-review-graph tool achieves a median 65× token reduction compared to whole-corpus reading, with reductions ranging from 36× to 376× across six real-world repositories.

The code-review-graph open-source project dramatically reduces LLM token consumption by replacing whole-repository context with targeted graph queries. Understanding the benchmark results for token efficiency helps developers evaluate the cost savings of structured code review versus naïvely feeding entire source trees to language models.

How Token Efficiency Is Measured

The benchmark methodology, documented in docs/REPRODUCING.md, establishes two distinct measurements:

1. Whole-corpus baseline: The raw token count required to feed every source file of a repository to an LLM
2. Graph-query cost: The token count of the structured response returned by the graph (search hits plus neighbor edges) for a typical review question

The evaluation runs across six real-world open-source projects totaling 13 commits, comparing the full-corpus baseline against graph-query costs using deterministic embeddings and pinned upstream SHAs.

Benchmark Results for Token Efficiency

The results demonstrate dramatic reductions in token usage when using the graph-based approach versus whole-corpus reading.

Repository	Snapshot SHA	Naïve-Corpus Tokens	Avg Graph Tokens	Reduction
fastapi	22381558	948,793	2,653	375.6×
flask	a29f88ce	143,594	2,196	71.0×
code-review-graph	84bde354	208,821	3,190	68.1×
gin	5c00df8a	166,868	2,766	61.9×
httpx	b55d4635	142,356	2,661	60.6×
express	b4ab7d65	136,052	3,936	36.0×

Source: README.md Benchmarks section

Key Metrics

According to the benchmark data published in the README, the median per-question token reduction is approximately 65×. The efficiency gains range from 36× (Express) to 376× (FastAPI), with larger codebases like FastAPI showing the most dramatic savings due to their substantial whole-corpus token counts.

Technical Architecture Behind Token Savings

The token efficiency stems from the tool's structural extraction approach. Rather than processing raw text, code-review-graph uses Tree-sitter to extract functions, classes, calls, and imports into a compact SQLite database.

At query time, the system returns only directly relevant nodes plus a limited "blast-radius" of neighbors (default ≤500 nodes). This typically yields responses of approximately 2,000–3,500 tokens instead of the 100,000-plus tokens required for complete source trees.

The context_savings metadata attached to MCP responses records exact token savings for each call, making the benefit transparent to downstream agents.

Running the Token Efficiency Benchmark

You can reproduce these benchmark results for token efficiency using the provided scripts and modules.

To run the full benchmark suite:

code-review-graph eval

For the standalone token comparison:

python -m code_review_graph.token_benchmark

Programmatically, invoke the benchmark using:

from code_review_graph.eval.benchmarks.token_efficiency import run_token_efficiency

store = ...          # GraphStore from code_review_graph.main.build_graph()

repo = ...           # Repository object for the target

results = run_token_efficiency(store, repo)

print(f"Corpus tokens: {results['naive_corpus_tokens']}")
print(f"Graph tokens : {results['avg_graph_tokens']}")
print(f"Reduction    : {results['avg_graph_tokens'] / results['naive_corpus_tokens']:.2f}×")

The formal benchmark implementation in code_review_graph/eval/benchmarks/token_efficiency.py compares the full get_review_context() JSON output against changed-file content to generate the reduction ratios.

Summary

• code-review-graph achieves a median 65× token reduction compared to whole-corpus reading across six test repositories.
• Token savings range from 36× to 376×, with larger projects like FastAPI showing the highest efficiency gains.
• The benchmark methodology measures raw corpus tokens against graph-query response tokens, documented in docs/REPRODUCING.md.
• Structural extraction via Tree-sitter and limited blast-radius queries (≤500 nodes) enable the compact 2,000–3,500 token responses.
• Reproducibility scripts in code_review_graph/token_benchmark.py and code_review_graph/eval/benchmarks/token_efficiency.py allow independent verification of results.

Frequently Asked Questions

What is the median token reduction achieved by code-review-graph?

The benchmark results for token efficiency show a median reduction of approximately 65× when comparing graph-based queries to whole-corpus reading. This means that for a typical code review question, the tool uses roughly 1/65th of the tokens required to process the entire repository.

Which repository showed the highest token efficiency in the benchmarks?

FastAPI demonstrated the highest efficiency with a 375.6× reduction, reducing 948,793 corpus tokens to just 2,653 graph tokens. This represents the maximum savings in the test suite, though results vary by codebase size and structure.

How can I reproduce these token efficiency benchmarks?

Run code-review-graph eval for the full suite or python -m code_review_graph.token_benchmark for standalone testing. The reproducibility documentation in docs/REPRODUCING.md specifies pinned SHAs and deterministic embeddings to ensure consistent results across machines.

What causes the variation in token savings between different repositories?

The variation depends on the ratio between total repository size and the localized context needed for review questions. Larger projects like FastAPI (948K tokens) yield higher absolute savings than smaller ones like Express (136K tokens), though both achieve significant reductions over naïve corpus feeding.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}