Running Evaluation Benchmarks and Interpreting Results in code-review-graph

The code-review-graph repository provides a reproducible evaluation framework that measures token efficiency and impact accuracy across real-world repositories, outputting structured CSV results that can be aggregated into markdown reports.

Running evaluation benchmarks in code-review-graph allows you to validate the efficiency and accuracy of graph-based code review context retrieval against baseline methods. The framework ships with pre-configured YAML specifications for popular open-source repositories and automated CLI tooling to orchestrate the entire pipeline from cloning to report generation.

The Evaluation Framework Architecture

The evaluation system is built around three core components defined in code_review_graph/eval/runner.py: a Benchmark Registry mapping names to implementation functions (lines 28-36), YAML-based Benchmark Configs that pin specific commit SHAs for reproducibility (lines 62-77), and the Evaluation Runner that orchestrates execution (lines 47-74).

Benchmark Registry and Configuration

The BENCHMARK_REGISTRY dictionary in runner.py dynamically routes benchmark names to their corresponding Python modules. Each benchmark is a callable that receives a repository path, a GraphStore instance, and a configuration dictionary.

Benchmark configurations are YAML files stored in code_review_graph/eval/configs/. Each config specifies:

• The repository URL and exact commit SHA
• A list of test commits for evaluation
• Metadata for reproducibility

The load_config() and load_all_configs() functions (lines 62-77) hydrate these YAML files into Python dictionaries used by the runner.

How to Run Evaluation Benchmarks

The framework exposes a CLI command that wraps the core run_eval() function. This handles repository cloning, graph construction, benchmark execution, and CSV serialization.

Basic CLI Usage

Install the evaluation extras to ensure all dependencies are available:

pip install "code-review-graph[eval]"

Run the complete benchmark suite across all configured repositories:

code-review-graph eval

Execute specific benchmarks on selected repositories:

code-review-graph eval --repos fastapi --benchmarks token_efficiency

Redirect output to a custom directory:

code-review-graph eval --output-dir ./my-results

The Five-Stage Execution Pipeline

When you invoke code-review-graph eval, the run_eval() function executes these stages:

1. Configuration Loading – Loads YAML configs for specified repos (or all repos if none specified)
2. Repository Cloning – clone_or_update() performs full clones (no --depth flag) at exact commits to ensure reproducibility (lines 80-91 in runner.py)
3. Graph Construction – Instantiates GraphStore from code_review_graph/graph.py and calls full_build() to parse the entire repository
4. Benchmark Execution – Iterates over benchmark_names (defaults to all keys in BENCHMARK_REGISTRY) and invokes each benchmark function (e.g., token_efficiency.run)
5. Result Persistence – write_csv() (lines 35-44) serializes results to CSV files in the output directory (default: evaluate/results)

Key Benchmarks Explained

The framework currently implements two primary benchmarks: Token Efficiency and Impact Accuracy. Each benchmark is implemented as a standalone module in code_review_graph/eval/benchmarks/.

Token Efficiency Benchmark

Located in code_review_graph/eval/benchmarks/token_efficiency.py, this benchmark measures context reduction by comparing three token-counting strategies:

• Naïve count: Summing tokens from full changed files using _count_file_tokens
• Standard count: Counting tokens in the raw git diff via _count_diff_tokens
• Graph count: Serializing the JSON output from get_review_context and counting tokens with _count_tokens

The token approximation uses a simple heuristic: len(text) // 4 (lines 21-24). The benchmark calculates two critical ratios:

• naive_to_graph_ratio: How many times smaller the graph context is compared to reading full files
• standard_to_graph_ratio: How the graph compares to standard diff-based approaches

If get_review_context raises an exception, the row is marked with status="error" and excluded from aggregate statistics (lines 9-15). The aggregate() function (lines 25-43) computes median and mean ratios across successful runs.

Impact Accuracy Benchmark

Implemented in code_review_graph/eval/benchmarks/impact_accuracy.py, this benchmark measures the precision and recall of blast-radius impact analysis. According to the README, the current implementation reports an F1 score of approximately 0.71, with recall=1.0 representing a graph-derived upper bound rather than true recall.

Interpreting Results and Metrics

After execution, the framework generates one CSV file per repository per benchmark (e.g., evaluate/results/fastapi_token_efficiency.csv). These CSVs contain raw measurements that can be aggregated into high-level insights.

Understanding the Numbers

The README documents headline metrics derived from these CSVs:

• Token Reduction: The median across six evaluated repositories shows an 82× reduction compared to naïve full-corpus reads (with the fastest repository achieving 528× reduction)
• Impact Accuracy: F1 ~0.71 indicates the precision of the graph-based impact detection
• Build Performance: Node counts, edge counts, and search latency metrics characterize graph construction overhead

Generating Human-Readable Reports

Convert raw CSVs into markdown tables matching the README format using the reporter module:

python -m code_review_graph.eval.reporter ./evaluate/results > BENCHMARK_REPORT.md

The generate_report() function in code_review_graph/eval/reporter.py (lines 15-34) parses the CSV files and formats aggregated statistics into publication-ready tables.

Programmatic Benchmark Execution

You can invoke benchmarks directly from Python for custom analysis or integration with existing pipelines:

from pathlib import Path
from code_review_graph.graph import GraphStore
from code_review_graph.incremental import full_build, get_db_path
from code_review_graph.eval.benchmarks import token_efficiency

# Initialize repository path and graph store

repo = Path("/tmp/fastapi")
store = GraphStore(get_db_path(repo))

# Build the knowledge graph

full_build(repo, store)

# Define test configuration

config = {
    "name": "fastapi",
    "test_commits": [{"sha": "0227991a", "description": "example"}],
}

# Execute benchmark

results = token_efficiency.run(repo, store, config)
summary = token_efficiency.aggregate(results)
print(summary)

This approach is useful for debugging specific commits or integrating the evaluation into CI/CD pipelines where CLI invocation is undesirable.

Summary

• The evaluation framework centers on code_review_graph/eval/runner.py, which orchestrates the complete benchmark lifecycle from configuration to CSV output.
• Token efficiency benchmarks in token_efficiency.py demonstrate 82× median token reduction by comparing graph-based context retrieval against full-file and diff-based baselines.
• Results are persisted as CSV files in evaluate/results/ and can be transformed into markdown reports using code_review_graph/eval/reporter.py.
• The CLI command code-review-graph eval accepts --repos and --benchmarks filters for targeted evaluation, while the underlying run_eval() function provides programmatic access.
• All benchmarks use pinned commit SHAs from YAML configs to ensure fully reproducible results across different environments.

Frequently Asked Questions

How do I run a specific benchmark on a single repository?

Use the --repos and --benchmarks flags with the CLI: code-review-graph eval --repos fastapi --benchmarks token_efficiency. This filters the benchmark registry to execute only the specified benchmark against the named repository configuration, skipping all other repos and tests.

What does the "naive_to_graph_ratio" metric mean in token efficiency results?

This ratio compares the token count of reading entire changed files versus using the graph-based get_review_context output. A ratio of 82× means the graph approach consumes 82 times fewer tokens than reading full files, indicating significant efficiency gains for LLM context windows.

Where are benchmark results stored after execution?

By default, results are written to the evaluate/results/ directory as CSV files named {repo}_{benchmark}.csv. You can override this location using the --output-dir CLI argument or the output_dir parameter when calling run_eval() programmatically.

How is token count approximated in the evaluation framework?

The framework uses a character-to-token ratio heuristic defined in token_efficiency.py (lines 21-24): def _count_tokens(text: str) -> int: return len(text) // 4. This approximates that one token equals approximately four characters, which provides a language-agnostic estimation suitable for comparing relative efficiency across different context retrieval methods.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}