# Running Evaluation Benchmarks and Interpreting Results in code-review-graph

> Learn to run evaluation benchmarks and interpret results in code-review-graph. Discover how this framework measures token efficiency and impact accuracy for real-world repositories.

- Repository: [Tirth Kanani/code-review-graph](https://github.com/tirth8205/code-review-graph)
- Tags: performance
- Published: 2026-07-18

---

**The code-review-graph repository provides a reproducible evaluation framework that measures token efficiency and impact accuracy across real-world repositories, outputting structured CSV results that can be aggregated into markdown reports.**

Running evaluation benchmarks in code-review-graph allows you to validate the efficiency and accuracy of graph-based code review context retrieval against baseline methods. The framework ships with pre-configured YAML specifications for popular open-source repositories and automated CLI tooling to orchestrate the entire pipeline from cloning to report generation.

## The Evaluation Framework Architecture

The evaluation system is built around three core components defined in [`code_review_graph/eval/runner.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/runner.py): a **Benchmark Registry** mapping names to implementation functions (lines 28-36), YAML-based **Benchmark Configs** that pin specific commit SHAs for reproducibility (lines 62-77), and the **Evaluation Runner** that orchestrates execution (lines 47-74).

### Benchmark Registry and Configuration

The `BENCHMARK_REGISTRY` dictionary in [`runner.py`](https://github.com/tirth8205/code-review-graph/blob/main/runner.py) dynamically routes benchmark names to their corresponding Python modules. Each benchmark is a callable that receives a repository path, a `GraphStore` instance, and a configuration dictionary.

Benchmark configurations are YAML files stored in `code_review_graph/eval/configs/`. Each config specifies:
- The repository URL and exact commit SHA
- A list of test commits for evaluation
- Metadata for reproducibility

The `load_config()` and `load_all_configs()` functions (lines 62-77) hydrate these YAML files into Python dictionaries used by the runner.

## How to Run Evaluation Benchmarks

The framework exposes a CLI command that wraps the core `run_eval()` function. This handles repository cloning, graph construction, benchmark execution, and CSV serialization.

### Basic CLI Usage

Install the evaluation extras to ensure all dependencies are available:

```bash
pip install "code-review-graph[eval]"

```

Run the complete benchmark suite across all configured repositories:

```bash
code-review-graph eval

```

Execute specific benchmarks on selected repositories:

```bash
code-review-graph eval --repos fastapi --benchmarks token_efficiency

```

Redirect output to a custom directory:

```bash
code-review-graph eval --output-dir ./my-results

```

### The Five-Stage Execution Pipeline

When you invoke `code-review-graph eval`, the `run_eval()` function executes these stages:

1. **Configuration Loading** – Loads YAML configs for specified repos (or all repos if none specified)
2. **Repository Cloning** – `clone_or_update()` performs full clones (no `--depth` flag) at exact commits to ensure reproducibility (lines 80-91 in [`runner.py`](https://github.com/tirth8205/code-review-graph/blob/main/runner.py))
3. **Graph Construction** – Instantiates `GraphStore` from [`code_review_graph/graph.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/graph.py) and calls `full_build()` to parse the entire repository
4. **Benchmark Execution** – Iterates over `benchmark_names` (defaults to all keys in `BENCHMARK_REGISTRY`) and invokes each benchmark function (e.g., `token_efficiency.run`)
5. **Result Persistence** – `write_csv()` (lines 35-44) serializes results to CSV files in the output directory (default: `evaluate/results`)

## Key Benchmarks Explained

The framework currently implements two primary benchmarks: **Token Efficiency** and **Impact Accuracy**. Each benchmark is implemented as a standalone module in `code_review_graph/eval/benchmarks/`.

### Token Efficiency Benchmark

Located in [`code_review_graph/eval/benchmarks/token_efficiency.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/benchmarks/token_efficiency.py), this benchmark measures context reduction by comparing three token-counting strategies:

- **Naïve count**: Summing tokens from full changed files using `_count_file_tokens`
- **Standard count**: Counting tokens in the raw git diff via `_count_diff_tokens`
- **Graph count**: Serializing the JSON output from `get_review_context` and counting tokens with `_count_tokens`

The token approximation uses a simple heuristic: `len(text) // 4` (lines 21-24). The benchmark calculates two critical ratios:
- `naive_to_graph_ratio`: How many times smaller the graph context is compared to reading full files
- `standard_to_graph_ratio`: How the graph compares to standard diff-based approaches

If `get_review_context` raises an exception, the row is marked with `status="error"` and excluded from aggregate statistics (lines 9-15). The `aggregate()` function (lines 25-43) computes median and mean ratios across successful runs.

### Impact Accuracy Benchmark

Implemented in [`code_review_graph/eval/benchmarks/impact_accuracy.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/benchmarks/impact_accuracy.py), this benchmark measures the precision and recall of blast-radius impact analysis. According to the README, the current implementation reports an F1 score of approximately 0.71, with recall=1.0 representing a graph-derived upper bound rather than true recall.

## Interpreting Results and Metrics

After execution, the framework generates one CSV file per repository per benchmark (e.g., `evaluate/results/fastapi_token_efficiency.csv`). These CSVs contain raw measurements that can be aggregated into high-level insights.

### Understanding the Numbers

The README documents headline metrics derived from these CSVs:

- **Token Reduction**: The median across six evaluated repositories shows an 82× reduction compared to naïve full-corpus reads (with the fastest repository achieving 528× reduction)
- **Impact Accuracy**: F1 ~0.71 indicates the precision of the graph-based impact detection
- **Build Performance**: Node counts, edge counts, and search latency metrics characterize graph construction overhead

### Generating Human-Readable Reports

Convert raw CSVs into markdown tables matching the README format using the reporter module:

```bash
python -m code_review_graph.eval.reporter ./evaluate/results > BENCHMARK_REPORT.md

```

The `generate_report()` function in [`code_review_graph/eval/reporter.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/reporter.py) (lines 15-34) parses the CSV files and formats aggregated statistics into publication-ready tables.

## Programmatic Benchmark Execution

You can invoke benchmarks directly from Python for custom analysis or integration with existing pipelines:

```python
from pathlib import Path
from code_review_graph.graph import GraphStore
from code_review_graph.incremental import full_build, get_db_path
from code_review_graph.eval.benchmarks import token_efficiency

# Initialize repository path and graph store

repo = Path("/tmp/fastapi")
store = GraphStore(get_db_path(repo))

# Build the knowledge graph

full_build(repo, store)

# Define test configuration

config = {
    "name": "fastapi",
    "test_commits": [{"sha": "0227991a", "description": "example"}],
}

# Execute benchmark

results = token_efficiency.run(repo, store, config)
summary = token_efficiency.aggregate(results)
print(summary)

```

This approach is useful for debugging specific commits or integrating the evaluation into CI/CD pipelines where CLI invocation is undesirable.

## Summary

- The evaluation framework centers on [`code_review_graph/eval/runner.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/runner.py), which orchestrates the complete benchmark lifecycle from configuration to CSV output.
- **Token efficiency** benchmarks in [`token_efficiency.py`](https://github.com/tirth8205/code-review-graph/blob/main/token_efficiency.py) demonstrate 82× median token reduction by comparing graph-based context retrieval against full-file and diff-based baselines.
- Results are persisted as CSV files in `evaluate/results/` and can be transformed into markdown reports using [`code_review_graph/eval/reporter.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/eval/reporter.py).
- The CLI command `code-review-graph eval` accepts `--repos` and `--benchmarks` filters for targeted evaluation, while the underlying `run_eval()` function provides programmatic access.
- All benchmarks use pinned commit SHAs from YAML configs to ensure fully reproducible results across different environments.

## Frequently Asked Questions

### How do I run a specific benchmark on a single repository?

Use the `--repos` and `--benchmarks` flags with the CLI: `code-review-graph eval --repos fastapi --benchmarks token_efficiency`. This filters the benchmark registry to execute only the specified benchmark against the named repository configuration, skipping all other repos and tests.

### What does the "naive_to_graph_ratio" metric mean in token efficiency results?

This ratio compares the token count of reading entire changed files versus using the graph-based `get_review_context` output. A ratio of 82× means the graph approach consumes 82 times fewer tokens than reading full files, indicating significant efficiency gains for LLM context windows.

### Where are benchmark results stored after execution?

By default, results are written to the `evaluate/results/` directory as CSV files named `{repo}_{benchmark}.csv`. You can override this location using the `--output-dir` CLI argument or the `output_dir` parameter when calling `run_eval()` programmatically.

### How is token count approximated in the evaluation framework?

The framework uses a character-to-token ratio heuristic defined in [`token_efficiency.py`](https://github.com/tirth8205/code-review-graph/blob/main/token_efficiency.py) (lines 21-24): `def _count_tokens(text: str) -> int: return len(text) // 4`. This approximates that one token equals approximately four characters, which provides a language-agnostic estimation suitable for comparing relative efficiency across different context retrieval methods.