How to Run Tests and Evaluate Retrieval Quality for New Languages in code-graph-rag

Run the full test suite with pytest -n auto and benchmark CALLS-edge retrieval for any language by declaring constants in evals/constants.py, implementing an oracle that emits ground-truth edges, and registering a new CLI command in evals/cli.py.

The vitali87/code-graph-rag repository provides a comprehensive evaluation framework that validates both core library correctness and per-language retrieval accuracy. Whether you are extending support to Kotlin, Ruby, or another target, you must first ensure the existing test suite passes, then implement a language-specific oracle that compares the graph’s CALLS edges against a ground-truth source. This guide walks through the exact file paths and function signatures required to integrate a new language into the evaluation pipeline.

Install the Development and Test Stack

Before running any tests or evaluations, install the optional test dependencies declared in [pyproject.toml](https://github.com/vitali87/code-graph-rag/blob/main/pyproject.toml).

Using uv (recommended):

git clone https://github.com/vitali87/code-graph-rag.git
cd code-graph-rag
uv sync --group test

Using pip:

pip install -r requirements.txt
pip install "code-graph-rag[test]"

This pulls in pytest-asyncio, testcontainers, and parallelization libraries required by the suite.

Run the Generic Test Suite

Execute the full collection to verify that core graph operations work correctly across all existing language front-ends.

pytest -n auto

The parallelized suite (configured via the test extra in pyproject.toml) validates:

• Graph capture and ingestion – codebase_rag/tests/test_capture_*
• Incremental update correctness – codebase_rag/tests/test_incremental_*
• Dead-code detection – codebase_rag/tests/test_dead_code_*
• Import resolution and inheritance – codebase_rag/tests/test_imports_*

All tests must pass before you add a new language; they guarantee that the query engine behaves correctly regardless of the specific tree-sitter grammar in use.

Benchmark Retrieval Quality for a New Language

Adding a language requires five concrete steps: constants declaration, oracle implementation, retrieval driver creation, CLI registration, and execution.

Extend the Constants File

Declare language-specific metadata in [evals/constants.py](https://github.com/vitali87/code-graph-rag/blob/main/evals/constants.py). For a hypothetical Kotlin addition, append definitions for file suffixes, output filenames, and edge formatting templates:

# In evals/constants.py

KOTLIN_DEFAULT_TARGET = "."
KOTLIN_RETRIEVAL_SCORES_FILENAME = "kotlin_retrieval_scores.csv"
KOTLIN_RETRIEVAL_DIFF_FILENAME = "kotlin_retrieval_diff.json"
KOTLIN_RETRIEVAL_DIFF_PREFIX = "kotlin-retrieval:"
KOTLIN_RETRIEVAL_LABEL = "graph"
KOTLIN_RETRIEVAL_TITLE = "cgr multi-language retrieval: Kotlin CALLS vs kotlin-cli oracle"
KOTLIN_CALL_EDGE_REPR = "{file} -> {name}"

These constants are referenced throughout the evaluation code via the ec alias (from . import constants as ec) and control CSV headers, JSON keys, and CLI help strings.

Implement the Oracle

Create a module (e.g., evals/kotlin_oracle.py) that exports two functions: an availability check and a ground-truth generator. The oracle must return only first-party symbols and file-level CALL edges (caller file path + callee simple name).

import shutil
from pathlib import Path
from typing import tuple, set, frozenset

from . import constants as ec

CallEdge = tuple[str, str]

def kotlin_available() -> bool:
    """Return True if the external kotlin-cli binary is on PATH."""
    return shutil.which(ec.KOTLIN_CLI_BIN) is not None

def run_kotlin_call_oracle(target: Path) -> tuple[set[CallEdge], frozenset[str]]:
    """
    Execute the external oracle and parse its JSON output into:
    - A set of (caller_path, callee_name) edges
    - A frozenset of declared symbols (used to filter third-party calls)
    """
    # Implementation similar to evals/go_retrieval.py lines 31-53

    # ... parse external tool output ...

    return edges, declared_symbols

See the Go implementation in [evals/go_retrieval.py](https://github.com/vitali87/code-graph-rag/blob/main/evals/go_retrieval.py) (lines 31-53) for the exact edge-parsing pattern.

Create the Retrieval Driver

Add a new file (e.g., evals/kotlin_retrieval.py) that orchestrates the comparison. The driver must implement three core functions: oracle_kotlin_call_edges, cgr_kotlin_call_edges, and score_kotlin_retrieval.

from pathlib import Path
from typing import Annotated

import typer
from loguru import logger

from codebase_rag import constants as cs
from . import constants as ec
from . import logs as ls
from .cgr_graph import _capture
from .kotlin_oracle import kotlin_available, run_kotlin_call_oracle
from .score import _prf
from .structure_report import render, write_outputs
from .types_defs import DiffBucket, LocationStats, ScoreResult

console_target = Path(ec.KOTLIN_DEFAULT_TARGET)
_CALLS = cs.RelationshipType.CALLS.value
CallEdge = tuple[str, str]

def oracle_kotlin_call_edges(target: Path) -> tuple[set[CallEdge], frozenset[str]]:
    return run_kotlin_call_oracle(target)

def cgr_kotlin_call_edges(target: Path, project: str, declared: frozenset[str]) -> set[CallEdge]:
    """Capture the graph and extract CALL edges for Kotlin files."""
    ingestor = _capture(target, project)
    
    # Map (label, uid) → source file path for .kt files

    caller_path = {
        (str(label), str(uid)): str(props[cs.KEY_PATH])
        for (label, uid), props in ingestor.nodes.items()
        if props.get(cs.KEY_PATH) and str(props[cs.KEY_PATH]).endswith(".kt")
    }
    
    edges: set[CallEdge] = set()
    for from_label, from_val, rel_type, _to_label, to_val in ingestor.rels:
        if rel_type != _CALLS:
            continue
        path = caller_path.get((str(from_label), str(from_val)))
        if not path:
            continue
        name = str(to_val).split(cs.SEPARATOR_DOT)[-1]
        if name in declared:
            edges.add((path, name))
    return edges

def _edge_repr(edge: CallEdge) -> str:
    return ec.KOTLIN_CALL_EDGE_REPR.format(file=edge[0], name=edge[1])

def score_kotlin_retrieval(cgr: set[CallEdge], oracle: set[CallEdge]) -> ScoreResult:
    """Calculate precision, recall, and F1; generate diff buckets."""
    rows, diff = [], {}
    row = _prf(ec.Category.RETRIEVAL.value, ec.KOTLIN_RETRIEVAL_LABEL, cgr, oracle)
    if row:
        rows.append(row)
        diff[ec.KOTLIN_RETRIEVAL_DIFF_PREFIX + ec.KOTLIN_RETRIEVAL_LABEL] = DiffBucket(
            missing=[_edge_repr(e) for e in sorted(oracle - cgr)],
            extra=[_edge_repr(e) for e in sorted(cgr - oracle)],
        )
    return ScoreResult(rows=rows, location=LocationStats(0, 0, 0, 0.0, 0), diff=diff)

def main(
    target: Annotated[Path, typer.Option(help="Kotlin source directory")] = console_target,
    project_name: Annotated[str, typer.Option(help="cgr project name")] = "",
    out_dir: Annotated[Path, typer.Option(help="Output directory")] = Path(ec.DEFAULT_OUT_DIR),
):
    if not kotlin_available():
        logger.error(ls.KOTLIN_ORACLE_MISSING)
        raise typer.Exit(code=1)

    target = target.resolve()
    project = project_name or target.name

    logger.info(f"Running oracle on {target}")
    oracle_edges, declared = oracle_kotlin_call_edges(target)
    
    logger.info(f"Capturing graph for {project}")
    cgr_edges = cgr_kotlin_call_edges(target, project, declared)
    
    result = score_kotlin_retrieval(cgr_edges, oracle_edges)
    write_outputs(result, out_dir, ec.KOTLIN_RETRIEVAL_SCORES_FILENAME, ec.KOTLIN_RETRIEVAL_DIFF_FILENAME)
    render(result, ec.KOTLIN_RETRIEVAL_TITLE)

if __name__ == "__main__":
    typer.run(main)

The _capture helper (from [codebase_rag/cgr_graph.py](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/cgr_graph.py)) initializes the ingestor, while _prf (from [evals/score.py](https://github.com/vitali87/code-graph-rag/blob/main/evals/score.py)) computes precision, recall, and F1 metrics.

Register the CLI Command

Wire the new benchmark into the evaluation CLI by editing [evals/cli.py](https://github.com/vitali87/code-graph-rag/blob/main/evals/cli.py):

# evals/cli.py

from .kotlin_retrieval import main as kotlin_main

# After the Typer app definition

app.command()(kotlin_main)

This exposes the command as cgr eval kotlin-retrieval (or whatever naming convention you choose).

Execute the Benchmark

Run the retrieval evaluation against a local codebase:

cgr eval kotlin-retrieval --target path/to/kotlin/project \
                          --project-name my_kotlin_proj \
                          --out-dir evals/results

The command produces two artifacts:

• kotlin_retrieval_scores.csv – Contains precision, recall, and F1 scores formatted according to constants.CSV_FIELDS.
• kotlin_retrieval_diff.json – Lists missing and extra edges for debugging discrepancies between the graph and the oracle.

Verify and Iterate

Review the console output (rendered by cli._render) which prints TP/FP/FN counts alongside the F1 score. If metrics are below threshold, iterate on the tree-sitter grammar or parser adapters in the main library until the CALLS-edge retrieval matches the oracle’s ground truth. Use the JSON diff to identify specific missing or spurious edges (e.g., src/main.kt -> processUser).

Summary

• Install test dependencies using uv sync --group test or the equivalent pip command to access pytest and plugins.
• Run the generic suite with pytest -n auto to confirm core graph operations (capture, incremental updates, dead-code detection) function correctly.
• Declare language constants in evals/constants.py to define output filenames, CLI labels, and edge string templates.
• Implement an oracle that returns (caller_file, callee_name) tuples and a set of declared first-party symbols.
• Build a retrieval driver that captures the graph via _capture, filters for CALL edges, and compares against the oracle using _prf.
• Register the command in evals/cli.py and execute via cgr eval <language>-retrieval to generate CSV scores and JSON diffs.

Frequently Asked Questions

What does the oracle need to return for a valid comparison?

The oracle must return a set of tuples where each tuple contains (caller_file_path, callee_simple_name) representing a CALL relationship, plus a frozenset of declared symbols used to filter out third-party library calls. The evaluation logic in score.py compares these ground-truth edges against the edges extracted from the code-graph-rag ingestor to calculate precision and recall.

Can I run evaluations for multiple languages in a single command?

No, each language has a distinct CLI subcommand (e.g., go-retrieval, scala-retrieval, kotlin-retrieval) registered in evals/cli.py. You must invoke each separately, though you can script a loop that runs cgr eval <lang>-retrieval across multiple projects and aggregates the resulting CSV files.

Where does the graph capture logic live?

The _capture function used in retrieval drivers is defined in [codebase_rag/cgr_graph.py](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/cgr_graph.py). It initializes a codebase_rag ingestor, walks the target directory, and populates nodes and relationships (including the CALLS edges) that your evaluation code will later query.

How do I interpret the retrieval scores CSV?

The CSV file generated by write_outputs contains columns defined in evals/constants.CSV_FIELDS, including precision (ratio of correct edges to total retrieved), recall (ratio of correct edges to total oracle edges), and F1 (harmonic mean of the two). A high F1 indicates that the graph accurately captures the call structure for your new language compared to the external oracle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}