# How to Run Tests and Evaluate Retrieval Quality for New Languages in code-graph-rag > Easily run tests and evaluate retrieval quality for new languages in code-graph-rag. Benchmark CALLS-edge retrieval by implementing an oracle and registering a new CLI command. - Repository: [Vitali Avagyan/code-graph-rag](https://github.com/vitali87/code-graph-rag) - Tags: how-to-guide - Published: 2026-08-13 --- **Run the full test suite with `pytest -n auto` and benchmark CALLS-edge retrieval for any language by declaring constants in [`evals/constants.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/constants.py), implementing an oracle that emits ground-truth edges, and registering a new CLI command in [`evals/cli.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/cli.py).** The `vitali87/code-graph-rag` repository provides a comprehensive evaluation framework that validates both core library correctness and per-language retrieval accuracy. Whether you are extending support to Kotlin, Ruby, or another target, you must first ensure the existing test suite passes, then implement a language-specific oracle that compares the graph’s **CALLS edges** against a ground-truth source. This guide walks through the exact file paths and function signatures required to integrate a new language into the evaluation pipeline. ## Install the Development and Test Stack Before running any tests or evaluations, install the optional test dependencies declared in [[`pyproject.toml`](https://github.com/vitali87/code-graph-rag/blob/main/pyproject.toml)](https://github.com/vitali87/code-graph-rag/blob/main/pyproject.toml). Using **uv** (recommended): ```bash git clone https://github.com/vitali87/code-graph-rag.git cd code-graph-rag uv sync --group test ``` Using **pip**: ```bash pip install -r requirements.txt pip install "code-graph-rag[test]" ``` This pulls in `pytest-asyncio`, `testcontainers`, and parallelization libraries required by the suite. ## Run the Generic Test Suite Execute the full collection to verify that core graph operations work correctly across all existing language front-ends. ```bash pytest -n auto ``` The **parallelized suite** (configured via the `test` extra in [`pyproject.toml`](https://github.com/vitali87/code-graph-rag/blob/main/pyproject.toml)) validates: - **Graph capture and ingestion** – `codebase_rag/tests/test_capture_*` - **Incremental update correctness** – `codebase_rag/tests/test_incremental_*` - **Dead-code detection** – `codebase_rag/tests/test_dead_code_*` - **Import resolution and inheritance** – `codebase_rag/tests/test_imports_*` All tests must pass before you add a new language; they guarantee that the query engine behaves correctly regardless of the specific tree-sitter grammar in use. ## Benchmark Retrieval Quality for a New Language Adding a language requires five concrete steps: constants declaration, oracle implementation, retrieval driver creation, CLI registration, and execution. ### Extend the Constants File Declare language-specific metadata in [[`evals/constants.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/constants.py)](https://github.com/vitali87/code-graph-rag/blob/main/evals/constants.py). For a hypothetical Kotlin addition, append definitions for file suffixes, output filenames, and edge formatting templates: ```python # In evals/constants.py KOTLIN_DEFAULT_TARGET = "." KOTLIN_RETRIEVAL_SCORES_FILENAME = "kotlin_retrieval_scores.csv" KOTLIN_RETRIEVAL_DIFF_FILENAME = "kotlin_retrieval_diff.json" KOTLIN_RETRIEVAL_DIFF_PREFIX = "kotlin-retrieval:" KOTLIN_RETRIEVAL_LABEL = "graph" KOTLIN_RETRIEVAL_TITLE = "cgr multi-language retrieval: Kotlin CALLS vs kotlin-cli oracle" KOTLIN_CALL_EDGE_REPR = "{file} -> {name}" ``` These constants are referenced throughout the evaluation code via the `ec` alias (`from . import constants as ec`) and control CSV headers, JSON keys, and CLI help strings. ### Implement the Oracle Create a module (e.g., [`evals/kotlin_oracle.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/kotlin_oracle.py)) that exports two functions: an availability check and a ground-truth generator. The oracle must return **only first-party symbols** and **file-level CALL edges** (caller file path + callee simple name). ```python import shutil from pathlib import Path from typing import tuple, set, frozenset from . import constants as ec CallEdge = tuple[str, str] def kotlin_available() -> bool: """Return True if the external kotlin-cli binary is on PATH.""" return shutil.which(ec.KOTLIN_CLI_BIN) is not None def run_kotlin_call_oracle(target: Path) -> tuple[set[CallEdge], frozenset[str]]: """ Execute the external oracle and parse its JSON output into: - A set of (caller_path, callee_name) edges - A frozenset of declared symbols (used to filter third-party calls) """ # Implementation similar to evals/go_retrieval.py lines 31-53 # ... parse external tool output ... return edges, declared_symbols ``` See the Go implementation in [[`evals/go_retrieval.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/go_retrieval.py)](https://github.com/vitali87/code-graph-rag/blob/main/evals/go_retrieval.py) (lines 31-53) for the exact edge-parsing pattern. ### Create the Retrieval Driver Add a new file (e.g., [`evals/kotlin_retrieval.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/kotlin_retrieval.py)) that orchestrates the comparison. The driver must implement three core functions: `oracle_kotlin_call_edges`, `cgr_kotlin_call_edges`, and `score_kotlin_retrieval`. ```python from pathlib import Path from typing import Annotated import typer from loguru import logger from codebase_rag import constants as cs from . import constants as ec from . import logs as ls from .cgr_graph import _capture from .kotlin_oracle import kotlin_available, run_kotlin_call_oracle from .score import _prf from .structure_report import render, write_outputs from .types_defs import DiffBucket, LocationStats, ScoreResult console_target = Path(ec.KOTLIN_DEFAULT_TARGET) _CALLS = cs.RelationshipType.CALLS.value CallEdge = tuple[str, str] def oracle_kotlin_call_edges(target: Path) -> tuple[set[CallEdge], frozenset[str]]: return run_kotlin_call_oracle(target) def cgr_kotlin_call_edges(target: Path, project: str, declared: frozenset[str]) -> set[CallEdge]: """Capture the graph and extract CALL edges for Kotlin files.""" ingestor = _capture(target, project) # Map (label, uid) → source file path for .kt files caller_path = { (str(label), str(uid)): str(props[cs.KEY_PATH]) for (label, uid), props in ingestor.nodes.items() if props.get(cs.KEY_PATH) and str(props[cs.KEY_PATH]).endswith(".kt") } edges: set[CallEdge] = set() for from_label, from_val, rel_type, _to_label, to_val in ingestor.rels: if rel_type != _CALLS: continue path = caller_path.get((str(from_label), str(from_val))) if not path: continue name = str(to_val).split(cs.SEPARATOR_DOT)[-1] if name in declared: edges.add((path, name)) return edges def _edge_repr(edge: CallEdge) -> str: return ec.KOTLIN_CALL_EDGE_REPR.format(file=edge[0], name=edge[1]) def score_kotlin_retrieval(cgr: set[CallEdge], oracle: set[CallEdge]) -> ScoreResult: """Calculate precision, recall, and F1; generate diff buckets.""" rows, diff = [], {} row = _prf(ec.Category.RETRIEVAL.value, ec.KOTLIN_RETRIEVAL_LABEL, cgr, oracle) if row: rows.append(row) diff[ec.KOTLIN_RETRIEVAL_DIFF_PREFIX + ec.KOTLIN_RETRIEVAL_LABEL] = DiffBucket( missing=[_edge_repr(e) for e in sorted(oracle - cgr)], extra=[_edge_repr(e) for e in sorted(cgr - oracle)], ) return ScoreResult(rows=rows, location=LocationStats(0, 0, 0, 0.0, 0), diff=diff) def main( target: Annotated[Path, typer.Option(help="Kotlin source directory")] = console_target, project_name: Annotated[str, typer.Option(help="cgr project name")] = "", out_dir: Annotated[Path, typer.Option(help="Output directory")] = Path(ec.DEFAULT_OUT_DIR), ): if not kotlin_available(): logger.error(ls.KOTLIN_ORACLE_MISSING) raise typer.Exit(code=1) target = target.resolve() project = project_name or target.name logger.info(f"Running oracle on {target}") oracle_edges, declared = oracle_kotlin_call_edges(target) logger.info(f"Capturing graph for {project}") cgr_edges = cgr_kotlin_call_edges(target, project, declared) result = score_kotlin_retrieval(cgr_edges, oracle_edges) write_outputs(result, out_dir, ec.KOTLIN_RETRIEVAL_SCORES_FILENAME, ec.KOTLIN_RETRIEVAL_DIFF_FILENAME) render(result, ec.KOTLIN_RETRIEVAL_TITLE) if __name__ == "__main__": typer.run(main) ``` The `_capture` helper (from [[`codebase_rag/cgr_graph.py`](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/cgr_graph.py)](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/cgr_graph.py)) initializes the ingestor, while `_prf` (from [[`evals/score.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/score.py)](https://github.com/vitali87/code-graph-rag/blob/main/evals/score.py)) computes **precision**, **recall**, and **F1** metrics. ### Register the CLI Command Wire the new benchmark into the evaluation CLI by editing [[`evals/cli.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/cli.py)](https://github.com/vitali87/code-graph-rag/blob/main/evals/cli.py): ```python # evals/cli.py from .kotlin_retrieval import main as kotlin_main # After the Typer app definition app.command()(kotlin_main) ``` This exposes the command as `cgr eval kotlin-retrieval` (or whatever naming convention you choose). ### Execute the Benchmark Run the retrieval evaluation against a local codebase: ```bash cgr eval kotlin-retrieval --target path/to/kotlin/project \ --project-name my_kotlin_proj \ --out-dir evals/results ``` The command produces two artifacts: - **`kotlin_retrieval_scores.csv`** – Contains precision, recall, and F1 scores formatted according to `constants.CSV_FIELDS`. - **[`kotlin_retrieval_diff.json`](https://github.com/vitali87/code-graph-rag/blob/main/kotlin_retrieval_diff.json)** – Lists missing and extra edges for debugging discrepancies between the graph and the oracle. ## Verify and Iterate Review the console output (rendered by `cli._render`) which prints TP/FP/FN counts alongside the **F1 score**. If metrics are below threshold, iterate on the tree-sitter grammar or parser adapters in the main library until the **CALLS-edge retrieval** matches the oracle’s ground truth. Use the JSON diff to identify specific missing or spurious edges (e.g., `src/main.kt -> processUser`). ## Summary - **Install test dependencies** using `uv sync --group test` or the equivalent pip command to access pytest and plugins. - **Run the generic suite** with `pytest -n auto` to confirm core graph operations (capture, incremental updates, dead-code detection) function correctly. - **Declare language constants** in [`evals/constants.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/constants.py) to define output filenames, CLI labels, and edge string templates. - **Implement an oracle** that returns `(caller_file, callee_name)` tuples and a set of declared first-party symbols. - **Build a retrieval driver** that captures the graph via `_capture`, filters for CALL edges, and compares against the oracle using `_prf`. - **Register the command** in [`evals/cli.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/cli.py) and execute via `cgr eval -retrieval` to generate CSV scores and JSON diffs. ## Frequently Asked Questions ### What does the oracle need to return for a valid comparison? The oracle must return a **set of tuples** where each tuple contains `(caller_file_path, callee_simple_name)` representing a CALL relationship, plus a **frozenset of declared symbols** used to filter out third-party library calls. The evaluation logic in [`score.py`](https://github.com/vitali87/code-graph-rag/blob/main/score.py) compares these ground-truth edges against the edges extracted from the code-graph-rag ingestor to calculate precision and recall. ### Can I run evaluations for multiple languages in a single command? No, each language has a distinct CLI subcommand (e.g., `go-retrieval`, `scala-retrieval`, `kotlin-retrieval`) registered in [`evals/cli.py`](https://github.com/vitali87/code-graph-rag/blob/main/evals/cli.py). You must invoke each separately, though you can script a loop that runs `cgr eval -retrieval` across multiple projects and aggregates the resulting CSV files. ### Where does the graph capture logic live? The `_capture` function used in retrieval drivers is defined in [[`codebase_rag/cgr_graph.py`](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/cgr_graph.py)](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/cgr_graph.py). It initializes a `codebase_rag` ingestor, walks the target directory, and populates nodes and relationships (including the CALLS edges) that your evaluation code will later query. ### How do I interpret the retrieval scores CSV? The CSV file generated by `write_outputs` contains columns defined in `evals/constants.CSV_FIELDS`, including **precision** (ratio of correct edges to total retrieved), **recall** (ratio of correct edges to total oracle edges), and **F1** (harmonic mean of the two). A high F1 indicates that the graph accurately captures the call structure for your new language compared to the external oracle.