# How to Tune the Similarity Threshold for Duplicate Code Detection in Code‑Graph‑RAG

> Learn how to tune the similarity threshold for duplicate code detection in Code-Graph-RAG. Adjust the threshold via CLI, API, or configuration for precise results.

- Repository: [Vitali Avagyan/code-graph-rag](https://github.com/vitali87/code-graph-rag)
- Tags: how-to-guide
- Published: 2026-09-08

---

**Tune the Jaccard similarity threshold in Code‑Graph‑RAG by passing a float between 0 and 1 via the `--threshold` CLI flag, the `threshold` parameter in the `find_duplicate_code` Python API, or by overriding the `DUPLICATES_DEFAULT_THRESHOLD` configuration constant.**

Code‑Graph‑RAG detects near-duplicate functions by comparing abstract syntax tree (AST) fingerprints using a Jaccard similarity index. Understanding how to tune the similarity threshold for duplicate code detection allows you to control the granularity of matches, from loose structural similarities to exact copy-paste blocks.

## Understanding the Jaccard Similarity Threshold

Code‑Graph‑RAG measures function similarity by computing the **Jaccard index** on sets of **branch-overlap tokens** extracted from AST fingerprints. These tokens describe the structural shape of each function independent of variable names or formatting.

The `threshold` parameter defines the minimum Jaccard similarity score (a float in the closed interval **[0, 1]**) that a pair of functions must exceed to be reported as near-duplicates. The default value is **0.8**, requiring an 80% overlap between token sets.

## Where the Threshold Is Implemented in the Source Code

The core detection engine lives in [`codebase_rag/duplicates.py`](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/duplicates.py) according to the vitali87/code-graph-rag source code.

The default threshold is injected via the constant `cs.DUPLICATES_DEFAULT_THRESHOLD` at line 44. Input validation occurs in the `_validation_error` function (lines 55-57), which enforces that the threshold remains within the valid **[0, 1]** range. The actual comparison happens in `_match_duplicate_shapes` at line 561, where the calculated Jaccard score is evaluated against your specified threshold to determine if a pair qualifies as a duplicate.

## Three Methods to Tune the Similarity Threshold

### Command-Line Interface (CLI)

Pass the `--threshold` flag to the `cgr duplicates` command to adjust the cutoff for a specific execution:

```bash
cgr duplicates --threshold 0.9 --min-size 20 --project-name my-project

```

This approach is documented in [`docs/guide/cli-reference.md`](https://github.com/vitali87/code-graph-rag/blob/main/docs/guide/cli-reference.md) and [`docs/guide/duplicates.md`](https://github.com/vitali87/code-graph-rag/blob/main/docs/guide/duplicates.md).

### Python API

Programmatically control the threshold by passing the `threshold` argument to the `find_duplicate_code` function:

```python
from codebase_rag.duplicates import find_duplicate_code

# project is an ingested CodeGraphProject instance

duplicates = await find_duplicate_code(
    project,
    threshold=0.7,      # Require 70% Jaccard similarity

    min_size=15         # Ignore functions smaller than 15 nodes

)

for grp in duplicates:
    print(f"Group of {len(grp)} near-duplicates (score ≥ {grp[0].score})")
    for fn in grp:
        print(f"  – {fn.file}:{fn.line}")

```

### Configuration File

Override the default threshold permanently for all executions by setting `duplicates.threshold` in a JSON or YAML configuration file. This updates the `cs.DUPLICATES_DEFAULT_THRESHOLD` constant that the codebase uses when no explicit threshold is provided.

## Selecting the Optimal Threshold Value

Choosing the right threshold involves a signal-to-noise trade-off:

- **Low threshold (e.g., 0.5)**: Surfaces loosely related functions and broad structural patterns. Useful for discovering all forms of code reuse, but may generate noisy reports with many false positives.
- **High threshold (e.g., 0.95)**: Reports only near-identical copy-paste blocks. Minimizes false positives and focuses refactoring efforts on exact duplicates.

For large codebases with many small helper functions, start with the default **0.8** and raise the value if the report is too verbose. For focused refactoring targeting exact duplicates, raise the threshold toward **1.0** while potentially lowering the `min_size` parameter to catch smaller replicated blocks.

## Working with the Jaccard Utility Directly

For custom experiments or manual validation, you can access the underlying Jaccard calculator implemented in the codebase:

```python
from codebase_rag.utils.dependencies import jaccard

set_a = {"if", "return", "assign", "call"}
set_b = {"if", "return", "call", "loop"}

similarity = jaccard(set_a, set_b)  # Returns 0.75

```

## Summary

- Code‑Graph‑RAG uses the **Jaccard index** on AST branch-overlap tokens to measure function similarity.
- Valid threshold values range from **0 to 1**, with a default of **0.8** (80% overlap).
- Adjust the threshold via the **CLI** (`--threshold` flag), **Python API** (`threshold` parameter in `find_duplicate_code`), or **configuration constants** (`DUPLICATES_DEFAULT_THRESHOLD`).
- Lower thresholds detect broad structural similarities; higher thresholds isolate exact duplicates.

## Frequently Asked Questions

### What happens if I set the similarity threshold to 0 or 1?

Setting the threshold to **0** reports every function pair as a duplicate, resulting in maximum noise and unusable results. Setting it to **1** requires identical token sets with zero variation, reporting only exact structural duplicates with perfect AST fingerprint matches.

### Why does Code‑Graph‑RAG use the Jaccard index for duplicate detection?

The **Jaccard index** effectively compares the shape of AST fingerprints by calculating the intersection over union of branch-overlap token sets. This approach is robust to variable renaming and minor formatting changes while accurately catching structural similarities between functions.

### Can I tune the threshold without modifying the source code?

Yes. You can adjust the threshold without editing repository files by using the **`--threshold` CLI flag** when running `cgr duplicates` or by passing the `threshold` argument to the `find_duplicate_code` Python function. Both methods override the default without requiring changes to [`codebase_rag/duplicates.py`](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/duplicates.py).

### Where is the threshold validation enforced in the codebase?

Input validation occurs in [`codebase_rag/duplicates.py`](https://github.com/vitali87/code-graph-rag/blob/main/codebase_rag/duplicates.py) within the `_validation_error` function (lines 55-57), which validates that the threshold falls within the valid **[0, 1]** range before the detection algorithm begins processing. If the value is outside this range, the function raises a validation error immediately.