How to Tune the Similarity Threshold for Duplicate Code Detection in Code‑Graph‑RAG

Tune the Jaccard similarity threshold in Code‑Graph‑RAG by passing a float between 0 and 1 via the --threshold CLI flag, the threshold parameter in the find_duplicate_code Python API, or by overriding the DUPLICATES_DEFAULT_THRESHOLD configuration constant.

Code‑Graph‑RAG detects near-duplicate functions by comparing abstract syntax tree (AST) fingerprints using a Jaccard similarity index. Understanding how to tune the similarity threshold for duplicate code detection allows you to control the granularity of matches, from loose structural similarities to exact copy-paste blocks.

Understanding the Jaccard Similarity Threshold

Code‑Graph‑RAG measures function similarity by computing the Jaccard index on sets of branch-overlap tokens extracted from AST fingerprints. These tokens describe the structural shape of each function independent of variable names or formatting.

The threshold parameter defines the minimum Jaccard similarity score (a float in the closed interval [0, 1]) that a pair of functions must exceed to be reported as near-duplicates. The default value is 0.8, requiring an 80% overlap between token sets.

Where the Threshold Is Implemented in the Source Code

The core detection engine lives in codebase_rag/duplicates.py according to the vitali87/code-graph-rag source code.

The default threshold is injected via the constant cs.DUPLICATES_DEFAULT_THRESHOLD at line 44. Input validation occurs in the _validation_error function (lines 55-57), which enforces that the threshold remains within the valid [0, 1] range. The actual comparison happens in _match_duplicate_shapes at line 561, where the calculated Jaccard score is evaluated against your specified threshold to determine if a pair qualifies as a duplicate.

Three Methods to Tune the Similarity Threshold

Command-Line Interface (CLI)

Pass the --threshold flag to the cgr duplicates command to adjust the cutoff for a specific execution:

cgr duplicates --threshold 0.9 --min-size 20 --project-name my-project

This approach is documented in docs/guide/cli-reference.md and docs/guide/duplicates.md.

Python API

Programmatically control the threshold by passing the threshold argument to the find_duplicate_code function:

from codebase_rag.duplicates import find_duplicate_code

# project is an ingested CodeGraphProject instance

duplicates = await find_duplicate_code(
    project,
    threshold=0.7,      # Require 70% Jaccard similarity

    min_size=15         # Ignore functions smaller than 15 nodes

)

for grp in duplicates:
    print(f"Group of {len(grp)} near-duplicates (score ≥ {grp[0].score})")
    for fn in grp:
        print(f"  – {fn.file}:{fn.line}")

Configuration File

Override the default threshold permanently for all executions by setting duplicates.threshold in a JSON or YAML configuration file. This updates the cs.DUPLICATES_DEFAULT_THRESHOLD constant that the codebase uses when no explicit threshold is provided.

Selecting the Optimal Threshold Value

Choosing the right threshold involves a signal-to-noise trade-off:

• Low threshold (e.g., 0.5): Surfaces loosely related functions and broad structural patterns. Useful for discovering all forms of code reuse, but may generate noisy reports with many false positives.
• High threshold (e.g., 0.95): Reports only near-identical copy-paste blocks. Minimizes false positives and focuses refactoring efforts on exact duplicates.

For large codebases with many small helper functions, start with the default 0.8 and raise the value if the report is too verbose. For focused refactoring targeting exact duplicates, raise the threshold toward 1.0 while potentially lowering the min_size parameter to catch smaller replicated blocks.

Working with the Jaccard Utility Directly

For custom experiments or manual validation, you can access the underlying Jaccard calculator implemented in the codebase:

from codebase_rag.utils.dependencies import jaccard

set_a = {"if", "return", "assign", "call"}
set_b = {"if", "return", "call", "loop"}

similarity = jaccard(set_a, set_b)  # Returns 0.75

Summary

• Code‑Graph‑RAG uses the Jaccard index on AST branch-overlap tokens to measure function similarity.
• Valid threshold values range from 0 to 1, with a default of 0.8 (80% overlap).
• Adjust the threshold via the CLI (--threshold flag), Python API (threshold parameter in find_duplicate_code), or configuration constants (DUPLICATES_DEFAULT_THRESHOLD).
• Lower thresholds detect broad structural similarities; higher thresholds isolate exact duplicates.

Frequently Asked Questions

What happens if I set the similarity threshold to 0 or 1?

Setting the threshold to 0 reports every function pair as a duplicate, resulting in maximum noise and unusable results. Setting it to 1 requires identical token sets with zero variation, reporting only exact structural duplicates with perfect AST fingerprint matches.

Why does Code‑Graph‑RAG use the Jaccard index for duplicate detection?

The Jaccard index effectively compares the shape of AST fingerprints by calculating the intersection over union of branch-overlap token sets. This approach is robust to variable renaming and minor formatting changes while accurately catching structural similarities between functions.

Can I tune the threshold without modifying the source code?

Yes. You can adjust the threshold without editing repository files by using the --threshold CLI flag when running cgr duplicates or by passing the threshold argument to the find_duplicate_code Python function. Both methods override the default without requiring changes to codebase_rag/duplicates.py.

Where is the threshold validation enforced in the codebase?

Input validation occurs in codebase_rag/duplicates.py within the _validation_error function (lines 55-57), which validates that the threshold falls within the valid [0, 1] range before the detection algorithm begins processing. If the value is outside this range, the function raises a validation error immediately.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}