How Semantic Search Combines Multiple Signals for Code Analysis

The semantic search engine in codebase-memory-mcp calculates a unified similarity score by weighting and summing orthogonal signals—lexical, structural, and contextual—then applying a proximity boost to prioritize geographically related code.

The codebase-memory-mcp repository implements a sophisticated semantic search system that merges multiple heterogeneous signals into a single similarity metric. This approach enables robust code analysis by capturing similarities across lexical tokens, API signatures, and structural patterns. Understanding how these signals combine reveals why the engine can identify functionally related code even when variable names and implementations differ significantly.

Signal Architecture and Weight Configuration

The semantic search system uses a configurable weighting scheme to balance the relative importance of each signal type. These weights are defined as constants in src/semantic/semantic.c and must sum to approximately 1.0 to maintain normalized scoring.

Default Weight Constants

The default weight table is declared at lines 40-49 in semantic.c:

#define CBM_SEM_W_TFIDF          0.20F   // lexical token overlap
#define CBM_SEM_W_RI             0.25F   // Random Indexing similarity
#define CBM_SEM_W_MINHASH        0.10F   // Jaccard on MinHash sketches
#define CBM_SEM_W_API            0.15F   // API‑signature similarity
#define CBM_SEM_W_TYPE           0.10F   // type‑signature similarity
#define CBM_SEM_W_DECORATOR      0.05F   // decorator pattern similarity
#define CBM_SEM_W_STRUCT_PROFILE 0.10F   // structural (AST) profile similarity
#define CBM_SEM_W_DATAFLOW       0.05F   // data‑flow similarity (future extension)

These constants populate a cbm_sem_config_t structure via cbm_sem_get_config() (lines 9-22). The configuration supports runtime overrides through environment variables such as CBM_SEMANTIC_THRESHOLD and CBM_SEMANTIC_ENABLED, allowing users to tune sensitivity without recompiling.

The Seven Orthogonal Signals

Each function is represented by a cbm_sem_func_t struct (defined in semantic.h) that stores vectors for distinct signal types. The extraction and vector creation occur during the semantic pipeline pass (pass_semantic.c, line 526).

The system captures seven distinct signals:

• TF-IDF (Weight: 0.20): Sparse vectors of token frequencies generated by cbm_sem_tokenize, capturing lexical overlap between function bodies.
• Random Indexing (Weight: 0.25): Dense 4-bit scalar-quantized vectors produced by cbm_rsq_ip, representing semantic meaning through distributed word representations.
• MinHash (Weight: 0.10): 64-bit sketches used for fast Jaccard similarity estimation via cbm_minhash_jaccard.
• API Signatures (Weight: 0.15): Random-indexed vectors hashing function calls and external library usage.
• Type Signatures (Weight: 0.10): Encoded representations of parameter types and return types.
• Decorator Patterns (Weight: 0.05): Vectors capturing annotations and decorator usage patterns.
• Structural Profiles (Weight: 0.10): Fixed-size float arrays from ast_profile.c describing control-flow shapes and AST characteristics.

Additionally, the system tracks file_path for each function to compute directory proximity bonuses during final scoring.

The Combined Scoring Algorithm

The core combinator logic resides in cbm_sem_combined_score (semantic.c, lines 1601-1660). This function implements a four-stage pipeline to derive the final similarity value between two functions.

MinHash Early-Out Optimization

Before computing expensive semantic signals, the algorithm checks for high structural similarity using MinHash. If both functions possess MinHash sketches and their Jaccard similarity exceeds CBM_MINHASH_JACCARD_THRESHOLD, the function returns 0.0 immediately to preserve the edge emission budget for the similarity pass:

if (a->has_minhash && b->has_minhash) {
    double early_j = cbm_minhash_jaccard(...);
    if (early_j >= CBM_MINHASH_JACCARD_THRESHOLD) return 0.0F;
}

Weighted Signal Aggregation

For functions passing the early-out check, the algorithm computes a weighted sum across all signal dimensions:

float score = cfg->w_tfidf * sparse_tfidf_cosine(a, b);
score += cfg->w_ri      * cbm_rsq_ip(&a->ri_code, &b->ri_code);
score += cfg->w_minhash * (float)j;               // Jaccard
score += cfg->w_api     * cbm_rsq_ip(&a->api_code, &b->api_code);
score += cfg->w_type    * cbm_rsq_ip(&a->type_code, &b->type_code);
score += cfg->w_decorator * cbm_rsq_ip(&a->deco_code, &b->deco_code);
score += cfg->w_struct_profile * small_cosine(...);

Each component uses the appropriate similarity metric—cosine similarity for TF-IDF and structural profiles, inner product for random-indexed vectors, and Jaccard for MinHash sketches.

Directory Proximity Multiplier

The raw semantic score is then multiplied by a proximity factor computed by cbm_sem_proximity (semantic.c, lines 19-53). This function analyzes how many directory components two file paths share, boosting scores for functions located in similar project locations:

score *= cbm_sem_proximity(a->file_path, b->file_path);

This localization bonus ensures that related functions within the same module or subsystem rank higher than semantically similar but geographically distant code.

Score Normalization

Finally, the algorithm clamps the result to the valid cosine similarity range of [0, 1]:

if (score > CBM_SEM_UNIT_POS) score = CBM_SEM_UNIT_POS;
if (score < 0.0F)           score = 0.0F;

The resulting float represents the definitive semantic similarity used when emitting SEMANTICALLY_RELATED edges in pass_semantic_edges.c (line 852).

Pipeline Integration and CLI Usage

The semantic search system integrates with the analysis pipeline through the mcp command-line interface. The public entry point is the semantic_query argument processed in src/mcp/mcp.c (lines 369-394).

Running Semantic Queries

Users can execute keyword-based searches that leverage the combined scoring algorithm:

$ mcp --semantic-query '["http", "handler"]' --project myproj

The CLI parses the JSON keyword array, builds TF-IDF vectors, and invokes run_semantic_query_core(), which uses cbm_sem_combined_score to rank candidates. Results include the computed similarity score:

{
  "qn": "myproject/src/parser/json_decode.c:decode",
  "label": "FUNC",
  "file": "src/parser/json_decode.c",
  "score": 0.842
}

Direct API Usage

Developers can also compute scores programmatically using the C API:

#include "semantic/semantic.h"

int main(void) {
    cbm_sem_func_t func_a, func_b;
    // ...populate fields (tfidf, ri_code, minhash, etc.)...

    cbm_sem_config_t cfg = cbm_sem_get_config();   // uses default weights
    float similarity = cbm_sem_combined_score(&func_a, &func_b, &cfg);
    printf("Combined semantic similarity = %.3f\n", similarity);
    return 0;
}

Runtime Configuration

Adjust behavioral thresholds without recompilation:

export CBM_SEMANTIC_ENABLED=1               # turn on the semantic pipeline

export CBM_SEMANTIC_THRESHOLD=0.45          # raise the minimum edge score

Summary

• Seven orthogonal signals—including TF-IDF, Random Indexing, MinHash, API signatures, and AST profiles—are weighted and combined to create a robust similarity metric.
• MinHash early-out optimization prevents redundant computation when functions are already structurally similar.
• Directory proximity boosting ensures geographically related code receives higher relevance scores.
• Clamped normalization guarantees all output scores remain within the [0, 1] cosine similarity range.
• Runtime configurability via environment variables allows tuning of thresholds without source modification.

Frequently Asked Questions

What signals does the semantic search use?

The system employs seven distinct signals: TF-IDF for lexical token overlap, Random Indexing for semantic vector similarity, MinHash for structural Jaccard estimation, API signatures for external call patterns, type signatures for parameter/return type analysis, decorator patterns for annotation matching, and structural profiles for AST shape comparison. Each signal targets a different aspect of code meaning to ensure comprehensive similarity detection.

How does the proximity boost affect search results?

The cbm_sem_proximity function (lines 19-53 in semantic.c) calculates a multiplier based on shared directory path components between two source files. Functions in the same directory or module receive higher final scores than equally similar functions located in distant project locations. This prioritizes cohesive code relationships over coincidental similarities across large codebases.

Can I customize the signal weights?

Currently, the default weights are hardcoded in semantic.c (lines 40-49) and loaded into cbm_sem_config_t via cbm_sem_get_config(). While the environment variables CBM_SEMANTIC_THRESHOLD and CBM_SEMANTIC_ENABLED allow runtime threshold adjustment, custom weight overrides require modifying the source configuration structure and recompiling the project.

What is the MinHash early-out optimization?

The MinHash early-out is a performance optimization in cbm_sem_combined_score that checks Jaccard similarity before computing expensive semantic signals. If two functions share a MinHash Jaccard score above CBM_MINHASH_JACCARD_THRESHOLD, the function returns 0.0 immediately. This prevents the pipeline from wasting computational resources on pairs that are already structurally identical, reserving the semantic edge budget for more meaningful comparisons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}