How Semantic Search with Embeddings Works in codebase-memory-mcp: On-Device Vector Search Deep Dive

The codebase-memory-mcp project implements semantic search using pre-computed token embeddings from nomic-embed-code stored in a static lookup table, enabling cosine similarity comparisons entirely on-device without external services.

The DeusData/codebase-memory-mcp repository provides a Model Context Protocol (MCP) server that performs semantic code search without network dependencies. Understanding how this semantic search with embeddings works internally reveals a sophisticated three-stage pipeline that bridges Python-based ML inference with high-performance C runtime execution.

Architecture Overview

The semantic search system operates through a distinct separation between offline embedding generation and runtime vector lookup. This architecture ensures zero-latency external calls during query execution.

Stage 1: Embedding Extraction

The Python script scripts/extract_nomic_vectors.py handles the heavy lifting of model interaction. It loads the nomic-embed-code transformer model, filters the tokenizer vocabulary to retain only code-relevant identifiers, and performs full inference on each token. The script applies optional simulated attention post-processing and writes quantized int8 vector blobs along with corresponding C header files.

Stage 2: Static Vector Compilation

The generated artifacts—including code_vectors.bin, code_vectors.h, code_vectors_blob.S, and code_tokens.h—are compiled directly into the binary via Makefile.cbm. These headers expose PRETRAINED_VECTOR_BLOB and the token mapping array PRETRAINED_TOKENS, embedding the entire embedding table into the executable.

Stage 3: Runtime Query Execution

At runtime, the C library src/semantic/semantic.c loads these pre-computed vectors. When a client submits a semantic_query array through src/mcp/mcp.c, the runtime constructs a query vector by averaging token embeddings and compares it against the corpus using cosine similarity. The top-k matches are returned as JSON.

Embedding Generation Pipeline

The quality of semantic search depends heavily on the preprocessing and inference logic implemented in the extraction script.

Token Filtering and Preprocessing

The extraction process employs is_code_relevant to discard BPE noise, punctuation, and non-Latin characters before processing. The clean_token function normalizes remaining tokens by stripping BPE markers (lstrip), removing underscores, and converting to lowercase. This filtering typically reduces the vocabulary to approximately 42,839 code-relevant tokens from the original model vocabulary.

Model Inference and Vector Processing

Tokens are fed to the model using the prompt format "search_query: <token>". The pipeline performs mean-pooling over non-padding tokens, truncates vectors to OUTPUT_DIM = 768 dimensions, and applies L2-normalization. After processing the full corpus, a mean-centering step subtracts the corpus-wide mean vector to reduce anisotropy, followed by re-normalization.

The optional simulated attention mechanism implements a self-attention-like post-process that blends each vector with the mean of its K nearest neighbors across several iterations, improving contextual similarity between related code tokens.

Quantization and Storage

Final vectors are scaled to the range [-127, 127] and stored as int8 values in code_vectors.bin. The binary layout follows a strict schema: [int32 count][int32 dim] followed by count×dim int8 values. This quantization reduces the embedding table to approximately 13.2 MB while preserving search quality.

Runtime Semantic Scoring

Query execution follows a streamlined path optimized for low-latency retrieval.

Tokenization and Query Vector Construction

When processing a semantic query, the runtime tokenizer (cbm_sem_tokenize in src/semantic/semantic.c) splits identifiers by delimiters and camel-case transitions, expands common abbreviations, and retrieves each token's embedding via pretrained_vec_at. The system sums these embeddings and applies L2-normalization to create the final query vector.

Similarity Computation and Ranking

Cosine similarity is computed between the query vector and every stored token vector using the formula dot_product / (norms). The cbm_sem_topk function identifies the highest-scoring matches, which are then blended with other ranking signals including TF-IDF, Random Indexing, and MinHash using configurable weights (CBM_SEM_W_TFIDF, CBM_SEM_W_RI) defined in semantic.c.

Implementation Examples

Extracting Embeddings

Generate the static vector table using the extraction script:

# Install required Python packages

pip install torch transformers sentence-transformers

# Generate embeddings (one-time operation)

python3 scripts/extract_nomic_vectors.py \
    --output-dir vendored/nomic \
    --device cpu

Expected output:

step 1: loading model + tokenizer... loaded in 12.3s
step 2: filtering vocabulary... code-relevant (deduplicated): 42,839 tokens
step 3: extracting embeddings... extracted 42,839 vectors × 768d
step 4: simulated attention... completed in 38.7s
step 5: writing output files... code_vectors.bin (13.2 MB)

Recompiling the Binary

Embed the new vectors into the executable:

make -f Makefile.cbm clean
make -f Makefile.cbm

The build process automatically links code_vectors_blob.S and the generated headers, producing the final codebase-mcp binary with embedded embeddings.

Performing Queries via CLI

Execute semantic searches through the command line:

# Search for code related to "send" and "publish"

codebase-mcp query \
    --semantic_query send publish \
    --path ./my-project \
    --limit 5

Sample JSON response:

{
  "semantic_query": [
    {"file":"src/net/transport.c","score":0.873},
    {"file":"src/pubsub/broker.c","score":0.862},
    {"file":"src/cli/cli.c","score":0.845}
  ]
}

Direct C API Usage

Integrate semantic search into C applications using the exposed API:

#include "nomic/code_vectors.h"
#include "semantic/semantic.h"

float query_vec[PRETRAINED_DIM];
int tokens[MAX_TOKENS];

// Tokenize query string
int n = cbm_sem_tokenize("send", tokens, MAX_TOKENS);

// Build query vector from token embeddings
cbm_sem_average_embedding(tokens, n, query_vec);

int topk = 10;
int idxs[topk];
float sims[topk];

// Retrieve top-k matches
cbm_sem_topk(query_vec, PRETRAINED_DIM, idxs, sims, topk);

for (int i = 0; i < topk; ++i) {
    printf("%s  (%.3f)\n", PRETRAINED_TOKENS[idxs[i]], sims[i]);
}

Summary

• Three-stage pipeline: Embedding extraction via Python, static compilation into C headers, and runtime cosine similarity search in src/semantic/semantic.c
• Zero external dependencies: All embeddings from nomic-embed-code are baked into the binary, eliminating network latency and service dependencies
• Efficient storage: 768-dimensional vectors are quantized to int8 and stored in code_vectors.bin using a compact binary layout
• Hybrid scoring: Semantic similarity scores blend with TF-IDF, Random Indexing, and MinHash using configurable weights in semantic.c
• Token preprocessing: Runtime tokenization handles camelCase, abbreviations, and delimiter splitting via cbm_sem_tokenize

Frequently Asked Questions

How does codebase-memory-mcp avoid external API calls for embeddings?

The project embeds all token vectors directly into the compiled binary via code_vectors_blob.S and code_vectors.h. During the build process, Makefile.cbm links the quantized embedding table as static data, allowing src/semantic/semantic.c to access PRETRAINED_VECTOR_BLOB without network requests. This design ensures deterministic, offline-capable semantic search.

What is the purpose of the simulated attention step in embedding generation?

The simulated_attention function in scripts/extract_nomic_vectors.py performs a neighborhood-aware post-processing that blends each token's embedding with the mean of its K nearest neighbors. This technique, applied after mean-centering, enhances the contextual relationships between related code tokens and mitigates anisotropy in the embedding space, improving retrieval accuracy for semantically related but lexically distinct terms.

How does the runtime compute similarity between query and code tokens?

The runtime in src/semantic/semantic.c first constructs a query vector by averaging the embeddings of query tokens (retrieved via pretrained_vec_at), then normalizes the result. It calculates cosine similarity using the dot product divided by the product of L2 norms against all pre-computed token vectors. The cbm_sem_topk function identifies the highest-scoring matches for inclusion in the JSON response.

Why use int8 quantization instead of float32 for storing embeddings?

Quantizing 768-dimensional float32 vectors to int8 reduces storage from approximately 164 MB to 13.2 MB for 42,839 tokens—a reduction that enables embedding the entire table within the executable binary. The scaling to [-127, 127] preserves relative vector directions necessary for cosine similarity while significantly reducing memory footprint and cache pressure during the brute-force similarity computations in cbm_sem_topk.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}