# How Semantic Search with Embeddings Works in codebase-memory-mcp: On-Device Vector Search Deep Dive

> Explore semantic search with embeddings in codebase-memory-mcp. Discover how on-device vector search uses nomic-embed-code for efficient, private codebase analysis.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: deep-dive
- Published: 2026-07-16

---

**The codebase-memory-mcp project implements semantic search using pre-computed token embeddings from nomic-embed-code stored in a static lookup table, enabling cosine similarity comparisons entirely on-device without external services.**

The DeusData/codebase-memory-mcp repository provides a Model Context Protocol (MCP) server that performs semantic code search without network dependencies. Understanding how this semantic search with embeddings works internally reveals a sophisticated three-stage pipeline that bridges Python-based ML inference with high-performance C runtime execution.

## Architecture Overview

The semantic search system operates through a distinct separation between offline embedding generation and runtime vector lookup. This architecture ensures zero-latency external calls during query execution.

### Stage 1: Embedding Extraction

The Python script [`scripts/extract_nomic_vectors.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/extract_nomic_vectors.py) handles the heavy lifting of model interaction. It loads the `nomic-embed-code` transformer model, filters the tokenizer vocabulary to retain only code-relevant identifiers, and performs full inference on each token. The script applies optional **simulated attention** post-processing and writes quantized `int8` vector blobs along with corresponding C header files.

### Stage 2: Static Vector Compilation

The generated artifacts—including `code_vectors.bin`, [`code_vectors.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/code_vectors.h), `code_vectors_blob.S`, and [`code_tokens.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/code_tokens.h)—are compiled directly into the binary via `Makefile.cbm`. These headers expose `PRETRAINED_VECTOR_BLOB` and the token mapping array `PRETRAINED_TOKENS`, embedding the entire embedding table into the executable.

### Stage 3: Runtime Query Execution

At runtime, the C library [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c) loads these pre-computed vectors. When a client submits a `semantic_query` array through [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c), the runtime constructs a query vector by averaging token embeddings and compares it against the corpus using cosine similarity. The top-k matches are returned as JSON.

## Embedding Generation Pipeline

The quality of semantic search depends heavily on the preprocessing and inference logic implemented in the extraction script.

### Token Filtering and Preprocessing

The extraction process employs `is_code_relevant` to discard BPE noise, punctuation, and non-Latin characters before processing. The `clean_token` function normalizes remaining tokens by stripping BPE markers (`lstrip`), removing underscores, and converting to lowercase. This filtering typically reduces the vocabulary to approximately 42,839 code-relevant tokens from the original model vocabulary.

### Model Inference and Vector Processing

Tokens are fed to the model using the prompt format `"search_query: <token>"`. The pipeline performs mean-pooling over non-padding tokens, truncates vectors to `OUTPUT_DIM = 768` dimensions, and applies L2-normalization. After processing the full corpus, a mean-centering step subtracts the corpus-wide mean vector to reduce anisotropy, followed by re-normalization.

The optional **simulated attention** mechanism implements a self-attention-like post-process that blends each vector with the mean of its *K* nearest neighbors across several iterations, improving contextual similarity between related code tokens.

### Quantization and Storage

Final vectors are scaled to the range `[-127, 127]` and stored as `int8` values in `code_vectors.bin`. The binary layout follows a strict schema: `[int32 count][int32 dim]` followed by `count×dim` int8 values. This quantization reduces the embedding table to approximately 13.2 MB while preserving search quality.

## Runtime Semantic Scoring

Query execution follows a streamlined path optimized for low-latency retrieval.

### Tokenization and Query Vector Construction

When processing a semantic query, the runtime tokenizer (`cbm_sem_tokenize` in [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c)) splits identifiers by delimiters and camel-case transitions, expands common abbreviations, and retrieves each token's embedding via `pretrained_vec_at`. The system sums these embeddings and applies L2-normalization to create the final query vector.

### Similarity Computation and Ranking

Cosine similarity is computed between the query vector and every stored token vector using the formula `dot_product / (norms)`. The `cbm_sem_topk` function identifies the highest-scoring matches, which are then blended with other ranking signals including TF-IDF, Random Indexing, and MinHash using configurable weights (`CBM_SEM_W_TFIDF`, `CBM_SEM_W_RI`) defined in [`semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/semantic.c).

## Implementation Examples

### Extracting Embeddings

Generate the static vector table using the extraction script:

```bash

# Install required Python packages

pip install torch transformers sentence-transformers

# Generate embeddings (one-time operation)

python3 scripts/extract_nomic_vectors.py \
    --output-dir vendored/nomic \
    --device cpu

```

Expected output:

```

step 1: loading model + tokenizer... loaded in 12.3s
step 2: filtering vocabulary... code-relevant (deduplicated): 42,839 tokens
step 3: extracting embeddings... extracted 42,839 vectors × 768d
step 4: simulated attention... completed in 38.7s
step 5: writing output files... code_vectors.bin (13.2 MB)

```

### Recompiling the Binary

Embed the new vectors into the executable:

```bash
make -f Makefile.cbm clean
make -f Makefile.cbm

```

The build process automatically links `code_vectors_blob.S` and the generated headers, producing the final `codebase-mcp` binary with embedded embeddings.

### Performing Queries via CLI

Execute semantic searches through the command line:

```bash

# Search for code related to "send" and "publish"

codebase-mcp query \
    --semantic_query send publish \
    --path ./my-project \
    --limit 5

```

Sample JSON response:

```json
{
  "semantic_query": [
    {"file":"src/net/transport.c","score":0.873},
    {"file":"src/pubsub/broker.c","score":0.862},
    {"file":"src/cli/cli.c","score":0.845}
  ]
}

```

### Direct C API Usage

Integrate semantic search into C applications using the exposed API:

```c
#include "nomic/code_vectors.h"
#include "semantic/semantic.h"

float query_vec[PRETRAINED_DIM];
int tokens[MAX_TOKENS];

// Tokenize query string
int n = cbm_sem_tokenize("send", tokens, MAX_TOKENS);

// Build query vector from token embeddings
cbm_sem_average_embedding(tokens, n, query_vec);

int topk = 10;
int idxs[topk];
float sims[topk];

// Retrieve top-k matches
cbm_sem_topk(query_vec, PRETRAINED_DIM, idxs, sims, topk);

for (int i = 0; i < topk; ++i) {
    printf("%s  (%.3f)\n", PRETRAINED_TOKENS[idxs[i]], sims[i]);
}

```

## Summary

- **Three-stage pipeline**: Embedding extraction via Python, static compilation into C headers, and runtime cosine similarity search in [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c)
- **Zero external dependencies**: All embeddings from `nomic-embed-code` are baked into the binary, eliminating network latency and service dependencies
- **Efficient storage**: 768-dimensional vectors are quantized to `int8` and stored in `code_vectors.bin` using a compact binary layout
- **Hybrid scoring**: Semantic similarity scores blend with TF-IDF, Random Indexing, and MinHash using configurable weights in [`semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/semantic.c)
- **Token preprocessing**: Runtime tokenization handles camelCase, abbreviations, and delimiter splitting via `cbm_sem_tokenize`

## Frequently Asked Questions

### How does codebase-memory-mcp avoid external API calls for embeddings?

The project embeds all token vectors directly into the compiled binary via `code_vectors_blob.S` and [`code_vectors.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/code_vectors.h). During the build process, `Makefile.cbm` links the quantized embedding table as static data, allowing [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c) to access `PRETRAINED_VECTOR_BLOB` without network requests. This design ensures deterministic, offline-capable semantic search.

### What is the purpose of the simulated attention step in embedding generation?

The `simulated_attention` function in [`scripts/extract_nomic_vectors.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/extract_nomic_vectors.py) performs a neighborhood-aware post-processing that blends each token's embedding with the mean of its *K* nearest neighbors. This technique, applied after mean-centering, enhances the contextual relationships between related code tokens and mitigates anisotropy in the embedding space, improving retrieval accuracy for semantically related but lexically distinct terms.

### How does the runtime compute similarity between query and code tokens?

The runtime in [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c) first constructs a query vector by averaging the embeddings of query tokens (retrieved via `pretrained_vec_at`), then normalizes the result. It calculates cosine similarity using the dot product divided by the product of L2 norms against all pre-computed token vectors. The `cbm_sem_topk` function identifies the highest-scoring matches for inclusion in the JSON response.

### Why use int8 quantization instead of float32 for storing embeddings?

Quantizing 768-dimensional float32 vectors to `int8` reduces storage from approximately 164 MB to 13.2 MB for 42,839 tokens—a reduction that enables embedding the entire table within the executable binary. The scaling to `[-127, 127]` preserves relative vector directions necessary for cosine similarity while significantly reducing memory footprint and cache pressure during the brute-force similarity computations in `cbm_sem_topk`.