# How DeusData/codebase-memory-mcp Implements Semantic Search and Embeddings for Code

> Learn how DeusData codebase memory mcp implements semantic search and embeddings. Discover its hybrid Python-C pipeline for efficient code vectorization and fast cosine similarity.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: deep-dive
- Published: 2026-07-15

---

**DeusData/codebase-memory-mcp implements semantic search and embeddings through a hybrid Python-to-C pipeline that extracts 768-dimensional vectors from the nomic-embed-code transformer, quantizes them to int8, and embeds them as a static binary blob for zero-dependency runtime cosine similarity calculations.**

The DeusData/codebase-memory-mcp project (also referred to as Instagit) provides a deterministic, offline-capable semantic search and embeddings system for source code analysis. Unlike traditional vector databases that require heavy runtime dependencies, this implementation compiles transformer-based embeddings directly into the executable, enabling fast in-memory lookups without Python or GPU requirements.

## Embedding Extraction Pipeline in Python

The foundation rests on [`scripts/extract_nomic_vectors.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/extract_nomic_vectors.py), which orchestrates the one-time generation of pre-computed embeddings from the transformer model.

### Token Filtering and Model Inference

The script loads the **nomic-embed-code** transformer (a 7B parameter model) and processes only code-relevant tokens from its vocabulary. The `is_code_relevant` and `clean_token` functions filter the vocabulary to retain identifier-shaped strings such as `myVar` and `load_data`, discarding natural language and special tokens. Inference runs in configurable batches (`BATCH_SIZE`) across CPU, CUDA, or MPS devices.

### Vector Processing and Quantization

Raw hidden-state vectors undergo mean-pooling and truncation to `OUTPUT_DIM = 768`. The pipeline applies L2-normalization and mean-centering to reduce anisotropy, followed by optional refinement via `simulated_attention`. Finally, the float32 vectors are quantized to int8 (range -127 to 127) to minimize binary size while preserving cosine similarity accuracy.

### C Header Generation

The extraction script generates three critical artifacts in `vendored/nomic/`:

- [`code_vectors.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/code_vectors.h): Declares `PRETRAINED_TOKEN_COUNT`, `PRETRAINED_DIM`, and the `pretrained_vec_at(i)` accessor
- [`code_tokens.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/code_tokens.h): Defines `PRETRAINED_TOKENS[]`, a static string array mapping indices to token strings
- `code_vectors_blob.S`: An assembler file embedding the binary blob via `.incbin`

## Static C Runtime Integration

The runtime implementation eliminates external file dependencies by embedding vectors directly into the executable.

### Binary Blob Embedding

The `vendored/nomic/code_vectors_blob.S` file uses assembly directives to include `code_vectors.bin` as a raw data section within the compiled binary. This ensures that `pretrained_vec_at(i)` returns a pointer to int8 data without filesystem access, supporting fully static linking.

### Runtime Lookup Interface

Client code includes [`vendored/nomic/code_vectors.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/vendored/nomic/code_vectors.h) to access the embedding table. The `pretrained_vec_at` function provides O(1) indexed access to the quantized vectors, while `PRETRAINED_TOKENS` enables linear search for string-to-index resolution.

## Semantic Scoring Engine

The [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c) module implements the actual search logic, combining transformer embeddings with traditional information retrieval signals.

### Tokenization Strategy

The `cbm_sem_tokenize` function splits identifiers using delimiters and camel-case boundaries, converting tokens to lowercase. This normalization ensures that `camelCase` and `camel_case` map to the same semantic representations.

### Multi-Signal Similarity Computation

The `cbm_sem_combined_score` function aggregates multiple complementary signals:

- **Embedding signal**: Cosine similarity between L2-normalized float vectors (de-quantized from int8), weighted by `CBM_SEM_W_RI`
- **TF-IDF, Random Indexing, MinHash**: Traditional lexical signals
- **API signatures**: Structural code patterns

Proximity boosts (`CBM_SEM_PROX_MAX_BOOST`) increase scores for symbols appearing in the same source file.

### Configuration and Tuning

Environment variables control runtime behavior:

- `CBM_SEMANTIC_ENABLED`: Toggle the entire semantic pipeline
- `CBM_SEMANTIC_THRESHOLD`: Configure similarity cutoff values

## Practical Implementation Examples

Generating the embedding lookup (run once):

```bash

# Install required Python packages

pip install torch transformers sentence-transformers

# Extract vectors and generate C headers

python scripts/extract_nomic_vectors.py --output-dir vendored/nomic

```

Using the embeddings from C:

```c
#include "nomic/code_vectors.h"
#include "semantic/semantic.h"

/* Find the index of a token string */
int token_index(const char *tok) {
    for (int i = 0; i < PRETRAINED_TOKEN_COUNT; ++i) {
        if (strcmp(PRETRAINED_TOKENS[i], tok) == 0) return i;
    }
    return -1;
}

/* Retrieve a normalised float vector for a token */
void get_token_vec(const char *tok, float out[PRETRAINED_DIM]) {
    int idx = token_index(tok);
    if (idx < 0) { memset(out, 0, sizeof(float)*PRETRAINED_DIM); return; }

    const int8_t *q = pretrained_vec_at(idx);
    for (int d = 0; d < PRETRAINED_DIM; ++d) {
        out[d] = (float)q[d] / 127.0f;
    }
    
    // L2-normalise
    float norm = 0.0f;
    for (int d = 0; d < PRETRAINED_DIM; ++d) norm += out[d]*out[d];
    norm = sqrtf(norm);
    for (int d = 0; d < PRETRAINED_DIM; ++d) out[d] /= norm;
}

```

Computing semantic similarity between identifiers:

```c
float semantic_similarity(const char *a, const char *b) {
    float vec_a[PRETRAINED_DIM], vec_b[PRETRAINED_DIM];
    get_token_vec(a, vec_a);
    get_token_vec(b, vec_b);

    float dot = 0.0f;
    for (int i = 0; i < PRETRAINED_DIM; ++i) dot += vec_a[i] * vec_b[i];
    return dot;  // Cosine similarity in [-1, 1]
}

```

## Summary

- **Hybrid architecture**: Python handles heavy transformer inference once; C handles runtime similarity with zero dependencies
- **Space-efficient storage**: Float32 vectors are quantized to int8 and embedded as binary blobs via assembly `.incbin` in `code_vectors_blob.S`
- **Multi-signal approach**: Combines nomic-embed-code vectors with TF-IDF, Random Indexing, and MinHash in `cbm_sem_combined_score`
- **Configurable runtime**: Environment variables `CBM_SEMANTIC_ENABLED` and `CBM_SEMANTIC_THRESHOLD` control activation and cutoffs
- **File-based proximity**: `CBM_SEM_PROX_MAX_BOOST` favors local symbol relationships within the same file

## Frequently Asked Questions

### What embedding model does DeusData/codebase-memory-mcp use?

The system uses the **nomic-embed-code** transformer, a 7B parameter model specifically trained for code understanding. The [`scripts/extract_nomic_vectors.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/extract_nomic_vectors.py) script performs full inference on the model's vocabulary and exports the hidden states as quantized vectors using `is_code_relevant` filtering and `simulated_attention` refinement.

### How does the system achieve zero-dependency runtime operation?

All embedding vectors are pre-computed and embedded directly into the executable via the `code_vectors_blob.S` assembler file using `.incbin`. This eliminates the need for Python, PyTorch, or external model files at runtime, allowing `pretrained_vec_at()` to perform pure in-memory lookups from the static binary section.

### What is the vector dimension and precision?

Vectors are 768-dimensional (`PRETRAINED_DIM = 768`) and stored as int8 quantized values (range -127 to 127). At runtime, [`src/semantic/semantic.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/semantic/semantic.c) promotes these to float32 and applies L2-normalization before computing cosine similarity.

### Can I adjust the weight of semantic embeddings versus lexical signals?

Yes. The `CBM_SEM_W_RI` constant defines the weight of the Random Indexing/embedding signal within `cbm_sem_combined_score`. You can also tune the proximity boost via `CBM_SEM_PROX_MAX_BOOST` and disable semantic search entirely using the `CBM_SEMANTIC_ENABLED` environment variable.