How DeusData/codebase-memory-mcp Implements Semantic Search and Embeddings for Code

DeusData/codebase-memory-mcp implements semantic search and embeddings through a hybrid Python-to-C pipeline that extracts 768-dimensional vectors from the nomic-embed-code transformer, quantizes them to int8, and embeds them as a static binary blob for zero-dependency runtime cosine similarity calculations.

The DeusData/codebase-memory-mcp project (also referred to as Instagit) provides a deterministic, offline-capable semantic search and embeddings system for source code analysis. Unlike traditional vector databases that require heavy runtime dependencies, this implementation compiles transformer-based embeddings directly into the executable, enabling fast in-memory lookups without Python or GPU requirements.

Embedding Extraction Pipeline in Python

The foundation rests on scripts/extract_nomic_vectors.py, which orchestrates the one-time generation of pre-computed embeddings from the transformer model.

Token Filtering and Model Inference

The script loads the nomic-embed-code transformer (a 7B parameter model) and processes only code-relevant tokens from its vocabulary. The is_code_relevant and clean_token functions filter the vocabulary to retain identifier-shaped strings such as myVar and load_data, discarding natural language and special tokens. Inference runs in configurable batches (BATCH_SIZE) across CPU, CUDA, or MPS devices.

Vector Processing and Quantization

Raw hidden-state vectors undergo mean-pooling and truncation to OUTPUT_DIM = 768. The pipeline applies L2-normalization and mean-centering to reduce anisotropy, followed by optional refinement via simulated_attention. Finally, the float32 vectors are quantized to int8 (range -127 to 127) to minimize binary size while preserving cosine similarity accuracy.

C Header Generation

The extraction script generates three critical artifacts in vendored/nomic/:

• code_vectors.h: Declares PRETRAINED_TOKEN_COUNT, PRETRAINED_DIM, and the pretrained_vec_at(i) accessor
• code_tokens.h: Defines PRETRAINED_TOKENS[], a static string array mapping indices to token strings
• code_vectors_blob.S: An assembler file embedding the binary blob via .incbin

Static C Runtime Integration

The runtime implementation eliminates external file dependencies by embedding vectors directly into the executable.

Binary Blob Embedding

The vendored/nomic/code_vectors_blob.S file uses assembly directives to include code_vectors.bin as a raw data section within the compiled binary. This ensures that pretrained_vec_at(i) returns a pointer to int8 data without filesystem access, supporting fully static linking.

Runtime Lookup Interface

Client code includes vendored/nomic/code_vectors.h to access the embedding table. The pretrained_vec_at function provides O(1) indexed access to the quantized vectors, while PRETRAINED_TOKENS enables linear search for string-to-index resolution.

Semantic Scoring Engine

The src/semantic/semantic.c module implements the actual search logic, combining transformer embeddings with traditional information retrieval signals.

Tokenization Strategy

The cbm_sem_tokenize function splits identifiers using delimiters and camel-case boundaries, converting tokens to lowercase. This normalization ensures that camelCase and camel_case map to the same semantic representations.

Multi-Signal Similarity Computation

The cbm_sem_combined_score function aggregates multiple complementary signals:

• Embedding signal: Cosine similarity between L2-normalized float vectors (de-quantized from int8), weighted by CBM_SEM_W_RI
• TF-IDF, Random Indexing, MinHash: Traditional lexical signals
• API signatures: Structural code patterns

Proximity boosts (CBM_SEM_PROX_MAX_BOOST) increase scores for symbols appearing in the same source file.

Configuration and Tuning

Environment variables control runtime behavior:

• CBM_SEMANTIC_ENABLED: Toggle the entire semantic pipeline
• CBM_SEMANTIC_THRESHOLD: Configure similarity cutoff values

Practical Implementation Examples

Generating the embedding lookup (run once):

# Install required Python packages

pip install torch transformers sentence-transformers

# Extract vectors and generate C headers

python scripts/extract_nomic_vectors.py --output-dir vendored/nomic

Using the embeddings from C:

#include "nomic/code_vectors.h"
#include "semantic/semantic.h"

/* Find the index of a token string */
int token_index(const char *tok) {
    for (int i = 0; i < PRETRAINED_TOKEN_COUNT; ++i) {
        if (strcmp(PRETRAINED_TOKENS[i], tok) == 0) return i;
    }
    return -1;
}

/* Retrieve a normalised float vector for a token */
void get_token_vec(const char *tok, float out[PRETRAINED_DIM]) {
    int idx = token_index(tok);
    if (idx < 0) { memset(out, 0, sizeof(float)*PRETRAINED_DIM); return; }

    const int8_t *q = pretrained_vec_at(idx);
    for (int d = 0; d < PRETRAINED_DIM; ++d) {
        out[d] = (float)q[d] / 127.0f;
    }
    
    // L2-normalise
    float norm = 0.0f;
    for (int d = 0; d < PRETRAINED_DIM; ++d) norm += out[d]*out[d];
    norm = sqrtf(norm);
    for (int d = 0; d < PRETRAINED_DIM; ++d) out[d] /= norm;
}

Computing semantic similarity between identifiers:

float semantic_similarity(const char *a, const char *b) {
    float vec_a[PRETRAINED_DIM], vec_b[PRETRAINED_DIM];
    get_token_vec(a, vec_a);
    get_token_vec(b, vec_b);

    float dot = 0.0f;
    for (int i = 0; i < PRETRAINED_DIM; ++i) dot += vec_a[i] * vec_b[i];
    return dot;  // Cosine similarity in [-1, 1]
}

Summary

• Hybrid architecture: Python handles heavy transformer inference once; C handles runtime similarity with zero dependencies
• Space-efficient storage: Float32 vectors are quantized to int8 and embedded as binary blobs via assembly .incbin in code_vectors_blob.S
• Multi-signal approach: Combines nomic-embed-code vectors with TF-IDF, Random Indexing, and MinHash in cbm_sem_combined_score
• Configurable runtime: Environment variables CBM_SEMANTIC_ENABLED and CBM_SEMANTIC_THRESHOLD control activation and cutoffs
• File-based proximity: CBM_SEM_PROX_MAX_BOOST favors local symbol relationships within the same file

Frequently Asked Questions

What embedding model does DeusData/codebase-memory-mcp use?

The system uses the nomic-embed-code transformer, a 7B parameter model specifically trained for code understanding. The scripts/extract_nomic_vectors.py script performs full inference on the model's vocabulary and exports the hidden states as quantized vectors using is_code_relevant filtering and simulated_attention refinement.

How does the system achieve zero-dependency runtime operation?

All embedding vectors are pre-computed and embedded directly into the executable via the code_vectors_blob.S assembler file using .incbin. This eliminates the need for Python, PyTorch, or external model files at runtime, allowing pretrained_vec_at() to perform pure in-memory lookups from the static binary section.

What is the vector dimension and precision?

Vectors are 768-dimensional (PRETRAINED_DIM = 768) and stored as int8 quantized values (range -127 to 127). At runtime, src/semantic/semantic.c promotes these to float32 and applies L2-normalization before computing cosine similarity.

Can I adjust the weight of semantic embeddings versus lexical signals?

Yes. The CBM_SEM_W_RI constant defines the weight of the Random Indexing/embedding signal within cbm_sem_combined_score. You can also tune the proximity boost via CBM_SEM_PROX_MAX_BOOST and disable semantic search entirely using the CBM_SEMANTIC_ENABLED environment variable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}