How Semantic Search Works in Codebase-Memory-MCP with Bundled Nomic Embeddings

Codebase-Memory-MCP implements fully-local semantic search by compiling 768-dimensional int8-quantized Nomic embeddings directly into the binary, enabling zero-dependency cosine similarity matching against enriched document vectors without external API calls.

Codebase-Memory-MCP is a local-first code search engine that embeds semantic understanding directly into its static binary. Unlike cloud-dependent alternatives, this repository implements semantic search with bundled Nomic embeddings by baking the nomic-embed-code model into the executable at compile time, eliminating runtime network requirements entirely.

Embedding the Nomic Model into the Binary

The system uses a two-phase approach to embed the nomic-embed-code model (40,000 tokens, 768 dimensions) directly into the compiled artifact.

Offline Model Extraction

The Python script scripts/extract_nomic_vectors.py downloads the Hugging Face model nomic-ai/nomic-embed-code and prepares two compile-time artifacts:

• vendored/nomic/code_tokens.h – Contains the token vocabulary (approximately 40,000 tokens)
• vendored/nomic/code_vectors.bin – Contains int8-quantized 768-dimensional vectors for every token

These files are then transformed into an assembler blob at vendored/nomic/code_vectors_blob.S for static linking.

Compile-Time Binary Inclusion

The build system embeds these vectors as a static data section. In Makefile.cbm, the UNIXCODER_BLOB_SRC variable points to vendored/nomic/code_vectors_blob.S, which is linked into the final binary. At runtime, the executable contains the complete embedding table as a built-in data section, requiring no external model files.

Runtime Semantic Search Pipeline

Lazy Token-to-Index Mapping

When the semantic module first initializes, semantic.c constructs an in-memory hash table via ensure_pretrained_map(). This double-checked locking mechanism maps token strings to their integer indices in the pretrained table, ensuring thread-safe lazy initialization without startup overhead.

Query Tokenization and Vector Lookup

The function cbm_sem_tokenize() in src/semantic/semantic.c splits user queries into tokens, expanding common abbreviations (e.g., "err" → "error"). For each token, cbm_sem_random_index() performs a lookup:

1. Check the pretrained map for the token
2. If found, read int8 values from pretrained_vec_at(idx) and de-quantize by dividing by CBM_SEM_INT8_MAX
3. If absent, generate a sparse random vector using xxHash-based hashing

The vectors are summed and normalized via cbm_sem_normalize() to produce a single 768-dimensional query vector.

Document Vector Enrichment

During indexing, each source file becomes a corpus entry where tokens are transformed into dense vectors using the same pretrained lookup. These vectors undergo Random Indexing, co-occurrence context analysis, and Reflective Random Indexing (RRI) enrichment, storing the final result in corpus_entry_t.enriched_vec.

Computing Semantic Similarity

Semantic scoring occurs in run_semantic_query() within src/mcp/mcp.c. The system computes cosine similarity between the query vector and each document's enriched vector using cbm_sem_cosine(). This cosine score becomes one of eleven signals combined for final ranking, merging semantic similarity with TF-IDF, structural matching, and API signatures.

Practical Implementation Examples

CLI Semantic Search

Invoke semantic search via the command line:

search_graph --semantic_query send publish

This translates to a JSON request containing the semantic query array, as validated in tests/test_cli.c.

Building Query Vectors Programmatically

cbm_sem_vec_t qvec = {0};
const char *keywords[] = {"send", "publish"};

/* Tokenize and accumulate vectors */
for (int i = 0; i < 2; ++i) {
    char *tokens[32];
    int cnt = cbm_sem_tokenize(keywords[i], tokens, 32);
    for (int t = 0; t < cnt; ++t) {
        cbm_sem_random_index(tokens[t], &qvec);
        free(tokens[t]);
    }
}

/* Normalize for cosine comparison */
cbm_sem_normalize(&qvec);

Computing Cosine Similarity

float score = cbm_sem_cosine(&qvec, &doc_entry->enriched_vec);

Key Source Files

• src/semantic/semantic.c – Core implementation of tokenization, vector lookup, and normalization
• src/semantic/semantic.h – API declarations and CBM_SEM_DIM = 768 constant
• vendored/nomic/code_tokens.h – 40K token vocabulary
• vendored/nomic/code_vectors_blob.S – Assembler blob embedding the quantized vectors
• scripts/extract_nomic_vectors.py – Offline model extraction script
• src/mcp/mcp.c – MCP JSON API integration via run_semantic_query()

Summary

• Fully-local execution: The nomic-embed-code model (768-dimensional, int8-quantized) compiles directly into the binary via code_vectors_blob.S, eliminating API keys and network dependencies
• Lazy initialization: The ensure_pretrained_map() function builds the token-to-index hash table on first use with thread-safe double-checked locking
• Hybrid vector generation: cbm_sem_random_index() falls back to xxHash-based random vectors for out-of-vocabulary tokens while using pretrained Nomic embeddings for known tokens
• Enriched document representation: Corpus entries combine pretrained embeddings with Random Indexing and Reflective Random Indexing in corpus_entry_t.enriched_vec
• Cosine similarity scoring: The cbm_sem_cosine() function compares query vectors against enriched document vectors as one of eleven ranking signals

Frequently Asked Questions

How does Codebase-Memory-MCP handle tokens not in the Nomic vocabulary?

When cbm_sem_random_index() encounters an unknown token, it generates a deterministic sparse random vector using xxHash-based hashing. This ensures every query term receives a vector representation, preventing vocabulary gaps from causing search failures while maintaining the distributional properties of the embedding space.

Why use int8 quantization for the Nomic embeddings?

The embeddings use int8 quantization to minimize binary size while preserving semantic quality. During runtime, values are de-quantized by dividing by CBM_SEM_INT8_MAX before vector arithmetic. This compression reduces memory footprint without requiring external model loading, keeping the entire search system self-contained in a single executable.

What is Reflective Random Indexing (RRI) in document enrichment?

RRI augments the pretrained Nomic vectors with co-occurrence statistics and contextual information captured during corpus indexing. As implemented in src/semantic/semantic.c, this technique enriches the static embeddings with dynamic corpus-specific semantics, storing the result in corpus_entry_t.enriched_vec to improve matching accuracy for domain-specific terminology.

Does semantic search require external dependencies or API keys?

No. Because the Nomic embeddings are compiled into the binary through vendored/nomic/code_vectors_blob.S and referenced via pretrained_vec_at(), the system performs all vector operations locally. The Makefile.cbm build process embeds the 40,000-token vocabulary and 768-dimensional vectors as static data, enabling instant semantic search without Docker containers, network requests, or configuration files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}