How to Configure Embedding Providers for Vector Similarity Search in ai-memory

To configure embedding providers in ai-memory, add an [embedder] table to your config.toml specifying the provider name, model, and credentials, then start the server to enable automatic vector generation for semantic search.

The akitaonrails/ai-memory project implements vector similarity search by storing dense embeddings for every page when an embedder is configured. Setting up this capability requires declaring your provider in the configuration file and ensuring the appropriate environment variables are available at runtime. Once active, the system automatically generates embeddings on writes and supports hybrid search combining vector similarity with FTS5 and graph results.

Configure the Embedding Provider in config.toml

Add an [embedder] table to your config.toml (or config.default.toml) to declare which provider to use. The table requires the provider name, the specific model identifier, and relies on provider-specific environment variables for authentication.

OpenAI Configuration

For OpenAI embeddings, export OPENAI_API_KEY and specify the model in your configuration:

[embedder]
provider = "openai"
model = "text-embedding-ada-002"

Google Vertex AI Configuration

Google's Vertex AI embedder requires GOOGLE_APPLICATION_CREDENTIALS pointing to a service-account JSON file:

[embedder]
provider = "google"
model = "textembedding-gecko@001"

Ollama Local Configuration

For local development with Ollama, set OLLAMA_BASE_URL (defaulting to http://localhost:11434) and choose any locally-served model:

[embedder]
provider = "ollama"
model = "nomic-embed-text"

# Optional: dim = 1024  # overrides model-reported dimension

Runtime Initialization and Factory Pattern

According to the akitaonrails/ai-memory source code, the embedder is constructed via the factory in crates/ai-memory-llm/src/factory.rs. During server startup, the system calls:

let embedder = make_embedder(&config.embedder)?;   // returns Arc<dyn Embedder>
let state = AdminState::new(...).with_embedder(embedder);

If no [embedder] table exists, the embedder remains None, causing vector-dependent endpoints to return 503 Service Unavailable (as implemented in crates/ai-memory-mcp/src/admin.rs).

Automatic Embedding on Page Writes

Once configured, ai-memory automatically generates vectors during content creation. In crates/ai-memory-wiki/src/wiki.rs, the Wiki::write_page method checks for an active embedder:

if let Some(embedder) = &self.embedder {
    let vec = embedder.embed_document(&final_body).await?;
    store_embedding(page_id,
        embedder.provider().to_string(),
        embedder.model().to_string(),
        embedder.dim(),
        vec).await?;
}

This ensures every new or updated page receives an embedding without manual intervention.

Back-filling Existing Pages

To generate embeddings for pages created before configuration, use the /admin/embed endpoint implemented in crates/ai-memory-mcp/src/admin.rs. This endpoint iterates over pages lacking embeddings and invokes embed_document, respecting the provider/model/dim triple and deleting stale rows when models change (via store::writer.rs::delete_stale_page_embeddings).

Trigger a back-fill using curl:

curl -X POST http://127.0.0.1:49374/admin/embed \
     -H "Content-Type: application/json" \
     -d '{"reembed":true,"dry_run":false}'

Hybrid Vector Similarity Search

The query endpoint (/admin/search with query_vec=true) performs hybrid retrieval by constructing query embeddings and running Reciprocal Rank Fusion (RRF) over three sources: FTS5 text matches, entity-matching rows, and stored page embeddings using cosine similarity. As seen in crates/ai-memory-mcp/src/server.rs (lines 1520-1530), vector scoring executes only when an embedder is configured; otherwise, the system falls back to pure FTS5, entity, and graph scoring.

Example hybrid search request:

curl -X POST http://127.0.0.1:49374/admin/search \
     -H "Content-Type: application/json" \
     -d '{"query":"machine learning", "query_vec":true, "limit":10}'

Programmatic Configuration

For custom binaries, construct the embedder manually and attach it to the Wiki instance:

use std::sync::Arc;
use ai_memory_llm::{Embedder, openai::OpenAIEmbedder};
use ai_memory_wiki::Wiki;

let embedder: Arc<dyn Embedder> = Arc::new(OpenAIEmbedder::new(
    std::env::var("OPENAI_API_KEY")?,
    "text-embedding-ada-002".to_string(),
));

let wiki = Wiki::new(...).with_embedder(embedder);

Summary

• Add an [embedder] table to config.toml with provider, model, and optional dim keys to configure embedding providers for vector similarity search in ai-memory.
• Export provider-specific environment variables (OPENAI_API_KEY, GOOGLE_APPLICATION_CREDENTIALS, or OLLAMA_BASE_URL) before starting the server.
• The factory in crates/ai-memory-llm/src/factory.rs instantiates the embedder; missing configuration results in HTTP 503 for vector operations.
• Page writes automatically generate embeddings via Wiki::write_page in crates/ai-memory-wiki/src/wiki.rs.
• Use the /admin/embed endpoint to back-fill existing content when enabling embeddings on existing databases.
• Hybrid queries in crates/ai-memory-mcp/src/server.rs combine vector similarity with FTS5 and graph search for comprehensive results.

Frequently Asked Questions

What happens if I don't configure an embedder in ai-memory?

If no [embedder] table is present in the configuration, the embedder remains uninitialized (None). Any request to /admin/embed or hybrid vector search endpoints will return HTTP 503, and the system will use only FTS5, entity, and graph-based retrieval without vector similarity.

Can I switch embedding models after data is already stored?

Yes, but changing the provider, model, or dimension invalidates existing embeddings. The /admin/embed endpoint detects model changes and calls delete_stale_page_embeddings to remove incompatible vectors before generating new ones with the updated configuration.

Does ai-memory support local embedding without external API calls?

Yes, the Ollama provider enables fully local vector generation. Configure provider = "ollama" with a local model like nomic-embed-text, ensure your Ollama server is running at OLLAMA_BASE_URL (default http://localhost:11434), and the system will generate embeddings without sending data to external services.

How does hybrid search rank results when combining vectors and text?

The system uses Reciprocal Rank Fusion (RRF) to combine scores from FTS5 text matches, entity relationships, and cosine similarity between stored embeddings and the query vector. This ensures semantic similarity and keyword relevance both influence the final ranking, with vector contributions only included when an embedder is active.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}